Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add create_or_replace_document_objects method to weaviate provider #36177

Merged
merged 36 commits into from
Dec 23, 2023
Merged

Add create_or_replace_document_objects method to weaviate provider #36177

merged 36 commits into from
Dec 23, 2023
+699 −11

Conversation

utkarsharma2
Copy link
Contributor

When inserting the data into weaviate database, it's common to override the existing data, if users go about doing that themselves, there are lots of cases and API failures that need to be handled. This PR aims to provide a helper function that can handle all cases.

mpgreg
mpgreg suggested changes Dec 12, 2023
View reviewed changes
Copy link

@mpgreg mpgreg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Still need to address upsert needs and also error handling. See comments.

sunank200
sunank200 suggested changes Dec 13, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the ineFor ingesting data, it should determine objects to insert, objects to delete and objects that remain unchanged for each chunk. This would simplify the logic.

@utkarsharma2 utkarsharma2 changed the title Add create_or_replace_objects method to weaviate provider Add create_or_replace_document_objects method to weaviate provider Dec 20, 2023
sunank200
sunank200 reviewed Dec 20, 2023
View reviewed changes
josh-fell
josh-fell reviewed Dec 21, 2023
View reviewed changes
sunank200
sunank200 approved these changes Dec 22, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@utkarsharma2 utkarsharma2 merged commit 75d74b1 into apache:main Dec 23, 2023
52 checks passed
@utkarsharma2 utkarsharma2 deleted the WeaviateIngest branch December 23, 2023 06:15
This was referenced Dec 28, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@potiuk potiuk potiuk approved these changes

@sunank200 sunank200 sunank200 approved these changes

@josh-fell josh-fell josh-fell approved these changes

@phanikumv phanikumv Awaiting requested review from phanikumv

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy

@pankajkoti pankajkoti Awaiting requested review from pankajkoti

@mpgreg mpgreg Awaiting requested review from mpgreg

Assignees
No one assigned
Labels
area:providers provider:weaviate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@utkarsharma2 @potiuk @sunank200 @mpgreg @josh-fell