[WIP] Fix Merge Into Partial Updates with Global Index #13083
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
The USP of partial updates is that users don't have to specify all fields in the merge into command. However, with global index and partition path updates, merge into command will fail because the expectation is that full record is provided to
HoodieIndexUtils::mergeIncomingWithExistingRecordWithExpressionPayload
. This PR attempts to fix the behavior by doing partial merge and building full record i.e. get the merged record and then fill in the missing fields from existing record. The PR still uses record merger API in both the above method as well asHoodieMergedReadHandle#doMergedRead
. Ideally, we would want to use the filegroup reader inHoodieMergedReadHandle
instad of record merger. That's a larger refactoring and for now, maybe we can just error out with message that merge into partial updates are not supported with global index.Impact
Fix Merge Into Partial Updates with Global Index
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist