You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Logic to apply any job tracker updates for an ingest commit transaction based on the transaction and the table state.
This is to allow us to perform the tracker updates in somewhere following the transaction log.
Analysis
Deserialise add files transaction, give to transaction log head, and perform action on it based on the state before the transaction.
For failures, e.g. when an ingest job is run twice and the second commit should fail, we still want the tracker to be updated. We can commit the transaction to the log regardless of whether it is valid or not, then detect this at read time like we do for compactions.
We'll be removing the job tracker update from the state store committer, so we'll need to replicate the test coverage of that from StateStoreCommitterTest.
The text was updated successfully, but these errors were encountered:
Background
Description
Logic to apply any job tracker updates for an ingest commit transaction based on the transaction and the table state.
This is to allow us to perform the tracker updates in somewhere following the transaction log.
Analysis
Deserialise add files transaction, give to transaction log head, and perform action on it based on the state before the transaction.
For failures, e.g. when an ingest job is run twice and the second commit should fail, we still want the tracker to be updated. We can commit the transaction to the log regardless of whether it is valid or not, then detect this at read time like we do for compactions.
We'll be removing the job tracker update from the state store committer, so we'll need to replicate the test coverage of that from StateStoreCommitterTest.
The text was updated successfully, but these errors were encountered: