Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply ingest job tracker update against state from transaction #4253

Closed
patchwork01 opened this issue Feb 13, 2025 · 0 comments · Fixed by #4257
Closed

Apply ingest job tracker update against state from transaction #4253

patchwork01 opened this issue Feb 13, 2025 · 0 comments · Fixed by #4257
Assignees

Comments

@patchwork01
Copy link
Collaborator

patchwork01 commented Feb 13, 2025

Background

Description

Logic to apply any job tracker updates for an ingest commit transaction based on the transaction and the table state.
This is to allow us to perform the tracker updates in somewhere following the transaction log.

Analysis

Deserialise add files transaction, give to transaction log head, and perform action on it based on the state before the transaction.

For failures, e.g. when an ingest job is run twice and the second commit should fail, we still want the tracker to be updated. We can commit the transaction to the log regardless of whether it is valid or not, then detect this at read time like we do for compactions.

We'll be removing the job tracker update from the state store committer, so we'll need to replicate the test coverage of that from StateStoreCommitterTest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants