Load CDK: CheckpointManager support for Index-Based Checkpointing #53663
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/indexed-based-bookkeeping
branch
from
7b4f628 to
6069fa9
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
from
8754a59 to
ded3918
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/indexed-based-bookkeeping
branch
from
6069fa9 to
5714081
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
2 times, most recently
from
4c5afe6 to
ebe0036
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/indexed-based-bookkeeping
branch
from
5714081 to
c5d56a4
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
from
ebe0036 to
89ffc87
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/indexed-based-bookkeeping
branch
from
c5d56a4 to
c34e0df
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
from
89ffc87 to
6c8c579
Compare
edgao
reviewed
Feb 18, 2025
airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/state/CheckpointManager.kt
Outdated
Show resolved
Hide resolved
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/indexed-based-bookkeeping
branch
3 times, most recently
from
3570a29 to
d20d91d
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
from
6c8c579 to
a8575e3
Compare
edgao
approved these changes
Feb 19, 2025
| @@ -155,7 +164,13 @@ abstract class StreamsCheckpointManager<T> : CheckpointManager<DestinationStream | |||
| val head = globalCheckpoints.peek() | |||
| val allStreamsPersisted = | |||
| head.streamIndexes.all { (stream, index) -> | |||
| syncManager.getStreamManager(stream).areRecordsPersistedUntil(index) | |||
| if (!checkpointById) { | |||
There was a problem hiding this comment.
nit: invert this if statement (i.e. if (checkpointById) { ... CheckpointId()... } else {...})
| ======= | ||
| * True if all records in the stream have been marked as completed AND the stream has been marked as | ||
| * complete. | ||
| >>>>>>> 3570a293f9b (addressed comments) |
There was a problem hiding this comment.
looks like a borked git merge
There was a problem hiding this comment.
yeah fixed it locally
Base automatically changed from
jschmidt/load-cdk/indexed-based-bookkeeping
to
master
February 19, 2025 21:06
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
from
a8575e3 to
4261c93
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
from
4261c93 to
70a91da
Compare
johnny-schmidt
force-pushed
the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
from
70a91da to
2bf1b3e
Compare
johnny-schmidt
deleted the
jschmidt/load-cdk/id-bookkeeping-ckpt-manager
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #53646. This adds the corresponding support in the checkpoint manager for index-based accounting.