feat: add snapshot walop #25755
Draft
feat: add snapshot walop #25755
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- The main change is to detach wal and snapshot, in a way all 3 of the
following things can happen
- flush the wal buffer only (already handled, before this commit)
- flush wal buffer and snapshot (already handled, before this commit)
- snapshot without flushing wal buffer (introduced in this commit)
This is achieved by introducing another method `snapshot` in
`WalFileNotifier` trait. The main dependency between wal and snapshot
is the `wal_file_number`, since this is tracked in `SnapshotTracker`
separately we can switch to using `SnapshotTracker`'s
`last_wal_sequence_number` instead of the one that comes through the
`WalContents`.
- A higher level background loop is introduced that checks the overall
table buffer size every `N` seconds and if it is greater than a
threshold (`X`) then it calls `snapshot` method. Both `N` and `X` are
configurable through cli. `N` defaults to 10s and `X` defaults to 70%
- Some refactoring of code so that existing methods can be reused when
only snapshotting
closes: #25685
praveen-influx
force-pushed
the
praveen/add-snapshot-walop
branch
from
ce49d73 to
280e40e
Compare
praveen-influx
force-pushed
the
praveen/add-snapshot-walop
branch
from
280e40e to
90a7261
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.