feat(master): async pruning of orphan nodes #877
Conversation
Co-authored-by: Julien Robert <[email protected]>
Co-authored-by: mmsqe <[email protected]> Co-authored-by: Julien Robert <[email protected]>
Co-authored-by: Julien Robert <[email protected]>
Co-authored-by: khanh-notional <[email protected]>
Co-authored-by: Elias Naur <[email protected]>
…806) (#821) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marko Baricevic <[email protected]>
#822) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#805) (#825) Co-authored-by: Matt, Park <[email protected]>
Co-authored-by: Marko Baricevic <[email protected]>
Co-authored-by: cool-developer <[email protected]>
…n error (backport #843) (#844) Co-authored-by: Emmanuel T Odeke <[email protected]>
Co-authored-by: cool-developer <[email protected]>
…er (backport #846) (#847) Co-authored-by: Emmanuel T Odeke <[email protected]>
This reverts commit d8c630d.
WalkthroughThe recent updates focus on enhancing performance, ensuring thread safety, and improving code clarity. Key changes include upgrading the Go version in the CI workflow, introducing thread safety measures in the Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
czarcas7ic
changed the title
Co-authored-by: cool-developer <[email protected]>
cool-develope
changed the title
| legacyRootKeyFormat.Scan(itr.Key(), &curVersion) | ||
| rootKeys = append(rootKeys, itr.Key()) | ||
| if prevVersion > 0 { | ||
| if err := ndb.traverseOrphans(prevVersion, curVersion, func(orphan *Node) error { |
There was a problem hiding this comment.
@cool-develope Isn't it true that v1 does not store orphans explicitly, and identifies them by traversing trees at version n and n+1? This being the case, why can deleteLegacyVersions not simply iterate on the orphan key prefix in leveldb and delete all of them? Am I missing something?
There was a problem hiding this comment.
yes, it can simple iterate orphan values
| @@ -46,6 +49,9 @@ func (b *BatchWithFlusher) estimateSizeAfterSetting(key []byte, value []byte) (i | |||
| // the batch is flushed to disk, cleared, and a new one is created with buffer pre-allocated to threshold. | |||
| // The addition entry is then added to the batch. | |||
| func (b *BatchWithFlusher) Set(key, value []byte) error { | |||
| b.mtx.Lock() | |||
There was a problem hiding this comment.
If each batch accumulates in memory before flushing to disk doesn't it make sense to lock at write instead of Set/Delete?
| counter++ | ||
| if counter == 1000 { | ||
| counter = 0 | ||
| time.Sleep(1000 * time.Millisecond) |
There was a problem hiding this comment.
Is this sleep to create a write gap so that SaveVersion doesn't block too frequently from the newly introduced mutex in BatchWithFlusher while a long prune operation is ongoing?
If deleteLegacyVersions is only called once on migration I guess it's OK, but this sleep is hard to understand and could result in unexpected wait times from the main I/o thread. I'm also curious about the overhead of mutex/set release on every Set/Delete call.
|
@czarcas7ic could you please wire up this PR? only encoding and fastKeyFormatter features |
|
@cool-develope Just verifying, you only want the FastPrefixFormatter change as well as encoding.go change, and remove everything else? My apologies, I am between multiple tasks right now so am losing context on this. |
we can close it, -> #923 |
|
Thanks, my apologies for the extra overhead that this may have caused! |
This feature prevents the multiple hour long waiting period when chains upgrade from previous versions of IAVL to IAVL v1. The time comes from pruning orphan nodes. This synchronously prunes them. This branch has been NOT been tested against osmosis mainnet, however the v1.x.x version of this PR has.