Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshotter: Improve updates #20796

Closed
wants to merge 4 commits into from
Closed

Conversation

holiman
Copy link
Contributor

@holiman holiman commented Mar 23, 2020

This PR was originally opened against the snapshot-5 branch over at @karalabe as karalabe#27 . That PR contains some graphs from running it.

Background:

When we execute block n , we execute transactions serially.
After tx 1, we call Finalize, which puts the address for each pending object into pending.
Similarly, we for each of those, we call obj.Finalize, which does a similar thing for each modified slot.

Idea

So, during finalize, we also send each addr off onto a channel (non-blocking). And on that channel, there's a little
precacher-thread which loads it from the trie and warms up the cache.

So the snapshotter allows us to parallelize the trie-reads instead of doing it serially. At the end of the block, when we finally reach the update-phase, the trie caches will be prepopulated already, and the update will be quick.

@holiman holiman changed the title Improve updates Snapshotter: Improve updates Mar 23, 2020
@holiman holiman force-pushed the improve_updates branch from 447b777 to 3ed1810 Compare May 7, 2020 10:00
@holiman
Copy link
Contributor Author

holiman commented May 7, 2020

I rebased this on master.
However, a pretty natural follow-up to this would be:

  1. Instead of just filling in a state trie that we eventually discard, we keep the state trie around. At a later stage, when the statedb wants to actually commit, it would ask the prefetcher : "do you have a trie with root X". And the prefetcher would hand over a mostly-filled trie.
  2. Instead of discarding N storage tries, we'd store them in a map, and hand over to the statedb for Commit.
  3. The prefetcher tries would be cleared out when the next block operations start happening.

Currently, the only thing we do is warm up some caches, but we still have to walk tries and unmarshal blobs. With that proposed change, we'd immediately obtain nearly-finished tries to operate on.

Sounds good?

@holiman holiman mentioned this pull request May 8, 2020
3 tasks
@holiman holiman closed this May 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants