New historic state cache

[michaelsproul]

Proposed Changes

Attempt to unify the new HDiffBuffer cache with the old "historic state cache" by creating a cache that holds either or both kinds of states, and lazily converts between the formats as required.

Additional Info

Performance testing on mainnet with --hierarchy-exponents 5,7,11 shows much improved performance for sequential queries. Most queries after the first are served in 400-700ms.

[michaelsproul]

This is working quite well for the sequential query case. The tricky part is managing when to build caches. Previously we were re-building them for every state that required replay, which was obviously sub-optimal. Now we ensure that any state in the historic state cache has all caches built, which makes replay faster at the expense of slowing down the loading of snapshot/diff states.

On the test machine the loads for snapshot/diff states are now around 6-7s (ouch), but each subsequent state is more like 0.5-1s (on average 700ms).

I thought about moving the cache build from when the snapshot/diff state is loaded to the next slot, which would keep the total amortised time the same, but reduce the worst-case time for the diff state by several seconds (maybe from 6-7s down to 3s). It seems cache building only takes 1.8-2.5s in the worst case, so there's still some time going missing for the snapshot state.

[michaelsproul]

Block replay is quick now with built caches, although it's a bit slower at epoch boundaries:

Oct 11 06:27:33.922 DEBG Replayed blocks, replay_time_ms: 10, target_slot: 8999423, service: freezer_db, module: store::hot_cold_store:1865
Oct 11 06:27:35.536 DEBG Replayed blocks, replay_time_ms: 836, target_slot: 8999424, service: freezer_db, module: store::hot_cold_store:1865

In this commit (52a9066) I made it so that we replay 1 block in preference to applying a diff for states that would normally be constructed by diffing. This may be the wrong choice, as applying a diff should be ~150ms, and converting HDiffBuffer -> state should be only ~400ms or so, for 550ms total. Although this doesn't mesh with the benchmark results from prior to this commit, where diff states were taking around 4-6s! Maybe cloning the several hundred MB of hdiff buffer is taking more time than we think (could be good to measure this). Another advantage of favouring replay where possible (i.e. for sequential access) is that it minimises memory fragmentation.

[michaelsproul]

This was a regression from a looong time ago.

[michaelsproul]

I think this is ready for review.

Some further optimisations are described here, but I think current performance is now very good.

• More historic state cache optimisations #6485

[dapplion]

These two misses are different. Technically we got a hit as the cache is allowing us to skip some work. Maybe we can have a metric STORE_BEACON_HISTORIC_STATE_CACHE_RESULT with a label:

• hit_exact
• hit_replay
• miss

[michaelsproul]

I need to think about this some more. Will merge as-is for now