use snappy-framed format for compressing bellatrix+ database entries #3551
Conversation
| db.blocks[BeaconBlockFork.Phase0].get(key.data, decode).expectDb() and success or | ||
| db.v0.getPhase0BlockSSZ(key, data) | ||
|
|
||
| proc getBlockSZ*( |
There was a problem hiding this comment.
Why are getBlockSZ(phase0) and getBlockSZ(altair) separate functions? Bellatrix differs in that it does assign rather than snappy.decode.
There was a problem hiding this comment.
phase0 reads from an additional table.. yeah, there's better ways to factor the code I'm sure but ..
| # TODO rollback is needed to deal with bug - use `noRollback` to ignore: | ||
| # https://github.com/nim-lang/Nim/issues/14126 | ||
| # TODO RVO is inefficient for large objects: | ||
| # https://github.com/nim-lang/Nim/issues/13879 |
There was a problem hiding this comment.
Is RVO still inefficient for large objects? That Nim issue is closed.
There was a problem hiding this comment.
ah, yes.. there's nim-lang/Nim#15571 now instead
There was a problem hiding this comment.
longer story would be that to remove the TODO, one would need to examine the C code again and see what's going on - issue might be closed but that doesn't mean it's actually fixed for our use case
arnetheduck
force-pushed
the
snappy-frame-db
branch
2 times, most recently
from
80e7f9f to
23f9abf
Compare
arnetheduck
changed the title
`.era` files and Req/Resp protocols use framed formats - aligning the database with these makes for less recompression work overall as gossip is sent only once while req/resp repeats (potentially) - this also allows efficient pruning-to-era where snappy-recompression is the major cycle thief.
arnetheduck
force-pushed
the
snappy-frame-db
branch
from
19dddbc to
05fd6d8
Compare
.erafiles and Req/Resp protocols use framed formats - aligning thedatabase with these makes for less recompression work overall as gossip
is sent only once while req/resp repeats (potentially) - this also
allows efficient pruning-to-era where snappy-recompression is the major
cycle thief.