Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-evaluate possible solutions for persistent storage in wasm #518

Open
zvolin opened this issue Jan 21, 2025 · 1 comment
Open

Re-evaluate possible solutions for persistent storage in wasm #518

zvolin opened this issue Jan 21, 2025 · 1 comment
Labels
node-wasm Related to `lumina-node-wasm`

Comments

@zvolin
Copy link
Member

zvolin commented Jan 21, 2025

Lumina in browser uses indexeddb api for persistent storage and our schema looks like this:headers table

hash: string // hex, has index
height: u64 // has index
header: bytes // header encoded with protobuf

header ranges table (has only 2 entries, one for daser, one for syncer)

ranges: [(u64, u64)] // array of pairs (from_height, to_height)

sampling metadata table (stores cids that will need to be pruned):

height: u64
status: Unknown | Accepted | Rejected
cids: [bytes]

blockstore table:

cid: bytes
data: bytes

We've measured mainnet using chromium and firefox. There seem to be a big difference between those 2 browsers, firefox storage usage is a few times bigger. They use different databases under the hood, firefox goes with sqlite and chromium uses leveldb.1 hour of syncing + dasing:
chrome: ~35.7MB, ~103x103 square
firefox: ~178MB, ~107x107 square24 hours of syncing + dasing:
chrome: ~825.5MB, ~93x93 square
firefox: ~3.6GB, ~93x93 square1h but without storing signatures and validator set:
chrome: ~19.4MB, -45%
firefox: ~87.5MB, -50%

@zvolin
Copy link
Member Author

zvolin commented Jan 21, 2025

There's a huge discrepancy between firefox and chrome indexeddb disk usage. We should re-check all available solutions and see if there's something that works better than idb

@zvolin zvolin added the node-wasm Related to `lumina-node-wasm` label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
node-wasm Related to `lumina-node-wasm`
Projects
None yet
Development

No branches or pull requests

1 participant