forknet: support arbitrary shard layouts #12938
Conversation
|
I know it is kind of difficult for reviewers to test this sorry... I have been testing w some ad hoc scripts, but next week in a separate PR maybe I can clean some of them up so testing is easier |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12938 +/- ##
==========================================
- Coverage 70.37% 70.26% -0.11%
==========================================
Files 854 855 +1
Lines 175421 176151 +730
Branches 175421 176151 +730
==========================================
+ Hits 123444 123780 +336
- Misses 46848 47223 +375
- Partials 5129 5148 +19
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| } | ||
| None => shard_layout.clone(), | ||
| }; | ||
|
|
||
| // Flat state can be at different heights for different shards. | ||
| // That is fine, we'll simply lookup state root for each . | ||
| let fork_heads = get_fork_heads(&all_shard_uids, store.clone())?; |
There was a problem hiding this comment.
I had to look into the function get_fork_heads, is a misleading name 🤣
| epoch_length, | ||
| num_seats, | ||
| block_height, | ||
| flat_head.height + 1, |
There was a problem hiding this comment.
Can you clarify why we need to add +1?
There was a problem hiding this comment.
I honestly don't know.. I bet it doesn't really matter too much. But here I was just making sure to keep the old behavior because in the current code, we do let block_height = block_height + 1; before writing it to FORK_TOOL_BLOCK_HEIGHT
| }; | ||
|
|
||
| let mut link_allowances = Vec::new(); | ||
| // TODO: maybe do something other than this |
There was a problem hiding this comment.
A few options:
- compute average allowance from all links
- use max allowance every time
| Some(root) => commit_to_existing_state(shard_tries, shard_uid, root, updates)?, | ||
| None => { | ||
| let state_root = commit_to_new_state(shard_tries, shard_uid, updates)?; | ||
| // TODO: load memtrie |
There was a problem hiding this comment.
What's the purpose of loading memtrie here?
There was a problem hiding this comment.
I think it will all just be a lot faster if there ends up being lots of updates to that shard. thats why right now this fork-network command loads memtries for all shards before it does all the rewriting so itll be faster (although it uses an insane amount of memory)
| struct InitCmd { | ||
| /// If given, the shard layout in this file will be used to generate the forked genesis state | ||
| #[arg(long)] | ||
| pub shard_layout_file: Option<PathBuf>, |
There was a problem hiding this comment.
In practice, this shard layout file, is it supposed to be a json file with the same content of the field shard_layout in genesis?
There was a problem hiding this comment.
Yup, just copy pasting it to a new file should work
#12938 changed the format of the data we write to the `FORK_TOOL` Misc column keys in the `init` command, but didn't keep backward compatibility. But this should have been done because we need to run `fork-network` again when starting a forknet so we can run the `set-validators` and `finalize` commands. So this PR reads the values the old way if there's nothing in the `FORK_TOOL_FLAT_HEAD` key
We'd like to use forknet with more shards than we have currently on mainnet, or just any arbitrary shard layout. But it won't work to just change the shard layout in the genesis file, since we need to rewrite everything in the tries. So this PR implements support for an optional custom shard layout when generating the state.
To do this, we use the existing
StorageMutatorthat keeps a list of updates per shard, and we change it to be aware of which shards belong to the source shard layout and which belong to the target. We still have one thread per source shard, but now it writes state to shards that possibly don't exist in the shard layout.This also requires writing a new bandwidth scheduler state in each target shard that doesn't exist in the source shard layout