VM: Diff-based Touched Accounts Checkpointing #2581

holgerd77 · 2023-03-13T08:57:45Z

Part of #1536

Extracted from #2578

PR switches to diff-based touched accounts checkpointing in the VM. This helps to solve the first of Shanghai dDoS attacks going into a recursive DELEGATECALL starting at block 2283416 (DAO HF) and repeated in subsequent block txs (with "normally" running blocks in between) and leading our client to crash with heap out of memory ("Javascript Ineffective mark-compacts near heap limit Allocation failed") (a bit in contrast to the attack in #1536 described by me as well as just stating the block execution getting very slow interestingly, not sure if we might have added additional (memory) weight here in the meantime. So this is on an 8GB Apple MacBook Air M1 machine) (I do know about this option to just increase the Node.js memory limit but I explicity did not want to just do that, seems not a "solution" to me).

Here is a screenshot (don't get confused by block-number-misleading "Latest local block" message):

When looking at the debug logs of the block/tx run by running:

DEBUG=ethjs,evm:*,evm:*:*,vm:*,vm:*:*,statemanager:* npm run client:start -- --discDns=false --discV4=false --maxPeers=0 --executeBlocks=2283416

one can see that hundreds of DELEGATECALLs are executed subsequently, each one happening from within the previous delegate call and each creating a new checkpoint without committing for 100s of delegate calls. With the previous checkpointing implementation for touched accounts this lead to the whole account structure being copied over and over again, even when there are no changes, leading to a lot of memory bloat.

Situation will get better at the Tangerine Whistle HF at block 2463000 increasing the gas costs for call operations and other I/O heavy opcodes (in case of DELEGATECALL from 40 to 700, so pretty substantially), so that such an attack is not that effective any more.

This is nevertheless a lasting and structural change improving the overall VM efficiency. The functionality (change) should be sufficiently covered by existing tests.

Note that this change alone is not solving the out-of-memory crash though yet, I will submit a subsequent PR with a separate cache improvement which will completely solve the issue. PR is nevertheless open for review and can be merged. 🙂

We have a structurally similar problem with the way we handle accessed storage checkpointing in the VM, which we should also tackle (also open for grasps, a bit more complex but also not too much, solution should be structurally very similar). For now I "solved" this for the current context I am working on by guarding the cache to just get used for Berlin HF and upwards, which generally makes sense since before it is just an unnecessarily build-up structure.

Small third change in this PR is to use the direct memory access (see #2573 for more context) for the MLOAD opcode, which should be very much safe since this will be converted to BigInt directly afterwards anyhow. So this just saves an unnecessary in-between copy.

holgerd77 · 2023-03-13T09:11:43Z

Ah, interestingly some selected tests are failing here. Would be glad on some support on this (during the week would be enough), if someone has some ressources to spare (@jochem-brouwer or @ScottyPoi e.g.?).

codecov · 2023-03-13T09:11:54Z

Codecov Report

Merging #2581 (a9bab81) into master (cbf9a04) will decrease coverage by 0.47%.
The diff coverage is 97.39%.

Additional details and impacted files

Flag	Coverage Δ
block	`90.13% <ø> (ø)`
blockchain	`90.40% <ø> (ø)`
client	`?`
common	`?`
devp2p	`91.56% <ø> (-0.20%)`	⬇️
ethash	`∅ <ø> (∅)`
evm	`79.31% <100.00%> (ø)`
rlp	`∅ <ø> (?)`
statemanager	`89.61% <ø> (ø)`
trie	`90.36% <ø> (ø)`
tx	`94.16% <ø> (ø)`
util	`?`
vm	`84.53% <97.36%> (+0.66%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

jochem-brouwer · 2023-03-13T09:14:13Z

Will take a look this afternoon! 😄

jochem-brouwer · 2023-03-13T09:31:59Z

packages/vm/src/eei/vmState.ts

@@ -61,8 +70,9 @@ export class VmState implements EVMStateAccess {
   * Partial implementation, called from the subclass.
   */
  async checkpoint(): Promise<void> {
-    this._touchedStack.push(new Set(Array.from(this._touched)))
-    this._accessedStorage.push(new Map())
+    if (this._common.gteHardfork(Hardfork.Berlin)) {


I think it is cleaner if we check for EIP 2929 active here

jochem-brouwer · 2023-03-13T15:23:21Z

packages/vm/src/eei/vmState.ts

@@ -286,6 +325,7 @@ export class VmState implements EVMStateAccess {
      }
    }
    this._touched.clear()
+    this._touchedStack = {}


The touchedStack is not empty for some reason here, which makes me not super confident that this is all correct. After running npm run test:blockchain -- --fork=Istanbul --test=RevertPrecompiledTouch_d0g0v0_Istanbul there are still fields at the 0 and 2 keys.

jochem-brouwer · 2023-03-13T18:06:51Z

I think that the current approach is not correct @holgerd77. Here is a situation:

Tx calls a contract. This contract touches the address A. Now, the contract calls itself again (checkpoint), it now touches A again, and now this call frame reverts. This deletes A from the touched accounts. However, it is clear that A should be touched. After finishing the Tx, A should be checked and if it is empty, delete the account from the state.

jochem-brouwer · 2023-03-13T18:14:58Z

I think in order to fix:

Remember at which height the account was touched first
If a frame reverts, only delete accounts from _touched if the current height matches the height when it was first touched.

holgerd77 · 2023-03-14T08:30:26Z

I think that the current approach is not correct @holgerd77. Here is a situation:

Tx calls a contract. This contract touches the address A. Now, the contract calls itself again (checkpoint), it now touches A again, and now this call frame reverts. This deletes A from the touched accounts. However, it is clear that A should be touched. After finishing the Tx, A should be checked and if it is empty, delete the account from the state.

Ok, I totally trust you without even trying to understand your point. 😜 This was a first try on this, since generally it would be good to have this optimized towards a diff-based setup. Would you have the ressources to work a bit on this during the next days (until the end of the week, or maybe mid next week?) and integrate your suggestions/fixes?

jochem-brouwer · 2023-03-14T10:58:27Z

I tried integrating it yesterday, but this still fails tests (introduced some new ones and fixed some old ones). I will definitely have time to work on this this week!

jochem-brouwer · 2023-03-14T13:47:21Z

Ok, this logic is starting to get very confusing and very complex. Let's see if CI passes now though.

In this state, I do not want to merge this (in case this passes) because the logic is all over the place. Maybe we should create a more general journal class which does this diff-based journaling, with the commit/revert/checkpoint logic?

jochem-brouwer · 2023-03-14T13:52:21Z

Ok, Istanbul state tests passed locally so I expect CI to pass.

jochem-brouwer · 2023-03-14T14:05:34Z

BTW, side note, since this targets the Shanghai DDoS attacks - we still have this problem: #2406

(For this specific attack we will run out-of-memory on mainnet also)

acolytec3

This looks great. Have worked through the code and I think this approach makes sense!

jochem-brouwer · 2023-03-15T10:17:31Z

Ok great! Let's discuss in the call this afternoon if we want to merge as-is, because I think this current approach (1) works but is (2) rather hackish :)

jochem-brouwer · 2023-03-15T20:20:01Z

Added general journaling. I have not yet looked if we can also add this to the access lists, so the journaling might change in another PR. Ready for review.

…, haha

…n commit

holgerd77 · 2023-03-17T10:37:35Z

Rebased this via UI.

Not sure if this a new development, but just to mention that various VM extended blockchain test runs were failing for this lately.

acolytec3 · 2023-03-17T17:14:10Z

I've started looking at the logs. Looks like the jorunaling isn't properly catching accounts that need to be cleaned up on revert (at least in some instances). I've compared the VM output between master and this branch and see where a touched account is being cleaned up in master and not in this branch so will need to research further.

jochem-brouwer · 2023-03-17T19:04:12Z

Hmm, weird, the tests failing do not seem super related to the touched accounts. I also ran the Istanbul blockchain test locally, and it passed, so I assumed all was OK (but I rechecked, also fails here now, so not sure what went on there)

acolytec3

This looks great. Pending passing CI with the fix on the journal height, lets merge this.

* Added v7 release reference in main README table (#2562) * common: Schedule shanghai on goerli (#2563) * common: Schedule shanghai on goerli * update timestamp * util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564) * util/tx: Shift ssz back to case dependency free ES2019 compatible version * update package lock * update karma ecma version * VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568) * EVM: Memory Fix & Other Optimizations (#2570) * EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:* * VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging) * EVM: CALL/CREATE debug exit msg differentiation * EVM: avoid buffer copy in memory read (performance) * EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification * EVM: Added EIP check for transient storage checkpointing * EVM: Precompile Debug Logger Improvements (#2572) * EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging * EVM: Added remaining precompile debug loggers * EVM: added error cases to BLS precompile debug log * EVM: Added missing precompile return value debug logs * Small fixes * tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575) * EVM: Avoid memory.read() Memory Copy (#2573) * EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode * EVM: Add direct memory read to all calling opcodes * EVM: Copy over memory on IDENTITY precompile * EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow) * Some optimizations * blockchain: fix merge->clique transition (#2571) * Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577) * client/engine: ensure finalized/safe blocks are in canonical chain * client: engine-api: fix finalized block check * client/tests: fix forkchoice updated test * client: add fcu tests to check if blocks are part of canonical chain * client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579) * client/engine: ensure invalid blockhash response matches spec (#2583) * client/engine: delete invalid skeleton blocks (#2584) Co-authored-by: acolytec3 <[email protected]> * Setup to dev/test snapsync with sim architecture (#2574) * Setup to dev/test snapsync with sim architecture * modfiy single-run to setup a lodestar<>geth node to snapsync from * setup an ethereumjs inline client and get it to peer with geth * cleanup setup a bit * snapsync run spec * get the snap testdev sim working * finalize the test infra and update usage doc * enhance coverage * Use geth RPC to connect to ethJS * refac wait for snap sync completion --------- Co-authored-by: acolytec3 <[email protected]> * client: Add safe and finalized blockoptions to the chain (#2585) * client: Add safe and finalized blockoptions to the chain * fix tests * fix more tests * fix remaining * cleanup * enhance coverage * unset scheduled goerli timestamp based hfs colliding with test * Client: Small Debug Helpers and CLI Improvements (#2586) * Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks * Client -> Execution: NumBlocksPerIteration (default: 50) as an option * Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0 * Apply suggestions from code review Co-authored-by: acolytec3 <[email protected]> * Apply suggestions from code review --------- Co-authored-by: acolytec3 <[email protected]> * common: Schedule Shanghai on mainnet! (#2591) * common: Schedule Shanghai on mainnet! * clear hf timestamp for test * VM: Diff-based Touched Accounts Checkpointing (#2581) * VM: Switched to a more efficient diff-based way of touched account checkpointing * VM: move accessed storage inefficient checkpointing problem to berlin, haha * EVM: avoid memory copy in MLOAD opcode function * Remove console.log() in EVM * vmState: ensure touched accounts delete stack gets properly updated on commit * vm/eei: save touched height * vm/vmState: new possible fix for touched accounts * vm/vmState: another attempt to fix touched accounts journaling * vm: add journaling * Check correct journal height on revert --------- Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: acolytec3 <[email protected]> --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]>

* V7 update to master 1 (#2593) * Added v7 release reference in main README table (#2562) * common: Schedule shanghai on goerli (#2563) * common: Schedule shanghai on goerli * update timestamp * util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564) * util/tx: Shift ssz back to case dependency free ES2019 compatible version * update package lock * update karma ecma version * VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568) * EVM: Memory Fix & Other Optimizations (#2570) * EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:* * VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging) * EVM: CALL/CREATE debug exit msg differentiation * EVM: avoid buffer copy in memory read (performance) * EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification * EVM: Added EIP check for transient storage checkpointing * EVM: Precompile Debug Logger Improvements (#2572) * EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging * EVM: Added remaining precompile debug loggers * EVM: added error cases to BLS precompile debug log * EVM: Added missing precompile return value debug logs * Small fixes * tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575) * EVM: Avoid memory.read() Memory Copy (#2573) * EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode * EVM: Add direct memory read to all calling opcodes * EVM: Copy over memory on IDENTITY precompile * EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow) * Some optimizations * blockchain: fix merge->clique transition (#2571) * Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577) * client/engine: ensure finalized/safe blocks are in canonical chain * client: engine-api: fix finalized block check * client/tests: fix forkchoice updated test * client: add fcu tests to check if blocks are part of canonical chain * client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579) * client/engine: ensure invalid blockhash response matches spec (#2583) * client/engine: delete invalid skeleton blocks (#2584) Co-authored-by: acolytec3 <[email protected]> * Setup to dev/test snapsync with sim architecture (#2574) * Setup to dev/test snapsync with sim architecture * modfiy single-run to setup a lodestar<>geth node to snapsync from * setup an ethereumjs inline client and get it to peer with geth * cleanup setup a bit * snapsync run spec * get the snap testdev sim working * finalize the test infra and update usage doc * enhance coverage * Use geth RPC to connect to ethJS * refac wait for snap sync completion --------- Co-authored-by: acolytec3 <[email protected]> * client: Add safe and finalized blockoptions to the chain (#2585) * client: Add safe and finalized blockoptions to the chain * fix tests * fix more tests * fix remaining * cleanup * enhance coverage * unset scheduled goerli timestamp based hfs colliding with test * Client: Small Debug Helpers and CLI Improvements (#2586) * Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks * Client -> Execution: NumBlocksPerIteration (default: 50) as an option * Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0 * Apply suggestions from code review Co-authored-by: acolytec3 <[email protected]> * Apply suggestions from code review --------- Co-authored-by: acolytec3 <[email protected]> * common: Schedule Shanghai on mainnet! (#2591) * common: Schedule Shanghai on mainnet! * clear hf timestamp for test * VM: Diff-based Touched Accounts Checkpointing (#2581) * VM: Switched to a more efficient diff-based way of touched account checkpointing * VM: move accessed storage inefficient checkpointing problem to berlin, haha * EVM: avoid memory copy in MLOAD opcode function * Remove console.log() in EVM * vmState: ensure touched accounts delete stack gets properly updated on commit * vm/eei: save touched height * vm/vmState: new possible fix for touched accounts * vm/vmState: another attempt to fix touched accounts journaling * vm: add journaling * Check correct journal height on revert --------- Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: acolytec3 <[email protected]> --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> * First pass - most util changes * Fix most account stuff * Fix account test * Many byte fixes * util: fix constants tests * remaining fixes * Turn off ci jobs * monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes * util: update description of bytes exporT * util: remove unused import * common: use bytesToHex helper instead of toString('hex') * trie: refactor non-test files to Uint8Array * util: add binary string utils * util: remove extra Uint8Array.from * util: remove arrToBufArray util * trie: adjust tests and fix outstanding issues * util: remove binarystring utils and add compareBytes and randomBytes util * common: refactor common with Uint8Array * util: accept 0x-prefixed and non-prefixed hex strings in toBytes * tx: refactor Buffer -> Uint8Array * tx: remove unused import * util: revert toBytes update * block: refactor Buffer -> uint8array * block: adjust import * trie: refactor remaining buffer instances * move devp2p to uint8Array * statemanager: refactor buffer -> uint8array * util: simplify zeros * util: add concatBytesUnsafe * ethash: partial migration * ethash: update examples * ethash: wip fixes * ethash: more WIP * ethash: ensure fnv input is read from mix, not mixView * blockchain: migrate to uint8array * ethash: renable all tests * ethash: fix bytesReverse * Fix miner tests * many hexToBytes moves * most of evm/vm moves * evm: Fix all tests * vm: more fixes * More fixes * vm: fix receipts encoding * vm: fix tester * client: refactor buffer to uint8array * client: additional uint8 adjustments * client: fix most tests * fix remaining client unit tests * reactivate most CI * client: fix les test/protocol * turn client CI on * util: fix name typo * lint * Fix withdrawals * remove buffarraytoarr * Remove bufArrtoArr references * Lint * fix examples * Fix difficulty test * lint * block: update randomBytes import * replace randombytes import * client: fix sim test util * vm: fix example * devp2p: update snappy typing and fix tests * Fix tests * Remove additional buffer references * rustbn fixes * add 0x prefix to precompile address * Remove `node-ip` dependency and buffer references in devp2p * Switch slice to subarray * evm: fix blake2f * Merge fixes * more merge commit fixes * more test fixes * Address all the feedback * fix dns test * Update packages/util/src/bytes.ts Co-authored-by: Jochem Brouwer <[email protected]> * Fix return type for baToJSON * util: instantiate hexByByte array * Remove baToJson * rebase fixes * Fix event typing * Revert outdated initcode changes * lint --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]>

* `Buffer` to `Uint8Array` conversion (#2566) * V7 update to master 1 (#2593) * Added v7 release reference in main README table (#2562) * common: Schedule shanghai on goerli (#2563) * common: Schedule shanghai on goerli * update timestamp * util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564) * util/tx: Shift ssz back to case dependency free ES2019 compatible version * update package lock * update karma ecma version * VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568) * EVM: Memory Fix & Other Optimizations (#2570) * EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:* * VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging) * EVM: CALL/CREATE debug exit msg differentiation * EVM: avoid buffer copy in memory read (performance) * EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification * EVM: Added EIP check for transient storage checkpointing * EVM: Precompile Debug Logger Improvements (#2572) * EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging * EVM: Added remaining precompile debug loggers * EVM: added error cases to BLS precompile debug log * EVM: Added missing precompile return value debug logs * Small fixes * tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575) * EVM: Avoid memory.read() Memory Copy (#2573) * EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode * EVM: Add direct memory read to all calling opcodes * EVM: Copy over memory on IDENTITY precompile * EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow) * Some optimizations * blockchain: fix merge->clique transition (#2571) * Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577) * client/engine: ensure finalized/safe blocks are in canonical chain * client: engine-api: fix finalized block check * client/tests: fix forkchoice updated test * client: add fcu tests to check if blocks are part of canonical chain * client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579) * client/engine: ensure invalid blockhash response matches spec (#2583) * client/engine: delete invalid skeleton blocks (#2584) Co-authored-by: acolytec3 <[email protected]> * Setup to dev/test snapsync with sim architecture (#2574) * Setup to dev/test snapsync with sim architecture * modfiy single-run to setup a lodestar<>geth node to snapsync from * setup an ethereumjs inline client and get it to peer with geth * cleanup setup a bit * snapsync run spec * get the snap testdev sim working * finalize the test infra and update usage doc * enhance coverage * Use geth RPC to connect to ethJS * refac wait for snap sync completion --------- Co-authored-by: acolytec3 <[email protected]> * client: Add safe and finalized blockoptions to the chain (#2585) * client: Add safe and finalized blockoptions to the chain * fix tests * fix more tests * fix remaining * cleanup * enhance coverage * unset scheduled goerli timestamp based hfs colliding with test * Client: Small Debug Helpers and CLI Improvements (#2586) * Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks * Client -> Execution: NumBlocksPerIteration (default: 50) as an option * Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0 * Apply suggestions from code review Co-authored-by: acolytec3 <[email protected]> * Apply suggestions from code review --------- Co-authored-by: acolytec3 <[email protected]> * common: Schedule Shanghai on mainnet! (#2591) * common: Schedule Shanghai on mainnet! * clear hf timestamp for test * VM: Diff-based Touched Accounts Checkpointing (#2581) * VM: Switched to a more efficient diff-based way of touched account checkpointing * VM: move accessed storage inefficient checkpointing problem to berlin, haha * EVM: avoid memory copy in MLOAD opcode function * Remove console.log() in EVM * vmState: ensure touched accounts delete stack gets properly updated on commit * vm/eei: save touched height * vm/vmState: new possible fix for touched accounts * vm/vmState: another attempt to fix touched accounts journaling * vm: add journaling * Check correct journal height on revert --------- Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: acolytec3 <[email protected]> --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> * First pass - most util changes * Fix most account stuff * Fix account test * Many byte fixes * util: fix constants tests * remaining fixes * Turn off ci jobs * monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes * util: update description of bytes exporT * util: remove unused import * common: use bytesToHex helper instead of toString('hex') * trie: refactor non-test files to Uint8Array * util: add binary string utils * util: remove extra Uint8Array.from * util: remove arrToBufArray util * trie: adjust tests and fix outstanding issues * util: remove binarystring utils and add compareBytes and randomBytes util * common: refactor common with Uint8Array * util: accept 0x-prefixed and non-prefixed hex strings in toBytes * tx: refactor Buffer -> Uint8Array * tx: remove unused import * util: revert toBytes update * block: refactor Buffer -> uint8array * block: adjust import * trie: refactor remaining buffer instances * move devp2p to uint8Array * statemanager: refactor buffer -> uint8array * util: simplify zeros * util: add concatBytesUnsafe * ethash: partial migration * ethash: update examples * ethash: wip fixes * ethash: more WIP * ethash: ensure fnv input is read from mix, not mixView * blockchain: migrate to uint8array * ethash: renable all tests * ethash: fix bytesReverse * Fix miner tests * many hexToBytes moves * most of evm/vm moves * evm: Fix all tests * vm: more fixes * More fixes * vm: fix receipts encoding * vm: fix tester * client: refactor buffer to uint8array * client: additional uint8 adjustments * client: fix most tests * fix remaining client unit tests * reactivate most CI * client: fix les test/protocol * turn client CI on * util: fix name typo * lint * Fix withdrawals * remove buffarraytoarr * Remove bufArrtoArr references * Lint * fix examples * Fix difficulty test * lint * block: update randomBytes import * replace randombytes import * client: fix sim test util * vm: fix example * devp2p: update snappy typing and fix tests * Fix tests * Remove additional buffer references * rustbn fixes * add 0x prefix to precompile address * Remove `node-ip` dependency and buffer references in devp2p * Switch slice to subarray * evm: fix blake2f * Merge fixes * more merge commit fixes * more test fixes * Address all the feedback * fix dns test * Update packages/util/src/bytes.ts Co-authored-by: Jochem Brouwer <[email protected]> * Fix return type for baToJSON * util: instantiate hexByByte array * Remove baToJson * rebase fixes * Fix event typing * Revert outdated initcode changes * lint --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]> * Devp2p status fix * Remove buffer detritus * Switch db to view * buffer cleanup * Correctly parse heads from DB * Fix encodings * Cast db values to uint8array * Fix db bug in ethash * Remove unused peerId check * Add pow miner test * Finish test * Move pow test to integration tests * client: lint * rename test helper * Add timeout to PoW test * Fix test runner * remove console log --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]>

* `Buffer` to `Uint8Array` conversion (#2566) * V7 update to master 1 (#2593) * Added v7 release reference in main README table (#2562) * common: Schedule shanghai on goerli (#2563) * common: Schedule shanghai on goerli * update timestamp * util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564) * util/tx: Shift ssz back to case dependency free ES2019 compatible version * update package lock * update karma ecma version * VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568) * EVM: Memory Fix & Other Optimizations (#2570) * EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:* * VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging) * EVM: CALL/CREATE debug exit msg differentiation * EVM: avoid buffer copy in memory read (performance) * EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification * EVM: Added EIP check for transient storage checkpointing * EVM: Precompile Debug Logger Improvements (#2572) * EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging * EVM: Added remaining precompile debug loggers * EVM: added error cases to BLS precompile debug log * EVM: Added missing precompile return value debug logs * Small fixes * tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575) * EVM: Avoid memory.read() Memory Copy (#2573) * EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode * EVM: Add direct memory read to all calling opcodes * EVM: Copy over memory on IDENTITY precompile * EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow) * Some optimizations * blockchain: fix merge->clique transition (#2571) * Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577) * client/engine: ensure finalized/safe blocks are in canonical chain * client: engine-api: fix finalized block check * client/tests: fix forkchoice updated test * client: add fcu tests to check if blocks are part of canonical chain * client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579) * client/engine: ensure invalid blockhash response matches spec (#2583) * client/engine: delete invalid skeleton blocks (#2584) Co-authored-by: acolytec3 <[email protected]> * Setup to dev/test snapsync with sim architecture (#2574) * Setup to dev/test snapsync with sim architecture * modfiy single-run to setup a lodestar<>geth node to snapsync from * setup an ethereumjs inline client and get it to peer with geth * cleanup setup a bit * snapsync run spec * get the snap testdev sim working * finalize the test infra and update usage doc * enhance coverage * Use geth RPC to connect to ethJS * refac wait for snap sync completion --------- Co-authored-by: acolytec3 <[email protected]> * client: Add safe and finalized blockoptions to the chain (#2585) * client: Add safe and finalized blockoptions to the chain * fix tests * fix more tests * fix remaining * cleanup * enhance coverage * unset scheduled goerli timestamp based hfs colliding with test * Client: Small Debug Helpers and CLI Improvements (#2586) * Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks * Client -> Execution: NumBlocksPerIteration (default: 50) as an option * Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0 * Apply suggestions from code review Co-authored-by: acolytec3 <[email protected]> * Apply suggestions from code review --------- Co-authored-by: acolytec3 <[email protected]> * common: Schedule Shanghai on mainnet! (#2591) * common: Schedule Shanghai on mainnet! * clear hf timestamp for test * VM: Diff-based Touched Accounts Checkpointing (#2581) * VM: Switched to a more efficient diff-based way of touched account checkpointing * VM: move accessed storage inefficient checkpointing problem to berlin, haha * EVM: avoid memory copy in MLOAD opcode function * Remove console.log() in EVM * vmState: ensure touched accounts delete stack gets properly updated on commit * vm/eei: save touched height * vm/vmState: new possible fix for touched accounts * vm/vmState: another attempt to fix touched accounts journaling * vm: add journaling * Check correct journal height on revert --------- Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: acolytec3 <[email protected]> --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> * First pass - most util changes * Fix most account stuff * Fix account test * Many byte fixes * util: fix constants tests * remaining fixes * Turn off ci jobs * monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes * util: update description of bytes exporT * util: remove unused import * common: use bytesToHex helper instead of toString('hex') * trie: refactor non-test files to Uint8Array * util: add binary string utils * util: remove extra Uint8Array.from * util: remove arrToBufArray util * trie: adjust tests and fix outstanding issues * util: remove binarystring utils and add compareBytes and randomBytes util * common: refactor common with Uint8Array * util: accept 0x-prefixed and non-prefixed hex strings in toBytes * tx: refactor Buffer -> Uint8Array * tx: remove unused import * util: revert toBytes update * block: refactor Buffer -> uint8array * block: adjust import * trie: refactor remaining buffer instances * move devp2p to uint8Array * statemanager: refactor buffer -> uint8array * util: simplify zeros * util: add concatBytesUnsafe * ethash: partial migration * ethash: update examples * ethash: wip fixes * ethash: more WIP * ethash: ensure fnv input is read from mix, not mixView * blockchain: migrate to uint8array * ethash: renable all tests * ethash: fix bytesReverse * Fix miner tests * many hexToBytes moves * most of evm/vm moves * evm: Fix all tests * vm: more fixes * More fixes * vm: fix receipts encoding * vm: fix tester * client: refactor buffer to uint8array * client: additional uint8 adjustments * client: fix most tests * fix remaining client unit tests * reactivate most CI * client: fix les test/protocol * turn client CI on * util: fix name typo * lint * Fix withdrawals * remove buffarraytoarr * Remove bufArrtoArr references * Lint * fix examples * Fix difficulty test * lint * block: update randomBytes import * replace randombytes import * client: fix sim test util * vm: fix example * devp2p: update snappy typing and fix tests * Fix tests * Remove additional buffer references * rustbn fixes * add 0x prefix to precompile address * Remove `node-ip` dependency and buffer references in devp2p * Switch slice to subarray * evm: fix blake2f * Merge fixes * more merge commit fixes * more test fixes * Address all the feedback * fix dns test * Update packages/util/src/bytes.ts Co-authored-by: Jochem Brouwer <[email protected]> * Fix return type for baToJSON * util: instantiate hexByByte array * Remove baToJson * rebase fixes * Fix event typing * Revert outdated initcode changes * lint --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]> * Devp2p status fix * Remove buffer detritus * Switch db to view * buffer cleanup * Correctly parse heads from DB * Fix encodings * Cast db values to uint8array * Fix db bug in ethash * Remove unused peerId check * Add pow miner test * Finish test * Move pow test to integration tests * client: lint * rename test helper * Add timeout to PoW test * Update eip4844 txs to decoupled blobs spec * fix sharding spec and blobtx * fix the sharding muli client run * fix args * fix compatibility with latest lodestar branch * update test help * Fix fee market test * lint * Move all kzg stuff to util * Various cleanup * Update to latest c-kzg * Fix utils * Remove outdated kzg references * Fix client tests * Update c-kzg dep * Fix karma, remove Buffer references --------- Co-authored-by: acolytec3 <[email protected]> Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]>

* V7 update to master 1 (#2593) * Added v7 release reference in main README table (#2562) * common: Schedule shanghai on goerli (#2563) * common: Schedule shanghai on goerli * update timestamp * util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564) * util/tx: Shift ssz back to case dependency free ES2019 compatible version * update package lock * update karma ecma version * VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568) * EVM: Memory Fix & Other Optimizations (#2570) * EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:* * VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging) * EVM: CALL/CREATE debug exit msg differentiation * EVM: avoid buffer copy in memory read (performance) * EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification * EVM: Added EIP check for transient storage checkpointing * EVM: Precompile Debug Logger Improvements (#2572) * EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging * EVM: Added remaining precompile debug loggers * EVM: added error cases to BLS precompile debug log * EVM: Added missing precompile return value debug logs * Small fixes * tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575) * EVM: Avoid memory.read() Memory Copy (#2573) * EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode * EVM: Add direct memory read to all calling opcodes * EVM: Copy over memory on IDENTITY precompile * EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow) * Some optimizations * blockchain: fix merge->clique transition (#2571) * Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577) * client/engine: ensure finalized/safe blocks are in canonical chain * client: engine-api: fix finalized block check * client/tests: fix forkchoice updated test * client: add fcu tests to check if blocks are part of canonical chain * client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579) * client/engine: ensure invalid blockhash response matches spec (#2583) * client/engine: delete invalid skeleton blocks (#2584) Co-authored-by: acolytec3 <[email protected]> * Setup to dev/test snapsync with sim architecture (#2574) * Setup to dev/test snapsync with sim architecture * modfiy single-run to setup a lodestar<>geth node to snapsync from * setup an ethereumjs inline client and get it to peer with geth * cleanup setup a bit * snapsync run spec * get the snap testdev sim working * finalize the test infra and update usage doc * enhance coverage * Use geth RPC to connect to ethJS * refac wait for snap sync completion --------- Co-authored-by: acolytec3 <[email protected]> * client: Add safe and finalized blockoptions to the chain (#2585) * client: Add safe and finalized blockoptions to the chain * fix tests * fix more tests * fix remaining * cleanup * enhance coverage * unset scheduled goerli timestamp based hfs colliding with test * Client: Small Debug Helpers and CLI Improvements (#2586) * Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks * Client -> Execution: NumBlocksPerIteration (default: 50) as an option * Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0 * Apply suggestions from code review Co-authored-by: acolytec3 <[email protected]> * Apply suggestions from code review --------- Co-authored-by: acolytec3 <[email protected]> * common: Schedule Shanghai on mainnet! (#2591) * common: Schedule Shanghai on mainnet! * clear hf timestamp for test * VM: Diff-based Touched Accounts Checkpointing (#2581) * VM: Switched to a more efficient diff-based way of touched account checkpointing * VM: move accessed storage inefficient checkpointing problem to berlin, haha * EVM: avoid memory copy in MLOAD opcode function * Remove console.log() in EVM * vmState: ensure touched accounts delete stack gets properly updated on commit * vm/eei: save touched height * vm/vmState: new possible fix for touched accounts * vm/vmState: another attempt to fix touched accounts journaling * vm: add journaling * Check correct journal height on revert --------- Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: acolytec3 <[email protected]> --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> * First pass - most util changes * Fix most account stuff * Fix account test * Many byte fixes * util: fix constants tests * remaining fixes * Turn off ci jobs * monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes * util: update description of bytes exporT * util: remove unused import * common: use bytesToHex helper instead of toString('hex') * trie: refactor non-test files to Uint8Array * util: add binary string utils * util: remove extra Uint8Array.from * util: remove arrToBufArray util * trie: adjust tests and fix outstanding issues * util: remove binarystring utils and add compareBytes and randomBytes util * common: refactor common with Uint8Array * util: accept 0x-prefixed and non-prefixed hex strings in toBytes * tx: refactor Buffer -> Uint8Array * tx: remove unused import * util: revert toBytes update * block: refactor Buffer -> uint8array * block: adjust import * trie: refactor remaining buffer instances * move devp2p to uint8Array * statemanager: refactor buffer -> uint8array * util: simplify zeros * util: add concatBytesUnsafe * ethash: partial migration * ethash: update examples * ethash: wip fixes * ethash: more WIP * ethash: ensure fnv input is read from mix, not mixView * blockchain: migrate to uint8array * ethash: renable all tests * ethash: fix bytesReverse * Fix miner tests * many hexToBytes moves * most of evm/vm moves * evm: Fix all tests * vm: more fixes * More fixes * vm: fix receipts encoding * vm: fix tester * client: refactor buffer to uint8array * client: additional uint8 adjustments * client: fix most tests * fix remaining client unit tests * reactivate most CI * client: fix les test/protocol * turn client CI on * util: fix name typo * lint * Fix withdrawals * remove buffarraytoarr * Remove bufArrtoArr references * Lint * fix examples * Fix difficulty test * lint * block: update randomBytes import * replace randombytes import * client: fix sim test util * vm: fix example * devp2p: update snappy typing and fix tests * Fix tests * Remove additional buffer references * rustbn fixes * add 0x prefix to precompile address * Remove `node-ip` dependency and buffer references in devp2p * Switch slice to subarray * evm: fix blake2f * Merge fixes * more merge commit fixes * more test fixes * Address all the feedback * fix dns test * Update packages/util/src/bytes.ts Co-authored-by: Jochem Brouwer <[email protected]> * Fix return type for baToJSON * util: instantiate hexByByte array * Remove baToJson * rebase fixes * Fix event typing * Revert outdated initcode changes * lint --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]>

* `Buffer` to `Uint8Array` conversion (#2566) * V7 update to master 1 (#2593) * Added v7 release reference in main README table (#2562) * common: Schedule shanghai on goerli (#2563) * common: Schedule shanghai on goerli * update timestamp * util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564) * util/tx: Shift ssz back to case dependency free ES2019 compatible version * update package lock * update karma ecma version * VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568) * EVM: Memory Fix & Other Optimizations (#2570) * EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:* * VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging) * EVM: CALL/CREATE debug exit msg differentiation * EVM: avoid buffer copy in memory read (performance) * EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification * EVM: Added EIP check for transient storage checkpointing * EVM: Precompile Debug Logger Improvements (#2572) * EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging * EVM: Added remaining precompile debug loggers * EVM: added error cases to BLS precompile debug log * EVM: Added missing precompile return value debug logs * Small fixes * tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575) * EVM: Avoid memory.read() Memory Copy (#2573) * EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode * EVM: Add direct memory read to all calling opcodes * EVM: Copy over memory on IDENTITY precompile * EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow) * Some optimizations * blockchain: fix merge->clique transition (#2571) * Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577) * client/engine: ensure finalized/safe blocks are in canonical chain * client: engine-api: fix finalized block check * client/tests: fix forkchoice updated test * client: add fcu tests to check if blocks are part of canonical chain * client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579) * client/engine: ensure invalid blockhash response matches spec (#2583) * client/engine: delete invalid skeleton blocks (#2584) Co-authored-by: acolytec3 <[email protected]> * Setup to dev/test snapsync with sim architecture (#2574) * Setup to dev/test snapsync with sim architecture * modfiy single-run to setup a lodestar<>geth node to snapsync from * setup an ethereumjs inline client and get it to peer with geth * cleanup setup a bit * snapsync run spec * get the snap testdev sim working * finalize the test infra and update usage doc * enhance coverage * Use geth RPC to connect to ethJS * refac wait for snap sync completion --------- Co-authored-by: acolytec3 <[email protected]> * client: Add safe and finalized blockoptions to the chain (#2585) * client: Add safe and finalized blockoptions to the chain * fix tests * fix more tests * fix remaining * cleanup * enhance coverage * unset scheduled goerli timestamp based hfs colliding with test * Client: Small Debug Helpers and CLI Improvements (#2586) * Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks * Client -> Execution: NumBlocksPerIteration (default: 50) as an option * Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0 * Apply suggestions from code review Co-authored-by: acolytec3 <[email protected]> * Apply suggestions from code review --------- Co-authored-by: acolytec3 <[email protected]> * common: Schedule Shanghai on mainnet! (#2591) * common: Schedule Shanghai on mainnet! * clear hf timestamp for test * VM: Diff-based Touched Accounts Checkpointing (#2581) * VM: Switched to a more efficient diff-based way of touched account checkpointing * VM: move accessed storage inefficient checkpointing problem to berlin, haha * EVM: avoid memory copy in MLOAD opcode function * Remove console.log() in EVM * vmState: ensure touched accounts delete stack gets properly updated on commit * vm/eei: save touched height * vm/vmState: new possible fix for touched accounts * vm/vmState: another attempt to fix touched accounts journaling * vm: add journaling * Check correct journal height on revert --------- Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: acolytec3 <[email protected]> --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> * First pass - most util changes * Fix most account stuff * Fix account test * Many byte fixes * util: fix constants tests * remaining fixes * Turn off ci jobs * monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes * util: update description of bytes exporT * util: remove unused import * common: use bytesToHex helper instead of toString('hex') * trie: refactor non-test files to Uint8Array * util: add binary string utils * util: remove extra Uint8Array.from * util: remove arrToBufArray util * trie: adjust tests and fix outstanding issues * util: remove binarystring utils and add compareBytes and randomBytes util * common: refactor common with Uint8Array * util: accept 0x-prefixed and non-prefixed hex strings in toBytes * tx: refactor Buffer -> Uint8Array * tx: remove unused import * util: revert toBytes update * block: refactor Buffer -> uint8array * block: adjust import * trie: refactor remaining buffer instances * move devp2p to uint8Array * statemanager: refactor buffer -> uint8array * util: simplify zeros * util: add concatBytesUnsafe * ethash: partial migration * ethash: update examples * ethash: wip fixes * ethash: more WIP * ethash: ensure fnv input is read from mix, not mixView * blockchain: migrate to uint8array * ethash: renable all tests * ethash: fix bytesReverse * Fix miner tests * many hexToBytes moves * most of evm/vm moves * evm: Fix all tests * vm: more fixes * More fixes * vm: fix receipts encoding * vm: fix tester * client: refactor buffer to uint8array * client: additional uint8 adjustments * client: fix most tests * fix remaining client unit tests * reactivate most CI * client: fix les test/protocol * turn client CI on * util: fix name typo * lint * Fix withdrawals * remove buffarraytoarr * Remove bufArrtoArr references * Lint * fix examples * Fix difficulty test * lint * block: update randomBytes import * replace randombytes import * client: fix sim test util * vm: fix example * devp2p: update snappy typing and fix tests * Fix tests * Remove additional buffer references * rustbn fixes * add 0x prefix to precompile address * Remove `node-ip` dependency and buffer references in devp2p * Switch slice to subarray * evm: fix blake2f * Merge fixes * more merge commit fixes * more test fixes * Address all the feedback * fix dns test * Update packages/util/src/bytes.ts Co-authored-by: Jochem Brouwer <[email protected]> * Fix return type for baToJSON * util: instantiate hexByByte array * Remove baToJson * rebase fixes * Fix event typing * Revert outdated initcode changes * lint --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]> * Devp2p status fix * Remove buffer detritus * Switch db to view * buffer cleanup * Correctly parse heads from DB * Fix encodings * Cast db values to uint8array * Fix db bug in ethash * Remove unused peerId check * Add pow miner test * Finish test * Move pow test to integration tests * client: lint * rename test helper * Add timeout to PoW test * Fix test runner * remove console log --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]>

* `Buffer` to `Uint8Array` conversion (#2566) * V7 update to master 1 (#2593) * Added v7 release reference in main README table (#2562) * common: Schedule shanghai on goerli (#2563) * common: Schedule shanghai on goerli * update timestamp * util/tx: Shift ssz back to case dependency free ES2019 compatible version (#2564) * util/tx: Shift ssz back to case dependency free ES2019 compatible version * update package lock * update karma ecma version * VM: some optimization on the bnadd/bnmul precompiles to only copy over the necessary 128 bytes as input for the WASM call (#2568) * EVM: Memory Fix & Other Optimizations (#2570) * EVM: Rename evm debug logger to evm:evm (one for package, one for class), consistency, also, logger will otherwise be left out when run with evm:* * VM: Rename message checkpoint to state checkpoint in debug message (there is a dedicated message checkpoint msg along msg logging) * EVM: CALL/CREATE debug exit msg differentiation * EVM: avoid buffer copy in memory read (performance) * EVM: Rewrite runCall() checkpoint/revert conditional for readability/simplification * EVM: Added EIP check for transient storage checkpointing * EVM: Precompile Debug Logger Improvements (#2572) * EVM: Added general precompile debug logger setup, first ECRECOVER exemplary debug logging * EVM: Added remaining precompile debug loggers * EVM: added error cases to BLS precompile debug log * EVM: Added missing precompile return value debug logs * Small fixes * tx: ensure eip3860 txs can have more than max_initcode_size data if to field is non-empty (#2575) * EVM: Avoid memory.read() Memory Copy (#2573) * EVM: added avoidCopy parameter to memory.read() function, first test on CREATE opcode * EVM: Add direct memory read to all calling opcodes * EVM: Copy over memory on IDENTITY precompile * EVM: remove length checks and return buffer 0-filling in Memory.read() (memory is uncoditionally being extended properly anyhow) * Some optimizations * blockchain: fix merge->clique transition (#2571) * Client: ensure safe/finalized blocks are part of the canonical chain on forkchoiceUpdated (#2577) * client/engine: ensure finalized/safe blocks are in canonical chain * client: engine-api: fix finalized block check * client/tests: fix forkchoice updated test * client: add fcu tests to check if blocks are part of canonical chain * client/engine: ensure payload has a valid timestamp forkchoiceUpdated (#2579) * client/engine: ensure invalid blockhash response matches spec (#2583) * client/engine: delete invalid skeleton blocks (#2584) Co-authored-by: acolytec3 <[email protected]> * Setup to dev/test snapsync with sim architecture (#2574) * Setup to dev/test snapsync with sim architecture * modfiy single-run to setup a lodestar<>geth node to snapsync from * setup an ethereumjs inline client and get it to peer with geth * cleanup setup a bit * snapsync run spec * get the snap testdev sim working * finalize the test infra and update usage doc * enhance coverage * Use geth RPC to connect to ethJS * refac wait for snap sync completion --------- Co-authored-by: acolytec3 <[email protected]> * client: Add safe and finalized blockoptions to the chain (#2585) * client: Add safe and finalized blockoptions to the chain * fix tests * fix more tests * fix remaining * cleanup * enhance coverage * unset scheduled goerli timestamp based hfs colliding with test * Client: Small Debug Helpers and CLI Improvements (#2586) * Client: new constant MAX_TOLERATED_BLOCK_TIME for execution, added warning for slowly executed blocks * Client -> Execution: NumBlocksPerIteration (default: 50) as an option * Client: only restart RLPx server or log peer stats if max peers is set to be greater than 0 * Apply suggestions from code review Co-authored-by: acolytec3 <[email protected]> * Apply suggestions from code review --------- Co-authored-by: acolytec3 <[email protected]> * common: Schedule Shanghai on mainnet! (#2591) * common: Schedule Shanghai on mainnet! * clear hf timestamp for test * VM: Diff-based Touched Accounts Checkpointing (#2581) * VM: Switched to a more efficient diff-based way of touched account checkpointing * VM: move accessed storage inefficient checkpointing problem to berlin, haha * EVM: avoid memory copy in MLOAD opcode function * Remove console.log() in EVM * vmState: ensure touched accounts delete stack gets properly updated on commit * vm/eei: save touched height * vm/vmState: new possible fix for touched accounts * vm/vmState: another attempt to fix touched accounts journaling * vm: add journaling * Check correct journal height on revert --------- Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: acolytec3 <[email protected]> --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> * First pass - most util changes * Fix most account stuff * Fix account test * Many byte fixes * util: fix constants tests * remaining fixes * Turn off ci jobs * monorepo: bigIntToUnpaddedBuffer -> bigIntToUnpaddedBytes * util: update description of bytes exporT * util: remove unused import * common: use bytesToHex helper instead of toString('hex') * trie: refactor non-test files to Uint8Array * util: add binary string utils * util: remove extra Uint8Array.from * util: remove arrToBufArray util * trie: adjust tests and fix outstanding issues * util: remove binarystring utils and add compareBytes and randomBytes util * common: refactor common with Uint8Array * util: accept 0x-prefixed and non-prefixed hex strings in toBytes * tx: refactor Buffer -> Uint8Array * tx: remove unused import * util: revert toBytes update * block: refactor Buffer -> uint8array * block: adjust import * trie: refactor remaining buffer instances * move devp2p to uint8Array * statemanager: refactor buffer -> uint8array * util: simplify zeros * util: add concatBytesUnsafe * ethash: partial migration * ethash: update examples * ethash: wip fixes * ethash: more WIP * ethash: ensure fnv input is read from mix, not mixView * blockchain: migrate to uint8array * ethash: renable all tests * ethash: fix bytesReverse * Fix miner tests * many hexToBytes moves * most of evm/vm moves * evm: Fix all tests * vm: more fixes * More fixes * vm: fix receipts encoding * vm: fix tester * client: refactor buffer to uint8array * client: additional uint8 adjustments * client: fix most tests * fix remaining client unit tests * reactivate most CI * client: fix les test/protocol * turn client CI on * util: fix name typo * lint * Fix withdrawals * remove buffarraytoarr * Remove bufArrtoArr references * Lint * fix examples * Fix difficulty test * lint * block: update randomBytes import * replace randombytes import * client: fix sim test util * vm: fix example * devp2p: update snappy typing and fix tests * Fix tests * Remove additional buffer references * rustbn fixes * add 0x prefix to precompile address * Remove `node-ip` dependency and buffer references in devp2p * Switch slice to subarray * evm: fix blake2f * Merge fixes * more merge commit fixes * more test fixes * Address all the feedback * fix dns test * Update packages/util/src/bytes.ts Co-authored-by: Jochem Brouwer <[email protected]> * Fix return type for baToJSON * util: instantiate hexByByte array * Remove baToJson * rebase fixes * Fix event typing * Revert outdated initcode changes * lint --------- Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: g11tech <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]> * Devp2p status fix * Remove buffer detritus * Switch db to view * buffer cleanup * Correctly parse heads from DB * Fix encodings * Cast db values to uint8array * Fix db bug in ethash * Remove unused peerId check * Add pow miner test * Finish test * Move pow test to integration tests * client: lint * rename test helper * Add timeout to PoW test * Update eip4844 txs to decoupled blobs spec * fix sharding spec and blobtx * fix the sharding muli client run * fix args * fix compatibility with latest lodestar branch * update test help * Fix fee market test * lint * Move all kzg stuff to util * Various cleanup * Update to latest c-kzg * Fix utils * Remove outdated kzg references * Fix client tests * Update c-kzg dep * Fix karma, remove Buffer references --------- Co-authored-by: acolytec3 <[email protected]> Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: Jochem Brouwer <[email protected]> Co-authored-by: Gabriel Rocheleau <[email protected]>

holgerd77 added PR state: needs review PR state: merge ready type: refactor type: performance type: enhancement package: vm package: evm labels Mar 13, 2023

holgerd77 mentioned this pull request Mar 13, 2023

Statemanager: Add Separate Cache-Free StateManager version for Client Execution / copy() Options Fix #2582

Closed

jochem-brouwer reviewed Mar 13, 2023

View reviewed changes

jochem-brouwer added the type: test all hardforks This special label enables VM state and blockchain tests for all hardforks on the respective PR. label Mar 13, 2023

jochem-brouwer added PR/Issue state: blocked and removed PR state: merge ready labels Mar 14, 2023

jochem-brouwer mentioned this pull request Mar 14, 2023

Remove touched height map #2589

Closed

acolytec3 previously approved these changes Mar 14, 2023

View reviewed changes

jochem-brouwer dismissed acolytec3’s stale review via ba08168 March 15, 2023 20:18

jochem-brouwer added PR state: merge ready and removed PR/Issue state: blocked labels Mar 15, 2023

jochem-brouwer requested a review from acolytec3 March 15, 2023 20:20

holgerd77 and others added 8 commits March 17, 2023 11:36

VM: move accessed storage inefficient checkpointing problem to berlin…

097ba6a

…, haha

EVM: avoid memory copy in MLOAD opcode function

3dc802e

Remove console.log() in EVM

1a234ed

vmState: ensure touched accounts delete stack gets properly updated o…

7b422ad

…n commit

vm/eei: save touched height

464749b

vm/vmState: new possible fix for touched accounts

a0bfbc0

vm/vmState: another attempt to fix touched accounts journaling

a7f3df9

vm: add journaling

a5fb9be

holgerd77 force-pushed the vm-diff-based-touched-account-checkpointing branch from 4f58424 to a5fb9be Compare March 17, 2023 10:37

holgerd77 mentioned this pull request Mar 17, 2023

Statemanager: Cache Rewrite (Account/Storage) & Refactoring (Interface/Tests) (old) #2592

Closed

Check correct journal height on revert

a9bab81

acolytec3 approved these changes Mar 17, 2023

View reviewed changes

jochem-brouwer approved these changes Mar 17, 2023

View reviewed changes

jochem-brouwer merged commit f5ab7fc into master Mar 17, 2023

holgerd77 deleted the vm-diff-based-touched-account-checkpointing branch June 6, 2023 08:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VM: Diff-based Touched Accounts Checkpointing #2581

VM: Diff-based Touched Accounts Checkpointing #2581

holgerd77 commented Mar 13, 2023 •

edited

Loading

holgerd77 commented Mar 13, 2023

codecov bot commented Mar 13, 2023 •

edited

Loading

jochem-brouwer commented Mar 13, 2023

jochem-brouwer Mar 13, 2023

jochem-brouwer Mar 13, 2023

jochem-brouwer commented Mar 13, 2023

jochem-brouwer commented Mar 13, 2023

holgerd77 commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

acolytec3 left a comment

jochem-brouwer commented Mar 15, 2023

jochem-brouwer commented Mar 15, 2023

holgerd77 commented Mar 17, 2023

acolytec3 commented Mar 17, 2023

jochem-brouwer commented Mar 17, 2023

acolytec3 left a comment

VM: Diff-based Touched Accounts Checkpointing #2581

VM: Diff-based Touched Accounts Checkpointing #2581

Conversation

holgerd77 commented Mar 13, 2023 • edited Loading

holgerd77 commented Mar 13, 2023

codecov bot commented Mar 13, 2023 • edited Loading

Codecov Report

jochem-brouwer commented Mar 13, 2023

jochem-brouwer Mar 13, 2023

Choose a reason for hiding this comment

jochem-brouwer Mar 13, 2023

Choose a reason for hiding this comment

jochem-brouwer commented Mar 13, 2023

jochem-brouwer commented Mar 13, 2023

holgerd77 commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

jochem-brouwer commented Mar 14, 2023

acolytec3 left a comment

Choose a reason for hiding this comment

jochem-brouwer commented Mar 15, 2023

jochem-brouwer commented Mar 15, 2023

holgerd77 commented Mar 17, 2023

acolytec3 commented Mar 17, 2023

jochem-brouwer commented Mar 17, 2023

acolytec3 left a comment

Choose a reason for hiding this comment

holgerd77 commented Mar 13, 2023 •

edited

Loading

codecov bot commented Mar 13, 2023 •

edited

Loading