Various Hive related fixes #2532

acolytec3 · 2023-02-15T20:33:21Z

New fixes related to Hive tests

Adds a new --loadBlocksFromRlp CLI parameter that allows for loading RLP encoded blocks on chain start that are fed via the ./chain.rlp path in various Hive tests. Needed by the Hive tests
Add better handling of reorgs that go pre-merge -- fixes hive tests run by this command - go run ./hive.go --client ethereumjs --sim ethereum/engine --sim.limit "engine-transition/Single Block PoW Re-org to Higher-Total-Difficulty Chain, Equal Height"
Skip extradata check on PoW genesis blocks (since some Hive tests with PoW genesis parameters have extradata that does not conform to PoW extradata requirements

codecov · 2023-02-15T20:37:45Z

Codecov Report

Merging #2532 (73095a8) into master (1a7094d) will decrease coverage by 1.09%.
The diff coverage is 78.21%.

❗ Current head 73095a8 differs from pull request most recent head e980e76. Consider uploading reports for the commit e980e76 to get more accurate results

Additional details and impacted files

Flag	Coverage Δ
block	`90.13% <100.00%> (+0.01%)`	⬆️
blockchain	`?`
client	`87.64% <77.77%> (-0.06%)`	⬇️
common	`?`
devp2p	`?`
ethash	`∅ <ø> (∅)`
evm	`83.36% <ø> (-0.01%)`	⬇️
rlp	`?`
statemanager	`?`
trie	`?`
tx	`93.71% <ø> (ø)`
util	`?`
vm	`83.87% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

acolytec3 · 2023-02-18T11:22:50Z

Additional things needed to properly run this code with hive:

Replace the dockerfile in hive/clients/ethereumjs/Dockerfile with below which adds the new reading of ./chain.rlp

FROM node:18-alpine as build
RUN apk update && apk add --no-cache bash git jq curl python3 gcc make g++ rsync && rm -rf /var/cache/apk/*

RUN mkdir /ethereumjs-monorepo
WORKDIR /ethereumjs-monorepo

COPY ethereumjs-monorepo/package.json .
COPY ethereumjs-monorepo/package-lock.json .
RUN npm i --ignore-scripts

COPY ethereumjs-monorepo-master/ ./
RUN npm i

COPY ethereumjs-monorepo /ejsrepo
RUN rsync -a /ejsrepo/* .
RUN git status
RUN npm i

WORKDIR /

ADD ethereumjs.sh /ethereumjs.sh
ADD mapper.jq /mapper.jq
ADD jwtsecret /jwtsecret
RUN chmod +x /ethereumjs.sh


# Inject the enode id retriever script
RUN mkdir /hive-bin
ADD enode.sh /hive-bin/enode.sh
RUN chmod +x /hive-bin/enode.sh

ADD genesis.json /genesis.json

# Export the usual networking ports to allow outside access to the node
EXPOSE 8545 8546 8551 8547 30303 30303/udp

# NodeJS applications have a default memory limit of 2.5GB.
# This limit is bit tight, it is recommended to raise the limit
# since memory may spike during certain network conditions.
ENV NODE_OPTIONS=--max_old_space_size=6144

ENTRYPOINT ["/ethereumjs.sh"]

Replace hive/clients/ethereumjs/ethereumjs.sh with this:

#!/bin/bash

# Startup script to initialize and boot a ethereum-js instance.
#
# This script assumes the following files:
#  - `genesis.json` file is located in the filesystem root (mandatory)
#  - `chain.rlp` file is located in the filesystem root (optional)
#  - `blocks` folder is located in the filesystem root (optional)
#  - `keys` folder is located in the filesystem root (optional)
#
# This script assumes the following environment variables:
#
#  - HIVE_BOOTNODE                enode URL of the remote bootstrap node
#  - HIVE_NETWORK_ID              network ID number to use for the eth protocol
#  - HIVE_TESTNET                 whether testnet nonces (2^20) are needed
#  - HIVE_NODETYPE                sync and pruning selector (archive, full, light)
#
# Forks:
#
#  - HIVE_FORK_HOMESTEAD          block number of the homestead hard-fork transition
#  - HIVE_FORK_DAO_BLOCK          block number of the DAO hard-fork transition
#  - HIVE_FORK_DAO_VOTE           whether the node support (or opposes) the DAO fork
#  - HIVE_FORK_TANGERINE          block number of Tangerine Whistle transition
#  - HIVE_FORK_SPURIOUS           block number of Spurious Dragon transition
#  - HIVE_FORK_BYZANTIUM          block number for Byzantium transition
#  - HIVE_FORK_CONSTANTINOPLE     block number for Constantinople transition
#  - HIVE_FORK_PETERSBURG         block number for ConstantinopleFix/PetersBurg transition
#  - HIVE_FORK_ISTANBUL           block number for Istanbul transition
#  - HIVE_FORK_MUIRGLACIER        block number for Muir Glacier transition
#  - HIVE_FORK_BERLIN             block number for Berlin transition
#  - HIVE_FORK_LONDON             block number for London
#
# Clique PoA:
#
#  - HIVE_CLIQUE_PERIOD           enables clique support. value is block time in seconds.
#  - HIVE_CLIQUE_PRIVATEKEY       private key for clique mining
#
# Other:
#
#  - HIVE_MINER                   enable mining. value is coinbase address.
#  - HIVE_MINER_EXTRA             extra-data field to set for newly minted blocks
#  - HIVE_SKIP_POW                if set, skip PoW verification during block import
#  - HIVE_LOGLEVEL                client loglevel (0-5)
#  - HIVE_GRAPHQL_ENABLED         enables graphql on port 8545
#  - HIVE_LES_SERVER              set to '1' to enable LES server


# Immediately abort the script on any error encountered
set -e

ethereumjs="node /ethereumjs-monorepo/packages/client/dist/bin/cli.js"
FLAGS="--gethGenesis ./genesis.json --rpc --rpcEngine --saveReceipts --rpcEnginePort 8551 --ws --logLevel debug --rpcDebug --transports rlpx --isSingleNode"


# Configure the chain.
mv /genesis.json /genesis-input.json
jq -f /mapper.jq /genesis-input.json > /genesis.json

# Dump genesis
echo "Supplied genesis state:"
cat /genesis.json

# Import clique signing key.
if [ "$HIVE_CLIQUE_PRIVATEKEY" != "" ]; then
    # Create password file.
    echo "Importing clique key..."
    echo "$HIVE_CLIQUE_PRIVATEKEY" > ./private_key.txt
    # Ensure password file is used when running ethereumjs in mining mode.
    if [ "$HIVE_MINER" != "" ]; then
        FLAGS="$FLAGS --mine --unlock ./private_key.txt --minerCoinbase $HIVE_MINER"
    fi
fi

if [ "$HIVE_TERMINAL_TOTAL_DIFFICULTY" != "" ]; then
    FLAGS="$FLAGS --jwt-secret ./jwtsecret"
fi


# Load the test chain if present
echo "Loading initial blockchain..."
if [ -f /chain.rlp ]; then
  FLAGS="$FLAGS --loadBlocksFromRlp=/chain.rlp"
  else
  echo "Warning: chain.rlp not found."
fi

FLAGS="$FLAGS --bootnodes=$HIVE_BOOTNODE"
echo "Running ethereumjs with flags $FLAGS"
$ethereumjs $FLAGS

acolytec3 · 2023-02-18T11:25:11Z

Use this command -- go run ./hive.go --client ethereumjs --sim ethereum/engine --sim.limit "engine-transition/Single Block PoW Re-org to Higher-Total-Difficulty Chain, Equal Height" --docker.output to run one of the engine-transition tests that loads block data on client start. It's currently failing because we don't handle the reorg case correctly when going back from PoS to a different PoW terminal block.

jochem-brouwer

Some remarks :)

jochem-brouwer · 2023-02-18T12:40:39Z

packages/block/src/header.ts

@@ -354,8 +354,10 @@ export class BlockHeader {
    if (this._common.consensusAlgorithm() === ConsensusAlgorithm.Ethash) {
      // PoW/Ethash
      if (
+        number > BigInt(0) &&


Ok, this disables genesis extradata check by default, I am not sure if we want this?

This could be tricky since this check is in common and doesn't have access to the clients config flags that would indicate a test environment. We could have the hive simulator remove the extradata field if we're on PoW and that should circumvent the issue

jochem-brouwer · 2023-02-18T12:41:32Z

packages/client/bin/cli.ts

@@ -292,6 +295,10 @@ const args: ClientOpts = yargs(hideBin(process.argv))
      'To run client in single node configuration without need to discover the sync height from peer. Particularly useful in test configurations. This flag is automically activated in the "dev" mode',
    boolean: true,
  })
+  .option('loadBlocksFromRlp', {
+    describe: 'path to a file of RLP encoded blocks',


We should take a look how other clients import these blocks. I would be more in favor to only allow this if some kind of "test network" flag is provided.

It's pretty similar. They look for a specific flag and load them if so. Why would this be an issue for us as currently designed? This flag never gets set for a production run.

jochem-brouwer · 2023-02-18T12:42:25Z

packages/client/bin/cli.ts

+          }`
+        )
+      } catch {
+        config.logger.info('Encountered invalid block while preloading chain')


If this is the case, should we not immediately quit the client, instead of proceeding?

https://github.com/ethereum/hive/blob/master/docs/clients.md No, we're supposed to import to to the last valid block, per the hive docs

jochem-brouwer · 2023-02-18T12:52:41Z

packages/client/bin/cli.ts

+        )
+      } catch {
+        config.logger.info('Encountered invalid block while preloading chain')
+        done = true


Ahh, yes, that should have been obvious to me. Will update.

holgerd77 · 2023-02-20T09:03:28Z

Ok, I would regard this as too much work-in-progress as to consider for tomorrows releases.

Please do not merge before releases are published.

g11tech

lgtm!

holgerd77

Looks good

holgerd77

LGTM

acolytec3 added PR state: WIP package: client labels Feb 15, 2023

jochem-brouwer reviewed Feb 18, 2023

View reviewed changes

g11tech added PR state: needs review and removed PR state: WIP labels Feb 20, 2023

g11tech force-pushed the preload-blocks-on-client-start branch from 4c39892 to 7edeab8 Compare February 20, 2023 09:46

g11tech added the package: blockchain label Feb 20, 2023

g11tech previously approved these changes Feb 20, 2023

View reviewed changes

acolytec3 changed the title ~~Add new client param for preloading blocks on client start~~ Various Hive related fixes Feb 21, 2023

acolytec3 mentioned this pull request Feb 21, 2023

Add support for loading chain.rlp to ethereumjs ethereum/hive#729

Merged

holgerd77 previously approved these changes Feb 21, 2023

View reviewed changes

acolytec3 and others added 12 commits February 21, 2023 12:26

add block preload parameter

6ff0515

Skip extradata check on genesis pow blocks

64f0daf

Reconfigure block preloading

cc0f93a

open client

341f99c

Fix chain rlp loading

65d42ea

fix resetting canonical head on reorg scenarios

937b242

remove unnecessary hf setting in chain putblocks

fa90e89

add and use reset canonical head for chain

dd5e902

fix blockchain test

381f61a

fix breaking skeleton spec

34d7910

fix reset canonical header and add fix

1c488dc

Adjust loop exit check and logging

3f00fc4

holgerd77 force-pushed the preload-blocks-on-client-start branch from 73095a8 to 3f00fc4 Compare February 21, 2023 11:26

add jsdoc

e980e76

g11tech dismissed holgerd77’s stale review via e980e76 February 21, 2023 11:34

g11tech dismissed their stale review via e980e76 February 21, 2023 11:34

holgerd77 approved these changes Feb 21, 2023

View reviewed changes

holgerd77 merged commit 8cd95e5 into master Feb 21, 2023

holgerd77 deleted the preload-blocks-on-client-start branch February 21, 2023 11:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various Hive related fixes #2532

Various Hive related fixes #2532

acolytec3 commented Feb 15, 2023 •

edited

Loading

codecov bot commented Feb 15, 2023 •

edited

Loading

acolytec3 commented Feb 18, 2023

acolytec3 commented Feb 18, 2023

jochem-brouwer left a comment

jochem-brouwer Feb 18, 2023

acolytec3 Feb 18, 2023

jochem-brouwer Feb 18, 2023

acolytec3 Feb 18, 2023

jochem-brouwer Feb 18, 2023

acolytec3 Feb 18, 2023

jochem-brouwer Feb 18, 2023

acolytec3 Feb 18, 2023

holgerd77 commented Feb 20, 2023

g11tech left a comment

holgerd77 left a comment

holgerd77 left a comment

Various Hive related fixes #2532

Various Hive related fixes #2532

Conversation

acolytec3 commented Feb 15, 2023 • edited Loading

codecov bot commented Feb 15, 2023 • edited Loading

Codecov Report

acolytec3 commented Feb 18, 2023

acolytec3 commented Feb 18, 2023

jochem-brouwer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holgerd77 commented Feb 20, 2023

g11tech left a comment

Choose a reason for hiding this comment

holgerd77 left a comment

Choose a reason for hiding this comment

holgerd77 left a comment

Choose a reason for hiding this comment

acolytec3 commented Feb 15, 2023 •

edited

Loading

codecov bot commented Feb 15, 2023 •

edited

Loading