Remaining Hive Cancun Failures

[spencer-tb]

Description

I'm tagging the last failures here for the hive cancun tests. Specifically from the engine-cancun simulator.

(note: on hivecancun all other failures pass locally and are due to something internal to the CI).

ForkchoiceUpdatedV3 To Request Shanghai Payload, Null Beacon Root
ForkchoiceUpdatedV2 To Request Shanghai Payload, Non-Null Beacon Root
ForkchoiceUpdatedV2 To Request Cancun Payload, Non-Null Beacon Root
Invalid PayloadAttributes, Missing BeaconRoot, Syncing=False
Invalid PayloadAttributes, Missing BeaconRoot, Syncing=True
Inconsistent Head in ForkchoiceState
Inconsistent Safe in ForkchoiceState
Inconsistent Finalized in ForkchoiceState
Valid NewPayload->ForkchoiceUpdated on Syncing Client

For the first 5 tests:

ForkchoiceUpdatedV3 To Request Shanghai Payload, Null Beacon Root
ForkchoiceUpdatedV2 To Request Shanghai Payload, Non-Null Beacon Root
ForkchoiceUpdatedV2 To Request Cancun Payload, Non-Null Beacon Root
Invalid PayloadAttributes, Missing BeaconRoot, Syncing=False
Invalid PayloadAttributes, Missing BeaconRoot, Syncing=True

These can all be explained from the execution apis spec update in ethereum/execution-apis#498.
We are essentially sending an fcu with invalid payload attributes and hence expect an error reponse of INVALID_PAYLOAD_ATTRIBUTES (-38003). Ethereumjs is returning INVALID_PARAMS (-32602). The only exception is for the last case where Syncing=True, where we expect no error response, as from the execution apis PR above, if a client is syncing no payload attributes validations should be ran.

For these 3 tests:

Inconsistent Head in ForkchoiceState
Inconsistent Safe in ForkchoiceState
Inconsistent Finalized in ForkchoiceState

As we are sending an fcu, where the head, safe and finalized block hashes are inconistent with the previous - we expect and error response of INVALID_FORKCHOICE_STATE (-38002). Ethereumjs is returning INVALID_PARAMS (-32602).

For the last test its a bit more complex:

Valid NewPayload->ForkchoiceUpdated on Syncing Client

We are essentially testing what happens when we send a valid payload to a client that is syncing. The predominant part of the test is successful in ethereumjs, but its what happens at the end of the test where we get the failure (hive test code).

Initial successful test steps:

• At first we start with 2 clients, with a focus on testing the "primary" client. At the start of the test both clients are up to date with the first 5 blocks, where the primary client is the block producer. Here is are last fcu's:
  • Primary client: fcuv3(lastest=0xad62..., safe=0x3175..., finalized=0xafef...) -> VALID
  • Secondary client: fcuv3(lastest=0xad62..., safe=0x3175..., finalized=0xafef...) -> VALID
• We then disconnect the primary client, the secondary becomes the block producer and produces a new block. The secondary client then executes this block 0xf1ef... (newPayload). Both clients then forkchoice update to it, where the secondary client is successful (VALID) and then primary is not (SYNCING).
  • Primary client: fcuv3(lastest=0xf1ef..., safe=0xad62..., finalized=0x3175...) -> SYNCING
  • Secondary client: fcuv3(lastest=0xf1ef..., safe=0xad62..., finalized=0x3175...) -> VALID
• A new block is produced from the secondary (valid newpayload), and attempted to execute on the primary client 0x953a.... It won't be able to validate it (ACCEPTED/SYNCING). We then try to fcu with a reference to this payload (SYNCING).
  • Primary client: enpv3(...blockhash=0x953a...) -> ACCEPTED
  • Primary client: fcuv3(lastest=0x953a..., safe=0x953a..., finalized=0x953a...) -> SYNCING
• We then send the previous payload 0xf1ef... to the primary client (newPayload) expecting a VALID response.
  • Primary client: enpv3(...blockhash=0xf1ef...) -> VALID
• This is then followed by the new block payload again 0x953a... on both clients. Both clients then fcu to this latest payload (VALID).
  • Primary client: enpv3(...blockhash=0x953a...) -> VALID
  • Primary client: fcuv3(lastest=0x953a..., safe=0x953a..., finalized=0x953a...) -> VALID
  • Secondary client: enpv3(...blockhash=0x953a...) -> VALID
  • Secondary client: fcuv3(lastest=0x953a..., safe=0xf1ef..., finalized=0xad62...)) -> VALID

*Failing test step:

When we disconnect the secondary client at the end of the test, the primary becomes the block producer. At this stage the CL mock sends an fcuv3 to primary client with the previous forkchoice state information:

• Primary client: fcuv3(lastest=0x953a..., safe=0xf1ef..., finalized=0xad62...)) -> SYNCING

This should not be SYNCING, it should be VALID, and is where the failure in the test resides.

Here are the logs for ethereumjs and the logs for geth (passing client) as reference.

Note the last stage at the bottom of the logs where we remove the secondary client.

Steps to reproduce

Using the latest version of hive run the following command:

./hive --sim ethereum/engine --client ethereumjs --client-file configs/cancun.yaml --docker.output --sim.limit engine-cancun/"<test_name>"

[holgerd77]

Can we close this?

[jochem-brouwer]

I am rather sure this is not yet resolved (I apparently have a draft PR open 😅 )

[spencer-tb]

We can take a look later this week too! We are updating/migrating some of the hive CI this week :)

[holgerd77]

Still a lot of hive-cancun tests failing on our side:
https://hivecancun.ethdevops.io/suite.html?suiteid=1726820541-5dd90b36c00d30a4cc553d1f8748dc56.json&suitename=engine-cancun

Likely worth to look into that, since it will increase the robustness of our client.