Fix e2e_2_pxes

[ludamad]

Failing in master frequently
had to skip "transfers funds from user A to B via PXE A followed by transfer from B to A via PXE B"

[ludamad]

@benesjan can you assign or self-assign someone to work on this, per your git blame?

[spalladino]

I looked briefly into this: in the failed run I got, the pxe-synchronizer updated the header while in the middle of a simulation. And this 17-month-old TODO suggests we should pause synching while simulating.

Eg this run worked:

  aztec:simulator:private_execution [VERBOSE] Executing external function SchnorrAccount:entrypoint@0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e +0ms
  aztec:pxe:simulator_oracle [VERBOSE] Found 1 logs for secret 0x2b17f4bed295f347d7f815968632becaa4f64c044302db6ccc146507c7b68103 as recipient 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e. Incrementing index to 1 at contract: SchnorrAccount(0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e) +65ms
  aztec:simulator:unconstrained_execution [VERBOSE] Executing unconstrained function 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e:0x00000000(compute_note_hash_and_optionally_a_nullifier) +0ms
  aztec:pxe:simulator_oracle [VERBOSE] Added incoming note for contract 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e at slot 0x0000000000000000000000000000000000000000000000000000000000000001 with nullifier 0x171139759a51648147deb4d4d775095c860b18e57e5221502871a8a568d1b66b +43ms
  aztec:pxe:simulator_oracle [VERBOSE] Added outgoing note for contract 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e at slot 0x0000000000000000000000000000000000000000000000000000000000000001 +0ms
  aztec:simulator:private_execution [VERBOSE] Executing external function Token:transfer@0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce +0ms
  aztec:pxe:simulator_oracle [VERBOSE] Found 1 logs for secret 0x0c6c699f89f0c41172b2bf40c19fe9aca84e86469406e671cfb78ad286d665b2 as recipient 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e. Incrementing index to 1 at contract: Token(0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce) +80ms
  aztec:simulator:unconstrained_execution [VERBOSE] Executing unconstrained function 0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce:0x00000000(compute_note_hash_and_optionally_a_nullifier) +0ms
  aztec:simulator:unconstrained_execution [VERBOSE] Executing unconstrained function 0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce:0x00000000(compute_note_hash_and_optionally_a_nullifier) +0ms
  aztec:pxe:simulator_oracle [VERBOSE] Added incoming note for contract 0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce at slot 0x08e33a0afb8db56e1cf31e8121bc9c56ae0c1d55d6a4167a0063ec9479aa8094 with nullifier 0x03493dc5151f2722d1aedae699ee1544c9c25079b8362dfc0702fda9d7db9884 +62ms
  aztec:sequencer [VERBOSE] Sequencer sync check succeeded worldStateNumber=6 worldStateHash=0x1a75212ddf067bba687d1c0a47a3dfd795adda66631b845d300b882dbc192624 l2BlockSourceNumber=6 l2BlockSourceHash=0x1a75212ddf067bba687d1c0a47a3dfd795adda66631b845d300b882dbc192624 p2pNumber=6 l1ToL2MessageSourceNumber=6 +1s
  aztec:pxe:simulator_oracle [VERBOSE] Incrementing secret 0x13ba2ef1840b3a1ce4c29600937ee0d7c7eaceb0f0389f73845f36bf1e0dd3e9 as sender 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e for recipient: 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e at contract: Token(0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce) +62ms
  aztec:sequencer [VERBOSE] Can propose block 7 at slot 9 +6ms
  aztec:pxe:simulator_oracle [VERBOSE] Incrementing secret 0x0c6c699f89f0c41172b2bf40c19fe9aca84e86469406e671cfb78ad286d665b2 as sender 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e for recipient: 0x1a3962241b3f9e738dd26ca750a78fd15ef655ed9294c7085fe46792deb729a3 at contract: Token(0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce) +55ms
  aztec:pxe:simulator_oracle [VERBOSE] Incrementing secret 0x0c6c699f89f0c41172b2bf40c19fe9aca84e86469406e671cfb78ad286d665b2 as sender 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e for recipient: 0x1a3962241b3f9e738dd26ca750a78fd15ef655ed9294c7085fe46792deb729a3 at contract: Token(0x0c7eef44875d1ea36beb2796ba014a95ecd4f6c74ecf6598c8e92648b7aa37ce) +40ms
  aztec:pxe_service_e6e454 [VERBOSE] Simulation completed for 0x2788edc063f57df429fd75f8b2140dc62be25e076c9099c6e66d4e73dc28352e:entrypoint +455ms

And this run failed:

  aztec:simulator:private_execution [VERBOSE] Executing external function SchnorrAccount:entrypoint@0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab +0ms
  aztec:pxe:simulator_oracle [VERBOSE] Found 1 logs for secret 0x0a815f0ad41232389a7ed581433efd6c925487a513d34cc748453b216d76604e as recipient 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab. Incrementing index to 1 at contract: SchnorrAccount(0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab) +64ms
  aztec:simulator:unconstrained_execution [VERBOSE] Executing unconstrained function 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab:0x00000000(compute_note_hash_and_optionally_a_nullifier) +0ms
  aztec:pxe:simulator_oracle [VERBOSE] Added incoming note for contract 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab at slot 0x0000000000000000000000000000000000000000000000000000000000000001 with nullifier 0x29a682a0918e47b5f3a3a2cda8ff34cf7d147b06bf22d7061b5873ea020faf0d +47ms
  aztec:pxe:simulator_oracle [VERBOSE] Added outgoing note for contract 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab at slot 0x0000000000000000000000000000000000000000000000000000000000000001 +0ms
  aztec:simulator:private_execution [VERBOSE] Executing external function Token:transfer@0x08bf6dec356912a02c989da751d8f6b8d1a85068e63bc4fabf50e59aa4f8b58b +0ms
  aztec:archiver [VERBOSE] Retrieved no new L1 -> L2 messages between L1 blocks 27 and 27. +1s
  aztec:pxe_synchronizer_160fd7 [VERBOSE] Processing blocks 6 to 6 +4s
  aztec:pxe:simulator_oracle [VERBOSE] Found 1 logs for secret 0x179de997d178beac65ab2f7eeff6fad0e881a6ea659919ded2203a529625676e as recipient 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab. Incrementing index to 1 at contract: Token(0x08bf6dec356912a02c989da751d8f6b8d1a85068e63bc4fabf50e59aa4f8b58b) +106ms
  aztec:archiver [VERBOSE] Updating the proven block number to 6 and epoch to 0 +9ms
  aztec:archiver [VERBOSE] No blocks to retrieve from 27 to 27 +1ms
  aztec:pxe:simulator_oracle [VERBOSE] Incrementing secret 0x11404865509713b4eb77d2ef1b3825860edeb50424ccec79d491cc21e721aa0a as sender 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab for recipient: 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab at contract: Token(0x08bf6dec356912a02c989da751d8f6b8d1a85068e63bc4fabf50e59aa4f8b58b) +67ms
  aztec:sequencer [VERBOSE] Sequencer sync check succeeded worldStateNumber=6 worldStateHash=0x0c7358919fc2fb950c47a7d7f32f404e1c02518d45ef02a26eaf24176d039c15 l2BlockSourceNumber=6 l2BlockSourceHash=0x0c7358919fc2fb950c47a7d7f32f404e1c02518d45ef02a26eaf24176d039c15 p2pNumber=6 l1ToL2MessageSourceNumber=6 +1s
  aztec:world_state [VERBOSE] Chain proven at block 6 +1s
  aztec:world_state [VERBOSE] Chain finalized at block 6 +0ms
  aztec:sequencer [VERBOSE] Can propose block 7 at slot 9 +19ms
  aztec:pxe:simulator_oracle [VERBOSE] Incrementing secret 0x179de997d178beac65ab2f7eeff6fad0e881a6ea659919ded2203a529625676e as sender 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab for recipient: 0x1ba62689268fdab7eb59f51a421d285b4076926abe0f9a0d9c725e63458ba192 at contract: Token(0x08bf6dec356912a02c989da751d8f6b8d1a85068e63bc4fabf50e59aa4f8b58b) +69ms
  aztec:pxe:simulator_oracle [VERBOSE] Incrementing secret 0x179de997d178beac65ab2f7eeff6fad0e881a6ea659919ded2203a529625676e as sender 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab for recipient: 0x1ba62689268fdab7eb59f51a421d285b4076926abe0f9a0d9c725e63458ba192 at contract: Token(0x08bf6dec356912a02c989da751d8f6b8d1a85068e63bc4fabf50e59aa4f8b58b) +39ms
  aztec:pxe_service_160fd7 [VERBOSE] Simulation completed for 0x0d79e7712436cf0fd4c869cdb4b4fa981ba5b44aba40dcead6e495775118b1ab:entrypoint +440ms
  aztec:pxe_service_160fd7 [ERROR] Error: Circuit execution failed: membership check failed
    at Object.<anonymous>.module.exports.__wbg_constructor_f8f83aa6ec3644b9 (/usr/src/noir/packages/acvm_js/nodejs/acvm_js.js:666:17)
    at acvm_js::js_execution_error::JsExecutionError::new::hebdb1fd5d8918eaa (wasm://wasm/0097ce1e:1:1275824)
    at acvm_js::execute::ProgramExecutor<B>::execute_circuit::{{closure}}::hdef4865f37ab532b (wasm://wasm/0097ce1e:1:431725)
    at acvm_js::execute::execute_program_with_native_program_and_return::{{closure}}::h4055275b2107668e (wasm://wasm/0097ce1e:1:1110308)
    at acvm_js::execute::execute_program_with_native_type_return::{{closure}}::hfd113d3fb49d29e9 (wasm://wasm/0097ce1e:1:1173605)
    at wasm_bindgen_futures::future_to_promise::{{closure}}::{{closure}}::h0452d56dcba8f2f3 (wasm://wasm/0097ce1e:1:1147352)
    at wasm_bindgen_futures::queue::Queue::new::{{closure}}::he31c90183f9be757 (wasm://wasm/0097ce1e:1:1320490)
    at <dyn core::ops::function::FnMut<(A,)>+Output = R as wasm_bindgen::closure::WasmClosure>::describe::invoke::h624a3dfbf78ca22a (wasm://wasm/0097ce1e:1:1670461)
    at __wbg_adapter_22 (/usr/src/noir/packages/acvm_js/nodejs/acvm_js.js:220:10)
    at real (/usr/src/noir/packages/acvm_js/nodejs/acvm_js.js:205:20) {
  callStack: [Array],
  rawAssertionPayload: [Object],
  brilligFunctionId: 0
} +435ms

Not sure if it's worth trying to fix given syncing is going away, but I'll leave that with the fairies. Copying @nventuro since @benesjan is on vacation.

[Thunkar]

Probably similar to what was happening here: #10316

I'll take this!