Replica not in sync with mainnet.optimism.io

[Enigmatic331]

Describe the bug
A transaction deploying a contract to mainnet.optimism.io was done (0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef) - When querying from mainnet.optimism.io I can see that the transaction receipt is valid with status 0x1, however on my replica the transaction receipt returns 0x0.

mainnet.optimism.io

replica:

To Reproduce
Steps to reproduce the behavior:

1. Clone develop branch
2. yarn, yarn build, make l2geth
3. I am using the below settings for the .env and startup script for l2geth (likely I might be missing something obvious on these)

.env

DATA_TRANSPORT_LAYER__DB_PATH=./db
DATA_TRANSPORT_LAYER__SERVER_HOSTNAME=0.0.0.0
DATA_TRANSPORT_LAYER__SERVER_PORT=7878
DATA_TRANSPORT_LAYER__ADDRESS_MANAGER=0x668E5b997b9aE88a56cd40409119d4Db9e6d752E
DATA_TRANSPORT_LAYER__L1_START_HEIGHT=12410727
DATA_TRANSPORT_LAYER__L1_RPC_ENDPOINT=<L1 URL>
DATA_TRANSPORT_LAYER__CONFIRMATIONS=20
DATA_TRANSPORT_LAYER__POLLING_INTERVAL=5000
DATA_TRANSPORT_LAYER__LOGS_PER_POLLING_INTERVAL=2000
DATA_TRANSPORT_LAYER__SYNC_FROM_L1=false
DATA_TRANSPORT_LAYER__SYNC_FROM_L2=true
DATA_TRANSPORT_LAYER__L2_RPC_ENDPOINT=https://mainnet.optimism.io
DATA_TRANSPORT_LAYER__L2_CHAIN_ID=10
DATA_TRANSPORT_LAYER__TRANSACTIONS_PER_POLLING_INTERVAL=1000
DATA_TRANSPORT_LAYER__DANGEROUSLY_CATCH_ALL_ERRORS=true
DATA_TRANSPORT_LAYER__LEGACY_SEQUENCER_COMPATIBILITY=false
DATA_TRANSPORT_LAYER__STOP_L2_SYNC_AT_BLOCK=9007199254740991
DATA_TRANSPORT_LAYER__DEFAULT_BACKEND=l2
L1_NODE_WEB3_URL=<L1 URL>

start.sh

#!/bin/bash

IS_VERIFIER=TRUE
DATADIR=c:/Optimism/node
ETH1_CHAIN_ID=1
TARGET_GAS_LIMIT=9000000
CHAIN_ID=10
ETH1_CTC_DEPLOYMENT_HEIGHT=12410807
ETH1_L1_GATEWAY_ADDRESS=0xe681F80966a8b1fFadECf8068bD6F99034791c95
ETH1_L1_CROSS_DOMAIN_MESSENGER_ADDRESS=0x902e5fF5A99C4eC1C21bbab089fdabE32EF0A5DF
ADDRESS_MANAGER_OWNER_ADDRESS=0x9BA6e03D8B90dE867373Db8cF1A58d2F7F006b3A
ROLLUP_STATE_DUMP_PATH=https://storage.googleapis.com/optimism/mainnet/4.json
ROLLUP_CLIENT_HTTP=http://localhost:7878
ROLLUP_POLL_INTERVAL=1s
ROLLUP_TIMESTAMP_REFRESH=3m
CACHE=1024
RPC_PORT=8545
VERBOSITY=3

USAGE="
Start the Sequencer or Verifier with most configuration pre-set.

CLI Arguments:
  -h|--help                              - help message
  -v|--verifier                          - start in verifier mode
  --datadir                              - data directory to use
  --chainid                              - layer two chain id to use, must match contracts on L1
  --eth1.chainid                         - eth1 chain id
  --eth1.ctcdeploymentheight             - eth1 ctc deploy height
  --eth1.l1crossdomainmessengeraddress   - eth1 l1 xdomain messenger address
  --eth1.ctcdeploymentheight             - eth1 ctc deployment height
  --rollup.statedumppath                 - http path to the initial state dump
  --rollup.clienthttp                    - rollup client http
  --rollup.pollinterval                  - polling interval for the rollup client
  --rollup.timestamprefresh              - timestamp refresh interval
  --cache                                - geth cache size
  --targetgaslimit                       - gas per block
"

while (( "$#" )); do
    case "$1" in
        -h|--help)
            echo "$USAGE"
            exit 0
            ;;
        -v|--verifier)
            IS_VERIFIER=true
            shift 1
            ;;
        --verbosity)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                VERBOSITY="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --datadir)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                DATADIR="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --chainid)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                CHAIN_ID="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --rpcport)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                RPC_PORT="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --eth1.chainid)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ETH1_CHAIN_ID="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --eth1.ctcdeploymentheight)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ETH1_CTC_DEPLOYMENT_HEIGHT="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --eth1.l1gatewayaddress)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ETH1_L1_GATEWAY_ADDRESS="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --eth1.l1crossdomainmessengeraddress)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ETH1_L1_CROSS_DOMAIN_MESSENGER_ADDRESS="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --eth1.ctcdeploymentheight)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ADDRESS_MANAGER_OWNER_ADDRESS="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --rollup.statedumppath)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ROLLUP_STATE_DUMP_PATH="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --rollup.clienthttp)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ROLLUP_CLIENT_HTTP="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --rollup.pollinterval)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ROLLUP_POLL_INTERVAL="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --rollup.timestamprefresh)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                ROLLUP_TIMESTAMP_REFRESH="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --cache)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                CACHE="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        --targetgaslimit)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
                TARGET_GASLIMIT="$2"
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        *)
            echo "Unknown argument $1" >&2
            shift
            ;;
    esac
done

cmd="./geth.exe"
cmd="$cmd --eth1.syncservice"
cmd="$cmd --datadir $DATADIR"
cmd="$cmd --rollup.maxcalldatasize=40000"
cmd="$cmd --eth1.chainid $ETH1_CHAIN_ID"
cmd="$cmd --networkid $CHAIN_ID"
cmd="$cmd --eth1.l1crossdomainmessengeraddress $ETH1_L1_CROSS_DOMAIN_MESSENGER_ADDRESS"
cmd="$cmd --rollup.addressmanagerowneraddress $ADDRESS_MANAGER_OWNER_ADDRESS"
cmd="$cmd --rollup.statedumppath $ROLLUP_STATE_DUMP_PATH"
cmd="$cmd --eth1.ctcdeploymentheight $ETH1_CTC_DEPLOYMENT_HEIGHT"
if [[ ! -z $ETH1_L1_GATEWAY_ADDRESS ]]; then
    cmd="$cmd --eth1.l1ethgatewayaddress $ETH1_L1_GATEWAY_ADDRESS"
fi
cmd="$cmd --rollup.clienthttp $ROLLUP_CLIENT_HTTP"
cmd="$cmd --rollup.pollinterval $ROLLUP_POLL_INTERVAL"
cmd="$cmd --rollup.timestamprefresh $ROLLUP_TIMESTAMP_REFRESH"
cmd="$cmd --cache $CACHE"
cmd="$cmd --rpc"
cmd="$cmd --dev"
cmd="$cmd --chainid $CHAIN_ID"
cmd="$cmd --rpcaddr 0.0.0.0"
cmd="$cmd --rpcport $RPC_PORT"
cmd="$cmd --rpcvhosts '*'"
cmd="$cmd --rpccorsdomain '*'"
cmd="$cmd --rpcvhosts '*'"
cmd="$cmd --wsaddr 0.0.0.0"
cmd="$cmd --wsport 8546"
cmd="$cmd --wsorigins '*'"
cmd="$cmd --rpcapi 'eth,net,rollup,web3,debug'"
cmd="$cmd --gasprice 0"
cmd="$cmd --nousb"
cmd="$cmd --gcmode=archive"
cmd="$cmd --ipcdisable"
if [[ ! -z "$IS_VERIFIER" ]]; then
    cmd="$cmd --rollup.verifier"
fi
cmd="$cmd --verbosity=$VERBOSITY"

echo -e "Running:\nTARGET_GAS_LIMIT=$TARGET_GAS_LIMIT USING_OVM=true $cmd"
eval env TARGET_GAS_LIMIT=$TARGET_GAS_LIMIT USING_OVM=true $cmd

4. Query for transaction receipt on 0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef

Currently still digging about to see what else I may be missing... Any ideas will be very much appreciated 🙇

[tynes]

A fix to the DTL released here: https://github.com/ethereum-optimism/optimism/releases/tag/%40eth-optimism%2Fdata-transport-layer%400.3.1

This patch fixed the problems that I was experiencing locally. Let me know if that fixes the problem for you

[tynes]

Can you share the exact git commit that you are using? Can you confirm that the state roots are matching for the genesis block?

[tynes]

Can reproduce locally:

diff \
    <(ETH_RPC_URL=http://127.0.0.1:8145 seth tx 0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef)
    <(ETH_RPC_URL=https://mainnet.optimism.io seth tx 0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef)

[Enigmatic331]

Ah sorry Mark was away tinkering around with Kovan+Develop branch earlier and missed your replies...

I did a fresh setup 2 hours back so I guess I was on commit cb2e9ad

Should I go ahead with trying DTL v0.3.1?

Thanks a ton!

[tynes]

Hey @Enigmatic331, the fix should be here: #881

Going to get a release out asap after this goes through code review

[tynes]

Just created new releases:

• https://github.com/ethereum-optimism/optimism/releases/tag/%40eth-optimism%2Fl2geth%400.3.3
• https://github.com/ethereum-optimism/optimism/releases/tag/%40eth-optimism%2Fdata-transport-layer%400.3.2

Feel free to reopen if the problem persists

[Enigmatic331]

Thanks so much @tynes!

One question - Albeit I know blockHash isn't really a thing but I'm worried it'd impact other areas of the database - Is it alright that the receipt is returning a different blockHash (mainnet.optimism.io being 0xbc5658f9eff19648ac60664880fcd2cbe5f595c7c944852610c056ee79cd4365 vs 0x18ebee690c2b482c528d394abe05331bb805af62af053bd8154c0b2ac454a7c8)?

Data below is what is returned from getTransactionReceipt for 0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef on my replica:

   "jsonrpc":"2.0",
   "id":1,
   "result":{
      "blockHash":"0x18ebee690c2b482c528d394abe05331bb805af62af053bd8154c0b2ac454a7c8",
      "blockNumber":"0x1842",
      "contractAddress":"0x5aec87824aab25a2157df405975003c6fd2ae319",
      "cumulativeGasUsed":"0x2853fd",
      "from":"0x3d080421c9dd5fb387d6e3124f7e1c241ade9568",
      "gasUsed":"0x2853fd",
      "logs":[
         {
            "address":"0x4200000000000000000000000000000000000006",
            "topics":[
               "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
               "0x0000000000000000000000003d080421c9dd5fb387d6e3124f7e1c241ade9568",
               "0x0000000000000000000000004200000000000000000000000000000000000005"
            ],
            "data":"0x0000000000000000000000000000000000000000000000000000000000000000",
            "blockNumber":"0x1842",
            "transactionHash":"0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef",
            "transactionIndex":"0x0",
            "blockHash":"0x18ebee690c2b482c528d394abe05331bb805af62af053bd8154c0b2ac454a7c8",
            "logIndex":"0x0",
            "removed":false
         },
         {
            "address":"0x5aec87824aab25a2157df405975003c6fd2ae319",
            "topics":[
               "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
               "0x0000000000000000000000000000000000000000000000000000000000000000",
               "0x0000000000000000000000003d080421c9dd5fb387d6e3124f7e1c241ade9568"
            ],
            "data":"0x0000000000000000000000000000000000000000000000302379bf2ca2e00000",
            "blockNumber":"0x1842",
            "transactionHash":"0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef",
            "transactionIndex":"0x0",
            "blockHash":"0x18ebee690c2b482c528d394abe05331bb805af62af053bd8154c0b2ac454a7c8",
            "logIndex":"0x1",
            "removed":false
         }
      ],
      "logsBloom":"0x00000000000000000000000200000000000000000000000000040000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000009040000000000000010000000000000000000000400000000020000000000000000008800000000000000000000000010000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000008000000000000000000000020000000000000000000000010000000000000000000000000000000000000000000",
      "status":"0x1",
      "to":null,
      "transactionHash":"0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef",
      "transactionIndex":"0x0"
   }
}

[Enigmatic331]

The stateRoot does agree though, so not sure if this is an issue:

mainnet.optimism.io

{
   "jsonrpc":"2.0",
   "id":1,
   "result":{
      "difficulty":"0x2",
      "extraData":"0xd98301090a846765746889676f312e31342e3135856c696e75780000000000004c68e5fd3a42875b3e89cdd1f47dd574386fa9e42043efad3c9e046ff4797c262cb52383c1dac1cf2a732b20f2cf149782577b16741eff779326fc65d769cd0f01",
      "gasLimit":"0x895440",
      "gasUsed":"0x2853fd",
      "hash":"0xbc5658f9eff19648ac60664880fcd2cbe5f595c7c944852610c056ee79cd4365",
      "logsBloom":"0x00000000000000000000000200000000000000000000000000040000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000009040000000000000010000000000000000000000400000000020000000000000000008800000000000000000000000010000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000008000000000000000000000020000000000000000000000010000000000000000000000000000000000000000000",
      "miner":"0x0000000000000000000000000000000000000000",
      "mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000",
      "nonce":"0x0000000000000000",
      "number":"0x1842",
      "parentHash":"0x8a82730a91b085896e97967a98c6bf3473ed7e6f7c5c77d46258a091b33f0baa",
      "receiptsRoot":"0xcd889a50d393a55c72b5e7d577642635d05290ab11a2996ab12ec50c41a5bc1b",
      "sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "size":"0x1c90",
      "stateRoot":"0x4e4550b3000c3b5867ff74dc9555d12d5e7354975e634a3f7d99dbad7c4074b3",
      "timestamp":"0x609e9094",
      "totalDifficulty":"0x3085",
      "transactions":[
         "0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef"
      ],
      "transactionsRoot":"0xf40699cb8da8bad3393bba72568d935583ac2cea3614532095217b644dc3a776",
      "uncles":[
         
      ]
   }
}

replica

{
   "jsonrpc":"2.0",
   "id":1,
   "result":{
      "difficulty":"0x2",
      "extraData":"0xda8301090a846765746888676f312e31362e338777696e646f777300000000001b314c51f2d5d2f14309d640bcb3bea849bcfe93972133691e50caa521f31abc1354a7224514864151a4984996e62f7723f0c5ae9e83fc8c8e25da7bc4c8745c01",
      "gasLimit":"0x895440",
      "gasUsed":"0x2853fd",
      "hash":"0x18ebee690c2b482c528d394abe05331bb805af62af053bd8154c0b2ac454a7c8",
      "logsBloom":"0x00000000000000000000000200000000000000000000000000040000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000009040000000000000010000000000000000000000400000000020000000000000000008800000000000000000000000010000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000008000000000000000000000020000000000000000000000010000000000000000000000000000000000000000000",
      "miner":"0x0000000000000000000000000000000000000000",
      "mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000",
      "nonce":"0x0000000000000000",
      "number":"0x1842",
      "parentHash":"0x381896e793396fde144a5307f6bcae3b4cf25e39351eed6428eefc56ef9595c0",
      "receiptsRoot":"0xcd889a50d393a55c72b5e7d577642635d05290ab11a2996ab12ec50c41a5bc1b",
      "sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "size":"0x1c90",
      "stateRoot":"0x4e4550b3000c3b5867ff74dc9555d12d5e7354975e634a3f7d99dbad7c4074b3",
      "timestamp":"0x609e9094",
      "totalDifficulty":"0x3085",
      "transactions":[
         "0xa8aa13df617039192ec6507aff76ea88a3f16e725b28105813fb545e23ca50ef"
      ],
      "transactionsRoot":"0xf40699cb8da8bad3393bba72568d935583ac2cea3614532095217b644dc3a776",
      "uncles":[
         
      ]
   }
}

[Enigmatic331]

Ok there's a difference on the parentHash so might be something from an earlier block... Hmmmmmm....

ADD: Checking back from block(0), and seems that the blockHash is different from the very start so I might just be pedantic and this is a non-issue, so just to check.... :)

[tynes]

@Enigmatic331 Currently the blockhashes are non-canonical because we are using Clique consensus with a single signer and no peers for each node. Every node creates its own signer key and includes a signature in the extra data field of the block. The extra data is included in the block hash so every node generates a different block hash. This could be fixed by hardcoding a single clique signer key in the node but a better solution is being worked on in #875. This implements a custom consensus.Engine which basically does a no-op for many of the consensus engine methods and deterministically fills in the unused block fields with zero bytes

[Enigmatic331]

Aha! Thanks so much Tynes - That makes total sense. Should be the same thing which was bothering me over at Kovan as well so good to know it's not really an "issue" per se.

Great - Gonna get to work.