trace_block returns "insufficient funds for gas * price + value" error on block 21030627

[quick-pawiromitchel]

System information

Erigon version: erigon/2.60.4/linux-amd64/go1.22.5

OS & Version: Linux

Chain/Network: Mainnet

Steps to reproduce the behaviour

{
  "method": "trace_block",
  "params": [
    "0x140e6e3"
  ],
  "id": 1,
  "jsonrpc": "2.0"
}

Returns

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32000,
    "message": "first run for txIndex 85 error: insufficient funds for gas * price + value: address 0x0b92619DdE55C0cbf828d32993a7fB004E00c84B have 1414044108792614043 want 1414044806692614043"
  }
}

Backtrace

Oct 29 13:25:37 erigon[1105]: [WARN] [10-29|13:25:37.367] [rpc] served                             conn=xxxxx:59459 method=trace_block reqid=1 t=34.504079ms err="first run for txIndex 85 error: insufficient funds for gas * price + value: address 0x0b92619DdE55C0cbf828d32993a7fB004E00c84B have 1414044108792614043 want 1414044806692614043"

[yperbasis]

Fixed by PR #12559 and will be released in v2.60.10.

[Tommel71]

@yperbasis There seems to be something wrong, we still get the error on 2.60.10 with the newest docker container and the git_tag seems to be 2.60.9 still

○ → docker run erigontech/erigon:latest
[INFO] [11-06|14:44:41.097] logging to file system                   log dir=/home/erigon/.local/share/erigon/logs file prefix=erigon log level=info json=false
[INFO] [11-06|14:44:41.098] Build info                               git_branch=release/2.60 git_tag=v2.60.9-4-g3afee08bff git_commit=3afee08bfff043a731485e0a92dd8650823d8884
[INFO] [11-06|14:44:41.098] Starting Erigon on Ethereum mainnet...

[yperbasis]

@yperbasis There seems to be something wrong, we still get the error on 2.60.10 with the newest docker container and the git_tag seems to be 2.60.9 still

○ → docker run erigontech/erigon:latest
[INFO] [11-06|14:44:41.097] logging to file system                   log dir=/home/erigon/.local/share/erigon/logs file prefix=erigon log level=info json=false
[INFO] [11-06|14:44:41.098] Build info                               git_branch=release/2.60 git_tag=v2.60.9-4-g3afee08bff git_commit=3afee08bfff043a731485e0a92dd8650823d8884
[INFO] [11-06|14:44:41.098] Starting Erigon on Ethereum mainnet...

@VBulikov @lystopad could you check this please

[lystopad]

Hi, all

"git_tag" in this case is the output of git describe --tags '--match=v*' --dirty see here

This outputs basically means -- the last tag found in git was "v2.60.9" and current git repository state is 4 commits on top of that tag and latest short commit id is "3afee08bff". More details could be found here.

It is done so, because we decided to TAG git repository after release.
Let me know if you would like to discuss/change it.

P.S.
In the following PR #12658 I changed short commit id to be 7 chars which will match short commit IDs in github history (easy to find).

[Tommel71]

@yperbasis @lystopad

It is not just the tag but more importantly that the issue described in this thread persists.

[yperbasis]

@Tommel71 trace_block now works fine for 0x140e6e3 on my v2.60.10 node (albeit I built it locally instead of using the docker image).

[soad003]

@yperbasis for us (same team as @Tommel71), the issue persists after updating to the v2.60.10 docker image.

The curl call returns an error as before. Do you have any advice, is there some cache we need to reset or could it be an issue with the docker image?

[yperbasis]

I can reproduce the issue locally when I use erigon from erigon_v2.60.9_linux_amd64v2.tar.gz, but erigon from erigon_v2.60.10_linux_amd64v2.tar.gz works fine for me. Perhaps there's something wrong with the docker image.

[tohadv]

Hi! I have the same error! I try to build erigon and rpcdaemon, try to use 2.60.10 v1 and v2 and still have it :(

[mdragaschnig]

[..] Perhaps there's something wrong with the docker image.

That's possibly the case; using the official docker image

erigontech/erigon                                     v2.60.10          sha256:87975922b4506b5e0956e8523a4638a63bbeeb01941bcfe5f82a55343d3c2846   c2ee3624690f   8 days ago      597MB

produces

curl -X POST http://erigon2.60.10 -H 'Content-Type: application/json' -d '{"method":"trace_block","params":["0x140e6e3"],"id":1,"jsonrpc":"2.0"}'`
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"first run for txIndex 85 error: insufficient funds for gas * price + value: address 0x0b92619DdE55C0cbf828d32993a7fB004E00c84B have 1414044108792614043 want 1414044806692614043"}}

[yperbasis]

Sorry, I can't reproduce the issue and am out of ideas, so closing.

[32ethers]

I have the same issue, my erigon version is erigon version 2.60.10-3afee08b, chifra version is chifra version GHC-TrueBlocks//3.5.0-release, but actually it is complied from the latest code in master branch, version should above 3.8

Is that because when my node reach height 21030627, my version is 2.60.9?

[mdragaschnig]

Sorry, I can't reproduce the issue and am out of ideas, so closing.

Thanks for testing; so did the docker image version work for you?

I've just tried with the latest docker image v2.61.0-beta

git_branch=release/2.61 git_tag=v2.61.0-beta1 git_commit=9dc3435f893801cbff5d2cec74f66ec97d4da74c

and here the insufficient funds error still exists.

As @32ethers said; can the problem be that the database was written with a pre 2.60.10 version?

[yperbasis]

Thanks for testing; so did the docker image version work for you?

I've just tried with the latest docker image v2.61.0-beta

git_branch=release/2.61 git_tag=v2.61.0-beta1 git_commit=9dc3435f893801cbff5d2cec74f66ec97d4da74c

and here the insufficient funds error still exists.

As @32ethers said; can the problem be that the database was written with a pre 2.60.10 version?

No, this is a pure code change, not related to the database. That's why I'm a bit lost why v2.60.10 or v2.61.0-beta aren't working for you.

[32ethers]

Thanks for testing; so did the docker image version work for you?
I've just tried with the latest docker image v2.61.0-beta
git_branch=release/2.61 git_tag=v2.61.0-beta1 git_commit=9dc3435f893801cbff5d2cec74f66ec97d4da74c
and here the insufficient funds error still exists.
As @32ethers said; can the problem be that the database was written with a pre 2.60.10 version?

No, this is a pure code change, not related to the database. That's why I'm a bit lost why v2.60.10 or v2.61.0-beta aren't working for you.

Another guess:
chifra get data from rpc, and I start my rpc service with rpcdaemon, not erigon. Will that be an issue?

By the way, this is my rpcdaemon command:

./rpcdaemon --datadir="/node1/eth/"  --txpool.api.addr=localhost:9070  --private.api.addr=localhost:9070  --http.addr=0.0.0.0 --http.api=eth,erigon,web3,net,debug,trace,txpool

[yperbasis]

@32ethers yes, you need to update rpcdaemon to v2.60.10 as well.

[32ethers]

I remove source code and compile again, and switch to 2.61.0 tag, still not work. here is what i did:

rm -rf erigon
git clone https://github.com/erigontech/erigon.git
cd erigon
make
build/bin/erigon --version # ensure version is 2.61
cp -r build/bin /node1/erigon_2.61.0-beta1
cd /node1/erigon_2.61.0-beta1
./rpcdaemon --datadir="/node1/eth/"  --txpool.api.addr=localhost:9070  --private.api.addr=localhost:9070  --http.addr=0.0.0.0 --http.api=eth,erigon,web3,net,debug,trace,txpool

And still got the error, rpcdaemon will give a warning, and chifra will raise a error ant read that height repeatly.

Query with curl, still get the error:

Did I miss something? Do I have some cache to clean elsewhere?

[32ethers]

Hi @yperbasis

I tested again, turn out if you use rpc in erigon, it will work, but rpcdaemon doesn't.

so you should consider fix rpcdaemon as well.

[yperbasis]

The remote rpcdaemon case is fixed by PR #13125/#13128/#13129