trie/database on insert - panic: runtime error: invalid memory address or nil pointer dereference #17280

illya13 · 2018-07-30T12:50:37Z

... I think bug was introduced with #16810
with double linked list management and I think db.newest has invalid value here
https://github.com/ethereum/go-ethereum/blob/master/trie/database.go#L316

v1.8.10 is working stable for the same scenario

System information

Geth version: v1.8.11, v1.8.12, master
OS & Version: Linux

Expected behaviour

stable env

Actual behaviour

multiple nodes are crashing

Steps to reproduce the behaviour

private PoA net
submit 700+ tx
it will crash

Backtrace

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x69c3c2]

goroutine 50 [running]:
github.com/ethereum/go-ethereum/trie.(*Database).insert(0xc420128e10, 0xf2024b11e0b30949, 0xd0489f37f7c96afd, 0xd5e22bd469762fec, 0x22e64f58cfef7c06, 0xc485887680, 0x22, 0x226, 0x10c59e0, 0xc42cb87310)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/database.go:316 +0x3b2
github.com/ethereum/go-ethereum/trie.(*hasher).store(0xc425752880, 0x10c59e0, 0xc42cb87310, 0xc420128e10, 0x10c5900, 0xc42cb87310, 0x10c59e0, 0xc42cb87360, 0x0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:190 +0x2a0
github.com/ethereum/go-ethereum/trie.(*hasher).hash(0xc425752880, 0x10c59e0, 0xc42cb862d0, 0xc420128e10, 0x0, 0x10c77e0, 0xc4260080a0, 0x10c77e0, 0xc4260080a0, 0x0, ...)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:98 +0x17a
github.com/ethereum/go-ethereum/trie.(*hasher).hashChildren(0xc425752880, 0x10c5960, 0xc446479b80, 0xc420128e10, 0x10c7701, 0x10c77e0, 0xc462823dc0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:148 +0x18f
github.com/ethereum/go-ethereum/trie.(*hasher).hash(0xc425752880, 0x10c5960, 0xc446479b80, 0xc420128e10, 0x0, 0x10c77e0, 0xc462823dc0, 0x10c77e0, 0xc462823dc0, 0x0, ...)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:94 +0x10f
github.com/ethereum/go-ethereum/trie.(*hasher).hashChildren(0xc425752880, 0x10c5960, 0xc446479900, 0xc420128e10, 0x10c7701, 0x10c77e0, 0xc45874ba20, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:148 +0x18f
github.com/ethereum/go-ethereum/trie.(*hasher).hash(0xc425752880, 0x10c5960, 0xc446479900, 0xc420128e10, 0x0, 0x10c77e0, 0xc469588200, 0x10c77e0, 0xc469588340, 0x0, ...)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:94 +0x10f
github.com/ethereum/go-ethereum/trie.(*hasher).hashChildren(0xc425752880, 0x10c5960, 0xc446479680, 0xc420128e10, 0x1, 0xc4001bec80, 0x1b51b20, 0x100, 0x32, 0x32)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:148 +0x18f
github.com/ethereum/go-ethereum/trie.(*hasher).hash(0xc425752880, 0x10c5960, 0xc446479680, 0xc420128e10, 0xc4405e5201, 0xb767cc668fd83b7d, 0x1, 0x8, 0x0, 0xc4415ed4e8, ...)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/hasher.go:94 +0x10f
github.com/ethereum/go-ethereum/trie.(*Trie).hashRoot(0xc42e0da2a0, 0xc420128e10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:475 +0xe6
github.com/ethereum/go-ethereum/trie.(*Trie).Commit(0xc42e0da2a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4415ed5b0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:460 +0x71
github.com/ethereum/go-ethereum/trie.(*SecureTrie).Commit(0xc42e0da2a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xd98d1adcf954d59e, 0xc4415ed7d0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/trie/secure_trie.go:155 +0x49e
github.com/ethereum/go-ethereum/core/state.(*stateObject).CommitTrie(0xc45ad59a40, 0x10cba20, 0xc4202b4420, 0xc48108d078, 0x1)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/core/state/state_object.go:229 +0x9a
github.com/ethereum/go-ethereum/core/state.(*StateDB).Commit(0xc45ad58ee0, 0xc4415edd01, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/core/state/statedb.go:603 +0x31e
github.com/ethereum/go-ethereum/core.(*BlockChain).WriteBlockWithState(0xc42034c000, 0xc42b8f2630, 0xc423b2e238, 0x1, 0x1, 0xc45ad58ee0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/core/blockchain.go:902 +0x5e8
github.com/ethereum/go-ethereum/miner.(*worker).wait(0xc4202211e0)
        /home/travis/gopath/src/github.com/ethereum/go-ethereum/miner/worker.go:319 +0x252
created by github.com/ethereum/go-ethereum/miner.newWorker
...

The text was updated successfully, but these errors were encountered:

illya13 · 2018-07-30T12:59:05Z

CC: @karalabe

karalabe · 2018-07-30T13:19:20Z

Do you have a script to repro this? It would be really valuable.

illya13 · 2018-07-30T13:20:44Z

no I don't :(
It was used in our app and I was able to face the issue after geth upgrade

karalabe · 2018-07-30T13:34:35Z

Hmm, I did have a code modification that handles some very very weird corner case around db.newest, but until now I couldn't really figure out any possibility to trigger it (i.e. I can't figure out an edge case that would hit this code path).

Could you please:

Try out trie: handle removing the freshest node too #17282 and see if it makes a difference.
Please detail your exact PoA setup, particularly what your block times are, what you start your nodes with, how fast do the nodes crash, etc.

illya13 · 2018-07-30T13:38:29Z

interval: 1s
network is stable if no huge load
crash very fast i.e. in 10-15min if do load (600-1k tx)
started on AWS, T2 nodes (4G RAM, 2CPU)

nohup geth --datadir data --networkid=18 --unlock "4a..40" --password data/password.txt --gasprice "1000000000" --targetgaslimit '671088640' --nodiscover --mine --minerthreads 1 --nodekeyhex=e3..33 --ethstats 'node1:...' > data/geth.log 2>&1 &
nohup geth --datadir data --networkid=18 --unlock "9d..94" --password data/password.txt --gasprice "1000000000" --targetgaslimit '671088640' --nodiscover --mine --minerthreads 1 --nodekeyhex=31..5c --ethstats 'node2:...' > data/geth.log 2>&1 &
nohup geth --datadir data --networkid=18 --unlock "9d..ab" --password data/password.txt --gasprice "1000000000" --targetgaslimit '671088640' --nodiscover --mine --minerthreads 1 --nodekeyhex=ee..ae --ethstats 'node3:...' > data/geth.log 2>&1 &
nohup geth --datadir data --networkid=18 --unlock "12..7b" --password data/password.txt --gasprice "1000000000" --targetgaslimit '671088640' --nodiscover --mine --minerthreads 1 --nodekeyhex=44..6c --ethstats 'node4:...' > data/geth.log 2>&1 &

illya13 · 2018-07-30T13:41:08Z

#17282 - do you have any linux/x64 builds ?

karalabe · 2018-07-30T13:41:42Z

Hmm, that's an interesting stress test scenario (to push out a block every second from multiple nodes with many transactions). My bet is that there's some weird race between mining blocks and incoming blocks from the network. I'll try to set up some repro for it, though that will probably take some time.

illya13 · 2018-07-30T13:43:31Z

tx comes from separate (non validating) gw node
v1.8.10 is working well

karalabe · 2018-07-30T15:23:55Z

I'm trying to repro this with a local stress test. For now I've hit a previously reported issue, so I'll commit a fix for that. Hopefully I can trigger this crash too somehow.

illya13 · 2018-07-30T15:33:25Z

Ok, if you have any builds to test - please let me know
ATM It is hard for me to build and redistribute
But quite easy to download and test over env

karalabe · 2018-07-31T07:52:33Z

Could you try with latest master? I've merged in two fixes, one for the miner, one for the flush-list. I don't think either will fix your issue btw, since they appear to be different causes, but lets see. The code is definitely more correct now.

illya13 · 2018-07-31T08:08:59Z

trying

Version: 1.8.13-unstable
Git Commit: d927cbb638419866de47929a723b5f2b8922c88e
Architecture: amd64
Go Version: go1.10.3
Operating System: linux

illya13 · 2018-07-31T08:27:33Z

did 6 rounds of bulk testing, 10k tx in total,
so far - stable and faster than v1.8.10,
thx

will report more during day ...

illya13 · 2018-07-31T11:19:40Z

... still up

So I think we can mark it as fixed in master

karalabe · 2018-08-01T10:47:49Z

Thanks for the confirm. I'll close it, but please open a new issue if it still persists. This is definitely a concerning issue.

illya13 · 2018-08-01T12:01:02Z

👍

karalabe closed this as completed Aug 1, 2018

hello2mao mentioned this issue Nov 14, 2018

trie/database on insert - panic: runtime error: invalid memory address or nil pointer dereference #18094

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

trie/database on insert - panic: runtime error: invalid memory address or nil pointer dereference #17280

trie/database on insert - panic: runtime error: invalid memory address or nil pointer dereference #17280

illya13 commented Jul 30, 2018 •

edited

Loading

illya13 commented Jul 30, 2018

karalabe commented Jul 30, 2018

illya13 commented Jul 30, 2018

karalabe commented Jul 30, 2018 •

edited

Loading

illya13 commented Jul 30, 2018 •

edited

Loading

illya13 commented Jul 30, 2018

karalabe commented Jul 30, 2018

illya13 commented Jul 30, 2018 •

edited

Loading

karalabe commented Jul 30, 2018

illya13 commented Jul 30, 2018

karalabe commented Jul 31, 2018

illya13 commented Jul 31, 2018

illya13 commented Jul 31, 2018

illya13 commented Jul 31, 2018 •

edited

Loading

karalabe commented Aug 1, 2018

illya13 commented Aug 1, 2018

trie/database on insert - panic: runtime error: invalid memory address or nil pointer dereference #17280

trie/database on insert - panic: runtime error: invalid memory address or nil pointer dereference #17280

Comments

illya13 commented Jul 30, 2018 • edited Loading

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Backtrace

illya13 commented Jul 30, 2018

karalabe commented Jul 30, 2018

illya13 commented Jul 30, 2018

karalabe commented Jul 30, 2018 • edited Loading

illya13 commented Jul 30, 2018 • edited Loading

illya13 commented Jul 30, 2018

karalabe commented Jul 30, 2018

illya13 commented Jul 30, 2018 • edited Loading

karalabe commented Jul 30, 2018

illya13 commented Jul 30, 2018

karalabe commented Jul 31, 2018

illya13 commented Jul 31, 2018

illya13 commented Jul 31, 2018

illya13 commented Jul 31, 2018 • edited Loading

karalabe commented Aug 1, 2018

illya13 commented Aug 1, 2018

illya13 commented Jul 30, 2018 •

edited

Loading

karalabe commented Jul 30, 2018 •

edited

Loading

illya13 commented Jul 30, 2018 •

edited

Loading

illya13 commented Jul 30, 2018 •

edited

Loading

illya13 commented Jul 31, 2018 •

edited

Loading