panic: Failed to process committed block: Wrong Block.Header.AppHash. #180

dingyo777 · 2020-01-07T00:32:19Z

I've started new full node-binary.

spec is like below

ubuntu 18.04
full node version : 0.6.3-hotfix
I copied related config : app.toml, config.toml, genesis.json

But I got below error at around block number 1082013

related logs are like below

I[2020-01-06|22:47:04.484] Executed block                               module=state height=1082013 validTxs=0 invalidTxs=1
...
panic: Failed to process committed block (1082014:39C2F09916BD3415C40BD825406C09A022C98FFCDBDFFFC7DADA0E822F60F6B9): Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

May I get your favor how to resolve this issue?

The text was updated successfully, but these errors were encountered:

chainwhisper · 2020-01-07T02:47:02Z

Did you use state-sync or fast-sync?

The default way is using state-sync and if your node is syncing in mainnet, you should start from height round 59MM. It will take less than an hour to get synced.

dingyo777 · 2020-01-07T02:59:54Z

@huangsuyu
Thank you always for your support~!!

currently, my full node is fast-sync

What I want to is download all the block info from block number 0.

Then In that case, what should I config?

My full node-binary config is like this:

fast_sync = true
state_sync_reactor = false
state_sync_height = -1
hot_sync = true
hot_sync_timeout = "3s"
hot_sync_reactor = true
...

chainwhisper · 2020-01-08T01:20:16Z

Hello, @dingyo777

I have tested fast-sync in mainnet with my machine and it worked well.

Are you using the same config files from v0.6.3-hf?

dingyo777 · 2020-01-08T02:05:49Z

Hello ~ @huangsuyu

Thank you for your confirmation.

Yes, I have same config from v0.6.3-hf and I modified some filed as mentioned above comment.

Dose your node work well including block number 1082013?

My node also works well until block number 1082013, but when while syncing 1082013, it failed.

chainwhisper · 2020-01-08T06:36:26Z

@dingyo777 yes, I used fast-sync and there was no issue.

My advice is you reset your node and start again.

ArseniiPetrovich · 2020-01-22T05:27:32Z

I think we run into this issue as well. Logs are a bit different but still.
Logs from our node:

Jan 22 05:21:54 ip-172-31-18-149 bnbchaind: ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected 377FA176F7DD9CB5F11318DD9497C239D1D93636D9DCB842DAB4C5CB7AA347B6, got F9B249154ECD284A1CA4781DFFF0EBCA883A6E8E2BAC494C312731880B42F7EE

At the same time only 1 of our 2 nodes failed, second one is still active. Any idea of what could be wrong?

pablofullana · 2020-01-24T15:46:18Z

hi everyone, any updates on @ArseniiPetrovich issue?

chainwhisper · 2020-01-30T06:20:53Z

Hello, @ArseniiPetrovich could you please set your log level to debug and share more info here?

I cannot reproduce in my end

dingyo777 · 2020-02-03T01:09:52Z

very very weird.

I reinstall 0.6.3-hotfix on another linux ubuntu machine (18.04)
I copied all the related config under ~/.bnbchaind/config

At the same time, I reinstalled 0.6.3-hotfix on original server.

But both stopped at block number 1082013 with same error like

panic: Failed to process committed block (1082014:39C2F09916BD3415C40BD825406C09A022C98FFCDBDFFFC7DADA0E822F60F6B9): Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

chainwhisper · 2020-02-03T13:34:45Z

very very weird.

I reinstall 0.6.3-hotfix on another linux ubuntu machine (18.04)
I copied all the related config under ~/.bnbchaind/config

At the same time, I reinstalled 0.6.3-hotfix on original server.

But both stopped at block number 1082013 with same error like
panic: Failed to process committed block (1082014:39C2F09916BD3415C40BD825406C09A022C98FFCDBDFFFC7DADA0E822F60F6B9): Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

can you please change log level to debug and share more details?

Thanks!

ArseniiPetrovich · 2020-02-03T14:24:54Z

Hey, @huangsuyu, not sure if I'm doing it correct, but here the output:

[root@ip-172-31-18-149 ~]# /home/ec2-user/node-binary/fullnode/prod/0.6.3-hotfix/linux/bnbchaind start --home=/home/ec2-user/.bnbchaind --log_level *:info --trace
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected 377FA176F7DD9CB5F11318DD9497C239D1D93636D9DCB842DAB4C5CB7AA347B6, got F9B249154ECD284A1CA4781DFFF0EBCA883A6E8E2BAC494C312731880B42F7EE

I think I should get a bit more verbose output, but that is all I got.

ArseniiPetrovich · 2020-02-03T14:39:33Z

Oh, sorry. I forgot about bnc.log file. Here it is - it was too big to paste it here, so I've uploaded it to my GDrive.
https://drive.google.com/file/d/1TrF3TWvUiCcn_RCxF5LkYX9As45rA6lN/view?usp=sharing
ZIP file format.

dingyo777 · 2020-02-04T02:05:00Z

@huangsuyu
Thank you always~!
I changed log level to debug at config.toml but nothing special.
Below is some part of messages right before die

D[2020-02-04|02:02:52.154] Got recentPrices is only debug-level log.

I[2020-02-04|02:02:52.153] Relaying block for order book                module=dex height=1082011
I[2020-02-04|02:02:52.153] replayed all tx. Starting match              module=dex height=1082011
I[2020-02-04|02:02:52.153] No new orders for any pair, give up matching module=dexkeeper
I[2020-02-04|02:02:52.153] No order comes in for the block              module=dexkeeper
I[2020-02-04|02:02:52.153] Relaying block for order book                module=dex height=1082012
I[2020-02-04|02:02:52.153] replayed all tx. Starting match              module=dex height=1082012
I[2020-02-04|02:02:52.153] No new orders for any pair, give up matching module=dexkeeper
I[2020-02-04|02:02:52.153] No order comes in for the block              module=dexkeeper
I[2020-02-04|02:02:52.153] Relaying block for order book                module=dex height=1082013
I[2020-02-04|02:02:52.153] Skip tx when replay                          module=dex height=1082013 idx=0
I[2020-02-04|02:02:52.153] replayed all tx. Starting match              module=dex height=1082013
I[2020-02-04|02:02:52.153] No new orders for any pair, give up matching module=dexkeeper
I[2020-02-04|02:02:52.153] No order comes in for the block              module=dexkeeper
D[2020-02-04|02:02:52.154] Got recentPrices                             module=main lastSeq=1081 recentPrices=map[]
I[2020-02-04|02:02:52.217] Starting multiAppConn                        module=proxy impl=multiAppConn
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=query impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=state impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=mempool impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=consensus impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting EventBus                            module=events impl=EventBus
I[2020-02-04|02:02:52.217] Starting PubSub                              module=pubsub impl=PubSub
I[2020-02-04|02:02:52.234] Starting TxIndexerService                    module=txindex impl=TxIndexerService
I[2020-02-04|02:02:52.234] Starting BlockIndexerService                 module=blockindex impl=BlockIndexerService
I[2020-02-04|02:02:52.235] Starting indexHub                            module=indexer_hub impl=indexHub
I[2020-02-04|02:02:52.235] ABCI Handshake App Info                      module=consensus height=1082013 hash=AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0 software-version= protocol-version=0
I[2020-02-04|02:02:52.235] ABCI Replay Blocks                           module=consensus appHeight=1082013 storeHeight=1082014 stateHeight=1082013
I[2020-02-04|02:02:52.235] Replay last block using real app             module=consensus
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

Above logs are made when I retried on failed node-binary.

So let me start again after resetting all the data file and related config file with debug level. And then I will attach the result here

chainwhisper · 2020-02-07T06:33:05Z

hello, @dingyo777

I failed to reproduce your issue at my end. I have enabled fast-sync and downloaded all the blocks.

my config.toml:

fast_sync = true

# As state sync is an experimental feature, this switch can totally disable it on core network nodes (validator, witness)
state_sync_reactor = true

# If this node is many days behind the tip of the chain, StateSyncHeight
# allows them to catchup quickly by downloading app state (without historical blocks)
# in parallel and start syncing block afterwards
# <0 - turn off state sync
# =0 - sync from peer's latest height
# >0 - sync from that height
state_sync_height = -1

# Only take effect when HotSyncReactor is true.
# If true, will sync blocks use hot sync protocol
# If false, still use tendermint consensus protocol, but can still handle other peers sync request.
hot_sync = true

# The max wait time for subscribe a block.
# Only take effect when hot_sync is true
hot_sync_timeout = "3s"

# It will benefit fullnode and witness who do not need consensus by saving network and cpu resources.
# Recommend the node that is not validator to turn on.
hot_sync_reactor = true

dingyo777 · 2020-02-10T08:45:15Z

@huangsuyu
Really thank you so much for your support~!!
Then let me retry with the config you attached.

ArseniiPetrovich · 2020-02-10T17:24:46Z

What about my issue, @huangsuyu ?

chainwhisper · 2020-02-11T04:54:16Z

What about my issue, @huangsuyu ?

please share your config.toml and app.toml.

I failed to reproduce your problem at my end.

ArseniiPetrovich · 2020-02-11T09:23:25Z

Sure, here it is, @huangsuyu
config_and_app_toml.zip

ares0x · 2020-02-24T09:01:12Z

Hi, I ran into a problem. My blocks were syncing normally the other day. Today I saw that he made a mistake.

version:0.6.3

panic:
github.com/tendermint/tendermint/blockchain/hot.(*BlockPool).applyBlock(0xc000e82000, 0xc0fc88ed80) /home/suyu/go/pkg/mod/github.com/binance-chain/[email protected]/blockchain/hot/pool.go:566 +0x494 github.com/tendermint/tendermint/blockchain/hot.(*BlockPool).hotSyncRoutine(0xc000e82000) /home/suyu/go/pkg/mod/github.com/binance-chain/[email protected]/blockchain/hot/pool.go:383 +0x3c6 created by github.com/tendermint/tendermint/blockchain/hot.(*BlockPool).SwitchToHotSync /home/suyu/go/pkg/mod/github.com/binance-chain/[email protected]/blockchain/hot/pool.go:198 +0x172 ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B

@huangsuyu

config:
fast_sync = true state_sync_reactor = true state_sync_height = 0 hot_sync = true hot_sync_timeout = "3s" hot_sync_reactor = true

chainwhisper · 2020-02-25T03:08:55Z

Hello, @404killer

Could you upgrade to hotfix version v0.6.3-hf1? you can use this script

ares0x · 2020-02-25T03:12:28Z

Can I use this script to upgrade the version of the node directly?

Hello, @404killer

Could you upgrade to hotfix version v0.6.3-hf1? you can use this script

Can I use this script to upgrade the version of the node directly?

ares0x · 2020-02-25T03:13:24Z

Can I use this script to upgrade the version of the node directly?

Hello, @404killer
Could you upgrade to hotfix version v0.6.3-hf1? you can use this script

Can I use this script to upgrade the version of the node directly?

And do I need to resynchronize my block data?

ares0x · 2020-02-25T03:20:10Z

`... Choose version of Binance Chain node to install

0.5.8
0.5.9
0.5.10
0.6.0
0.6.1
0.6.2
0.6.3
`

My current version is 0.6.3. There is no v0.6.3-hf1

chainwhisper · 2020-02-25T03:23:22Z

`... Choose version of Binance Chain node to install

0.5.8

0.5.9

0.5.10

0.6.0

0.6.1

0.6.2

0.6.3
`

My current version is 0.6.3. There is no v0.6.3-hf1

This script is already updated, please download the latest version.

To recover, please follow this instruction here: https://docs.binance.org/fullnodeissue.html#forget-to-upgrade

You need to use recovery tool to revert one block and restart with hotfix version.

ares0x · 2020-02-25T03:24:10Z

`... Choose version of Binance Chain node to install

0.5.8

0.5.9

0.5.10

0.6.0

0.6.1

0.6.2

0.6.3
`

My current version is 0.6.3. There is no v0.6.3-hf1

This script is already updated, please download the latest version.

To recover, please follow this instruction here: https://docs.binance.org/fullnodeissue.html#forget-to-upgrade

You need to use recovery tool to revert one block and restart with hotfix version.

Ok, I'll test it first. Thank you very much

wannabit-yongjoo · 2020-02-28T04:14:16Z

Checked "state_recover" working well.

Used like
state_recover 1181875 $BNCHOME

ares0x · 2020-03-09T03:29:44Z

I upgraded the version of node-binary to 0.6.3-hotfix to fix the problem.And thank you very much for your help. @huangsuyu

kenorb · 2021-11-13T15:23:47Z

See: cosmos/cosmos-sdk#10281

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic: Failed to process committed block: Wrong Block.Header.AppHash. #180

panic: Failed to process committed block: Wrong Block.Header.AppHash. #180

dingyo777 commented Jan 7, 2020

chainwhisper commented Jan 7, 2020

dingyo777 commented Jan 7, 2020

chainwhisper commented Jan 8, 2020

dingyo777 commented Jan 8, 2020 •

edited

Loading

chainwhisper commented Jan 8, 2020

ArseniiPetrovich commented Jan 22, 2020

pablofullana commented Jan 24, 2020

chainwhisper commented Jan 30, 2020

dingyo777 commented Feb 3, 2020

chainwhisper commented Feb 3, 2020 •

edited

Loading

ArseniiPetrovich commented Feb 3, 2020

ArseniiPetrovich commented Feb 3, 2020

dingyo777 commented Feb 4, 2020 •

edited

Loading

chainwhisper commented Feb 7, 2020

dingyo777 commented Feb 10, 2020

ArseniiPetrovich commented Feb 10, 2020

chainwhisper commented Feb 11, 2020

ArseniiPetrovich commented Feb 11, 2020

ares0x commented Feb 24, 2020 •

edited

Loading

chainwhisper commented Feb 25, 2020

ares0x commented Feb 25, 2020

ares0x commented Feb 25, 2020

ares0x commented Feb 25, 2020

chainwhisper commented Feb 25, 2020

ares0x commented Feb 25, 2020

wannabit-yongjoo commented Feb 28, 2020 •

edited

Loading

ares0x commented Mar 9, 2020

kenorb commented Nov 13, 2021 •

edited

Loading

panic: Failed to process committed block: Wrong Block.Header.AppHash. #180

panic: Failed to process committed block: Wrong Block.Header.AppHash. #180

Comments

dingyo777 commented Jan 7, 2020

chainwhisper commented Jan 7, 2020

dingyo777 commented Jan 7, 2020

chainwhisper commented Jan 8, 2020

dingyo777 commented Jan 8, 2020 • edited Loading

chainwhisper commented Jan 8, 2020

ArseniiPetrovich commented Jan 22, 2020

pablofullana commented Jan 24, 2020

chainwhisper commented Jan 30, 2020

dingyo777 commented Feb 3, 2020

chainwhisper commented Feb 3, 2020 • edited Loading

ArseniiPetrovich commented Feb 3, 2020

ArseniiPetrovich commented Feb 3, 2020

dingyo777 commented Feb 4, 2020 • edited Loading

chainwhisper commented Feb 7, 2020

dingyo777 commented Feb 10, 2020

ArseniiPetrovich commented Feb 10, 2020

chainwhisper commented Feb 11, 2020

ArseniiPetrovich commented Feb 11, 2020

ares0x commented Feb 24, 2020 • edited Loading

chainwhisper commented Feb 25, 2020

ares0x commented Feb 25, 2020

ares0x commented Feb 25, 2020

ares0x commented Feb 25, 2020

chainwhisper commented Feb 25, 2020

ares0x commented Feb 25, 2020

wannabit-yongjoo commented Feb 28, 2020 • edited Loading

ares0x commented Mar 9, 2020

kenorb commented Nov 13, 2021 • edited Loading

dingyo777 commented Jan 8, 2020 •

edited

Loading

chainwhisper commented Feb 3, 2020 •

edited

Loading

dingyo777 commented Feb 4, 2020 •

edited

Loading

ares0x commented Feb 24, 2020 •

edited

Loading

wannabit-yongjoo commented Feb 28, 2020 •

edited

Loading

kenorb commented Nov 13, 2021 •

edited

Loading