Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can not sync pyrmont testnet #9450

Closed
terencechain opened this issue Aug 24, 2021 · 11 comments · Fixed by #9452
Closed

Can not sync pyrmont testnet #9450

terencechain opened this issue Aug 24, 2021 · 11 comments · Fixed by #9452
Labels
Priority: High High priority item

Comments

@terencechain
Copy link
Member

Can not sync pyrmont testnet post fork in a period of non finality. Node is stuck with:

WARN initial-sync: Range is not processed error=Range had no valid blocks to process

Some evidence gathered so far:

  • These nodes were restarted during a period of non-finality
  • Peer counts appear much lower than before. From 75 to only 25.

Some targets to consider:

  • p2p changes
  • state processing changes
  • (lack of) init-sync changes for hf1
@terencechain terencechain added the Priority: High High priority item label Aug 24, 2021
@prestonvanloon prestonvanloon mentioned this issue Aug 24, 2021
Closed
@prestonvanloon prestonvanloon linked a pull request Aug 24, 2021 that will close this issue
Patch Initial Sync For Non Finalized Blocks #9452
Merged
1 task
@prestonvanloon
Copy link
Member

#9452 Seems to resolve this issue. Keeping it open for a little while until we have more data.

@prestonvanloon
Copy link
Member

Waiting for confirmation from a node syncing from genesis.

@gregbkr
Copy link

gregbkr commented Aug 25, 2021 •
edited
Loading

I have the same issue with command since 20.08:
/home/eth/prysm/prysm.sh beacon-chain --pyrmont --http-web3provider=https://goerli.prylabs.net --accept-terms-of-use

So I tried to use infura ETH1 provider instead
home/eth/prysm/prysm.sh beacon-chain --pyrmont --http-web3provider=https://goerli.infura.io/v3/8a1ad835e4224cc0a9ffa0f36c733dcc --accept-terms-of-use

But got the message:

sudo journalctl -u prysm-beacon -f
-- Logs begin at Thu 2021-03-11 13:43:56 UTC. --
Aug 25 20:59:41 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:41" level=info msg="Connected to eth1 proof-of-work chain" endpoint="https://goerli.infura.io/***" prefix=powchain
Aug 25 20:59:42 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:42" level=info msg="Chain genesis time reached" genesisStateRoot=2bb257ca66d05a047a65fe43a5f457b674de445d917cca029efb09b3ba4758c4 genesisTime="2020-11-18 12:00:07 +0000 UTC" genesisValidators=100010 prefix=slotutil
Aug 25 20:59:44 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:44" level=info msg="Starting initial chain sync..." prefix=initial-sync
Aug 25 20:59:44 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:44" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=0
Aug 25 20:59:44 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:44" level=info msg="Started discovery v5" ENR="enr:-LK4QAqsuZUwKueFAEcWViI3GCWc0-BLaVnLAKWKB1a0prCXPdKHbIvPA_VyXou0jZQP4CD4-UbYnUpI_oDs0aAPX_oBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpA7CIeVAAAgCf__________gmlkgnY0gmlwhKwfF8uJc2VjcDI1NmsxoQOlCPnUS7KkVzOmsWAq9-zU7GpPWQbx_Xj8aDfZpxHu4YN0Y3CCMsiDdWRwgi7g" prefix=p2p
Aug 25 20:59:44 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:44" level=info msg="Node started p2p server" multiAddr="/ip4/172.31.23.203/tcp/13000/p2p/16Uiu2HAmPmDZP26P7zXdei3p6rHgkfWrWwsPXr1mfmTz8KY1vKCt" prefix=p2p
Aug 25 20:59:49 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:49" level=info msg="New gRPC client connected to beacon node" addr="127.0.0.1:36400" prefix=rpc
Aug 25 20:59:49 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:49" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=2
Aug 25 20:59:54 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:54" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=2
Aug 25 20:59:59 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 20:59:59" level=info msg="Processing block batch of size 27 starting from  0x3d0fbdd6... 1972737/2018699 - estimated time remaining 9h27m25s" blocksPerSecond=1.4 peers=3 prefix=initial-sync
Aug 25 21:00:18 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:18" level=info msg="Processing block batch of size 28 starting from  0xb4d65306... 1972768/2018700 - estimated time remaining 4h38m22s" blocksPerSecond=2.8 peers=5 prefix=initial-sync
Aug 25 21:00:24 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:24" level=info msg="Synced to finalized epoch - now syncing blocks up to current head" headSlot=2018701 prefix=initial-sync syncedSlot=1972817
Aug 25 21:00:24 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:24" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:24 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:24" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:24 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:24" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:24 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:24" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:24 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:24" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:41 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:41" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:41 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:41" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:41 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:41" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:41 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:41" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:43 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:43" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:43 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:43" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:43 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:43" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:43 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:43" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:43 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:43" level=warning msg="Range is not processed" error="Range had no valid blocks to process" prefix=initial-sync
Aug 25 21:00:44 ip-172-31-23-203 prysm.sh[2858]: time="2021-08-25 21:00:44" level=info msg="Peer summary" activePeers=10 inbound=3 outbound=7 prefix=p2p


/prysm.sh beacon-chain --version
Latest Prysm version is v1.4.3.
Beacon chain is up to date.
Verifying binary integrity.
beacon-chain-v1.4.3-linux-amd64: OK
gpg: Signature made Fri Jul 30 18:56:42 2021 UTC
gpg:                using RSA key 0AE0051D647BA3C1A917AF4072E33E4DF1A5036E
gpg: Good signature from "Preston Van Loon <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 0AE0 051D 647B A3C1 A917  AF40 72E3 3E4D F1A5 036E
Verified /home/eth/prysm/dist/beacon-chain-v1.4.3-linux-amd64 has been signed by Prysmatic Labs.
Starting Prysm beacon-chain --version
beacon-chain version Prysm/v1.4.3/8bca66ac6408a03af52d65541f58384007ed50ef. Built at: 2021-07-30 18:51:56+00:00

Not sure what to do now.

Thanks!

@tkachevaa
Copy link

tkachevaa commented Aug 27, 2021 •
edited
Loading

After 2021-08-22 I got the same loop errors - no good blocks in batch.
Today i download actual version of prysm.bat and also edit it with this - set USE_PRYSM_VERSION=v2.0.0-rc.0
Now initial sync is going with more success , but some times got the same errors. Hope to sync finally and start to validate.
Maybe the old version of prysm.bat (my was from 2020-11) is not working after 2021-08-22.

@iSOcH
Copy link

iSOcH commented Aug 27, 2021

You definitely need 2.0.0 for pyrmont testnet after the altair fork.
But this issue is about the new version also having problems, especially since/during the non-finality tests

The errors of my node now changed after restarting it like 12h ago:

eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x439281003306611f9d7dd144a78f2ecbf1c0bd64443ccdc2a0e5ac342d2eec20" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 48 starting from  0x5e98559d... 2025822/2028557 - estimated time remaining 30s" blocksPerSecond=89.2 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x51f6132c05886741717ba5b5fc2ebe908dca31e5eb1ac7a06694c6f5d749e936" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 48 starting from  0x8779917d... 2025886/2028557 - estimated time remaining 29s" blocksPerSecond=91.6 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xa98840ae63c3147733b5ceb8cba8a50c74571f212d62c36d777a89516c4d7d49" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 37 starting from  0x573ac213... 2025950/2028557 - estimated time remaining 27s" blocksPerSecond=93.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x43cfbd9541ceb493074417fbad4df3c4eaf81d6db512dc2aa4fb147132cabdb9" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 51 starting from  0xf2a2b888... 2026014/2028557 - estimated time remaining 26s" blocksPerSecond=96.0 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8cc16f348db1a24b94e60b46a38340bdf726ba593f75933f911fc3ae388b47d4" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 50 starting from  0x1f6094b4... 2025566/2028558 - estimated time remaining 37s" blocksPerSecond=79.3 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x49062e933cc68dd228a165e5a12b5082d5374b40b6589fd07b6d1f224d126abe" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 50 starting from  0x4c1d8de8... 2025630/2028558 - estimated time remaining 35s" blocksPerSecond=81.8 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x11fca032bbc701a419cd26e076a38e669036ad7389f567d5698a82520d53e6c2" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 48 starting from  0xee70212f... 2025696/2028558 - estimated time remaining 33s" blocksPerSecond=84.2 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x2fa61c284991e209355d27193722ba0003c9bdae6380c0f4a0ee68720bcaab32" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 52 starting from  0xaac9e556... 2025758/2028558 - estimated time remaining 32s" blocksPerSecond=86.8 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x439281003306611f9d7dd144a78f2ecbf1c0bd64443ccdc2a0e5ac342d2eec20" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 48 starting from  0x5e98559d... 2025822/2028558 - estimated time remaining 30s" blocksPerSecond=89.2 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x51f6132c05886741717ba5b5fc2ebe908dca31e5eb1ac7a06694c6f5d749e936" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 48 starting from  0x8779917d... 2025886/2028558 - estimated time remaining 29s" blocksPerSecond=91.6 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xa98840ae63c3147733b5ceb8cba8a50c74571f212d62c36d777a89516c4d7d49" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 37 starting from  0x573ac213... 2025950/2028558 - estimated time remaining 27s" blocksPerSecond=93.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x43cfbd9541ceb493074417fbad4df3c4eaf81d6db512dc2aa4fb147132cabdb9" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 51 starting from  0xf2a2b888... 2026014/2028558 - estimated time remaining 26s" blocksPerSecond=96.0 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8cc16f348db1a24b94e60b46a38340bdf726ba593f75933f911fc3ae388b47d4" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 50 starting from  0x1f6094b4... 2025566/2028558 - estimated time remaining 31s" blocksPerSecond=96.0 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x49062e933cc68dd228a165e5a12b5082d5374b40b6589fd07b6d1f224d126abe" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 50 starting from  0x4c1d8de8... 2025630/2028558 - estimated time remaining 29s" blocksPerSecond=98.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x11fca032bbc701a419cd26e076a38e669036ad7389f567d5698a82520d53e6c2" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 52 starting from  0xaac9e556... 2025758/2028558 - estimated time remaining 27s" blocksPerSecond=101.1 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x439281003306611f9d7dd144a78f2ecbf1c0bd64443ccdc2a0e5ac342d2eec20" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 48 starting from  0x5e98559d... 2025822/2028558 - estimated time remaining 26s" blocksPerSecond=103.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x51f6132c05886741717ba5b5fc2ebe908dca31e5eb1ac7a06694c6f5d749e936" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 48 starting from  0x8779917d... 2025886/2028558 - estimated time remaining 25s" blocksPerSecond=105.9 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xa98840ae63c3147733b5ceb8cba8a50c74571f212d62c36d777a89516c4d7d49" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 51 starting from  0xf2a2b888... 2026014/2028558 - estimated time remaining 23s" blocksPerSecond=108.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8cc16f348db1a24b94e60b46a38340bdf726ba593f75933f911fc3ae388b47d4" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=info msg="Processing block batch of size 50 starting from  0x1f6094b4... 2025566/2028558 - estimated time remaining 34s" blocksPerSecond=86.8 peers=23 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x49062e933cc68dd228a165e5a12b5082d5374b40b6589fd07b6d1f224d126abe" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=info msg="Processing block batch of size 50 starting from  0x4c1d8de8... 2025630/2028558 - estimated time remaining 32s" blocksPerSecond=89.3 peers=23 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x11fca032bbc701a419cd26e076a38e669036ad7389f567d5698a82520d53e6c2" prefix=initial-sync

@nisdas
Copy link
Member

nisdas commented Aug 27, 2021

hey @iSOcH @tkachevaa , this will be patched in our next v2 release candidate. #9452 has the fix, our tentative time for our next release will be most likely at the start of next week.

@tkachevaa
Copy link

You definitely need 2.0.0 for pyrmont testnet after the altair fork.
But this issue is about the new version also having problems, especially since/during the non-finality tests

The errors of my node now changed after restarting it like 12h ago:

eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x439281003306611f9d7dd144a78f2ecbf1c0bd64443ccdc2a0e5ac342d2eec20" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 48 starting from  0x5e98559d... 2025822/2028557 - estimated time remaining 30s" blocksPerSecond=89.2 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x51f6132c05886741717ba5b5fc2ebe908dca31e5eb1ac7a06694c6f5d749e936" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 48 starting from  0x8779917d... 2025886/2028557 - estimated time remaining 29s" blocksPerSecond=91.6 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xa98840ae63c3147733b5ceb8cba8a50c74571f212d62c36d777a89516c4d7d49" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 37 starting from  0x573ac213... 2025950/2028557 - estimated time remaining 27s" blocksPerSecond=93.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x43cfbd9541ceb493074417fbad4df3c4eaf81d6db512dc2aa4fb147132cabdb9" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=info msg="Processing block batch of size 51 starting from  0xf2a2b888... 2026014/2028557 - estimated time remaining 26s" blocksPerSecond=96.0 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:40" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8cc16f348db1a24b94e60b46a38340bdf726ba593f75933f911fc3ae388b47d4" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 50 starting from  0x1f6094b4... 2025566/2028558 - estimated time remaining 37s" blocksPerSecond=79.3 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x49062e933cc68dd228a165e5a12b5082d5374b40b6589fd07b6d1f224d126abe" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 50 starting from  0x4c1d8de8... 2025630/2028558 - estimated time remaining 35s" blocksPerSecond=81.8 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x11fca032bbc701a419cd26e076a38e669036ad7389f567d5698a82520d53e6c2" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 48 starting from  0xee70212f... 2025696/2028558 - estimated time remaining 33s" blocksPerSecond=84.2 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x2fa61c284991e209355d27193722ba0003c9bdae6380c0f4a0ee68720bcaab32" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 52 starting from  0xaac9e556... 2025758/2028558 - estimated time remaining 32s" blocksPerSecond=86.8 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x439281003306611f9d7dd144a78f2ecbf1c0bd64443ccdc2a0e5ac342d2eec20" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 48 starting from  0x5e98559d... 2025822/2028558 - estimated time remaining 30s" blocksPerSecond=89.2 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x51f6132c05886741717ba5b5fc2ebe908dca31e5eb1ac7a06694c6f5d749e936" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 48 starting from  0x8779917d... 2025886/2028558 - estimated time remaining 29s" blocksPerSecond=91.6 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xa98840ae63c3147733b5ceb8cba8a50c74571f212d62c36d777a89516c4d7d49" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 37 starting from  0x573ac213... 2025950/2028558 - estimated time remaining 27s" blocksPerSecond=93.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x43cfbd9541ceb493074417fbad4df3c4eaf81d6db512dc2aa4fb147132cabdb9" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=info msg="Processing block batch of size 51 starting from  0xf2a2b888... 2026014/2028558 - estimated time remaining 26s" blocksPerSecond=96.0 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:43" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8cc16f348db1a24b94e60b46a38340bdf726ba593f75933f911fc3ae388b47d4" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 50 starting from  0x1f6094b4... 2025566/2028558 - estimated time remaining 31s" blocksPerSecond=96.0 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x49062e933cc68dd228a165e5a12b5082d5374b40b6589fd07b6d1f224d126abe" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 50 starting from  0x4c1d8de8... 2025630/2028558 - estimated time remaining 29s" blocksPerSecond=98.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x11fca032bbc701a419cd26e076a38e669036ad7389f567d5698a82520d53e6c2" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 52 starting from  0xaac9e556... 2025758/2028558 - estimated time remaining 27s" blocksPerSecond=101.1 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x439281003306611f9d7dd144a78f2ecbf1c0bd64443ccdc2a0e5ac342d2eec20" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 48 starting from  0x5e98559d... 2025822/2028558 - estimated time remaining 26s" blocksPerSecond=103.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x51f6132c05886741717ba5b5fc2ebe908dca31e5eb1ac7a06694c6f5d749e936" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 48 starting from  0x8779917d... 2025886/2028558 - estimated time remaining 25s" blocksPerSecond=105.9 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xa98840ae63c3147733b5ceb8cba8a50c74571f212d62c36d777a89516c4d7d49" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=info msg="Processing block batch of size 51 starting from  0xf2a2b888... 2026014/2028558 - estimated time remaining 23s" blocksPerSecond=108.5 peers=26 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:46" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x8cc16f348db1a24b94e60b46a38340bdf726ba593f75933f911fc3ae388b47d4" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=info msg="Processing block batch of size 50 starting from  0x1f6094b4... 2025566/2028558 - estimated time remaining 34s" blocksPerSecond=86.8 peers=23 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x49062e933cc68dd228a165e5a12b5082d5374b40b6589fd07b6d1f224d126abe" prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=info msg="Processing block batch of size 50 starting from  0x4c1d8de8... 2025630/2028558 - estimated time remaining 32s" blocksPerSecond=89.3 peers=23 prefix=initial-sync
eth2_1       | time="2021-08-27 05:51:50" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x11fca032bbc701a419cd26e076a38e669036ad7389f567d5698a82520d53e6c2" prefix=initial-sync

Now i got the same errors

@tkachevaa
Copy link

With v2.0.0-rc.1 problem was solved, synced and start to validate

@pdyraga
Copy link

pdyraga commented Sep 1, 2021

I tried to use beacon-chain-v2.0.0-rc.1-linux-amd64 on Pyrmont today, starting it with just --pyrmont and --config-file option` with the following config file content:

http-web3provider: 'https://goerli.infura.io/v3/<censored>'
datadir: '/home/eth2/pyrmont/data/beacon'
log-file: '/var/log/eth2/pyrmont/beacon.log'
accept-terms-of-use: true

The data dir was empty.

This is what I've got:

time="2021-09-01 09:26:54" level=warning msg="Running on Pyrmont Testnet" prefix=flags
time="2021-09-01 09:26:54" level=info msg="Checking DB" database-path="/home/eth2/pyrmont/data/beacon/beaconchaindata" prefix=node
time="2021-09-01 09:26:54" level=info msg="Deposit contract: 0x8c5fecdc472e27bc447696f431e425d02dd46a8c" prefix=node
time="2021-09-01 09:26:54" level=info msg="Waiting for state to be initialized" prefix=initial-sync
time="2021-09-01 09:26:54" level=info msg="Starting beacon node" prefix=node version="Prysm/v2.0.0-rc.1/49dce52ae963966975e9c4555e2c032f5a6baa9e. Built at: 2021-08-31 15:13:25+00:00"
time="2021-09-01 09:26:54" level=info msg="gRPC server listening on port" address="127.0.0.1:4000" prefix=rpc
time="2021-09-01 09:26:54" level=warning msg="You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=rpc
time="2021-09-01 09:26:54" level=info msg="Starting API middleware" API middleware address="127.0.0.1:3501" prefix=gateway
time="2021-09-01 09:26:54" level=info msg="Starting gRPC gateway" address="127.0.0.1:3500" prefix=gateway
time="2021-09-01 09:26:54" level=info msg="Waiting to reach the validator deposit threshold to start the beacon chain..." prefix=blockchain
time="2021-09-01 09:26:55" level=info msg="Connected to eth1 proof-of-work chain" endpoint="https://goerli.infura.io/***" prefix=powchain
time="2021-09-01 09:26:55" level=info msg="New gRPC client connected to beacon node" addr="127.0.0.1:47796" prefix=rpc
time="2021-09-01 09:27:00" level=info msg="Processing deposits from Ethereum 1 chain" deposits=512 genesisValidators=512 prefix=powchain
time="2021-09-01 09:27:01" level=info msg="Processing deposits from Ethereum 1 chain" deposits=1024 genesisValidators=1024 prefix=powchain
time="2021-09-01 09:27:07" level=info msg="Processing deposits from Ethereum 1 chain" deposits=1536 genesisValidators=1536 prefix=powchain
time="2021-09-01 09:27:09" level=info msg="Processing deposits from Ethereum 1 chain" deposits=2048 genesisValidators=2048 prefix=powchain
time="2021-09-01 09:27:12" level=info msg="Processing deposits from Ethereum 1 chain" deposits=2560 genesisValidators=2560 prefix=powchain
time="2021-09-01 09:27:14" level=info msg="Processing deposits from Ethereum 1 chain" deposits=3072 genesisValidators=3072 prefix=powchain
time="2021-09-01 09:27:16" level=info msg="Processing deposits from Ethereum 1 chain" deposits=3584 genesisValidators=3584 prefix=powchain

(...)

time="2021-09-01 10:47:13" level=info msg="Processing deposits from Ethereum 1 chain" deposits=98304 genesisValidators=98304 prefix=powchain
time="2021-09-01 10:47:49" level=info msg="Processing deposits from Ethereum 1 chain" deposits=98816 genesisValidators=98816 prefix=powchain
time="2021-09-01 10:48:25" level=info msg="Processing deposits from Ethereum 1 chain" deposits=99328 genesisValidators=99328 prefix=powchain
time="2021-09-01 10:49:02" level=info msg="Processing deposits from Ethereum 1 chain" deposits=99840 genesisValidators=99840 prefix=powchain
time="2021-09-01 10:49:17" level=info msg="Minimum number of validators reached for beacon-chain to start" ChainStartTime=2020-11-18 12:00:07 +0000 UTC prefix=powchain
time="2021-09-01 10:49:25" level=info msg="Initialized beacon chain genesis state" prefix=blockchain
time="2021-09-01 10:49:25" level=info msg="Chain genesis time reached" genesisStateRoot=eccac996d92ce12bd7d071d331b930ef21375e3a3db83fa62140a9ef50340a14 genesisTime="2020-11-18 12:00:07 +0000 UTC" genesisValidators=100010 prefix=slotutil
time="2021-09-01 10:49:25" level=info msg="Starting initial chain sync..." prefix=initial-sync
time="2021-09-01 10:49:25" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=0
time="2021-09-01 10:49:25" level=info msg="Started discovery v5" ENR="enr:-Ly4QHrqxydyCvROmkcWOBH8tUdt3ABbkYj_CgohjPlQ5zm2cJy-ggFuK3LnDzpbPvmJM0S1BSoainPh_VTZ9ngiOPEBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhAoBAASJc2VjcDI1NmsxoQP1CYryrRzdMwl0-XGacQu6zSbR_2th6mAX-1WSnm1HvYhzeW5jbmV0cwCDdGNwgjLIg3VkcIIu4A" prefix=p2p
time="2021-09-01 10:49:25" level=info msg="Node started p2p server" multiAddr="/ip4/10.1.0.4/tcp/13000/p2p/16Uiu2HAmV9WgBA61VrJD5ub4tFvq3NqPedVRMgRbnaTADvAzpK2t" prefix=p2p
time="2021-09-01 10:49:30" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=1
time="2021-09-01 10:49:35" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=1
time="2021-09-01 10:49:44" level=info msg="Processing block batch of size 45 starting from  0x30c04689... 1/2066048 - estimated time remaining 255h4m3s" blocksPerSecond=2.2 peers=6 prefix=initial-sync
time="2021-09-01 10:49:44" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x6a89af5df908893eedbed10ba4c13fc13d5653ce57db637e3bfded73a987bb87" prefix=initial-sync
time="2021-09-01 10:49:44" level=info msg="Processing block batch of size 46 starting from  0xe73d20ea... 66/2066048 - estimated time remaining 126h7m41s" blocksPerSecond=4.5 peers=6 prefix=initial-sync
time="2021-09-01 10:49:44" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x891fcd12de76b6694bc5e39bd9992b9971fc07f67b201e53fd5565dec59dae6f" prefix=initial-sync
time="2021-09-01 10:49:44" level=info msg="Processing block batch of size 41 starting from  0x542c17b2... 128/2066048 - estimated time remaining 86h56m58s" blocksPerSecond=6.6 peers=6 prefix=initial-sync
time="2021-09-01 10:49:44" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xab7ed10549204cb9d0ba79225e96662284647784353f545d38d781890d9a4306" prefix=initial-sync
time="2021-09-01 10:49:44" level=info msg="Processing block batch of size 51 starting from  0xc0c0c2df... 193/2066048 - estimated time remaining 62h42m56s" blocksPerSecond=9.2 peers=6 prefix=initial-sync
time="2021-09-01 10:49:44" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x6ad92bbe5c8bb74faeffda56614d99b7db194c1880e98cbc9b279e66e2687cdf" prefix=initial-sync
time="2021-09-01 10:49:44" level=info msg="Processing block batch of size 49 starting from  0x23e933aa... 256/2066048 - estimated time remaining 49h28m5s" blocksPerSecond=11.6 peers=6 prefix=initial-sync
time="2021-09-01 10:49:44" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0x22998c70826834834190e011cedb821bc09aa2766a991d91ad77f8996822ba13" prefix=initial-sync
time="2021-09-01 10:49:44" level=info msg="Processing block batch of size 55 starting from  0x714a9815... 321/2066048 - estimated time remaining 39h59m13s" blocksPerSecond=14.3 peers=6 prefix=initial-sync
time="2021-09-01 10:49:44" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xce893382401052d31462e3dc04587b66123d55dc8f7a3bce2cdcc4389f26ac87" prefix=initial-sync
time="2021-09-01 10:49:44" level=info msg="Processing block batch of size 61 starting from  0x1db33102... 384/2066048 - estimated time remaining 32h58m36s" blocksPerSecond=17.4 peers=6 prefix=initial-sync
time="2021-09-01 10:49:44" level=warning msg="Batch is not processed" error="beacon node doesn't have a parent in db with root: 0xdb832584ba9e66ec397f3ec47173efec129e03e7065c7d8f2b500f79b89981d9" prefix=initial-sync

and it continues to be like that for the last 3 hours.

@nisdas
Copy link
Member

nisdas commented Sep 1, 2021

So this is out of our control @pdyraga . Geth had a bug which broke blockhash computation for clique networks.
ethereum/go-ethereum#23463

You can either run your own updated geth node with the fix or wait till its an official release and infura updates its nodes with the fix.

Alternatively, you can run with a --genesis-state=/path/to/genesis flag and use the genesis state here:
https://github.com/eth2-clients/eth2-networks/blob/master/shared/pyrmont/genesis.ssz

@kolydart
Copy link

kolydart commented Sep 2, 2021

I started my first testnet attempts 3 days ago and I kept receiving similar errors (beacon node doesn't have a parent in db with root: 0x...). I thought, if bugs like this is the norm, I should stay away. @nisdas thanks for the explanation.

@gregbkr gregbkr mentioned this issue Sep 13, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Priority: High High priority item
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Patch Initial Sync For Non Finalized Blocks prysmaticlabs/prysm
8 participants
@pdyraga @gregbkr @prestonvanloon @kolydart @iSOcH @terencechain @nisdas @tkachevaa