Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'GasLimit' field cannot be greater than a block's gas limit! #9103

Open
8 of 18 tasks
stuberman opened this issue Jul 30, 2022 · 20 comments
Open
8 of 18 tasks

'GasLimit' field cannot be greater than a block's gas limit! #9103

stuberman opened this issue Jul 30, 2022 · 20 comments
Labels
area/proving Area: Proving kind/bug Kind: Bug LM-tech-debt need/analysis Hint: Needs Analysis P1 P1: Must be resolved
Milestone

Comments

@stuberman
Copy link

stuberman commented Jul 30, 2022

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
  • This is not a new feature request. If it is, please file a feature request instead.
  • This is not an enhancement request. If it is, please file a improvement suggestion instead.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to lotus.

Lotus component

  • lotus daemon - chain sync
  • lotus miner - mining and block production
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt)
  • lotus miner/market - storage deal
  • lotus miner/market - retrieval deal
  • lotus miner/market - data transfer
  • lotus client
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Lotus Version

lotus-miner version
Daemon:  1.16.1+mainnet+git.a2fa07d35+api1.5.0
Local: lotus-miner version 1.16.1+mainnet+git.a2fa07d35

Describe the Bug

Unable to post one of 4 deadlines two days in a row:
See closed issue 8997

lotus-miner proving info

Miner: f01278
Current Epoch: 2029687
Proving Period Boundary: 2029
Proving Period Start: 2029549 (1 hour 9 minutes ago)
Next Period Start: 2032429 (in 22 hours 51 minutes)

Faults: 2125 (58.00%)
Recovering: 2125
Deadline Index: 2
Deadline Sectors: 2126
Deadline Open: 2029669 (9 minutes ago)
Deadline Close: 2029729 (in 21 minutes)
Deadline Challenge: 2029649 (19 minutes ago)
Deadline FaultCutoff: 2029599 (44 minutes ago)

lotus-miner proving deadlines

Miner: f01278

deadline partitions sectors (faults) proven partitions
0 1 1449 (0) 0
1 1 53 (0) 0
2 1 2215 (2125) 0 (current)
3 1 37 (0) 0

Logging Information

2022-07-30T15:58:14.500 INFO storage_proofs_core::compound_proof > snark_proof:finish
2022-07-30T15:58:14.501 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2022-07-30T15:58:14.703 INFO filcrypto::util::types > generate_window_post: end
2022-07-30T15:58:14.727Z	INFO	stores	stores/http_handler.go:189	SERVE Alloc check /remote/unsealed/s-t01278-2830/8/allocated/0/34091302912
2022-07-30T15:58:14.763Z	INFO	storageminer	storage/wdpost_run.go:672	computing window post	{"batch": 0, "elapsed": 581.534384937, "skip": 0, "err": null}
2022-07-30T15:58:14.794 INFO filcrypto::util::types > verify_window_post: start
2022-07-30T15:58:14.795Z	INFO	stores	stores/http_handler.go:189	SERVE Alloc check /remote/unsealed/s-t01278-2830/8/allocated/0/34091302912
2022-07-30T15:58:14.824 INFO filecoin_proofs::api::window_post > verify_window_post:start
2022-07-30T15:58:14.826 INFO filecoin_proofs::caches > trying parameters memory cache for: WINDOW_POST[34359738368]-verifying-key
2022-07-30T15:58:14.826 INFO filecoin_proofs::caches > found params in memory cache for WINDOW_POST[34359738368]-verifying-key
2022-07-30T15:58:17.387 INFO filecoin_proofs::api::window_post > verify_window_post:finish
2022-07-30T15:58:17.388 INFO filcrypto::util::types > verify_window_post: end
2022-07-30T15:58:19.967Z	ERROR	storageminer	storage/wdpost_run.go:181	submit window post failed: pushing message to mpool:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).submitPoStMessage
        /home/stuart/lotus/storage/wdpost_run.go:885
  - mpool push: failed to push message: message not valid for block inclusion: 'GasLimit' field cannot be greater than a block's gas limit
2022-07-30T15:58:19.968Z	WARN	storageminer	storage/wdpost_changehandler.go:494	Aborted window post Submitting (Deadline: &{CurrentEpoch:2029669 PeriodStart:2029549 Index:2 Open:2029669 Close:2029729 Challenge:2029649 FaultCutoff:2029599 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})
2022-07-30T15:58:30.917Z	ERROR	storageminer	storage/wdpost_run.go:181	submit window post failed: pushing message to mpool:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).submitPoStMessage
        /home/stuart/lotus/storage/wdpost_run.go:885
  - mpool push: failed to push message: message not valid for block inclusion: 'GasLimit' field cannot be greater than a block's gas limit
2022-07-30T15:58:30.917Z	WARN	storageminer	storage/wdpost_changehandler.go:494	Aborted window post Submitting (Deadline: &{CurrentEpoch:2029669 PeriodStart:2029549 Index:2 Open:2029669 Close:2029729 Challenge:2029649 FaultCutoff:2029599 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})
2022-07-30T15:58:36.307Z	INFO	miner	miner/miner.go:479	completed mineOne	{"tookMilliseconds": 8, "forRound": 2029678, "baseEpoch": 2029677, "baseDeltaSeconds": 6, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2125523, "lookbackEpochs": 900, "networkPowerAtLookback": "20724732860518498304", "minerPowerAtLookback": "375987163136000", "isEligible": true, "isWinner": false, "error": null}
2022-07-30T15:59:30.823Z	ERROR	storageminer	storage/wdpost_run.go:181	submit window post failed: pushing message to mpool:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).submitPoStMessage
        /home/stuart/lotus/storage/wdpost_run.go:885
  - mpool push: failed to push message: message not valid for block inclusion: 'GasLimit' field cannot be greater than a block's gas limit
2022-07-30T15:59:30.823Z	WARN	storageminer	storage/wdpost_changehandler.go:494	Aborted window post Submitting (Deadline: &{CurrentEpoch:2029669 PeriodStart:2029549 Index:2 Open:2029669 Close:2029729 Challenge:2029649 FaultCutoff:2029599 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})
2022-07-30T16:00:04.052Z	ERROR	storageminer	storage/wdpost_run.go:181	submit window post failed: pushing message to mpool:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).submitPoStMessage
        /home/stuart/lotus/storage/wdpost_run.go:885
  - mpool push: failed to push message: message not valid for block inclusion: 'GasLimit' field cannot be greater than a block's gas limit
2022-07-30T16:00:04.053Z	WARN	storageminer	storage/wdpost_changehandler.go:494	Aborted window post Submitting (Deadline: &{CurrentEpoch:2029669 PeriodStart:2029549 Index:2 Open:2029669 Close:2029729 Challenge:2029649 FaultCutoff:2029599 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:

Repo Steps

See closed issue -this is not resolved in v1.16.1

#8997

@stuberman
Copy link
Author

Additional note: This deadline (2) was not compacted but deadlines 0 and 1 had been compacted last week.

@jennijuju
Copy link
Member

Did you set the MaxPartitionsPerPoStMessage And MaxPartitionsPerRecoveryMessage?

@jennijuju
Copy link
Member

I would recommend lower it, especially the recovery one first!

@stuberman
Copy link
Author

I only have one partition (first four deadlines)
Where is this parameter? Is this new?

@stuberman
Copy link
Author

BTW - I found it in lotus-miner config default.
My miner is using the defaults (not set) so should be '0' (unlimited)????

@stuberman
Copy link
Author

stuberman commented Jul 30, 2022

It is important to note that I have been running v1.16.1 without problem for the last few weeks and this problem only showed itself two days ago on July 29. (So today is my second day of faulting on deadline 2.)

Frankly this makes no sense to me. I only have one partition and changing from default (0 unlimited) to '1' should make no difference. I did now add the [Proving] block and reset the value to '1'.

This is going to keep costing me penalties until this bug is fixed.

In some cases when submitting DeclareFaultsRecovered messages,
there may be too many recoveries to fit in a BlockGasLimit.
In those cases it may be necessary to set this value to something low (eg 1);
Note that setting this value lower may result in less efficient gas use - more messages will be sent than needed,
resulting in more total gas use (but each message will have lower gas limit)

type: int
env var: LOTUS_PROVING_MAXPARTITIONSPERRECOVERYMESSAGE
MaxPartitionsPerRecoveryMessage = 0

@Reiers Reiers added area/proving Area: Proving need/analysis Hint: Needs Analysis and removed need/triage labels Jul 30, 2022
@stuberman
Copy link
Author

Cost so far

Time Message ID From To Net Income Type
2022-07-30 12:24:00 N/A f01278 f099 -34.2931 FIL Burn
2022-07-29 12:24:00 N/A f01278 f099 -34.2571 FIL Burn

@stuberman
Copy link
Author

New .lotusminer/config.toml settings

[Proving]
MaxPartitionsPerPoStMessage = 1
MaxPartitionsPerRecoveryMessage = 1

@rjan90
Copy link
Contributor

rjan90 commented Jul 31, 2022

First of all, this issue is above my understanding of how the GasLimit is calculated, so these are just some observations for further context:

  • Your GasLimit for the windowPoSt-message for deadline 2 (27th of July) before the power-loss on the 28th of July was 38,586,733 which is well below the block gas limit of 10B. So there should not be a problem to post a message containing 2200+ sectors.
  • The same goes for the DeclareFaultsRecovered message posted on July 30th, which had a gas limit of 923,057,115 and where able to be posted to the chain.

Do you have the epochs before and after the block inclusion: 'GasLimit' field error for the windowPoSt messages?

[Proving]
MaxPartitionsPerPoStMessage = 1
MaxPartitionsPerRecoveryMessage = 1

Changing from 0 to 1 should not have any impact at all, since you only have 1 partition. Changing this value only makes sense when you have more then 2-3 partitions in a deadline.

Only thing I can think of that maybe could be beneficial before someone more knowledge about GasLimits takes a look at it is to enable LOTUS_VM_ENABLE_TRACING=1 to get traces in case it happens again.

@stuberman
Copy link
Author

Do you have the epochs before and after the block inclusion: 'GasLimit' field error for the windowPoSt messages?

I have since restarted the lotus-miner with the new settings so no older logs now. After the wdpost fails again in a few hours I can collect more log info.

I also added the envvar "LOTUS_VM_ENABLE_TRACING=1" to the miner/lotus node.

@stuberman
Copy link
Author

Additional context

July 28 loss of all power due to Internet outage on all four deadlines (chain not in sync unable to PoST any messages)
July 29 all four deadlines declare "DeclareFaultsRecovered" but only three deadlines PoST successfully, deadline '2' throws 'GasLimit' error
July 30 deadline '2' declares "DeclareFaultsRecovered" and again deadline '2' throws 'GasLimit' error

@stuberman
Copy link
Author

stuberman commented Jul 31, 2022

Start of wdpost for July 31 (both deadline 0 and deadline 2 (DeclareFaultsRecovered for 2125 sectors)

2022-07-31T14:53:36.053Z WARN storageminer storage/wdpost_run.go:237 Checked sectors {"checked": 1443, "good": 1443}
2022-07-31T14:53:36.072Z WARN storageminer storage/wdpost_run.go:237 Checked sectors {"checked": 2125, "good": 2125}
2022-07-31T14:53:36.098Z INFO storageminer storage/wdpost_run.go:657 running window post {"chain-random": "uK+N5DVTvX2ERJS5WF1DnSozPR9i9uAQcEpNinyb9GI=", "deadline": {"CurrentEpoch":2032410,"PeriodStart":2032429,"Index":0,"Open":2032429,"Close":2032489,"Challenge":2032409,"FaultCutoff":2032359,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70}, "height": "2032410", "skipped": 0}
2022-07-31T14:53:36.098Z INFO advmgr sector-storage/manager_post.go:81 GenerateWindowPoSt run at lotus-miner
2022-07-31T14:53:36.108 INFO filcrypto::util::types > generate_window_post: start
2022-07-31T14:53:36.133 INFO filecoin_proofs::api::window_post > generate_window_post:start
2022-07-31T14:53:36.133 INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[34359738368]
2022-07-31T14:53:36.133 INFO filecoin_proofs::caches > no params in memory cache for Window_POST[34359738368]
2022-07-31T14:53:36.164 INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: FallbackPoSt::PublicParams{sector_size: 34359738368, challenge_count: 10, sector_count: 2349}
2022-07-31T14:53:36.164 INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.params" exist
2022-07-31T14:53:36.164 INFO storage_proofs_core::parameter_cache > checking cache_path: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.params" for parameters
2022-07-31T14:53:36.164 INFO storage_proofs_core::parameter_cache > Verify production parameters is false
2022-07-31T14:59:14.666 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2022-07-31T14:59:15.164 INFO storage_proofs_core::compound_proof > snark_proof:start
2022-07-31T14:59:43.405 INFO bellperson::groth16::prover > synthesis time: 28.163577451s
2022-07-31T14:59:43.405 INFO bellperson::groth16::prover > starting proof timer2022-07-31T14:59:15.241 INFO bellperson::groth16::prover > Bellperson 0.21.0 is being used!
2022-07-31T14:59:47.149 INFO bellperson::gpu::locks > GPU is available for FFT!
2022-07-31T14:59:47.149 INFO bellperson::gpu::program > Using kernel on CUDA.
2022-07-31T14:59:47.218 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2022-07-31T14:59:47.218 INFO bellperson::gpu::fft > FFT: Device 0: NVIDIA GeForce RTX 2080 Ti
2022-07-31T14:59:47.218 INFO bellperson::domain > GPU FFT kernel instantiated!
2022-07-31T15:00:09.549 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2022-07-31T15:00:09.550 INFO bellperson::gpu::program > Using kernel on CUDA.
2022-07-31T15:00:09.577 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
2022-07-31T15:00:09.577 INFO bellperson::gpu::multiexp > Multiexp: Device 0: NVIDIA GeForce RTX 2080 Ti (Chunk-size: 7488999)
2022-07-31T15:00:09.578 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2022-07-31T15:04:11.109 INFO bellperson::groth16::prover > prover time: 267.703182821s
2022-07-31T15:04:12.932 INFO storage_proofs_core::compound_proof > snark_proof:finish
2022-07-31T15:04:12.933 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2022-07-31T15:04:12.942Z INFO stores stores/http_handler.go:189 SERVE Alloc check /remote/unsealed/s-t01278-3833/8/allocated/0/34091302912
2022-07-31T15:04:12.988 INFO filcrypto::util::types > generate_window_post: end
2022-07-31T15:04:12.998Z INFO storageminer storage/wdpost_run.go:672 computing window post {"batch": 0, "elapsed": 636.899598358, "skip": 0, "err": null}
2022-07-31T15:04:13.001 INFO filcrypto::util::types > verify_window_post: start
2022-07-31T15:04:13.002Z INFO stores stores/http_handler.go:189 SERVE Alloc check /remote/unsealed/s-t01278-3833/8/allocated/0/34091302912
2022-07-31T15:04:13.012 INFO filecoin_proofs::api::window_post > verify_window_post:start
2022-07-31T15:04:13.013 INFO filecoin_proofs::caches > trying parameters memory cache for: WINDOW_POST[34359738368]-verifying-key
2022-07-31T15:04:13.013 INFO filecoin_proofs::caches > no params in memory cache for WINDOW_POST[34359738368]-verifying-key
2022-07-31T15:04:13.046 INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: FallbackPoSt::PublicParams{sector_size: 34359738368, challenge_count: 10, sector_count: 2349}
2022-07-31T15:04:13.047 INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk" exist
2022-07-31T15:04:13.047 INFO storage_proofs_core::parameter_cache > checking cache_path: "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk" for verifying key
2022-07-31T15:04:14.108 INFO storage_proofs_core::parameter_cache > read verifying key from cache "/home/stuart/parms/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk
2022-07-31T15:04:15.626 INFO filecoin_proofs::api::window_post > verify_window_post:finish
2022-07-31T15:04:15.626 INFO filcrypto::util::types > verify_window_post: end
2022-07-31T15:04:17.337Z INFO storageminer storage/wdpost_run.go:888 Submitted window post: bafy2bzacebyqsyvpofy2tf72ywuicp4xh347gvk2vblsk3bujcjvg5myt4aww (deadline 0)

lotus-miner proving info

Miner: f01278
Current Epoch: 2032453
Proving Period Boundary: 2029
Proving Period Start: 2032429 (12 minutes ago)
Next Period Start: 2035309 (in 23 hours 48 minutes)

Faults: 2125 (58.00%)
Recovering: 2125
Deadline Index: 0
Deadline Sectors: 1443
Deadline Open: 2032429 (12 minutes ago)
Deadline Close: 2032489 (in 18 minutes)
Deadline Challenge: 2032409 (22 minutes ago)
Deadline FaultCutoff: 2032359 (47 minutes ago)

lotus-miner proving deadlines

Miner: f01278

deadline partitions sectors (faults) proven partitions
0 1 1449 (0) 1 (current)
1 1 53 (0) 0
2 1 2219 (2125) 0
3 1 37 (0) 0
4 0 0 (0) 0
5 0 0 (0) 0

@stuberman
Copy link
Author

July 31 Deadline 2 wdpost

2022-07-31T15:48:37.957Z WARN storageminer storage/wdpost_run.go:237 Checked sectors {"checked": 2129, "good": 2129}
2022-07-31T15:48:38.014Z INFO storageminer storage/wdpost_run.go:657 running window post {"chain-random": "T4cOGrdYOVZyq6EuEMgBJfR/3UnHFXvQlacIpC4hIfo=", "deadline": {"CurrentEpoch":2032530,"PeriodStart":2032429,"Index":2,"Open":2032549,"Close":2032609,"Challenge":2032529,"FaultCutoff":2032479,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70}, "height": "2032530", "skipped": 0}
2022-07-31T15:48:38.014Z INFO advmgr sector-storage/manager_post.go:81 GenerateWindowPoSt run at lotus-miner
2022-07-31T15:48:38.031 INFO filcrypto::util::types > generate_window_post: start
2022-07-31T15:48:38.068 INFO filecoin_proofs::api::window_post > generate_window_post:start
2022-07-31T15:48:38.068 INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[34359738368]
2022-07-31T15:48:38.068 INFO filecoin_proofs::caches > found params in memory cache for Window_POST[34359738368]
2022-07-31T15:48:40.546 INFO storage_proofs_core::compound_proof > vanilla_proofs:start
2022-07-31T15:53:37.767 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2022-07-31T15:53:38.527 INFO storage_proofs_core::compound_proof > snark_proof:start
2022-07-31T15:53:38.626 INFO bellperson::groth16::prover > Bellperson 0.21.0 is being used!
2022-07-31T15:54:05.268 INFO bellperson::groth16::prover > synthesis time: 26.641769226s
2022-07-31T15:54:05.268 INFO bellperson::groth16::prover > starting proof timer
2022-07-31T15:54:08.701 INFO bellperson::gpu::locks > GPU is available for FFT!
2022-07-31T15:54:08.701 INFO bellperson::gpu::program > Using kernel on CUDA.
2022-07-31T15:54:08.752 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2022-07-31T15:54:08.752 INFO bellperson::gpu::fft > FFT: Device 0: NVIDIA GeForce RTX 2080 Ti
2022-07-31T15:54:08.752 INFO bellperson::domain > GPU FFT kernel instantiated!
2022-07-31T15:54:30.452 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2022-07-31T15:54:30.454 INFO bellperson::gpu::program > Using kernel on CUDA.
2022-07-31T15:54:30.472 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
2022-07-31T15:54:30.472 INFO bellperson::gpu::multiexp > Multiexp: Device 0: NVIDIA GeForce RTX 2080 Ti (Chunk-size: 7488999)
2022-07-31T15:54:30.472 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2022-07-31T15:58:06.435 INFO bellperson::groth16::prover > prover time: 241.165598194s
2022-07-31T15:58:08.820 INFO storage_proofs_core::compound_proof > snark_proof:finish
2022-07-31T15:58:08.821 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2022-07-31T15:58:08.891 INFO filcrypto::util::types > generate_window_post: end
2022-07-31T15:58:08.899Z INFO storageminer storage/wdpost_run.go:672 computing window post {"batch": 0, "elapsed": 570.884984786, "skip": 0, "err": null}
2022-07-31T15:58:08.904 INFO filcrypto::util::types > verify_window_post: start
2022-07-31T15:58:08.919 INFO filecoin_proofs::api::window_post > verify_window_post:start
2022-07-31T15:58:08.921 INFO filecoin_proofs::caches > trying parameters memory cache for: WINDOW_POST[34359738368]-verifying-key
2022-07-31T15:58:08.921 INFO filecoin_proofs::caches > found params in memory cache for WINDOW_POST[34359738368]-verifying-key
2022-07-31T15:58:09.409 INFO filecoin_proofs::api::window_post > verify_window_post:finish
2022-07-31T15:58:09.409 INFO filcrypto::util::types > verify_window_post: end
2022-07-31T15:58:19.419Z ERROR storageminer storage/wdpost_run.go:181 submit window post failed: pushing message to mpool:
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).submitPoStMessage
/home/stuart/lotus/storage/wdpost_run.go:885

  • mpool push: failed to push message: message not valid for block inclusion: 'GasLimit' field cannot be greater than a block's gas limit
    2022-07-31T15:58:19.419Z WARN storageminer storage/wdpost_changehandler.go:494 Aborted window post Submitting (Deadline: &{CurrentEpoch:2032549 PeriodStart:2032429 Index:2 Open:2032549 Close:2032609 Challenge:2032529 FaultCutoff:2032479 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})

2022-07-31T15:58:30.893Z ERROR storageminer storage/wdpost_run.go:181 submit window post failed: pushing message to mpool:
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).submitPoStMessage
/home/stuart/lotus/storage/wdpost_run.go:885

  • mpool push: failed to push message: message not valid for block inclusion: 'GasLimit' field cannot be greater than a block's gas limit
    2022-07-31T15:58:30.893Z WARN storageminer storage/wdpost_changehandler.go:494 Aborted window post Submitting (Deadline: &{CurrentEpoch:2032549 PeriodStart:2032429 Index:2 Open:2032549 Close:2032609 Challenge:2032529 FaultCutoff:2032479 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})

2022-07-31T15:58:32.272Z ERROR storageminer storage/wdpost_run.go:181 submit window post failed: pushing message to mpool:
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).submitPoStMessage
/home/stuart/lotus/storage/wdpost_run.go:885

  • mpool push: failed to push message: message not valid for block inclusion: 'GasLimit' field cannot be greater than a block's gas limit
    2022-07-31T15:58:32.272Z WARN storageminer storage/wdpost_changehandler.go:494 Aborted window post Submitting (Deadline: &{CurrentEpoch:2032549 PeriodStart:2032429 Index:2 Open:2032549 Close:2032609 Challenge:2032529 FaultCutoff:2032479 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})

And continues to repeat....

@stuberman
Copy link
Author

stuberman commented Jul 31, 2022

lotus-miner proving info

Miner: f01278
Current Epoch: 2032563
Proving Period Boundary: 2029
Proving Period Start: 2032429 (1 hour 7 minutes ago)
Next Period Start: 2035309 (in 22 hours 53 minutes)

Faults: 2125 (58.00%)
Recovering: 2125
Deadline Index: 2
Deadline Sectors: 2129
Deadline Open: 2032549 (7 minutes ago)
Deadline Close: 2032609 (in 23 minutes)
Deadline Challenge: 2032529 (17 minutes ago)
Deadline FaultCutoff: 2032479 (42 minutes ago)

lotus-miner proving info (40 minutes later)

Miner: f01278
Current Epoch: 2032645
Proving Period Boundary: 2029
Proving Period Start: 2032429 (1 hour 48 minutes ago)
Next Period Start: 2035309 (in 22 hours 12 minutes)

Faults: 2127 (58.00%)
Recovering: 0
Deadline Index: 3
Deadline Sectors: 36
Deadline Open: 2032609 (18 minutes ago)
Deadline Close: 2032669 (in 12 minutes)
Deadline Challenge: 2032589 (28 minutes ago)
Deadline FaultCutoff: 2032539 (53 minutes ago)

lotus-miner proving deadlines
Miner: f01278

deadline partitions sectors (faults) proven partitions
0 1 1449 (0) 0
1 1 53 (0) 0
2 1 2219 (2127) 0
3 1 37 (0) 1 (current)
4 0 0 (0) 0
5 0 0 (0) 0

@stuberman
Copy link
Author

Lotus log - select messages

2022-07-31T15:58:19.414Z WARN fullnode full/mpool.go:171 Push from ID address (f03197), adjusting to f3q3crofnq3sxmbyqwxsx7qcmjuykr4uk7jgcrxlrdrk7ovjkmlbhgqhkcomeeqcum5l6h4pugxigrbolz5fkq
2022-07-31T15:58:19.419Z WARN rpc [email protected]/handler.go:279 error in RPC call to 'Filecoin.MpoolPushMessage': mpool push: failed to push message:

2022-07-31T15:58:30.892Z WARN fullnode full/mpool.go:171 Push from ID address (f03197), adjusting to f3q3crofnq3sxmbyqwxsx7qcmjuykr4uk7jgcrxlrdrk7ovjkmlbhgqhkcomeeqcum5l6h4pugxigrbolz5fkq
2022-07-31T15:58:30.893Z WARN rpc [email protected]/handler.go:279 error in RPC call to 'Filecoin.MpoolPushMessage': mpool push: failed to push message:

@stuberman
Copy link
Author

stuberman commented Jul 31, 2022

Cost so far -102.7519 FIL x '$10.30 = < $ 1,058.34 >

Time Message ID From To Net Income Type
2022-07-31 12:24:00 N/A f01278 f099 -34.2017 FIL Burn
2022-07-30 12:24:00 N/A f01278 f099 -34.2931 FIL Burn
2022-07-29 12:24:00 N/A f01278 f099 -34.2571 FIL Burn

@zenzenless
Copy link

A temporary remedy: maybe you can reduce the number of sectors in DeclareFaultsRecovered and split them into two messages to send.

@stuberman
Copy link
Author

James, how would I do that?

@jennijuju
Copy link
Member

@stuberman we are looking into this rn and hopefully give you some tools, soon

@magik6k
Copy link
Contributor

magik6k commented Aug 2, 2022

With #9109 it looks like your sector number allocations are pretty good, which invalidates one of the theories as to why gas use is so high here

$ ./lsm --actor f01278 proving deadline 0
Deadline Index:           0
Partitions:               1
Proven Partitions:        0
Current:                  false

Partition Index:          0
	All Sectors:               1449 (bitfield - runs 158+158=316 - 3425 0s 1449 1s - 0 inv - 185 B 185B)
	Live Sectors:              1443 (bitfield - runs 159+159=318 - 3431 0s 1443 1s - 0 inv - 184 B 184B)
	Active Sectors:            1443 (bitfield - runs 159+159=318 - 3431 0s 1443 1s - 0 inv - 184 B 184B)
	Faulty Sectors:            0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)
	Recovering Sectors:        0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)

$ ./lsm --actor f01278 proving deadline 1
Deadline Index:           1
Partitions:               1
Proven Partitions:        0
Current:                  false

Partition Index:          0
	All Sectors:               53 (bitfield - runs 45+45=90 - 4544 0s 53 1s - 0 inv - 68 B 68B)
	Live Sectors:              52 (bitfield - runs 45+45=90 - 4545 0s 52 1s - 0 inv - 67 B 67B)
	Active Sectors:            52 (bitfield - runs 45+45=90 - 4545 0s 52 1s - 0 inv - 67 B 67B)
	Faulty Sectors:            0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)
	Recovering Sectors:        0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)

$ ./lsm --actor f01278 proving deadline 2
Deadline Index:           2
Partitions:               1
Proven Partitions:        0
Current:                  false

Partition Index:          0
	All Sectors:               2225 (bitfield - runs 176+176=352 - 2664 0s 2225 1s - 0 inv - 220 B 220B)
	Live Sectors:              2130 (bitfield - runs 204+204=408 - 2759 0s 2130 1s - 0 inv - 239 B 239B)
	Active Sectors:            0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)
	Faulty Sectors:            2128 (bitfield - runs 204+204=408 - 2759 0s 2128 1s - 0 inv - 239 B 239B)
	Recovering Sectors:        0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)

$ ./lsm --actor f01278 proving deadline 3
Deadline Index:           3
Partitions:               1
Proven Partitions:        0
Current:                  false

Partition Index:          0
	All Sectors:               37 (bitfield - runs 36+36=72 - 4603 0s 37 1s - 0 inv - 54 B 54B)
	Live Sectors:              36 (bitfield - runs 35+35=70 - 4604 0s 36 1s - 0 inv - 53 B 53B)
	Active Sectors:            36 (bitfield - runs 35+35=70 - 4604 0s 36 1s - 0 inv - 53 B 53B)
	Faulty Sectors:            0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)
	Recovering Sectors:        0 (bitfield - runs 0+0=0 - 0 0s 0 1s - 0 inv - 1 B 1B)

The second partition has ~400 runs in it's bitfields, which isn't much different from other miners, so there may be something else going on here

@rjan90 rjan90 added this to the LM-Tech-Debt milestone Mar 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/proving Area: Proving kind/bug Kind: Bug LM-tech-debt need/analysis Hint: Needs Analysis P1 P1: Must be resolved
Projects
Status: No status
Development

No branches or pull requests

7 participants