Off-Chain Window PoSt

[Stebalien]

See #42. WIP.

[TippyFlitsUK]

This would be extremely welcome.

[cwhiggins]

I would say that I am not for this proposal. I prefer what @jbenet proposed with a control plane and data plane,#24 (comment), I feel that is more in line with the type of community we should promote. It was compared to making sure emergency service vehicles are able to get where they are going, people move out of the way not because of what they can gain.
In this FIP, the POST mechanism is reduced to this possiblity, ### Disputing Proofs Is Rational
Currently, there are two reasons to dispute an invalid proof:

1. To maximize profits as a miner by minimizing the power of other miners.

and I don't think that this possiblity is appropriate for our communities goals.

[Stebalien]

@cwhiggins this proposal is complementary to the "fast lane" proposals, it doesn't conflict with them. I completely agree that "necessary" messages should be prioritized.

I'm fielding this proposal because:

1. Unlike the fast-lane proposals, this proposal actually increases chain throughput. Even with a fast-lane for window posts, the network will eventually grow to a size where window posts take all chain bandwidth. That would break the network. To be clear, this won't happen for a long time.
2. All the fast lane proposals I've seen (there are several) are significantly more difficult to implement than this proposal. This proposal requires some changes to the actors logic and state. Those proposals will require complex incentive analysis, changes to message selection, and likely changes to the block header format itself.

That's why I'm proposing this solution now.

To maximize profits as a miner by minimizing the power of other miners.

and I don't think that this possiblity is appropriate for our communities goals.

The goal is to ensure that miners are correctly storing data, that's it. I'm providing both an altruistic and a selfish reason:

1. Altruistically, some parties will dispute incorrect window posts to ensure that data storage on Filecoin is reliable. By reliably disputing these proofs, miners are highly incentivized to correctly store and prove their storage or risk penalties.
2. Selfishly, some parties will dispute incorrect window posts to maximize their own relative power. Power in a blockchain is a (mostly) zero sum game. The less power you have, the more power I have.

[NSC-FIL]

I'm for this enhancement if the it doesn't slow down the sealing process.

[cwhiggins]

Will this have to happen two days in a row before a miner is penalized? How often in a day can it be requested a miner prove their storage? Will it scale with size of miner, say only one request per day for 10Tib miner, 1000 for 10Pib?

[cwhiggins]

Also, the end points of your response 1,2 you use the term "incorrect window posts", the requestor does not know the state of the Post. It is a fishing expedition. The more resources you have, the more you can go fish. Sorry, I don't like this implementation.

[Stebalien]

Will this have to happen two days in a row before a miner is penalized?

No. A miner will still be able to "miss" a single window post without paying penalties, but they won't be able to submit a bad proof.

How often in a day can it be requested a miner prove their storage? Will it scale with size of miner, say only one request per day for 10Tib miner, 1000 for 10Pib?

Miners will still need to prove their storage as normal using the normal proof mechanisms. The only change here is that the network won't validate these proofs unless some other miner submits a new message disputing the proof.

[Stebalien]

Also, the end points of your response 1,2 you use the term "incorrect window posts", the requestor does not know the state of the Post. It is a fishing expedition. The more resources you have, the more you can go fish. Sorry, I don't like this implementation.

I think you're misunderstanding the proposal. This proposal requires no additional work from miners proving their storage.

[cwhiggins]

Could please clarify what will happen when some one submits a dispute to a proof?Will the miner at that point just be required to submit a saved message from the Post?

[Stebalien]

Could please clarify what will happen when some one submits a dispute to a proof?Will the miner at that point just be required to submit a saved message from the Post?

This process is explained in detail in the "Change: SubmitWindowedPoSt" and "New: DisputeWindowedPoSt" sections.

Simple summary (but you should still actually read the proposal):

1. When a proof is "optimistically" accepted (but not verified immediately) it is recorded in the state tree.
2. Later, another miner can "dispute" this proof by submitting the index in the state tree where the proof is recorded on chain.

[cwhiggins]

so only other miners can dispute proofs, as they are then doing the work to verify against chain? It seems to me there still needs to be some sort of engagement with the miner in question at the time of the dispute? I have read the proposal, obvious by my first post, really, had to go there?

[Stebalien]

It seems to me there still needs to be some sort of engagement with the miner in question at the time of the dispute?

No, the proof is recorded in the on-chain state when the miner first submits it. The disputer just tells the chain to check the proof.

Steps:

1. The miner submits a proof. <-- only miner interaction
2. New: The chain stores it temporarily in the miner's state.
3. New: Someone disputes it (calls DisputeWindowedPoSt).
4. The network verifies the proof.

The only difference is the addition of steps 2 and 3 to make step 4 (the expensive part) optional unless someone notices that the proof is incorrect.

[funktafide]

I think it looks like a god idea, but the possibility of large miners spamming bad proofs is somewhat concerning. Obviously you don't want to penalize someone for 1 bad proof, but I think once the number reaches a specific point, the penalty should be stiff.

[BrunoZell]

Does the network power only decrease when the dispute lands on chain? What happens in the intermediate time the miner had unlawful power to his name? Is it just the slashing penalty or are possible block rewards he got from that power deducted as well? (It seems difficult to get right though as it's usually just a fraction of his power that got lost). Also, will the process of continuously validating proofs and filing disputes be automated when running a node?

In general I am all for doing as much work off chain as possible. However, I feel it is very weird that miners need to pay each other in order to get paid. Especially considering that WindowPoST is very much expected and also required. As soon as a sealed sector is known the system should make room for the expected PoSTs, but not for more. Maybe a variable block size or a fixed fee set at sector commit?

[Stebalien]

@funktafide

Obviously you don't want to penalize someone for 1 bad proof...

That's actually not obvious to me and I'm curious what you're thinking here. At the moment, there are pretty stiff penalties (~5.1 days worth of block rewards) for submitting any bad proofs (note, these aren't just missed proofs, they're actually incorrect). The idea is to check these proofs thoroughly locally before submitting them.

@BrunoZell

Great questions! I'll try to work my answers into the proposal.

Does the network power only decrease when the dispute lands on chain?

Unfortunately, yes. Actually, a bit longer because Filecoin determines the winning miner by looking Finality (900) epochs back in the power table.

What happens in the intermediate time the miner had unlawful power to his name?

They will continue to win blocks as normal.

Is it just the slashing penalty or are possible block rewards he got from that power deducted as well?

The penalty scales with the amount of incorrectly proven power which is also proportional to the expected number of blocks won.

Specifically, this proposal currently sets the penalty to 5.1 "BR". That's the expected block reward earned by the incorrectly proven sectors over 5.1 days.

So in theory, a miner could "get lucky" and win more than it loses. But a miner is significantly more likely to lose more than it wins.

Also, will the process of continuously validating proofs and filing disputes be automated when running a node?

The network is secure as long as someone checks the proofs. I expect the Filecoin Foundation to sponsor one, at least, given that the cost of running one such node is reasonably low.

For now, any "disputer" process will likely be a special node but I'd personally love to see a world where all lotus nodes randomly sample and check proofs at some user configurable frequency.

In general I am all for doing as much work off chain as possible. However, I feel it is very weird that miners need to pay each other in order to get paid. As soon as a sealed sector is known the system should make room for the expected PoSTs, but not for more. Maybe a variable block size or a fixed fee set at sector commit?

I think I'm missing some context here but I'll answer as best I can. Specifically, could you expand on "miners need to pay each other in order to get paid" and "maybe a variable block size or a fixed fee set at sector commit"?

There are a number of reasons to verify posts off-chain, but I they all boil down to: it frees up chain bandwidth. Basically, we don't need to synchronously and deterministically verify all proofs in lockstep, so why do it? We can let nodes verify proofs in whatever manner works for them, and report issues to the chain, This enables:

• Buffering. It's ok to fall behind a bit.
• Parallelism. It's possible to perform some operations in parallel on-chain, but it's tricky to get right and optimize.
• Caching. Caching in a deterministic environment is tricky and error prone so Lotus does some caching but not nearly as much as it could. However, an off-chain proof verification system can cache the inputs to the proof verification logic and avoid re-loading them from the state tree.

[funktafide]

@Stebalien I think I confused a couple different sections when reading. I was looking at the Risks section.

A large miner could spam bad proofs to try to overwhelm the chain with the subsequent verification cost.
We'd have to pick penalties/limits to avoid this.
Each partition can only be used once in this way before a valid proof is needed to restore its power.
A the moment, we have chain bandwidth to handle challenging every proof, and if we greatly reduce congestion,
headroom here will increase by >= 10x
Future proof aggregation techniques may support submitting a challenge for multiple proofs with sub-linear verification
cost.

I was referring to a large miner spamming bad proofs from that section. So if someone disputes a proof 1 time and its actually a valid proof, does the miner that disputes it get penalized? I was thinking the first time you wouldn't, but if you are spamming to overwhelm the chain, the penalty would get increasingly higher. It looks like you are already looking at this, so my post was a bit redundant. This was the part that kind of stuck out to me as a bit concerning since there are some pretty large miners out there.

[jennijuju]

Can one proof being disputed multiple times within 1800 epochs by the same challenger or different challengers?

[BrunoZell]

@funktafide The poential of spam is not limited to this - all transactions can be used to spam the network. The attacker would be paying the fees according to the computational load that is required to process the transaction in form of gas usage and simultaneously would drive up the base fee as intended by EIP1559.

@Stebalien

miners need to pay each other in order to get paid

...in a senes that they need to pay transaction fees for their sector commits and window PoSTs in order to mine blocks, and when they get elected to mine a block they are rewarded with the block reward plus the fees payed by all other miners. Specifically I dislike that gaining power in the network is coupled to the transaction fee market. The tx fee market is mainly dictated by the capacity and energy consumption of the blockchain.

IMO it is essential to open blockchain systems that everyone can gain power in the network regardless of congestion and fee market. In fact, I belive congestion should be an additional incentive for brand new miners to get these sweet transaction fees. Having new miners first pay those high fees to get power results in the opposite effect or even a death spral where existing miners will stop adding new storage due to falling profits.

Maybe a variable block size or a fixed fee set at sector commit?

So how could we solve that? Two ideas that come to my mind.

1. Whenever a new sector is sealed and committed, the network knows for how long the sector is expected to stay alive. This publicly verifiable information can be used to make room in the chain for all expected window PoSTs, possibly by having miners always include those transactions regardless of block size. This would decouple those messages from the gas market and miners would essentially give each other network capacity. I see two problems with this approach: What's the incentive for miners to follow this protocol instead of leaving out messages and claim they didn't knew about them? Especially considering they would gain power in not including such PoSTs. And could such a mechanic block future developments of making file modifications possible or adding other new features that do not make the amount of window PoSTs predicateble?

2. Somehow fix the fee for all window PoST at commit time. After thinking more about this it really doesn't make any sense so I won't elaborate any furhter here 😄

[irenegia]

I am not fully convinced that "disputing proofs is rational": the arguments used here to support that statement are true but what about miners try do be "free riders"?
That is, a miner can choose to let the others do the verification and disputing work (zero work for him, same "rewards" as before).
cc @sa8

[Stebalien]

In terms of "minimizing power", I agree that the argument is very weak and I'm happy to remove it. From my perspective, the only argument that really works is the "altruistic" (ish) one. That is, large enough players will verify proofs because:

1. It should be relatively cheap. At the moment, everyone is verifying all proofs. Even if we grow by 10x or 100x, verifying proofs should still be relatively inexpensive compared to the cost of actually mining on Filecoin.
2. If a significant fraction of proofs on Filecoin are incorrect and go undisputed, people will lose trust in the Filecoin network devaluing the token.

[Stebalien]

Updated. How does the new text read?

[irenegia]

The security considerations section misses this points:

• Short fork attack may be possible and some invalid proof may not be disputed.
  If the challenge for an invalid proof arrives close (ie, < 200 epochs) to the end of the challenge window, the miner (the one who submitted the invalid proof) could fork using selfish mining and create a chain where there is no challenges for his invalid proof.
• About "Power & sector extension attacks": Note that after this FIP the power for invalid proofs is removed only when the invalid proof is detected and therefor, from the deadline to this moment, there is some small PT inflation we did not have before.

[Stebalien]

done.

[irenegia]

5.1?
I think it is 5.51 (FF + 2*BR), and for completeness I would add the this value is chosen to give a negative profit for miner who sends invalid proofs when we have that an invalid proof is detected in 12h (after deadline) with probability ≥ 1/3.

[Stebalien]

Done.

[f8-ptrk]

The miner submits a proof. <-- only miner interaction
New: The chain stores it temporarily in the miner's state.
New: Someone disputes it (calls DisputeWindowedPoSt).
The network verifies the proof.

who is "paying" for the last step?

case 1: the Dispute is not justified --> the proof is valid
case 2: the Dispute is justified --> the proof is invalid

in addition: what keeps miners from collectively ignoring DisputeWindowedPoSt messages?

[anorth]

So if someone disputes a proof 1 time and its actually a valid proof, does the miner that disputes it get penalized

@funktafide the party that disputes it pays the gas cost of checking the proof validity, but earns no reward that might cover that cost if the dispute fails. This is not any more prone to spam than any other method. @Stebalien was worried about the case of a miner submitting many bad proofs, thus kind of obligating the network to process valid disputes for them all. It's a very expensive attack. (I expect future proof batch/aggregation techniques to dramatically reduce the cost of disputing en-masse.)

Can one proof being disputed multiple times within 1800 epochs by the same challenger or different challengers

@jennijuju an invalid proof can be disputed only once. A valid proof can be disputed repeatedly (at cost to the disputer, and will always fail to penalize/reward).

everyone can gain power in the network regardless of congestion and fee market

@BrunoZell thanks this is a great perspective. I think there's a lot of support for that vision long term, we'd love your help figuring it out. The control/data plane model might be a good fit, or others. But let's take that discussion elsewhere (#24, #47, or a new issue) – it's complementary to this technique of moving most proof verification off-chain, which we will want anyway.

[BrunoZell]

I was suspecting this would be out of scope for this issue, still finding my way around FIL development. Thanks for the pointers!

[anorth]

A suggestion you are free to ignore: I find that formatting markdown files with a line break after every sentence makes targeting review comments much easier. It's also more sympathetic to view in a non-wrapping editor (like many people's code editors).

[anorth]

We need to describe the reward above, and then outline its potential for extraction here.

[Stebalien]

Mostly done. We still need to pin down the exact reward.

[anorth]

Address benefits specifically to small miners

[anorth]

This change increases the length of chain history that must be accessible at low cost by the VM in order to fetch a challenge seed from dispute period + challenge lookback in the past. This period is less that the period required for pre-commit randomness, so likely no implementation change, but FYI @magik6k @arajasek to ensure that both are taken as bounds.

[Stebalien]

A suggestion you are free to ignore: I find that formatting markdown files with a line break after every sentence makes targeting review comments much easier. It's also more sympathetic to view in a non-wrapping editor (like many people's code editors).

Yeah, I'll switch it. I've stopped inserting line breaks because I've optimized my editor work around the lack there of... but I agree.

[anorth]

I think this is untrue if there is a non-zero reward. We should note here the maximum fund extraction rate for a self-destructing miner (and that it is lower than that already possible from declaring consensus faults). cc @Kubuxu for details.

[Stebalien]

Yep. Updating now.

[Stebalien]

The rate is a bit tricky to calculate. The rate for slashing is one reward per day but the rate here is one reward per partition per 2 days. On top of that, the fine to reward ratio depends on how "full" each partition is.

I'd like to consider setting a minimum fine. That would make it easy to reason about this in terms of a tax. E.g., if the minimum fine is 3x the reward, the "advance" would have a 75% tax.