Building block specific metadata for each finalized block within the consensus.

[imyourm8]

Short intro for what I am trying to accomplish:

Using Grandpa + Babe

Each block has a modified header (ideally) to include metadata-specific data for light clients to be able to query Full nodes for data to check if they indeed do have it.

A high-level algorithm looks like this:

1. Get finalized block data somehow.
2. Build metadata based on block's body bytes (Kate Commitment)
3. Save that metadata on-chain + header data for light clients (probably it's not possible to modify header after block is finalized)
4. Other non-proposing nodes must be able to quickly verify that metadata for validity (this process is must faster).

There are a few problems with the whole process:

• Metadata processing is long ~12 seconds on average.
• The result must be validated within the consensus or afterward and trigger slashing in case the validity check is failed.

Myself, I checked a few options based on documentation:

Block import pipeline

That was the most obvious place to go, do it the last step right before the Client, but it seems like I can't access on-chain storage within it. Let's assume I can, will 12 seconds process break anything?

Offchain worker

That was my second obvious choice due to the fact, that it is automatically called after the block is finalized. But the function itself only accepts block number, and I can't get quick access to the block data there. Using RPC localhost call sounds a bit hacky and I would prefer direct access to on-chain data, if possible.

The other problem with offchain worker would be implementing custom slashing because the network must ensure, that metadata is built for each block, with no exceptions.

Is it possible to do all these using Substrate?

[danforbes]

@imyourm8 - I conferred with @shawntabrizi on this topic and he pointed out this PR, which removed a Finality Tracker pallet from our core frame directory of pallets. You may want to consider cribbing this pallet and adding it directly to your runtime, or at least it may help guide your solution to the problem you've described. Please let me know if this is helpful paritytech/substrate#7228

[danforbes]

Thank you @tomusdrw for pointing out this existing Issue, which is related to this topic, I believe paritytech/substrate#6742

[danforbes]

@imyourm8 - can you please provide an update/response of whether or not the above resources are helpful?

[imyourm8]

@danforbes Thank you for providing some information. Unfortunately, paritytech/substrate#7228 wasn't very helpful, I still don't know how I can possibly access block data after the finalization stage.

Let's say we drop off-chain worker for a moment. Is there any way I can access the best block data after it has been finalized?

[imyourm8]

Also, how can I modify block building logic? I want to take N block body bytes and add another N bytes as an extension, effectively doubling the block size.

[imyourm8]

For now - we have decided to fork Substrate and modify its internals to meet our requirements.

[tomusdrw]

@imyourm8 I think we need a bit more details on what exactly your use case is.

From what I understand you want to feed back on-chain processed finality information and you want this on-chain information to be verified by consensus as well.

If the verification takes 12 seconds, then it naturally limits your block times. Such long block import times feel like an obvious DoS vector as well, cause I imagine this will vary heavily depending on the hardware.
IMHO what you are trying to do is not really well suited for block import. You should probably take a look at some kind of off-chain verification strategies, which have proper incentive schemes that would give you strong economic guarantees of the on-chain data being correct.

Now for the technical details:

1. If you decide that the whole process should happen on-chain (in the runtime), you may feed finality information to the runtime via mechanism called InherentData. Together with BlockProposer customisation, block author can inject any kind of external data it wants, see here: https://github.com/paritytech/polkadot/blob/086a0aeb90c43d62e7d4fe4249f06b6497a480bf/validation/src/block_production.rs#L135
2. Please take a look at DigestItem mechanism, for adding extra data to Header from within the runtime code.
3. If you are not planning to integrate with Polkadot, you can take a look at HostFunction extensions, to create new runtime-available API to fetch finality information without relying on InherentData injection.

I don't know if I understood your case correctly, but my best guess towards how this should be implemented is this:

1. You start a new background service, which listens to finality notifications from GRANDPA and every time there is new data, starts constructing the metadata. (service is active only if role.is_authority() == true)
2. You customize block production logic (see link above) and use Inherent to inject metadata information to make it available for the runtime.
3. If you choose to verify on-chain (i.e. no off-chain verification game), you need to add HostFunctions to retrieve any additional information that is requried to verify the metadata injected by the authority (unless the verification is completely stateless)
4. If the verification is successful you alter some on-chain data and add an extra DigestItem with light-client related data.

You can also use Off-chain Workers instead of InherentData to inject metadata for verification, but InherentData can be more strictly verified (i.e. blocks without it may be considered incorrect).

Let me know if that helps.

[nuke-web3]

Hey all - is this still an open issue? If so, let's reopen it.
Thanks for the details, this well be good for the community to reference latter in any case 🙇🏼