Link retrievals to miners

[bajtos]

Currently, SPARK checks the retrieval of CIDs stored in Filecoin but does not know which deal it deal it is testing and which miner is expected to serve the retrieval.

We need to link the retrieval tasks to deals and miners.

There are different ways how to accomplish this.

1. Convert DealProposal data into ContextID string, using the same algorithm as Boost/Frisbii. Then we can filter IPNI response to find the advertisement matching the ContextID of the deal we are testing.
2. Alternatively, we may be able to map provider addresses to miners, but this is less robust.

After a bit of exploration in CheckerNetwork/fil-deal-context-id#1, I decided to use a different approach:

• Let Spark checkers query the on-chain miner info (method Filecoin.StateMinerInfo) to find miner's libp2p identity. This identity should be the same as the identity provided in the IPNI response.

Tasks

• Modify fil-deal-ingester to include MinerID in the data stored in Spark DB.
• Modify spark-api to return MinerID in per-round tasks.
• Modify spark checker to (1) Resolve MinerID to ProviderID (2) filter IPNI response using ProviderID (3) report MinerID and ProviderID in the measurement fields
• Modify spark-api and spark-publish to record & publish MinerID fields
• Modify spark-evaluate to include MinerID in the task/deal id (see feat: remove provider info from task definition spark-evaluate#143)
• Update spark-evaluate to calculate per-miner stats for each round
• Expose these stats via spark-stats and visualise them in our internal & public dashboards
• Write a blog post explaining how we solve the problem of determining how to retrieve content stored in a given Filecoin deal

Cleanup/refactoring opportunities

• Rename getTaskId to getDealId, see feat: new stat rate_of_deals_advertising_http  spark-evaluate#151 (comment)

Out of scope

• Measure and visualise requests per storage provider → Measure and visualise requests per storage provider #64
• Modify fraud detection (committee building) in spark-evaluate to cross-check ProviderID → Find majorities in Spark measurements #59
• Update Spark protocol docs: CID Sampling and Retrieval With Attestation

[bajtos]

Cleanup/refactoring opportunities

Rename getTaskId to getDealId, see feat: new stat rate_of_deals_advertising_http spark-evaluate#151 (comment)

I feel the name "task id" better describes the value, so I am going to keep it as it is now.

[bajtos]

Write a blog post explaining how we solve the problem of determining how to retrieve content stored in a given Filecoin deal

This will be a series of posts.

1. How SPARK samples Filecoin deals: draft
2. Blog post: How Spark Maps FIL+ LDN Deals to Content Identifiers #100
3. Blog post: How SPARK discovers retrieval providers #101
4. Blog post: Downloading content stored on Filecoin #102