API improvements, Direct data querying. Update on new task On-Demand Data Streaming.

Release 1.7.8: API improvements, Direct data querying. Update on new task On-Demand Data Streaming.

Hello @dataverse・13 ! We are excited to announce the latest update that adds on-demand data streaming capabilities to SN13! This feature allows validators to request small chunks of data (<1k rows) from miners through the API. As well as direct possibility to query data from the network to the API users.

Key Features

On-Demand Data Streaming

• Validators can request data directly from top miners
• Supports both X (Twitter) and Reddit data sources
• Built-in validation and reward system

API Bucket quering

• Validators can request any timebucket from the network via API.

*New Scoring Info for Miners*

• Pre-implemented on-demand handling in miner template
• Freedom to customize scraping implementations
• Automatic reward system:
• Top 50% miners by trust participate in validation
• 50% chance of validation per request
• Successful validation: +0.1% credibility boost
• Failed validation: 1% credibility decrease
  here we will include a link to scoring.md that in dev

Other Updates:

HF Updates:

• Added miner hotkey and UID in Hugging Face scoring logs.

Dynamic Desirability Updates:

• Added endpoint to validator API to get the latest unscaled JSON desirability submission for a given hotkey.
  • If no hotkey is specified, returns the current total dynamic list used by the validator.
• Used subtensor.commit to update retrieval and upload functions to be compatible with dTao update.
• Small config change to MinerEvaluator to be compatible with dTao update.
• Increased total label scale factor pool for the dynamic list to 100.
  • (All dynamically incentivized labels across all validators will have their scale factors add up to 100).
• Added JSON formatting when displaying current dynamic desirability list for readability improvements in wandb logs.

Happy Scraping!