Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest historical block hashes #3518

Open
leoyvens opened this issue May 2, 2022 · 5 comments
Open

Ingest historical block hashes #3518

leoyvens opened this issue May 2, 2022 · 5 comments

Comments

@leoyvens
Copy link
Collaborator

leoyvens commented May 2, 2022

An issue affecting the network, which sends queries by block hash, is that graph node will fail if it cannot find the block hash in its block cache. The block ingestor will ingest blocks within the reorg threshold, but holes can happen and nothing is done to fill them.

Since historical queries mean that all block hashes are potentially necessary, a way to solve this is to run a backfill ingestor job that ingests all historical hashes.

In terms of implementation, we'd want to be able to have only the block hash and number rather than the whole header, which is not valid in the current block cache structure. Also we'd probably want to leverage the firehose and not need a json-rpc implementation of the backfill.

@lutter
Copy link
Collaborator

lutter commented May 2, 2022

This would also address #1467

@azf20
Copy link
Contributor

azf20 commented May 3, 2022

I think this is supported for Firehose Blockstreams @maoueh ?

@leoyvens
Copy link
Collaborator Author

leoyvens commented Aug 4, 2022

After #3738 we'll also need to ingest the timestamps to continue supporting the _meta field.

@azf20
Copy link
Contributor

azf20 commented Aug 7, 2023

We could alternatively lazily fetch blocks when there is a query (similar to #4768 )

@fordN fordN added the Stale label Apr 9, 2024
@madumas
Copy link
Collaborator

madumas commented May 29, 2024

A note only to point out that this is a pain point on the network. Indexers may be forced to purge the block cache, and there is no easy way to "refill" it. The lazy fetch solution would be great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants