feat: serve CAR blocks #68
Conversation
alanshaw
changed the title
| } | ||
| } | ||
|
|
||
| const obj = await env.CARPARK.get(`${dataCid}/${dataCid}.car`, { range }) |
There was a problem hiding this comment.
Could be cool to try this with roundabout. I think it should work and let us redirect to presigned url.
Probably good to keep as is and track issue to attempt cost optimisation :)
There was a problem hiding this comment.
Hmm, yes but we probably don't want to give out those URLs to anyone via the gateway. Someone will try to use/abuse and they also expire...
There was a problem hiding this comment.
So the usage would be the same as here? it would be a redirect that response will be read right away right? So, expiration would not be a problem, and if we perform request to roundabout within the worker and redirect to presigned URL then user will not even know how URL was created.
Use/abuse would be the same as via freeway no. So, rate limits is really the solution we seem to have to avoid abuse
🤖 I have created a release *beep* *boop* --- ## [2.10.0](v2.9.2...v2.10.0) (2023-07-31) ### Features * serve CAR blocks ([#68](#68)) ([0b95438](0b95438)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This PR adds a new handler that allows freeway to serve CAR files directly from R2 (CARPARK).
Requesting a CID from the gateway with the CAR codec
0x0202will invoke this new handler.Obviously CARs are not served in the same way as regular IPFS data, since we can serve the data directly from an R2 bucket. To deal with this difference, there's a new middlewares that detects the CAR codec and calls the correct handler, passing through to the regular handlers if the CAR codec is not detected.
The handler supports
HEADrequests, allowing you to get the file size of the item you're requesting. It also supports the HTTPRangeheader, allowing you to extract byte ranges from within a CAR, which you might do if you fetch the index for the CAR.The idea is that we can use these handlers to build a system that runs on content-claims. The flow might be something like:
partin the partition claimincludesCID specified in the inclusion claims (You must first fetch partition claims for each index CID to find out which CAR file they can be found in).Rangerequests to export the whole DAG or a sub-DAG directly from theparts, using the index data as a guide for which blocks to extract.Note: if exporting a full DAG you might want to just stop after (1) and import all data in all CARs into IPFS.