From 75b88f9c8675a9dcc8c1a8702eeb76932603c78a Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 11:51:35 +0300 Subject: [PATCH 01/25] Create architecture-options.md --- .../docs/concepts/architecture-options.md | 206 ++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 src/pages/docs/concepts/architecture-options.md diff --git a/src/pages/docs/concepts/architecture-options.md b/src/pages/docs/concepts/architecture-options.md new file mode 100644 index 0000000..01a769e --- /dev/null +++ b/src/pages/docs/concepts/architecture-options.md @@ -0,0 +1,206 @@ +## Architecture options + +## Possible architectures of using web3.storage to upload + +UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate: + +- Client-server: You (the developer) own the Space, and your user uploads to your backend infra before you upload it to the service +- Delegated: You own the Space, but you give a delegated UCAN token to your user's Agent to upload directly to the service (rather than needing to touch the upload in your backend) +- (Most complex) User-owned: Your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it + +In the How-tos section of the docs, we focused on the first two options, as they are the most familiar today. However, you can implement each of these in a number of ways, but we talk through some considerations when implementing a given option. + +#### [Client-server](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#client-server) + +``` +mermaid + +sequenceDiagram + +participant User + +w3up-client in backend-\>\>w3up-client in backend: Client set with Agent with delegation from Space + +User-\>\>w3up-client in backend: Upload data + +w3up-client in backend-\>\>web3.storage w3up service: Upload data + +``` + +- For your backend to be scalable, you might consider using serverless workers or a queue in front of a server +- In either case, you'll need a registered Space, and your client instance in your backend to have an Agent with a delegation from this Space + - (Recommended) It's likely easiest to create and register your Space using [w3cli](https://github.com/web3-storage/w3cli) rather than using w3up-client to do so (especially if your backend isn't persistent); you can then generate your own Agent and delegate the ability to upload to your Space using something like [this example](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation) + - If your backend is persistent, you can do this or do everything in the client directly ([create Space](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#creating-and-registering-spaces) and [get delegation](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#delegating-from-space-to-agent)) +- After this, once your user uploads data to your backend, you can run any of the upload methods + +#### [Delegated](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#delegated) + +```mermaid + +sequenceDiagram + +participant w3up-client in user + +participant w3up-client in backend + +participant web3.storage w3up service + +w3up-client in backend-\>\>w3up-client in backend: Client created with Agent and delegation from Space + +w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent + +w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID + +w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID + +w3up-client in user-\>\>web3.storage w3up service: Upload data + +``` + +- You will likely have w3up-client running in your end-user's client code, as well as backend code that's able to generate UCANs that delegate the ability to upload and pass them to your users (e.g., w3up-client running in a serverless worker) +- For your backend to be scalable, you might consider using serverless workers or a queue in front of a server +- As the developer, you'll need a registered Space, and your client instance in your backend to have an Agent with a delegation from this Space + - (Recommended) It's likely easiest to create and register your Space using [w3cli](https://github.com/web3-storage/w3cli) rather than using w3up-client to do so (especially if your backend isn't persistent); you can then generate your own Agent and delegate the ability to upload to your Space using something like [this example](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation) + - If your backend is persistent, you can do this or do everything in the client directly ([create Space](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#creating-and-registering-spaces) and [get delegation](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#delegating-from-space-to-agent)) +- Your user does not need a registered Space - just an Agent with a delegation from your Space + - w3up-client in the end user environment should have a unique Agent for each user, which should happen by default (since when w3up-client is instantiated it creates a new Agent anyway, or uses the one in local Store) + - From there, when your end user is ready to upload, they should request from your backend a delegation from your developer-owned Space to their Agent (which can be derived via [client.agent()](https://github.com/web3-storage/w3up/blob/main/packages/w3up-client/docs-Client#agent)) + - In your backend, you can call [client.createDelegation()](https://github.com/web3-storage/w3up/blob/main/packages/w3up-client/docs-Client#createDelegation) passing in the Agent object from client.agent() in your end user's instance, and passing through options? params to limit the scope of the delegation (e.g., store/add, upload/add, expiration time) + - You can serialize this using delegation.archive() and send it to your user + - The end user instance of the client should not need to call client.authorize(email), as it is not claiming any delegations via email address (but rather getting the delegation directly from your backend) +- Once your user receives the delegation, they can deserialize it using [ucanto.Delegation.extract()](https://github.com/web3-storage/ucanto/blob/c8999a59852b61549d163532a83bac62290b629d/packages/core/src/delegation.js#L399) and pass it in using client.addSpace(), and from there they can run any of the upload methods + - Note that this alone does not give visibility into which of your end users are uploading what; to track this, you'll probably need them to send you that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking) +- A code example that does this can be found below + +``` +import { CarReader } from '@ipld/car'; + +import \* as DID from '@ipld/dag-ucan/did'; + +import \* as Delegation from '@ucanto/core/delegation'; + +import { importDAG } from '@ucanto/core/delegation'; + +import \* as Signer from '@ucanto/principal/ed25519'; + +import \* as Client from '@web3-storage/w3up-client'; + +async function backend(did: string) { + +// Load client with specific private key + +const principal = Signer.parse(process.env.KEY); + +const client = await Client.create({ principal }); + +// Add proof that this agent has been delegated capabilities on the space + +const proof = await parseProof(process.env.PROOF); + +const space = await client.addSpace(proof); + +await client.setCurrentSpace(space.did()); + +// Create a delegation for a specific DID + +const audience = DID.parse(did); + +const abilities = ['store/add', 'upload/add']; + +const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24; // 24 hours from now + +const delegation = await client.createDelegation(audience, abilities, { + +expiration, + +}); + +// Serialize the delegation and send it to the client + +const archive = await delegation.archive(); + +return archive.ok; + +} + +/\*\* @param {string} data Base64 encoded CAR file \*/ + +async function parseProof(data) { + +const blocks = []; + +const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); + +for await (const block of reader.blocks()) { + +blocks.push(block); + +} + +return importDAG(blocks); + +} + +async function frontend() { + +// Create a new client + +const client = await Client.create(); + +// Fetch the delegation from the backend + +const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; + +const response = await fetch(apiUrl); + +const data = await response.arrayBuffer(); + +// Deserialize the delegation + +const delegation = await Delegation.extract(new Uint8Array(data)); + +if (!delegation.ok) { + +throw new Error('Failed to extract delegation'); + +} + +// Add proof that this agent has been delegated capabilities on the space + +const space = await client.addSpace(delegation.ok); + +client.setCurrentSpace(space.did()); + +// READY to go! + +} +``` + +#### [User-owned](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#user-owned) + +``` +mermaid + +sequenceDiagram + +participant User + +participant Application backend + +participant web3.storage w3up service + +Application backend-\>\>User: Front end code that includes w3up-client + +User-\>\>web3.storage w3up service: (If needed) Create Space and register it + +User-\>\>web3.storage w3up service: (If needed) Use Agent email verification to "log in" to Space + +User-\>\>web3.storage w3up service: Upload data using w3up-client + +``` + +- If you want your user to own their own Space, you'll likely be relying on the w3up-client methods to create a Space, authorize the Space, and authorize the Agent on the end user-side; from there they can run any of the upload methods + - Doing this does take some of the UX out of your control; for instance, when web3.storage fully launches with w3up, your users will have to set up their payment methods with web3.storage + - Note that this alone does not give visibility into which of your end users are uploading what; to track this, you'll probably need them to send you that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking) +- There is a world of possibilities with your users "bringing their own identity" for their Space; you could explore how crypto wallet private keys, Apple Passkey, and more might map to Space DIDs and have the client use those +- If you have code snippet(s) that works for you, please share them in a PR or [Github issue](https://github.com/web3-storage/w3up/issues) and we'll link them here! From 7e97455f5e3d37479f1100b4485347b02a0af60c Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 11:52:33 +0300 Subject: [PATCH 02/25] Create car.md --- src/pages/docs/concepts/car.md | 600 +++++++++++++++++++++++++++++++++ 1 file changed, 600 insertions(+) create mode 100644 src/pages/docs/concepts/car.md diff --git a/src/pages/docs/concepts/car.md b/src/pages/docs/concepts/car.md new file mode 100644 index 0000000..03b0b01 --- /dev/null +++ b/src/pages/docs/concepts/car.md @@ -0,0 +1,600 @@ +## Content Archive (CAR) files + +Working with Content Archives (CAR) files + +When you upload files to web3.storage using the [client library](https://web3.storage/docs/reference/js-client-library/), your data is converted into a graph of data structures, which are then packed into a format called a Content Archive (CAR) before being sent to the web3.storage service. + +For most use cases, you never need to know about this process, as the conversion happens behind the scenes when using the client library. However, you might want to know more about how to manipulate CAR files directly, especially if you're using `upload` and `store` separately in web3.storage. + +## What is a Content Archive? + +The [Content Archive format](https://ipld.io/specs/transport/car/) is a way of packaging up [content addressed data](https://web3.storage/docs/concepts/content-addressing/) into archive files that can be easily stored and transferred. You can think of them like [TAR files](https://en.wikipedia.org/wiki/Tar_(computing)) that are designed for storing collections of content addressed data. + +The type of data stored in CARs is defined by [IPLD](https://ipld.io/), or InterPlanetary Linked Data. IPLD is a specification and set of implementations for structured data types that can link to each other using a hash-based Content Identifier (CID). Data linked in this way forms a Directed Acyclic Graph, or DAG, and you'll likely see a few references to DAGs in the documentation for IPLD and IPFS. + +IPFS files are one example of IPLD data, but IPLD can also be used to access data from Ethereum, Git, and other hash-addressed systems. You can also use IPLD as a general purpose format for your structured data, sort of like a Web3-flavored JSON. See Advanced IPLD formats below for more information. + +## [CARs and web3.storage](https://web3.storage/docs/how-tos/work-with-car-files/#cars-and-web3storage) + +When the web3.storage client and CLI pack up regular files into a CAR to store on IPFS, the CAR contains data encoded in the same format used by IPFS when importing files using the command line or other IPFS APIs. + +This format uses an IPLD "codec" called [dag-pb](https://ipld.io/docs/codecs/known/dag-pb/), which uses [Protocol Buffers](https://developers.google.com/protocol-buffers) to encode an object graph. Inside the graph are [UnixFS objects](https://docs.ipfs.io/concepts/file-systems/#unix-file-system-unixfs) that describe the files and their contents. We do this client-side for a few reasons. + +First, formatting everything on the client allows us to calculate the root Content Identifier for the data you're uploading before we send any data to the remote service. This means that you can compare the CID returned by the web3.storage service to the one you calculated locally, and you don't have to trust the service to do the right thing. + +Another reason to use CARs is to support large files, which would otherwise hit size limits on the web3.storage backend platform. The data in a CAR is already chunked into small blocks, which makes CARs easy to shard into small pieces that can be uploaded in batches. + +CAR files are a format that pretty much any IPFS tool or implementation can interact with. You can export data from your personal IPFS node into a CAR file and upload it to web3.storage using `w3 up --car` or `client.uploadCar`. As a result, we dive into the various ways you might interact with CAR files. + +## [Command line tools](https://web3.storage/docs/how-tos/work-with-car-files/#command-line-tools) + +There are a few ways to create and interact with CAR files from the command line. + +### [ipfs-car](https://web3.storage/docs/how-tos/work-with-car-files/#ipfs-car) + +The [ipfs-car](https://github.com/web3-storage/ipfs-car) JavaScript package includes a command-line tool for easily creating, unpacking, and verifying CAR files. + +To install it, you'll need [Node.js](https://nodejs.org/) - we recommend the latest stable version. + +You can install the command globally: + +```npm install -g ipfs-car``` + +Or run the command with npx without installing it to your PATH: + +```npx ipfs-car --help``` + +The --pack flag will create a new CAR file from a collection of input files: + +```ipfs-car --packpath/to/files --outputpath/to/write/a.car``` + +Or extract files from a CAR with --unpack: + +```ipfs-car --unpackpath/to/my.car --output/path/to/unpack/files/to``` + +You can also list the contents of a CAR with --list: + +```ipfs-car --listpath/to/my.car``` + +For more usage information, run ipfs-car --help. + +### Kubo + +Kubo is the reference implementation of the IPFS protocol. Among many other features, go-ipfs supports exporting any IPFS object graph into a CAR file and importing data from CAR files into your local IPFS repository. + +The [ipfs dag export](https://docs.ipfs.io/reference/cli/#ipfs-dag-export) command will fetch an IPFS object graph by its Content ID (CID), writing a stream of CAR data to standard output. + +To create a CAR file using go-ipfs, you can redirect the output of ipfs dag export to a file: + +``` +cid="bafybeigdmvh2wgmryq5ovlfu4bd3yiljokhzdep7abpe4c4lrf6rukkx4m" +ipfs dag export $cid > path/to/output.car +``` + +Note that you should replace the value of cid inside the quotes with the CID you want to export. + +If you don't have the CID in your local IPFS repository, the dag export command will try to fetch it over the IPFS network. + +To add the contents of a CAR file to your local IPFS repository, you can use ipfs dag import: + +```ipfs dag import path/to/input.car``` + +## Javascript libraries + +#### ipfs-car + +The ipfs-car package includes library functions for packing and unpacking files into CARs, using the IPFS UnixFs data model. The library includes the same functionality as the ipfs-car command line utility [described above](https://web3.storage/docs/how-tos/work-with-car-files/#ipfs-car). + +See the [ipfs-car README](https://github.com/web3-storage/ipfs-car#api) for API documentation and usage examples. + +#### @ipld/car + +The [@ipld/car](https://github.com/ipld/js-car)[package](https://github.com/ipld/js-car) contains the main JavaScript implementation of the CAR specification and is used by ipfs-car under the hood. If you want to store non-file data using [advanced IPLD formats](https://web3.storage/docs/how-tos/work-with-car-files/#advanced-ipld-formats), you should use @ipld/car directly. + +@ipld/car also provides the CarReader interface used by the web3.storage client's [putCar](https://web3.storage/docs/reference/js-client-library/#store-car-files)[method](https://web3.storage/docs/reference/js-client-library/#store-car-files). + +Here's a simple example of loading a CAR file from a Node.js stream and storing it with web3.storage: + +``` +import { reateReadStream } from 'fs'; + +import { CarReader } from '@ipld/car'; + +async function storeCarFile(filename) { + +const inStream = createReadStream(filename); + +const car = await CarReader.fromIterable(inStream); + +const client = makeStorageClient(); + +const cid = await client.putCar(car); + +console.log('Stored CAR file! CID:',cid); + +} +``` + +CarReader.fromIterable accepts any iterable of Uint8Array data, including Node.js streams. If you have all your CAR data in a single Uint8Array already, you can use [CarReader.fromBytes](https://github.com/ipld/js-car#CarReader__fromBytes) instead. + +The CarReader type shown above will read the entire contents of the CAR into memory, which may cause issues with large files. On Node.js, you can use [CarIndexedReader](https://github.com/ipld/js-car#carindexedreader), which reads CAR data from disk directly and uses less memory than CarReader. + +## [Advanced IPLD formats](https://web3.storage/docs/how-tos/work-with-car-files/#advanced-ipld-formats) + +IPLD can also be used as a general purpose data format like JSON. In fact, you can use JSON directly as IPLD just by using a special convention for linking to other IPLD objects. This convention is defined in the [dag-json](https://ipld.io/docs/codecs/known/dag-json/)["codec"](https://ipld.io/docs/codecs/known/dag-json/). + +Here's an example of a dag-json object: + +``` +{ + +"name": "Have you seen this dog?", + +"description": "I have now...", + +"image":{"/":"bafybeihkqv2ukwgpgzkwsuz7whmvneztvxglkljbs3zosewgku2cfluvba"} + +} +``` +The image field uses the special "link type" to reference another IPLD object. The link is just a regular JSON object with a single key named /, whose value is a Content Identifier. + +Although dag-json is familiar and easy to use, we recommend using the similar [dag-cbor](https://ipld.io/docs/codecs/known/dag-cbor/)[codec](https://ipld.io/docs/codecs/known/dag-cbor/) instead. dag-cbor uses the [Concise Binary Object Representation](https://cbor.io/) to more efficiently encode data, especially binary data which must be Base64-encoded when using dag-json. + +### [Examples](https://web3.storage/docs/how-tos/work-with-car-files/#examples) + +Below are some examples of working with dag-cbor data and sending it to web3.storage. + +First, you'll need to import some things: + +``` +import {Web3Storage} from 'web3.storage' + +import {CarReader} from '@ipld/car' + +import {encode} from 'multiformats/block' + +import *as cbor from '@ipld/dag-cbor' + +import {sha256} from 'multiformats/hashes/sha2' + +//#end region cbor Link Example + +//#region make UnixFs File + +import {importer} from 'ipfs-unixfs-importer' + +import {MemoryBlockStore} from 'ipfs-car/blockstore/memory' +``` + +Now we'll define a convenience function to encode an IPLD block of CBOR data and hash with SHA2-256: + +``` +encodeCborBlock(value) + +async function encodeCborBlock (value) { + return encode({ value, codec: cbor, hasher: sha256 }) +} +``` + +And a function to make a CAR from a collection of blocks and a root CID: + +``` +makeCar(rootCID, ipldBlocks) + +asyncfunction makeCar (rootCID,ipldBlocks){ + +return new CarReader (1,[rootCID],ipldBlocks) + +} +``` + +#### Storing simple CBOR data + +Using the helpers above, you can make a CAR file with a single block of simple CBOR data and send it to web3.storage: + +``` +simpleCborExample() + +async function simpleCborExample(){ + +//encode the value into an IPLD block and store with web3.storage + +const block = await encodeCborBlock({hello:'world'}) + +const car = await makeCar(block.cid,[block]) + +//upload to web3.storage using put Car + +const client = new Web3Storage({token:process.env.WEB3STORAGE\_TOKEN}) + +console.log('🤖Storing simple CBOR object...') + +const cid = await client.putCar(car) + +console.log(`🎉Done storing simple CBOR object.CID:${cid}`) + +console.log(`💡If you have ipfs installed, try:ipfsdagget${cid}\n`) + +} +``` + + +If you have the IPFS command line app installed, you can view the object you stored with the [ipfs dag get](https://docs.ipfs.io/reference/cli/#ipfs-dag-get)[command](https://docs.ipfs.io/reference/cli/#ipfs-dag-get), for example: + +``` +ipfs dag get bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae + +{ "hello" : "world"} +``` + +Note that the example output has been indented with [jq](https://stedolan.github.io/jq/) for clarity.The real command will output a compact dag-json representation of the CBOR data without any extra whitespace. + +#### CBOR with IPLD links + +You can link from one CBOR object to another using CIDs: + +``` +cborLinkExample() + +async function cborLinkExample(){ + +//Encode a simple object to get its CID + +const addressBlock = await encodeCborBlock({email:'zaphod@beeblebrox.galaxy'}) + +//Now we can use the CID to link to the object from another object + +const personBlock = await encodeCborBlock({ + +title:'Galactic President', + +description:'Just this guy, you know?', + +contact:addressBlock.cid + +}) + +//pack everything into a CAR + +const car = await makeCar(personBlock.cid,[personBlock,addressBlock]) + +//upload to web3.storage using putCar + +const client = new Web3Storage({token:process.env.WEB3STORAGE\_TOKEN}) + +console.log('🤖 Storing CBOR objects with CID links between them...') + +const cid = await client.putCar(car) + +console.log('🎉Stored linked data using dag-cbor.RootCID:',cid) + +console.log(`💡If you have ipfs installed, try: ipfsdagget${cid}`) + +console.log(`🔗You can also traverse the link by path:ipfsdagget${cid}/contact\n`) + +} + +async function make UnixFsFile(source){ + +const blockstore = new MemoryBlockStore() + +//taken from https://github.com/web3-storage/ipfs-car/blob/main/src/pack/constants.ts + +//but with wrapWithDirectory overriden to false + +const unixFs Options={ + +cidVersion:1, + +chunker:'fixed', + +maxChunkSize:262144, + +hasher:sha256, + +rawLeaves:true, + +wrapWithDirectory:false, + +maxChildrenPerNode:174 + +} + +const importStream = await importer(source,blockstore,unixFsOptions) + +let root = null + +for await(const entry of importStream){ + +root = entry + +} + +const blocks=[] + +for await(const block of blockstore.blocks()){ + +blocks.push(block) + +} + +await blockstore.close() + +return{root,blocks} + +} + +//#end region make UnixFs File + +//#region cbor Link To File Example + +async function cborLinkToFileExample(){ + +const source=[{ + +path: 'example.txt', + +content: new TextEncoder().encode('Some plain text, encoded to UTF-8') + +}] + +const {root,blocks} = await makeUnixFsFile(source) + +const cborBlock = await encodeCborBlock({ + +description:'A CBOR object that references a UnixFS file object by CID', + +file:root.cid + +}) + +blocks.push(cborBlock) + +const car = await makeCar(cborBlock.cid,blocks) + +const client = new Web3Storage({token:process.env.WEB3STORAGE_TOKEN}) + +console.log('🤖 Storing a CBOR object that links to a UnixFS file by CID...') + +const cid = await client.putCar(car) + +console.log('🎉Stored dag-cbor object that links to a unixfs file.RootCID:',cid) + +console.log(`💡If you have ipfs installed, try: ipfsdagget${cid}`) + +console.log(`💾You can view the linked file with ipfs: ipfscat${cid}/file`) + +console.log('🔗View linked file via IPFS gateway:',`https://${cid}.ipfs.dweb.link/file`) + +} + +//#end region cbor Link To File Example + +simpleCborExample() + +.then(cborLinkExample) + +.then(cborLinkToFileExample) + +.catch(console.error) +``` + +As with simple objects, you can use ipfs dag get to show the outer object: + +```ipfs dag get bafyreieq6bftbe3o46lrdbzj6vrvyee4njfschajxgmpxwbqex3czifhry``` + + +``` +{ + +"contact":{ + +"/":"bafyreicp2g6ez5exmw5uxsns7kkwtxr5z4vyx4xkdci6xpy2vou3zqc6me" + +}, + +"description" : "Just this guy, you know?", + +"title":"Galactic President" + +} +``` + +The contact field above contains an IPLD link, which can be included in the ipfs dag get command to resolve the linked object: + +```ipfs dag get bafyreieq6bftbe3o46lrdbzj6vrvyee4njfschajxgmpxwbqex3czifhry/contact``` + +```{"email":"zaphod@beeblebrox.galaxy"}``` + +#### Linking from CBOR to an IPFS file + +Our final example is a little more complex. We're going to store a file in the same UnixFS format that IPFS uses, and link to it from a CBOR object. + +First, we'll encode a file into UnixFS format. Normally, this is done by the client library, but we want to get the CID of the file object to use for our link before we send the file off to web3.storage, so we'll construct the UnixFS object ourselves. + +Here's a helper function to make a UnixFS file and encode it to an IPLD block: + +``` +makeUnixFsFile(source) + +import { importer } from 'ipfs-unixfs-importer' +import { MemoryBlockStore } from 'ipfs-car/blockstore/memory' +// #endregion imports + +// #region encodeCborBlock +async function encodeCborBlock (value) { + return encode({ value, codec: cbor, hasher: sha256 }) +} +// #endregion encodeCborBlock + +// #region makeCar +async function makeCar (rootCID, ipldBlocks) { + return new CarReader(1, [rootCID], ipldBlocks) +} +// #endregion makeCar + +// #region simpleCborExample +async function simpleCborExample () { + // encode the value into an IPLD block and store with web3.storage + const block = await encodeCborBlock({ hello: 'world' }) + const car = await makeCar(block.cid, [block]) + + // upload to web3.storage using putCar + const client = new Web3Storage({ token: process.env.WEB3STORAGE_TOKEN }) + console.log('🤖 Storing simple CBOR object...') + const cid = await client.putCar(car) + console.log(`🎉 Done storing simple CBOR object. CID: ${cid}`) + console.log(`💡 If you have ipfs installed, try: ipfs dag get ${cid}\n`) +} +// #endregion simpleCborExample + +// #region cborLinkExample +async function cborLinkExample () { + // Encode a simple object to get its CID + const addressBlock = await encodeCborBlock({ email: 'zaphod@beeblebrox.galaxy' }) + + // Now we can use the CID to link to the object from another object + const personBlock = await encodeCborBlock({ + title: 'Galactic President', + description: 'Just this guy, you know?', + contact: addressBlock.cid + }) + + // pack everything into a CAR + const car = await makeCar(personBlock.cid, [personBlock, addressBlock]) + + // upload to web3.storage using putCar + const client = new Web3Storage({ token: process.env.WEB3STORAGE_TOKEN }) + + console.log('🤖 Storing CBOR objects with CID links between them...') + const cid = await client.putCar(car) + console.log('🎉 Stored linked data using dag-cbor. Root CID:', cid) + console.log(`💡 If you have ipfs installed, try: ipfs dag get ${cid}`) + console.log(`🔗 You can also traverse the link by path: ipfs dag get ${cid}/contact\n`) +} +async function makeUnixFsFile (source) { + const blockstore = new MemoryBlockStore() + // taken from https://github.com/web3-storage/ipfs-car/blob/main/src/pack/constants.ts + // but with wrapWithDirectory overriden to false + const unixFsOptions = { + cidVersion: 1, + chunker: 'fixed', + maxChunkSize: 262144, + hasher: sha256, + rawLeaves: true, + wrapWithDirectory: false, + maxChildrenPerNode: 174 + } + const importStream = await importer(source, blockstore, unixFsOptions) + let root = null + for await (const entry of importStream) { + root = entry + } + const blocks = [] + for await (const block of blockstore.blocks()) { + blocks.push(block) + } + await blockstore.close() + return { root, blocks } +} +``` +The helper returns a root block, which we can link to by CID, as well as a blocks array containing the encoded file data. When we create the CAR to send to web3.storage, it's important to include all the file blocks as well as the CBOR block. + +``` +cborLinkToFileExample() + +asyncfunctioncborLinkToFileExample(){ + +constsource=[{ + +path:'example.txt', + +content:newTextEncoder().encode('Someplaintext,encodedtoUTF-8') + +}] + +const{root,blocks}=awaitmakeUnixFsFile(source) + +constcborBlock=awaitencodeCborBlock({ + +description:'ACBORobjectthatreferencesaUnixFSfileobjectbyCID', + +file:root.cid + +}) + +blocks.push(cborBlock) + +constcar=awaitmakeCar(cborBlock.cid,blocks) + +constclient=newWeb3Storage({token:process.env.WEB3STORAGE\_TOKEN}) + +console.log('🤖StoringaCBORobjectthatlinkstoaUnixFSfilebyCID...') + +constcid=awaitclient.putCar(car) + +console.log('🎉Storeddag-cborobjectthatlinkstoaunixfsfile.RootCID:',cid) + +console.log(`💡Ifyouhaveipfsinstalled,try:ipfsdagget${cid}`) + +console.log(`💾Youcanviewthelinkedfilewithipfs:ipfscat${cid}/file`) + +console.log('🔗ViewlinkedfileviaIPFSgateway:',`https://${cid}.ipfs.dweb.link/file`) + +} +``` + +As before, we can view the root block with ipfs dag get: + +``` +ipfs dag get bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy + +{ + +"description" : "A CBOR object that references a UnixFS file object by CID", + +"file":{ + +"/":"bafkreihmlglmfpadbk4fy72ljniveedbqicysoe5zhqqkgkuso3e6xyns4" + +} + +} +``` + +Since the file data is plain text, you can use ipfs dag get to fetch its contents: + +``` +ipfs dag get bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy/file + +"Some plain text, encoded to UTF-8" +``` + +Notice that the file content is wrapped in quotes because dag get is interpreting the content as a JSON string. + +To avoid this, or to fetch binary files, you can use ipfs get to download the file: + +``` +ipfs get bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy/file + + +Saving file(s) to file + +33B/33B[===============================================================]100.00%0s +``` + +Note that the IPFS HTTP gateway currently does not support rendering CBOR data, so the root object is not directly viewable via the gateway. See the note about gateway support below for more information. + +However, the gateway _can_ traverse the IPLD links inside our CBOR object, so you can link to the file by path and the gateway will resolve the linked file. For example: + +[https://bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy.ipfs.dweb.link/file](https://bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy.ipfs.dweb.link/file). + +##### Gateway support + +Although web3.storage supports storing CAR files with dag-cbor content by default and can accept other codecs with the decoders option, the IPFS HTTP gateway does not currently "speak" these formats and will not return such data over HTTP. Please follow [this issue](https://github.com/ipfs/go-ipfs/issues/8234) to track the development of this feature. + +### [Enabling IPLD codecs in the client library](https://web3.storage/docs/how-tos/work-with-car-files/#enabling-ipld-codecs-in-the-client-library) + +By default, the client's [putCar](https://web3.storage/docs/reference/js-client-library/#store-car-files)[method](https://web3.storage/docs/reference/js-client-library/#store-car-files) will accept data encoded using the dag-pb, dag-cbor, or raw codecs. If you want to use another codec like dag-json, you must include the codec in the decoders option to putCar. + +See the [putCar](https://web3.storage/docs/reference/js-client-library/#parameters-5)[parameter reference](https://web3.storage/docs/reference/js-client-library/#parameters-5) for more details and an example that uses dag-json. From 199961e15e445151b9a293fd9be2b8110f01c2ac Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 11:53:15 +0300 Subject: [PATCH 03/25] Create content-addressing.md --- src/pages/docs/concepts/content-addressing.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 src/pages/docs/concepts/content-addressing.md diff --git a/src/pages/docs/concepts/content-addressing.md b/src/pages/docs/concepts/content-addressing.md new file mode 100644 index 0000000..e64dc83 --- /dev/null +++ b/src/pages/docs/concepts/content-addressing.md @@ -0,0 +1,47 @@ +## Content addressing + +Content addressing in brief + +web3.storage's decentralized file storage relies on _content addressing_ to find, reference, and retrieve your files on the network. Content addressing is a technique for organizing and locating data in a system in which the key used to locate content is derived from the content itself, rather than its location. While you don't need to understand content addressing to be able to incorporate web3.storage in your apps and services, if you're curious about what's going on under the hood, read on. + +### [The basic problem](https://web3.storage/docs/concepts/content-addressing/#the-basic-problem) + +Consider what happens when you resolve a link like web3.storage/docs/concepts/content-addressing. First, your operating system queries a global shared key-value store, split into many domains — you may know this as the Domain Name System (DNS). The DNS returns an IP address that your network card can use to send HTTP requests over the network, where this site's naming conventions turn the key /concepts/content-addressing into a response payload. + +The problem is, components of an address like web3.storage/docs/concepts/content-addressing are _mutable_, meaning they can change over time. In the context of the web, where _everything_ is mutable and dynamic, this is just the way it's always been. As a result, [link rot](https://en.wikipedia-on-ipfs.org/wiki/Link_rot) is just something we've all learned to live with. + +### [CIDs: Location-independent, globally unique keys](https://web3.storage/docs/concepts/content-addressing/#cids-location-independent-globally-unique-keys) + +However, thanks to content addressing, link rot may become a thing of the past. A content-addressed system such as web3.storage is like our key-value-based DNS, with one significant difference: You no longer get to choose the keys. Instead, the keys are derived directly from the file contents using an algorithm that will always generate the same key for the same content. + +As a result, we no longer need to coordinate among multiple writers to our store by splitting the key space into domains and locations on file systems. There's now one universal domain: the domain of all possible values. If multiple people add the same value, there's no collision in the key space. They just each get the same key back from the put method, with one additional benefit: The availability and performance of retrievals on the network is increased. This gives our keys _location independence_. There's one other important result: Each individual key is a unique signature for the data itself, ensuring _verifiability_ that the key matches the content and the content hasn't been altered. + +This type of key is called a _content identifier (CID)_. Once you know the CID of a file on the web3.storage network, you have all you need for the network to locate and return the file back to you. + +[web3.storage CIDs under the hood](https://web3.storage/docs/concepts/content-addressing/#web3storage-cids-under-the-hood) + +web3.storage uses CIDs to make its decentralized file storage work, with help from [IPFS](https://ipfs.io/) for locating files and making sure they're always available. + +Content addressing is the basis of the peer-to-peer hypermedia protocol IPFS (the InterPlanetary File System), which web3.storage uses to locate files. When web3.storage stores your data on IPFS, it can be retrieved from any IPFS node that has a copy of that data. This can make data transfers more efficient and reduce the load on any single node. As each user fetches a piece of data, they keep a local copy around to help other users who might request it later. + +In addition to web3.storage making it easy to get your data onto the content-addressed IPFS network, it also provides long-term persistence for your files using the decentralized Filecoin storage network. The Filecoin network incentivizes participants to provide storage space for files on the network. By combining IPFS and Filecoin storage into one easy-to-use service, web3.storage makes it simple to store, locate, and retrieve your files on the decentralized web. + +There are two types of CIDs that web3.storage interacts with: + +- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. This is what the vast majority of IPFS users will be using. +- Shard CIDs: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. + +In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs. + +### [Summary](https://web3.storage/docs/concepts/content-addressing/#summary) + +Using content addressing for locating files rather than the legacy web's method of location-dependent addressing responds to several critical weaknesses of the legacy web: + +- Content addressing solves for the problem behind link rot — the mutability of location-dependent storage systems — by using a hashing algorithm to generate a unique CID for each file that can be used as the lookup key for a file rather than a URL. +- In addition to making sure files don't get lost if they're moved, content addressing also ensures that users intending to retrieve a specific version of a file will be guaranteed to retrieve that version for as long as it exists anywhere on the network. + +### [Learn more](https://web3.storage/docs/concepts/content-addressing/#learn-more) + +Want a deep dive into content addressing, how it works, and why it's important? Check out [ProtoSchool](https://proto.school/content-addressing/) for an in-depth look at content addressing on the decentralized web, plus a wealth of other interactive tutorials on DWeb concepts, protocols, and tools. + +Want a technical explanation of how IPFS CIDs work in particular? Have a look at the [official IPFS docs](https://docs.ipfs.io/concepts/content-addressing/). From b0f8a7421d2961dab7ab5b144e0091bbb759196e Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 11:53:41 +0300 Subject: [PATCH 04/25] Create filecoin-storage.md --- src/pages/docs/concepts/filecoin-storage.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 src/pages/docs/concepts/filecoin-storage.md diff --git a/src/pages/docs/concepts/filecoin-storage.md b/src/pages/docs/concepts/filecoin-storage.md new file mode 100644 index 0000000..e95a336 --- /dev/null +++ b/src/pages/docs/concepts/filecoin-storage.md @@ -0,0 +1,13 @@ +## Filecoin storage + +### Filecoin storage + +The Filecoin network is a great building block for any decentralized storage system. Independent storage providers periodically must cryptographically prove that they are physically storing your specific data for a specific duration of time. When they submit these proofs to the network, other nodes verify these proofs, and this is what ends up on the Filecoin blockchain. So, anyone at any given moment can trustlessly verify that specific content is persisted, the number of copies on the network, and with who they are stored with. Read more about Filecoin proofs [here](https://filecoin.io/blog/posts/what-sets-us-apart-filecoin-s-proof-system/). + +Because of the open nature of the Filecoin network allowing anyone to participate, it's very inexpensive to store data on the Filecoin network. As a result, web3.storage uses the network today to back up all its data on the decentralized web. In the future, we will launch new products that allow users to take advantage of Filecoin for "colder," disruptively expensive data storage. + +The great thing about IPFS is that, regardless of where your data is sitting (on web3.storage's dedicated servers, Filecoin, on your local IPFS node, or anywhere else), you access your data the same way - using the data's content address - and allows you to tune how many copies you have if you'd like, which has tradeoffs in terms of cost, availability, and read performance! + +When storing data on Filecoin, you enter storage "deals" that have a finite duration. web3.storage service currently renews deals for you to ensure your data deals never expire. + +[https://web3.storage/docs/concepts/w3link/](https://web3.storage/docs/concepts/w3link/) From 3641c64269c2d4b7e3c53db5b54d51e2a4e3c3f5 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 11:54:14 +0300 Subject: [PATCH 05/25] Create index.md --- src/pages/docs/concepts/index.md | 151 +++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 src/pages/docs/concepts/index.md diff --git a/src/pages/docs/concepts/index.md b/src/pages/docs/concepts/index.md new file mode 100644 index 0000000..4e21fe0 --- /dev/null +++ b/src/pages/docs/concepts/index.md @@ -0,0 +1,151 @@ +## Concepts + +## UCANs and web3.storage + +How web3.storage uses UCAN + +For authorization, w3up services use [ucanto](https://github.com/web3-storage/ucanto), a Remote Procedure Call (RPC) framework built around [UCAN](https://ucan.xzy/), or User Controlled Authorization Networks. UCANs are a powerful capability-based authorization system that allows fine-grained sharing of permissions through a process called _delegation_ on top of [public key cryptography](https://en.wikipedia.org/wiki/Public-key_cryptography). + +You can think about UCAN replacing bearer tokens in traditional APIs for authorization with w3up. Since any actor can be represented by a cryptographic keypair and permissions can be delegated to them, users can interact with w3up directly in cases where a developer might have needed to previously run additional back-end infrastructure to keep API keys secure. This can be extended even to have end users using applications integrated with w3up using their own keypair-based identity. + +### How w3up and w3up-client use UCANs + +Our client and CLI use ucanto to take care of the details of UCANs for you, but a few of the underlying terms and concepts may "bubble up" to the surface of the API, so we'll cover the basics. We'll also go over some terms that are specific to web3.storage that you might not have encountered elsewhere. + +UCAN-based APIs are centered around _capabilities_, which are comprised of an _ability_ and a _resource_. Together, the ability and resource determine what action a client can perform and what objects in the system can be acted upon. When invoking a service method, a client will present a UCAN token that includes an ability and resource, along with _proofs_ that verify that they should be allowed to exercise the capability. The proof might be signed directly by the capability owner, or have a chain of signatures (_delegations_) where the actor invoking the capability has been verifiably delegated permission to do so. + +#### [Space](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#space) + +When you upload data to w3up, your uploads are linked to a unique _Space_ that acts as a "namespace" for the data you upload. Each Space corresponds to a _DID_, or [Decentralized Identity Document](https://www.w3.org/TR/did-core/). In web3.storage's implementation of w3up, these Space DIDs generally use the key DID method, of the form did:key:publicKey with a corresponding private signing key. + +When creating a Space, it generates this private key and did:key for you locally. To use web3.storage, you then register a Space by associating it with your email address. From there, when invoking storage capabilities with web3.storage, the Space did:key is the "resource" portion of the capability, while the ability is an action like store/add or store/remove. (A Space registered with web3.storage is imperfectly analogous to an "account" with web3.storage.) + +Under the hood in the email registration process, your Space delegates the capabilities needed to use w3up to your email address, and this delegation is stored by web3.storage. If you need access to your Space in the future from any device, web3.storage allows you to reclaim those capabilities the same way you would reset a password in other services - using an email verification process. This means you don't need to store or manage Space private keys to use w3up - just create a new space, register it with w3up and use it from as many devices as you like. More on this "sign in" process is detailed in the next section on Agents. + +#### Agent + +To invoke a capability like store/add on a Space using the client or CLI, the client must have an _Agent_. Like a Space, an Agent corresponds to a did:key whose private key is generated locally. An Agent is useful once the client or CLI has a UCAN delegation where a registered Space(s) delegates the Agent its capabilities. (An imperfect analogy is Agent to login session.) + +The delegation from a Space to your Agent that w3up-client needs can be passed either by verifying the email address the Space is registered to and claiming the UCAN delegation (authorize(email) then capability.access.claim) or directly if you have the UCAN delegation available locally (addSpace(delegation)). + +#### Delegation to other actors + +Just like Spaces can delegate permissions to Agents you own, you can also delegate permissions to other actors' Agents. One common application of this could be you delegating permission to upload to your Space to your users. Here's a code snippet demonstrating this from the Upload section: + +``` +import { CarReader } from '@ipld/car'; + +import \* as DID from '@ipld/dag-ucan/did'; + +import \* as Delegation from '@ucanto/core/delegation'; + +import { importDAG } from '@ucanto/core/delegation'; + +import \* as Signer from '@ucanto/principal/ed25519'; + +import \* as Client from '@web3-storage/w3up-client'; + +async function backend(did: string) { + +// Load client with specific private key + +const principal = Signer.parse(process.env.KEY); + +const client = await Client.create({ principal }); + +// Add proof that this agent has been delegated capabilities on the space + +const proof = await parseProof(process.env.PROOF); + +const space = await client.addSpace(proof); + +await client.setCurrentSpace(space.did()); + +// Create a delegation for a specific DID + +const audience = DID.parse(did); + +const abilities = ['store/add', 'upload/add']; + +const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24; // 24 hours from now + +const delegation = await client.createDelegation(audience, abilities, { + +expiration, + +}); + +// Serialize the delegation and send it to the client + +const archive = await delegation.archive(); + +return archive.ok; + +} + +/\*\* @param {string} data Base64 encoded CAR file \*/ + +async function parseProof(data) { + +const blocks = []; + +const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); + +for await (const block of reader.blocks()) { + +blocks.push(block); + +} + +return importDAG(blocks); + +} + +async function frontend() { + +// Create a new client + +const client = await Client.create(); + +// Fetch the delegation from the backend + +const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; // backend method is exposed at this API URL + +const response = await fetch(apiUrl); + +const data = await response.arrayBuffer(); + +// Deserialize the delegation + +const delegation = await Delegation.extract(new Uint8Array(data)); + +if (!delegation.ok) { + +throw new Error('Failed to extract delegation'); + +} + +// Add proof that this agent has been delegated capabilities on the space + +const space = await client.addSpace(delegation.ok); + +client.setCurrentSpace(space.did()); + +// READY to go! + +} +``` + +You can see the following flow: + +- When `backend` function is called in the developer's backend: + - It's passed the DID of the user's Agent + - Backend client initializes with an Agent that has permission to the developer's Space + - It then generates a UCAN delegated to the user Agent DID passed in with only the `store/add` and `upload/add` abilities (to give the user ability to upload) and set to expire in 24 hours +- When `frontend` function is called in the user's environment: + - An Agent DID is created + - The `backend` function hosted at an API endpoint is called, passing in the Agent DID + - The client is set up with a UCAN delegating upload capabilties to the Agent + - It's now ready to upload! + +However, there's other interesting possibilities - for instance, you could create an app where your users make Spaces and delegate permission to your app to read their uploads. Read the Architecture options section to explore more. From b5e19633e8a742c2bede996bd92539f1a4ddbc49 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 11:55:58 +0300 Subject: [PATCH 06/25] Create ipfs-gateways.md --- src/pages/docs/concepts/ipfs-gateways.md | 43 ++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 src/pages/docs/concepts/ipfs-gateways.md diff --git a/src/pages/docs/concepts/ipfs-gateways.md b/src/pages/docs/concepts/ipfs-gateways.md new file mode 100644 index 0000000..32151e5 --- /dev/null +++ b/src/pages/docs/concepts/ipfs-gateways.md @@ -0,0 +1,43 @@ +## IPFS Gateways + +### IPFS HTTP Gateways + +web3.storage uses the [InterPlanetary File System (IPFS)](https://ipfs.io/) as a key part of its storage and retrieval infrastructure. + +The IPFS network is a peer-to-peer network of computers that share resources to efficiently provide content to anyone that requests it. Computers that join the IPFS network supply blocks of data using a hash-based Content Identifiers (CIDs). + +To make IPFS data accessible outside of the peer-to-peer network, [special IPFS nodes called "gateways"](https://docs.ipfs.io/concepts/ipfs-gateway/) act as bridges between the HTTP protocol that all web browsers understand and IPFS-native protocols. + +As more browsers like [Brave](https://brave.com/ipfs-support/) and [Opera](https://blogs.opera.com/tips-and-tricks/2021/02/opera-crypto-files-for-keeps-ipfs-unstoppable-domains/) adopt native IPFS support, the need for gateways will naturally lessen over time. Today, you can reach the widest audience by using HTTP gateways in your web applications, but it's a great idea to also surface the original ipfs:// URI for your content, so that IPFS-native browsers can access the content directly through Bitswap. + +For more information about fetching content that you uploaded through an IPFS HTTP gateway, see the Retrieve section. + +## [Types of gateway](https://web3.storage/docs/concepts/w3link/#types-of-gateway) + +The official [IPFS documentation on gateways](https://docs.ipfs.io/concepts/ipfs-gateway/) is helpful for understanding the types of gateways in the IPFS ecosystem and how they're used. + +One of the key things to understand for our purposes is the different [resolution styles](https://docs.ipfs.io/concepts/ipfs-gateway/#resolution-style) that can be used to fetch content using gateway URLs. + +If you check the [list of public gateways](https://ipfs.github.io/public-gateway-checker/), you'll see that some support "subdomain" style URLs, while others only support path-style URLs. Below is a short refresher on the distinction between the two styles. + +### [Path style URLs](https://web3.storage/docs/concepts/w3link/#path-style-urls) + +A "path style" URL puts the IPFS CID into the path portion of the gateway URL, like this: + +[https://w3s.link/ipfs/bafkreied5tvfci25k5td56w4zgj3owxypjgvmpwj5n7cvzgp5t4ittatfy](https://w3s.link/ipfs/bafkreied5tvfci25k5td56w4zgj3owxypjgvmpwj5n7cvzgp5t4ittatfy) + +If the CID points to a directory listing, you can append the name of a file within the directory to fetch the file: + +[https://w3s.link/ipfs/bafybeid4gwmvbza257a7rx52bheeplwlaogshu4rgse3eaudfkfm7tx2my/hi-gateway.txt](https://w3s.link/ipfs/bafybeid4gwmvbza257a7rx52bheeplwlaogshu4rgse3eaudfkfm7tx2my/hi-gateway.txt) + +### [Subdomain style URLs](https://web3.storage/docs/concepts/w3link/#subdomain-style-urls) + +A "subdomain style" gateway URL puts the CID into the host portion of the URL, as a subdomain of the gateway host, like this: + +[https://bafkreied5tvfci25k5td56w4zgj3owxypjgvmpwj5n7cvzgp5t4ittatfy.ipfs.w3s.link](https://bafkreied5tvfci25k5td56w4zgj3owxypjgvmpwj5n7cvzgp5t4ittatfy.ipfs.w3s.link/) + +If the CID points to a directory listing, you can use the path portion of the URL to specify the filename: + +[https://bafybeid4gwmvbza257a7rx52bheeplwlaogshu4rgse3eaudfkfm7tx2my.ipfs.w3s.link/hi-gateway.txt](https://bafybeid4gwmvbza257a7rx52bheeplwlaogshu4rgse3eaudfkfm7tx2my.ipfs.w3s.link/hi-gateway.txt) + +This is the preferred style for serving web assets over HTTP gateways, because web browsers provide security isolation on a per-domain basis. Using the subdomain style, every CID gets its own "namespace" for things like cookies and local storage, which isolates things from other web content stored on IPFS. From e10ab03ac6cb7f14f1e562a7bc9dee7a4f690787 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 11:56:28 +0300 Subject: [PATCH 07/25] Create upload-vs-store.md --- src/pages/docs/concepts/upload-vs-store.md | 86 ++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 src/pages/docs/concepts/upload-vs-store.md diff --git a/src/pages/docs/concepts/upload-vs-store.md b/src/pages/docs/concepts/upload-vs-store.md new file mode 100644 index 0000000..5608421 --- /dev/null +++ b/src/pages/docs/concepts/upload-vs-store.md @@ -0,0 +1,86 @@ +## Upload vs. Store in web3.storage + +## Upload vs. Store capabilities in web3.storage + +There are two types of content identifiers (CIDs) that web3.storage interacts with: + +- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. +- Shard CIDs: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. + +One example of where you might see both is uploading a large file with the CLI: + +``` +w3 up gb.file + +1 file (1GB) + +bagbaierao... + +bagbaieraq... + +bagbaieraj... + +bagbaierai... + +bagbaierax... + +bagbaieraf... + +bagbaierac... + +bagbaierax... + +bagbaierax... + +bagbaiera4... + +⁂ Stored 1 file + +⁂ https://w3s.link/ipfs/bafybeiaxwvg4... +``` + +The CLI sharded the 1GB upload into 10 shards, each with a `bag…`-prefixed CID. The content CID of the file itself is included in the `w3s.link/ipfs/bafy…` link at the bottom. + +In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs. + +However, if you're interested in learning more about how web3.storage uses both, read on! + +### Upload vs. Store + +There are two similar-sounding, complementary, but separate concepts in web3.storage: Upload and Store. The place you most readily see this is in the lower-level client methods like `Client.capabilities.upload.*` and `Client.capabilities.store.*`, and CLI methods like `w3 can upload *`, and `w3 can store *`. + +The main difference is that upload methods interact with content CIDs and store methods interact with shard CIDs. However, both are needed to make your uploads available to the IPFS network! + +Let's break it down. In the CLI example above where we called `w3 up file`, we saw that both shard CIDs and a content CID were returned. That's because `w3 up` calls both `store` and `upload` were called under the hood (but abstracts the complexity for any user that doesn't care about it). + +The CLI and client first take the upload and converts it into an DAG (directed acyclic graph). This is what IPFS uses to generate the content CID - each node in the graph has its own CID, with the graph's leafs containing the upload's data, and the root node of the graph the content CID of the entire graph. However, to send the data to web3.storage, the content has to be in a different form. web3.storage achieves this by converting the graph into a set of CAR file shards. + +Each CAR shard has a CID associated with the data itself (i.e., as it sits on disk serialized, with a header for instructions on how the blocks within the serialized data should be arranged, etc.). The CLI and client send each of these shards one-by-one to web3.storage by invoking the `store/add` UCAN capability under the hood (e.g., `w3 can store add`)! + +However, from web3.storage's perspective, it doesn't necessarily know whether the set of CAR shards it was sent from the series of `store/add`s corresponds to a single content CID that the user cares about, multiple content CIDs, or none at all (e.g., the CAR files sent represent an incomplete graph). As a result, `upload/add` allows the user to explicitly register a content CID with a set of shard CIDs. This is primarily done for the user's sake - it makes it easier to track which content CIDs (what they're using to fetch data from the IPFS network) correspond to which shard CIDs (what is physically being stored with web3.storage). + +In cases where the user is uploading a whole file or directory like in the example above, it's safe to know that the series of CAR shards uploaded by the user correspond to the file's content CID that it cares about. That's why the higher-level `w3 up` method is appropriate to use, and should represent the vast majority of upload use cases by web3.storage users. + +### When should I care about Store and shard CIDs? + +There's a few cases to note when caring about shard CIDs themselves and the `store` series of capabilities. + +**Sharing data across content CIDs** + +Because IPFS interacts with DAGs, you can actually share blocks between different content CIDs. One simple example would be in a directory: + +\ + +If a user uploads this directory using `w3 up`, the content CID for this directory will be registered with their account. This is actually the root CID of the directory itself, but isn't the only content CID in reality that was made available on the network: each file within this directory actually has their own content CID (highlighted). + +Let's say you want to be able to have two different directories registered as uploads in your account: one that has 3 of the files above, and another that has all of them. One way to do this would be to upload each file separately, then locally compute the content CIDs of each directory. You can then call `w3 can upload add` to register each of these directories. + +This results in only a single copy of each file being stored, but you being able to interact with two different directories! + +**Removing data from account** + +web3.storage tracks usage for payment (i.e., how much storage is utilized by a user) using the volume of data associated with shard CIDs. This should make sense after learning about the difference between `store` and `upload` - web3.storage is storing the CAR shards themselves, and `upload`s are more users tracking. + +Fortunately, this shouldn't make things any more complicated - we go into more detail below, but in general, when you remove a content CID from your account, you'll want to remove the shard CIDs as well (e.g., in the client calling `Client.remove(contentCID, shards=True)`). + +However, if you are a power user interacting with shard CIDs as well (like in the previous section), then you need to be more cautious about removing shard CIDs from your account. (This is why the default for the client and CLI methods is for shards to be maintained after removing a content CID). Learn more about how to do this in the Remove section. From f64e6798554a4379ace390e8339daa3bb5b9d7f9 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:04:49 +0300 Subject: [PATCH 08/25] Create how-tos.md --- src/pages/docs/concepts/how-tos.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 src/pages/docs/concepts/how-tos.md diff --git a/src/pages/docs/concepts/how-tos.md b/src/pages/docs/concepts/how-tos.md new file mode 100644 index 0000000..fe7357d --- /dev/null +++ b/src/pages/docs/concepts/how-tos.md @@ -0,0 +1,23 @@ +# How-tos + +## Create Account and Space + +**How to create an account** + +In the How-tos section of the docs, we show the most familiar, straightforward way to use web3.storage: by setting up an account for you, the developer, that you use in your application. For an overview of the various ways web3.storage can be integrated with your application, check out the "Architecture options" concepts section. + +You first need to create and register a Space you'll upload to. A Space acts as a namespace for your uploads. It is created locally and associated with a private key, and is then registered with web3.storage and associated with your email address. But don't worry about keeping track of the Space's private key! web3.storage's email authorization system allows this private key to be treated as a throwaway key. + +**The easiest way to create and register a Space is by using the CLI.** + +1. Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli` +2. Run `w3 authorize [alice@example.com](mailto:alice@example.com)` in the command line using your email address. This will send an email to your inbox with a link for validation. +3. Once you click on the validation link, you'll be taken to a webpage where you can enter your payment information and select a plan (like our Free tier). +4. Create a new Space for storing your data and register it: +w3 space create + +**w3 space register** + +Separately, you can visit console.web3.storage, sign up with your email and select a plan, and create a space using the UI, but we recommend that developers get familiar with the CLI since it's a powerful tool for many things you might want to do. + +The Space you create can be used to upload data using the CLI, the w3up client, or when you log into the web console. We discuss these various ways to upload in the next section, Store. From 50bf689b5db7a90c21749730472157158336ffcb Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:06:28 +0300 Subject: [PATCH 09/25] Create index.md --- src/pages/docs/concepts/{how-tos.md => how-tos/index.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename src/pages/docs/concepts/{how-tos.md => how-tos/index.md} (100%) diff --git a/src/pages/docs/concepts/how-tos.md b/src/pages/docs/concepts/how-tos/index.md similarity index 100% rename from src/pages/docs/concepts/how-tos.md rename to src/pages/docs/concepts/how-tos/index.md From fd203289420713c1e51e1151190f4b769551abd5 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:07:21 +0300 Subject: [PATCH 10/25] Rename src/pages/docs/concepts/how-tos/index.md to src/pages/docs/how-tos/index.md --- src/pages/docs/{concepts => }/how-tos/index.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename src/pages/docs/{concepts => }/how-tos/index.md (100%) diff --git a/src/pages/docs/concepts/how-tos/index.md b/src/pages/docs/how-tos/index.md similarity index 100% rename from src/pages/docs/concepts/how-tos/index.md rename to src/pages/docs/how-tos/index.md From 2b47f33d5ea98f5b1e30b39c51ee928994dce2f0 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:07:55 +0300 Subject: [PATCH 11/25] Create list.md --- src/pages/docs/how-tos/list.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 src/pages/docs/how-tos/list.md diff --git a/src/pages/docs/how-tos/list.md b/src/pages/docs/how-tos/list.md new file mode 100644 index 0000000..a61adf7 --- /dev/null +++ b/src/pages/docs/how-tos/list.md @@ -0,0 +1,34 @@ +## List + +### How to list files uploaded to web3.storage + +In this how-to guide, you'll learn about the different ways that you can list the files that you've uploaded to web3.storage. Once you've [stored some files](https://web3.storage/docs/how-tos/store/) using web3.storage, you'll want to see a list of what you've uplodaded. There are two ways you can do this: + +- Programmatically using the web3.storage client or CLI +- Using the web3.storage console + +## [Using the web3.storage client](https://web3.storage/docs/how-tos/list/#using-the-web3storage-client) or CLI + +You can also access a listing of your uploads from your code using the web3.storage client. In the example below, this guide walks through how to use the JavaScript client library to fetch a complete listing of all the data you've uploaded using web3.storage. + +For instructions on how to set up your client instance or CLI, check out the Upload section. + +\ + +Here's what you call, here's what the output looks like. (client and CLI) + +Timestamps. Data size? + +Today, like other developer object storage solutions, no sorting or querying by timestamp to keep things scalable. + +Can get nested list of shard CIDs, or look up what the shard CIDs are for an individual upload. Client and CLI. If you'd like to learn more check out Upload vs. Store section + +Filecoin deal info and what fields mean + +## [Using the web3.storage](https://web3.storage/docs/how-tos/list/#using-the-web3storage-website)console + +You can see a list of everything you've uploaded to web3.storage in your console on the website. If you don't need to work with this list programmatically, using the website may be a simpler choice. + +\ + +This console provides a convenient overview of your stored data, including links to view your files in your browser via an [IPFS gateway](https://docs.ipfs.io/concepts/ipfs-gateway/) and information about how the data is being stored on the decentralized storage networks that web3.storage uses under the hood. From eb17a6d8a53f811e0aa1607eb10c61d3b7698bfe Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:08:19 +0300 Subject: [PATCH 12/25] Create plan.md --- src/pages/docs/how-tos/plan.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 src/pages/docs/how-tos/plan.md diff --git a/src/pages/docs/how-tos/plan.md b/src/pages/docs/how-tos/plan.md new file mode 100644 index 0000000..2ad1e55 --- /dev/null +++ b/src/pages/docs/how-tos/plan.md @@ -0,0 +1,17 @@ +## View or change plan settings + +To view or change account settings, navigate to the web console. Sign in if you need to, and click "Plan settings." + +\ + +You can see your current plan, and select a new plan if you'd like to move to it. + +\ + +You can also see your current payment method on file and change it if you need to. + +\ + +You can access your previous invoices by clicking on "See invoices." + +\ From 480ba8d7287f956262e260c29975320b836a845b Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:08:38 +0300 Subject: [PATCH 13/25] Create receipts.md --- src/pages/docs/how-tos/receipts.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 src/pages/docs/how-tos/receipts.md diff --git a/src/pages/docs/how-tos/receipts.md b/src/pages/docs/how-tos/receipts.md new file mode 100644 index 0000000..b765817 --- /dev/null +++ b/src/pages/docs/how-tos/receipts.md @@ -0,0 +1,9 @@ +## Querying UCAN receipts + +You'll be able to get the receipt for a given task. This is especially useful for getting more information about async tasks that are found in receipts for other UCAN invocations. To read more about how web3.storage uses UCANs and why they're so useful, check out the UCANs and web3.storage section. + +e.g. in the case of filecoin/offer, you'll get an instant receipt with an async task of filecoin/accept. + +You'll be able to ask for the receipt for filecoin/accept to get the aggregate CID and information of the deal the aggregate was first added to (when the task completes). + +You can also then invoke deal/info with the aggregate CID to get the current deal information. From 02902cbf111606c2510e9aa1758d4d550d4c15d1 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:09:10 +0300 Subject: [PATCH 14/25] Create remove.md --- src/pages/docs/how-tos/remove.md | 124 +++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 src/pages/docs/how-tos/remove.md diff --git a/src/pages/docs/how-tos/remove.md b/src/pages/docs/how-tos/remove.md new file mode 100644 index 0000000..98cb0eb --- /dev/null +++ b/src/pages/docs/how-tos/remove.md @@ -0,0 +1,124 @@ +## Remove + +### Removing data from your account + +You might want to remove data from being associated with your account. You can do so using the JS client, CLI, or web app. + +Note that there is a minimum 30 day retention period for uploaded data, and even once removed, the data might persist on the public IPFS network. + +**CAUTION** + +⚠️❗ Public Data 🌎: All data uploaded to w3up is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using w3up. + +⚠️❗ Permanent Data ♾️: Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future. + +**Removing uploads (content CIDs) vs. stores (shard CIDs)** + +web3.storage tracks two different things for its users to support content addressing. These concepts were first introduced in the Upload section: + +- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. +- Shard CIDs: The CID of the serialized shards of data itself (CAR files) that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. + +web3.storage tracks usage for payment (i.e., how much storage is utilized by a user) using the volume of data associated with shard CIDs. However, in general, most users will be interacting with content CIDs (this is how you fetch your data from the network), with shard CIDs more of an implementation detail (how data gets chunked, serialized into CAR files, and stored for uploads). + +Fortunately, this shouldn't make things any more complicated - we go into more detail below, but in general, when you remove a content CID from your account, you'll want to remove the shard CIDs as well (e.g., in the client calling `Client.remove(contentCID, shards=True)`). + +However, if you are a power user interacting with shard CIDs as well (e.g., using the client's `capability.store.*` or CLI's `w3 can store *` methods), then you need to be more cautious about removing shard CIDs from your account. (This is why the default for the client and CLI methods is for shards to be maintained after removing a content CID). You can read more about why you might want to interact with shard CIDs directly and the implications in the Upload vs. Store section. + +**Using the client or CLI** + +If you followed the Upload section, you should already have your client or CLI set up with an Agent for your Space. From there, to remove a content CID from your account, you'll generally be using: + +- Client: `Client.remove(contentCID)` +- CLI: `w3 rm \` + +If you initially uploaded your content by using the recommended upload methods (e.g., used `Client.upload()` or `w3 up`) and didn't interact with CAR shards at all when uploading, we recommend removing the shard CIDs associated with the content CID from your account. Otherwise, you will still be paying for the data stored with web3.storage (as mentioned above). The easiest way to do that is to set the `shards` parameter as `True`: + +- Client: `Client.remove(contentCID, shards=True)` +- CLI: `w3 rm \ --shards` in the CLI). + +A full example of this is: + +``` +import { create } from '@web3-storage/w3up-client' + +import \* as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys + +const principal = Signer.parse(process.env.KEY) // Agent private key + +const client = await create({ principal }) + +async function main () { + +// Load client with specific private key + +const principal = Signer.parse(process.env.KEY) + +const client = await Client.create({ principal }) + +// Add proof that this agent has been delegated capabilities on the space + +const proof = await parseProof(process.env.PROOF) + +const space = await client.addSpace(proof) + +await client.setCurrentSpace(space.did()) + +// remove content previously uploaded, including the underlying shards + +client.remove('bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu', shards=True) + +} + +/\*\* @param {string} data Base64 encoded CAR file \*/ + +async function parseProof (data) { + +const blocks = [] + +const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) + +for await (const block of reader.blocks()) { + +blocks.push(block) + +} + +return importDAG(blocks) + +} +``` +**Removing content CIDs and shard CIDs separately** + +If you have managed your shard CIDs and upload CIDs separately (e.g., used `Client.capability.store.add()` and `Client.capability.upload.add()` in the client or `w3 can store add` and `w3 can upload add` in the CLI), you'll want to remove the upload CIDs and underlying shard CIDs separately as well. You can read more about why you might want to interact with shard CIDs directly and the implications in the Upload vs. Store section. + +To remove shard CIDs and upload CIDs separately, you'll generally do this by: + +- Client: + - If you registered a content CID you want to remove using `Client.capability.upload.add(contentCID)`… + - (If you don't know which shard CIDs are associated with the content CID) Run `Client.capability.upload.listShards(contentCID)`, which returns a list of shard CIDs + - Remove it using `Client.capability.upload.remove(contentCID)` + - Remove the shard CIDs that you'd like to + - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) + - Remove the shard CIDs one-by-one using `Client.capability.store.remove(shardCID)` +- CLI: + - If you registered a content CID you want to remove using `w3 can upload add \`… + - (If you don't know which shard CIDs are associated with the content CID) Run `w3 can upload ls \ --shards, which returns a list of shard CIDs + - Remove it using `w3 can upload rm \` + - Remove the shard CIDs that you'd like to + - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) + - Remove the shard CIDs one-by-one using `w3 can store rm \` + +**Using the web console** + +If you want to remove uploads via the web app, first navigate to the web console. + +\ + +Click the checkbox next to the upload CIDs you'd like to remove. + +\ + +By default, the underlying shard CIDs are also selected, which you can see by expanding the content CID using the dropdown arrow. If you would like to retain the shard CIDs, click "Unselect shard CIDs" next to the upload CID. Note that the shard CIDs must be deleted to no longer pay for their storage, and that most users will want to always delete the shard CIDs associated with their upload CIDs (e.g., if you've only uploaded content through the web console). + +\ From 99b4494509238b6034fb595ec2e8278de03d9a80 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:09:56 +0300 Subject: [PATCH 15/25] Create retrieve.md --- src/pages/docs/how-tos/retrieve.md | 99 ++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 src/pages/docs/how-tos/retrieve.md diff --git a/src/pages/docs/how-tos/retrieve.md b/src/pages/docs/how-tos/retrieve.md new file mode 100644 index 0000000..0a3d2f6 --- /dev/null +++ b/src/pages/docs/how-tos/retrieve.md @@ -0,0 +1,99 @@ +## Retrieve + +## How to retrieve data from web3.storage + +In this how-to guide, you'll learn several methods for retrieving data from web3.storage. + +All data stored using web3.storage is made available for retrieval via [IPFS](https://ipfs.io/), the InterPlanetary File System. IPFS is a distributed, peer-to-peer network for storing and sharing content-addressed data. This guide shows you several ways to retrieve your data from IPFS: + +- In your browser using an [HTTP gateway](https://web3.storage/docs/how-tos/retrieve/#using-an-ipfs-http-gateway). +- Using the Saturn dCDN. +- In your terminal using the [IPFS command-line tools](https://web3.storage/docs/how-tos/retrieve/#using-the-ipfs-command-line). +- In your terminal using [curl or Powershell](https://web3.storage/docs/how-tos/retrieve/#using-curl-or-powershell). + +When retrieving any data, you'll be using the content CID of the upload (prefixed by `bafy…`). + +**Using an IPFS HTTP gateway** + +You can easily fetch any data stored using web3.storage using an IPFS HTTP gateway. Because IPFS is a peer-to-peer, decentralized network, you can use any public HTTP gateway to fetch your data. In this guide, we'll use the gateway at w3s.link (which is optimized for data stored with web3.storage), but you can see more worldwide gateways on the [IPFS Public Gateway Checker](https://ipfs.github.io/public-gateway-checker/). + +You can use an IPFS gateway to view a list of all the files in that directory from your browser. To do so, simply create a gateway URL. For example, if your CID is bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu, you can make a URL for the w3s.link gateway as follows: `https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/`. Follow that link, and you'll see a page similar to this: + +![](RackMultipart20231107-1-k3fcf9_html_5bb9fefc33b3be31.png) + +If you want to link directly to a file within that directory, just add the file path after the CID portion of the link. For example: [bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg](https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg) could be used as a shareable link for your new favorite wallpaper. + +**Tip** + +Your web3.storage console page includes IPFS gateway links to all the content you've uploaded. + +### [Setting the filename for downloads via gateways](https://web3.storage/docs/how-tos/retrieve/#setting-the-filename-for-downloads-via-gateways) + +When downloading files from an HTTP gateway, web browsers will set the default filename for the downloaded file based on the path component of the gateway link. For example, if you use your browser's "Save link as..." feature on the following link, it should prompt you to save a file named treehouse.jpeg: + +[https://bafybeicfnbaeigdtklwkrj35r4wtfppix732zromsadvgiu33mowah74yq.ipfs.w3s.link/treehouse.jpeg](https://bafybeicfnbaeigdtklwkrj35r4wtfppix732zromsadvgiu33mowah74yq.ipfs.w3s.link/treehouse.jpeg) + +In the link above, the CID bafybeicfnbaeigdtklwkrj35r4wtfppix732zromsadvgiu33mowah74yq points to an IPFS directory listing, which maps from the filename treehouse.jpeg to the CID for the image itself. + +Since the web3.storage client wraps your uploaded files in a directory by default, this is the most common kind of gateway link you're likely to need, and your users should get nice filenames when they download their content. + +However, the behavior is a bit different if you make a gateway link directly to the image CID: + +- [https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/](https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/) +- [https://ipfs.io/ipfs/bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei](https://ipfs.io/ipfs/bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei) + +Both of the URLs above link directly to the CID of the image, without an associated filename. The first URL uses the recommended "subdomain" URL format for gateway links, while the second form uses a "path prefix" format that you may see in use elsewhere in the IPFS ecosystem. + +Depending on which style of link you use, your browser will prompt you to save a file with a generic name like download, or with the CID as the filename. + +If you have such a link, you can override the default filename by adding a query string parameter to your link of the form ?filename=\. For example, the following link will save as treehouse.jpeg, even though it links directly to the image by CID: + +[https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/?filename=treehouse.jpeg](https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/?filename=treehouse.jpeg) + +**Other gateway endpoints** + +Graph API, CAR fetching… + +## [Using the IPFS command line](https://web3.storage/docs/how-tos/retrieve/#using-the-ipfs-command-line) + +If you have the [IPFS command line interface](https://docs.ipfs.io/how-to/command-line-quick-start/) installed, you can use it directly to fetch data without going through a gateway. This also works if you've installed [IPFS Desktop](https://docs.ipfs.io/install/ipfs-desktop/), which includes the IPFS CLI. + +To get the whole bundle and save it to a directory, run the following command: + +``` +ipfs get bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu +``` + +If you want to get a specific file out of the bundle, add its name onto the end of the ipfs get bafybie... command: + +``` +ipfs get bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu/youareanonsense.jpg +``` + +## [Using curl or Powershell](https://web3.storage/docs/how-tos/retrieve/#using-curl-or-powershell) + +Sometimes you may need to just download a specific file to your computer using the command line. Unix-based operating systems, like Linux and macOS, can use curl. Windows users can use Powershell. + +- Linux +- macOS +- Windows + +1. Open a terminal window. +2. Use curl to download your file: + +``` +curl https://\.ipfs.w3s.link/\-o~/\ +``` + +1.  +Replace \, \, and \ with their respective values. + | Variable | Replace with | Example | | --- | --- | --- | | \ | The CID of the file you want to download. | bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq | | \ | The _name_ of the file that you originally uploaded to web3.storage. | example.txt | | \ | The path and filename that you want curl to save the file to. This can be different to \. | Desktop/output-file.txt | + Your complete command should look something like this: + +``` +curl https://bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq.ipfs.w3s.link/example.txt-o~/output-file.txt +``` + +## Next steps + +Next, you'll learn about how to [list](https://web3.storage/docs/how-tos/list/) From 6fe652b3cfb3e92afa7771e59f9f0ae788122327 Mon Sep 17 00:00:00 2001 From: Jay Date: Mon, 13 Nov 2023 12:10:31 +0300 Subject: [PATCH 16/25] Create upload.md --- src/pages/docs/how-tos/upload.md | 604 +++++++++++++++++++++++++++++++ 1 file changed, 604 insertions(+) create mode 100644 src/pages/docs/how-tos/upload.md diff --git a/src/pages/docs/how-tos/upload.md b/src/pages/docs/how-tos/upload.md new file mode 100644 index 0000000..534aae9 --- /dev/null +++ b/src/pages/docs/how-tos/upload.md @@ -0,0 +1,604 @@ +## Upload + +**How to upload data using web3.storage** + +In this how-to guide, you'll learn how to store data programmatically for your development projects using the web3.storage client library in JavaScript using your (developer-owned) Space. This includes various architecture options for the data pipeline for your users to upload to web3.storage, which then makes your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin. + +Later in this section, we also cover uploading data using the CLI or web console. If you just want to quickly store a few files using web3.storage rather than include upload functionality in an app or service you're building, you may want to hop down there. + +**CAUTION** + +⚠️❗ Public Data 🌎: All data uploaded to w3up is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using w3up. + +⚠️❗ Permanent Data ♾️: Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future. + +**web3.storage API authorization** + +In the previous section, you created a Space that has a unique DID. To use the client to upload data to this Space, you need to give it permission to do so. This is done by passing an Agent into the client and a UCAN token delegating permissions to this Agent. An Agent has a DID with an underlying private key (like a Space), but can be thought of as parallel to a login session (whereas the Space can be thought of more like a bucket or account). To learn more about how UCAN works and how web3.storage uses it, read our docs on UCAN authorization. + +**Using the Javascript client to upload** + +This section discusses using the web3.storage JavaScript client, w3up-client, with your (developer-owned) Space in your application. web3.storage's Javascript client provides a simple interface for storing data using syntax inspired by familiar web APIs such as [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API)and [File](https://developer.mozilla.org/en-US/docs/Web/API/File). + +``` +mermaid + +flowchart TD + +B[w3up-client instance] --\>|Automatic if specific Agent is not passed when client object created|B(Pass local Agent DID and key) + +S --\> C(Get UCAN delegation from Space to Agent) + +C --\> D(Upload to Space using Agent) + +``` + +All uses of w3up-client to upload with web3.storage follow the flow above. + +**Installing the client** + +In your JavaScript project, add the web3.storage package to your dependencies: + +``` +npminstall@web3-storage/w3up-client +``` + +**Creating a server-side client instance** + +The package provides a [static](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#create)[create](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#create)[function](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#create) that returns a [Client](https://web3-storage.github.io/w3up-client/classes/client.Client.html)[object](https://web3-storage.github.io/w3up-client/classes/client.Client.html). How you initialize it depends on the backend environment. + +**Claim delegation via email validation: For persistent backend only** + +``` +mermaid + +sequenceDiagram + +Client-\>\>web3.storage service: Here is my email address and Agent DID + +web3.storage service--\>\>Client: Please click the link to validate + +Client--\>\>web3.storage service: Email address validated + +web3.storage service-\>\>Client: Here is a UCAN delegating permission from Space DID to Agent DID + +``` + +You can use web3.storage's email authorization flow to give permissions to your server-side client. This can be good if your backend environment will be persistent (otherwise it would be prohibitive to click an email validation link every time the client is re-instantiated. + +``` +import { create } from '@web3-storage/w3up-client' + +const client = await create() +``` + +By default, clients will create a new [Agent](https://web3-storage.github.io/w3protocol/classes/_web3_storage_access.Agent.html) and put it in a persistent local [Store](https://github.com/web3-storage/w3up/tree/main/packages/access-client) if it can't find an existing one to load (so the next time the client is initialized on the same device, it will use the same Agent). + +Then you can authorize your Agent with your email address. Calling authorize will cause an email to be sent to the given address. + +``` +await client.authorize('zaphod@beeblebrox.galaxy') +``` + +Once a user clicks the confirmation link in the email, the authorize method will resolve. Make sure to check for errors, as authorize will fail if the email is not confirmed within the expiration timeout. Authorization needs to happen only once per agent. This also claims all delegations available with your email address, so from there, you can select the Space you'd like to use. + +``` +await client.setCurrentSpace(space.did()) # select the relevant Space DID that is associated with your account +``` + +**Bring your own Agent: For any backend (including non-persistent and/or serverless)** + +``` +mermaid + +sequenceDiagram + +Developer-\>\>Developer: Create Agent private key and DID + +Developer-\>\>Developer: Delegate UCAN from Space to Agent + +Developer-\>\>Client: Here is my Agent private key and UCAN delegating permissions + +``` + +An option that works for any backend environment is to define an Agent and delegate a UCAN from your Space to this Agent before initializing the client. This is especially useful if you're using the client in a serverless Node environment (e.g., Lambda). + +In your command line wherever the CLI is configured with the Space you want to use (e.g., where you created the Space): + +``` +# the following command returns what will be your Agent private key and DID +npx ucan-key ed --json + +# store the private key in environment variable KEY + +# the following command returns the UCAN that will delegate + +# make sure `w3 space use` is set to the Space you intend on using + +w3 delegation create \ | base64 + +# store the output in environment variable PROOF + +# if you want to limit permissions being passed to the Agent, you can specify which permissions to give, e.g., `--can 'store/add' --can 'upload/add'` limits to just being able to upload +``` + + +Then, when you initialize and configure the client, you can pass in this Agent and UCAN. + +``` +import { create } from '@web3-storage/w3up-client' + +import \* as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys + +async function main () { + +// Load client with specific private key + +const principal = Signer.parse(process.env.KEY) + +const client = await Client.create({ principal }) + +// Add proof that this agent has been delegated capabilities on the space + +const proof = await parseProof(process.env.PROOF) + +const space = await client.addSpace(proof) + +await client.setCurrentSpace(space.did()) + +// READY to go! + +} + +/\*\* @param {string} data Base64 encoded CAR file \*/ + +async function parseProof (data) { + +const blocks = [] + +const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) + +for await (const block of reader.blocks()) { + +blocks.push(block) + +} + +return importDAG(blocks) + +} +``` + +If you're doing this in a non-persistent or serverless backend, you might consider using an in-memory [Store](https://github.com/web3-storage/w3up/tree/main/packages/access-client) for your Agent information rather than the default on-disk: + +``` +import { StoreMemory } from '@web3-storage/access/stores/store-memory' + +const client = await Client.create({ principal, store: new MemoryStore() }) +``` + +**Uploading to web3.storage** + +Now that your backend client instance is set up with being able to interact with your Space, you're ready to upload! Call uploadFile to upload a single file, or uploadDirectory to upload multiple files. + +There are two main options to getting content into your Space: + +- Upload data to web3.storage from the backend client itself (e.g., you're storing data that your users are uploading to your backend) +- Upload data to web3.storage directly from your user's environment (like your application's user's browser) by delegating a UCAN that has permission to upload to your Space + +**Upload from backend client directly** + +``` +mermaid + +sequenceDiagram + +User-\>\>w3up-client in backend: Upload data + +w3up-client in backend-\>\>web3.storage service: Upload data + +``` + +You are already set up to upload using your client instance as data becomes available to your backend - you can call uploadFile or uploadDirectory with it. Here's a complete example of what this might look like: + +``` +import { create } from '@web3-storage/w3up-client' + +import \* as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys + +const principal = Signer.parse(process.env.KEY) // Agent private key + +const client = await create({ principal }) + +async function main () { + +// Load client with specific private key + +const principal = Signer.parse(process.env.KEY) + +const client = await Client.create({ principal }) + +// Add proof that this agent has been delegated capabilities on the space + +const proof = await parseProof(process.env.PROOF) + +const space = await client.addSpace(proof) + +await client.setCurrentSpace(space.did()) + +// READY to go! + +} + +/\*\* @param {string} data Base64 encoded CAR file \*/ + +async function parseProof (data) { + +const blocks = [] + +const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) + +for await (const block of reader.blocks()) { + +blocks.push(block) + +} + +return importDAG(blocks) + +} +``` + +**Delegate UCAN for your user to upload directly** + +``` +mermaid + +sequenceDiagram + +participant w3up-client in user + +participant w3up-client in backend + +participant web3.storage service + +w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent + +w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID + +w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID + +w3up-client in user-\>\>web3.storage service: Upload data + +``` + +Your backend instance can also be used to delegate upload permissions directly to your user to upload. The code snippet below shows an example of how you might set up a client instance in your application frontend and how it might interact with your backend client. You can see how the frontend client Agent DID is used for the backend client to delegate permissions to; from there, it will be the frontend client that will call the `upload` method. + +``` +import { CarReader } from '@ipld/car'; + +import \* as DID from '@ipld/dag-ucan/did'; + +import \* as Delegation from '@ucanto/core/delegation'; + +import { importDAG } from '@ucanto/core/delegation'; + +import \* as Signer from '@ucanto/principal/ed25519'; + +import \* as Client from '@web3-storage/w3up-client'; + +async function backend(did: string) { + +// Load client with specific private key + +const principal = Signer.parse(process.env.KEY); + +const client = await Client.create({ principal }); + +// Add proof that this agent has been delegated capabilities on the space + +const proof = await parseProof(process.env.PROOF); + +const space = await client.addSpace(proof); + +await client.setCurrentSpace(space.did()); + +// Create a delegation for a specific DID + +const audience = DID.parse(did); + +const abilities = ['store/add', 'upload/add']; + +const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24; // 24 hours from now + +const delegation = await client.createDelegation(audience, abilities, { + +expiration, + +}); + +// Serialize the delegation and send it to the client + +const archive = await delegation.archive(); + +return archive.ok; + +} + +/\*\* @param {string} data Base64 encoded CAR file \*/ + +async function parseProof(data) { + +const blocks = []; + +const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); + +for await (const block of reader.blocks()) { + +blocks.push(block); + +} + +return importDAG(blocks); + +} + +async function frontend() { + +// Create a new client + +const client = await Client.create(); + +// Fetch the delegation from the backend + +const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; // backend method is exposed at this API URL + +const response = await fetch(apiUrl); + +const data = await response.arrayBuffer(); + +// Deserialize the delegation + +const delegation = await Delegation.extract(new Uint8Array(data)); + +if (!delegation.ok) { + +throw new Error('Failed to extract delegation'); + +} + +// Add proof that this agent has been delegated capabilities on the space + +const space = await client.addSpace(delegation.ok); + +client.setCurrentSpace(space.did()); + +// READY to go! + +} +``` + +**[Preparing files and uploading](https://web3.storage/docs/how-tos/store/#preparing-files-for-upload)** + +You are now ready to upload using the client! In general, the easiest way to upload data is using the uploadFile or uploadDirectory method. + +uploadFile expects a "Blob like" input, which can be a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) when running in a browser. On node.js, see the [filesFromPath](https://github.com/web3-storage/files-from-path)[library](https://github.com/web3-storage/files-from-path), which can load compatible objects from the local filesystem. By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish. + +uploadDirectory requires File-like objects instead of Blobs, as the file's name property is used to build the directory hierarchy. + +**Tip** + +When uploading multiple files, try to give each file a unique name. All the files in a storeDirectory request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name. + +You can control the directory layout and create nested directory structures by using / delimited paths in your filenames: +``` +const files = [ + +new File(['some-file-content'], 'readme.md'), + +new File(['import foo'], 'src/main.py'), + +new File([someBinaryData], 'images/example.png'), + +] + +const directoryCid = await client.storeDirectory(files) +``` + +In the example above, directoryCid resolves to an IPFS directory with the following layout: + +``` +. + +├──images + +│└──example.png + +├──readme.md + +└──src + +└──main.py +``` + +There are a few different ways of creating File objects available, depending on your platform. + +In the browser, you can use a [file input element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/file) to allow the user to select files for upload: + +``` +functiongetFiles(){ + +constfileInput=document.querySelector('input[type="file"]') + +returnfileInput.files + +} +``` + +You can also manually create File objects using the native File constructor provided by the browser runtime. This is useful when you want to store data created by your application, instead of files from the user's computer. + +``` +functionmakeFileObjects(){ + +//YoucancreateFileobjectsfromaBlobofbinarydata + +//see:https://developer.mozilla.org/en-US/docs/Web/API/Blob + +//Herewe'rejuststoringaJSONobject,butyoucanstoreimages, + +//audio,orwhateveryouwant! + +constobj={hello:'world'} + +constblob=newBlob([JSON.stringify(obj)],{type:'application/json'}) + +constfiles=[ + +newFile(['contents-of-file-1'],'plain-utf8.txt'), + +newFile([blob],'hello.json') + +] + +returnfiles + +} +``` + +In Node.js, the web3.storage package exports some helpful utility functions from the [files-from-path](https://www.npmjs.com/package/files-from-path)[module](https://www.npmjs.com/package/files-from-path) that allow you to easily read File objects from the local file system. The getFilesFromPath helper asynchronously returns an array of Files that you can use directly with the put client method: + +``` +import{getFilesFromPath}from'web3.storage' +``` + +If you expect to be loading a lot of large files, you may be better served by the [filesFromPath](https://github.com/web3-storage/files-from-path#filesfrompath)[helper](https://github.com/web3-storage/files-from-path#filesfrompath). It reduces memory pressure by yielding File objects one by one as they're loaded from disk, instead of loading everything into memory. You can then issue multiple put requests to send each file to web3.storage. + +You can also manually create File objects by importing a Node.js implementation of File from the web3.storage package. This is useful when you want to store data created by your application, instead of files from the user's computer. + +``` +import{File}from'web3.storage' + +asyncfunctiongetFiles(path){ + +constfiles=awaitgetFilesFromPath(path) + +console.log(`read${files.length}file(s)from${path}`) + +returnfiles + +} + +functionmakeFileObjects(){ + +//YoucancreateFileobjectsfromaBufferofbinarydata + +//see:https://nodejs.org/api/buffer.html + +//Herewe'rejuststoringaJSONobject,butyoucanstoreimages, + +//audio,orwhateveryouwant! + +constobj={hello:'world'} + +constbuffer=Buffer.from(JSON.stringify(obj)) + +constfiles=[ + +newFile(['contents-of-file-1'],'plain-utf8.txt'), + +newFile([buffer],'hello.json') + +] + +returnfiles + +} +``` + +**Using the CLI or web console to upload** + +You can also use the CLI or the web3.storage web console to upload. + +**CLI** + +If you followed the Create account and Space section, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions: + +1. (If not yet installed) Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli`. +2. Run `w3 authorize [alice@example.com](mailto:alice@example.com)` in the command line using your email address. Click on the validation link sent to your email. +3. After successfully running `authorize`, your CLI Agent has been delegated access to all Spaces associated with your email address. You can see a list of these Spaces using `w3 space ls` and select the one you'd like to upload to using `w3 space use \`. + +When the right Space is selected, you are ready to upload! You can do so by running `w3 up \`. + +There are a few useful flags (check out the reference docs to see a full list): + +``` +--no-wrap //Don't wrap input files with a directory. +-H, --hidden //Include paths that start with ".". +-c, --car //File is a CAR file. +``` + +**Web console** + +You can also upload data using the web console. First, go to console.web3.storage. Enter your email if you're not yet logged in (and click on the link sent to your inbox). + + + +Once logged in, select the Space you'd like to upload to. + + + +From there, just click the Upload box and select your file or directory. + + + +**Content vs. shard CIDs** + +There are two types of content identifiers (CIDs) that web3.storage interacts with: + +- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. +- Shard CIDs: The CID of the serialized shards of data itself (CAR files) that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. + +One example of where you might see both is uploading a large file with the CLI: + + +``` +w3 up gb.file + +1 file (1GB) + +bagbaierao... + +bagbaieraq... + +bagbaieraj... + +bagbaierai... + +bagbaierax... + +bagbaieraf... + +bagbaierac... + +bagbaierax... + +bagbaierax... + +bagbaiera4... + +⁂ Stored 1 file + +⁂ https://w3s.link/ipfs/bafybeiaxwvg4... +``` + +The CLI sharded the 1GB upload into 10 shards, each with a `bag…`-prefixed CID. The content CID of the file itself is included in the `w3s.link/ipfs/bafy…` link at the bottom. + +In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs. + +However, there are some cases where interacting with shard CIDs can be useful. This requires using the lower-level methods that the client and CLI expose (`capability.*` methods in the client, and `w3 can *` methods in the CLI). You can read more about the differences in the Upload vs. Store section. + +**Next steps** + +Learn more about how to fetch your data using the CID in the next section, Retrieve. + +[https://web3.storage/docs/how-tos/retrieve/](https://web3.storage/docs/how-tos/retrieve/) From 9b2f19bc4db450cb2b907b3124ad30fafac333a6 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Mon, 13 Nov 2023 12:42:17 +0300 Subject: [PATCH 17/25] fix: comment out missing image --- src/pages/docs/how-tos/retrieve.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/pages/docs/how-tos/retrieve.md b/src/pages/docs/how-tos/retrieve.md index 0a3d2f6..c04399c 100644 --- a/src/pages/docs/how-tos/retrieve.md +++ b/src/pages/docs/how-tos/retrieve.md @@ -19,7 +19,9 @@ You can easily fetch any data stored using web3.storage using an IPFS HTTP gatew You can use an IPFS gateway to view a list of all the files in that directory from your browser. To do so, simply create a gateway URL. For example, if your CID is bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu, you can make a URL for the w3s.link gateway as follows: `https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/`. Follow that link, and you'll see a page similar to this: + If you want to link directly to a file within that directory, just add the file path after the CID portion of the link. For example: [bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg](https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg) could be used as a shareable link for your new favorite wallpaper. From 6bd2d53bc7a5f8a1c232c6190ecb2829a5e98f18 Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 10:07:48 +0000 Subject: [PATCH 18/25] chore: fix formatting in architecture-options.md License: MIT Signed-off-by: Oli Evans --- .../docs/concepts/architecture-options.md | 170 ++++++------------ 1 file changed, 51 insertions(+), 119 deletions(-) diff --git a/src/pages/docs/concepts/architecture-options.md b/src/pages/docs/concepts/architecture-options.md index 01a769e..90710b2 100644 --- a/src/pages/docs/concepts/architecture-options.md +++ b/src/pages/docs/concepts/architecture-options.md @@ -1,6 +1,6 @@ -## Architecture options +# Architecture options -## Possible architectures of using web3.storage to upload +Possible architectures of using web3.storage to upload UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate: @@ -10,21 +10,14 @@ UCAN opens up a number of options in how to integrate with w3up: Should you, the In the How-tos section of the docs, we focused on the first two options, as they are the most familiar today. However, you can implement each of these in a number of ways, but we talk through some considerations when implementing a given option. -#### [Client-server](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#client-server) - -``` -mermaid +## Client-server +```mermaid sequenceDiagram - participant User - w3up-client in backend-\>\>w3up-client in backend: Client set with Agent with delegation from Space - User-\>\>w3up-client in backend: Upload data - w3up-client in backend-\>\>web3.storage w3up service: Upload data - ``` - For your backend to be scalable, you might consider using serverless workers or a queue in front of a server @@ -33,28 +26,18 @@ w3up-client in backend-\>\>web3.storage w3up service: Upload data - If your backend is persistent, you can do this or do everything in the client directly ([create Space](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#creating-and-registering-spaces) and [get delegation](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#delegating-from-space-to-agent)) - After this, once your user uploads data to your backend, you can run any of the upload methods -#### [Delegated](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#delegated) +## Delegated ```mermaid - sequenceDiagram - participant w3up-client in user - participant w3up-client in backend - participant web3.storage w3up service - w3up-client in backend-\>\>w3up-client in backend: Client created with Agent and delegation from Space - w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent - w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID - w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID - w3up-client in user-\>\>web3.storage w3up service: Upload data - ``` - You will likely have w3up-client running in your end-user's client code, as well as backend code that's able to generate UCANs that delegate the ability to upload and pass them to your users (e.g., w3up-client running in a serverless worker) @@ -72,131 +55,80 @@ w3up-client in user-\>\>web3.storage w3up service: Upload data - Note that this alone does not give visibility into which of your end users are uploading what; to track this, you'll probably need them to send you that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking) - A code example that does this can be found below -``` -import { CarReader } from '@ipld/car'; - -import \* as DID from '@ipld/dag-ucan/did'; - -import \* as Delegation from '@ucanto/core/delegation'; - -import { importDAG } from '@ucanto/core/delegation'; - -import \* as Signer from '@ucanto/principal/ed25519'; - -import \* as Client from '@web3-storage/w3up-client'; +```js +import { CarReader } from '@ipld/car' +import * as DID from '@ipld/dag-ucan/did' +import * as Delegation from '@ucanto/core/delegation' +import { importDAG } from '@ucanto/core/delegation' +import * as Signer from '@ucanto/principal/ed25519' +import * as Client from '@web3-storage/w3up-client' async function backend(did: string) { + // Load client with specific private key + const principal = Signer.parse(process.env.KEY) + const client = await Client.create({ principal }) -// Load client with specific private key - -const principal = Signer.parse(process.env.KEY); - -const client = await Client.create({ principal }); - -// Add proof that this agent has been delegated capabilities on the space - -const proof = await parseProof(process.env.PROOF); - -const space = await client.addSpace(proof); - -await client.setCurrentSpace(space.did()); - -// Create a delegation for a specific DID - -const audience = DID.parse(did); - -const abilities = ['store/add', 'upload/add']; - -const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24; // 24 hours from now - -const delegation = await client.createDelegation(audience, abilities, { - -expiration, + // Add proof that this agent has been delegated capabilities on the space + const proof = await parseProof(process.env.PROOF) + const space = await client.addSpace(proof) + await client.setCurrentSpace(space.did()) -}); + // Create a delegation for a specific DID + const audience = DID.parse(did) + const abilities = ['store/add', 'upload/add'] + const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24 // 24 hours from now + const delegation = await client.createDelegation(audience, abilities, { expiration }) -// Serialize the delegation and send it to the client - -const archive = await delegation.archive(); - -return archive.ok; + // Serialize the delegation and send it to the client + const archive = await delegation.archive() + return archive.ok; } -/\*\* @param {string} data Base64 encoded CAR file \*/ - +/** @param {string} data Base64 encoded CAR file */ async function parseProof(data) { - -const blocks = []; - -const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); - -for await (const block of reader.blocks()) { - -blocks.push(block); - -} - -return importDAG(blocks); - + const blocks = [] + const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) + for await (const block of reader.blocks()) { + blocks.push(block) + } + return importDAG(blocks) } async function frontend() { + // Create a new client + const client = await Client.create() -// Create a new client - -const client = await Client.create(); - -// Fetch the delegation from the backend - -const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; - -const response = await fetch(apiUrl); - -const data = await response.arrayBuffer(); - -// Deserialize the delegation - -const delegation = await Delegation.extract(new Uint8Array(data)); + // Fetch the delegation from the backend + const apiUrl = `/api/w3up-delegation/${client.agent().did()}` + const response = await fetch(apiUrl) + const data = await response.arrayBuffer(); -if (!delegation.ok) { + // Deserialize the delegation + const delegation = await Delegation.extract(new Uint8Array(data)); + if (!delegation.ok) { + throw new Error('Failed to extract delegation'); + } -throw new Error('Failed to extract delegation'); - -} - -// Add proof that this agent has been delegated capabilities on the space - -const space = await client.addSpace(delegation.ok); - -client.setCurrentSpace(space.did()); - -// READY to go! + // Add proof that this agent has been delegated capabilities on the space + const space = await client.addSpace(delegation.ok); + client.setCurrentSpace(space.did()) + // READY to go! } ``` -#### [User-owned](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#user-owned) - -``` -mermaid +## User-owned +```mermaid sequenceDiagram - participant User - participant Application backend - participant web3.storage w3up service - Application backend-\>\>User: Front end code that includes w3up-client - User-\>\>web3.storage w3up service: (If needed) Create Space and register it - User-\>\>web3.storage w3up service: (If needed) Use Agent email verification to "log in" to Space - User-\>\>web3.storage w3up service: Upload data using w3up-client - ``` - If you want your user to own their own Space, you'll likely be relying on the w3up-client methods to create a Space, authorize the Space, and authorize the Agent on the end user-side; from there they can run any of the upload methods From 2c3864cb76cb5c49c1c102423fa88ffb82401d7f Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 11:09:53 +0000 Subject: [PATCH 19/25] chore: fix car and remove old examples License: MIT Signed-off-by: Oli Evans --- src/pages/docs/concepts/_meta.json | 3 +- src/pages/docs/concepts/car.md | 538 ++--------------------------- 2 files changed, 38 insertions(+), 503 deletions(-) diff --git a/src/pages/docs/concepts/_meta.json b/src/pages/docs/concepts/_meta.json index 4b03ee8..a989ebb 100644 --- a/src/pages/docs/concepts/_meta.json +++ b/src/pages/docs/concepts/_meta.json @@ -1,4 +1,5 @@ { "ucan": "UCAN", - "did": "DID" + "did": "DID", + "car": "CAR" } \ No newline at end of file diff --git a/src/pages/docs/concepts/car.md b/src/pages/docs/concepts/car.md index 03b0b01..ebf7263 100644 --- a/src/pages/docs/concepts/car.md +++ b/src/pages/docs/concepts/car.md @@ -1,6 +1,4 @@ -## Content Archive (CAR) files - -Working with Content Archives (CAR) files +# Content Archive (CAR) files When you upload files to web3.storage using the [client library](https://web3.storage/docs/reference/js-client-library/), your data is converted into a graph of data structures, which are then packed into a format called a Content Archive (CAR) before being sent to the web3.storage service. @@ -14,7 +12,7 @@ The type of data stored in CARs is defined by [IPLD](https://ipld.io/), or Inter IPFS files are one example of IPLD data, but IPLD can also be used to access data from Ethereum, Git, and other hash-addressed systems. You can also use IPLD as a general purpose format for your structured data, sort of like a Web3-flavored JSON. See Advanced IPLD formats below for more information. -## [CARs and web3.storage](https://web3.storage/docs/how-tos/work-with-car-files/#cars-and-web3storage) +## CARs and web3.storage When the web3.storage client and CLI pack up regular files into a CAR to store on IPFS, the CAR contains data encoded in the same format used by IPFS when importing files using the command line or other IPFS APIs. @@ -26,35 +24,37 @@ Another reason to use CARs is to support large files, which would otherwise hit CAR files are a format that pretty much any IPFS tool or implementation can interact with. You can export data from your personal IPFS node into a CAR file and upload it to web3.storage using `w3 up --car` or `client.uploadCar`. As a result, we dive into the various ways you might interact with CAR files. -## [Command line tools](https://web3.storage/docs/how-tos/work-with-car-files/#command-line-tools) +## Command line tools There are a few ways to create and interact with CAR files from the command line. -### [ipfs-car](https://web3.storage/docs/how-tos/work-with-car-files/#ipfs-car) +### ipfs-car The [ipfs-car](https://github.com/web3-storage/ipfs-car) JavaScript package includes a command-line tool for easily creating, unpacking, and verifying CAR files. -To install it, you'll need [Node.js](https://nodejs.org/) - we recommend the latest stable version. - You can install the command globally: -```npm install -g ipfs-car``` - -Or run the command with npx without installing it to your PATH: - -```npx ipfs-car --help``` +```sh +npm install -g ipfs-car +``` The --pack flag will create a new CAR file from a collection of input files: -```ipfs-car --packpath/to/files --outputpath/to/write/a.car``` +```sh +ipfs-car --pack path/to/files --output path/to/write/a.car +``` -Or extract files from a CAR with --unpack: +Extract files from a CAR with --unpack: -```ipfs-car --unpackpath/to/my.car --output/path/to/unpack/files/to``` +```sh +ipfs-car --unpack path/to/my.car --output path/to/unpack/files/to +``` -You can also list the contents of a CAR with --list: +List the contents of a CAR with --list: -```ipfs-car --listpath/to/my.car``` +```sh +ipfs-car --list path/to/my.car +``` For more usage information, run ipfs-car --help. @@ -66,7 +66,7 @@ The [ipfs dag export](https://docs.ipfs.io/reference/cli/#ipfs-dag-export) comma To create a CAR file using go-ipfs, you can redirect the output of ipfs dag export to a file: -``` +```sh cid="bafybeigdmvh2wgmryq5ovlfu4bd3yiljokhzdep7abpe4c4lrf6rukkx4m" ipfs dag export $cid > path/to/output.car ``` @@ -77,7 +77,9 @@ If you don't have the CID in your local IPFS repository, the dag export command To add the contents of a CAR file to your local IPFS repository, you can use ipfs dag import: -```ipfs dag import path/to/input.car``` +```sh +ipfs dag import path/to/input.car +``` ## Javascript libraries @@ -95,23 +97,16 @@ The [@ipld/car](https://github.com/ipld/js-car)[package](https://github.com/ipld Here's a simple example of loading a CAR file from a Node.js stream and storing it with web3.storage: -``` -import { reateReadStream } from 'fs'; - -import { CarReader } from '@ipld/car'; +```js +import { createReadStream } from 'fs' +import { CarReader } from '@ipld/car' async function storeCarFile(filename) { - -const inStream = createReadStream(filename); - -const car = await CarReader.fromIterable(inStream); - -const client = makeStorageClient(); - -const cid = await client.putCar(car); - -console.log('Stored CAR file! CID:',cid); - + const inStream = createReadStream(filename) + const car = await CarReader.fromIterable(inStream) + const client = makeStorageClient() + const cid = await client.uploadCar(car) + console.log('Stored CAR file! CID:',cid) } ``` @@ -123,478 +118,17 @@ The CarReader type shown above will read the entire contents of the CAR into mem IPLD can also be used as a general purpose data format like JSON. In fact, you can use JSON directly as IPLD just by using a special convention for linking to other IPLD objects. This convention is defined in the [dag-json](https://ipld.io/docs/codecs/known/dag-json/)["codec"](https://ipld.io/docs/codecs/known/dag-json/). -Here's an example of a dag-json object: +Here's an example of a `dag-json` object: -``` +```json { - -"name": "Have you seen this dog?", - -"description": "I have now...", - -"image":{"/":"bafybeihkqv2ukwgpgzkwsuz7whmvneztvxglkljbs3zosewgku2cfluvba"} - + "name": "Have you seen this dog?", + "description": "I have now...", + "image":{"/":"bafybeihkqv2ukwgpgzkwsuz7whmvneztvxglkljbs3zosewgku2cfluvba"} } ``` + The image field uses the special "link type" to reference another IPLD object. The link is just a regular JSON object with a single key named /, whose value is a Content Identifier. Although dag-json is familiar and easy to use, we recommend using the similar [dag-cbor](https://ipld.io/docs/codecs/known/dag-cbor/)[codec](https://ipld.io/docs/codecs/known/dag-cbor/) instead. dag-cbor uses the [Concise Binary Object Representation](https://cbor.io/) to more efficiently encode data, especially binary data which must be Base64-encoded when using dag-json. -### [Examples](https://web3.storage/docs/how-tos/work-with-car-files/#examples) - -Below are some examples of working with dag-cbor data and sending it to web3.storage. - -First, you'll need to import some things: - -``` -import {Web3Storage} from 'web3.storage' - -import {CarReader} from '@ipld/car' - -import {encode} from 'multiformats/block' - -import *as cbor from '@ipld/dag-cbor' - -import {sha256} from 'multiformats/hashes/sha2' - -//#end region cbor Link Example - -//#region make UnixFs File - -import {importer} from 'ipfs-unixfs-importer' - -import {MemoryBlockStore} from 'ipfs-car/blockstore/memory' -``` - -Now we'll define a convenience function to encode an IPLD block of CBOR data and hash with SHA2-256: - -``` -encodeCborBlock(value) - -async function encodeCborBlock (value) { - return encode({ value, codec: cbor, hasher: sha256 }) -} -``` - -And a function to make a CAR from a collection of blocks and a root CID: - -``` -makeCar(rootCID, ipldBlocks) - -asyncfunction makeCar (rootCID,ipldBlocks){ - -return new CarReader (1,[rootCID],ipldBlocks) - -} -``` - -#### Storing simple CBOR data - -Using the helpers above, you can make a CAR file with a single block of simple CBOR data and send it to web3.storage: - -``` -simpleCborExample() - -async function simpleCborExample(){ - -//encode the value into an IPLD block and store with web3.storage - -const block = await encodeCborBlock({hello:'world'}) - -const car = await makeCar(block.cid,[block]) - -//upload to web3.storage using put Car - -const client = new Web3Storage({token:process.env.WEB3STORAGE\_TOKEN}) - -console.log('🤖Storing simple CBOR object...') - -const cid = await client.putCar(car) - -console.log(`🎉Done storing simple CBOR object.CID:${cid}`) - -console.log(`💡If you have ipfs installed, try:ipfsdagget${cid}\n`) - -} -``` - - -If you have the IPFS command line app installed, you can view the object you stored with the [ipfs dag get](https://docs.ipfs.io/reference/cli/#ipfs-dag-get)[command](https://docs.ipfs.io/reference/cli/#ipfs-dag-get), for example: - -``` -ipfs dag get bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae - -{ "hello" : "world"} -``` - -Note that the example output has been indented with [jq](https://stedolan.github.io/jq/) for clarity.The real command will output a compact dag-json representation of the CBOR data without any extra whitespace. - -#### CBOR with IPLD links - -You can link from one CBOR object to another using CIDs: - -``` -cborLinkExample() - -async function cborLinkExample(){ - -//Encode a simple object to get its CID - -const addressBlock = await encodeCborBlock({email:'zaphod@beeblebrox.galaxy'}) - -//Now we can use the CID to link to the object from another object - -const personBlock = await encodeCborBlock({ - -title:'Galactic President', - -description:'Just this guy, you know?', - -contact:addressBlock.cid - -}) - -//pack everything into a CAR - -const car = await makeCar(personBlock.cid,[personBlock,addressBlock]) - -//upload to web3.storage using putCar - -const client = new Web3Storage({token:process.env.WEB3STORAGE\_TOKEN}) - -console.log('🤖 Storing CBOR objects with CID links between them...') - -const cid = await client.putCar(car) - -console.log('🎉Stored linked data using dag-cbor.RootCID:',cid) - -console.log(`💡If you have ipfs installed, try: ipfsdagget${cid}`) - -console.log(`🔗You can also traverse the link by path:ipfsdagget${cid}/contact\n`) - -} - -async function make UnixFsFile(source){ - -const blockstore = new MemoryBlockStore() - -//taken from https://github.com/web3-storage/ipfs-car/blob/main/src/pack/constants.ts - -//but with wrapWithDirectory overriden to false - -const unixFs Options={ - -cidVersion:1, - -chunker:'fixed', - -maxChunkSize:262144, - -hasher:sha256, - -rawLeaves:true, - -wrapWithDirectory:false, - -maxChildrenPerNode:174 - -} - -const importStream = await importer(source,blockstore,unixFsOptions) - -let root = null - -for await(const entry of importStream){ - -root = entry - -} - -const blocks=[] - -for await(const block of blockstore.blocks()){ - -blocks.push(block) - -} - -await blockstore.close() - -return{root,blocks} - -} - -//#end region make UnixFs File - -//#region cbor Link To File Example - -async function cborLinkToFileExample(){ - -const source=[{ - -path: 'example.txt', - -content: new TextEncoder().encode('Some plain text, encoded to UTF-8') - -}] - -const {root,blocks} = await makeUnixFsFile(source) - -const cborBlock = await encodeCborBlock({ - -description:'A CBOR object that references a UnixFS file object by CID', - -file:root.cid - -}) - -blocks.push(cborBlock) - -const car = await makeCar(cborBlock.cid,blocks) - -const client = new Web3Storage({token:process.env.WEB3STORAGE_TOKEN}) - -console.log('🤖 Storing a CBOR object that links to a UnixFS file by CID...') - -const cid = await client.putCar(car) - -console.log('🎉Stored dag-cbor object that links to a unixfs file.RootCID:',cid) - -console.log(`💡If you have ipfs installed, try: ipfsdagget${cid}`) - -console.log(`💾You can view the linked file with ipfs: ipfscat${cid}/file`) - -console.log('🔗View linked file via IPFS gateway:',`https://${cid}.ipfs.dweb.link/file`) - -} - -//#end region cbor Link To File Example - -simpleCborExample() - -.then(cborLinkExample) - -.then(cborLinkToFileExample) - -.catch(console.error) -``` - -As with simple objects, you can use ipfs dag get to show the outer object: - -```ipfs dag get bafyreieq6bftbe3o46lrdbzj6vrvyee4njfschajxgmpxwbqex3czifhry``` - - -``` -{ - -"contact":{ - -"/":"bafyreicp2g6ez5exmw5uxsns7kkwtxr5z4vyx4xkdci6xpy2vou3zqc6me" - -}, - -"description" : "Just this guy, you know?", - -"title":"Galactic President" - -} -``` - -The contact field above contains an IPLD link, which can be included in the ipfs dag get command to resolve the linked object: - -```ipfs dag get bafyreieq6bftbe3o46lrdbzj6vrvyee4njfschajxgmpxwbqex3czifhry/contact``` - -```{"email":"zaphod@beeblebrox.galaxy"}``` - -#### Linking from CBOR to an IPFS file - -Our final example is a little more complex. We're going to store a file in the same UnixFS format that IPFS uses, and link to it from a CBOR object. - -First, we'll encode a file into UnixFS format. Normally, this is done by the client library, but we want to get the CID of the file object to use for our link before we send the file off to web3.storage, so we'll construct the UnixFS object ourselves. - -Here's a helper function to make a UnixFS file and encode it to an IPLD block: - -``` -makeUnixFsFile(source) - -import { importer } from 'ipfs-unixfs-importer' -import { MemoryBlockStore } from 'ipfs-car/blockstore/memory' -// #endregion imports - -// #region encodeCborBlock -async function encodeCborBlock (value) { - return encode({ value, codec: cbor, hasher: sha256 }) -} -// #endregion encodeCborBlock - -// #region makeCar -async function makeCar (rootCID, ipldBlocks) { - return new CarReader(1, [rootCID], ipldBlocks) -} -// #endregion makeCar - -// #region simpleCborExample -async function simpleCborExample () { - // encode the value into an IPLD block and store with web3.storage - const block = await encodeCborBlock({ hello: 'world' }) - const car = await makeCar(block.cid, [block]) - - // upload to web3.storage using putCar - const client = new Web3Storage({ token: process.env.WEB3STORAGE_TOKEN }) - console.log('🤖 Storing simple CBOR object...') - const cid = await client.putCar(car) - console.log(`🎉 Done storing simple CBOR object. CID: ${cid}`) - console.log(`💡 If you have ipfs installed, try: ipfs dag get ${cid}\n`) -} -// #endregion simpleCborExample - -// #region cborLinkExample -async function cborLinkExample () { - // Encode a simple object to get its CID - const addressBlock = await encodeCborBlock({ email: 'zaphod@beeblebrox.galaxy' }) - - // Now we can use the CID to link to the object from another object - const personBlock = await encodeCborBlock({ - title: 'Galactic President', - description: 'Just this guy, you know?', - contact: addressBlock.cid - }) - - // pack everything into a CAR - const car = await makeCar(personBlock.cid, [personBlock, addressBlock]) - - // upload to web3.storage using putCar - const client = new Web3Storage({ token: process.env.WEB3STORAGE_TOKEN }) - - console.log('🤖 Storing CBOR objects with CID links between them...') - const cid = await client.putCar(car) - console.log('🎉 Stored linked data using dag-cbor. Root CID:', cid) - console.log(`💡 If you have ipfs installed, try: ipfs dag get ${cid}`) - console.log(`🔗 You can also traverse the link by path: ipfs dag get ${cid}/contact\n`) -} -async function makeUnixFsFile (source) { - const blockstore = new MemoryBlockStore() - // taken from https://github.com/web3-storage/ipfs-car/blob/main/src/pack/constants.ts - // but with wrapWithDirectory overriden to false - const unixFsOptions = { - cidVersion: 1, - chunker: 'fixed', - maxChunkSize: 262144, - hasher: sha256, - rawLeaves: true, - wrapWithDirectory: false, - maxChildrenPerNode: 174 - } - const importStream = await importer(source, blockstore, unixFsOptions) - let root = null - for await (const entry of importStream) { - root = entry - } - const blocks = [] - for await (const block of blockstore.blocks()) { - blocks.push(block) - } - await blockstore.close() - return { root, blocks } -} -``` -The helper returns a root block, which we can link to by CID, as well as a blocks array containing the encoded file data. When we create the CAR to send to web3.storage, it's important to include all the file blocks as well as the CBOR block. - -``` -cborLinkToFileExample() - -asyncfunctioncborLinkToFileExample(){ - -constsource=[{ - -path:'example.txt', - -content:newTextEncoder().encode('Someplaintext,encodedtoUTF-8') - -}] - -const{root,blocks}=awaitmakeUnixFsFile(source) - -constcborBlock=awaitencodeCborBlock({ - -description:'ACBORobjectthatreferencesaUnixFSfileobjectbyCID', - -file:root.cid - -}) - -blocks.push(cborBlock) - -constcar=awaitmakeCar(cborBlock.cid,blocks) - -constclient=newWeb3Storage({token:process.env.WEB3STORAGE\_TOKEN}) - -console.log('🤖StoringaCBORobjectthatlinkstoaUnixFSfilebyCID...') - -constcid=awaitclient.putCar(car) - -console.log('🎉Storeddag-cborobjectthatlinkstoaunixfsfile.RootCID:',cid) - -console.log(`💡Ifyouhaveipfsinstalled,try:ipfsdagget${cid}`) - -console.log(`💾Youcanviewthelinkedfilewithipfs:ipfscat${cid}/file`) - -console.log('🔗ViewlinkedfileviaIPFSgateway:',`https://${cid}.ipfs.dweb.link/file`) - -} -``` - -As before, we can view the root block with ipfs dag get: - -``` -ipfs dag get bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy - -{ - -"description" : "A CBOR object that references a UnixFS file object by CID", - -"file":{ - -"/":"bafkreihmlglmfpadbk4fy72ljniveedbqicysoe5zhqqkgkuso3e6xyns4" - -} - -} -``` - -Since the file data is plain text, you can use ipfs dag get to fetch its contents: - -``` -ipfs dag get bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy/file - -"Some plain text, encoded to UTF-8" -``` - -Notice that the file content is wrapped in quotes because dag get is interpreting the content as a JSON string. - -To avoid this, or to fetch binary files, you can use ipfs get to download the file: - -``` -ipfs get bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy/file - - -Saving file(s) to file - -33B/33B[===============================================================]100.00%0s -``` - -Note that the IPFS HTTP gateway currently does not support rendering CBOR data, so the root object is not directly viewable via the gateway. See the note about gateway support below for more information. - -However, the gateway _can_ traverse the IPLD links inside our CBOR object, so you can link to the file by path and the gateway will resolve the linked file. For example: - -[https://bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy.ipfs.dweb.link/file](https://bafyreid7hvce4pzcy56s4hwu7xrt3dnnzzfvilzfwsadvf6q4eqild6ndy.ipfs.dweb.link/file). - -##### Gateway support - -Although web3.storage supports storing CAR files with dag-cbor content by default and can accept other codecs with the decoders option, the IPFS HTTP gateway does not currently "speak" these formats and will not return such data over HTTP. Please follow [this issue](https://github.com/ipfs/go-ipfs/issues/8234) to track the development of this feature. - -### [Enabling IPLD codecs in the client library](https://web3.storage/docs/how-tos/work-with-car-files/#enabling-ipld-codecs-in-the-client-library) - -By default, the client's [putCar](https://web3.storage/docs/reference/js-client-library/#store-car-files)[method](https://web3.storage/docs/reference/js-client-library/#store-car-files) will accept data encoded using the dag-pb, dag-cbor, or raw codecs. If you want to use another codec like dag-json, you must include the codec in the decoders option to putCar. - -See the [putCar](https://web3.storage/docs/reference/js-client-library/#parameters-5)[parameter reference](https://web3.storage/docs/reference/js-client-library/#parameters-5) for more details and an example that uses dag-json. From b4283b9bb1974d8190f38d56167b7c2d58a05535 Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 13:57:36 +0000 Subject: [PATCH 20/25] chore: many fixes License: MIT Signed-off-by: Oli Evans --- src/pages/_meta.json | 2 +- src/pages/docs/_meta.json | 8 +- src/pages/docs/concepts/_meta.json | 16 +- .../docs/concepts/architecture-options.md | 4 +- src/pages/docs/concepts/content-addressing.md | 14 +- src/pages/docs/concepts/filecoin-storage.md | 6 +- src/pages/docs/concepts/ipfs-gateways.md | 10 +- .../{index.md => ucans-and-web3storage.md} | 152 ++--- src/pages/docs/concepts/upload-vs-store.md | 40 +- src/pages/docs/how-tos/_meta.json | 9 + .../how-tos/{index.md => create-account.md} | 9 +- src/pages/docs/how-tos/upload.md | 560 +++++------------- src/pages/docs/quickstart.md | 95 +++ src/pages/docs/w3up-client.md | 6 +- 14 files changed, 373 insertions(+), 558 deletions(-) rename src/pages/docs/concepts/{index.md => ucans-and-web3storage.md} (69%) create mode 100644 src/pages/docs/how-tos/_meta.json rename src/pages/docs/how-tos/{index.md => create-account.md} (91%) create mode 100644 src/pages/docs/quickstart.md diff --git a/src/pages/_meta.json b/src/pages/_meta.json index 7c46b9b..72ece55 100644 --- a/src/pages/_meta.json +++ b/src/pages/_meta.json @@ -1,6 +1,6 @@ { "docs": { - "title": "🏠", + "title": "Docs", "type": "page" } } \ No newline at end of file diff --git a/src/pages/docs/_meta.json b/src/pages/docs/_meta.json index 0d0d10c..5231dfa 100644 --- a/src/pages/docs/_meta.json +++ b/src/pages/docs/_meta.json @@ -1,6 +1,8 @@ { - "index": "Start", + "index": "Welcome", + "quickstart": "Quickstart", + "how-tos": "How to", + "concepts": "Concepts", "w3cli": "Command line", - "w3up-client": "JS Client", - "concepts": "Concepts" + "w3up-client": "JS Client" } \ No newline at end of file diff --git a/src/pages/docs/concepts/_meta.json b/src/pages/docs/concepts/_meta.json index a989ebb..655b93d 100644 --- a/src/pages/docs/concepts/_meta.json +++ b/src/pages/docs/concepts/_meta.json @@ -1,5 +1,15 @@ { - "ucan": "UCAN", - "did": "DID", - "car": "CAR" + "ucans-and-web3storage": "UCANs and web3.storage", + "content-addressing": "Content-addressing", + "upload-vs-store": "Upload vs. Store", + "architecture-options": "Architecture options", + "car": "Content Archive (CAR) files", + "filecoin-storage": "Filecoin", + "ipfs-gateways": "IPFS Gateways", + "did": { + "display": "hidden" + }, + "ucan": { + "display": "hidden" + } } \ No newline at end of file diff --git a/src/pages/docs/concepts/architecture-options.md b/src/pages/docs/concepts/architecture-options.md index 90710b2..79b3f56 100644 --- a/src/pages/docs/concepts/architecture-options.md +++ b/src/pages/docs/concepts/architecture-options.md @@ -1,12 +1,10 @@ # Architecture options -Possible architectures of using web3.storage to upload - UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate: - Client-server: You (the developer) own the Space, and your user uploads to your backend infra before you upload it to the service - Delegated: You own the Space, but you give a delegated UCAN token to your user's Agent to upload directly to the service (rather than needing to touch the upload in your backend) -- (Most complex) User-owned: Your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it +- User-owned: Your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it In the How-tos section of the docs, we focused on the first two options, as they are the most familiar today. However, you can implement each of these in a number of ways, but we talk through some considerations when implementing a given option. diff --git a/src/pages/docs/concepts/content-addressing.md b/src/pages/docs/concepts/content-addressing.md index e64dc83..050e86e 100644 --- a/src/pages/docs/concepts/content-addressing.md +++ b/src/pages/docs/concepts/content-addressing.md @@ -1,16 +1,14 @@ -## Content addressing - -Content addressing in brief +# Content addressing web3.storage's decentralized file storage relies on _content addressing_ to find, reference, and retrieve your files on the network. Content addressing is a technique for organizing and locating data in a system in which the key used to locate content is derived from the content itself, rather than its location. While you don't need to understand content addressing to be able to incorporate web3.storage in your apps and services, if you're curious about what's going on under the hood, read on. -### [The basic problem](https://web3.storage/docs/concepts/content-addressing/#the-basic-problem) +## The basic problem Consider what happens when you resolve a link like web3.storage/docs/concepts/content-addressing. First, your operating system queries a global shared key-value store, split into many domains — you may know this as the Domain Name System (DNS). The DNS returns an IP address that your network card can use to send HTTP requests over the network, where this site's naming conventions turn the key /concepts/content-addressing into a response payload. The problem is, components of an address like web3.storage/docs/concepts/content-addressing are _mutable_, meaning they can change over time. In the context of the web, where _everything_ is mutable and dynamic, this is just the way it's always been. As a result, [link rot](https://en.wikipedia-on-ipfs.org/wiki/Link_rot) is just something we've all learned to live with. -### [CIDs: Location-independent, globally unique keys](https://web3.storage/docs/concepts/content-addressing/#cids-location-independent-globally-unique-keys) +## CIDs: Location-independent, globally unique keys However, thanks to content addressing, link rot may become a thing of the past. A content-addressed system such as web3.storage is like our key-value-based DNS, with one significant difference: You no longer get to choose the keys. Instead, the keys are derived directly from the file contents using an algorithm that will always generate the same key for the same content. @@ -18,7 +16,7 @@ As a result, we no longer need to coordinate among multiple writers to our store This type of key is called a _content identifier (CID)_. Once you know the CID of a file on the web3.storage network, you have all you need for the network to locate and return the file back to you. -[web3.storage CIDs under the hood](https://web3.storage/docs/concepts/content-addressing/#web3storage-cids-under-the-hood) +## web3.storage CIDs under the hood web3.storage uses CIDs to make its decentralized file storage work, with help from [IPFS](https://ipfs.io/) for locating files and making sure they're always available. @@ -33,14 +31,14 @@ There are two types of CIDs that web3.storage interacts with: In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs. -### [Summary](https://web3.storage/docs/concepts/content-addressing/#summary) +## Summary Using content addressing for locating files rather than the legacy web's method of location-dependent addressing responds to several critical weaknesses of the legacy web: - Content addressing solves for the problem behind link rot — the mutability of location-dependent storage systems — by using a hashing algorithm to generate a unique CID for each file that can be used as the lookup key for a file rather than a URL. - In addition to making sure files don't get lost if they're moved, content addressing also ensures that users intending to retrieve a specific version of a file will be guaranteed to retrieve that version for as long as it exists anywhere on the network. -### [Learn more](https://web3.storage/docs/concepts/content-addressing/#learn-more) +## Learn more Want a deep dive into content addressing, how it works, and why it's important? Check out [ProtoSchool](https://proto.school/content-addressing/) for an in-depth look at content addressing on the decentralized web, plus a wealth of other interactive tutorials on DWeb concepts, protocols, and tools. diff --git a/src/pages/docs/concepts/filecoin-storage.md b/src/pages/docs/concepts/filecoin-storage.md index e95a336..5ad1bf4 100644 --- a/src/pages/docs/concepts/filecoin-storage.md +++ b/src/pages/docs/concepts/filecoin-storage.md @@ -1,8 +1,8 @@ -## Filecoin storage +# Filecoin storage -### Filecoin storage +The Filecoin network is a great building block for any decentralized storage system. Independent storage providers periodically must cryptographically prove that they are physically storing your specific data for a specific duration of time. When they submit these proofs to the network, other nodes verify these proofs, and this is what ends up on the Filecoin blockchain. So, anyone at any given moment can trustlessly verify that specific content is persisted, the number of copies on the network, and with who they are stored with. -The Filecoin network is a great building block for any decentralized storage system. Independent storage providers periodically must cryptographically prove that they are physically storing your specific data for a specific duration of time. When they submit these proofs to the network, other nodes verify these proofs, and this is what ends up on the Filecoin blockchain. So, anyone at any given moment can trustlessly verify that specific content is persisted, the number of copies on the network, and with who they are stored with. Read more about Filecoin proofs [here](https://filecoin.io/blog/posts/what-sets-us-apart-filecoin-s-proof-system/). +Read more about Filecoin proofs [here](https://filecoin.io/blog/posts/what-sets-us-apart-filecoin-s-proof-system/). Because of the open nature of the Filecoin network allowing anyone to participate, it's very inexpensive to store data on the Filecoin network. As a result, web3.storage uses the network today to back up all its data on the decentralized web. In the future, we will launch new products that allow users to take advantage of Filecoin for "colder," disruptively expensive data storage. diff --git a/src/pages/docs/concepts/ipfs-gateways.md b/src/pages/docs/concepts/ipfs-gateways.md index 32151e5..b96c2e5 100644 --- a/src/pages/docs/concepts/ipfs-gateways.md +++ b/src/pages/docs/concepts/ipfs-gateways.md @@ -1,6 +1,4 @@ -## IPFS Gateways - -### IPFS HTTP Gateways +# IPFS Gateways web3.storage uses the [InterPlanetary File System (IPFS)](https://ipfs.io/) as a key part of its storage and retrieval infrastructure. @@ -12,7 +10,7 @@ As more browsers like [Brave](https://brave.com/ipfs-support/) and [Opera](https For more information about fetching content that you uploaded through an IPFS HTTP gateway, see the Retrieve section. -## [Types of gateway](https://web3.storage/docs/concepts/w3link/#types-of-gateway) +## Types of gateway The official [IPFS documentation on gateways](https://docs.ipfs.io/concepts/ipfs-gateway/) is helpful for understanding the types of gateways in the IPFS ecosystem and how they're used. @@ -20,7 +18,7 @@ One of the key things to understand for our purposes is the different [resolutio If you check the [list of public gateways](https://ipfs.github.io/public-gateway-checker/), you'll see that some support "subdomain" style URLs, while others only support path-style URLs. Below is a short refresher on the distinction between the two styles. -### [Path style URLs](https://web3.storage/docs/concepts/w3link/#path-style-urls) +### Path style URLs A "path style" URL puts the IPFS CID into the path portion of the gateway URL, like this: @@ -30,7 +28,7 @@ If the CID points to a directory listing, you can append the name of a file with [https://w3s.link/ipfs/bafybeid4gwmvbza257a7rx52bheeplwlaogshu4rgse3eaudfkfm7tx2my/hi-gateway.txt](https://w3s.link/ipfs/bafybeid4gwmvbza257a7rx52bheeplwlaogshu4rgse3eaudfkfm7tx2my/hi-gateway.txt) -### [Subdomain style URLs](https://web3.storage/docs/concepts/w3link/#subdomain-style-urls) +### Subdomain style URLs A "subdomain style" gateway URL puts the CID into the host portion of the URL, as a subdomain of the gateway host, like this: diff --git a/src/pages/docs/concepts/index.md b/src/pages/docs/concepts/ucans-and-web3storage.md similarity index 69% rename from src/pages/docs/concepts/index.md rename to src/pages/docs/concepts/ucans-and-web3storage.md index 4e21fe0..da0252f 100644 --- a/src/pages/docs/concepts/index.md +++ b/src/pages/docs/concepts/ucans-and-web3storage.md @@ -1,20 +1,16 @@ -## Concepts - -## UCANs and web3.storage - -How web3.storage uses UCAN +# UCANs and web3.storage For authorization, w3up services use [ucanto](https://github.com/web3-storage/ucanto), a Remote Procedure Call (RPC) framework built around [UCAN](https://ucan.xzy/), or User Controlled Authorization Networks. UCANs are a powerful capability-based authorization system that allows fine-grained sharing of permissions through a process called _delegation_ on top of [public key cryptography](https://en.wikipedia.org/wiki/Public-key_cryptography). You can think about UCAN replacing bearer tokens in traditional APIs for authorization with w3up. Since any actor can be represented by a cryptographic keypair and permissions can be delegated to them, users can interact with w3up directly in cases where a developer might have needed to previously run additional back-end infrastructure to keep API keys secure. This can be extended even to have end users using applications integrated with w3up using their own keypair-based identity. -### How w3up and w3up-client use UCANs +## How w3up and w3up-client use UCANs Our client and CLI use ucanto to take care of the details of UCANs for you, but a few of the underlying terms and concepts may "bubble up" to the surface of the API, so we'll cover the basics. We'll also go over some terms that are specific to web3.storage that you might not have encountered elsewhere. UCAN-based APIs are centered around _capabilities_, which are comprised of an _ability_ and a _resource_. Together, the ability and resource determine what action a client can perform and what objects in the system can be acted upon. When invoking a service method, a client will present a UCAN token that includes an ability and resource, along with _proofs_ that verify that they should be allowed to exercise the capability. The proof might be signed directly by the capability owner, or have a chain of signatures (_delegations_) where the actor invoking the capability has been verifiably delegated permission to do so. -#### [Space](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#space) +### Space When you upload data to w3up, your uploads are linked to a unique _Space_ that acts as a "namespace" for the data you upload. Each Space corresponds to a _DID_, or [Decentralized Identity Document](https://www.w3.org/TR/did-core/). In web3.storage's implementation of w3up, these Space DIDs generally use the key DID method, of the form did:key:publicKey with a corresponding private signing key. @@ -22,117 +18,77 @@ When creating a Space, it generates this private key and did:key for you locally Under the hood in the email registration process, your Space delegates the capabilities needed to use w3up to your email address, and this delegation is stored by web3.storage. If you need access to your Space in the future from any device, web3.storage allows you to reclaim those capabilities the same way you would reset a password in other services - using an email verification process. This means you don't need to store or manage Space private keys to use w3up - just create a new space, register it with w3up and use it from as many devices as you like. More on this "sign in" process is detailed in the next section on Agents. -#### Agent +### Agent To invoke a capability like store/add on a Space using the client or CLI, the client must have an _Agent_. Like a Space, an Agent corresponds to a did:key whose private key is generated locally. An Agent is useful once the client or CLI has a UCAN delegation where a registered Space(s) delegates the Agent its capabilities. (An imperfect analogy is Agent to login session.) The delegation from a Space to your Agent that w3up-client needs can be passed either by verifying the email address the Space is registered to and claiming the UCAN delegation (authorize(email) then capability.access.claim) or directly if you have the UCAN delegation available locally (addSpace(delegation)). -#### Delegation to other actors +### Delegation to other actors Just like Spaces can delegate permissions to Agents you own, you can also delegate permissions to other actors' Agents. One common application of this could be you delegating permission to upload to your Space to your users. Here's a code snippet demonstrating this from the Upload section: -``` +```javascript import { CarReader } from '@ipld/car'; - -import \* as DID from '@ipld/dag-ucan/did'; - -import \* as Delegation from '@ucanto/core/delegation'; - +import * as DID from '@ipld/dag-ucan/did'; +import * as Delegation from '@ucanto/core/delegation'; import { importDAG } from '@ucanto/core/delegation'; - -import \* as Signer from '@ucanto/principal/ed25519'; - -import \* as Client from '@web3-storage/w3up-client'; +import * as Signer from '@ucanto/principal/ed25519'; +import * as Client from '@web3-storage/w3up-client'; async function backend(did: string) { - -// Load client with specific private key - -const principal = Signer.parse(process.env.KEY); - -const client = await Client.create({ principal }); - -// Add proof that this agent has been delegated capabilities on the space - -const proof = await parseProof(process.env.PROOF); - -const space = await client.addSpace(proof); - -await client.setCurrentSpace(space.did()); - -// Create a delegation for a specific DID - -const audience = DID.parse(did); - -const abilities = ['store/add', 'upload/add']; - -const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24; // 24 hours from now - -const delegation = await client.createDelegation(audience, abilities, { - -expiration, - -}); - -// Serialize the delegation and send it to the client - -const archive = await delegation.archive(); - -return archive.ok; - + // Load client with specific private key + const principal = Signer.parse(process.env.KEY); + const client = await Client.create({ principal }); + + // Add proof that this agent has been delegated capabilities on the space + const proof = await parseProof(process.env.PROOF); + const space = await client.addSpace(proof); + await client.setCurrentSpace(space.did()); + + // Create a delegation for a specific DID + const audience = DID.parse(did); + const abilities = ['store/add', 'upload/add']; + const expiration = Math.floor(Date.now() / 1000) + 60 * 60 * 24; // 24 hours from now + const delegation = await client.createDelegation(audience, abilities, { + expiration, + }); + + // Serialize the delegation and send it to the client + const archive = await delegation.archive(); + return archive.ok; } -/\*\* @param {string} data Base64 encoded CAR file \*/ - +/** @param {string} data Base64 encoded CAR file */ async function parseProof(data) { - -const blocks = []; - -const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); - -for await (const block of reader.blocks()) { - -blocks.push(block); - -} - -return importDAG(blocks); - + const blocks = []; + const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); + for await (const block of reader.blocks()) { + blocks.push(block); + } + return importDAG(blocks); } async function frontend() { + // Create a new client + const client = await Client.create(); -// Create a new client - -const client = await Client.create(); - -// Fetch the delegation from the backend - -const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; // backend method is exposed at this API URL - -const response = await fetch(apiUrl); - -const data = await response.arrayBuffer(); - -// Deserialize the delegation - -const delegation = await Delegation.extract(new Uint8Array(data)); - -if (!delegation.ok) { - -throw new Error('Failed to extract delegation'); - -} - -// Add proof that this agent has been delegated capabilities on the space - -const space = await client.addSpace(delegation.ok); + // Fetch the delegation from the backend + const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; // backend method is exposed at this API URL + const response = await fetch(apiUrl); + const data = await response.arrayBuffer(); -client.setCurrentSpace(space.did()); + // Deserialize the delegation + const delegation = await Delegation.extract(new Uint8Array(data)); + if (!delegation.ok) { + throw new Error('Failed to extract delegation'); + } -// READY to go! + // Add proof that this agent has been delegated capabilities on the space + const space = await client.addSpace(delegation.ok); + client.setCurrentSpace(space.did()); + // READY to go! } ``` @@ -145,7 +101,7 @@ You can see the following flow: - When `frontend` function is called in the user's environment: - An Agent DID is created - The `backend` function hosted at an API endpoint is called, passing in the Agent DID - - The client is set up with a UCAN delegating upload capabilties to the Agent + - The client is set up with a UCAN delegating upload capabilities to the Agent - It's now ready to upload! -However, there's other interesting possibilities - for instance, you could create an app where your users make Spaces and delegate permission to your app to read their uploads. Read the Architecture options section to explore more. +However, there's other interesting possibilities - for instance, you could create an app where your users make Spaces and delegate permission to your app to read their uploads. Read the [Architecture options](/docs/concepts/architecture-options/) section to explore more. diff --git a/src/pages/docs/concepts/upload-vs-store.md b/src/pages/docs/concepts/upload-vs-store.md index 5608421..0609727 100644 --- a/src/pages/docs/concepts/upload-vs-store.md +++ b/src/pages/docs/concepts/upload-vs-store.md @@ -1,41 +1,27 @@ -## Upload vs. Store in web3.storage - -## Upload vs. Store capabilities in web3.storage +# Upload vs. Store in web3.storage There are two types of content identifiers (CIDs) that web3.storage interacts with: -- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. -- Shard CIDs: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. +1. Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. +2. Shard CIDs: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. One example of where you might see both is uploading a large file with the CLI: -``` +```sh w3 up gb.file - -1 file (1GB) - + 1 file (1GB) bagbaierao... - bagbaieraq... - bagbaieraj... - bagbaierai... - bagbaierax... - bagbaieraf... - bagbaierac... - bagbaierax... - bagbaierax... - bagbaiera4... ⁂ Stored 1 file - ⁂ https://w3s.link/ipfs/bafybeiaxwvg4... ``` @@ -45,7 +31,7 @@ In the vast majority of cases, users should focus on content CIDs, as this is wh However, if you're interested in learning more about how web3.storage uses both, read on! -### Upload vs. Store +## Upload vs. Store There are two similar-sounding, complementary, but separate concepts in web3.storage: Upload and Store. The place you most readily see this is in the lower-level client methods like `Client.capabilities.upload.*` and `Client.capabilities.store.*`, and CLI methods like `w3 can upload *`, and `w3 can store *`. @@ -61,26 +47,24 @@ However, from web3.storage's perspective, it doesn't necessarily know whether th In cases where the user is uploading a whole file or directory like in the example above, it's safe to know that the series of CAR shards uploaded by the user correspond to the file's content CID that it cares about. That's why the higher-level `w3 up` method is appropriate to use, and should represent the vast majority of upload use cases by web3.storage users. -### When should I care about Store and shard CIDs? +## When should I care about Store and shard CIDs? There's a few cases to note when caring about shard CIDs themselves and the `store` series of capabilities. -**Sharing data across content CIDs** - -Because IPFS interacts with DAGs, you can actually share blocks between different content CIDs. One simple example would be in a directory: +### Sharing data across content CIDs -\ +Because IPFS interacts with DAGs, you can actually share blocks between different content CIDs. One simple example would be in a directory -If a user uploads this directory using `w3 up`, the content CID for this directory will be registered with their account. This is actually the root CID of the directory itself, but isn't the only content CID in reality that was made available on the network: each file within this directory actually has their own content CID (highlighted). +If a user uploads a directory using `w3 up`, the content CID for this directory will be registered with their account. This is actually the root CID of the directory itself, but isn't the only content CID in reality that was made available on the network: each file within this directory actually has their own content CID (highlighted). Let's say you want to be able to have two different directories registered as uploads in your account: one that has 3 of the files above, and another that has all of them. One way to do this would be to upload each file separately, then locally compute the content CIDs of each directory. You can then call `w3 can upload add` to register each of these directories. This results in only a single copy of each file being stored, but you being able to interact with two different directories! -**Removing data from account** +### Removing data from an account web3.storage tracks usage for payment (i.e., how much storage is utilized by a user) using the volume of data associated with shard CIDs. This should make sense after learning about the difference between `store` and `upload` - web3.storage is storing the CAR shards themselves, and `upload`s are more users tracking. -Fortunately, this shouldn't make things any more complicated - we go into more detail below, but in general, when you remove a content CID from your account, you'll want to remove the shard CIDs as well (e.g., in the client calling `Client.remove(contentCID, shards=True)`). +Fortunately, this shouldn't make things any more complicated - we go into more detail below, but in general, when you remove a content CID from your account, you'll want to remove the shard CIDs as well (e.g., in the client calling `Client.remove(contentCID, shards=true)`). However, if you are a power user interacting with shard CIDs as well (like in the previous section), then you need to be more cautious about removing shard CIDs from your account. (This is why the default for the client and CLI methods is for shards to be maintained after removing a content CID). Learn more about how to do this in the Remove section. diff --git a/src/pages/docs/how-tos/_meta.json b/src/pages/docs/how-tos/_meta.json new file mode 100644 index 0000000..d339ebe --- /dev/null +++ b/src/pages/docs/how-tos/_meta.json @@ -0,0 +1,9 @@ +{ + "create-account": "Create account and space", + "upload": "Upload", + "retrieve": "Retrieve", + "list": "List", + "remove": "Remove", + "plan": "Change price plan", + "receipts": "Query UCAN receipts" +} diff --git a/src/pages/docs/how-tos/index.md b/src/pages/docs/how-tos/create-account.md similarity index 91% rename from src/pages/docs/how-tos/index.md rename to src/pages/docs/how-tos/create-account.md index fe7357d..374bb19 100644 --- a/src/pages/docs/how-tos/index.md +++ b/src/pages/docs/how-tos/create-account.md @@ -1,14 +1,13 @@ -# How-tos -## Create Account and Space +# Create Account and Space -**How to create an account** +## How to create an account In the How-tos section of the docs, we show the most familiar, straightforward way to use web3.storage: by setting up an account for you, the developer, that you use in your application. For an overview of the various ways web3.storage can be integrated with your application, check out the "Architecture options" concepts section. You first need to create and register a Space you'll upload to. A Space acts as a namespace for your uploads. It is created locally and associated with a private key, and is then registered with web3.storage and associated with your email address. But don't worry about keeping track of the Space's private key! web3.storage's email authorization system allows this private key to be treated as a throwaway key. -**The easiest way to create and register a Space is by using the CLI.** +## The easiest way to create and register a Space is by using the CLI 1. Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli` 2. Run `w3 authorize [alice@example.com](mailto:alice@example.com)` in the command line using your email address. This will send an email to your inbox with a link for validation. @@ -16,7 +15,7 @@ You first need to create and register a Space you'll upload to. A Space acts as 4. Create a new Space for storing your data and register it: w3 space create -**w3 space register** +## `w3 space register` Separately, you can visit console.web3.storage, sign up with your email and select a plan, and create a space using the UI, but we recommend that developers get familiar with the CLI since it's a powerful tool for many things you might want to do. diff --git a/src/pages/docs/how-tos/upload.md b/src/pages/docs/how-tos/upload.md index 534aae9..69296fe 100644 --- a/src/pages/docs/how-tos/upload.md +++ b/src/pages/docs/how-tos/upload.md @@ -1,6 +1,6 @@ -## Upload +# Upload -**How to upload data using web3.storage** +## How to upload data using web3.storage In this how-to guide, you'll learn how to store data programmatically for your development projects using the web3.storage client library in JavaScript using your (developer-owned) Space. This includes various architecture options for the data pipeline for your users to upload to web3.storage, which then makes your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin. @@ -12,63 +12,67 @@ Later in this section, we also cover uploading data using the CLI or web console ⚠️❗ Permanent Data ♾️: Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future. -**web3.storage API authorization** +## web3.storage API authorization In the previous section, you created a Space that has a unique DID. To use the client to upload data to this Space, you need to give it permission to do so. This is done by passing an Agent into the client and a UCAN token delegating permissions to this Agent. An Agent has a DID with an underlying private key (like a Space), but can be thought of as parallel to a login session (whereas the Space can be thought of more like a bucket or account). To learn more about how UCAN works and how web3.storage uses it, read our docs on UCAN authorization. -**Using the Javascript client to upload** +## Using the Command line `w3cli` -This section discusses using the web3.storage JavaScript client, w3up-client, with your (developer-owned) Space in your application. web3.storage's Javascript client provides a simple interface for storing data using syntax inspired by familiar web APIs such as [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API)and [File](https://developer.mozilla.org/en-US/docs/Web/API/File). +If you followed the Create account and Space section, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions: +1. (If not yet installed) Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli`. +2. Run `w3 authorize [alice@example.com](mailto:alice@example.com)` in the command line using your email address. Click on the validation link sent to your email. +3. After successfully running `authorize`, your CLI Agent has been delegated access to all Spaces associated with your email address. You can see a list of these Spaces using `w3 space ls` and select the one you'd like to upload to using `w3 space use \`. + +When the right Space is selected, you are ready to upload! You can do so by running `w3 up \`. + +There are a few useful flags (check out the reference docs to see a full list): + +``` +--no-wrap //Don't wrap input files with a directory. +-H, --hidden //Include paths that start with ".". +-c, --car //File is a CAR file. ``` -mermaid -flowchart TD +## Using the Javascript client -B[w3up-client instance] --\>|Automatic if specific Agent is not passed when client object created|B(Pass local Agent DID and key) +This section discusses using the web3.storage JavaScript client, w3up-client, with your (developer-owned) Space in your application. web3.storage's Javascript client provides a simple interface for storing data using syntax inspired by familiar web APIs such as [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API)and [File](https://developer.mozilla.org/en-US/docs/Web/API/File). +```mermaid +flowchart TD +B[w3up-client instance] --\>|Automatic if specific Agent is not passed when client object created|B(Pass local Agent DID and key) S --\> C(Get UCAN delegation from Space to Agent) - C --\> D(Upload to Space using Agent) - ``` All uses of w3up-client to upload with web3.storage follow the flow above. -**Installing the client** +### Installing the client In your JavaScript project, add the web3.storage package to your dependencies: -``` -npminstall@web3-storage/w3up-client +```sh +npm install @web3-storage/w3up-client ``` -**Creating a server-side client instance** +### Creating a server-side client instance The package provides a [static](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#create)[create](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#create)[function](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#create) that returns a [Client](https://web3-storage.github.io/w3up-client/classes/client.Client.html)[object](https://web3-storage.github.io/w3up-client/classes/client.Client.html). How you initialize it depends on the backend environment. -**Claim delegation via email validation: For persistent backend only** - -``` -mermaid +### Claim delegation via email validation: For persistent backend only +```mermaid sequenceDiagram - Client-\>\>web3.storage service: Here is my email address and Agent DID - web3.storage service--\>\>Client: Please click the link to validate - Client--\>\>web3.storage service: Email address validated - web3.storage service-\>\>Client: Here is a UCAN delegating permission from Space DID to Agent DID - ``` You can use web3.storage's email authorization flow to give permissions to your server-side client. This can be good if your backend environment will be persistent (otherwise it would be prohibitive to click an email validation link every time the client is re-instantiated. -``` +```javascript import { create } from '@web3-storage/w3up-client' - const client = await create() ``` @@ -76,108 +80,80 @@ By default, clients will create a new [Agent](https://web3-storage.github.io/w3p Then you can authorize your Agent with your email address. Calling authorize will cause an email to be sent to the given address. -``` +```javascript await client.authorize('zaphod@beeblebrox.galaxy') ``` Once a user clicks the confirmation link in the email, the authorize method will resolve. Make sure to check for errors, as authorize will fail if the email is not confirmed within the expiration timeout. Authorization needs to happen only once per agent. This also claims all delegations available with your email address, so from there, you can select the Space you'd like to use. -``` +```javascript await client.setCurrentSpace(space.did()) # select the relevant Space DID that is associated with your account ``` -**Bring your own Agent: For any backend (including non-persistent and/or serverless)** +### Bring your own Agent -``` -mermaid +For any backend (including non-persistent and/or serverless) +```mermaid sequenceDiagram - Developer-\>\>Developer: Create Agent private key and DID - Developer-\>\>Developer: Delegate UCAN from Space to Agent - Developer-\>\>Client: Here is my Agent private key and UCAN delegating permissions - ``` An option that works for any backend environment is to define an Agent and delegate a UCAN from your Space to this Agent before initializing the client. This is especially useful if you're using the client in a serverless Node environment (e.g., Lambda). In your command line wherever the CLI is configured with the Space you want to use (e.g., where you created the Space): -``` +```shell # the following command returns what will be your Agent private key and DID -npx ucan-key ed --json - # store the private key in environment variable KEY +npx ucan-key ed --json # the following command returns the UCAN that will delegate - # make sure `w3 space use` is set to the Space you intend on using - -w3 delegation create \ | base64 +# if you want to limit permissions being passed to the Agent, you can specify which permissions to give, e.g., `--can 'store/add' --can 'upload/add'` limits to just being able to upload +w3 delegation create | base64 # store the output in environment variable PROOF - -# if you want to limit permissions being passed to the Agent, you can specify which permissions to give, e.g., `--can 'store/add' --can 'upload/add'` limits to just being able to upload ``` - Then, when you initialize and configure the client, you can pass in this Agent and UCAN. -``` +```javascript import { create } from '@web3-storage/w3up-client' - -import \* as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys +import * as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys async function main () { - -// Load client with specific private key - -const principal = Signer.parse(process.env.KEY) - -const client = await Client.create({ principal }) - -// Add proof that this agent has been delegated capabilities on the space - -const proof = await parseProof(process.env.PROOF) - -const space = await client.addSpace(proof) - -await client.setCurrentSpace(space.did()) - -// READY to go! - + // Load client with specific private key + const principal = Signer.parse(process.env.KEY) + const client = await Client.create({ principal }) + // Add proof that this agent has been delegated capabilities on the space + const proof = await parseProof(process.env.PROOF) + const space = await client.addSpace(proof) + await client.setCurrentSpace(space.did()) + // READY to go! } -/\*\* @param {string} data Base64 encoded CAR file \*/ - +/** @param {string} data Base64 encoded CAR file \*/ async function parseProof (data) { - -const blocks = [] - -const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) - -for await (const block of reader.blocks()) { - -blocks.push(block) - -} - -return importDAG(blocks) - + const blocks = [] + const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) + for await (const block of reader.blocks()) { + blocks.push(block) + } + return importDAG(blocks) } ``` If you're doing this in a non-persistent or serverless backend, you might consider using an in-memory [Store](https://github.com/web3-storage/w3up/tree/main/packages/access-client) for your Agent information rather than the default on-disk: -``` +```javascript import { StoreMemory } from '@web3-storage/access/stores/store-memory' - const client = await Client.create({ principal, store: new MemoryStore() }) ``` -**Uploading to web3.storage** +### Uploading to web3.storage Now that your backend client instance is set up with being able to interact with your Space, you're ready to upload! Call uploadFile to upload a single file, or uploadDirectory to upload multiple files. @@ -186,199 +162,111 @@ There are two main options to getting content into your Space: - Upload data to web3.storage from the backend client itself (e.g., you're storing data that your users are uploading to your backend) - Upload data to web3.storage directly from your user's environment (like your application's user's browser) by delegating a UCAN that has permission to upload to your Space -**Upload from backend client directly** - -``` -mermaid +### Upload from backend client directly +```mermaid sequenceDiagram - User-\>\>w3up-client in backend: Upload data - w3up-client in backend-\>\>web3.storage service: Upload data - ``` -You are already set up to upload using your client instance as data becomes available to your backend - you can call uploadFile or uploadDirectory with it. Here's a complete example of what this might look like: +You are already set up to upload using your client instance as data becomes available to your backend - you can call uploadFile or uploadDirectory with it. -``` +```javascript import { create } from '@web3-storage/w3up-client' +import { filesFromPaths } from 'files-from-path' -import \* as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys - -const principal = Signer.parse(process.env.KEY) // Agent private key - -const client = await create({ principal }) - -async function main () { - -// Load client with specific private key - -const principal = Signer.parse(process.env.KEY) - -const client = await Client.create({ principal }) - -// Add proof that this agent has been delegated capabilities on the space - -const proof = await parseProof(process.env.PROOF) - -const space = await client.addSpace(proof) - -await client.setCurrentSpace(space.did()) - -// READY to go! - -} - -/\*\* @param {string} data Base64 encoded CAR file \*/ - -async function parseProof (data) { - -const blocks = [] - -const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) - -for await (const block of reader.blocks()) { - -blocks.push(block) +// e.g "./best-gifs" +const path = process.env.PATH_TO_FILES +const files = await filesFromPaths([path]) -} - -return importDAG(blocks) - -} +const client = await create() +const directoryCid = await client.storeDirectory(files) ``` -**Delegate UCAN for your user to upload directly** +In the example above, `directoryCid` resolves to an IPFS directory. -``` -mermaid -sequenceDiagram +### Delegate UCAN for your user to upload directly +```mermaid +sequenceDiagram participant w3up-client in user - participant w3up-client in backend - participant web3.storage service - w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent - w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID - w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID - w3up-client in user-\>\>web3.storage service: Upload data - ``` Your backend instance can also be used to delegate upload permissions directly to your user to upload. The code snippet below shows an example of how you might set up a client instance in your application frontend and how it might interact with your backend client. You can see how the frontend client Agent DID is used for the backend client to delegate permissions to; from there, it will be the frontend client that will call the `upload` method. -``` +```javascript import { CarReader } from '@ipld/car'; - -import \* as DID from '@ipld/dag-ucan/did'; - -import \* as Delegation from '@ucanto/core/delegation'; - +import * as DID from '@ipld/dag-ucan/did'; +import * as Delegation from '@ucanto/core/delegation'; import { importDAG } from '@ucanto/core/delegation'; - -import \* as Signer from '@ucanto/principal/ed25519'; - -import \* as Client from '@web3-storage/w3up-client'; +import * as Signer from '@ucanto/principal/ed25519'; +import * as Client from '@web3-storage/w3up-client'; async function backend(did: string) { - -// Load client with specific private key - -const principal = Signer.parse(process.env.KEY); - -const client = await Client.create({ principal }); - -// Add proof that this agent has been delegated capabilities on the space - -const proof = await parseProof(process.env.PROOF); - -const space = await client.addSpace(proof); - -await client.setCurrentSpace(space.did()); - -// Create a delegation for a specific DID - -const audience = DID.parse(did); - -const abilities = ['store/add', 'upload/add']; - -const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24; // 24 hours from now - -const delegation = await client.createDelegation(audience, abilities, { - -expiration, - -}); - -// Serialize the delegation and send it to the client - -const archive = await delegation.archive(); - -return archive.ok; - + // Load client with specific private key + const principal = Signer.parse(process.env.KEY); + const client = await Client.create({ principal }); + + // Add proof that this agent has been delegated capabilities on the space + const proof = await parseProof(process.env.PROOF); + const space = await client.addSpace(proof); + await client.setCurrentSpace(space.did()); + + // Create a delegation for a specific DID + const audience = DID.parse(did); + const abilities = ['store/add', 'upload/add']; + const expiration = Math.floor(Date.now() / 1000) + 60 * 60 * 24; // 24 hours from now + const delegation = await client.createDelegation(audience, abilities, { + expiration, + }); + + // Serialize the delegation and send it to the client + const archive = await delegation.archive(); + return archive.ok; } -/\*\* @param {string} data Base64 encoded CAR file \*/ - +/** @param {string} data Base64 encoded CAR file */ async function parseProof(data) { - -const blocks = []; - -const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); - -for await (const block of reader.blocks()) { - -blocks.push(block); - -} - -return importDAG(blocks); - + const blocks = []; + const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')); + for await (const block of reader.blocks()) { + blocks.push(block); + } + return importDAG(blocks); } async function frontend() { + // Create a new client + const client = await Client.create(); -// Create a new client - -const client = await Client.create(); - -// Fetch the delegation from the backend - -const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; // backend method is exposed at this API URL - -const response = await fetch(apiUrl); - -const data = await response.arrayBuffer(); - -// Deserialize the delegation - -const delegation = await Delegation.extract(new Uint8Array(data)); + // Fetch the delegation from the backend + const apiUrl = `/api/w3up-delegation/${client.agent().did()}`; + const response = await fetch(apiUrl); + const data = await response.arrayBuffer(); -if (!delegation.ok) { + // Deserialize the delegation + const delegation = await Delegation.extract(new Uint8Array(data)); + if (!delegation.ok) { + throw new Error('Failed to extract delegation'); + } -throw new Error('Failed to extract delegation'); - -} - -// Add proof that this agent has been delegated capabilities on the space - -const space = await client.addSpace(delegation.ok); - -client.setCurrentSpace(space.did()); - -// READY to go! + // Add proof that this agent has been delegated capabilities on the space + const space = await client.addSpace(delegation.ok); + client.setCurrentSpace(space.did()); + // READY to go! } ``` -**[Preparing files and uploading](https://web3.storage/docs/how-tos/store/#preparing-files-for-upload)** +### Preparing files and uploading You are now ready to upload using the client! In general, the easiest way to upload data is using the uploadFile or uploadDirectory method. @@ -386,20 +274,15 @@ uploadFile expects a "Blob like" input, which can be a [Blob](https://developer. uploadDirectory requires File-like objects instead of Blobs, as the file's name property is used to build the directory hierarchy. -**Tip** - -When uploading multiple files, try to give each file a unique name. All the files in a storeDirectory request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name. +**Tip** When uploading multiple files, give each file a unique name. All the files in a `storeDirectory` request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name. You can control the directory layout and create nested directory structures by using / delimited paths in your filenames: -``` -const files = [ +```javascript +const files = [ new File(['some-file-content'], 'readme.md'), - new File(['import foo'], 'src/main.py'), - new File([someBinaryData], 'images/example.png'), - ] const directoryCid = await client.storeDirectory(files) @@ -409,195 +292,78 @@ In the example above, directoryCid resolves to an IPFS directory with the follow ``` . - ├──images - -│└──example.png - +| └──example.png ├──readme.md - └──src - └──main.py ``` -There are a few different ways of creating File objects available, depending on your platform. +There are a few different ways of creating `File` objects available, depending on your platform. In the browser, you can use a [file input element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/file) to allow the user to select files for upload: -``` -functiongetFiles(){ - -constfileInput=document.querySelector('input[type="file"]') - -returnfileInput.files - +```javascript +function getFiles () { + const fileInput = document.querySelector('input[type="file"]') + return fileInput.files } ``` You can also manually create File objects using the native File constructor provided by the browser runtime. This is useful when you want to store data created by your application, instead of files from the user's computer. -``` -functionmakeFileObjects(){ - -//YoucancreateFileobjectsfromaBlobofbinarydata - -//see:https://developer.mozilla.org/en-US/docs/Web/API/Blob - -//Herewe'rejuststoringaJSONobject,butyoucanstoreimages, - -//audio,orwhateveryouwant! - -constobj={hello:'world'} - -constblob=newBlob([JSON.stringify(obj)],{type:'application/json'}) - -constfiles=[ - -newFile(['contents-of-file-1'],'plain-utf8.txt'), - -newFile([blob],'hello.json') - -] - -returnfiles - +```javascript +function makeFileObjects () { + // You can create File objects from a Blob of binary data + // see: https://developer.mozilla.org/en-US/docs/Web/API/Blob + // Here we're just storing a JSON object, but you can store images, + // audio, or whatever you want! + const obj = { hello: 'world' } + const blob = new Blob([JSON.stringify(obj)], { type: 'application/json' }) + + const files = [ + new File(['contents-of-file-1'], 'plain-utf8.txt'), + new File([blob], 'hello.json') + ] + return files } ``` -In Node.js, the web3.storage package exports some helpful utility functions from the [files-from-path](https://www.npmjs.com/package/files-from-path)[module](https://www.npmjs.com/package/files-from-path) that allow you to easily read File objects from the local file system. The getFilesFromPath helper asynchronously returns an array of Files that you can use directly with the put client method: +In Node.js, the [files-from-path](https://www.npmjs.com/package/files-from-path)[module](https://www.npmjs.com/package/files-from-path) module reads File objects from the local file system. The `getFilesFromPaths` helper asynchronously returns an array of Files that you can use directly with the put client method: -``` -import{getFilesFromPath}from'web3.storage' +```javascript +import { getFilesFromPaths } from 'files-from-path' ``` If you expect to be loading a lot of large files, you may be better served by the [filesFromPath](https://github.com/web3-storage/files-from-path#filesfrompath)[helper](https://github.com/web3-storage/files-from-path#filesfrompath). It reduces memory pressure by yielding File objects one by one as they're loaded from disk, instead of loading everything into memory. You can then issue multiple put requests to send each file to web3.storage. You can also manually create File objects by importing a Node.js implementation of File from the web3.storage package. This is useful when you want to store data created by your application, instead of files from the user's computer. -``` -import{File}from'web3.storage' - -asyncfunctiongetFiles(path){ - -constfiles=awaitgetFilesFromPath(path) - -console.log(`read${files.length}file(s)from${path}`) - -returnfiles - +```javascript +async function getFiles (path) { + const files = await getFilesFromPath(path) + console.log(`read ${files.length} file(s) from ${path}`) + return files } -functionmakeFileObjects(){ - -//YoucancreateFileobjectsfromaBufferofbinarydata - -//see:https://nodejs.org/api/buffer.html - -//Herewe'rejuststoringaJSONobject,butyoucanstoreimages, - -//audio,orwhateveryouwant! - -constobj={hello:'world'} - -constbuffer=Buffer.from(JSON.stringify(obj)) - -constfiles=[ - -newFile(['contents-of-file-1'],'plain-utf8.txt'), - -newFile([buffer],'hello.json') - -] - -returnfiles - +function makeFileObjects () { + // You can create File objects from a Buffer of binary data + // see: https://nodejs.org/api/buffer.html + // Here we're just storing a JSON object, but you can store images, + // audio, or whatever you want! + const obj = { hello: 'world' } + const buffer = Buffer.from(JSON.stringify(obj)) + const files = [ + new File(['contents-of-file-1'], 'plain-utf8.txt'), + new File([buffer], 'hello.json') + ] + return files } ``` -**Using the CLI or web console to upload** - -You can also use the CLI or the web3.storage web console to upload. - -**CLI** - -If you followed the Create account and Space section, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions: - -1. (If not yet installed) Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli`. -2. Run `w3 authorize [alice@example.com](mailto:alice@example.com)` in the command line using your email address. Click on the validation link sent to your email. -3. After successfully running `authorize`, your CLI Agent has been delegated access to all Spaces associated with your email address. You can see a list of these Spaces using `w3 space ls` and select the one you'd like to upload to using `w3 space use \`. - -When the right Space is selected, you are ready to upload! You can do so by running `w3 up \`. - -There are a few useful flags (check out the reference docs to see a full list): - -``` ---no-wrap //Don't wrap input files with a directory. --H, --hidden //Include paths that start with ".". --c, --car //File is a CAR file. -``` - -**Web console** - -You can also upload data using the web console. First, go to console.web3.storage. Enter your email if you're not yet logged in (and click on the link sent to your inbox). - - - -Once logged in, select the Space you'd like to upload to. - - - -From there, just click the Upload box and select your file or directory. - - - -**Content vs. shard CIDs** - -There are two types of content identifiers (CIDs) that web3.storage interacts with: - -- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. -- Shard CIDs: The CID of the serialized shards of data itself (CAR files) that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. - -One example of where you might see both is uploading a large file with the CLI: - - -``` -w3 up gb.file - -1 file (1GB) - -bagbaierao... - -bagbaieraq... - -bagbaieraj... - -bagbaierai... - -bagbaierax... - -bagbaieraf... - -bagbaierac... - -bagbaierax... - -bagbaierax... - -bagbaiera4... - -⁂ Stored 1 file - -⁂ https://w3s.link/ipfs/bafybeiaxwvg4... -``` - -The CLI sharded the 1GB upload into 10 shards, each with a `bag…`-prefixed CID. The content CID of the file itself is included in the `w3s.link/ipfs/bafy…` link at the bottom. - -In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs. -However, there are some cases where interacting with shard CIDs can be useful. This requires using the lower-level methods that the client and CLI expose (`capability.*` methods in the client, and `w3 can *` methods in the CLI). You can read more about the differences in the Upload vs. Store section. -**Next steps** +## Next steps Learn more about how to fetch your data using the CID in the next section, Retrieve. diff --git a/src/pages/docs/quickstart.md b/src/pages/docs/quickstart.md new file mode 100644 index 0000000..74f1f2e --- /dev/null +++ b/src/pages/docs/quickstart.md @@ -0,0 +1,95 @@ +# Quickstart + +Ready to get started using web3.storage? Get up and running in minutes by following this quickstart guide. In this guide, we'll walk through the following steps: + +1. Install the CLI. +2. Create a Space to upload your files and register it with web3.storage. +3. Upload a file. +4. Get your uploaded file using your browser or curl. + +This guide uses our CLI, w3cli, since it's the fastest way to get started using web3.storage programmatically. In the "How-tos" section of the docs, we also include instructions on using the Javascript client or web interface to create an account, store data, and more. + +## You will need + +Node.js version `18` or higher and npm version `7` or higher to complete this guide. Check your local versions like this: + +```shell +node --version && npm --version + +> v18.17.1 +> 7.18.1 +``` + +Install the CLI from npm using your command line: + +```sh +npm install -g @web3-storage/w3cli +``` + +## Create an account + +You need to create a web3.storage account associated with an email address and set it up so you can start uploading to a Space. The Space is created locally and associated with a private key, and is then registered with web3.storage and associated with your email address. But don't worry about keeping track of the Space's private key! web3.storage's email authorization system allows this private key to be treated as a throwaway key. + +1. Run `w3 authorize [alice@example.com](mailto:alice@example.com)` in the command line using your email address. This will sent an email to your inbox with a link for validation. +2. Once you click on the validation link, you'll be taken to a webpage where you can enter your payment information and select a plan (like our Free tier). +3. Create a new Space for storing your data and register it: + +```sh +w3 space create Documents # pick a good name! + +w3 space register +``` + +## Upload + +You can now upload a file or directory using the command line: + +```sh +$ w3 up lets-go.txt + 1 file 0.6KB +⁂ Stored 1 file +⁂ https://w3s.link/ipfs/bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow4 +``` + +The CLI content-addresses your files, packs them into 1 or more CAR files, and uploads them to web3.storage for indexing and inclusion in Filecoin storage deals. It will show an http gateway URL that includes the content CID (content identifier) of your upload e.g: + +https://w3s.link/ipfs/bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow4 + +## Get your file + +Your upload is now available over the public IPFS network using the content CID of your upload. The easiest way to fetch it is using the w3s.link gateway using the link that `w3 up` provided. w3s.link is optimized for content uploaded to web3.storage. + +```sh +curl -L 'https://w3s.link/ipfs/bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow4/lets-go.txt' + + ___. ________ __ +__ _ __ ____ \_ |__ \_____ \ _______/ |_ ____ _______ _____ ____ ____ +\ \/ \/ /_/ __ \ | __ \ _(__ < / ___/\ __\ / _ \ \_ __ \\__ \ / ___\ _/ __ \ + \ / \ ___/ | \_\ \ / \ \___ \ | | ( <_> ) | | \/ / __ \_ / /_/ >\ ___/ + \/\_/ \___ > |___ //______ / /\/____ > |__| \____/ |__| (____ / \___ / \___ > + \/ \/ \/ \/ \/ \/ /_____/ \/ +``` + + +You can also fetch your content p2p style over bitswap with an IPFS implementation like `helia` or `kubo`. + +```sh +ipfs cat bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow4/lets-go.txt + + ___. ________ __ +__ _ __ ____ \_ |__ \_____ \ _______/ |_ ____ _______ _____ ____ ____ +\ \/ \/ /_/ __ \ | __ \ _(__ < / ___/\ __\ / _ \ \_ __ \\__ \ / ___\ _/ __ \ + \ / \ ___/ | \_\ \ / \ \___ \ | | ( <_> ) | | \/ / __ \_ / /_/ >\ ___/ + \/\_/ \___ > |___ //______ / /\/____ > |__| \____/ |__| (____ / \___ / \___ > + \/ \/ \/ \/ \/ \/ /_____/ \/ +``` + +## Next steps + +Congratulations! You've just covered the basics of web3.storage. To learn more, take a look at these useful resources: + +- For a deep dive into storing files, including using the Javascript client to do so, visit the Store how-to guide. +- Read more about the power of UCANs and IPFS, and learn about the various options to integrate web3.storage with your application. +- Try out our image gallery example to see how easy it is to take advantage of these decentralized protocols using web3.storage. +- Visit the reference API section for more details on what else you can do with the web3.storage client and how to integrate it into your own projects. + diff --git a/src/pages/docs/w3up-client.md b/src/pages/docs/w3up-client.md index 4551d81..5a1f9bd 100644 --- a/src/pages/docs/w3up-client.md +++ b/src/pages/docs/w3up-client.md @@ -36,9 +36,9 @@ See the [client reference docs][reference-w3up-client#create] for more creation ## Create and register a space -When you upload things to web3.storage, each upload is associated with a "space," which is a unique identifier that acts as a namespace for your content. +When you upload things to web3.storage, each upload is associated with a "space," which is a unique identifier that acts as a namespace for your content. -Spaces are identified by DID using keys created locally on your devices. To use a space for uploads, it needs to be registered with the storage service by providing an email address. +Spaces are identified by DID using keys created locally on your devices. To use a space for uploads, it needs to be registered with the storage service by providing an email address. To create a space using `w3up-client`, use the [`createSpace` client method][reference-w3up-client#createSpace]: @@ -106,7 +106,7 @@ In the example above, `directoryCid` resolves to an IPFS directory with the foll The `uploadFile` and `uploadDirectory` methods described in the previous step both return a CID, or Content Identifier, encoded as a string. -To create a link to view your file on an IPFS gateway, create a URL of the form `https://${cid}.ipfs.${gatewayHost}`, where `${cid}` is the CID of the content you want to view, and `${gatewayHost}` is the domain of the gateway. To use our own gateway at `w3s.link`, your url would be `https://${cid}.ipfs.w3s.link`. +To create a link to view your file on an IPFS gateway, create a URL of the form `https://${cid}.ipfs.${gatewayHost}`, where `${cid}` is the CID of the content you want to view, and `${gatewayHost}` is the domain of the gateway. To use our own gateway at `w3s.link`, your url would be `https://${cid}.ipfs.w3s.link`. Opening the gateway URL in a browser will take you to your uploaded file, or a directory listing of files, depending on what you uploaded. From 0b240f4e1bb3225b6d782d95d91fee5193eef87a Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 14:09:37 +0000 Subject: [PATCH 21/25] chore: fix retrieve License: MIT Signed-off-by: Oli Evans --- src/pages/docs/how-tos/retrieve.md | 47 ++++++++++++------------------ 1 file changed, 19 insertions(+), 28 deletions(-) diff --git a/src/pages/docs/how-tos/retrieve.md b/src/pages/docs/how-tos/retrieve.md index c04399c..33a49e4 100644 --- a/src/pages/docs/how-tos/retrieve.md +++ b/src/pages/docs/how-tos/retrieve.md @@ -1,6 +1,4 @@ -## Retrieve - -## How to retrieve data from web3.storage +# Retrieve In this how-to guide, you'll learn several methods for retrieving data from web3.storage. @@ -13,23 +11,19 @@ All data stored using web3.storage is made available for retrieval via [IPFS](ht When retrieving any data, you'll be using the content CID of the upload (prefixed by `bafy…`). -**Using an IPFS HTTP gateway** +## Using an IPFS HTTP gateway You can easily fetch any data stored using web3.storage using an IPFS HTTP gateway. Because IPFS is a peer-to-peer, decentralized network, you can use any public HTTP gateway to fetch your data. In this guide, we'll use the gateway at w3s.link (which is optimized for data stored with web3.storage), but you can see more worldwide gateways on the [IPFS Public Gateway Checker](https://ipfs.github.io/public-gateway-checker/). You can use an IPFS gateway to view a list of all the files in that directory from your browser. To do so, simply create a gateway URL. For example, if your CID is bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu, you can make a URL for the w3s.link gateway as follows: `https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/`. Follow that link, and you'll see a page similar to this: - - If you want to link directly to a file within that directory, just add the file path after the CID portion of the link. For example: [bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg](https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg) could be used as a shareable link for your new favorite wallpaper. **Tip** Your web3.storage console page includes IPFS gateway links to all the content you've uploaded. -### [Setting the filename for downloads via gateways](https://web3.storage/docs/how-tos/retrieve/#setting-the-filename-for-downloads-via-gateways) +### Setting the filename When downloading files from an HTTP gateway, web browsers will set the default filename for the downloaded file based on the path component of the gateway link. For example, if you use your browser's "Save link as..." feature on the following link, it should prompt you to save a file named treehouse.jpeg: @@ -52,11 +46,8 @@ If you have such a link, you can override the default filename by adding a query [https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/?filename=treehouse.jpeg](https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/?filename=treehouse.jpeg) -**Other gateway endpoints** - -Graph API, CAR fetching… -## [Using the IPFS command line](https://web3.storage/docs/how-tos/retrieve/#using-the-ipfs-command-line) +## Using the `ipfs` command If you have the [IPFS command line interface](https://docs.ipfs.io/how-to/command-line-quick-start/) installed, you can use it directly to fetch data without going through a gateway. This also works if you've installed [IPFS Desktop](https://docs.ipfs.io/install/ipfs-desktop/), which includes the IPFS CLI. @@ -72,30 +63,30 @@ If you want to get a specific file out of the bundle, add its name onto the end ipfs get bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu/youareanonsense.jpg ``` -## [Using curl or Powershell](https://web3.storage/docs/how-tos/retrieve/#using-curl-or-powershell) +## Using `curl` -Sometimes you may need to just download a specific file to your computer using the command line. Unix-based operating systems, like Linux and macOS, can use curl. Windows users can use Powershell. - -- Linux -- macOS -- Windows +Sometimes you may need to just download a specific file to your computer using the command line. 1. Open a terminal window. 2. Use curl to download your file: -``` -curl https://\.ipfs.w3s.link/\-o~/\ +```shell +curl https://.ipfs.w3s.link/ -o + +# example +curl https://bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq.ipfs.w3s.link/example.txt -o ./output-file.txt ``` -1.  -Replace \, \, and \ with their respective values. - | Variable | Replace with | Example | | --- | --- | --- | | \ | The CID of the file you want to download. | bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq | | \ | The _name_ of the file that you originally uploaded to web3.storage. | example.txt | | \ | The path and filename that you want curl to save the file to. This can be different to \. | Desktop/output-file.txt | - Your complete command should look something like this: +Replace ``, ``, and `` with their respective values. + +| Variable | Replace with | Example | +| --- | --- | --- | +| `` | The CID of the file you want to download. | bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq | +| `` | The _name_ of the file that you originally uploaded to web3.storage. | example.txt | +| `` | The path and filename that you want curl to save the file to. This can be different to ``. | `./output-file.txt` | -``` -curl https://bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq.ipfs.w3s.link/example.txt-o~/output-file.txt -``` ## Next steps Next, you'll learn about how to [list](https://web3.storage/docs/how-tos/list/) + From b7c549594103f3550588fd812f578b99ec5df4e3 Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 14:20:09 +0000 Subject: [PATCH 22/25] chore: fix moar License: MIT Signed-off-by: Oli Evans --- src/pages/docs/how-tos/_meta.json | 10 +++- src/pages/docs/how-tos/list.md | 24 ++------ src/pages/docs/how-tos/remove.md | 93 +++++++++++-------------------- 3 files changed, 45 insertions(+), 82 deletions(-) diff --git a/src/pages/docs/how-tos/_meta.json b/src/pages/docs/how-tos/_meta.json index d339ebe..36d0f1d 100644 --- a/src/pages/docs/how-tos/_meta.json +++ b/src/pages/docs/how-tos/_meta.json @@ -4,6 +4,12 @@ "retrieve": "Retrieve", "list": "List", "remove": "Remove", - "plan": "Change price plan", - "receipts": "Query UCAN receipts" + "plan": { + "title": "Change price plan", + "display": "hidden" + }, + "receipts": { + "title": "Query UCAN receipts", + "display": "hidden" + } } diff --git a/src/pages/docs/how-tos/list.md b/src/pages/docs/how-tos/list.md index a61adf7..1afaf2b 100644 --- a/src/pages/docs/how-tos/list.md +++ b/src/pages/docs/how-tos/list.md @@ -1,34 +1,22 @@ -## List - -### How to list files uploaded to web3.storage +# How to list files uploaded to web3.storage In this how-to guide, you'll learn about the different ways that you can list the files that you've uploaded to web3.storage. Once you've [stored some files](https://web3.storage/docs/how-tos/store/) using web3.storage, you'll want to see a list of what you've uplodaded. There are two ways you can do this: - Programmatically using the web3.storage client or CLI - Using the web3.storage console -## [Using the web3.storage client](https://web3.storage/docs/how-tos/list/#using-the-web3storage-client) or CLI +## Using the JS client or CLI You can also access a listing of your uploads from your code using the web3.storage client. In the example below, this guide walks through how to use the JavaScript client library to fetch a complete listing of all the data you've uploaded using web3.storage. For instructions on how to set up your client instance or CLI, check out the Upload section. -\ - -Here's what you call, here's what the output looks like. (client and CLI) - -Timestamps. Data size? - -Today, like other developer object storage solutions, no sorting or querying by timestamp to keep things scalable. - -Can get nested list of shard CIDs, or look up what the shard CIDs are for an individual upload. Client and CLI. If you'd like to learn more check out Upload vs. Store section - -Filecoin deal info and what fields mean +Today, like other developer object storage solutions, there is no sorting or querying by timestamp to keep things scalable. -## [Using the web3.storage](https://web3.storage/docs/how-tos/list/#using-the-web3storage-website)console +You can get nested list of shard CIDs, or look up what the shard CIDs are for an individual upload. Client and CLI. If you'd like to learn more check out Upload vs. Store section -You can see a list of everything you've uploaded to web3.storage in your console on the website. If you don't need to work with this list programmatically, using the website may be a simpler choice. +## Using the console web UI -\ +You can see a list of everything you've uploaded to web3.storage in the [console](https://console.web3.storage) web app. If you don't need to work with this list programmatically, using the website may be a simpler choice. This console provides a convenient overview of your stored data, including links to view your files in your browser via an [IPFS gateway](https://docs.ipfs.io/concepts/ipfs-gateway/) and information about how the data is being stored on the decentralized storage networks that web3.storage uses under the hood. diff --git a/src/pages/docs/how-tos/remove.md b/src/pages/docs/how-tos/remove.md index 98cb0eb..5ba0560 100644 --- a/src/pages/docs/how-tos/remove.md +++ b/src/pages/docs/how-tos/remove.md @@ -1,6 +1,4 @@ -## Remove - -### Removing data from your account +# Removing data from your account You might want to remove data from being associated with your account. You can do so using the JS client, CLI, or web app. @@ -12,7 +10,7 @@ Note that there is a minimum 30 day retention period for uploaded data, and even ⚠️❗ Permanent Data ♾️: Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future. -**Removing uploads (content CIDs) vs. stores (shard CIDs)** +## Removing uploads (content CIDs) vs. stores (shard CIDs) web3.storage tracks two different things for its users to support content addressing. These concepts were first introduced in the Upload section: @@ -25,70 +23,54 @@ Fortunately, this shouldn't make things any more complicated - we go into more d However, if you are a power user interacting with shard CIDs as well (e.g., using the client's `capability.store.*` or CLI's `w3 can store *` methods), then you need to be more cautious about removing shard CIDs from your account. (This is why the default for the client and CLI methods is for shards to be maintained after removing a content CID). You can read more about why you might want to interact with shard CIDs directly and the implications in the Upload vs. Store section. -**Using the client or CLI** +## Using the client or CLI If you followed the Upload section, you should already have your client or CLI set up with an Agent for your Space. From there, to remove a content CID from your account, you'll generally be using: - Client: `Client.remove(contentCID)` -- CLI: `w3 rm \` +- CLI: `w3 rm ` If you initially uploaded your content by using the recommended upload methods (e.g., used `Client.upload()` or `w3 up`) and didn't interact with CAR shards at all when uploading, we recommend removing the shard CIDs associated with the content CID from your account. Otherwise, you will still be paying for the data stored with web3.storage (as mentioned above). The easiest way to do that is to set the `shards` parameter as `True`: - Client: `Client.remove(contentCID, shards=True)` -- CLI: `w3 rm \ --shards` in the CLI). +- CLI: `w3 rm --shards` in the CLI A full example of this is: -``` +```javascript import { create } from '@web3-storage/w3up-client' - -import \* as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys +import * as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys const principal = Signer.parse(process.env.KEY) // Agent private key - const client = await create({ principal }) async function main () { - -// Load client with specific private key - -const principal = Signer.parse(process.env.KEY) - -const client = await Client.create({ principal }) - -// Add proof that this agent has been delegated capabilities on the space - -const proof = await parseProof(process.env.PROOF) - -const space = await client.addSpace(proof) - -await client.setCurrentSpace(space.did()) - -// remove content previously uploaded, including the underlying shards - -client.remove('bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu', shards=True) - + // Load client with specific private key + const principal = Signer.parse(process.env.KEY) + const client = await Client.create({ principal }) + + // Add proof that this agent has been delegated capabilities on the space + const proof = await parseProof(process.env.PROOF) + const space = await client.addSpace(proof) + await client.setCurrentSpace(space.did()) + + // remove content previously uploaded, including the underlying shards + client.remove('bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu', shards=True) } -/\*\* @param {string} data Base64 encoded CAR file \*/ - +/** @param {string} data Base64 encoded CAR file */ async function parseProof (data) { - -const blocks = [] - -const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) - -for await (const block of reader.blocks()) { - -blocks.push(block) - + const blocks = [] + const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) + for await (const block of reader.blocks()) { + blocks.push(block) + } + return importDAG(blocks) } -return importDAG(blocks) - -} ``` -**Removing content CIDs and shard CIDs separately** + +## Removing content CIDs and shard CIDs separately If you have managed your shard CIDs and upload CIDs separately (e.g., used `Client.capability.store.add()` and `Client.capability.upload.add()` in the client or `w3 can store add` and `w3 can upload add` in the CLI), you'll want to remove the upload CIDs and underlying shard CIDs separately as well. You can read more about why you might want to interact with shard CIDs directly and the implications in the Upload vs. Store section. @@ -102,23 +84,10 @@ To remove shard CIDs and upload CIDs separately, you'll generally do this by: - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) - Remove the shard CIDs one-by-one using `Client.capability.store.remove(shardCID)` - CLI: - - If you registered a content CID you want to remove using `w3 can upload add \`… - - (If you don't know which shard CIDs are associated with the content CID) Run `w3 can upload ls \ --shards, which returns a list of shard CIDs - - Remove it using `w3 can upload rm \` + - If you registered a content CID you want to remove using `w3 can upload add ``… + - (If you don't know which shard CIDs are associated with the content CID) Run `w3 can upload ls `` --shards, which returns a list of shard CIDs + - Remove it using `w3 can upload rm `` - Remove the shard CIDs that you'd like to - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) - - Remove the shard CIDs one-by-one using `w3 can store rm \` - -**Using the web console** - -If you want to remove uploads via the web app, first navigate to the web console. - -\ - -Click the checkbox next to the upload CIDs you'd like to remove. - -\ - -By default, the underlying shard CIDs are also selected, which you can see by expanding the content CID using the dropdown arrow. If you would like to retain the shard CIDs, click "Unselect shard CIDs" next to the upload CID. Note that the shard CIDs must be deleted to no longer pay for their storage, and that most users will want to always delete the shard CIDs associated with their upload CIDs (e.g., if you've only uploaded content through the web console). + - Remove the shard CIDs one-by-one using `w3 can store rm `` -\ From 2e9a2f96339ecd487356e5becc54e88b3b73cb2a Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 14:28:33 +0000 Subject: [PATCH 23/25] chore: moves n fixes License: MIT Signed-off-by: Oli Evans --- src/pages/docs/_meta.json | 6 +++--- src/pages/docs/concepts/architecture-options.md | 2 +- src/pages/docs/concepts/car.md | 2 +- src/pages/docs/concepts/ipfs-gateways.md | 2 +- src/pages/docs/concepts/upload-vs-store.md | 2 +- src/pages/docs/{how-tos => how-to}/_meta.json | 0 src/pages/docs/{how-tos => how-to}/create-account.md | 11 +++++------ src/pages/docs/{how-tos => how-to}/list.md | 0 src/pages/docs/{how-tos => how-to}/plan.md | 0 src/pages/docs/{how-tos => how-to}/receipts.md | 0 src/pages/docs/{how-tos => how-to}/remove.md | 0 src/pages/docs/{how-tos => how-to}/retrieve.md | 2 +- src/pages/docs/{how-tos => how-to}/upload.md | 4 +--- 13 files changed, 14 insertions(+), 17 deletions(-) rename src/pages/docs/{how-tos => how-to}/_meta.json (100%) rename src/pages/docs/{how-tos => how-to}/create-account.md (91%) rename src/pages/docs/{how-tos => how-to}/list.md (100%) rename src/pages/docs/{how-tos => how-to}/plan.md (100%) rename src/pages/docs/{how-tos => how-to}/receipts.md (100%) rename src/pages/docs/{how-tos => how-to}/remove.md (100%) rename src/pages/docs/{how-tos => how-to}/retrieve.md (99%) rename src/pages/docs/{how-tos => how-to}/upload.md (99%) diff --git a/src/pages/docs/_meta.json b/src/pages/docs/_meta.json index 5231dfa..ccc4f37 100644 --- a/src/pages/docs/_meta.json +++ b/src/pages/docs/_meta.json @@ -1,8 +1,8 @@ { "index": "Welcome", "quickstart": "Quickstart", - "how-tos": "How to", - "concepts": "Concepts", "w3cli": "Command line", - "w3up-client": "JS Client" + "w3up-client": "JS Client", + "how-to": "How to", + "concepts": "Concepts" } \ No newline at end of file diff --git a/src/pages/docs/concepts/architecture-options.md b/src/pages/docs/concepts/architecture-options.md index 79b3f56..56d7ed2 100644 --- a/src/pages/docs/concepts/architecture-options.md +++ b/src/pages/docs/concepts/architecture-options.md @@ -1,4 +1,4 @@ -# Architecture options +# Possible architectures using web3.storage to upload UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate: diff --git a/src/pages/docs/concepts/car.md b/src/pages/docs/concepts/car.md index ebf7263..9dc2953 100644 --- a/src/pages/docs/concepts/car.md +++ b/src/pages/docs/concepts/car.md @@ -1,4 +1,4 @@ -# Content Archive (CAR) files +# Working with Content Archive (CAR) files When you upload files to web3.storage using the [client library](https://web3.storage/docs/reference/js-client-library/), your data is converted into a graph of data structures, which are then packed into a format called a Content Archive (CAR) before being sent to the web3.storage service. diff --git a/src/pages/docs/concepts/ipfs-gateways.md b/src/pages/docs/concepts/ipfs-gateways.md index b96c2e5..d7fb9e4 100644 --- a/src/pages/docs/concepts/ipfs-gateways.md +++ b/src/pages/docs/concepts/ipfs-gateways.md @@ -1,4 +1,4 @@ -# IPFS Gateways +# IPFS HTTP Gateways web3.storage uses the [InterPlanetary File System (IPFS)](https://ipfs.io/) as a key part of its storage and retrieval infrastructure. diff --git a/src/pages/docs/concepts/upload-vs-store.md b/src/pages/docs/concepts/upload-vs-store.md index 0609727..32deef9 100644 --- a/src/pages/docs/concepts/upload-vs-store.md +++ b/src/pages/docs/concepts/upload-vs-store.md @@ -1,4 +1,4 @@ -# Upload vs. Store in web3.storage +# Upload vs. Store capabilities in web3.storage There are two types of content identifiers (CIDs) that web3.storage interacts with: diff --git a/src/pages/docs/how-tos/_meta.json b/src/pages/docs/how-to/_meta.json similarity index 100% rename from src/pages/docs/how-tos/_meta.json rename to src/pages/docs/how-to/_meta.json diff --git a/src/pages/docs/how-tos/create-account.md b/src/pages/docs/how-to/create-account.md similarity index 91% rename from src/pages/docs/how-tos/create-account.md rename to src/pages/docs/how-to/create-account.md index 374bb19..c908190 100644 --- a/src/pages/docs/how-tos/create-account.md +++ b/src/pages/docs/how-to/create-account.md @@ -1,13 +1,12 @@ - -# Create Account and Space - -## How to create an account +# How to create an account In the How-tos section of the docs, we show the most familiar, straightforward way to use web3.storage: by setting up an account for you, the developer, that you use in your application. For an overview of the various ways web3.storage can be integrated with your application, check out the "Architecture options" concepts section. You first need to create and register a Space you'll upload to. A Space acts as a namespace for your uploads. It is created locally and associated with a private key, and is then registered with web3.storage and associated with your email address. But don't worry about keeping track of the Space's private key! web3.storage's email authorization system allows this private key to be treated as a throwaway key. -## The easiest way to create and register a Space is by using the CLI +## Using the CLI + +The easiest way to create and register a Space is by using the CLI 1. Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli` 2. Run `w3 authorize [alice@example.com](mailto:alice@example.com)` in the command line using your email address. This will send an email to your inbox with a link for validation. @@ -15,7 +14,7 @@ You first need to create and register a Space you'll upload to. A Space acts as 4. Create a new Space for storing your data and register it: w3 space create -## `w3 space register` +### `w3 space register` Separately, you can visit console.web3.storage, sign up with your email and select a plan, and create a space using the UI, but we recommend that developers get familiar with the CLI since it's a powerful tool for many things you might want to do. diff --git a/src/pages/docs/how-tos/list.md b/src/pages/docs/how-to/list.md similarity index 100% rename from src/pages/docs/how-tos/list.md rename to src/pages/docs/how-to/list.md diff --git a/src/pages/docs/how-tos/plan.md b/src/pages/docs/how-to/plan.md similarity index 100% rename from src/pages/docs/how-tos/plan.md rename to src/pages/docs/how-to/plan.md diff --git a/src/pages/docs/how-tos/receipts.md b/src/pages/docs/how-to/receipts.md similarity index 100% rename from src/pages/docs/how-tos/receipts.md rename to src/pages/docs/how-to/receipts.md diff --git a/src/pages/docs/how-tos/remove.md b/src/pages/docs/how-to/remove.md similarity index 100% rename from src/pages/docs/how-tos/remove.md rename to src/pages/docs/how-to/remove.md diff --git a/src/pages/docs/how-tos/retrieve.md b/src/pages/docs/how-to/retrieve.md similarity index 99% rename from src/pages/docs/how-tos/retrieve.md rename to src/pages/docs/how-to/retrieve.md index 33a49e4..cef22fa 100644 --- a/src/pages/docs/how-tos/retrieve.md +++ b/src/pages/docs/how-to/retrieve.md @@ -1,4 +1,4 @@ -# Retrieve +# How to retrieve data from web3.storage In this how-to guide, you'll learn several methods for retrieving data from web3.storage. diff --git a/src/pages/docs/how-tos/upload.md b/src/pages/docs/how-to/upload.md similarity index 99% rename from src/pages/docs/how-tos/upload.md rename to src/pages/docs/how-to/upload.md index 69296fe..8d38103 100644 --- a/src/pages/docs/how-tos/upload.md +++ b/src/pages/docs/how-to/upload.md @@ -1,6 +1,4 @@ -# Upload - -## How to upload data using web3.storage +# How to upload data using web3.storage In this how-to guide, you'll learn how to store data programmatically for your development projects using the web3.storage client library in JavaScript using your (developer-owned) Space. This includes various architecture options for the data pipeline for your users to upload to web3.storage, which then makes your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin. From b9e9d75131c32e4b7874846d6c1949a31548774b Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 14:55:25 +0000 Subject: [PATCH 24/25] chore: fix remove code blocks License: MIT Signed-off-by: Oli Evans --- src/pages/docs/how-to/remove.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/pages/docs/how-to/remove.md b/src/pages/docs/how-to/remove.md index 5ba0560..765a62c 100644 --- a/src/pages/docs/how-to/remove.md +++ b/src/pages/docs/how-to/remove.md @@ -84,10 +84,10 @@ To remove shard CIDs and upload CIDs separately, you'll generally do this by: - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) - Remove the shard CIDs one-by-one using `Client.capability.store.remove(shardCID)` - CLI: - - If you registered a content CID you want to remove using `w3 can upload add ``… - - (If you don't know which shard CIDs are associated with the content CID) Run `w3 can upload ls `` --shards, which returns a list of shard CIDs - - Remove it using `w3 can upload rm `` + - If you registered a content CID you want to remove using `w3 can upload add `… + - (If you don't know which shard CIDs are associated with the content CID) Run `w3 can upload ls --shards`, which returns a list of shard CIDs + - Remove it using `w3 can upload rm ` - Remove the shard CIDs that you'd like to - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) - - Remove the shard CIDs one-by-one using `w3 can store rm `` + - Remove the shard CIDs one-by-one using `w3 can store rm ` From f7b3e26bf98f10bc84151d379021b4514568b56c Mon Sep 17 00:00:00 2001 From: Oli Evans Date: Mon, 13 Nov 2023 14:57:53 +0000 Subject: [PATCH 25/25] chore: Apply suggestions from code review Co-authored-by: Vasco Santos --- src/pages/docs/concepts/architecture-options.md | 4 ++-- src/pages/docs/concepts/car.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/pages/docs/concepts/architecture-options.md b/src/pages/docs/concepts/architecture-options.md index 56d7ed2..1ce7606 100644 --- a/src/pages/docs/concepts/architecture-options.md +++ b/src/pages/docs/concepts/architecture-options.md @@ -3,7 +3,7 @@ UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate: - Client-server: You (the developer) own the Space, and your user uploads to your backend infra before you upload it to the service -- Delegated: You own the Space, but you give a delegated UCAN token to your user's Agent to upload directly to the service (rather than needing to touch the upload in your backend) +- Delegated: You own the Space, but you give a delegated UCAN token to your user's Agent to upload directly to the service, rather than needing to proxy the upload through your backend (no egress from your infrastructure) - User-owned: Your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it In the How-tos section of the docs, we focused on the first two options, as they are the most familiar today. However, you can implement each of these in a number of ways, but we talk through some considerations when implementing a given option. @@ -133,4 +133,4 @@ User-\>\>web3.storage w3up service: Upload data using w3up-client - Doing this does take some of the UX out of your control; for instance, when web3.storage fully launches with w3up, your users will have to set up their payment methods with web3.storage - Note that this alone does not give visibility into which of your end users are uploading what; to track this, you'll probably need them to send you that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking) - There is a world of possibilities with your users "bringing their own identity" for their Space; you could explore how crypto wallet private keys, Apple Passkey, and more might map to Space DIDs and have the client use those -- If you have code snippet(s) that works for you, please share them in a PR or [Github issue](https://github.com/web3-storage/w3up/issues) and we'll link them here! +- If you have code snippet(s) that works for you, please share them in a PR or [Github issue](https://github.com/web3-storage/www/issues) and we'll link them here! diff --git a/src/pages/docs/concepts/car.md b/src/pages/docs/concepts/car.md index 9dc2953..abb46de 100644 --- a/src/pages/docs/concepts/car.md +++ b/src/pages/docs/concepts/car.md @@ -1,6 +1,6 @@ # Working with Content Archive (CAR) files -When you upload files to web3.storage using the [client library](https://web3.storage/docs/reference/js-client-library/), your data is converted into a graph of data structures, which are then packed into a format called a Content Archive (CAR) before being sent to the web3.storage service. +When you upload files to web3.storage using the client library, your data is converted into a graph of data structures, which are then packed into a format called a Content Archive (CAR) before being sent to the web3.storage service. For most use cases, you never need to know about this process, as the conversion happens behind the scenes when using the client library. However, you might want to know more about how to manipulate CAR files directly, especially if you're using `upload` and `store` separately in web3.storage. @@ -20,7 +20,7 @@ This format uses an IPLD "codec" called [dag-pb](https://ipld.io/docs/codecs/kno First, formatting everything on the client allows us to calculate the root Content Identifier for the data you're uploading before we send any data to the remote service. This means that you can compare the CID returned by the web3.storage service to the one you calculated locally, and you don't have to trust the service to do the right thing. -Another reason to use CARs is to support large files, which would otherwise hit size limits on the web3.storage backend platform. The data in a CAR is already chunked into small blocks, which makes CARs easy to shard into small pieces that can be uploaded in batches. +Another reason to use CARs is to support large files, which would otherwise hit size limits on the web3.storage backend platform. The data in a CAR is already chunked into small blocks, which makes CARs easy to shard into small pieces that can be uploaded in batches. This also enables the web3.storage platform to get larger content into Filecoin deals. CAR files are a format that pretty much any IPFS tool or implementation can interact with. You can export data from your personal IPFS node into a CAR file and upload it to web3.storage using `w3 up --car` or `client.uploadCar`. As a result, we dive into the various ways you might interact with CAR files.