Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use Indexing Service when feature flag is present #132

Merged
merged 9 commits into from
Dec 3, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,790 changes: 1,830 additions & 960 deletions package-lock.json

Large diffs are not rendered by default.

8 changes: 5 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
"@microlabs/otel-cf-workers": "^1.0.0-rc.48",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/sdk-trace-base": "^1.27.0",
"@storacha/indexing-service-client": "^1.1.3",
"@ucanto/client": "^9.0.1",
"@ucanto/principal": "^9.0.1",
"@ucanto/transport": "^9.1.1",
Expand All @@ -51,17 +52,18 @@
},
"devDependencies": {
"@cloudflare/workers-types": "^4.20231218.0",
"@storacha/cli": "^1.0.1",
"@storacha/client": "file:../upload-service/packages/w3up-client",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs update before merge.

"@types/chai": "^5.0.0",
"@types/mocha": "^10.0.9",
"@types/node-fetch": "^2.6.11",
"@types/sinon": "^17.0.3",
"@web3-storage/content-claims": "^5.0.0",
"@web3-storage/public-bucket": "^1.1.0",
"@web3-storage/upload-client": "^16.1.1",
"@web3-storage/w3cli": "^7.8.2",
"carstream": "^2.1.0",
"chai": "^5.1.1",
"esbuild": "^0.18.20",
"esbuild": "^0.24.0",
"files-from-path": "^0.2.6",
"miniflare": "^3.20240909.5",
"mocha": "^10.7.3",
Expand All @@ -70,7 +72,7 @@
"standard": "^17.1.0",
"tree-kill": "^1.2.2",
"typescript": "^5.6.3",
"wrangler": "^3.86.1"
"wrangler": "^3.90.0"
},
"standard": {
"ignore": [
Expand Down
4 changes: 2 additions & 2 deletions scripts/delegate-serve.d.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
declare module '@web3-storage/w3cli/lib.js' {
import { Client } from '@web3-storage/w3up-client'
declare module '@storacha/cli/lib.js' {
import { Client } from '@storacha/client'
export declare function getClient(): Promise<Client>
}
188 changes: 131 additions & 57 deletions scripts/delegate-serve.js
Original file line number Diff line number Diff line change
@@ -1,73 +1,147 @@
import sade from 'sade'
import { getClient } from '@web3-storage/w3cli/lib.js'
import { getClient } from '@storacha/cli/lib.js'
import { Space } from '@web3-storage/capabilities'
import * as serve from '../src/capabilities/serve.js'
/**
* @import * as Ucanto from '@ucanto/interface'
*/

const cli = sade('delegate-serve.js [space] [token] [accountDID] [gatewayDID]')
/**
* @template {string} Method
* @param {string} str
* @param {Method} [method]
* @returns {str is Ucanto.DID<Method>}
*/
const isDID = (str, method) =>
str.startsWith(`did:${method ? `${method}:` : ''}`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use ucanto schema for this.

https://github.com/storacha/ucanto/blob/6de27c0f19790d9def1df2f2e299fa4f0996ded9/packages/core/src/schema/did.js#L8-L25

import { Schema } from '@ucanto/core'

Schema.did({ method }).is(str)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha! I couldn't figure out how to get that to work before. Not sure what wasn't clicking, but it all makes sense now. Pushed as Use Ucanto Schema.


cli
.option('--space', 'The space DID to delegate. If not provided, a new space will be created.')
.option('--token', 'The auth token to use. If not provided, the delegation will not be authenticated.')
/**
* @param {string} str
* @returns {str is `did:mailto:${string}:${string}`}
*/
const isMailtoDID = (str) => /^did:mailto:.*:.*$/.test(str)

sade('delegate-serve.js [space]')
.option(
'--token',
'The auth token to use. If not provided, the delegation will not be authenticated.'
)
.option('--accountDID', 'The account DID to use when creating a new space.')
.option('--gatewayDID', 'The gateway DID to use when delegating the space/content/serve capability. Defaults to did:web:staging.w3s.link.')
.option(
'--gatewayDID',
'The gateway DID to use when delegating the space/content/serve capability. Defaults to did:web:staging.w3s.link.'
)
.describe(
`Delegates ${Space.contentServe.can} to the Gateway for a test space generated by the script, with an optional auth token. Outputs a base64url string suitable for the stub_delegation query parameter. Pipe the output to pbcopy or similar for the quickest workflow.`
`Delegates ${Space.contentServe.can} to the Gateway for a test space generated by the script, with an optional auth token. Outputs a base64url string suitable for the stub_delegation query parameter.`
)
.action(async (space, token, accountDID, gatewayDID, options) => {
const { space: spaceOption, token: tokenOption, accountDID: accountDIDOption, gatewayDID: gatewayDIDOption } = options
space = spaceOption || undefined
token = tokenOption || undefined
accountDID = accountDIDOption || undefined
gatewayDID = gatewayDIDOption || 'did:web:staging.w3s.link'
const client = await getClient()
.action(
/**
* @param {string} [space]
* @param {object} [options]
* @param {string} [options.token]
* @param {string} [options.accountDID]
* @param {string} [options.gatewayDID]
*/
async (
space,
{ token, accountDID, gatewayDID = 'did:web:staging.w3s.link' } = {}
) => {
const client = await getClient()

space ??= await createSpace(client, accountDID)

let spaceDID
let proofs = []
if (!space) {
const provider = /** @type {`did:web:${string}`} */ (client.defaultProvider())
const account = client.accounts()[accountDID]
const newSpace = await client.agent.createSpace('test')
const provision = await account.provision(newSpace.did(), { provider })
if (provision.error) throw provision.error
await newSpace.save()
const authProof = await newSpace.createAuthorization(client.agent)
proofs = [authProof]
spaceDID = newSpace.did()
} else {
client.addSpace(space)
spaceDID = space
proofs = client.proofs([
if (!isDID(space)) {
throw new Error(`Invalid space DID: ${space}`)
}

const proofs = client.proofs([
{
can: Space.contentServe.can,
with: spaceDID
}
with: space,
},
])
}

/** @type {import('@ucanto/client').Principal<`did:${string}:${string}`>} */
const gatewayIdentity = {
did: () => gatewayDID
}
if (proofs.length === 0) {
throw new Error(
`No proofs found. Are you authorized to ${serve.star.can} ${space}?`
)
}

// @ts-expect-error - The client still needs to be updated to support the capability type
const delegation = await client.createDelegation(gatewayIdentity, [Space.contentServe.can], {
expiration: Infinity,
proofs
})
if (!isDID(gatewayDID)) {
throw new Error(`Invalid gateway DID: ${gatewayDID}`)
}

await client.capability.access.delegate({
delegations: [delegation]
})
const gatewayIdentity = {
did: () => gatewayDID,
}

const carResult = await delegation.archive()
if (carResult.error) throw carResult.error
const base64Url = Buffer.from(carResult.ok).toString('base64url')
process.stdout.write(`Agent Proofs: ${proofs.flatMap(p => p.capabilities).map(c => `${c.can} with ${c.with}`).join('\n')}\n`)
process.stdout.write(`Issuer: ${client.agent.issuer.did()}\n`)
process.stdout.write(`Audience: ${gatewayIdentity.did()}\n`)
process.stdout.write(`Space: ${spaceDID}\n`)
process.stdout.write(`Token: ${token ?? 'none'}\n`)
process.stdout.write(`Delegation: ${delegation.capabilities.map(c => `${c.can} with ${c.with}`).join('\n')}\n`)
process.stdout.write(`Stubs: stub_space=${spaceDID}&stub_delegation=${base64Url}&authToken=${token ?? ''}\n`)
})
// NOTE: This type assertion is wrong. It's a hack to let us use this
// ability. `client.createDelegation` currently only accepts abilities it
// knows about. That should probably be expanded, but this little script
// isn't going to be the reason to go change that, as it involves updating
// multiple packages.
const ability = /** @type {"*"} */ (Space.contentServe.can)

const delegation = await client.createDelegation(
gatewayIdentity,
[ability],
{
expiration: Infinity,
proofs,
}
)

await client.capability.access.delegate({
delegations: [delegation],
})

const carResult = await delegation.archive()
if (carResult.error) throw carResult.error
const base64Url = Buffer.from(carResult.ok).toString('base64url')
process.stdout.write(
`Agent Proofs: ${proofs
.flatMap((p) => p.capabilities)
.map((c) => `${c.can} with ${c.with}`)
.join('\n')}\n`
)
process.stdout.write(`Issuer: ${client.agent.issuer.did()}\n`)
process.stdout.write(`Audience: ${gatewayIdentity.did()}\n`)
process.stdout.write(`Space: ${space}\n`)
process.stdout.write(`Token: ${token ?? 'none'}\n`)
process.stdout.write(
`Delegation: ${delegation.capabilities
.map((c) => `${c.can} with ${c.with}`)
.join('\n')}\n`
)
process.stdout.write(
`Stubs: stub_space=${space}&stub_delegation=${base64Url}&authToken=${
token ?? ''
}\n`
)
}
)
.parse(process.argv)

cli.parse(process.argv)
/**
* @param {import('@storacha/client').Client} client
* @param {string} [accountDID]
*/
async function createSpace(client, accountDID) {
const provider = client.defaultProvider()
if (!isDID(provider, 'web')) {
throw new Error(`Invalid provider DID: ${provider}`)
}
if (!accountDID) {
throw new Error('Must provide an account DID to create a space')
}
if (!isMailtoDID(accountDID)) {
throw new Error(`Invalid account DID: ${accountDID}`)
}
const account = client.accounts()[accountDID]
const newSpace = await client.agent.createSpace('test')
const provision = await account.provision(newSpace.did(), { provider })
if (provision.error) throw provision.error
await newSpace.save()
await newSpace.createAuthorization(client.agent)
return newSpace.did()
}
40 changes: 8 additions & 32 deletions src/middleware/withAuthorizedSpace.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ import * as serve from '../capabilities/serve.js'

/**
* @import * as Ucanto from '@ucanto/interface'
* @import { Locator } from '@web3-storage/blob-fetcher'
* @import { IpfsUrlContext, Middleware } from '@web3-storage/gateway-lib'
* @import { LocatorContext } from './withLocator.types.js'
* @import { AuthTokenContext } from './withAuthToken.types.js'
Expand All @@ -27,7 +26,7 @@ import * as serve from '../capabilities/serve.js'
* >
* )}
*/
export function withAuthorizedSpace (handler) {
export function withAuthorizedSpace(handler) {
return async (request, env, ctx) => {
const { locator, dataCid } = ctx
const locRes = await locator.locate(dataCid.multihash)
Expand Down Expand Up @@ -68,7 +67,7 @@ export function withAuthorizedSpace (handler) {
...ctx,
space: selectedSpace,
delegationProofs,
locator: spaceScopedLocator(locator, selectedSpace)
locator: locator.scopeToSpaces([selectedSpace]),
})
} catch (error) {
// If all Spaces failed to authorize, throw the first error.
Expand Down Expand Up @@ -98,7 +97,7 @@ const authorize = async (space, ctx) => {
const relevantDelegationsResult = await ctx.delegationsStorage.find({
audience: ctx.gatewayIdentity.did(),
can: serve.transportHttp.can,
with: space
with: space,
})

if (relevantDelegationsResult.error) return relevantDelegationsResult
Expand All @@ -110,9 +109,9 @@ const authorize = async (space, ctx) => {
audience: ctx.gatewayIdentity,
with: space,
nb: {
token: ctx.authToken
token: ctx.authToken,
},
proofs: relevantDelegationsResult.ok
proofs: relevantDelegationsResult.ok,
})
.delegate()

Expand All @@ -121,7 +120,7 @@ const authorize = async (space, ctx) => {
capability: serve.transportHttp,
authority: ctx.gatewayIdentity,
principal: Verifier,
validateAuthorization: () => ok({})
validateAuthorization: () => ok({}),
})
if (accessResult.error) {
return accessResult
Expand All @@ -130,30 +129,7 @@ const authorize = async (space, ctx) => {
return {
ok: {
space,
delegationProofs: relevantDelegationsResult.ok
}
delegationProofs: relevantDelegationsResult.ok,
},
}
}

/**
* Wraps a {@link Locator} and locates content only from a specific Space.
*
* @param {Locator} locator
* @param {Ucanto.DID} space
* @returns {Locator}
*/
const spaceScopedLocator = (locator, space) => ({
locate: async (digest) => {
const locateResult = await locator.locate(digest)
if (locateResult.error) {
return locateResult
} else {
return {
ok: {
...locateResult.ok,
site: locateResult.ok.site.filter((site) => site.space === space)
}
}
}
}
})
17 changes: 2 additions & 15 deletions src/middleware/withDelegationStubs.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { Delegation } from '@ucanto/core'
import { spaceScopedLocator } from '../util.js'

/**
* @import * as Ucanto from '@ucanto/interface'
Expand Down Expand Up @@ -53,21 +54,7 @@ export const withDelegationStubs = (handler) => async (request, env, ctx) => {
delegationProofs: [], // Delegation proofs are set by withAuthorizedSpace handler
locator:
stubSpace && isDIDKey(stubSpace)
? {
locate: async (digest, options) => {
const locateResult = await ctx.locator.locate(digest, options)
if (locateResult.error) return locateResult
return {
ok: {
...locateResult.ok,
site: locateResult.ok.site.map((site) => ({
...site,
space: stubSpace
}))
}
}
}
}
? spaceScopedLocator(ctx.locator, [stubSpace])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a stubSpace?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Space given as a stub (stub_space).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha - for testing right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly—that's what this middleware is for.

: ctx.locator
})
}
Expand Down
Loading
Loading