Skip to content

Commit

Permalink
Content Loader API: include docs for the built-in loaders (#10710)
Browse files Browse the repository at this point in the history
* feat(content-loader): add docs for built-in loaders in the reference
* rewording and add details about generateId options
* move built-in loaders after what is a loader
* mention YAML support for glob
* rename two headings
* rewording and add a tool to test the pattern
* add links from the content collections guide to the reference page
* add missing link to the glob tool...
* fix broken links
* update fileName description
* oxford commas

@lunaria-track:src/content/docs/en/guides/content-collections.mdx;src/content/docs/en/reference/content-loader-reference.mdx;src/content/docs/en/reference/modules/astro-content.mdx

---------

Co-authored-by: Sarah Rainsberger <[email protected]>
Co-authored-by: liruifengv <[email protected]>
Co-authored-by: yanthomasdev <[email protected]>
  • Loading branch information
4 people authored Jan 20, 2025
1 parent 00323b3 commit 9807f1a
Show file tree
Hide file tree
Showing 4 changed files with 140 additions and 6 deletions.
6 changes: 3 additions & 3 deletions src/content/docs/en/guides/content-collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,11 +83,11 @@ The Content Layer API allows you to fetch your content (whether stored locally i

#### Built-in loaders

Astro provides two built-in loader functions (`glob()` and `file()`) for fetching your local content, as well as access to the API to construct your own loader and fetch remote data.
Astro provides [two built-in loader functions](/en/reference/content-loader-reference/#built-in-loaders) (`glob()` and `file()`) for fetching your local content, as well as access to the API to construct your own loader and fetch remote data.

The `glob()` loader creates entries from directories of Markdown, MDX, Markdoc, or JSON files from anywhere on the filesystem. It accepts a `pattern` of entry files to match using glob patterns supported by [micromatch](https://github.com/micromatch/micromatch#matching-features), and a base file path of where your files are located. Each entry's `id` will be automatically generated from its file name. Use this loader when you have one file per entry.
The [`glob()` loader](/en/reference/content-loader-reference/#glob-loader) creates entries from directories of Markdown, MDX, Markdoc, JSON, or YAML files from anywhere on the filesystem. It accepts a `pattern` of entry files to match using glob patterns supported by [micromatch](https://github.com/micromatch/micromatch#matching-features), and a base file path of where your files are located. Each entry's `id` will be automatically generated from its file name. Use this loader when you have one file per entry.

The `file()` loader creates multiple entries from a single local file. Each entry in the file must have a unique `id` key property. It accepts a `base` file path to your file and optionally a [`parser` function](#parser-function) for data files it cannot parse automatically. Use this loader when your data file can be parsed as an array of objects.
The [`file()` loader](/en/reference/content-loader-reference/#file-loader) creates multiple entries from a single local file. Each entry in the file must have a unique `id` key property. It accepts a `base` file path to your file and optionally a [`parser` function](#parser-function) for data files it cannot parse automatically. Use this loader when your data file can be parsed as an array of objects.

```ts title="src/content.config.ts" {5,9}
import { defineCollection, z } from 'astro:content';
Expand Down
136 changes: 135 additions & 1 deletion src/content/docs/en/reference/content-loader-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,140 @@ Astro loaders allow you to load data into [content collections](/en/guides/conte

Each collection needs [a loader defined in its schema](/en/guides/content-collections/#defining-the-collection-loader). You can define a loader inline in your project's `src/content.config.ts` file, share one loader between multiple collections, or even [publish your loader to NPM as a package](/en/reference/publish-to-npm/) to share with others and be included in our integrations library.

## Built-in loaders

Astro provides two built-in loaders to help you fetch your collections. Both offer options to suit a wide range of use cases.

### `glob()` loader

<p>

**Type:** <code>(options: GlobOptions) => <a href="#the-loader-object">Loader</a></code><br />
<Since v="5.0.0" />
</p>

The `glob()` loader creates entries from directories of files from anywhere on the filesystem. The supported file types are Markdown, MDX, Markdoc, JSON, and YAML files.

This loader accepts an object with the following properties: `pattern`, `base` (optional), and `generateId` (optional).

```ts title="src/content.config.ts" {2,6,11,17-21}
import { defineCollection } from 'astro:content';
import { glob } from 'astro/loaders';

const pages = defineCollection({
/* Retrieve all Markdown files in your pages directory. */
loader: glob({ pattern: "**/*.md", base: "./src/data/pages" }),
schema: /* ... */
});
const blog = defineCollection({
/* Retrieve all Markdown and MDX files in your blog directory. */
loader: glob({ pattern: "**/*.(md|mdx)", base: "./src/data/blog" }),
schema: /* ... */
});
const authors = defineCollection({
/* Retrieve all JSON files in your authors directory while retaining
* uppercase letters in the ID. */
loader: glob({
pattern: '**/*.json',
base: "./src/data/authors",
generateId: ({ entry }) => entry.replace(/\.json$/, ''),
}),
schema: /* ... */
});
```

#### `pattern`

<p>

**Type:** `string | string[]`
</p>

The `pattern` property accepts a string or an array of strings using glob matching (e.g. wildcards, globstars). The patterns must be relative to the base directory of entry files to match.

You can learn more about the syntax to use in the [micromatch documentation](https://github.com/micromatch/micromatch#matching-features). You can also verify the validity of your pattern using an online tool like the [DigitalOcean Glob Tool](https://www.digitalocean.com/community/tools/glob).

#### `base`

<p>

**Type:** `string | URL`<br />
**Default:** `"."`
</p>

A relative path or [URL](https://developer.mozilla.org/en-US/docs/Web/API/URL) to the directory from which to resolve the `pattern`.

#### `generateId()`

<p>

**Type:** `(options: GenerateIdOptions) => string`
</p>

A callback function that returns a unique string per entry in a collection. It accepts an object as parameter with the following properties:
* `entry` - the path to the entry file, relative to the base directory
* `base` - the base directory [URL](https://developer.mozilla.org/en-US/docs/Web/API/URL)
* `data` - the parsed, unvalidated data of the entry

By default it uses [`github-slugger`](https://github.com/Flet/github-slugger) to generate a slug with [kebab-cased](https://developer.mozilla.org/en-US/docs/Glossary/Kebab_case) words.

### `file()` loader

<p>

**Type:** <code>(fileName: string, options?: FileOptions) => <a href="#the-loader-object">Loader</a></code><br />
<Since v="5.0.0" />
</p>

The `file()` loader creates entries from a single file that contains an array of objects with a unique `id` field, or an object with string keys. It supports JSON or YAML, and you can provide a custom `parser` for data files it cannot parse by default.

This loader accepts a `fileName` property and an optional object as second argument:

```ts title="src/content.config.ts" {2,6,11-13}
import { defineCollection } from 'astro:content';
import { file } from 'astro/loaders';

const authors = defineCollection({
/* Retrieve all entries from a JSON file. */
loader: file("src/data/authors.json"),
schema: /* ... */
});
const products = defineCollection({
/* Retrieve all entries from a CSV file using a custom parser. */
loader: file("src/data/products.csv", {
parser: (fileContent) => { /* your parser logic */ },
}),
schema: /* ... */
});
```

#### `fileName`

<p>

**Type:** `string`
</p>

Sets the path to the file to load, relative to the root directory.

#### Options

<p>

**Type:** `FileOptions`
</p>

An optional object with the following properties:

##### `parser()`

<p>

**Type:** `(text: string) => Record<string, Record<string, unknown>> | Array<Record<string, unknown>>`
</p>

A callback function to create a collection from a file’s contents. Use it when you need to process file not supported by default (e.g. `.csv`) or when using [nested `.json` documents](/en/guides/content-collections/#nested-json-documents).

## Loader types

Loaders can be defined either as a simple function that returns an array of entries or with the more powerful object Content Loader API for more control over the loading process.
Expand Down Expand Up @@ -88,7 +222,7 @@ const blog = defineCollection({
});
```

## Loader API
## Object loader API

The API for [inline loaders](#inline-loaders) is very simple, and is shown above. This section shows the API for defining an object loader.

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/en/reference/modules/astro-content.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ This function accepts the following properties:

<p>

**Type:** <code>() => Promise&lt;Array&lt;\{ id: string, [key: string]: any }&gt; | Record&lt;string, Record&lt;string, any&gt;&gt;&gt; | <a href="/en/reference/content-loader-reference/#loader-api">Loader</a></code>
**Type:** <code>() => Promise&lt;Array&lt;\{ id: string, [key: string]: any }&gt; | Record&lt;string, Record&lt;string, any&gt;&gt;&gt; | <a href="/en/reference/content-loader-reference/#object-loader-api">Loader</a></code>
<Since v="5.0.0" />
</p>

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/zh-cn/reference/modules/astro-content.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ export const collections = { blog };

<p>

**Type:** <code>() => Promise&lt;Array&lt;\{ id: string, [key: string]: any }&gt; | Record&lt;string, Record&lt;string, any&gt;&gt;&gt; | <a href="/zh-cn/reference/content-loader-reference/#loader-api">Loader</a></code>
**Type:** <code>() => Promise&lt;Array&lt;\{ id: string, [key: string]: any }&gt; | Record&lt;string, Record&lt;string, any&gt;&gt;&gt; | <a href="/zh-cn/reference/content-loader-reference/#object-loader-api">Loader</a></code>
<Since v="5.0.0" />
</p>

Expand Down

0 comments on commit 9807f1a

Please sign in to comment.