Skip to content

Commit

Permalink
add info about + prefix and splits config page (#6932)
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Feb 26, 2025
2 parents 70d5e96 + bd14bc1 commit c58eb66
Show file tree
Hide file tree
Showing 12 changed files with 271 additions and 208 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ is_featured: false

We’re announcing that [dbt Server](https://github.com/dbt-labs/dbt-server) is officially deprecated and will no longer be maintained by dbt Labs going forward. You can continue to use the repository and fork it for your needs. We’re also looking for a maintainer of the repository from our community! If you’re interested, please reach out by opening an issue in the [repository](https://github.com/dbt-labs/dbt-server/issues).


<!-- truncate -->
## Why are we deprecating dbt Server?

At dbt Labs, we are continually working to build rich experiences that help our users scale collaboration around data. To achieve this vision, we need to take moments to think about which products are contributing to this goal, and sometimes make hard decisions about the ones that are not, so that we can focus on enhancing the most impactful ones.
Expand Down
203 changes: 6 additions & 197 deletions website/docs/reference/configs-and-properties.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
---
title: Configs, properties, what are they?
title: Configurations and properties, what are they?
sidebar_label: Configs and properties
intro_text: "Understand the difference between properties and configurations in dbt: properties describe resources, while configurations control how dbt builds them in the warehouse."
description: "This page explains the difference between properties and configurations in dbt."
pagination_next: "reference/define-configs"
---

Resources in your project—models, snapshots, seeds, tests, and the rest—can have a number of declared **properties**. Resources can also define **configurations**, which are a special kind of property that bring extra abilities. What's the distinction?
Resources in your project—models, snapshots, seeds, tests, and the rest—can have a number of declared _properties_. Resources can also define _configurations_ (configs), which are a special kind of property that bring extra abilities. What's the distinction?
- Properties are declared for resources one-by-one in `properties.yml` files. Configs can be defined there, nested under a `config` property. They can also be set one-by-one via a `config()` macro (right within `.sql` files), and for many resources at once in `dbt_project.yml`.
- Because configs can be set in multiple places, they are also applied hierarchically. An individual resource might _inherit_ or _override_ configs set elsewhere.
- You can select resources based on their config values using the `config:` selection method, but not the values of non-config properties.
Expand All @@ -21,198 +25,3 @@ Whereas you can use **configurations** to:
* Declare where a seed will be created in the database (`<database>.<schema>.<alias>`)
* Declare whether a resource should persist its descriptions as comments in the database
* Apply tags and "meta" properties

## Where can I define configs?

Depending on the resource type, configurations can be defined in the dbt project and also in an installed package by:

<VersionBlock firstVersion="1.9">

1. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses`, `tests/`, and more.
2. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, etc)
</VersionBlock>

<VersionBlock lastVersion="1.8">

1. Using a [`config()` Jinja macro](/reference/dbt-jinja-functions/config) within a `model`, `snapshot`, or `test` SQL file
2. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses/`, or `tests/` directory.
3. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, etc)
</VersionBlock>

### Config inheritance

The most specific config always takes precedence. This generally follows the order above: an in-file `config()` block --> properties defined in a `.yml` file --> config defined in the project file.

Note - Generic data tests work a little differently when it comes to specificity. See [test configs](/reference/data-test-configs).

Within the project file, configurations are also applied hierarchically. The most specific config always takes precedence. In the project file, for example, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model or directory of models, define the [resource path](/reference/resource-configs/resource-path) as nested dictionary keys.

Configurations in your root dbt project have _higher_ precedence than configurations in installed packages. This enables you to override the configurations of installed packages, providing more control over your dbt runs.


### Combining configs

Most configurations are "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Note that a few configs have different merge behavior:
- [`tags`](/tags) are additive. If a model has some tags configured in `dbt_project.yml`, and more tags applied in its `.sql` file, the final set of tags will include all of them.
- [`meta`](/reference/resource-configs/meta) dictionaries are merged (a more specific key-value pair replaces a less specific value with the same key)
- [`pre-hook` and `post-hook`](/reference/resource-configs/pre-hook-post-hook) are also additive.

## Where can I define properties?

In dbt, you can use `properties.yml` files to define properties for resources. You can declare properties in `.yml` files, in the same directory as your resources. You can name these files `whatever_you_want.yml` and nest them arbitrarily in sub-folders within each directory.

We highly recommend that you define properties in dedicated paths alongside the resources they're describing.

:::info

#### schema.yml files

Previous versions of the docs referred to these as `schema.yml` files — we've moved away from that terminology since the word `schema` is used to mean other things when talking about databases, and people often thought that you _had_ to name these files `schema.yml`.

Instead, we now refer to these files as `properties.yml` files. (Of course, you're still free to name your files `schema.yml`)

:::

### Which properties are _not_ also configs?

In dbt, you can define node configs in `properties.yml` files, in addition to `config()` blocks and `dbt_project.yml`. However, some special properties can only be defined in the `.yml` file and you cannot configure them using `config()` blocks or the `dbt_project.yml` file:

Certain properties are special, because:

- They have a unique Jinja rendering context
- They create new project resources
- They don't make sense as hierarchical configuration
- They're older properties that haven't yet been redefined as configs

These properties are:

- [`description`](/reference/resource-properties/description)
- [`tests`](/reference/resource-properties/data-tests)
- [`docs`](/reference/resource-configs/docs)
- [`columns`](/reference/resource-properties/columns)
- [`quote`](/reference/resource-properties/quote)
- [`source` properties](/reference/source-properties) (e.g. `loaded_at_field`, `freshness`)
- [`exposure` properties](/reference/exposure-properties) (e.g. `type`, `maturity`)
- [`macro` properties](/reference/macro-properties) (e.g. `arguments`)

## Example
Here's an example that defines both `sources` and `models` for a project:

<File name='models/jaffle_shop.yml'>

```yml
version: 2

sources:
- name: raw_jaffle_shop
description: A replica of the postgres database used to power the jaffle_shop app.
tables:
- name: customers
columns:
- name: id
description: Primary key of the table
tests:
- unique
- not_null

- name: orders
columns:
- name: id
description: Primary key of the table
tests:
- unique
- not_null

- name: user_id
description: Foreign key to customers

- name: status
tests:
- accepted_values:
values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']


models:
- name: stg_jaffle_shop__customers
config:
tags: ['pii']
columns:
- name: customer_id
tests:
- unique
- not_null

- name: stg_jaffle_shop__orders
config:
materialized: view
columns:
- name: order_id
tests:
- unique
- not_null
- name: status
tests:
- accepted_values:
values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']
config:
severity: warn


```

</File>


## Related documentation
You can find an exhaustive list of each supported property and config, broken down by resource type:
* Model [properties](/reference/model-properties) and [configs](/reference/model-configs)
* Source [properties](/reference/source-properties) and [configs](source-configs)
* Seed [properties](/reference/seed-properties) and [configs](/reference/seed-configs)
* Snapshot [properties](snapshot-properties)
* Analysis [properties](analysis-properties)
* Macro [properties](/reference/macro-properties)
* Exposure [properties](/reference/exposure-properties)

## FAQs
<FAQ path="Project/schema-yml-name" />
<FAQ path="Project/resource-yml-name" />
<FAQ path="Project/multiple-resource-yml-files" />
<FAQ path="Project/properties-not-in-config" />
<FAQ path="Project/why-version-2" />
<FAQ path="Project/yaml-file-extension" />

## Troubleshooting common errors

### Invalid test config given in [model name]

This error occurs when your `.yml` file does not conform to the structure expected by dbt. A full error message might look like:
```
* Invalid test config given in models/schema.yml near {'namee': 'event', ...}
Invalid arguments passed to "UnparsedNodeUpdate" instance: 'name' is a required property, Additional properties are not allowed ('namee' was unexpected)
```

While verbose, an error like this should help you track down the issue. Here, the `name` field was provided as `namee` by accident. To fix this error, ensure that your `.yml` conforms to the expected structure described in this guide.

### Invalid syntax in your schema.yml file

If your `.yml` file is not valid yaml, then dbt will show you an error like this:

```
Runtime Error
Syntax error near line 6
------------------------------
5 | - name: events
6 | description; "A table containing clickstream events from the marketing website"
7 |
Raw Error:
------------------------------
while scanning a simple key
in "<unicode string>", line 6, column 5:
description; "A table containing clickstream events from the marketing website"
^
```

This error occurred because a semicolon (`;`) was accidentally used instead of a colon (`:`) after the `description` field. To resolve issues like this, find the `.yml` file referenced in the error message and fix any syntax errors present in the file. There are online YAML validators that can be helpful here, but please be mindful of submitting sensitive information to third-party applications!

9 changes: 8 additions & 1 deletion website/docs/reference/dbt_project.yml.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
---
description: "Reference guide for configuring the dbt_project.yml file."
intro_text: "The dbt_project.yml file is a required file for all dbt projects. It contains important information that tells dbt how to operate your project."
---

Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project.
Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project. It works as follows:

- dbt uses [YAML](https://yaml.org/) in a few different places. If you're new to YAML, it would be worth learning how arrays, dictionaries, and strings are represented.

Expand Down Expand Up @@ -96,6 +97,12 @@ vars:
</File>
## The `+` prefix

import PlusPrefix from '/snippets/_plus-prefix.md';

<PlusPrefix />

## Naming convention

It's important to follow the correct YAML naming conventions for the configs in your `dbt_project.yml` file to ensure dbt can process them properly. This is especially true for resource types with more than one word.
Expand Down
51 changes: 51 additions & 0 deletions website/docs/reference/define-configs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
title: Define configs
sidebar_label: Define configs
intro_text: "Learn how to define configurations for your resources in a dbt project"
description: "Learn how to define configurations for your resources in a dbt project"
pagination_previous: "reference/configs-and-properties"
pagination_next: "reference/define-properties"
---

Depending on the resource type, you can define configurations in a dbt project and also in an installed package by:

<VersionBlock firstVersion="1.9">

1. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses`, `tests/`, and more.
2. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, and so on)
</VersionBlock>

<VersionBlock lastVersion="1.8">

1. Using a [`config()` Jinja macro](/reference/dbt-jinja-functions/config) within a `model`, `snapshot`, or `test` SQL file
2. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses/`, or `tests/` directory.
3. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, and so on)
</VersionBlock>

## Config inheritance

The most specific config always takes precedence. This generally follows the order above: an in-file `config()` block --> properties defined in a `.yml` file --> config defined in the project file.

Note - Generic data tests work a little differently when it comes to specificity. See [test configs](/reference/data-test-configs).

Within the project file, configurations are also applied hierarchically. The most specific config always takes precedence. In the project file, for example, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model or directory of models, define the [resource path](/reference/resource-configs/resource-path) as nested dictionary keys.

Configurations in your root dbt project have _higher_ precedence than configurations in installed packages. This enables you to override the configurations of installed packages, providing more control over your dbt runs.

## Combining configs

Most configurations are "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Note that a few configs have different merge behavior:
- [`tags`](/tags) are additive. If a model has some tags configured in `dbt_project.yml`, and more tags applied in its `.sql` file, the final set of tags will include all of them.
- [`meta`](/reference/resource-configs/meta) dictionaries are merged (a more specific key-value pair replaces a less specific value with the same key)
- [`pre-hook` and `post-hook`](/reference/resource-configs/pre-hook-post-hook) are also additive.

## The `+` prefix

import PlusPrefix from '/snippets/_plus-prefix.md';

<PlusPrefix />


import Example from '/snippets/_configs-properties.md' ;

<Example />
47 changes: 47 additions & 0 deletions website/docs/reference/define-properties.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: Define properties
sidebar_label: Define properties
intro_text: "Learn how to define properties for your resources in a properties.yml file"
description: "Learn how to define properties for your resources in a properties.yml file"
pagination_previous: "reference/define-configs"
---

In dbt, you can use `properties.yml` files to define properties for resources. You can declare properties in `.yml` files, in the same directory as your resources. You can name these files `whatever_you_want.yml` and nest them arbitrarily in sub-folders within each directory.

We highly recommend that you define properties in dedicated paths alongside the resources they're describing.

:::info

#### schema.yml files

Previous versions of the docs referred to these as `schema.yml` files — we've moved away from that terminology since the word `schema` is used to mean other things when talking about databases, and people often thought that you _had_ to name these files `schema.yml`.

Instead, we now refer to these files as `properties.yml` files. (Of course, you're still free to name your files `schema.yml`)
git pull
:::

### Which properties are _not_ also configs?

In dbt, you can define node configs in `properties.yml` files, in addition to `config()` blocks and `dbt_project.yml`. However, some special properties can only be defined in the `.yml` file and you cannot configure them using `config()` blocks or the `dbt_project.yml` file:

Certain properties are special, because:

- They have a unique Jinja rendering context
- They create new project resources
- They don't make sense as hierarchical configuration
- They're older properties that haven't yet been redefined as configs

These properties are:

- [`description`](/reference/resource-properties/description)
- [`tests`](/reference/resource-properties/data-tests)
- [`docs`](/reference/resource-configs/docs)
- [`columns`](/reference/resource-properties/columns)
- [`quote`](/reference/resource-properties/quote)
- [`source` properties](/reference/source-properties) (for example, `loaded_at_field`, `freshness`)
- [`exposure` properties](/reference/exposure-properties) (for example, `type`, `maturity`)
- [`macro` properties](/reference/macro-properties) (for example, `arguments`)

import Example from '/snippets/_configs-properties.md' ;

<Example />
2 changes: 1 addition & 1 deletion website/docs/reference/project-configs/version.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ A version tag in a `.yml` property file provides the control tag, which informs

Starting from version 1.5, dbt will no longer require this configuration in your resource `.yml` files. If you want to know more about why this tag was previously required, you can refer to the [FAQs](#faqs). For users on dbt version 1.4 or lower, this tag is required,

For more on property files, see their general [documentation](/reference/configs-and-properties#where-can-i-define-properties) on the same page.
For more on property files, see their general [documentation](/reference/define-properties) on the same page.

<Tabs
groupId="resource-version-configs"
Expand Down
Loading

0 comments on commit c58eb66

Please sign in to comment.