diff --git a/website/docs/docs/build/custom-aliases.md b/website/docs/docs/build/custom-aliases.md index 9876f534f8f..589d64f8510 100644 --- a/website/docs/docs/build/custom-aliases.md +++ b/website/docs/docs/build/custom-aliases.md @@ -114,6 +114,14 @@ The default implementation of `generate_alias_name` simply uses the supplied `al + + +### Managing different behaviors across packages + +See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) + + + ### Caveats #### Ambiguous database identifiers diff --git a/website/docs/docs/build/custom-databases.md b/website/docs/docs/build/custom-databases.md index 3df3d705837..300fd3147f1 100644 --- a/website/docs/docs/build/custom-databases.md +++ b/website/docs/docs/build/custom-databases.md @@ -87,6 +87,14 @@ The default implementation of `generate_database_name` simply uses the supplied + + +### Managing different behaviors across packages + +See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) + + + ## Considerations ### BigQuery diff --git a/website/docs/docs/build/custom-schemas.md b/website/docs/docs/build/custom-schemas.md index 156d2f50368..2c72bec12c6 100644 --- a/website/docs/docs/build/custom-schemas.md +++ b/website/docs/docs/build/custom-schemas.md @@ -178,7 +178,7 @@ The following context methods _are_ available in the `generate_schema_name` macr | Other macros in your project | Macro | ✅ | | Other macros in your packages | Macro | ✅ | -#### Which vars are available in generate_schema_name? +### Which vars are available in generate_schema_name? @@ -190,6 +190,14 @@ for more information on these changes. Globally-scoped variables and variables defined on the command line with [--vars](/docs/build/project-variables) are accessible in the `generate_schema_name` context. + + +### Managing different behaviors across packages + +See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) + + + ## Managing environments In the `generate_schema_name` macro examples shown above, the `target.name` context variable is used to change the schema name that dbt generates for models. If the `generate_schema_name` macro in your project uses the `target.name` context variable, you must additionally ensure that your different dbt environments are configured appropriately. While you can use any naming scheme you'd like, we typically recommend: diff --git a/website/docs/docs/build/project-variables.md b/website/docs/docs/build/project-variables.md index a84bbcb36a9..03d1cca43b2 100644 --- a/website/docs/docs/build/project-variables.md +++ b/website/docs/docs/build/project-variables.md @@ -86,18 +86,32 @@ You can find more information on defining dictionaries with YAML [here](https:// ### Variable precedence -Variables defined with the `--vars` command line argument override variables -defined in the `dbt_project.yml` file. They are globally scoped and will be -accessible to all packages included in the project. +Variables defined with the `--vars` command line argument override variables defined in the `dbt_project.yml` file. They are globally scoped and accessible to the root project and all installed packages. The order of precedence for variable declaration is as follows (highest priority first): + + 1. The variables defined on the command line with `--vars`. -3. The package-scoped variable declaration in the `dbt_project.yml` file -2. The global variable declaration in the `dbt_project.yml` file. +2. The package-scoped variable declaration in the root `dbt_project.yml` file +3. The global variable declaration in the root `dbt_project.yml` file. +4. If this node is defined in a package: variable declarations in that package's `dbt_project.yml` file +5. The variable's default argument (if one is provided). + + + + + +1. The variables defined on the command line with `--vars`. +2. The package-scoped variable declaration in the root `dbt_project.yml` file +3. The global variable declaration in the root `dbt_project.yml` file. 4. The variable's default argument (if one is provided). + + If dbt is unable to find a definition for a variable after checking these four places, then a compilation error will be raised. +**Note:** Variable scope is based on the node ultimately using that variable. Imagine the case where a model defined in the root project is calling a macro defined in an installed package. That macro, in turn, uses the value of a variable. The variable will be resolved based on the _root project's_ scope, rather than the package's scope. + diff --git a/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md b/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md index 4f7ddc2fd8a..865c6d3ba49 100644 --- a/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md +++ b/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md @@ -31,4 +31,6 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x, ### Quick hits -**Coming Soon** \ No newline at end of file +More consistency and flexibility around packages! Resources defined in a package will respect variable and global macro definitions within the scope of that package. +- `vars` defined in a package's `dbt_project.yml` are now available in the resolution order when compiling nodes in that package, though CLI `--vars` and the root project's `vars` will still take precedence. See ["Variable Precedence"](/docs/build/project-variables#variable-precedence) for details. +- `generate_x_name` macros (defining custom rules for database, schema, alias naming) follow the same pattern as other "global" macros for package-scoped overrides. See [macro dispatch](/reference/dbt-jinja-functions/dispatch) for an overview of the patterns that are possible. diff --git a/website/docs/reference/dbt-jinja-functions/dispatch.md b/website/docs/reference/dbt-jinja-functions/dispatch.md index 60938361960..169971e782a 100644 --- a/website/docs/reference/dbt-jinja-functions/dispatch.md +++ b/website/docs/reference/dbt-jinja-functions/dispatch.md @@ -167,6 +167,24 @@ dispatch: +### Managing different global overrides across packages + +You can override global behaviors in different ways for each project that is installed as a package. This holds true for all global macros: `generate_schema_name`, `create_table_as`, etc. When parsing or running a resource defined in a package, the definition of the global macro within that package takes precedence over the definition in the root project, because it's more specific to those resources. + +By combining package-level overrides and `dispatch`, it is possible to achieve three different patterns: + +1. **Package always wins.** As the developer of dbt models in a project that will be deployed elsewhere as a package, I want full control over the macros used to define & materialize my models. My macros should always take precedence for my models, and there should not be any way to override them. + +_Mechanism:_ Each project/package fully overrides the macro by its name, e.g. `generate_schema_name` or `create_table_as`. Do not use dispatch. + +2. **Conditional application (root project wins).** As the maintainer of one dbt project in a mesh of multiple, my team wants conditional application of these rules. When running my project standalone (in development), I want my to apply my custom behavior; but when installed as a package and deployed alongside several other projects (in production), I want the root-level project's rules to apply. + +_Mechanism:_ Each package implements its "local" override by registering a candidate for dispatch with an adapter prefix, e.g. `default__generate_schema_name` or `default__create_table_as`. Then, the root-level project can register its own candidate for dispatch (`default__generate_schema_name`), winning the default search order, or by explicitly overriding the macro by name (`generate_schema_name`). + +3. **Same rules everywhere all the time.** As a member of the data platform team responsible for consistency across teams at my organization, I want to create a "macro package" that every team can install & use. + +_Mechanism:_ Create a standalone package of candidate macros only, e.g. `default__generate_schema_name` or `default__create_table_as`. Add a [project-level `dispatch` configuration](/reference/project-configs/dispatch-config) in every project's `dbt_project.yml`. + ## For adapter plugin maintainers Most packages were initially designed to work on the four original dbt adapters. By using the `dispatch` macro and project config, it is possible to "shim" existing packages to work on other adapters, by way of third-party compatibility packages.