diff --git a/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.5.md b/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.5.md index 11c78bd4bfa..f69a46bb571 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.5.md +++ b/website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.5.md @@ -153,3 +153,4 @@ Run `dbt --help` to see new & improved help documentation :) - [Events and logging](/reference/events-logging): Added `node_relation` (`database`, `schema`, `identifier`) to the `node_info` dictionary, available on node-specific events - Support setting `--project-dir` via environment variable: [`DBT_PROJECT_DIR`](/reference/dbt_project.yml) - More granular configurations for logging (to set [log format](/reference/global-configs/logs#log-formatting), [log levels](/reference/global-configs/logs#log-level), and [colorization](/reference/global-configs/logs#color)) and [cache population](/reference/global-configs/cache#cache-population) +- [dbt overwrites the `manifest.json` file](/reference/node-selection/state-comparison-caveats#overwrites-the-manifestjson) during parsing, which means when you reference `--state` from the `target/ directory`, you may encounter a warning indicating that the saved manifest wasn't found. diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index c7b5b8d810e..01f0af3c1f7 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -254,6 +254,18 @@ There are two additional `state` selectors that complement `state:new` and `stat These selectors can help you shorten run times by excluding unchanged nodes. Currently, no subselectors are available at this time, but that might change as use cases evolve. +#### Overwrites the `manifest.json` + +import Overwritesthemanifest from '/snippets/_overwrites-the-manifest.md'; + + + +#### Recommendation + +import Recommendationoverwritesthemanifest from '/snippets/_recommendation-overwriting-manifest.md'; + + + ### tag The `tag:` method is used to select models that match a specified [tag](/reference/resource-configs/tags). diff --git a/website/docs/reference/node-selection/state-comparison-caveats.md b/website/docs/reference/node-selection/state-comparison-caveats.md index bf79634e178..4202fa6aeae 100644 --- a/website/docs/reference/node-selection/state-comparison-caveats.md +++ b/website/docs/reference/node-selection/state-comparison-caveats.md @@ -45,6 +45,17 @@ If you're a frequent user of `relationships` tests or data tests, or frequently dbt run -s "state:modified" dbt test -s "state:modified" --exclude "test_name:relationships" ``` +### Overwrites the `manifest.json` + +import Overwritesthemanifest from '/snippets/_overwrites-the-manifest.md'; + + + +#### Recommendation + +import Recommendationoverwritesthemanifest from '/snippets/_recommendation-overwriting-manifest.md'; + + ### False positives diff --git a/website/package-lock.json b/website/package-lock.json index 50c3a8de8ea..3fa85f32338 100644 --- a/website/package-lock.json +++ b/website/package-lock.json @@ -5,6 +5,7 @@ "requires": true, "packages": { "": { + "name": "website", "version": "0.0.0", "dependencies": { "@docusaurus/core": "3.7.0", diff --git a/website/snippets/_overwrites-the-manifest.md b/website/snippets/_overwrites-the-manifest.md new file mode 100644 index 00000000000..2a69423bfd2 --- /dev/null +++ b/website/snippets/_overwrites-the-manifest.md @@ -0,0 +1,7 @@ +dbt overwrites the `manifest.json` file during parsing, which means when you reference `--state` from the `target/ directory`, you may encounter a warning indicating that the saved manifest wasn't found. + + + +During the next job run, dbt follows a sequence of steps that lead to the issue. First, it overwrites `target/manifest.json` before it can be used for change detection. Then, when dbt tries to read `target/manifest.json` again to detect changes, it finds none because the previous state has already been overwritten/erased. + +Avoid setting `--state` and `--target-path` to the same path with state-dependent features like `--defer` and `state:modified` as it can lead to non-idempotent behavior and won't work as expected. \ No newline at end of file diff --git a/website/snippets/_recommendation-overwriting-manifest.md b/website/snippets/_recommendation-overwriting-manifest.md new file mode 100644 index 00000000000..119e0f8e81b --- /dev/null +++ b/website/snippets/_recommendation-overwriting-manifest.md @@ -0,0 +1,7 @@ +To prevent the `manifest.json` from being overwritten before dbt reads it for change detection, update your workflow using one of these methods: + +- Move the `manifest.json` to a dedicated folder (for example `state/`) after dbt generates it in the `target/ folder`. This makes sure dbt references the correct saved state instead of comparing the current state with the just-overwritten version. It also avoids issues caused by setting `--state` and `--target-path` to the same location, which can lead to non-idempotent behavior. + + +- Write the manifest to a different `--target-path` in the build stage (where dbt would generate the `target/manifest.json`) or before it gets overwritten during job execution to avoid issues with change detection. This allows dbt to detect changes instead of comparing the current state with the just-overwritten version. +- Pass the `--no-write-json` flag: `dbt --no-write-json ls --select state:modified --state target`: during the reproduction stage. \ No newline at end of file diff --git a/website/static/img/docs/reference/saved-manifest-not-found.png b/website/static/img/docs/reference/saved-manifest-not-found.png new file mode 100644 index 00000000000..50fc302f852 Binary files /dev/null and b/website/static/img/docs/reference/saved-manifest-not-found.png differ