[CT-965] [Bug] Variable scoping ignored for macros

[NiallRees]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I'm developing the dbt_artifacts package, and have an integration test project which installs the dbt_artifacts package using a local: file system reference:

packages:
  - local: ../

Within the integration test project's dbt_project.yml, I've defined these variables, the idea being that anything under the dbt_artifacts key is scoped only to the dbt_artifacts package.

vars:
  test_dbt_vars_1: dbt_vars_1
  test_dbt_vars_2: dbt_vars_2
  test_dbt_vars_3: dbt_vars_3
  dbt_artifacts:
    env_vars: [
      'TEST_ENV_VAR_1'
    ]
    dbt_vars: [
      'test_dbt_vars_1',
      'test_dbt_vars_2',
      'test_dbt_vars_3'
    ]

If I use {{ var('env_vars') }} in one of dbt_artifacts' models, the model renders that value correctly as ['TEST_ENV_VAR_1'].

However, if I add {{ var('env_vars') }} to a macro (upload_invocations) in the dbt_artifacts package which is called from the on-run-end hook of the integration test project:

on-run-end:
  - "{{ dbt_artifacts.upload_invocations() }}"

I get an error:

Runtime Error
  Compilation Error in operation artifacts_integration_tests-on-run-end-0 (./dbt_project.yml)
    Required var 'env_vars' not found in config:
    Vars supplied to artifacts_integration_tests-on-run-end-0 = {
        "dbt_artifacts": {
            "dbt_vars": [
                "test_dbt_vars_1",
                "test_dbt_vars_2",
                "test_dbt_vars_3"
            ],
            "env_vars": [
                "TEST_ENV_VAR_1"
            ]
        },
        "test_dbt_vars_1": "dbt_vars_1",
        "test_dbt_vars_2": "dbt_vars_2",
        "test_dbt_vars_3": "dbt_vars_3"
    }
    
    > in macro upload_invocations (macros/upload_invocations.sql)
    > called by operation artifacts_integration_tests-on-run-end-0 (./dbt_project.yml)
    > called by operation artifacts_integration_tests-on-run-end-0 (./dbt_project.yml)

Expected Behavior

From reading the docs here, I would expect the variable used in the package's macro to render identically to how it does in the package's model. Instead, it seems that I cannot scope a variable to the package if I want to reference it within a macro.

Steps To Reproduce

See above

Relevant log output

➜  dbt_artifacts_internal/integration_test_project ✗ dbt compile --target snowflake
dbtenv info:  Using dbt-snowflake==1.1.0 (max installed stable version for current adapter).
09:18:33  Running with dbt=1.1.1
09:18:33  Unable to do partial parsing because a project config has changed
09:18:36  Found 24 models, 3 tests, 1 snapshot, 0 analyses, 482 macros, 1 operation, 1 seed file, 12 sources, 1 exposure, 0 metrics
09:18:36  
09:18:39  Concurrency: 8 threads (target='snowflake')
09:18:39  
09:18:39  Encountered an error:
Runtime Error
  Compilation Error in operation artifacts_integration_tests-on-run-end-0 (./dbt_project.yml)
    Required var 'env_vars' not found in config:
    Vars supplied to artifacts_integration_tests-on-run-end-0 = {
        "dbt_artifacts": {
            "dbt_vars": [
                "test_dbt_vars_1",
                "test_dbt_vars_2",
                "test_dbt_vars_3"
            ],
            "env_vars": [
                "TEST_ENV_VAR_1"
            ]
        },
        "test_dbt_vars_1": "dbt_vars_1",
        "test_dbt_vars_2": "dbt_vars_2",
        "test_dbt_vars_3": "dbt_vars_3"
    }
    
    > in macro upload_invocations (macros/upload_invocations.sql)
    > called by operation artifacts_integration_tests-on-run-end-0 (./dbt_project.yml)
    > called by operation artifacts_integration_tests-on-run-end-0 (./dbt_project.yml)

Environment

- OS: Max OS
- Python: 3.8.9
- dbt: dbt-snowflake 1.1.0, dbt 1.1.1

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[lostmygithubaccount]

hello @NiallRees, thanks for opening! apologies for the unusually long triage time on this one. what you're noticing is expected behavior that we don't intend on changing, so I'm going to close this as won't fix. hopefully the explanation below is helpful, and I have logged an issue to improve the docs here

From reading the docs here, I would expect the variable used in the package's macro to render identically to how it does in the package's model. Instead, it seems that I cannot scope a variable to the package if I want to reference it within a macro.

macros are resolved within the context of the model calling them (and its project), not the macro's project. you should be able to fix what you're experience by defining the environment variable in the model's project. can you try that out and let us know if you're still experiencing issues?

[lostmygithubaccount]

docs issue: dbt-labs/docs.getdbt.com#1928