[Bug] Projects with restrict-access: true cannot leverage generic tests on private models

[nicholasyager]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Suppose you have a dbt project with restrict-access: true, preventing other projects from importing this project and leveraging protected models. Now suppose you have a model with a group defined and access set to private, and a column with a generic test -- let's say a not_null test for the sake of example. With this combination of settings in place, compilation will raise a DbtReferenceError on the test since the test itself is not part of the group.

__models.yml

models:
  - name: accounts
    description: >
      All accounts with whom we have done business. This is a very sensitive asset.
    access: private
    group: sales

    columns:
      - name: name
        description: Name of the account.
        tests:
          - not_null
          - unique

dbt_project.yml

...
restrict-access: true
...

Output

(venv-1.7) > $ dbt build
00:22:16  Running with dbt=1.7.14
00:22:17  Registered adapter: duckdb=1.7.5
00:22:17  Encountered an error:
Parsing Error
  Node test.revenue.not_null_accounts_name.a22fd7c4da attempted to reference node model.revenue.accounts, which is not allowed because the referenced node is private to the 'sales' group.

The specific defect can be resolved in manifest.py in the Manifest.is_invalid_private_ref method. This method should be updated to valuate to false then the calling node's resource_type is NodeType.Test.

Expected Behavior

I would expect a dbt test Node to not be beholden to private ref checks when derived from a generic test.

Steps To Reproduce

See description

Relevant log output

(venv-1.7) > $ dbt build
00:22:16  Running with dbt=1.7.14
00:22:17  Registered adapter: duckdb=1.7.5
00:22:17  Encountered an error:
Parsing Error
  Node test.revenue.not_null_accounts_name.a22fd7c4da attempted to reference node model.revenue.accounts, which is not allowed because the referenced node is private to the 'sales' group.

Environment

- OS: macos 14.4.1
- Python: 3.12.3
- dbt: 1.7.14 and 1.8.0

Which database adapter are you using with dbt?

snowflake, other (mention it in "Additional Context")

Additional Context

Specific cause of the defect is here:

dbt-core/core/dbt/contracts/graph/manifest.py

Line 1434 in 8fe7d65

is_private_ref = (

This issue is related to #8248

[jtcohen6]

Thanks for opening @nicholasyager!

I agree that this is a bug. But I think it's slightly different from #8248, because generic tests can belong to groups, and dbt automatically sets their group attribute to match the model they're defined on.

The current intended behavior is:

1. If a model is installed via a "package" dependency, and restrict-access is False for that package, then it should be treated exactly the same as resources within this project. (This maintains historical behavior for packages.)
2. If a model is installed via a "package" dependency, and restrict-access is True, then the difference in namespace (package_name) should restrict private and protected references.
3. If a model is installed as a "project" dependency, then only reference to public models is allowed. To that end, the access level and namespace (package_name) are sufficient for resolving that reference, as groups don't currently exist across projects.

The current issue is that dbt's application of (2) is too broad, applying restrict-access even to resources within the same package/group. Rather than changing this logic to skip private reference checks for tests, I think we need to change this logic to be something like the following:

    def is_invalid_private_ref(
        ...
        return is_private_ref and (
            # Invalid reference because the group does not match
            (hasattr(node, "group") and node.group and node.group != target_model.group)
            # Or, invalid because these are different namespaces (project/package) and restrict-access is enforced
            or (node.package_name != target_model.package_name and restrict_package_access)
        )

When I try that change locally, and your reproduction case, dbt passes (and does not raise a DbtReferenceError).

---

Speaking of groups across projects...

Feature: Support cross-project Group evaluation

It sounds like you might be thinking in a direction similar to this proposal:

• [CT-3528] [Feature] Define a "yeslist" of downstream projects that can reference specific protected model(s) #9340 (comment)

The idea there is, a model should be ref'able by specific other projects, based on some opt-in criteria shared by both. If we pursue that idea, it would make sense to add group (and whatever other necessary properties) to ModelNodeArgs (which I see came up in linked conversation).

Open to hearing your thoughts on #9340!

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.