Unable to connect to Databricks SQL Warehouse from within Dev Container

[pgoslatara]

Expected behavior

I can preview data using the dbt Power User extension.

Actual behavior

I always get a 403 access denied error message when I try to preview data.

Steps To Reproduce

I authenticate to Databricks via:

databricks auth login --host https://....azuredatabricks.net

I then run source .env where my .env file contains:

export DBT_ACCESS_TOKEN=$(databricks auth token 'https://....azuredatabricks.net/' -o json | jq -r '.access_token')

My profiles.yml file is:

---
my_dbt:
    target: dev
    outputs:
        dev:
            type: databricks
            catalog: transformation_preprod
            host: ....azuredatabricks.net
            http_path: /sql/1.0/warehouses/...
            schema: "{{ env_var('DBT_SCHEMA', 'must be set') }}"
            threads: 16
            token: "{{ env_var('DBT_ACCESS_TOKEN') }}"

From the command line I can successfully run dbt debug and dbt build. Using dbt Power User I can compile models successfully.

When I try to preview the data or run a model using dbt Power User I get 403 errors.

Log output/Screenshots

Clicking the "Run dbt Model" button:

10:15:34  Running with dbt=1.9.1
/usr/local/lib/python3.11/site-packages/pydantic/_internal/_config.py:345: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
  warnings.warn(message, UserWarning)
10:15:37  Registered adapter: databricks=1.9.1
10:15:38  Unable to do partial parsing because config vars, config profile, or config target have changed
10:15:38  Unable to do partial parsing because profile has changed
10:15:42  Found 18 models, 18 data tests, 18 sources, 1013 macros
10:15:42  
10:15:42  Concurrency: 16 threads (target='dev')
10:15:42  
10:15:42  Databricks adapter: Connection(session-id=Unknown) - Exception while trying to create connection: Error during request to server. Received 403 - FORBIDDEN. Confirm your authentication credentials.
Error properties: attempt=1/30, bounded-retry-delay=None, elapsed-seconds=0.1499040126800537/900.0, error-message=, http-code=None, method=OpenSession, no-retry-reason=non-retryable error, original-exception=Received 403 - FORBIDDEN. Confirm your authentication credentials., query-id=None, session-id=None
10:15:43  
10:15:43  Finished running  in 0 hours 0 minutes and 0.19 seconds (0.19s).
10:15:43  Encountered an error:
Database Error
  Database Error
    Error during request to server. Received 403 - FORBIDDEN. Confirm your authentication credentials.

Clicking the "Execute dbt SQL" button:

Database Error
Database Error Error during request to server. Received 403 - FORBIDDEN. Confirm your authentication credentials.

View Detailed Error 🚨
{
  "code": -1,
  "message": "Database Error\n  Database Error\n    Error during request to server. Received 403 - FORBIDDEN. Confirm your authentication credentials.",
  "data": "\"Error: Database Error\\n  Database Error\\n    Error during request to server. Received 403 - FORBIDDEN. Confirm your authentication credentials.\\n\\tat exports.QueryExecution.queryResult (/root/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.52.1/dist/extension.js:39270:126)\\n\\tat async QueryResultPanel_1.executeQuery (/root/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.52.1/dist/extension.js:48136:90)\""
}

Operating System

Debian

dbt version

Core: - installed: 1.9.1 - latest: 1.9.2 - Update available! Your version of dbt-core is out of date! You can find instructions for upgrading here: https://docs.getdbt.com/docs/installation Plugins: - spark: 1.9.0 - Up to date! - databricks: 1.9.1 - Update available! At least one plugin is out of date with dbt-core. You can find instructions for upgrading here: https://docs.getdbt.com/docs/installation

dbt Adapter

dbt-databricks

dbt Power User version

v0.52.1

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[mdesmet]

Whatever you do on the command line has no influence on the extension. We pick up the authentication from the dbt profile, nothing more is done.

[pgoslatara]

@mdesmet Understood. Any idea why the extensions can't connect to the Datacricks cluster? The only config performed via the CLI is to set the env var that contains a token.