Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for "Config" select/exclude to DbtDag & DbtTaskGroup parsers #104

Merged
merged 14 commits into from
Feb 3, 2023

Conversation

chrishronek
Copy link
Contributor

Closes 90

I added the label bug because I found syntax preventing tags from being read out of properties.yml files correctly.

@chrishronek chrishronek added bug Something isn't working enhancement New feature or request labels Jan 27, 2023
@chrishronek chrishronek self-assigned this Jan 27, 2023
@chrishronek chrishronek changed the title Add config materialized to list of configs Add support for "Config" select/exclude to DbtDag & DbtTaskGroup parsers Jan 27, 2023
@chrishronek
Copy link
Contributor Author

This will convert the DbtModelConfig to this:

DbtModel(name='orders', path='dbt/jaffle_shop/models/orders.sql', config=DbtModelConfig(config_selectors={'schema:stg', 'materialized:view', 'tags:tag_4', 'tags:tag_3', 'tags:yet_another_test_tag', 'tags:tag_1'}, upstream_models={'stg_payments', 'stg_orders'}))

...as opposed to this:

DbtModel(name='orders', path='dbt/jaffle_shop/models/orders.sql', config=DbtModelConfig(tags={'tag_4', 'tag_3'}, upstream_models={'stg_payments', 'stg_orders'}))

which will really simplify the process for a customer when it comes to using select/exclude because this is similar to how dbt does it (see here):

    jaffle_shop = DbtTaskGroup(
        dbt_project_name="jaffle_shop",
        conn_id="snowflake_chronek",
        dbt_args={"dbt_executable_path": "/usr/local/airflow/dbt_venv/bin/dbt"},
        test_behavior='after_all',
        select={ #this could also be an exclude parameter with the same value for the opposite effect
          "configs": ['schema:stg']
        },
        dag=dag,
    )

...additionally, it will be easier for our parser to do a single intersection instead of multiple for various selector types. LMK if you have any questions

@chrishronek chrishronek marked this pull request as ready for review February 1, 2023 00:56
@chrishronek chrishronek requested a review from a team as a code owner February 1, 2023 00:56
@chrishronek chrishronek requested a review from jlaneve February 1, 2023 17:51
@chrishronek chrishronek merged commit a4a920a into main Feb 3, 2023
@chrishronek chrishronek deleted the config-selector-90 branch February 3, 2023 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for "Config" select/exclude to DbtDag & DbtTaskGroup parsers
2 participants