Add dbt build operators

[dylanharper-qz]

Description

Add operators for dbt build command.

Related Issue(s)

closes Implement DbtBuild<EXECUTION_MODE>Operator in Astronomer Cosmos

Breaking Change?

None

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	197e947
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/65a805004fe50f00085d2b93

[dylanharper-qz]

This will let us run our third party dbt packages without incurring excess Astro costs (ie, not adding thousands of airflow tasks per day for those packages).

[tatiana]

This looks great, @dylanharper-qz , this is a very useful operator!

[raulgonzalezduran]

@dylanharper-qz @tatiana Is there any documentation or example on how to use this feature? It's not clear to me if it has to be used through RenderConfig.node_converters or there's a simpler way (just in case it's relevant, running in Kubernetes mode). Thanks

[dylanharper-qz]

@raulgonzalezduran this does not include all of the needed configuration details, but this is one way to use this operator:

from airflow.models.dag import DAG
from cosmos import DbtBuildLocalOperator, ProfileConfig
from cosmos.profiles import GoogleCloudServiceAccountDictProfileMapping
from pendulum import datetime

PROFILE_CONFIG = ProfileConfig(
    profile_name="default",
    target_name="prod",
    profile_mapping=GoogleCloudServiceAccountDictProfileMapping(
        conn_id=CONN_ID,
        profile_args={
            "project": PROD_PROJECT,
            "dataset": PROD_DATASET,
        },
    ),
)

with DAG(
    dag_id="dbt_hourly",
    start_date=datetime(2022, 11, 3),
    schedule_interval="@hourly",
) as dag:
    run_operator = DbtBuildLocalOperator(
        profile_config=PROFILE_CONFIG,
        project_dir=DBT_PROJECT_PATH,
        dbt_executable_path=DBT_EXECUTABLE_PATH,
        task_id="build",
        dbt_cmd_flags=["--select", "+tag:hourly"],
        vars=DBT_VARS,
        install_deps=True,
    )

[raulgonzalezduran]

Ok thanks for the fast response, I see in that case we won't render the project though.
In case it's helpful for anyone, I'm in the end passing the RenderConfig.node_converter dict and creating a custom duplication of cosmos.airflow.graph.generate_task_or_group with slight modifications to use DbtBuild operator:

from cosmos.constants import DbtResourceType
from dags.ETL.cosmos_custom_node_converter import generate_task_or_group
..
    run_finance = DbtTaskGroup(
        ..
        render_config=RenderConfig(
            load_method=LoadMode.DBT_MANIFEST,
            dbt_deps=False,
            test_behavior=TestBehavior.NONE,
            node_converters={DbtResourceType.MODEL: generate_task_or_group},
            select=['tag:domain-finance']
        )
    )

Although this is very flexible, if at some point this is worked on, I think it'll be helpful to include a more intuitive way to pass directly a dict {DbtResourceType, Operator}, but thanks for including DBT build!