Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] include test to br_me_cnpj #566

Merged
merged 17 commits into from
Nov 23, 2023
Merged

[feat] include test to br_me_cnpj #566

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
c256654
include test
laura-l-amaral Nov 17, 2023
0b48e1c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 17, 2023
e98e738
fix flow totest
laura-l-amaral Nov 17, 2023
5946e0e
Merge branch 'staging/br_me_cnpj' of https://github.com/basedosdados/…
laura-l-amaral Nov 17, 2023
6fee5ba
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 17, 2023
e8ead1f
Merge branch 'main' into staging/br_me_cnpj
laura-l-amaral Nov 17, 2023
04db860
Merge branch 'staging/br_me_cnpj' of https://github.com/basedosdados/…
laura-l-amaral Nov 17, 2023
36495a7
remove run_dbt retries
laura-l-amaral Nov 23, 2023
8032155
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2023
ace9d19
fix dbt_alias
laura-l-amaral Nov 23, 2023
96bbfdc
Merge branch 'staging/br_me_cnpj' of https://github.com/basedosdados/…
laura-l-amaral Nov 23, 2023
1361507
Merge branch 'main' into staging/br_me_cnpj
mergify[bot] Nov 23, 2023
d1da779
fix retries parameter
laura-l-amaral Nov 23, 2023
dd92fd7
Merge branch 'staging/br_me_cnpj' of https://github.com/basedosdados/…
laura-l-amaral Nov 23, 2023
850bc38
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2023
99a4a0e
small fix
laura-l-amaral Nov 23, 2023
f1ddce9
Merge branch 'staging/br_me_cnpj' of https://github.com/basedosdados/…
laura-l-amaral Nov 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion pipelines/datasets/br_me_cnpj/flows.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@
"table_id": table_id,
"mode": materialization_mode,
"dbt_alias": dbt_alias,
"dbt_command": "run and test",
},
labels=current_flow_labels,
run_name=f"Materialize {dataset_id}.{table_id}",
Expand Down Expand Up @@ -265,6 +266,7 @@
"table_id": table_id,
"mode": materialization_mode,
"dbt_alias": dbt_alias,
"dbt_command": "run and test",
},
labels=current_flow_labels,
run_name=f"Materialize {dataset_id}.{table_id}",
Expand Down Expand Up @@ -294,7 +296,7 @@
bq_project="basedosdados",
upstream_tasks=[wait_for_materialization],
)

## atualiza o diretório de empresas
with case(materialize_after_dump, True):
# Trigger DBT flow run
current_flow_labels = get_current_flow_labels()
Expand Down
8 changes: 4 additions & 4 deletions pipelines/datasets/br_me_cnpj/schedules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"table_id": "empresas",
"materialization_mode": "prod",
"materialize_after_dump": True,
"dbt_alias": False,
"dbt_alias": True,
},
),
]
Expand All @@ -43,7 +43,7 @@
"table_id": "socios",
"materialization_mode": "prod",
"materialize_after_dump": True,
"dbt_alias": False,
"dbt_alias": True,
},
),
]
Expand All @@ -61,7 +61,7 @@
"table_id": "simples",
"materialization_mode": "prod",
"materialize_after_dump": True,
"dbt_alias": False,
"dbt_alias": True,
},
),
]
Expand All @@ -79,7 +79,7 @@
"table_id": "estabelecimentos",
"materialization_mode": "prod",
"materialize_after_dump": True,
"dbt_alias": False,
"dbt_alias": True,
},
),
]
Expand Down
3 changes: 1 addition & 2 deletions pipelines/utils/execute_dbt_model/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,7 @@ def get_k8s_dbt_client(

@task(
checkpoint=False,
max_retries=constants.TASK_MAX_RETRIES.value,
retry_delay=timedelta(seconds=constants.TASK_RETRY_DELAY.value),
max_retries=0,
)
def run_dbt_model(
dbt_client: DbtClient,
Expand Down