Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add new bq job timeout and retry config #49

Conversation

hui-zheng
Copy link
Contributor

resolves #45

Description

This PR provides a fine-grained control of the timeout and retry to bq query with four dbt-profile configs

job_creation_timeout_seconds     # specific for initiate BQ job, to control the timeout of step 1, query()
job_execution_timeout_seconds    # specific for awaiting job result, to control the timeout of step 2, result()

job_retry_deadline_seconds       # to control the overal query, retry_deadline of _query_and_results()
job_retries                      # to control the overall query, retries of _query_and_results()

These settings would allow us to control the timeout behaviours of step 1 and step 2 on their own, hence maximizing our chances to mitigate different kinds of intermittent errors.

For example, we could set the configs below to fail faster at the step of BQ job creation, while allowing queries with long-running results.

job_creation_timeout_seconds=30
job_execution_timeout_seconds=1200
job_retry_deadline_seconds=1500
job_retries=3

NOTE:
job_execution_timeout_seconds is the renaming of the previous timeout config.
job_retries is the renaming of the previous retries config.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-bigquery next" section.

@cla-bot
Copy link

cla-bot bot commented Oct 30, 2021

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Hui Zheng.
This is most likely caused by a git client misconfiguration; please make sure to:

  1. check if your git client is configured with an email to sign commits git config --list | grep email
  2. If not, set it up using git config --global user.email [email protected]
  3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

@hui-zheng hui-zheng closed this Oct 30, 2021
@hui-zheng hui-zheng force-pushed the enhancement/bq-job-retry-timeout-configs branch from c363bb1 to bb9caab Compare October 30, 2021 04:51
@cla-bot cla-bot bot added the cla:yes label Oct 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] dbt-Bigquery retry for DML job BQ API errors (503 errors, etc. )
1 participant