-
Notifications
You must be signed in to change notification settings - Fork 103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(athena-adapter): support custom retry configurations for boto3 calls #494
feat(athena-adapter): support custom retry configurations for boto3 calls #494
Conversation
Overall looks good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lukealexmiller everything looks nice to me, just let's fix minor comments about naming and docs
@lukealexmiller do you think that you can make the changes that @svdimchenko requested within the day? then we include this in the next release. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution
Except @svdimchenko comments LGTM ✅
…get_boto3_config. add unit test to cover implementation
80c2467
to
ac351a5
Compare
Thanks for the feedback all, hopefully this is now ready in time for the next release! btw, most of the doc changes are due to |
Description
This PR introduces a new functionality to the Athena adapter allowing users to specify a custom number of retry attempts for boto3 calls within the dbt-athena-community plugin. The modification adds a parameter to get_boto3_config function, enabling it to accept the number of retries as an argument. This change enhances resilience and configurability by letting users define their retry strategies, accommodating for network inconsistencies or AWS rate limits.
Resolves #480.
Changes:
num_retries
parameter toget_boto3_config
function to accept custom retry attempts.num_boto3_retries
as an optional configuration inAthenaCredentials
to store the number of boto3 retries.get_effective_num_retries
method inAthenaCredentials
to determine the effective number of retries considering the optionalnum_boto3_retries
.get_boto3_config
withinAthenaConnectionManager
andAthenaAdapter
to pass the effective number of retries.test_get_boto3_config
to validate the retry configuration within the boto3 config.The default behavior remains unchanged if the
num_boto3_retries
is not set, with the system defaulting to the existingnum_retries
value.Testing:
The updated test ensures that the
user_agent_extra
remains correctly set and validates the new retry configuration is applied whennum_boto3_retries
is specified.Note:
Ensure your environment configuration aligns with the new changes by providing the
num_boto3_retries
value if custom retry behavior is desired.Models used to test - Optional
Checklist