Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bulk: add exponential backoff to addSSTable #120132

Merged
merged 1 commit into from
Mar 8, 2024

Conversation

ajstorm
Copy link
Collaborator

@ajstorm ajstorm commented Mar 8, 2024

Previously on retryable errors in addSSTable we'd retry immediately. In cases where we're failing because a remote node is down, we're more likely to be successful if we retry only after some backoff. This commit adds an exponential backoff which retries 10 times, with max duration ~30 seconds. This should at least allow for the toleration of lease transfers with downed nodes.

Release note: None

Fixes: #120050

@ajstorm ajstorm requested a review from dt March 8, 2024 14:47
@ajstorm ajstorm requested a review from a team as a code owner March 8, 2024 14:47
Copy link

blathers-crl bot commented Mar 8, 2024

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Member

@dt dt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

Previously on retryable errors in addSSTable we'd retry immediately.  In cases
where we're failing because a remote node is down, we're more likely to be
successful if we retry only after some backoff. This commit adds an exponential
backoff which retries 10 times, with max duration ~30 seconds. This should at
least allow for the toleration of lease transfers with downed nodes.

Release note: None

Fixes: cockroachdb#120050
@ajstorm ajstorm force-pushed the ajstorm-exp-backoff branch from 57e7586 to 16acddd Compare March 8, 2024 16:10
@ajstorm
Copy link
Collaborator Author

ajstorm commented Mar 8, 2024

TFTR

bors r=dt

@craig
Copy link
Contributor

craig bot commented Mar 8, 2024

Build succeeded:

@craig craig bot merged commit c14c165 into cockroachdb:master Mar 8, 2024
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bulk: retry with exponential backoff when adding SSTables
3 participants