roachprod/failure-injection: add initial framework for failure injection library #140548
+701
−50
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This helper will be used in the new failure injection library.
|
This change is |
DarrylWong
force-pushed
the
fi-lib
branch
2 times, most recently
from
8c62328 to
0029b12
Compare
This commit adds the framework for the failure injection library, as well as the first supported failure: iptables network partitions. This failure can be used on roachprod clusters to create bidirectional and asymmetric network partitions between node(s).
This registry will allow for future usage of the failure injection library through the CLI and the failure injection planner/controller.
This adds an integration test for the failure injection library. The test spins up a cluster and randomly selects a failure to inject. It then validates that the failure was correctly injected. Afterwards, it reverts the failure and validates that the failure was correctly cleaned up.
DarrylWong
requested review from
herkolategan and
srosenberg
and removed request for
a team
srosenberg
requested review from
golgeek,
shailendra-patel and
nameisbhaskar
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the initial framework for the failure injection library within roachprod. The
failurespackage can now be used which adds theFailureModeinterface. AFailureModedescribes a failure that can be injected into a roachprod cluster along with how to revert the failure. Additionally, it also adds the first supported failure:iptablesnetwork partitions.See individual commits for details.
Release note: none
Epic: https://cockroachlabs.atlassian.net/browse/CRDB-46439
Informs: #138970
I have a WIP branch here with rough implementations of the CLI, roachtest refactoring, and disk stall failures if you're curious how that works. I wanted to keep this PR small though to keep things reviewable + get feedback before I get too deep.