[rhythm] Add rollout-operator and allow simultaneous update of block-builder pods #4660
+365
−69
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mapno
requested review from
joe-elliott,
mdisibio,
yvrhdn,
zalegrala,
electron0zero,
ie-pham,
stoewer and
javiermolinar
as code owners
| 'kubernetes.namespace': $._config.namespace, | ||
| }, | ||
|
|
||
| rollout_operator_container:: |
There was a problem hiding this comment.
There is already a rollout-operator deployment in
Should we merge both?
There was a problem hiding this comment.
Good catch. I think we have to, to at least avoid conflicting deployments. But since the other rollout operator looks last updated more than 2y ago, and we don't use it ourselves, I would defer to this new one where any differences are found.
There was a problem hiding this comment.
Yes! I had forgotten, thanks!
| }, | ||
|
|
||
| rollout_operator_container:: | ||
| container.new('rollout-operator', $._images.rollout_operator) + |
There was a problem hiding this comment.
We can bump the existing version wich is outdated:
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Introduced the rollout-operator so the pods in the block-builder's statefulset can simultaneously update. Otherwise there is the risk of ownership conflicts in partitions during rollouts.
It solves the following scenario:
There are 6 partitions and 3 block-builders.
Partition assignment is as follows:
block-builder-0has assigned partitions0and3.block-builder-1has assigned partitions1and4.block-builder-2has assigned partitions2and5.If a new block-builder is deployed, partition assignment will change to
block-builder-0has assigned partitions0and4.block-builder-1has assigned partitions1and5.block-builder-2has assigned partitions2.block-builder-3has assigned partitions3.As soon as block-builders are rolled out, they'll have the new config, while older replicas will have the old config—ie. at one point,
block-builder-3andblock-builder-0both will own partition3.Alternatives
A previous attempt was made in #4642, but in the end discarded. It required Kubernetes >=v1.24 and
MaxUnavailableStatefulSetfeature gate enabled, which is in alpha and not supported in many environments.Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]