diff --git a/keps/prod-readiness/sig-apps/3541.yaml b/keps/prod-readiness/sig-apps/3541.yaml new file mode 100644 index 00000000000..49ccf553b25 --- /dev/null +++ b/keps/prod-readiness/sig-apps/3541.yaml @@ -0,0 +1,3 @@ +kep-number: 3541 +alpha: + approver: "@wojtek-t" diff --git a/keps/sig-apps/3541-add-enforcedRollingUpdate-to-statefulset-update-strategies/README.md b/keps/sig-apps/3541-add-enforcedRollingUpdate-to-statefulset-update-strategies/README.md new file mode 100644 index 00000000000..b489ba4498b --- /dev/null +++ b/keps/sig-apps/3541-add-enforcedRollingUpdate-to-statefulset-update-strategies/README.md @@ -0,0 +1,923 @@ + +# KEP-3541: Add enforcedRollingUpdate to StatefulSet update strategies + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [GA](#ga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Upgrade](#upgrade) + - [Downgrade](#downgrade) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [x] (R) Design details are appropriately documented +- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [x] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [x] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [x] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +Currently we support two rolling update strategies in statefulSet, one is `OnDelete`, another one is `RollingUpdate`, +which is the default one. And as designed, when we got a broken state with a statefulSet applying the default `RollingUpdate` +strategy, we have to repair this manually. + +Like we applied a broken statefulSet, and the corresponding pods will stuck in pending with failures. then we apply with +a fixed yaml file, the stuck pods will not turn to health for security design, see [Forced rollback](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#forced-rollback). + +But this is heavily complained by the community, see [issue-67250](https://github.com/kubernetes/kubernetes/issues/67250), +[issue-60164](https://github.com/kubernetes/kubernetes/issues/60164), [issue-109597](https://github.com/kubernetes/kubernetes/issues/109597) and so on, +we should provide a escape patch for end-users. + +## Motivation + + + +As described above, pods in a broken state won't turn to health even we apply with a fixed statefulSet yaml file, instead, +statefulSet controller will wait until we delete the pod manually. I think this was designed because of some security reasons for +statefulSet generally is used to manage stateful workloads, delete them directly may lead to some unexpected results like +data loss, this is intolerable. + +But not always that case, for example we may deploy a statefulset just for the uniqueness of Pods, automatically delete them will not +cause any serious problems, so we should treat the rolling-update strategies case by case, and better to hand the choices over to end-users. + +From the solutions of community users, we have two choices to avoid this, one is set the `podManagementPolicy` to `Parallel`, see [issue](https://github.com/jenkinsci/helm-charts/issues/688), +but we will update the pods in parallel. Another one is use a controller and find the broken pods to delete, see [PR](https://github.com/argoproj-labs/argocd-operator/pull/604). + +Instead of getting around this problem, we should provide a new rolling-update strategy named like `enforcedRollingUpdate` to update the pods anyway, +but taking security into consideration, when updating the latest ordinal pod failed, we'll wait until the pod turn to healthy. + +### Goals + + + +Provide a new rolling-update strategy `enforcedRollingUpdate` to statefulSet. + +### Non-Goals + + + +Change to current implementation to forced updating. + +## Proposal + + + +### User Stories (Optional) + + + +#### Story 1 + +We're running a CI/CD system and the end-to-end automation is essential, but with statefulset, +only the happy path is satisfied with the system. Or we have to implement the "GC" logic in the +controller. But actually it should be the statefulSet controller's responsibility to handle this +since it's a common problem. + +#### Story 2 + +I'm using statefulSet for experiments, but each time I got a broken statefulSet in rolling update, +I have to delete the pod manually after applying a fixed yaml file, it's quite annoying. + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + + +If we wrongly configure the statefulset with so called `enforcedRollingUpdate` strategy, it may lead to +some unexpected results since the controller will update the pod anyway. We should document this. + +## Design Details + + + +We'll add a new `statefulSetUpdateStrategyType` named `enforcedRollingUpdate`, see: + + ```golang + type StatefulSetUpdateStrategyType string + + const ( + RollingUpdateStatefulSetStrategyType StatefulSetUpdateStrategyType = "RollingUpdate" + OnDeleteStatefulSetStrategyType StatefulSetUpdateStrategyType = "OnDelete" + + // EnforcedRollingUpdateStatefulSetStrategyType indicates that update will be + // applied to all Pods in the StatefulSet with respect to the StatefulSet + // ordering constraints. When a scale operation is performed with this + // strategy, new Pods will be created from the specification version indicated + // by the StatefulSet's updateRevision. And whatever the pod status is healthy + // or broken, the rolling update process will not stuck. + EnforcedRollingUpdateStatefulSetStrategyType StatefulSetUpdateStrategyType = "EnforcedRollingUpdate" + ) + ``` + +When update statefulSet, we should change two places: +Place1: + +- Before, when updating statefulSet, we should make sure that current replicas should be Running, Ready and Available. +- Now when `StatefulSetUpdateStrategyType` is `EnforcedRollingUpdate`, we'll continue the process. + +Place2: + +- Before, when deleting so called condemned pods who are pods beyond the updated statefulSet replicas, we'll wait for +all predecessors to be Running and Ready prior to attempting a deletion. +- Now when StatefulSetUpdateStrategyType is `EnforcedRollingUpdate`, we'll continue the process. + +When updating pods that doesn't match the update revision, we'll keep the logic and if the largest ordinal pod failed to +turn to health, we'll not update the smaller ordinal pods. + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- `pkg/apis/apps/validation`: `2022-09-28` - `90.8%` +- `pkg/controller/statefulset`: `2022-09-28` - `85.5%` +- `pkg/registry/apps/statefulset`: `2022-09-28` - `66.7%` + +##### Integration tests + + + +We should cover below scenarios: + +- With update strategy set to `rollingUpdate`, broken statefulSet will not recover even applying with a fixed configuration. +- With update strategy set to `enforcedRollingUpdate`, broken statefulSet will recover after applying with a fixed configuration. +- With update strategy set to `enforcedRollingUpdate`, broken statefulSet will not rolling update after applying with another unfixed configuration. +- With update strategy set to `rollingUpdate` or `enforcedRollingUpdate`, and `podManagementPolicy` set to `parallel`, +broken statefulSet will recover after applying with a fixed configuration. + +##### e2e tests + + + +This feature only impacts the statefulset Controller, so integration tests should be enough. + +### Graduation Criteria + + + +#### Alpha + +- Feature implemented behind a feature flag. +- Unit and integration tests passed as designed in [TestPlan](#test-plan). + +#### Beta + +- Feature is enabled by default. +- Address reviews and bug reports from Alpha users. + +#### GA + +- No negative feedback from developers. + +### Upgrade / Downgrade Strategy + + + +#### Upgrade + +This feature is protected by the feature-gate `EnforcedRollingUpdateInStatefulSet`, and it's an opt-in strategy +for end-users to choose, it won't change previous behaviors if you don't configure anything. + +If you want to use the feature, you should enable this feature gate. + +#### Downgrade + +If you configured the rolling update strategy to `enforcedRollingUpdate`, when downgrading, you should reconfigure +this strategy to what we supported. + +### Version Skew Strategy + + + +No. It general remains the same version with api-server. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [x] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: EnforcedRollingUpdateInStatefulSet + - Components depending on the feature gate: + - kube-apiserver + - kube-controller-manager + +###### Does enabling the feature change any default behavior? + + + +No, it's backwards compatible. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +Yes, we can disable the feature gate. + +###### What happens if we reenable the feature if it was previously rolled back? + +This feature works again. + +###### Are there any tests for feature enablement/disablement? + + + +Yes, we'll cover this in unit tests. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +No. + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +No. + +###### Will enabling / using this feature result in introducing new API types? + + + +No. + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +No. + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +No. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +No. + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +No. + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +- 2022-09-26: Initial KEP Created + +## Drawbacks + + + +## Alternatives + + + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-apps/3541-add-enforcedRollingUpdate-to-statefulset-update-strategies/kep.yaml b/keps/sig-apps/3541-add-enforcedRollingUpdate-to-statefulset-update-strategies/kep.yaml new file mode 100644 index 00000000000..f602a6056a4 --- /dev/null +++ b/keps/sig-apps/3541-add-enforcedRollingUpdate-to-statefulset-update-strategies/kep.yaml @@ -0,0 +1,40 @@ +title: Add EnforcedRollingUpdate to StatefulSet Update Strategies +kep-number: 3541 +authors: + - "@kerthcet" +owning-sig: sig-apps +status: provisional +creation-date: 2022-09-26 +reviewers: + - TBD +approvers: + - TBD + +##### WARNING !!! ###### +# prr-approvers has been moved to its own location +# You should create your own in keps/prod-readiness +# Please make a copy of keps/prod-readiness/template/nnnn.yaml +# to keps/prod-readiness/sig-xxxxx/00000.yaml (replace with kep number) +#prr-approvers: + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.25" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.26" + beta: "v1.27" + stable: "" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: EnforcedRollingUpdateInStatefulSet + components: + - kube-controller-manager +disable-supported: true