Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit the number of components (kubelet, kube-proxy and rivers) that can be updated simultaneously #630

Merged
merged 7 commits into from
Jun 8, 2023

Conversation

zoetrope
Copy link
Member

@zoetrope zoetrope commented May 29, 2023

Add a new cli option max-concurrent-updates.
It can limit the number of components (kubelet, kube-proxy and rivers) that can be updated simultaneously.

This change will solve the following problems:

  • Prevent large amounts access to the Internet
  • Prevent large amounts access to APIServer
  • Prevent all kubelets from going down

Signed-off-by: zoetrope [email protected]

@zoetrope zoetrope self-assigned this May 29, 2023
zoetrope added 5 commits May 31, 2023 18:19
- Prevent large amounts access to the Internet
- Prevent large amounts access to APIServer
- Prevent all kubelets from going down

Signed-off-by: zoetrope <[email protected]>
Signed-off-by: zoetrope <[email protected]>
@zoetrope zoetrope force-pushed the concurrent-kubelet-restart branch from 73b4b03 to c6666a2 Compare May 31, 2023 09:24
@zoetrope zoetrope marked this pull request as ready for review May 31, 2023 09:57
@zoetrope zoetrope requested a review from morimoto-cybozu May 31, 2023 09:58
@zoetrope zoetrope changed the title Limit the number of kubelet restarts at one time Limit the number of components (kubelet, kube-proxy and rivers) that can be updated simultaneously May 31, 2023
Copy link
Contributor

@morimoto-cybozu morimoto-cybozu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code itself seems good to me.

}
return ops
}

func k8sOps(c *cke.Cluster, nf *NodeFilter, cs *cke.ClusterStatus) (ops []cke.Operator) {
func separateOperators(nodes []*cke.Node, createOps func(ns []*cke.Node) cke.Operator, maxConcurrentUpdates int) (ops []cke.Operator) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a trivial point, but splitOperators() would be more appropriate.
https://www.google.com/search?q=separate+slice+golang

flgCertsGCInterval = pflag.String("certs-gc-interval", "1h", "tidy interval for expired certificates")
flgSessionTTL = pflag.String("session-ttl", "60s", "leader session's TTL")
flgDebugSabakan = pflag.Bool("debug-sabakan", false, "debug sabakan integration")
flgMaxConcurrentUpdates = pflag.Int("max-concurrent-updates", 10, "the maximum number of components that can be updated simultaneously")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
flgMaxConcurrentUpdates = pflag.Int("max-concurrent-updates", 10, "the maximum number of components that can be updated simultaneously")
flgMaxConcurrentUpdates = pflag.Int("max-concurrent-updates", 10, "the maximum number of worker-node components that can be updated simultaneously")

The description above seems too simple.
It would be nice to add an explanation for how CKE uses this parameter.
How about updating the following section?

https://github.com/cybozu-go/cke/blob/v1.25.3/docs/cluster_overview.md#worker-nodes

CKE deploys each worker-node component concurrently on multiple nodes.
You can control the deployment concurrency by specifying the max-concurrent-updates command-line option.

@zoetrope zoetrope requested a review from morimoto-cybozu June 7, 2023 08:48
Copy link
Contributor

@morimoto-cybozu morimoto-cybozu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zoetrope zoetrope merged commit 6c17172 into main Jun 8, 2023
@zoetrope zoetrope deleted the concurrent-kubelet-restart branch June 8, 2023 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants