Cordon Drifted nodes before processing evictions

[sidewinder12s]

Tell us about your request

We currently use a tool which will roll all nodes in an ASG when the launch template configuration changes. This is usually in response to AMI Upgrades.

One of the features of this tool is that when it detects this change, it will cordon all nodes in the ASG before it starts replacing nodes. This ensures that workloads on evicted nodes do not reschedule onto nodes that are about to get killed again.

Can Karpenter implement this for it's Drift calculations?

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

We have quite a few services that do not handle restart well or rapid restarts well and would like to minimize how many times they get restarted.

Are you currently working around this issue?

No feature like this in Karpenter.

Additional Context

No response

Attachments

No response

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jukie]

Isn't this already the case?
https://github.com/aws/karpenter/blob/39b48fecef170ee725a12031a20cf72da09de860/designs/node-upgrades.md?plain=1#L26-L28

[njtran]

Hey @sidewinder12s, this is a similar to #622. Would this help in the expiration case as well?

[sidewinder12s]

I think so. I'd generally consider this ask be, if Karpenter knows it is removing a batch of nodes, it should cordon the batch before taking action to ensure we don't reschedule workloads multiple times while it is performing the change.

[sidewinder12s]

Isn't this already the case?

https://github.com/aws/karpenter/blob/39b48fecef170ee725a12031a20cf72da09de860/designs/node-upgrades.md?plain=1#L26-L28

My read of that section is it is not clear that Karpenter will cordon all drifted nodes before processing, or only as part of a cordon and drain action.

[njtran]

That makes sense. This was one of the first considerations we had in the deprovisioning logic a while ago. The idea was to not cordon the nodes to allow the capacity to be utilized in case the nodes couldn't be deprovisioned for an unknown length of time (e.g. do-not-evict pods, blocking PDBS, unable to create a replacement node). If a node could not be deprovisioned, keeping around capacity that couldn't be scheduled to could incur some unwanted costs.

All in all, I'm open to making this configurable. Can you share a bit of your requirements on expiry and drift for nodes? How important do you see deprovisioning is for a node that is expired or drifted?

[sidewinder12s]

The one behavior we'd had issues with in a custom batch autocaler was that our preferred use of the do-not-evict annotation was to not evict singleton pods that were processing things. However the pod would eventually finish. If we do not cordon the node, it is likely the node will never empty out so that it can be replaced because new work will keep coming in that can use it.

One of the worse uses was using it for workloads that cannot handle disruption, because it means we must take manual action to perform cluster maintenance.

I think in general we're ok letting things block actions, but I think we'd want something like MachineDisruptionGate to eventually force the controller to take action to bring the cluster back into conformance with config. We run many large clusters with many different users and as we scale we're finding it difficult to coordinate action if we give users too many toggles to block maintenance and basically requires us to move to something like RDS maintenance windows where we basically tell everyone, we'll do work during these hours if your workload cannot handle the normal action of the system. Setting a TTL on nodes as policy is another action we're taking to basically stop letting the cluster keep nodes forever (so through Policy and action making it explicit to customers that you will need to deal with disruption)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[jukie]

/remove-lifecycle rotten

[jmdeal]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jmdeal]

#1314 will address this issue, albeit through a slightly different mechanism. Rather than cordoning the nodes with a NoSchedule taint, Karpenter will apply a PreferNoSchedule taint to voluntary disruption candidates, and it will apply an in-memory NoSchedule taint to drifted nodes. We don't apply this in-memory taint to underutilized nodes since additional pods scheduling is the signal we use to cancel a consolidation decision. The in-memory taint combined with the PreferNoSchedule taint ensures that Karpenter will pre-spin new capacity for drifted nodes, rather than evicting and rescheduling pods to other drifted nodes.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[jmdeal]

/remove-lifecycle stale
/lifecycle frozen

Going to freeze this issue, #1314 should address this issue but there is a race condition that needs to be fixed first. Hopefully I should be able to prioritize this in the next few weeks.

[andyspiers]

#1314 has now been closed off without merging so where does that leave this issue?