Karpenter should taint nodes on startup with karpenter.sh/unregistered:NoSchedule and remove once the node is registered

[jonathan-innis]

Description

What problem are you trying to solve?

Karpenter currently doesn't taint nodes in its userData, but it chooses to propagate some labels, annotations, etc. down to nodes when they join the cluster. There is currently a race that is present between the Karpenter controller's apply of the labels to the node and the kube-scheduler being able to bind pods against the node. It's possible that Karpenter may not have applied labels yet when the kube-scheduler determines that it can start to schedule pods against the node.

This problem is more prevalent with DaemonSets, where they can tolerate the node.kubernetes.io/not-ready taint, meaning that they can be scheduled against the node as soon as it joins. This is definitely less of a problem for application pods that have to wait for the node to go ready, but can still be an issue if the Karpenter controller is down.

This also isn't a large issue, in general, since the Karpenter cloudproviders will generally pass all of the labels down from a NodeClaim into the relevant startup script that is passed to the kubelet. This means that those labels will be there atomically on kubelet start, which, again, removes the chance that this race can occur.

Realistically, we shoudn't have to rely on this behavior and should just take a stance, similar to the external cloudprovider implementation with its taint that a node gets tainted on startup with a known taint and Karpenter removes the taint from the node once it has finished its registration process.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

There is some relevant conversation around this topic that I found here as I was exploring what it would look like if Karpenter were to allow users to specify the node-role.kubernetes.io label in the NodePool configuration.

[sftim]

• I think it's reasonable to taint with the NoExecute form as well. Pods that want to tolerate it can do.
• We can emit metrics about how many nodes we launched and how many nodes made it to the point that we successfully removed this taint from that node (eg, per NodePool). Those counters would help make node launch failures more visible.

[jonathan-innis]

I think it's reasonable to taint with the NoExecute form as well

Do we know why the external cloudprovider implementation uses the NoSchedule taint vs. the NoExecute taint. I'd imagine that it realistically doesn't matter much if you're adding the taint on creation of the node since there's nothing that's going to be schedule there anyways.

[fmuyassarov]

Hi @jonathan-innis . Can I offer help to work on this task ?

[rschalo]

Do we know why the external cloudprovider implementation uses the NoSchedule taint vs. the NoExecute taint.

As far as I can tell, the NoSchedule taint more or less has the same behavior as NoExecute except that NoExecute is more flexible and evicts current pods with the toleration. If racing is a concern then doesn't the NoExecute taint make more sense because of that eviction behavior?

[jonathan-innis]

I'm not sure it really makes a huge difference because the taint exists on creation so nothing should schedule anyways.

[rschalo]

Fair enough, in that case then I think 'NoEffect' still works because if a cx would want a pod to tolerate the taint then they have more control over how with access to 'tolerationSeconds'. WDYT?

[jonathan-innis]

If the pod is never scheduled to the node in the first place, I don't think it makes any difference. Seems fine either way to me.