Node is "NotReady" and waiting at "Terminating" for hours #1573
Comments
k8s-ci-robot
added
the
needs-triage
|
This issue is currently awaiting triage. If Karpenter contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
omg, after 6h later, still pods at "Terminating" status and node is "NotReady".
btw, instance is m5.large. And I got new instance stdout logs: |
|
If you're willing to try Karpenter 1.0 (newly released), you might see better behavior or diagnostics. I'd give it a go, honestly. |
|
@sftim thanks for suggestion, I'll try it but why karpenter or K8S doesn't intervene this situation? 18h passed and they are still waiting NotReady and Terminating. Is there any parameter to force terminate notready nodes? 
|
|
HI @ibalat, |
|
hi @jigisha620 , actually, nodes had initialized because these nodes are becoming "Ready", then pods are being scheduling and finally after a while (~30-60mins later) node is passing "NotReady" status. So, they work properly for a while. I tried to upgrade v1.0.0 but still same problem occur. I am sharing my nodeclass, nodepool and nodeclaim configs. Btw, do you know why pods still waiting at "Terminating" status? K8s or karpenter can force delete them after a while? Is there any config for that? Also I found newly events, maybe they are related with this issue. Their repeat count so much 
|
|
I think that the snippet that you have shared with "No allowed disruptions for disruption reason" is not the problem here. The nodes that you have, were already in |
|
sure, between 05:58:24 and 06:09:12 3 nodes became NotReady and I saw them lively. But no related log :( You can see all logs between these times: |
|
new update: not deletable node (although terminationGracePeriod: 5m and passed more time) show some events, maybe it can help
Node's nodeclaim have events below:
pods in node are waiting "Terminating" state and don't have any event or log at describe. After I deleted nodeclaim manually, node deleted (But passed graceperiodtime). |
|
|
|
No prestop hook, finalizer or another thing. Just waiting like at screenshots. |
|
This is not necessarily an issue from Karpenter. To investigate further, we will have to take a look at the kubelet logs to know why pods remained stuck at Terminating. Since you are using an eks ami, you can run a script that's on your worker node at /etc/eks called log-collector-script which would help us get the kubelet logs. If you have AWS premium support then you can open a ticket to investigate those logs or you can send them over and I can try looking into them. |
|
when it happens, I couldn't login EC2, it doesn't response. But I could get stdout, it below. |
|
we see this too many times |
|
What do the disk IOPS, disk idle time, and memory metrics look like for the affected hosts? Could this be the problem described in bottlerocket-os/bottlerocket#4075 (comment)? (applicable to Bottlerocket, but also observed with AL2). |
|
I had removed karpenter and reinstalled cluster autoscaler. But I can test it again in this week. After test, I will share results with you |
|
This is a common thing with most kubernetes providers/autoscalers. For some reason the general position of k8s (as a whole) is to not touch nodes that are stuck like this. This is a philosophical dilemma essentially. |
|
We also experience this quite frequently, but only with nodes that Karpenter has scheduled for disruption. If I totally disallow Karpenter from replacing nodes by setting the node budget to 0, the issue does not occur at all. If I let Karpenter disrupt nodes, then we see the issue reappearing very frequently. Karpenter version 1.0.2 |
|
@ibalat Have you attempted cutting a ticket with AWS to investigate how the node got partitioned from the control plane? |
|
/assign @garvinp-stripe |
|
@engedaam: GitHub didn't allow me to assign the following users: garvinp-stripe. Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/assign @GnatorX |
No, I had removed karpenter because of the issues and couldn't try again. |
|
Is this issue only showing up with Karpenter? What is weird to me is that this is the normal way with how autoscaler handles partitioned nodes as mentioned by @dcherniv Official docs: However I wonder if you have something configured differently on your nodes between Karpenter vs cluster-autoscaler for node termination. |
Description
Observed Behavior:
{"level":"INFO","time":"2024-08-14T12:13:23.794Z","logger":"controller","message":"pod xxxx has a preferred Anti-Affinity which can prevent consolidation","commit":"490ef94","controller":"provisioner"}Expected Behavior:
Reproduction Steps (Please include YAML):
I don't have any idea. It occur periodically
Versions:
kubectl version): 1.30The text was updated successfully, but these errors were encountered: