Upgrade EKS-D packages - resolve EBS PVC attaching for a long time

[ilia-medvedev-codefresh]

What I'd like:
We are currently experiencing an issue on our EKS clusters with PVC provisioning taking a long time. Solution described in kubernetes-sigs/aws-ebs-csi-driver#1302 (comment) and are heavily utilizing Bottlerocket AMI's in our clusters.
The issue originates in Kubelet and is resolved in latest patch versions.
For Bottlerocket to include those patches EKS-D packages need to be updated (Like was done previously in #3119).
Minimal versions of EKS-D containing the fix:

• EKS-D 1.24-19
• EKS-D 1.25-13
• EKS-D 1.26-11
• EKS-D 1.27-5

[gthao313]

@ilia-medvedev-codefresh Thanks for opening this issue! Bottlerocket team will investigate on it.

[gthao313]

Update EKS-D PRs had been merged.
#3234
#3235
#3236
#3237

^ EKS-D bumps are not to the versions listed in the issue description

[stmcginnis]

EKS-D bumps are not to the versions listed in the issue description. Reopening until we can get to those releases.

[stmcginnis]

Ah, my mistake here. The description actually calls out EKS-D package versions and not K8s point releases. I mixed that up in my head. :D So this is good!

[diranged]

When will an updated 1.14.x release come out with these patches?

[stmcginnis]

Hey @diranged! These updates were included in the 1.14.2 release. A couple weeks ago 1.14.3 also came out, with an update for the EKS-D K8s 1.23 variants.

[diranged]

@stmcginnis,
Indeed you are right - I misread the release notes and thought that 1.26.6-6 was the update. Given that, we are actually rolling our clusters back right now from 1.26-1.14.3 to 1.26-1.14.1. After doing the upgrade from 1.14.1 -> 1.14.3, we began to see significant problems with pod terminations when PVCs (EBS or EFS) were involved. Virtually all of the symptoms discussed in kubernetes-sigs/aws-ebs-csi-driver#1302 (comment) and kubernetes/kubernetes#116138.

I wonder what is causing our issue… could it be we need to upgrade something else in conjunction with the Bottlerocket AMI update?

[stmcginnis]

Yikes! Would you mind opening a new issue on that? Sounds like something that will need to be tracked down, so good if we have a new issue to capture those details and resolution.

[diranged]

@stmcginnis Done at #3370

[shay-ul]

@diranged we're also on 1.14.3 nodes and still see this issue. Is it possible that something in the ebs-csi-driver needs to be adjusted, such as describe here kubernetes-sigs/aws-ebs-csi-driver#1302 (comment) ?
Would love to hear any insights so far. thanks

Edit: Our issue was resolved with updating ebs-csi-driver to the latest version.