unknown flag: --leader-elect, all leader-elect flags are unknown since 1.32.0

[idebeijer]

Which component are you using?:

/area cluster-autoscaler

What version of the component are you using?:

Component version:

1.32.0

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.32.0

What environment is this in?:

AWS but not with EKS just EC2 and self-managed k8s.

What did you expect to happen?:

When upgrading the cluster-autoscaler from 1.31.0 to 1.32.0 the --leader-elect flag would still exists.

What happened instead?:

The cluster-autoscaler crashed saying the --leader-elect flag is unknown:

❯ k logs cluster-autoscaler-7d767d94fd-dk97q
unknown flag: --leader-elect
Usage of ./cluster-autoscaler:
unknown flag: --leader-elect

How to reproduce it (as minimally and precisely as possible):

Start the cluster-autoscaler version 1.32.0+ and set any of the following flags:

      --leader-elect                                                       Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability. (default true)
      --leader-elect-lease-duration duration                               The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled. (default 15s)
      --leader-elect-renew-deadline duration                               The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than the lease duration. This is only applicable if leader election is enabled. (default 10s)
      --leader-elect-resource-lock string                                  The type of resource object that is used for locking during leader election. Supported options are 'leases'. (default "leases")
      --leader-elect-resource-name string                                  The name of resource object that is used for locking during leader election. (default "cluster-autoscaler")
      --leader-elect-resource-namespace string                             The namespace of resource object that is used for locking during leader election.
      --leader-elect-retry-period duration                                 The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled

Anything else we need to know?:

This bug is likely the result of c382519.

In the main func since above commit, kube_flag.InitFlags() is called before componentopts.BindLeaderElectionFlags(&leaderElection, pflag.CommandLine). When I check out the repository and call kube_flag.InitFlags() after componentopts.BindLeaderElectionFlags(&leaderElection, pflag.CommandLine) like before the above commit, then running go run main.go --help shows the flags again.

[x-504]

Hi, We are experiencing same issue on version 1.32.0

[aleksandra-malinowska]

Seems it isn't the first time it happened: #6115

Maybe we should add a comment to the code above componentopts.BindLeaderElectionFlags(&leaderElection, pflag.CommandLine) to not move it after the second kube_flag.InitFlags()?

[idebeijer]

Per your suggestion @aleksandra-malinowska, I have added a comment to try and prevent this bug from happening again in the future. (#7763)

I have also manually cherry picked the fix to cluster-autoscaler-release-1.32 since the bug is still there. It also includes the comment. (#7764)

[CameronHall]

Appreciate you fixing that so quickly @idebeijer. Is there any chance we can get a 1.32.1 release that includes the fix?

[idebeijer]

Hi @CameronHall, unfortunately I can't create a release as I'm not a maintainer and I'm fairly new to all this so I don't exactly know how the release process works for the cluster-autoscaler. I believe we can ask in the #sig-autoscaling Kubernetes Slack channel if there is any release planned or being worked on.

[Tatskaari]

Seems it isn't the first time it happened: #6115

We've had a great experience using this library for flags parsing that makes all this declarative:
https://github.com/jessevdk/go-flags

Not sure if there would be appetite to migrate but I would highly recommend if there is.