Fix BalanceSimilarNodeGroups when scaling from zero

[bpineau]

Use real-world nodeInfo from similar groups when BalanceSimilarNodeGroups.

Where available, the CA evaluates node groups capacities and labels using real-world nodes, for optimal accuracy (accounting for kubelet and system reserved resources, for instance).

It fallbacks to synthetic nodeInfos inferred from ASG/MIG/VMSs templates when no real nodes are available yet (ie. upscaling from zero/empty node group).

That asymmetric evaluation can prevent BalanceSimilarNodeGroups from working reliably when upscaling from zero:

• The first upscaled nodeGroup will get a node (capacity initially evaluated from ASG/MIG/VMSS template, then from the first real node, runtime labels)
• Other empty nodeGroups will likely be considered dissimilar to the first one: we're comparing ASG templates with real-world nodeInfo
• In the case of least-waste expander for instance, CA might prefer the nodegroup from real-world node, accounting for reserved resources (best usage) over the others

This change set implements Maciek Pytel suggestion (from a previous attempt at fixing this issue): we try to use real-world nodes examples from ~similar node groups, before a last resort fallback to synthetic nodeInfos built from ASG templates.

We compare nodeInfos using the FindSimilarNodeGroups processor primitives in order to leverage per cloud provider comparators (eg. label exclusions).

We look for similar nodegroups using synthetic nodeInfo from ASG/MIG (where available): comparing their nodeInfo from real-world nodes to a synthetic nodeInfo from the empty nodegroup wouldn't work (that's the root cause of this issue). When ~similar nodegroups are found that way, we use their nodeInfos built from real-world nodes as models for empty nodegroups (but keeping original location related labels).

Tested with AWS ASGs, GCP MIGs and Azure VMSS.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign feiskyer after the PR has been reviewed.
You can assign the PR to them by writing /assign @feiskyer in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[MaciekPytel]

I think this should be implemented as a processor:

• A custom implementation of NodeGroupSetProcessor (which may internally call to BalancingNodeGroupSetProcessor or reuse some of its code).
• Or, possibly, a NodeInfoProcessor (if you actually want to use template node from 'similar' group for binpacking and not just balancing).

Those processors were added specifically to allow to use custom logic to replace balancing (NodeGroupSetProcessor) and allow custom overrides to NodeInfo map produced by GetNodeInfosForGroups (NodeInfoProcessor). My reasons are:

• The current implementation adds even more complexity to what are already most complex pieces of code (and it spreads the logic across multiple parts of the code). Encapsulating all this logic in one place would improve readability.
  • The cache can be held in processor. There is no reason why a processor couldn't be stateful (and there is an existing example of a processor using internal cache: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/core/filter_out_schedulable.go#L34)
• Core logic should not assume any particular implementation of a processor. You can replace the whole balancing logic with something completely different by switching NodeGroupSetProcessor (ex. balance all nodegroups that have a certain label, or for a more concrete example - balance all MIGs in a GKE nodepool).

[mwielgus]

Ping. Please respond to @MaciekPytel comment.

[bpineau]

@mwielgus : Maciek comment makes perfect sense, I want to secure a bit of time to try implementing the suggested options (I suspect I'll come back with questions if you don't mind).

[bpineau]

Implemented @MaciekPytel suggestion in PR #3761 ; closing in favour of the new PR.