[WIP] Refactor OS/AMI flavor support adding more flexibility to configure launchtemplates

[alex-berger]

This PR is work in progress and actually a follow-up/alternative to #1346. It is not yet complete and I will only spend more time polishing it (and implementing tests) if it has any chance to be merged. If not, it might still serve as foundation for further discussions about how Karpenter could be improved to support more (real-world) use-cases.

I created this PR at this early stage (kind of in a hurry), to make sure my work/ideas are visible to others (@ellistarn), even though they might not be ready for merging yet. So, please let's focus on the overall design and the features it introduces before we go into more general code-review.

1. Issue, if available:

2. Description of changes:

• Use AMI metadata to figure out the DeviceName of the root volume. Guessing the DeviceName is a common source of errors and why should the user be required to specify it if the system can provide that information. User usually just wants to adjust the volume's type, size, IOPS or throughput and should not have to care about its device name at all.
• Use AMI metadata to obtain default device mappings and let the user override specific aspects of them (e.g. volume's size, type, IOPS, ...). This is important as some AMIs (e.g. Bottlerocket) use several volumes.
• Support advanced kubelet configurations needed to tune, harden and lock-down the kubelet. This is very important for day two operations and to comply with security best practices (needed to obtain certifications like for example ISO and SOC2, ...).
• Initial support for ContainerRuntime specific configuration (maybe it should be moved to Bottlerocket). This is important to achieve high availability and security by being able to add OCI registry mirrors at the infrastructure layer, which serve as pull-through caches and might also filter (reject) bad images.
• Refactor support for different OS/AMI flavors (also called AMIFamily on other branches) and add strongly typed configuration for each flavor. Currently this supports AmazonLinux, Bottlerocket, Ubuntu and Generic. This also introduces support for a subset of LaunchTemplate parameters for Karpenter managed LaunchTemplates. E.g. BlockDevices configuration, CPUCredits and ThreadsPerCore. ThreadsPerCore is important for security constrained environments where one is required to risks of CPU side channel attacks (like for example spectre).
• Still support pre-existing LaunchTemplates (not managed by Karpenter)
• Support palceholder for ClusterName in selectors/filters for securitygroups and subnets. This is important to allow anonymous clusters which are not aware of their cluster names to still select the appropriate subnets and security groups.

3. How was this change tested?

4. Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: link to issue
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: 57b1422

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/622b746d7ca3a200080552a9

[bwagner5]

👋 Hey apologies on the delay in getting to this. I haven't fully dug into the all the changes here, but I did notice that some of the recent changes to clean-up the cloud provider's AMI Family implementation (#1420.) have been rolled back in this PR. I think the AMI Family refactor addresses some of the clean-up, but obviously does not address some of the features you are suggesting. We are certainly planning on ramping up the amount of fields the Provisioner supports and the generated LTs should be easier to add to with the AMI Family refactor.

I think it would be useful to open issues for each of the features unlocked to promote further discussions on them. We have been thinking a lot about the generic AMI case and AMI versions. We're open to adding more kubelet params as well, it looks like this PR adds a significant number of them :) We can certainly use this PR as a mechanism to discuss these features and approaches since you've implemented a direction, but I think the features will need to be broken up into separate PRs once we decide on the direction.

And thank you for taking the time to discuss these and even implement a PoC! It is much appreciated!

[bwagner5]

We're starting to add more of these options in this PR: #1720 Thanks for brining up these issues! More to come!