Single OCI registry: Stage 1 - container images

[piontec]

Related epics

• Stage 2: Remove operatorkit dependency from rbac-operator #3056
• Retagger changes: Prepare monitoring and alerting for zot #3068

Tasks

We discussed the new registry architecture - RFC.

We have to get out of Docker Hub by the end of 2023 - this is called 'stage 1' in the RFC. This ticket tracks and groups tasks around that.

Step 0 - Preparations

Preview Give feedback

1. Planning and RFC related issue
2. boostrap a new ACR registry #2896
   2 of 2
   componenet/registries epic/2023registries team/honeybadger +3
   

Step 1 - Setting up the new registry

Preview Give feedback

1. configure a new instance of retagger to replicate public images to the new ACR registry #2895
   9 of 11
   componenet/registries epic/2023registries team/honeybadger +3
   
2. reconfigure all the CI/CD pipelines to push new builds to the new repository #2915
   7 of 7
   componenet/registries epic/2023registries team/honeybadger +3
   
3. replicate all the internal (our components) tagged images we have in the current public registry to the new ACR #2916
   componenet/registries epic/2023registries team/honeybadger +3
4. check which repositories are now really used for all the MCs and WCs #2926
   componenet/registries epic/2023registries team/honeybadger +3
5. Discover if we need geo-replication for ACR registries. #3025
   componenet/registries team/honeybadger +2
   
6. Update the docs about newly created registries and their features #3027
   3 of 6
   componenet/registries team/honeybadger +2
   

Step 1.5 - Handling private images

Preview Give feedback

1. Setup the new private images registry on ACR #3026
   componenet/registries team/honeybadger +2
2. architect-orb push-to-registries should allow building image using private base image #3020
   component/architect team/honeybadger team/null +3
   
3. Preapring and delivering imagePullSecrets to needed MCs #3064
   12 of 12
   componenet/registries epic/2023registries team/honeybadger +3
   
4. Switching private apps to using the new imagePullSecrets #3063
   componenet/registries epic/2023registries team/honeybadger +3
5. Updating docs about private registries #3062
   +0
6. Check how private images can be still pulled from quay/docker when we switch the global registry variable to gsoci #3041
   epic/2023registries team/honeybadger +2
   
7. Sync old versions of the private images to the gsociprivate repo #3084
   componenet/registries epic/2023registries team/honeybadger +3

Step 2 - Phase out Docker Hub usage

Preview Give feedback

1. https://github.com/giantswarm/giantswarm/issues/28933
2. https://github.com/giantswarm/giantswarm/issues/30502
3. Test what happens if we run a cluster that uses docker hub access credentials that become invalid #3409
   componenet/registries team/honeybadger +2
   
4. standarization of the registry domain config value for catalogs+apps #3179
   epic/2023registries honeybadger/appplatform needs/rfc +3
5. [on-hold] Make new vintage releases that can use the new registries #3080
   0 of 2
   componenet/registries epic/2023registries team/honeybadger +3
6. Replace quay.io with the new registry in the cluster chart #3112
   Ready area/kaas needs/refinement team/turtles +4
7. Use new GS registry for core images in the cluster chart #3110
   Ready area/kaas needs/refinement team/turtles +4
8. stage switch of the default registry to gsoci #3410
   0 of 44
   componenet/registries epic/2023registries team/honeybadger +3
9. reconfigure container runtimes on existing WCs to use the new registry #2971
   epic/2023registries team/honeybadger +2
   
10. Modify repositories' CircleCI config to push to gsoci instead of Docker Hub #2979
    4 of 54
    team/honeybadger +1
    
11. Use new GS registry as the default for CAPI #3108
    0 of 4
    Ready area/kaas provider/cluster-api-aws provider/cluster-api-azure provider/vcd provider/vsphere team/turtles topic/capi +8
12. prepare all the changes so that every new cluster release uses the new registry already #3031
    2 of 2
    +0
13. reconfigure MCs to use the new registry #3032
    epic/2023registries team/honeybadger +2
    
14. add metrics and dashboards that will allow us to monitor usage of the registries from clusters #3028
    componenet/registries epic/2023registries team/honeybadger +3
15. Modify app values to replace docker.io as container registry domain with gsoci.azurecr.io #3017
    team/honeybadger +1
    
16. Create audit policy to count pods without ImagePullSecret using docker.io image #3029
    +0
17. Update dashboards to visualize metrics from audit policy #3030
    +0
18. Figure out what to do with private images that we have on quay.io
19. figure out how we can make WC deployed apps to use the new registries #3044
    epic/2023registries team/honeybadger +2
20. Use new GS registry for core images in the cluster-aws chart #3109
    area/kaas provider/cluster-api-aws team/phoenix +3
    
21. Use new GS registry for core images in the cluster-azure chart #3176
    area/kaas team/phoenix +2
    
22. Replace quay.io with the new registry in the cluster-aws chart #3111
    area/kaas team/phoenix +2
    
23. Replace quay.io with the new registry in the cluster-azure chart #3177
    area/kaas team/phoenix +2
    
24. Configure the CAPA default apps to use the new registry #3168
    area/kaas team/phoenix +2
    
25. Configure the CAPZ default apps to use the new registry #3169
    area/kaas team/phoenix team/rocket +3
    
26. Replace quay.io with the new registry in the cluster-cloud-director chart #3244
    area/kaas team/rocket topic/capi +3
    
27. Replace quay.io with the new registry in the cluster-vsphere chart #3245
    area/kaas team/rocket topic/capi +3
    
28. Replace docker.io with the new registry in the cluster-vsphere chart #3246
    area/kaas team/rocket topic/capi +3
29. Configure the CAPV default apps to use the new registry #3247
    area/kaas needs/refinement team/rocket topic/capi +4
    
30. Configure the CAPVCD default apps to use the new registry #3248
    area/kaas team/rocket topic/capi +3

Step 3 - wrapping up

Preview Give feedback

1. cleanup old registries when they are not used anymore
2. update docs about registries
3. remove pushes to old registries from retagger

Security enhancements

Preview Give feedback

1. use cosign to sign images built on circle [blocked by circleci's support for cosign]
2. generate SBoMs automatically for all image builds and upload results to the registry

[marians]

We should announce to customers in time that this change is going to happen. Here are the details we should include IMO.

• Our docker.io credentials are going to be removed from the hosts (date to be discussed)
• When using images from docker.io in workloads, please provide your own ImagePullSecret. You can start doing this immediately, no need to wait.
• Otherwise you might get affected by (probably) IP-based rate limiting. Actually Docker does not disclose details. By using your own credentials via ImagePullSecrets, you make sure that pulls are accounted for via your account(s) and you have more control over this.
• Side note: most customers use their own registries and copy the images they need from Docker Hub over to their own registries. This is the method we would recommend.

Please let's verify this messaging and also fill in the missing date info.

Also in KaaS sync the suggestion was made that we approach customers directly who, based on our data, use images from docker.io.

In addition we may also give them a dedicated dashboard to help customers track which workloads are using docker.io images.

[marians]

📉 Metrics on containers using docker.io images are now available in this Grafana Cloud dashboard

[piontec]

Ad Step 2. Notes from Whites:

MCs:

• how base images registry domains are set up? are they configurable?
  for systemd images we use a configurable docker registry. Default is docker.io and we can configure it in installations repo.
  Example: https://github.com/giantswarm/installations/blob/d6a674395835a4b06e37126f485b4dc1cf94b1ad/akita/terraform/bootstrap.sh#L69
  Changing needs rolling of all MC nodes.

WCs:

• how base images registry domains are set up? are they configurable?
  This is controller by aws-operator and configurable via config repo.
  Default is docker.io (https://github.com/giantswarm/config/blob/main/default/config.yaml#L10)
  Can be overridden per installation (https://github.com/giantswarm/config/blob/main/installations/akita/config.yaml.patch#L17)
• how do apps get registry domain set up?
  this one I'm not 100% sure, but I think this is all controlled by app operator (https://github.com/giantswarm/config/blob/main/default/apps/app-operator/configmap-values.yaml.template#L2) and it should be identical to how we did for aws-operator in the previous point.
• how do we set up registry credentials?
  We set credentials as part of containerd config file (https://github.com/giantswarm/k8scloudconfig/blob/master/files/conf/containerd-config.toml#L45) and docker config file (https://github.com/giantswarm/k8scloudconfig/blob/5ddefddd6650b16ffd0f4388e95ec7e7c12aff0d/files/conf/kubelet-docker-config.json#L4).
  Value is injected by aws-operator and I believe is configured by MC in the config repo (example https://github.com/giantswarm/config/blob/6dea50d889fe8af4769d151d359d1e2ca10e27bc/installations/visitor/secret.yaml#L13)
  Metrics:
• check if we have containerd metrics about pulls and since which release
  I think this should be included in containerd 1.7.0.
  Not shipped with flatcar stable yet :(

[marians]

I looked for internal documentation regarding registries, to be updated/reviewed. So far I found

• Handbook: Registry mirrors
• Intranet: Creating container registries for retagger
• Intranet: Changes in tooling to handle multiple image registries
  

[marians]

Since it has gotten tedious to track PRs for more than 150 repos, I created this spreadsheet:

https://docs.google.com/spreadsheets/d/1CwK4a9Q9AKZq_n2rYNcMBfUaXAaxFXCm_gvzhMmjgCM/edit#gid=0

The idea is:

1. CircleCI PR should be merged first. Extra issue: Modify repositories' CircleCI config to push to gsoci instead of Docker Hub #2979
2. Then default domain can be adjusted via another PR. (Not all of the repos we treated in (1) actually have a chart or a default domain setting. In this case it is probably not a deployed workload that generates many pulls, so these aren't important.) - Extra issue: Modify app values to replace docker.io as container registry domain with gsoci.azurecr.io #3017
3. Then a release is made

[alex-dabija]

I tested the creation of a CAPA cluster with the new registry (gsoci.azurecr.io) with the following configuration:

global:
  components:
    containerd:
      containerRegistries:
        docker.io:
          - endpoint: gsoci.azurecr.io
        gsoci.azurecr.io:
          - endpoint: gsoci.azurecr.io
  connectivity:
    availabilityZoneUsageLimit: 3
    network: {}
    topology: {}
  controlPlane: {}
  metadata:
    name: alex02
    organization: giantswarm
  nodePools:
    nodepool0:
      instanceType: m5.xlarge
      maxSize: 3
      minSize: 3
      rootVolumeSizeGB: 300
  providerSpecific: {}

I couldn't remove docker.io entirely because of how Helm merges dicts.

The cluster had the following containerd config:

version = 2

# recommended defaults from https://github.com/containerd/containerd/blob/main/docs/ops.md#base-configuration
# set containerd as a subreaper on linux when it is not running as PID 1
subreaper = true
# set containerd's OOM score
oom_score = -999
disabled_plugins = []
[plugins."io.containerd.runtime.v1.linux"]
# shim binary name/path
shim = "containerd-shim"
# runtime binary name/path
runtime = "runc"
# do not use a shim when starting containers, saves on memory but
# live restore is not supported
no_shim = false

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
# setting runc.options unsets parent settings
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "quay.io/giantswarm/pause:3.9"

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://gsoci.azurecr.io",]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gsoci.azurecr.io"]
      endpoint = ["https://gsoci.azurecr.io",]
[plugins."io.containerd.grpc.v1.cri".registry.configs]

The cluster had the following images running:

kubectl --context gs-grizzly-alex02-clientcert get pods --all-namespaces -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c
      4 docker.io/giantswarm/aws-ebs-csi-driver:v1.21.0
      3 docker.io/giantswarm/aws-ebs-csi-volume-limiter:0.1.0
      7 docker.io/giantswarm/cert-exporter:2.8.4
      1 docker.io/giantswarm/cert-manager-cainjector:v1.12.4
      2 docker.io/giantswarm/cert-manager-webhook:v1.12.4
      2 docker.io/giantswarm/cilium-operator-generic:v1.14.3
     42 docker.io/giantswarm/cilium:v1.14.3
      3 docker.io/giantswarm/configmap-reload:v0.8.0
      3 docker.io/giantswarm/coredns:1.11.1
      1 docker.io/giantswarm/csi-attacher:v4.4.1
      3 docker.io/giantswarm/csi-node-driver-registrar:v2.9.0
      1 docker.io/giantswarm/csi-provisioner:v3.6.1
      1 docker.io/giantswarm/csi-resizer:v1.8.1
      3 docker.io/giantswarm/etcd:3.5.6-0
      1 docker.io/giantswarm/etcd-kubernetes-resources-count-exporter:1.8.0
      1 docker.io/giantswarm/external-dns:v0.11.0
      3 docker.io/giantswarm/grafana-agent:v0.37.2
      1 docker.io/giantswarm/hubble-relay:v1.14.3
      1 docker.io/giantswarm/hubble-ui-backend:v0.12.1
      1 docker.io/giantswarm/hubble-ui:v0.12.1
      3 docker.io/giantswarm/kube-apiserver:v1.24.16
      3 docker.io/giantswarm/kube-controller-manager:v1.24.16
      3 docker.io/giantswarm/kube-scheduler:v1.24.16
      1 docker.io/giantswarm/kube-state-metrics:v2.10.0
      4 docker.io/giantswarm/livenessprobe:v2.11.0
      6 docker.io/giantswarm/node-exporter:v1.3.1
      2 docker.io/giantswarm/prometheus-config-reloader:v0.68.0
      1 docker.io/giantswarm/prometheus-operator:v0.68.0
      1 docker.io/giantswarm/prometheus:v2.47.1
      6 docker.io/giantswarm/promtail:2.8.4
      2 docker.io/giantswarm/vpa-admission-controller:0.14.0
      1 docker.io/giantswarm/vpa-recommender:0.14.0
      1 docker.io/giantswarm/vpa-updater:0.14.0
      1 public.ecr.aws/gravitational/teleport-distroless:14.1.3
      3 quay.io/giantswarm/amazon-eks-pod-identity-webhook-gs:v0.2.0
      3 quay.io/giantswarm/aws-cloud-controller-manager:v1.24.1
      6 quay.io/giantswarm/capi-node-labeler-app:0.3.4
      1 quay.io/giantswarm/cert-manager-controller:v1.12.4
      2 quay.io/giantswarm/chart-operator:2.35.2
      1 quay.io/giantswarm/cluster-autoscaler:v1.27.3
      6 quay.io/giantswarm/docker-kubectl:1.25.4
      2 quay.io/giantswarm/metrics-server:v0.6.4
      6 quay.io/giantswarm/net-exporter:1.18.0

I'm assuming that the docker.io images are pulled from gsoci.azurecr.io because it's the only endpoint configured.

## TODOs for KaaS
- [ ] Make the `gsoci.azurecr.io` registry the default;
- [ ] Update `imageRepository: docker.io/giantswarm` for the kubeadm control plane configuration in the cluster-aws chart;
- [ ] Update `imageRepository: docker.io/giantswarm` for the kubeadm control plane configuration in the cluster chart;
- [ ] Make `public.ecr.aws/gravitational/teleport-distroless:14.1.3` available from our registry;
- [ ] Replace `quay.io` with the new registry.

[weatherhog]

@piontec what is missing here for stage 1. Can this be closed?

[piontec]

Stage 1 is complete, 1.5 as well, but this is a big epic that covers next steps, that are not done yet. For me it's a great overview of where we are in this big story and I would keep this open, even if we're not currently actively working on it.