kinder: add workflow for testing super-admin.conf #2949
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| - name: ci-kubernetes-e2e-kubeadm-kinder-super-admin-{{ dashVer .KubernetesVersion }} | ||
| cluster: k8s-infra-prow-build | ||
| interval: {{ .JobInterval }} | ||
| decorate: true | ||
| labels: | ||
| preset-dind-enabled: "true" | ||
| preset-kind-volume-mounts: "true" | ||
| annotations: | ||
| testgrid-dashboards: sig-cluster-lifecycle-kubeadm | ||
| testgrid-tab-name: kubeadm-kinder-super-admin-{{ dashVer .KubernetesVersion }} | ||
| testgrid-alert-email: [email protected] | ||
| description: "OWNER: sig-cluster-lifecycle (kinder); Uses kubeadm/kinder to create a cluster and test the super-admin.conf functionality" | ||
| testgrid-num-columns-recent: "20" | ||
| {{ .AlertAnnotations }} | ||
| decoration_config: | ||
| timeout: 60m | ||
| extra_refs: | ||
| - org: kubernetes | ||
| repo: kubernetes | ||
| base_ref: {{ branchFor .KubernetesVersion }} | ||
| path_alias: k8s.io/kubernetes | ||
| - org: kubernetes | ||
| repo: kubeadm | ||
| base_ref: main | ||
| path_alias: k8s.io/kubeadm | ||
| spec: | ||
| containers: | ||
| - image: gcr.io/k8s-staging-test-infra/kubekins-e2e:{{ .TestInfraImage }}-{{ imageVer .KubernetesVersion }} | ||
| command: | ||
| - runner.sh | ||
| - "../kubeadm/kinder/ci/kinder-run.sh" | ||
| args: | ||
| - {{ .WorkflowFile }} | ||
| securityContext: | ||
| privileged: true | ||
| resources: | ||
| limits: | ||
| memory: "9000Mi" | ||
| cpu: 2000m | ||
| requests: | ||
| memory: "9000Mi" | ||
| cpu: 2000m |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,299 @@ | ||
| # IMPORTANT! this workflow is imported by super-admin-* workflows. | ||
| version: 1 | ||
| summary: | | ||
| This workflow implements a sequence of tasks used test the proper functioning | ||
| of kubeadm with the super-admin.conf functionality. | ||
| vars: | ||
| # vars defines default values for variable used by tasks in this workflow; | ||
| # those values might be overridden when importing this files. | ||
| kubernetesVersion: v1.13.5 | ||
| upgradeVersion: v1.13.5 | ||
| controlPlaneNodes: 3 | ||
| workerNodes: 2 | ||
| baseImage: kindest/base:v20221102-76f15095 # has containerd | ||
| image: kindest/node:test | ||
| clusterName: kinder-super-admin | ||
| kubeadmVerbosity: 6 | ||
| tasks: | ||
| - name: pull-base-image | ||
| description: | | ||
| pulls kindest/base image with docker in docker and all the prerequisites necessary for running kind(er) | ||
| cmd: docker | ||
| args: | ||
| - pull | ||
| - "{{ .vars.baseImage }}" | ||
| - name: add-kubernetes-versions | ||
| description: | | ||
| creates a node-image-variant by adding a Kubernetes version | ||
| cmd: kinder | ||
| args: | ||
| - build | ||
| - node-image-variant | ||
| - --base-image={{ .vars.baseImage }} | ||
| - --image={{ .vars.image }} | ||
| - --with-init-artifacts={{ .vars.kubernetesVersion }} | ||
| - --with-upgrade-artifacts={{ .vars.upgradeVersion }} | ||
| - --loglevel=debug | ||
| timeout: 15m | ||
| - name: create-cluster | ||
| description: | | ||
| create a set of nodes ready for hosting the Kubernetes cluster | ||
| cmd: kinder | ||
| args: | ||
| - create | ||
| - cluster | ||
| - --name={{ .vars.clusterName }} | ||
| - --image={{ .vars.image }} | ||
| - --control-plane-nodes={{ .vars.controlPlaneNodes }} | ||
| - --worker-nodes={{ .vars.workerNodes }} | ||
| - --loglevel=debug | ||
| timeout: 5m | ||
| - name: pre-init | ||
| description: | | ||
| Run commands before kubeadm init is called on a primary CP node | ||
| cmd: /bin/bash | ||
| args: | ||
| - -c | ||
| - | | ||
| set -x | ||
| CMD=docker exec {{ .vars.clusterName }}-control-plane-1 | ||
|
|
||
| # Generate CA, and kubeconfig files | ||
| ${CMD} kubeadm init phase certs ca || exit 1 | ||
| ${CMD} kubeadm init phase kubeconfig admin || exit 1 | ||
| ${CMD} kubeadm init phase kubeconfig super-admin || exit 1 | ||
|
|
||
| # Both admin.conf and super-admin.conf must exist | ||
| ${CMD} test -f /etc/kubernetes/admin.conf || exit 1 | ||
| ${CMD} test -f /etc/kubernetes/super-admin.conf || exit 1 | ||
|
|
||
| # Check certificate subjects | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = kubeadm:cluster-admins, CN = kubernetes-admin" || exit 1 | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/super-admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = system:masters, CN = kubernetes-super-admin" || exit 1 | ||
|
|
||
| # Make sure that the check-expiration and renew commands do not return errors | ||
| ${CMD} sudo kubeadm certs renew admin.conf || exit 1 | ||
| ${CMD} sudo kubeadm certs renew super-admin.conf || exit 1 | ||
| ${CMD} kubeadm certs check-expiration || exit 1 | ||
|
|
||
| # Delete super-admin.conf and make sure check-expiration and renew do not return errors | ||
| ${CMD} rm -f /etc/kubernetes/super-admin.conf | ||
| ${CMD} sudo kubeadm certs renew super-admin.conf || exit 1 | ||
| ${CMD} kubeadm certs check-expiration || exit 1 | ||
|
|
||
| # Cleanup | ||
| ${CMD} rm -f /etc/kubernetes/pki/ca.* | ||
| ${CMD} rm -f /etc/kubernetes/*.conf | ||
|
|
||
| # Ensure exit status of 0 | ||
| exit 0 | ||
|
Comment on lines
+88
to
+89
There was a problem hiding this comment. this was added last minute since sometimes passing a bash script to in some cases in the past we have seen this forces an exit 0 status once the EOF for a script was reached, There was a problem hiding this comment. seems good as it reached the end of the script here. |
||
| timeout: 5m | ||
| - name: init | ||
| description: | | ||
| Initializes the Kubernetes cluster with version "initVersion" | ||
| by starting the boostrap control-plane nodes | ||
| cmd: kinder | ||
| args: | ||
| - do | ||
| - kubeadm-init | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| - --kubeadm-verbosity={{ .vars.kubeadmVerbosity }} | ||
| - --copy-certs=auto | ||
| timeout: 5m | ||
| - name: post-init | ||
| description: | | ||
| Run commands after kubeadm init is called on a primary CP node | ||
| cmd: /bin/bash | ||
| args: | ||
| - -c | ||
| - | | ||
| set -x | ||
| CMD=docker exec {{ .vars.clusterName }}-control-plane-1 | ||
|
|
||
| # Both admin.conf and super-admin.conf must exist | ||
| ${CMD} test -f /etc/kubernetes/admin.conf || exit 1 | ||
| ${CMD} test -f /etc/kubernetes/super-admin.conf || exit 1 | ||
|
|
||
| # Check certificate subjects | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = kubeadm:cluster-admins, CN = kubernetes-admin" || exit 1 | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/super-admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = system:masters, CN = kubernetes-super-admin" || exit 1 | ||
|
|
||
| # Delete super-admin.conf to make sure this version of kubeadm creates it on upgrade | ||
| ${CMD} rm -f /etc/kubernetes/super-admin.conf | ||
|
|
||
| # Ensure exit status of 0 | ||
| exit 0 | ||
| timeout: 5m | ||
| - name: join | ||
| description: | | ||
| Join the other nodes to the Kubernetes cluster | ||
| cmd: kinder | ||
| args: | ||
| - do | ||
| - kubeadm-join | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| - --kubeadm-verbosity={{ .vars.kubeadmVerbosity }} | ||
| - --copy-certs=auto | ||
| timeout: 10m | ||
| - name: post-join | ||
| description: | | ||
| Run commands after kubeadm join is called on a secondary CP node | ||
| cmd: /bin/bash | ||
| args: | ||
| - -c | ||
| - | | ||
| set -x | ||
| CMD=docker exec {{ .vars.clusterName }}-control-plane-2 | ||
|
|
||
| # admin.conf must exist | ||
| ${CMD} test -f /etc/kubernetes/admin.conf || exit 1 | ||
|
|
||
| # super-admin.conf must not exist | ||
| ${CMD} test -f /etc/kubernetes/super-admin.conf && exit 1 | ||
|
|
||
| # Check certificate subject | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = kubeadm:cluster-admins, CN = kubernetes-admin" || exit 1 | ||
|
|
||
| # Check if 'kubeadm init' created the RBAC permissions for the admin.conf user | ||
| ${CMD} kubectl -n kube-system --kubeconfig /etc/kubernetes/admin.conf get cm kubeadm-config || exit 1 | ||
|
|
||
| # Ensure exit status of 0 | ||
| exit 0 | ||
| timeout: 5m | ||
| - name: upgrade | ||
| description: | | ||
| upgrades the cluster to Kubernetes "upgradeVersion" | ||
| cmd: kinder | ||
| args: | ||
| - do | ||
| - kubeadm-upgrade | ||
| - --upgrade-version={{ .vars.kubernetesVersion }} | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| - --kubeadm-verbosity={{ .vars.kubeadmVerbosity }} | ||
| timeout: 15m | ||
| - name: post-upgrade-primary-cp | ||
| description: | | ||
| Run commands after kubeadm upgrade is called on the primary CP node | ||
| cmd: /bin/bash | ||
| args: | ||
| - -c | ||
| - | | ||
| set -x | ||
| CMD=docker exec {{ .vars.clusterName }}-control-plane-1 | ||
|
|
||
| # Both admin.conf and super-admin.conf must exist | ||
| ${CMD} test -f /etc/kubernetes/admin.conf || exit 1 | ||
| ${CMD} test -f /etc/kubernetes/super-admin.conf || exit 1 | ||
|
|
||
| # Check certificate subjects | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = kubeadm:cluster-admins, CN = kubernetes-admin" || exit 1 | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/super-admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = system:masters, CN = kubernetes-super-admin" || exit 1 | ||
|
|
||
| # Check if the admin.conf user still has RBAC permissions | ||
| ${CMD} kubectl -n kube-system --kubeconfig /etc/kubernetes/admin.conf get cm kubeadm-config || exit 1 | ||
|
|
||
| # Ensure exit status of 0 | ||
| exit 0 | ||
| timeout: 5m | ||
| - name: post-upgrade-secondary-cp | ||
| description: | | ||
| Run commands after kubeadm upgrade is called on a secondary CP node | ||
| cmd: /bin/bash | ||
| args: | ||
| - -c | ||
| - | | ||
| set -x | ||
| CMD=docker exec {{ .vars.clusterName }}-control-plane-2 | ||
|
|
||
| # admin.conf must exist | ||
| ${CMD} test -f /etc/kubernetes/admin.conf || exit 1 | ||
|
|
||
| # super-admin.conf must not exist | ||
| ${CMD} test -f /etc/kubernetes/super-admin.conf && exit 1 | ||
|
|
||
| # Check certificate subject | ||
| ${CMD} grep 'client-certificate-data' /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = kubeadm:cluster-admins, CN = kubernetes-admin" || exit 1 | ||
|
|
||
| # Check if the admin.conf user still has the RBAC permissions | ||
| ${CMD} kubectl -n kube-system --kubeconfig /etc/kubernetes/admin.conf get cm kubeadm-config || exit 1 | ||
|
|
||
| # Ensure exit status of 0 | ||
| exit 0 | ||
| timeout: 5m | ||
| - name: cluster-info | ||
| description: | | ||
| Runs cluster-info | ||
| cmd: kinder | ||
| args: | ||
| - do | ||
| - cluster-info | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| - name: e2e-kubeadm | ||
| description: | | ||
| Runs kubeadm e2e tests | ||
| cmd: kinder | ||
| args: | ||
| - test | ||
| - e2e-kubeadm | ||
| - --test-flags=--report-dir={{ .env.ARTIFACTS }} --report-prefix=e2e-kubeadm | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| timeout: 10m | ||
| - name: get-logs | ||
| description: | | ||
| Collects all the test logs | ||
| cmd: kinder | ||
| args: | ||
| - export | ||
| - logs | ||
| - --loglevel=debug | ||
| - --name={{ .vars.clusterName }} | ||
| - "{{ .env.ARTIFACTS }}" | ||
| force: true | ||
| timeout: 5m | ||
| # kind export log is know to be flaky, so we are temporary ignoring errors in order | ||
| # to make the test pass in case everything else passed | ||
| # see https://github.com/kubernetes-sigs/kind/issues/456 | ||
| ignoreError: true | ||
| - name: reset | ||
| description: | | ||
| Exec kubeadm reset | ||
| cmd: kinder | ||
| args: | ||
| - do | ||
| - kubeadm-reset | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| - --kubeadm-verbosity={{ .vars.kubeadmVerbosity }} | ||
| force: true | ||
| - name: post-reset | ||
| description: | | ||
| Run commands after kubeadm reset is called on the primary CP node | ||
| cmd: /bin/bash | ||
| args: | ||
| - -c | ||
| - | | ||
| set -x | ||
| CMD=docker exec {{ .vars.clusterName }}-control-plane-1 | ||
|
|
||
| # Both admin.conf and super-admin.conf must not exist after reset | ||
| ${CMD} test -f /etc/kubernetes/admin.conf && exit 1 | ||
| ${CMD} test -f /etc/kubernetes/super-admin.conf && exit 1 | ||
|
|
||
| # Ensure exit status of 0 | ||
| exit 0 | ||
| timeout: 5m | ||
| - name: delete | ||
| description: | | ||
| Deletes the cluster | ||
| cmd: kinder | ||
| args: | ||
| - delete | ||
| - cluster | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| force: true | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| version: 1 | ||
| summary: | | ||
| This workflow tests the proper functioning of the {{ .KubernetesVersion }} version of both kubeadm | ||
| and Kubernetes when using the super-admin.conf feature. | ||
| test grid > https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-super-admin-{{ dashVer .KubernetesVersion }} | ||
| config > https://git.k8s.io/test-infra/config/jobs/kubernetes/sig-cluster-lifecycle/{{ .TargetFile }} | ||
| vars: | ||
| kubernetesVersion: "\{\{ resolve `ci/{{ ciLabelFor .KubernetesVersion }}` \}\}" | ||
| upgradeVersion: "\{\{ resolve `ci/{{ ciLabelFor .KubernetesVersion }}` \}\}" | ||
| tasks: | ||
| - import: super-admin-tasks.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| # AUTOGENERATED by https://git.k8s.io/kubeadm/kinder/ci/tools/update-workflows | ||
| version: 1 | ||
| summary: | | ||
| This workflow tests the proper functioning of the latest version of both kubeadm | ||
| and Kubernetes when using the super-admin.conf feature. | ||
| test grid > https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-super-admin-latest | ||
| config > https://git.k8s.io/test-infra/config/jobs/kubernetes/sig-cluster-lifecycle/kubeadm-kinder-super-admin.yaml | ||
| vars: | ||
| kubernetesVersion: "{{ resolve `ci/latest` }}" | ||
| upgradeVersion: "{{ resolve `ci/latest` }}" | ||
| tasks: | ||
| - import: super-admin-tasks.yaml |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i forgot to add quotes here.
it causes these CMD commands to fail, must have happened when i synced the template and the real workflow in ci/workflows.
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-super-admin-latest/1719602416766160896/build-log.txt
https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-super-admin-latest
will send a PR and self LGTM.