From c657243db887bfd0379e2face227ba704d16e7cf Mon Sep 17 00:00:00 2001 From: Aleksandr Arefev <39635005+alexarefev@users.noreply.github.com> Date: Thu, 2 Nov 2023 10:06:15 +0300 Subject: [PATCH 01/12] CIS --- documentation/CIS.md | 131 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 documentation/CIS.md diff --git a/documentation/CIS.md b/documentation/CIS.md new file mode 100644 index 000000000..ba3dc573e --- /dev/null +++ b/documentation/CIS.md @@ -0,0 +1,131 @@ +# Disable anonymous authentication for `kube-apiserver` + +The `--anonymous-auth` option manages anonymous requests to the `kube-apiserver`. By default it enables anonymous requests. + +## Prerequisites + +* Working Kubernetes cluster +* The following RBAC resources: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: healthz +rules: +- nonResourceURLs: ["/readyz"] + verbs: ["get"] +- nonResourceURLs: ["/livez"] + verbs: ["get"] +- nonResourceURLs: ["/healthz"] + verbs: ["get"] +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: healthz + namespace: kube-system +--- +apiVersion: v1 +kind: Secret +metadata: + annotations: + kubernetes.io/service-account.name: healthz + name: token-healthz + namespace: kube-system +type: kubernetes.io/service-account-token +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: healthz +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: view +subjects: +- kind: ServiceAccount + name: healthz + namespace: kube-system +``` + +## Disabling procedure + +1. Add `anonymous-auth: "false"` into the `kubeadm-config` configmap e.g.: + +```yaml +apiVersion: v1 +data: + ClusterConfiguration: | + apiServer: + certSANs: + - 192.168.56.106 + - ubuntu + extraArgs: + anonymous-auth: "false" +... +``` + +2. Change the `kube-apiserver` manifest on each control plane nodes one by one according to the following example: + +```yaml +apiVersion: v1 +kind: Pod +metadata: +... + name: kube-apiserver + namespace: kube-system +spec: + containers: + - command: + - kube-apiserver + - --anonymous-auth=false +... + livenessProbe: + failureThreshold: 8 + httpGet: + host: 192.168.56.106 + path: /livez + port: 6443 + scheme: HTTPS + httpHeaders: + - name: Authorization + value: Bearer +... + readinessProbe: + failureThreshold: 3 + httpGet: + host: 192.168.56.106 + path: /readyz + port: 6443 + scheme: HTTPS + httpHeaders: + - name: Authorization + value: Bearer +... + startupProbe: + failureThreshold: 24 + httpGet: + host: 192.168.56.106 + path: /livez + port: 6443 + scheme: HTTPS + httpHeaders: + - name: Authorization + value: Bearer +... +``` + +Where TOKEN is the result of the following command: + +``` +kubectl -n kube-system get secret token-healthz -o jsonpath='{.data.token}' | base64 --decode +``` + +## Limitations + +If the `--anonymous-auth` is set to `false` the upgrade and node addition procedures need some changes in workflow. The upgrade procedure needs enabling `anonymous-auth` before the `kubeadm upgrade` run. + +The node addition procedure affects if the control plane node is being added. After new control plane node has successfully added, the [disabling procedure](#disabling-procedure) should be performed on that node. + +Besides, disabled anonymous resuests to `kube-apiserver` need changes in monitoring system, if the resources like `healthz`, `readyz`, and `livez` are used in the system. From fcdca8031f3ff2b57ab4d5f642920d675ac1ab3f Mon Sep 17 00:00:00 2001 From: Aleksandr Arefev <39635005+alexarefev@users.noreply.github.com> Date: Fri, 3 Nov 2023 08:34:01 +0300 Subject: [PATCH 02/12] CIS related doc update --- documentation/{CIS.md => internal/Hardening.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename documentation/{CIS.md => internal/Hardening.md} (100%) diff --git a/documentation/CIS.md b/documentation/internal/Hardening.md similarity index 100% rename from documentation/CIS.md rename to documentation/internal/Hardening.md From 3867df0df7dde857f000d2111acf7b66c7d72f33 Mon Sep 17 00:00:00 2001 From: Aleksandr Arefev <39635005+alexarefev@users.noreply.github.com> Date: Fri, 3 Nov 2023 09:12:53 +0300 Subject: [PATCH 03/12] encryption --- documentation/Maintenance.md | 219 --------------------------- documentation/internal/Hardening.md | 227 +++++++++++++++++++++++++++- 2 files changed, 223 insertions(+), 223 deletions(-) diff --git a/documentation/Maintenance.md b/documentation/Maintenance.md index 89c98fe32..1391cc6bb 100644 --- a/documentation/Maintenance.md +++ b/documentation/Maintenance.md @@ -1510,225 +1510,6 @@ plugins: ``` -## Data Encryption in Kubernetes - -The following section describes the Kubernetes cluster capabilities to store and manipulate encrypted data. - -### Enabling Encryption - -ETCD as a Kubernetes cluster storage can interact with encrypted data. The encryption/decryption procedures are the part of `kube-apiserver` functionality. - -An example of the `EncryptionConfiguration` file is as follows: - -```yaml -apiVersion: apiserver.config.k8s.io/v1 -kind: EncryptionConfiguration -resources: - - resources: - - secrets - - configmaps - providers: - - aesgcm: - keys: - - name: key1 - secret: c2VjcmV0IGlzIHNlY3VyZQ== - - name: key2 - secret: dGhpcyBpcyBwYXNzd29yZA== - - aescbc: - keys: - - name: key1 - secret: c2VjcmV0IGlzIHNlY3VyZQ== - - name: key2 - secret: dGhpcyBpcyBwYXNzd29yZA== - - secretbox: - keys: - - name: key1 - secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= - - identity: {} -``` - -It should be created preliminarily and placed in the `/etc/kubernetes/enc/` directory. - -The next step is to enable the encryption settings in `kubeadm-config`: -```yaml -data: - ClusterConfiguration: | - apiServer: - ... - extraArgs: - ... - encryption-provider-config: /etc/kubernetes/enc/enc.yaml - extraVolumes: - ... - - hostPath: /etc/kubernetes/enc - mountPath: /etc/kubernetes/enc - name: enc - pathType: DirectoryOrCreate -``` - -There is an `--encryption-provider-config` option that points to the `EncryptionConfiguration` file location. The `kube-apiserver` should have the following parts in the manifest yaml: - -```yaml -... -spec: - containers: - - command: - - kube-apiserver - ... - - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml - ... - volumeMounts: - - name: enc - mountPath: /etc/kubernetes/enc - readonly: true - ... - volumes: - - name: enc - hostPath: - path: /etc/kubernetes/enc - type: DirectoryOrCreate -``` - -In the above case, the `secrets` and `configmaps` are encrypted on the first key of the `aesgcm` provider, but the previously encrypted `secrets` and `configmaps` are decrypted on any keys of any providers that are matched. This approach allows to change both encryption providers and keys during the operation. The keys should be random strings in base64 encoding. `identity` is the default provider that does not provide any encryption at all. -For more information, refer to [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/). - -### Integration with External KMS - -There is an encryption provider `kms` that allows using an external `Key Management Service` for the key storage, therefore the keys are not stored in the `EncryptionConfiguration` file, which is more secure. The `kms` provider needs to deploy a KMS plugin for further use. -The `Trousseau` KMS plugin is an example. It works through a unix socket, therefore `Trousseau` pods must be run on the same nodes as `kube-apiserver`. In case of using the KMS provider, the `EncryptionConfiguration` is as follows (`Vault` is a KMS): - -```yaml -apiVersion: apiserver.config.k8s.io/v1 -kind: EncryptionConfiguration -resources: - - resources: - - secrets - - configmaps - providers: - - kms: - name: vaultprovider - endpoint: unix:///opt/vault-kms/vaultkms.socket - cachesize: 100 - timeout: 3s - - identity: {} -``` - -Also, unix socket must be available for `kube-apiserver`: - - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: kube-apiserver - namespace: kube-system -spec: - containers: - - command: - volumeMounts: - - mountPath: /opt/vault-kms/vaultkms.socket - name: vault-kms - ... - volumes: - - hostPath: - path: /opt/vault-kms/vaultkms.socket - type: Socket - name: vault-kms -``` - -The environment variable `VAULT_ADDR` matches the address of the `Vault` service and `--listen-addr` argument points to KMS plugin unix socket in the following example: - -```yaml -apiVersion: apps/v1 -kind: DaemonSet -metadata: - name: vault-kms-provider - namespace: kube-system - ... -spec: - template: - spec: - initContainers: - - name: vault-agent - image: vault - securityContext: - privileged: true - args: - - agent - - -config=/etc/vault/vault-agent-config.hcl - - -log-level=debug - env: - - name: VAULT_ADDR - value: http://vault-adress:8200 - ... - containers: - - name: vault-kms-provider - image: ghcr.io/ondat/trousseau:v1.1.3 - imagePullPolicy: Always - args: - - -v=5 - - --config-file-path=/opt/trousseau/config.yaml - - --listen-addr=unix:///opt/vault-kms/vaultkms.socket - - --zap-encoder=json - - --v=3 -``` - -For more information, refer to: -* [https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/) -* [https://github.com/ondat/trousseau/wiki/Trousseau-Deployment](https://github.com/ondat/trousseau/wiki/Trousseau-Deployment) - -### Disabling Encryption - -The first step of disabling encryption is to make the `identity` provider default for encryption. The enabling of `EncryptionConfiguration` should be similar to the following example: - -```yaml -apiVersion: apiserver.config.k8s.io/v1 -kind: EncryptionConfiguration -resources: - - resources: - - secrets - - configmaps - providers: - - identity: {} - - aesgcm: - keys: - - name: key1 - secret: c2VjcmV0IGlzIHNlY3VyZQ== - - name: key2 - secret: dGhpcyBpcyBwYXNzd29yZA== - - aescbc: - keys: - - name: key1 - secret: c2VjcmV0IGlzIHNlY3VyZQ== - - name: key2 - secret: dGhpcyBpcyBwYXNzd29yZA== - - secretbox: - keys: - - name: key1 - secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= -``` - -The next step is to replace all resources that were previously encrypted (e.g. `secrets`): - -```console -# kubectl get secrets --all-namespaces -o json | kubectl replace -f - -``` - -It is then possible to remove encryption settings from the `kubeadm-config` configmap and `kube-apiserver` manifest. - -### Maintenance and Operation Features - -* Since the `/etc/kubernetes/enc/enc.yaml` file has keys, access to the file must be restricted. For instance: -```console -# chmod 0700 /etc/kubernetes/enc/ -``` - -* The proper way for using encryption is to rotate the keys. The rotation procedure of the keys should take into consideration the fact that the `EncryptionConfiguration` file must be equal on each `control-plane` node. During the keys rotation procedure, some operation of getting the encrypted resources may be unsuccessful. -* The `kube-apiserver` has an `--encryption-provider-config-automatic-reload` option that allows applying a new `EncryptionConfiguration` without `kube-apiserver` reload. - -* ETCD restore procedures should take into consideration the keys rotation, otherwise some data may be unavailable due to keys that were used for encryption and is not available after restoration. The backup procedure may include an additional step that renews all encrypted data before the ETCD backup. This approach decreases the security level for data in ETCD backup, but it prevents any inconvenience in the future. Another option is not to delete the keys from `env.yml` even if they are not used for encryption/decryption anymore. -* External services that interact with ETCD may stop working due to encryption enabling. - ## Changing Cluster CIDR There might be a situation when you have to change the pod network used in a cluster. The default `podSubnet` (`10.128.0.0/14` for IPv4 and `fd02::/48` for IPv6) may be inappropriate for some reason. diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index ba3dc573e..f69616588 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -1,8 +1,8 @@ -# Disable anonymous authentication for `kube-apiserver` +## Disable anonymous authentication for `kube-apiserver` The `--anonymous-auth` option manages anonymous requests to the `kube-apiserver`. By default it enables anonymous requests. -## Prerequisites +### Prerequisites * Working Kubernetes cluster * The following RBAC resources: @@ -49,7 +49,7 @@ subjects: namespace: kube-system ``` -## Disabling procedure +### Disabling procedure 1. Add `anonymous-auth: "false"` into the `kubeadm-config` configmap e.g.: @@ -122,10 +122,229 @@ Where TOKEN is the result of the following command: kubectl -n kube-system get secret token-healthz -o jsonpath='{.data.token}' | base64 --decode ``` -## Limitations +### Limitations If the `--anonymous-auth` is set to `false` the upgrade and node addition procedures need some changes in workflow. The upgrade procedure needs enabling `anonymous-auth` before the `kubeadm upgrade` run. The node addition procedure affects if the control plane node is being added. After new control plane node has successfully added, the [disabling procedure](#disabling-procedure) should be performed on that node. Besides, disabled anonymous resuests to `kube-apiserver` need changes in monitoring system, if the resources like `healthz`, `readyz`, and `livez` are used in the system. + +## Data Encryption in Kubernetes + +The following section describes the Kubernetes cluster capabilities to store and manipulate encrypted data. + +### Enabling Encryption + +ETCD as a Kubernetes cluster storage can interact with encrypted data. The encryption/decryption procedures are the part of `kube-apiserver` functionality. + +An example of the `EncryptionConfiguration` file is as follows: + +```yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: EncryptionConfiguration +resources: + - resources: + - secrets + - configmaps + providers: + - aesgcm: + keys: + - name: key1 + secret: c2VjcmV0IGlzIHNlY3VyZQ== + - name: key2 + secret: dGhpcyBpcyBwYXNzd29yZA== + - aescbc: + keys: + - name: key1 + secret: c2VjcmV0IGlzIHNlY3VyZQ== + - name: key2 + secret: dGhpcyBpcyBwYXNzd29yZA== + - secretbox: + keys: + - name: key1 + secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= + - identity: {} +``` + +It should be created preliminarily and placed in the `/etc/kubernetes/enc/` directory. + +The next step is to enable the encryption settings in `kubeadm-config`: +```yaml +data: + ClusterConfiguration: | + apiServer: + ... + extraArgs: + ... + encryption-provider-config: /etc/kubernetes/enc/enc.yaml + extraVolumes: + ... + - hostPath: /etc/kubernetes/enc + mountPath: /etc/kubernetes/enc + name: enc + pathType: DirectoryOrCreate +``` + +There is an `--encryption-provider-config` option that points to the `EncryptionConfiguration` file location. The `kube-apiserver` should have the following parts in the manifest yaml: + +```yaml +... +spec: + containers: + - command: + - kube-apiserver + ... + - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml + ... + volumeMounts: + - name: enc + mountPath: /etc/kubernetes/enc + readonly: true + ... + volumes: + - name: enc + hostPath: + path: /etc/kubernetes/enc + type: DirectoryOrCreate +``` + +In the above case, the `secrets` and `configmaps` are encrypted on the first key of the `aesgcm` provider, but the previously encrypted `secrets` and `configmaps` are decrypted on any keys of any providers that are matched. This approach allows to change both encryption providers and keys during the operation. The keys should be random strings in base64 encoding. `identity` is the default provider that does not provide any encryption at all. +For more information, refer to [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/). + +### Integration with External KMS + +There is an encryption provider `kms` that allows using an external `Key Management Service` for the key storage, therefore the keys are not stored in the `EncryptionConfiguration` file, which is more secure. The `kms` provider needs to deploy a KMS plugin for further use. +The `Trousseau` KMS plugin is an example. It works through a unix socket, therefore `Trousseau` pods must be run on the same nodes as `kube-apiserver`. In case of using the KMS provider, the `EncryptionConfiguration` is as follows (`Vault` is a KMS): + +```yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: EncryptionConfiguration +resources: + - resources: + - secrets + - configmaps + providers: + - kms: + name: vaultprovider + endpoint: unix:///opt/vault-kms/vaultkms.socket + cachesize: 100 + timeout: 3s + - identity: {} +``` + +Also, unix socket must be available for `kube-apiserver`: + + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: kube-apiserver + namespace: kube-system +spec: + containers: + - command: + volumeMounts: + - mountPath: /opt/vault-kms/vaultkms.socket + name: vault-kms + ... + volumes: + - hostPath: + path: /opt/vault-kms/vaultkms.socket + type: Socket + name: vault-kms +``` + +The environment variable `VAULT_ADDR` matches the address of the `Vault` service and `--listen-addr` argument points to KMS plugin unix socket in the following example: + +```yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: vault-kms-provider + namespace: kube-system + ... +spec: + template: + spec: + initContainers: + - name: vault-agent + image: vault + securityContext: + privileged: true + args: + - agent + - -config=/etc/vault/vault-agent-config.hcl + - -log-level=debug + env: + - name: VAULT_ADDR + value: http://vault-adress:8200 + ... + containers: + - name: vault-kms-provider + image: ghcr.io/ondat/trousseau:v1.1.3 + imagePullPolicy: Always + args: + - -v=5 + - --config-file-path=/opt/trousseau/config.yaml + - --listen-addr=unix:///opt/vault-kms/vaultkms.socket + - --zap-encoder=json + - --v=3 +``` + +For more information, refer to: +* [https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/) +* [https://github.com/ondat/trousseau/wiki/Trousseau-Deployment](https://github.com/ondat/trousseau/wiki/Trousseau-Deployment) + +### Disabling Encryption + +The first step of disabling encryption is to make the `identity` provider default for encryption. The enabling of `EncryptionConfiguration` should be similar to the following example: + +```yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: EncryptionConfiguration +resources: + - resources: + - secrets + - configmaps + providers: + - identity: {} + - aesgcm: + keys: + - name: key1 + secret: c2VjcmV0IGlzIHNlY3VyZQ== + - name: key2 + secret: dGhpcyBpcyBwYXNzd29yZA== + - aescbc: + keys: + - name: key1 + secret: c2VjcmV0IGlzIHNlY3VyZQ== + - name: key2 + secret: dGhpcyBpcyBwYXNzd29yZA== + - secretbox: + keys: + - name: key1 + secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= +``` + +The next step is to replace all resources that were previously encrypted (e.g. `secrets`): + +```console +# kubectl get secrets --all-namespaces -o json | kubectl replace -f - +``` + +It is then possible to remove encryption settings from the `kubeadm-config` configmap and `kube-apiserver` manifest. + +### Maintenance and Operation Features + +* Since the `/etc/kubernetes/enc/enc.yaml` file has keys, access to the file must be restricted. For instance: +```console +# chmod 0700 /etc/kubernetes/enc/ +``` + +* The proper way for using encryption is to rotate the keys. The rotation procedure of the keys should take into consideration the fact that the `EncryptionConfiguration` file must be equal on each `control-plane` node. During the keys rotation procedure, some operation of getting the encrypted resources may be unsuccessful. +* The `kube-apiserver` has an `--encryption-provider-config-automatic-reload` option that allows applying a new `EncryptionConfiguration` without `kube-apiserver` reload. + +* ETCD restore procedures should take into consideration the keys rotation, otherwise some data may be unavailable due to keys that were used for encryption and is not available after restoration. The backup procedure may include an additional step that renews all encrypted data before the ETCD backup. This approach decreases the security level for data in ETCD backup, but it prevents any inconvenience in the future. Another option is not to delete the keys from `env.yml` even if they are not used for encryption/decryption anymore. +* External services that interact with ETCD may stop working due to encryption enabling. From a580c08b7b47cb2f7d761c8a5c35ac39b6adccfe Mon Sep 17 00:00:00 2001 From: Aleksandr Arefev <39635005+alexarefev@users.noreply.github.com> Date: Fri, 3 Nov 2023 09:33:47 +0300 Subject: [PATCH 04/12] kubelet crt --- documentation/Maintenance.md | 42 ----------------------------- documentation/Troubleshooting.md | 2 +- documentation/internal/Hardening.md | 42 +++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 43 deletions(-) diff --git a/documentation/Maintenance.md b/documentation/Maintenance.md index 1391cc6bb..33bece38d 100644 --- a/documentation/Maintenance.md +++ b/documentation/Maintenance.md @@ -1617,48 +1617,6 @@ data: 11. Check that everything works properly and remove the old ippool if necessary. -## Kubelet Server Certificate Approval - -The `kubelet` server certificate is self-signed by default, and is usually stored in the `/var/lib/kubelet/pki/kubelet.crt` file. To avoid using the self-signed `kubelet` server certificate, alter the `cluster.yaml` file in the following way: - -```yaml -... -services: - kubeadm_kubelet: - serverTLSBootstrap: true - rotateCertificates: true - kubeadm: - apiServer: - extraArgs: - kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt -... -``` - -These settings enforce `kubelet` on each node of the cluster to request certificate approval (for `kubelet` server part) from the default Kubernetes CA and rotate certificate in the future. The `kube-apiserver` machinery does not approve certificate requests for `kubelet` automatically. They might be approved manually by the following commans. Get the list of certificate requests: - -``` -# kubectl get csr -NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION -csr-2z6rv 12m kubernetes.io/kubelet-serving system:node:nodename-1 Pending -csr-424qg 89m kubernetes.io/kubelet-serving system:node:nodename-2 Pending -``` - -Approve the particular request: - -``` -kubectl certificate approve csr-424qg -``` - -These commands might be automated in several ways. - -### Auto Approval CronJob - -Basically, `CronJob` runs the approval command above for every CSR according to some schedule. - -### Auto Approval Service - -It is possible to install the kubelet-csr-approver service. For more information, refer to [[kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver)](https://github.com/postfinance/kubelet-csr-approver). This service approves CSR automatically when a CSR is created according to several settings. It is better to restrict nodes' IP addresses (`providerIpPrefixes` option) and FQDN templates (providerRegex). For more information, refer to the official documentation. - # Common Practice You should not run any containers on worker nodes that are not managed by `kubelet` so as not to break the `kube-scheduler` precision. diff --git a/documentation/Troubleshooting.md b/documentation/Troubleshooting.md index d17d886c6..cb1db5e91 100644 --- a/documentation/Troubleshooting.md +++ b/documentation/Troubleshooting.md @@ -1216,4 +1216,4 @@ Error from server: error dialing backend: remote error: tls: internal error **Root cause**: The `kubelet` server certificate is not approved, whereas the cluster has been configured not to use self-signed certificates for the `kubelet` server. -**Solution**: Perform CSR approval steps from the maintenance guide. Refer to the [Kubelet Server Certificate Approval](https://github.com/Netcracker/KubeMarine/blob/main/documentation/Maintenance.md#kubelet-server-certificate-approval) section for details. +**Solution**: Perform CSR approval steps from the maintenance guide. Refer to the [Kubelet Server Certificate Approval](https://github.com/Netcracker/KubeMarine/blob/main/documentation/internal/Hardening.md#kubelet-server-certificate-approval) section for details. diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index f69616588..dadf9e4ee 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -348,3 +348,45 @@ It is then possible to remove encryption settings from the `kubeadm-config` conf * ETCD restore procedures should take into consideration the keys rotation, otherwise some data may be unavailable due to keys that were used for encryption and is not available after restoration. The backup procedure may include an additional step that renews all encrypted data before the ETCD backup. This approach decreases the security level for data in ETCD backup, but it prevents any inconvenience in the future. Another option is not to delete the keys from `env.yml` even if they are not used for encryption/decryption anymore. * External services that interact with ETCD may stop working due to encryption enabling. + +## Kubelet Server Certificate Approval + +The `kubelet` server certificate is self-signed by default, and is usually stored in the `/var/lib/kubelet/pki/kubelet.crt` file. To avoid using the self-signed `kubelet` server certificate, alter the `cluster.yaml` file in the following way: + +```yaml +... +services: + kubeadm_kubelet: + serverTLSBootstrap: true + rotateCertificates: true + kubeadm: + apiServer: + extraArgs: + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt +... +``` + +These settings enforce `kubelet` on each node of the cluster to request certificate approval (for `kubelet` server part) from the default Kubernetes CA and rotate certificate in the future. The `kube-apiserver` machinery does not approve certificate requests for `kubelet` automatically. They might be approved manually by the following commans. Get the list of certificate requests: + +``` +# kubectl get csr +NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION +csr-2z6rv 12m kubernetes.io/kubelet-serving system:node:nodename-1 Pending +csr-424qg 89m kubernetes.io/kubelet-serving system:node:nodename-2 Pending +``` + +Approve the particular request: + +``` +kubectl certificate approve csr-424qg +``` + +These commands might be automated in several ways. + +### Auto Approval CronJob + +Basically, `CronJob` runs the approval command above for every CSR according to some schedule. + +### Auto Approval Service + +It is possible to install the kubelet-csr-approver service. For more information, refer to [[kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver)](https://github.com/postfinance/kubelet-csr-approver). This service approves CSR automatically when a CSR is created according to several settings. It is better to restrict nodes' IP addresses (`providerIpPrefixes` option) and FQDN templates (providerRegex). For more information, refer to the official documentation. From c4ec56bcc449dd8e591e764477df3d2e7932304c Mon Sep 17 00:00:00 2001 From: Aleksandr Arefev <39635005+alexarefev@users.noreply.github.com> Date: Fri, 3 Nov 2023 15:32:18 +0300 Subject: [PATCH 05/12] doc update --- documentation/internal/Hardening.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index dadf9e4ee..5940729ff 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -1,5 +1,16 @@ +## The purpose + +The current document discrbes manual steps or procedures that are not covered by `KubeMarine` code itself, but should be implemented to get production-ready Kubernetes cluster. + +`kube-bench` is a well-known open-source tool to check the Kubernetes cluster against the `CIS Kubernetes Benchmark`. The report is divided on several parts. Each check has its own unique number. The items could be identified by that number. + +Useful links: +[Kube-bench](https://github.com/aquasecurity/kube-bench) + ## Disable anonymous authentication for `kube-apiserver` +**1.2.1** + The `--anonymous-auth` option manages anonymous requests to the `kube-apiserver`. By default it enables anonymous requests. ### Prerequisites @@ -132,6 +143,9 @@ Besides, disabled anonymous resuests to `kube-apiserver` need changes in monitor ## Data Encryption in Kubernetes +**1.2.29** +**1.2.30** + The following section describes the Kubernetes cluster capabilities to store and manipulate encrypted data. ### Enabling Encryption @@ -351,6 +365,8 @@ It is then possible to remove encryption settings from the `kubeadm-config` conf ## Kubelet Server Certificate Approval +**1.2.5** + The `kubelet` server certificate is self-signed by default, and is usually stored in the `/var/lib/kubelet/pki/kubelet.crt` file. To avoid using the self-signed `kubelet` server certificate, alter the `cluster.yaml` file in the following way: ```yaml From 1ec83063bb32c2a465332a1a1a44ab0c64bf9342 Mon Sep 17 00:00:00 2001 From: Aleksandr Arefev <39635005+alexarefev@users.noreply.github.com> Date: Fri, 3 Nov 2023 15:57:09 +0300 Subject: [PATCH 06/12] doc update --- documentation/internal/Hardening.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index 5940729ff..6d2c0d916 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -9,7 +9,8 @@ Useful links: ## Disable anonymous authentication for `kube-apiserver` -**1.2.1** +**Kube-bench identifier**: +* 1.2.1 The `--anonymous-auth` option manages anonymous requests to the `kube-apiserver`. By default it enables anonymous requests. @@ -143,8 +144,9 @@ Besides, disabled anonymous resuests to `kube-apiserver` need changes in monitor ## Data Encryption in Kubernetes -**1.2.29** -**1.2.30** +**Kube-bench identifier**: +* 1.2.29 +* 1.2.30 The following section describes the Kubernetes cluster capabilities to store and manipulate encrypted data. @@ -365,7 +367,8 @@ It is then possible to remove encryption settings from the `kubeadm-config` conf ## Kubelet Server Certificate Approval -**1.2.5** +**Kube-bench identifier**: +* 1.2.5 The `kubelet` server certificate is self-signed by default, and is usually stored in the `/var/lib/kubelet/pki/kubelet.crt` file. To avoid using the self-signed `kubelet` server certificate, alter the `cluster.yaml` file in the following way: From 707542ff7c5df51a23bc378f440b2b4e7b5c4ccd Mon Sep 17 00:00:00 2001 From: sergey kryazhev Date: Fri, 10 Nov 2023 18:10:30 +0300 Subject: [PATCH 07/12] some fixes --- documentation/Maintenance.md | 8 ++++++++ documentation/internal/Hardening.md | 27 ++++++++++++++++++++------- 2 files changed, 28 insertions(+), 7 deletions(-) diff --git a/documentation/Maintenance.md b/documentation/Maintenance.md index 33bece38d..f7971f916 100644 --- a/documentation/Maintenance.md +++ b/documentation/Maintenance.md @@ -29,6 +29,8 @@ This section describes the features and steps for performing maintenance procedu - [Changing Cluster CIDR](#changing-cluster-cidr) - [Kubelet Server Certificate Approval](#kubelet-server-certificate-approval) - [Common Practice](#common-practice) + - [Security hardening guide](#security-hardening-guide) + - [Worker nodes should be managed by kubelet](#worker-nodes-should-be-managed-by-kubelet) # Prerequisites @@ -1619,4 +1621,10 @@ data: # Common Practice +## Security hardening guide + +[Security hardening guide](./internal/Hardening.md) + +## Worker nodes should be managed by kubelet + You should not run any containers on worker nodes that are not managed by `kubelet` so as not to break the `kube-scheduler` precision. diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index 6d2c0d916..7635a4374 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -1,3 +1,11 @@ +# Security Hardening Guide + +- [The purpose](#the-purpose) +- [Disable anonymous authentication for kube-apiserver](#disable-anonymous-authentication-for-kube-apiserver) +- [Data Encryption in Kubernetes](#data-encryption-in-kubernetes) +- [Kubelet Server Certificate Approval](#kubelet-server-certificate-approval) + + ## The purpose The current document discrbes manual steps or procedures that are not covered by `KubeMarine` code itself, but should be implemented to get production-ready Kubernetes cluster. @@ -5,11 +13,12 @@ The current document discrbes manual steps or procedures that are not covered by `kube-bench` is a well-known open-source tool to check the Kubernetes cluster against the `CIS Kubernetes Benchmark`. The report is divided on several parts. Each check has its own unique number. The items could be identified by that number. Useful links: -[Kube-bench](https://github.com/aquasecurity/kube-bench) +[kube-bench](https://github.com/aquasecurity/kube-bench) ## Disable anonymous authentication for `kube-apiserver` **Kube-bench identifier**: + * 1.2.1 The `--anonymous-auth` option manages anonymous requests to the `kube-apiserver`. By default it enables anonymous requests. @@ -59,7 +68,7 @@ subjects: - kind: ServiceAccount name: healthz namespace: kube-system -``` +``` ### Disabling procedure @@ -130,7 +139,7 @@ spec: Where TOKEN is the result of the following command: -``` +```console kubectl -n kube-system get secret token-healthz -o jsonpath='{.data.token}' | base64 --decode ``` @@ -145,6 +154,7 @@ Besides, disabled anonymous resuests to `kube-apiserver` need changes in monitor ## Data Encryption in Kubernetes **Kube-bench identifier**: + * 1.2.29 * 1.2.30 @@ -185,7 +195,8 @@ resources: It should be created preliminarily and placed in the `/etc/kubernetes/enc/` directory. -The next step is to enable the encryption settings in `kubeadm-config`: +The next step is to enable the encryption settings in `kubeadm-config`: + ```yaml data: ClusterConfiguration: | @@ -251,7 +262,6 @@ resources: Also, unix socket must be available for `kube-apiserver`: - ```yaml apiVersion: v1 kind: Pod @@ -310,6 +320,7 @@ spec: ``` For more information, refer to: + * [https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/) * [https://github.com/ondat/trousseau/wiki/Trousseau-Deployment](https://github.com/ondat/trousseau/wiki/Trousseau-Deployment) @@ -355,6 +366,7 @@ It is then possible to remove encryption settings from the `kubeadm-config` conf ### Maintenance and Operation Features * Since the `/etc/kubernetes/enc/enc.yaml` file has keys, access to the file must be restricted. For instance: + ```console # chmod 0700 /etc/kubernetes/enc/ ``` @@ -368,6 +380,7 @@ It is then possible to remove encryption settings from the `kubeadm-config` conf ## Kubelet Server Certificate Approval **Kube-bench identifier**: + * 1.2.5 The `kubelet` server certificate is self-signed by default, and is usually stored in the `/var/lib/kubelet/pki/kubelet.crt` file. To avoid using the self-signed `kubelet` server certificate, alter the `cluster.yaml` file in the following way: @@ -387,7 +400,7 @@ services: These settings enforce `kubelet` on each node of the cluster to request certificate approval (for `kubelet` server part) from the default Kubernetes CA and rotate certificate in the future. The `kube-apiserver` machinery does not approve certificate requests for `kubelet` automatically. They might be approved manually by the following commans. Get the list of certificate requests: -``` +```console # kubectl get csr NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION csr-2z6rv 12m kubernetes.io/kubelet-serving system:node:nodename-1 Pending @@ -396,7 +409,7 @@ csr-424qg 89m kubernetes.io/kubelet-serving system:node:nodename-2 Approve the particular request: -``` +```console kubectl certificate approve csr-424qg ``` From e2c1fd4476d1d31baa0a7bf5579f14ed938f8a56 Mon Sep 17 00:00:00 2001 From: pranavcracker <113030526+pranavcracker@users.noreply.github.com> Date: Mon, 13 Nov 2023 17:46:10 +0530 Subject: [PATCH 08/12] Added one more scenario --- documentation/internal/Hardening.md | 55 +++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index 7635a4374..62e982e2d 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -4,6 +4,7 @@ - [Disable anonymous authentication for kube-apiserver](#disable-anonymous-authentication-for-kube-apiserver) - [Data Encryption in Kubernetes](#data-encryption-in-kubernetes) - [Kubelet Server Certificate Approval](#kubelet-server-certificate-approval) +- [Disabling auto-mounting of tokens for service accounts](#Disabling auto-mounting of tokens for service accounts) ## The purpose @@ -239,6 +240,8 @@ spec: In the above case, the `secrets` and `configmaps` are encrypted on the first key of the `aesgcm` provider, but the previously encrypted `secrets` and `configmaps` are decrypted on any keys of any providers that are matched. This approach allows to change both encryption providers and keys during the operation. The keys should be random strings in base64 encoding. `identity` is the default provider that does not provide any encryption at all. For more information, refer to [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/). +As per CIS bechmark (kube-bench checks), `aesgcm` provider for encrypption is not recognized as appropriate provider. To fulfil this requirement, we have to configure `aescbc`, `secretxbox` or `kms` as encryption provider. + ### Integration with External KMS There is an encryption provider `kms` that allows using an external `Key Management Service` for the key storage, therefore the keys are not stored in the `EncryptionConfiguration` file, which is more secure. The `kms` provider needs to deploy a KMS plugin for further use. @@ -422,3 +425,55 @@ Basically, `CronJob` runs the approval command above for every CSR according to ### Auto Approval Service It is possible to install the kubelet-csr-approver service. For more information, refer to [[kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver)](https://github.com/postfinance/kubelet-csr-approver). This service approves CSR automatically when a CSR is created according to several settings. It is better to restrict nodes' IP addresses (`providerIpPrefixes` option) and FQDN templates (providerRegex). For more information, refer to the official documentation. + +## Disabling auto-mounting of tokens for service accounts + +To disable the auto-mounting of service account tokens, we need to create secrets associated with perticular service account and mount that secret as a volume to the pod's specification wherver necessary. + +To acchieve it, we have to follow below proedure - + +### Disable auto-mounting +To disbale the auto-mounting of token add `automountServiceAccountToken: false` falg to the service account properties as shwon below - + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: ingress-nginx + namespace: ingress-nginx +automountServiceAccountToken: false +... +``` + +### Create secret +Create a new kuebernetes secret of type `kubernetes.io/service-account-token` as below - +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: ingress-nginx-token + namespace: ingress-nginx + annotations: + kubernetes.io/service-account.name: ingress-nginx +type: kubernetes.io/service-account-token +``` + +### Mount the token thorugh secrets +Edit POD specification and mount the secret as volume to the pod as below - + +```yaml +... +volumeMounts: + - name: ingress-nginx-token + mountPath: /var/run/secrets/kubernetes.io/serviceaccount + +... + +volumes: + - name: service-account-token + secret: + secretName: calico-node-token +... +``` +After this restart the pod to reflect the changes and verify that the secret is mounted to the pod at the specified mount point. + From b308bd79713df9f9f68fe4c186acb7885faa401f Mon Sep 17 00:00:00 2001 From: pranavcracker <113030526+pranavcracker@users.noreply.github.com> Date: Wed, 15 Nov 2023 11:10:21 +0530 Subject: [PATCH 09/12] corrected the name for volume --- documentation/internal/Hardening.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index 62e982e2d..46e75918b 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -470,9 +470,9 @@ volumeMounts: ... volumes: - - name: service-account-token + - name: ingress-nginx-token secret: - secretName: calico-node-token + secretName: ingress-nginx-token ... ``` After this restart the pod to reflect the changes and verify that the secret is mounted to the pod at the specified mount point. From 42f4c432bd16bcd7a3a095231ce0275491a15f11 Mon Sep 17 00:00:00 2001 From: Shoaib Mohammed <94443646+shmo1218@users.noreply.github.com> Date: Fri, 17 Nov 2023 14:27:52 +0530 Subject: [PATCH 10/12] Update Maintenance.md --- documentation/Maintenance.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/documentation/Maintenance.md b/documentation/Maintenance.md index f7971f916..6197b73c9 100644 --- a/documentation/Maintenance.md +++ b/documentation/Maintenance.md @@ -29,8 +29,8 @@ This section describes the features and steps for performing maintenance procedu - [Changing Cluster CIDR](#changing-cluster-cidr) - [Kubelet Server Certificate Approval](#kubelet-server-certificate-approval) - [Common Practice](#common-practice) - - [Security hardening guide](#security-hardening-guide) - - [Worker nodes should be managed by kubelet](#worker-nodes-should-be-managed-by-kubelet) + - [Security Hardening Guide](#security-hardening-guide) + - [Worker Nodes Should be Managed by Kubelet](#worker-nodes-should-be-managed-by-kubelet) # Prerequisites @@ -1621,10 +1621,10 @@ data: # Common Practice -## Security hardening guide +## Security Hardening Guide -[Security hardening guide](./internal/Hardening.md) +For more information, refer to [Security Hardening Guide](./internal/Hardening.md). -## Worker nodes should be managed by kubelet +## Worker Nodes Should be Managed by Kubelet You should not run any containers on worker nodes that are not managed by `kubelet` so as not to break the `kube-scheduler` precision. From c91b863d7473245476d95fd2557f2bf3ac1cf156 Mon Sep 17 00:00:00 2001 From: Shoaib Mohammed <94443646+shmo1218@users.noreply.github.com> Date: Fri, 17 Nov 2023 14:58:20 +0530 Subject: [PATCH 11/12] Update Hardening.md --- documentation/internal/Hardening.md | 94 +++++++++++++++-------------- 1 file changed, 48 insertions(+), 46 deletions(-) diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index 46e75918b..2d223bc3d 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -1,32 +1,32 @@ # Security Hardening Guide -- [The purpose](#the-purpose) -- [Disable anonymous authentication for kube-apiserver](#disable-anonymous-authentication-for-kube-apiserver) +- [Overview](#overview) +- [Disable Anonymous Authentication for kube-apiserver](#disable-anonymous-authentication-for-kube-apiserver) - [Data Encryption in Kubernetes](#data-encryption-in-kubernetes) - [Kubelet Server Certificate Approval](#kubelet-server-certificate-approval) -- [Disabling auto-mounting of tokens for service accounts](#Disabling auto-mounting of tokens for service accounts) +- [Disabling Auto-Mounting of Tokens for Service Accounts](#disabling-auto-mounting-of-tokens-for-service-accounts) -## The purpose +## Overview -The current document discrbes manual steps or procedures that are not covered by `KubeMarine` code itself, but should be implemented to get production-ready Kubernetes cluster. +The current document describes the manual steps or procedures that are not covered by the `KubeMarine` code itself, but should be implemented to get a production-ready Kubernetes cluster. `kube-bench` is a well-known open-source tool to check the Kubernetes cluster against the `CIS Kubernetes Benchmark`. The report is divided on several parts. Each check has its own unique number. The items could be identified by that number. Useful links: [kube-bench](https://github.com/aquasecurity/kube-bench) -## Disable anonymous authentication for `kube-apiserver` +## Disable Anonymous Authentication for `kube-apiserver` -**Kube-bench identifier**: +**Kube-bench Identifier**: * 1.2.1 -The `--anonymous-auth` option manages anonymous requests to the `kube-apiserver`. By default it enables anonymous requests. +The `--anonymous-auth` option manages anonymous requests to the `kube-apiserver`. By default, it enables anonymous requests. ### Prerequisites -* Working Kubernetes cluster +* A working Kubernetes cluster. * The following RBAC resources: ```yaml @@ -71,9 +71,9 @@ subjects: namespace: kube-system ``` -### Disabling procedure +### Disabling Procedure -1. Add `anonymous-auth: "false"` into the `kubeadm-config` configmap e.g.: +1. Add `anonymous-auth: "false"` into the `kubeadm-config` configmap. For example: ```yaml apiVersion: v1 @@ -138,7 +138,7 @@ spec: ... ``` -Where TOKEN is the result of the following command: +Where, TOKEN is the result of the following command: ```console kubectl -n kube-system get secret token-healthz -o jsonpath='{.data.token}' | base64 --decode @@ -146,24 +146,24 @@ kubectl -n kube-system get secret token-healthz -o jsonpath='{.data.token}' | ba ### Limitations -If the `--anonymous-auth` is set to `false` the upgrade and node addition procedures need some changes in workflow. The upgrade procedure needs enabling `anonymous-auth` before the `kubeadm upgrade` run. +If the `--anonymous-auth` is set to "false", the upgrade and node addition procedures need some changes in the workflow. The upgrade procedure needs enabling `anonymous-auth` before the `kubeadm upgrade` run. -The node addition procedure affects if the control plane node is being added. After new control plane node has successfully added, the [disabling procedure](#disabling-procedure) should be performed on that node. +The node addition procedure is affected if the control plane node is being added. After the new control plane node is successfully added, the [Disabling Procedure](#disabling-procedure) should be performed on that node. -Besides, disabled anonymous resuests to `kube-apiserver` need changes in monitoring system, if the resources like `healthz`, `readyz`, and `livez` are used in the system. +Besides, disabled anonymous requests to `kube-apiserver` need changes in the monitoring system, if the resources like `healthz`, `readyz`, and `livez` are used in the system. ## Data Encryption in Kubernetes -**Kube-bench identifier**: +**Kube-bench Identifier**: * 1.2.29 * 1.2.30 -The following section describes the Kubernetes cluster capabilities to store and manipulate encrypted data. +The following section describes the Kubernetes cluster capabilities to store and manipulate the encrypted data. ### Enabling Encryption -ETCD as a Kubernetes cluster storage can interact with encrypted data. The encryption/decryption procedures are the part of `kube-apiserver` functionality. +ETCD as a Kubernetes cluster storage can interact with the encrypted data. The encryption/decryption procedures are the part of the `kube-apiserver` functionality. An example of the `EncryptionConfiguration` file is as follows: @@ -237,15 +237,15 @@ spec: type: DirectoryOrCreate ``` -In the above case, the `secrets` and `configmaps` are encrypted on the first key of the `aesgcm` provider, but the previously encrypted `secrets` and `configmaps` are decrypted on any keys of any providers that are matched. This approach allows to change both encryption providers and keys during the operation. The keys should be random strings in base64 encoding. `identity` is the default provider that does not provide any encryption at all. +In the above case, `secrets` and `configmaps` are encrypted on the first key of the `aesgcm` provider, but the previously encrypted `secrets` and `configmaps` are decrypted on any keys of any providers that are matched. This approach allows to change both encryption providers and keys during the operation. The keys should be random strings in base64 encoding. `identity` is the default provider that does not provide any encryption at all. For more information, refer to [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/). -As per CIS bechmark (kube-bench checks), `aesgcm` provider for encrypption is not recognized as appropriate provider. To fulfil this requirement, we have to configure `aescbc`, `secretxbox` or `kms` as encryption provider. +As per the CIS benchmark (kube-bench checks), the `aesgcm` provider for encryption is not recognized as an appropriate provider. To fulfil this requirement, we have to configure `aescbc`, `secretxbox`, or `kms` as an encryption provider. ### Integration with External KMS There is an encryption provider `kms` that allows using an external `Key Management Service` for the key storage, therefore the keys are not stored in the `EncryptionConfiguration` file, which is more secure. The `kms` provider needs to deploy a KMS plugin for further use. -The `Trousseau` KMS plugin is an example. It works through a unix socket, therefore `Trousseau` pods must be run on the same nodes as `kube-apiserver`. In case of using the KMS provider, the `EncryptionConfiguration` is as follows (`Vault` is a KMS): +The `Trousseau` KMS plugin is an example. It works through a unix socket, therefore `Trousseau` pods must be run on the same nodes as `kube-apiserver`. In case of using a KMS provider, the `EncryptionConfiguration` is as follows (`Vault` is a KMS): ```yaml apiVersion: apiserver.config.k8s.io/v1 @@ -263,7 +263,7 @@ resources: - identity: {} ``` -Also, unix socket must be available for `kube-apiserver`: +Also, the unix socket must be available for `kube-apiserver`: ```yaml apiVersion: v1 @@ -285,7 +285,7 @@ spec: name: vault-kms ``` -The environment variable `VAULT_ADDR` matches the address of the `Vault` service and `--listen-addr` argument points to KMS plugin unix socket in the following example: +The environment variable `VAULT_ADDR` matches the address of the `Vault` service and `--listen-addr` argument points to the KMS plugin unix socket in the following example: ```yaml apiVersion: apps/v1 @@ -329,7 +329,7 @@ For more information, refer to: ### Disabling Encryption -The first step of disabling encryption is to make the `identity` provider default for encryption. The enabling of `EncryptionConfiguration` should be similar to the following example: +The first step for disabling encryption is to make the `identity` provider default for encryption. The enabling of `EncryptionConfiguration` should be similar to the following example: ```yaml apiVersion: apiserver.config.k8s.io/v1 @@ -358,13 +358,13 @@ resources: secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= ``` -The next step is to replace all resources that were previously encrypted (e.g. `secrets`): +The next step is to replace all resources that were previously encrypted (for example, `secrets`): ```console # kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` -It is then possible to remove encryption settings from the `kubeadm-config` configmap and `kube-apiserver` manifest. +It is then possible to remove the encryption settings from the `kubeadm-config` configmap and `kube-apiserver` manifest. ### Maintenance and Operation Features @@ -374,15 +374,14 @@ It is then possible to remove encryption settings from the `kubeadm-config` conf # chmod 0700 /etc/kubernetes/enc/ ``` -* The proper way for using encryption is to rotate the keys. The rotation procedure of the keys should take into consideration the fact that the `EncryptionConfiguration` file must be equal on each `control-plane` node. During the keys rotation procedure, some operation of getting the encrypted resources may be unsuccessful. -* The `kube-apiserver` has an `--encryption-provider-config-automatic-reload` option that allows applying a new `EncryptionConfiguration` without `kube-apiserver` reload. - -* ETCD restore procedures should take into consideration the keys rotation, otherwise some data may be unavailable due to keys that were used for encryption and is not available after restoration. The backup procedure may include an additional step that renews all encrypted data before the ETCD backup. This approach decreases the security level for data in ETCD backup, but it prevents any inconvenience in the future. Another option is not to delete the keys from `env.yml` even if they are not used for encryption/decryption anymore. +* The proper way for using encryption is to rotate the keys. The rotation procedure of the keys should take into consideration the fact that the `EncryptionConfiguration` file must be equal on each `control-plane` node. During the keys' rotation procedure, some operation of getting the encrypted resources may be unsuccessful. +* The `kube-apiserver` has an `--encryption-provider-config-automatic-reload` option that allows to apply a new `EncryptionConfiguration` without `kube-apiserver` reload. +* ETCD restore procedures should take into consideration the keys' rotation, otherwise some data may be unavailable due to keys that were used for the encryption and is not available after restoration. The backup procedure may include an additional step that renews all encrypted data before the ETCD backup. This approach decreases the security level for the data in ETCD backup, but it prevents any inconvenience in the future. Another option is not to delete the keys from `env.yml` even if they are not used for encryption/decryption anymore. * External services that interact with ETCD may stop working due to encryption enabling. ## Kubelet Server Certificate Approval -**Kube-bench identifier**: +**Kube-bench Identifier**: * 1.2.5 @@ -401,7 +400,7 @@ services: ... ``` -These settings enforce `kubelet` on each node of the cluster to request certificate approval (for `kubelet` server part) from the default Kubernetes CA and rotate certificate in the future. The `kube-apiserver` machinery does not approve certificate requests for `kubelet` automatically. They might be approved manually by the following commans. Get the list of certificate requests: +These settings enforce `kubelet` on each node of the cluster to request certificate approval (for `kubelet` server part) from the default Kubernetes CA and rotate certificate in the future. The `kube-apiserver` machinery does not approve certificate requests for `kubelet` automatically. They can be approved manually by the following commands. Use the following command to get the list of certificate requests: ```console # kubectl get csr @@ -410,7 +409,7 @@ csr-2z6rv 12m kubernetes.io/kubelet-serving system:node:nodename-1 csr-424qg 89m kubernetes.io/kubelet-serving system:node:nodename-2 Pending ``` -Approve the particular request: +Use the following command to approve a particular request: ```console kubectl certificate approve csr-424qg @@ -420,20 +419,21 @@ These commands might be automated in several ways. ### Auto Approval CronJob -Basically, `CronJob` runs the approval command above for every CSR according to some schedule. +Generally, `CronJob` runs the approval command above for every CSR according to some schedule. ### Auto Approval Service -It is possible to install the kubelet-csr-approver service. For more information, refer to [[kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver)](https://github.com/postfinance/kubelet-csr-approver). This service approves CSR automatically when a CSR is created according to several settings. It is better to restrict nodes' IP addresses (`providerIpPrefixes` option) and FQDN templates (providerRegex). For more information, refer to the official documentation. +It is possible to install the kubelet-csr-approver service. For more information, refer to [kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver). This service approves the CSR automatically when a CSR is created according to several settings. It is better to restrict nodes' IP addresses (`providerIpPrefixes` option) and FQDN templates (providerRegex). For more information, refer to the official documentation. + +## Disabling Auto-Mounting of Tokens for Service Accounts -## Disabling auto-mounting of tokens for service accounts +To disable the auto-mounting of service account tokens, create secrets associated with a particular service account and mount that secret as a volume to the pod's specification wherever necessary. -To disable the auto-mounting of service account tokens, we need to create secrets associated with perticular service account and mount that secret as a volume to the pod's specification wherver necessary. +To achieve this, implement the following procedure. -To acchieve it, we have to follow below proedure - +### Disable Auto-Mounting -### Disable auto-mounting -To disbale the auto-mounting of token add `automountServiceAccountToken: false` falg to the service account properties as shwon below - +To disable auto-mounting of a token, add `automountServiceAccountToken: false` flag to the service account properties as shown below. ```yaml apiVersion: v1 @@ -445,8 +445,10 @@ automountServiceAccountToken: false ... ``` -### Create secret -Create a new kuebernetes secret of type `kubernetes.io/service-account-token` as below - +### Create Secret + +Create a new Kubernetes secret of type `kubernetes.io/service-account-token` as follows. + ```yaml apiVersion: v1 kind: Secret @@ -458,8 +460,9 @@ metadata: type: kubernetes.io/service-account-token ``` -### Mount the token thorugh secrets -Edit POD specification and mount the secret as volume to the pod as below - +### Mount the Token Through Secrets + +Edit the POD specification and mount the secret as a volume to the pod as follows. ```yaml ... @@ -475,5 +478,4 @@ volumes: secretName: ingress-nginx-token ... ``` -After this restart the pod to reflect the changes and verify that the secret is mounted to the pod at the specified mount point. - +After this, restart the pod to reflect the changes and verify that the secret is mounted to the pod at the specified mount point. From 9dfa05d524c6507760a891df85f42af891cc3b8e Mon Sep 17 00:00:00 2001 From: pranavcracker <113030526+pranavcracker@users.noreply.github.com> Date: Mon, 20 Nov 2023 15:33:17 +0530 Subject: [PATCH 12/12] Added Kube-bench Identifier --- documentation/internal/Hardening.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/documentation/internal/Hardening.md b/documentation/internal/Hardening.md index 2d223bc3d..107b6001a 100644 --- a/documentation/internal/Hardening.md +++ b/documentation/internal/Hardening.md @@ -427,6 +427,13 @@ It is possible to install the kubelet-csr-approver service. For more information ## Disabling Auto-Mounting of Tokens for Service Accounts +**Kube-bench Identifier**: + +* 5.1.5 + +Create explicit service accounts wherever a Kubernetes workload requires specific access to the Kubernetes API server. +Modify the configuration of each default service account to include this value `automountServiceAccountToken: false` + To disable the auto-mounting of service account tokens, create secrets associated with a particular service account and mount that secret as a volume to the pod's specification wherever necessary. To achieve this, implement the following procedure.