[CPREQ-7974] Kubernetes hardening

documentation/Maintenance.md

@@ -29,6 +29,8 @@ This section describes the features and steps for performing maintenance procedu
     - [Changing Cluster CIDR](#changing-cluster-cidr)
     - [Kubelet Server Certificate Approval](#kubelet-server-certificate-approval)
 - [Common Practice](#common-practice)
+  - [Security hardening guide](#security-hardening-guide)
+  - [Worker nodes should be managed by kubelet](#worker-nodes-should-be-managed-by-kubelet)
 
 # Prerequisites
 
@@ -1510,225 +1512,6 @@ plugins:
 
 ```
 
-## Data Encryption in Kubernetes
-
-The following section describes the Kubernetes cluster capabilities to store and manipulate encrypted data.
-
-### Enabling Encryption
-
-ETCD as a Kubernetes cluster storage can interact with encrypted data. The encryption/decryption procedures are the part of `kube-apiserver` functionality.
-
-An example of the `EncryptionConfiguration` file is as follows:
-
-```yaml
-apiVersion: apiserver.config.k8s.io/v1
-kind: EncryptionConfiguration
-resources:
-  - resources:
-      - secrets
-      - configmaps
-    providers:
-      - aesgcm:
-          keys:
-            - name: key1
-              secret: c2VjcmV0IGlzIHNlY3VyZQ==
-            - name: key2
-              secret: dGhpcyBpcyBwYXNzd29yZA==
-      - aescbc:
-          keys:
-            - name: key1
-              secret: c2VjcmV0IGlzIHNlY3VyZQ==
-            - name: key2
-              secret: dGhpcyBpcyBwYXNzd29yZA==
-      - secretbox:
-          keys:
-            - name: key1
-              secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
-      - identity: {}
-```
-
-It should be created preliminarily and placed in the `/etc/kubernetes/enc/` directory.
-
-The next step is to enable the encryption settings in `kubeadm-config`: 
-```yaml
-data:
-  ClusterConfiguration: |
-    apiServer:
-      ...
-      extraArgs:
-        ...
-        encryption-provider-config: /etc/kubernetes/enc/enc.yaml
-      extraVolumes:
-      ...
-      - hostPath: /etc/kubernetes/enc
-        mountPath: /etc/kubernetes/enc
-        name: enc
-        pathType: DirectoryOrCreate
-```
-
-There is an `--encryption-provider-config` option that points to the `EncryptionConfiguration` file location. The `kube-apiserver` should have the following parts in the manifest yaml:
-
-```yaml
-...
-spec:
-  containers:
-  - command:
-    - kube-apiserver
-     ...
-    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
-      ...
-    volumeMounts:
-    - name: enc
-      mountPath: /etc/kubernetes/enc
-      readonly: true
-       ...
-  volumes:
-  - name: enc
-    hostPath:
-      path: /etc/kubernetes/enc
-      type: DirectoryOrCreate
-```
-
-In the above case, the `secrets` and `configmaps` are encrypted on the first key of the `aesgcm` provider, but the previously encrypted `secrets` and `configmaps` are decrypted on any keys of any providers that are matched. This approach allows to change both encryption providers and keys during the operation. The keys should be random strings in base64 encoding. `identity` is the default provider that does not provide any encryption at all.
-For more information, refer to [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/).
-
-### Integration with External KMS
-
-There is an encryption provider `kms` that allows using an external `Key Management Service` for the key storage, therefore the keys are not stored in the `EncryptionConfiguration` file, which is more secure. The `kms` provider needs to deploy a KMS plugin for further use.
-The `Trousseau` KMS plugin is an example. It works through a unix socket, therefore `Trousseau` pods must be run on the same nodes as `kube-apiserver`. In case of using the KMS provider, the `EncryptionConfiguration` is as follows (`Vault` is a KMS):
-
-```yaml
-apiVersion: apiserver.config.k8s.io/v1
-kind: EncryptionConfiguration
-resources:
-  - resources:
-      - secrets
-      - configmaps
-    providers:
-      - kms:
-          name: vaultprovider
-          endpoint: unix:///opt/vault-kms/vaultkms.socket
-          cachesize: 100
-          timeout: 3s
-      - identity: {}
-```
-
-Also, unix socket must be available for `kube-apiserver`:
-
-
-```yaml
-apiVersion: v1
-kind: Pod
-metadata:
-  name: kube-apiserver
-  namespace: kube-system
-spec:
-  containers:
-  - command:
-    volumeMounts:
-    - mountPath: /opt/vault-kms/vaultkms.socket
-      name: vault-kms
-       ...
-  volumes:
-  - hostPath:
-      path: /opt/vault-kms/vaultkms.socket
-      type: Socket
-    name: vault-kms
-```
-
-The environment variable `VAULT_ADDR` matches the address of the `Vault` service and `--listen-addr` argument points to KMS plugin unix socket in the following example:
-
-```yaml
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: vault-kms-provider
-  namespace: kube-system
-    ...
-spec:
-  template:
-    spec:
-      initContainers:
-        - name: vault-agent
-          image: vault
-          securityContext:
-            privileged: true
-          args:
-            - agent
-            - -config=/etc/vault/vault-agent-config.hcl
-            - -log-level=debug
-          env:
-            - name: VAULT_ADDR
-              value: http://vault-adress:8200
-               ...
-      containers:
-        - name: vault-kms-provider
-          image: ghcr.io/ondat/trousseau:v1.1.3
-          imagePullPolicy: Always
-          args:
-            - -v=5
-            - --config-file-path=/opt/trousseau/config.yaml
-            - --listen-addr=unix:///opt/vault-kms/vaultkms.socket
-            - --zap-encoder=json
-            - --v=3
-```
-
-For more information, refer to:
-* [https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/)
-* [https://github.com/ondat/trousseau/wiki/Trousseau-Deployment](https://github.com/ondat/trousseau/wiki/Trousseau-Deployment)
-
-### Disabling Encryption
-
-The first step of disabling encryption is to make the `identity` provider default for encryption. The enabling of `EncryptionConfiguration` should be similar to the following example:
-
-```yaml
-apiVersion: apiserver.config.k8s.io/v1
-kind: EncryptionConfiguration
-resources:
-  - resources:
-      - secrets
-      - configmaps
-    providers:
-      - identity: {}
-      - aesgcm:
-          keys:
-            - name: key1
-              secret: c2VjcmV0IGlzIHNlY3VyZQ==
-            - name: key2
-              secret: dGhpcyBpcyBwYXNzd29yZA==
-      - aescbc:
-          keys:
-            - name: key1
-              secret: c2VjcmV0IGlzIHNlY3VyZQ==
-            - name: key2
-              secret: dGhpcyBpcyBwYXNzd29yZA==
-      - secretbox:
-          keys:
-            - name: key1
-              secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
-```
-
-The next step is to replace all resources that were previously encrypted (e.g. `secrets`):
-
-```console
-# kubectl get secrets --all-namespaces -o json | kubectl replace -f -
-```
-
-It is then possible to remove encryption settings from the `kubeadm-config` configmap and `kube-apiserver` manifest.
-
-### Maintenance and Operation Features
-
-* Since the `/etc/kubernetes/enc/enc.yaml` file has keys, access to the file must be restricted. For instance:
-```console
-# chmod 0700 /etc/kubernetes/enc/
-```
-
-* The proper way for using encryption is to rotate the keys. The rotation procedure of the keys should take into consideration the fact that the `EncryptionConfiguration` file must be equal on each `control-plane` node. During the keys rotation procedure, some operation of getting the encrypted resources may be unsuccessful.
-* The `kube-apiserver` has an `--encryption-provider-config-automatic-reload` option that allows applying a new `EncryptionConfiguration` without `kube-apiserver` reload.
-
-* ETCD restore procedures should take into consideration the keys rotation, otherwise some data may be unavailable due to keys that were used for encryption and is not available after restoration. The backup procedure may include an additional step that renews all encrypted data before the ETCD backup. This approach decreases the security level for data in ETCD backup, but it prevents any inconvenience in the future. Another option is not to delete the keys from `env.yml` even if they are not used for encryption/decryption anymore.
-* External services that interact with ETCD may stop working due to encryption enabling.
-
 ## Changing Cluster CIDR
 
 There might be a situation when you have to change the pod network used in a cluster. The default `podSubnet` (`10.128.0.0/14` for IPv4 and `fd02::/48` for IPv6) may be inappropriate for some reason.
@@ -1836,48 +1619,12 @@ data:
 
 11. Check that everything works properly and remove the old ippool if necessary.
 
-## Kubelet Server Certificate Approval
-
-The `kubelet` server certificate is self-signed by default, and is usually stored in the `/var/lib/kubelet/pki/kubelet.crt` file. To avoid using the self-signed `kubelet` server certificate, alter the `cluster.yaml` file in the following way:
-
-```yaml
-...
-services:
-  kubeadm_kubelet:
-    serverTLSBootstrap: true
-    rotateCertificates: true
-  kubeadm:
-    apiServer:
-      extraArgs:
-        kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt
-...
-```
-
-These settings enforce `kubelet` on each node of the cluster to request certificate approval (for `kubelet` server part) from the default Kubernetes CA and rotate certificate in the future. The `kube-apiserver` machinery does not approve certificate requests for `kubelet` automatically. They might be approved manually by the following commans. Get the list of certificate requests:
-
-```
-# kubectl get csr
-NAME        AGE     SIGNERNAME                          REQUESTOR                 REQUESTEDDURATION    CONDITION
-csr-2z6rv   12m     kubernetes.io/kubelet-serving       system:node:nodename-1    <none>               Pending
-csr-424qg   89m     kubernetes.io/kubelet-serving       system:node:nodename-2    <none>               Pending
-```
-
-Approve the particular request:
-
-```
-kubectl certificate approve csr-424qg
-```
-
-These commands might be automated in several ways.
-
-### Auto Approval CronJob
-
-Basically, `CronJob` runs the approval command above for every CSR according to some schedule.
+# Common Practice
 
-### Auto Approval Service
+## Security hardening guide
 
-It is possible to install the kubelet-csr-approver service. For more information, refer to [[kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver)](https://github.com/postfinance/kubelet-csr-approver). This service approves CSR automatically when a CSR is created according to several settings. It is better to restrict nodes' IP addresses (`providerIpPrefixes` option) and FQDN templates (providerRegex). For more information, refer to the official documentation.
+[Security hardening guide](./internal/Hardening.md)
 
-# Common Practice
+## Worker nodes should be managed by kubelet
 
 You should not run any containers on worker nodes that are not managed by `kubelet` so as not to break the `kube-scheduler` precision.

documentation/Troubleshooting.md

@@ -1216,4 +1216,4 @@ Error from server: error dialing backend: remote error: tls: internal error
 
 **Root cause**: The `kubelet` server certificate is not approved, whereas the cluster has been configured not to use self-signed certificates for the `kubelet` server.
 
-**Solution**: Perform CSR approval steps from the maintenance guide. Refer to the [Kubelet Server Certificate Approval](https://github.com/Netcracker/KubeMarine/blob/main/documentation/Maintenance.md#kubelet-server-certificate-approval) section for details.
+**Solution**: Perform CSR approval steps from the maintenance guide. Refer to the [Kubelet Server Certificate Approval](https://github.com/Netcracker/KubeMarine/blob/main/documentation/internal/Hardening.md#kubelet-server-certificate-approval) section for details.