Skip to content

Commit

Permalink
encryption
Browse files Browse the repository at this point in the history
  • Loading branch information
@alexarefev
alexarefev committed Nov 3, 2023
1 parent 19cf0e9 commit a6eb8de
Show file tree
Hide file tree
Showing 2 changed files with 223 additions and 223 deletions.
219 changes: 0 additions & 219 deletions documentation/Maintenance.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1509,225 +1509,6 @@ plugins:
```

## Data Encryption in Kubernetes

The following section describes the Kubernetes cluster capabilities to store and manipulate encrypted data.

### Enabling Encryption

ETCD as a Kubernetes cluster storage can interact with encrypted data. The encryption/decryption procedures are the part of `kube-apiserver` functionality.

An example of the `EncryptionConfiguration` file is as follows:

```yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- aesgcm:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- aescbc:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- secretbox:
keys:
- name: key1
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
- identity: {}
```

It should be created preliminarily and placed in the `/etc/kubernetes/enc/` directory.

The next step is to enable the encryption settings in `kubeadm-config`:
```yaml
data:
ClusterConfiguration: |
apiServer:
...
extraArgs:
...
encryption-provider-config: /etc/kubernetes/enc/enc.yaml
extraVolumes:
...
- hostPath: /etc/kubernetes/enc
mountPath: /etc/kubernetes/enc
name: enc
pathType: DirectoryOrCreate
```

There is an `--encryption-provider-config` option that points to the `EncryptionConfiguration` file location. The `kube-apiserver` should have the following parts in the manifest yaml:

```yaml
...
spec:
containers:
- command:
- kube-apiserver
...
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
...
volumeMounts:
- name: enc
mountPath: /etc/kubernetes/enc
readonly: true
...
volumes:
- name: enc
hostPath:
path: /etc/kubernetes/enc
type: DirectoryOrCreate
```

In the above case, the `secrets` and `configmaps` are encrypted on the first key of the `aesgcm` provider, but the previously encrypted `secrets` and `configmaps` are decrypted on any keys of any providers that are matched. This approach allows to change both encryption providers and keys during the operation. The keys should be random strings in base64 encoding. `identity` is the default provider that does not provide any encryption at all.
For more information, refer to [https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/).

### Integration with External KMS

There is an encryption provider `kms` that allows using an external `Key Management Service` for the key storage, therefore the keys are not stored in the `EncryptionConfiguration` file, which is more secure. The `kms` provider needs to deploy a KMS plugin for further use.
The `Trousseau` KMS plugin is an example. It works through a unix socket, therefore `Trousseau` pods must be run on the same nodes as `kube-apiserver`. In case of using the KMS provider, the `EncryptionConfiguration` is as follows (`Vault` is a KMS):

```yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- kms:
name: vaultprovider
endpoint: unix:///opt/vault-kms/vaultkms.socket
cachesize: 100
timeout: 3s
- identity: {}
```

Also, unix socket must be available for `kube-apiserver`:


```yaml
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
volumeMounts:
- mountPath: /opt/vault-kms/vaultkms.socket
name: vault-kms
...
volumes:
- hostPath:
path: /opt/vault-kms/vaultkms.socket
type: Socket
name: vault-kms
```

The environment variable `VAULT_ADDR` matches the address of the `Vault` service and `--listen-addr` argument points to KMS plugin unix socket in the following example:

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: vault-kms-provider
namespace: kube-system
...
spec:
template:
spec:
initContainers:
- name: vault-agent
image: vault
securityContext:
privileged: true
args:
- agent
- -config=/etc/vault/vault-agent-config.hcl
- -log-level=debug
env:
- name: VAULT_ADDR
value: http://vault-adress:8200
...
containers:
- name: vault-kms-provider
image: ghcr.io/ondat/trousseau:v1.1.3
imagePullPolicy: Always
args:
- -v=5
- --config-file-path=/opt/trousseau/config.yaml
- --listen-addr=unix:///opt/vault-kms/vaultkms.socket
- --zap-encoder=json
- --v=3
```

For more information, refer to:
* [https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/)
* [https://github.com/ondat/trousseau/wiki/Trousseau-Deployment](https://github.com/ondat/trousseau/wiki/Trousseau-Deployment)

### Disabling Encryption

The first step of disabling encryption is to make the `identity` provider default for encryption. The enabling of `EncryptionConfiguration` should be similar to the following example:

```yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- identity: {}
- aesgcm:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- aescbc:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- secretbox:
keys:
- name: key1
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
```

The next step is to replace all resources that were previously encrypted (e.g. `secrets`):

```console
# kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```

It is then possible to remove encryption settings from the `kubeadm-config` configmap and `kube-apiserver` manifest.

### Maintenance and Operation Features

* Since the `/etc/kubernetes/enc/enc.yaml` file has keys, access to the file must be restricted. For instance:
```console
# chmod 0700 /etc/kubernetes/enc/
```

* The proper way for using encryption is to rotate the keys. The rotation procedure of the keys should take into consideration the fact that the `EncryptionConfiguration` file must be equal on each `control-plane` node. During the keys rotation procedure, some operation of getting the encrypted resources may be unsuccessful.
* The `kube-apiserver` has an `--encryption-provider-config-automatic-reload` option that allows applying a new `EncryptionConfiguration` without `kube-apiserver` reload.

* ETCD restore procedures should take into consideration the keys rotation, otherwise some data may be unavailable due to keys that were used for encryption and is not available after restoration. The backup procedure may include an additional step that renews all encrypted data before the ETCD backup. This approach decreases the security level for data in ETCD backup, but it prevents any inconvenience in the future. Another option is not to delete the keys from `env.yml` even if they are not used for encryption/decryption anymore.
* External services that interact with ETCD may stop working due to encryption enabling.

## Changing Cluster CIDR

There might be a situation when you have to change the pod network used in a cluster. The default `podSubnet` (`10.128.0.0/14` for IPv4 and `fd02::/48` for IPv6) may be inappropriate for some reason.
Expand Down
Loading

0 comments on commit a6eb8de

Please sign in to comment.