Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

Commit

Permalink
docs: Clean up k8s with cri-containerd howto
Browse files Browse the repository at this point in the history 
This PR is based on #124 but has been reworked and updated to take into
account review feedback and extra cleanups to bring this howto in line
with the latest documentation requirements.

Fixes #127.

Contributions-from: T. Nichole Williams <[email protected]>
Signed-off-by: James O. D. Hunt <[email protected]>
  • Loading branch information
@jodh-intel
jodh-intel committed Sep 24, 2018
1 parent a64fccf commit 184a293
Showing 1 changed file with 176 additions and 134 deletions.
310 changes: 176 additions & 134 deletions how-to/how-to-use-k8s-with-cri-containerd-and-kata.md
View file
Original file line number Diff line number Diff line change
@@ -1,64 +1,82 @@
# How to use Kata Containers and CRI (containerd plugin) with Kubernetes

This document describes how to set up a single-machine Kubernetes cluster.
The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads.
* [Requirements](#requirements)
* [Install containerd with CRI plugin enabled](#install-containerd-with-cri-plugin-enabled)
* [Install Kata Containers](#install-kata-containers)
* [Install Kubernetes](#install-kubernetes)
* [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers)
* [Define the Kata runtime as the untrusted workload runtime](#define-the-kata-runtime-as-the-untrusted-workload-runtime)
* [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd)
* [Configure proxy - OPTIONAL](#configure-proxy---optional)
* [Start Kubernetes](#start-kubernetes)
* [Install a Pod Network](#install-a-pod-network)
* [Allow pods to run in the master node](#allow-pods-to-run-in-the-master-node)
* [Create an unstrusted pod using Kata Containers](#create-an-unstrusted-pod-using-kata-containers)
* [Delete created pod](#delete-created-pod)

This document describes how to set up a single-machine Kubernetes (k8s) cluster.

The Kubernetes cluster will use the
[CRI containerd plugin](https://github.com/containerd/cri) and
[Kata Containers](https://katacontainers.io) to launch untrusted workloads.

## Requirements

## Requirements
- Kubernetes, kubelet, kubeadm
- cri-containerd
- Kata Containers

For information about the supported version of these components see
Kata Containers [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml) file.
**Note:** For information about the supported versions of these components,
see the Kata Containers
[versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml)
file.

## Install containerd(with CRI plugin enabled)
## Install containerd with CRI plugin enabled

Follow the instructions from [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md)
- Follow the instructions from the
[CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md).

<!---
```bash
# Check if containerd is installed
$ command -v containerd
```
--->

## Install Kata Containers
- Check if `containerd` is now available
```bash
$ command -v containerd
```

Follow the instructions to [install Kata](https://github.com/kata-containers/documentation/blob/master/install/README.md).
## Install Kata Containers

<!---
```bash
# Check if kata-runtime is installed
$ command -v kata-runtime
# Check kata is well configured
$ kata-runtime kata-env
```
--->
Follow the instructions to
[install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md).

## Install Kubernetes
Install Kubernetes in your host. See kubeadm [installation](https://kubernetes.io/docs/tasks/tools/install-kubeadm/)
<!---
```bash
# Check if kubadm is installed
$ command -v kubeadm
```
--->

### Configure containerd to use Kata Containers
- Follow the instructions for
[kubeadm installation](https://kubernetes.io/docs/setup/independent/install-kubeadm/).

- Check `kubeadm` is now available

```bash
$ command -v kubeadm
```

## Configure containerd to use Kata Containers

The CRI `containerd` plugin supports configuration for two runtime types.

The CRI containerd plugin support configuration for two runtime types.
- **Default runtime:**

- Default runtime: A runtime that is used by default to run workloads.
- Untrusted workload runtime: A runtime that will be used run untrusted workloads.
A runtime that is used by default to run workloads.

#### Define the Kata runtime as `untrusted_workload_runtime`
- **Untrusted workload runtime:**

Configure the Kata runtime for untrusted workload with the [config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md)
`plugins.cri.containerd.untrusted_workload_runtime`.
A runtime that will be used to run untrusted workloads. This is appropriate
for workloads that require a higher degree of security isolation.

#### Define the Kata runtime as the untrusted workload runtime

Configure `containerd` to use the Kata runtime to run untrusted workloads by
setting the `plugins.cri.containerd.untrusted_workload_runtime`
[config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md):

Unless configured otherwise, the default runtime is set to `runc`.
```bash
# Configure containerd to use Kata as untrusted_workload_runtime
$ sudo mkdir -p /etc/containerd/
$ cat << EOT | sudo tee /etc/containerd/config.toml
[plugins]
Expand All @@ -69,28 +87,33 @@ $ cat << EOT | sudo tee /etc/containerd/config.toml
EOT
```

### Configure Kubelet to use containerd
**Note:** Unless configured otherwise, the default runtime is set to `runc`.

In order to allow kubelet use containerd (using CRI interface) configure the service to
point to containerd socket.
## Configure Kubelet to use containerd

```bash
# Configure k8s to use containerd
$ sudo mkdir -p /etc/systemd/system/kubelet.service.d/
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF
$ sudo systemctl daemon-reload
```
In order to allow kubelet to use containerd (using the CRI interface), configure the service to point to the `containerd` socket.

### Optional: Configure proxy
- Configure Kubernetes to use `containerd`

If you are behind a proxy this script will configure your proxy for docker
kubelet and containerd.
```bash
$ sudo mkdir -p /etc/systemd/system/kubelet.service.d/
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF
```
- Inform systemd about the new configuration
```bash
$ sudo systemctl daemon-reload
```
## Configure proxy - OPTIONAL
If you are behind a proxy, use the following script to configure your proxy for docker, kubelet, and containerd:
```bash
# Set proxys
$ services=(
'kubelet'
'containerd'
Expand All @@ -99,115 +122,134 @@ $ services=(
$ for s in "${services[@]}"; do
service_dir="/etc/systemd/system/${s}.service.d/"
sudo mkdir -p ${service_dir}
service_dir="/etc/systemd/system/${s}.service.d/"
sudo mkdir -p ${service_dir}
cat << EOT | sudo tee "${service_dir}/proxy.conf"
cat << EOT | sudo tee "${service_dir}/proxy.conf"
[Service]
Environment="HTTP_PROXY=${http_proxy}"
Environment="HTTPS_PROXY=${https_proxy}"
Environment="NO_PROXY=${no_proxy}"
EOT
done
$ sudo systemctl daemon-reload
```
### Start Kubernetes with kubeadm
## Start Kubernetes
```bash
# Mark sure containerd is up and running
$ sudo systemctl restart containerd
$ sudo systemctl status containerd
- Make sure `containerd` is up and running
# Prevent docker iptables rules conflict with k8s pod communication
$ sudo iptables -P FORWARD ACCEPT
```bash
$ sudo systemctl restart containerd
$ sudo systemctl status containerd
```
# Start cluster using kubeadm
$ sudo kubeadm init --skip-preflight-checks \
--cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
- Prevent conflicts between `docker` iptables (packet filtering) rules and k8s pod communication
$ export KUBECONFIG=/etc/kubernetes/admin.conf
```bash
$ sudo iptables -P FORWARD ACCEPT
```
$ sudo -E kubectl get nodes
$ sudo -E kubectl get pods
```
- Start cluster using `kubeadm`
### Install a pod network
Install a pod network plugin is needed to allow pods communicate with each other.
```bash
$ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
Install flannel plugging, by following the instructions in the section *Installing a pod network*
from [Using kubeadm to Create a Cluster ](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
guide.
$ export KUBECONFIG=/etc/kubernetes/admin.conf
<!---
```bash
# Install a pod network using flannel
# There is not a programmatic way to know last what flannel commit use
# See https://github.com/coreos/flannel/issues/995
$ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
--->
$ sudo -E kubectl get nodes
$ sudo -E kubectl get pods
```
## Install a Pod Network
```bash
# wait for pod network
$ timeout_dns=0
$ until [ "$timeout_dns" -eq "420" ]; do
if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
break
fi
sleep 1s
((timeout_dns+=1))
done

# check pod network is running
$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )
```
A pod network plugin is needed to allow pods to communicate with each other.
### Allow run pods in master node
- Install the `flannel` plugin by following the
[Using kubeadm to Create a Cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#instructions)
guide, starting from the **Installing a pod network** section.
By default, the cluster will not schedule pods in the master node to allow that run:
- Create a pod network using flannel
```bash
# allow master node run pods
$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
```
**Note:** There is not a programmatic way to know last what flannel commit use. See https://github.com/coreos/flannel/issues/995.
```bash
$ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
### Create a unstrusted pod using Kata Containers
- Wait for the pod network to become available
By default, all pods are created with the default runtime configured in CRI containerd plugin.
If a pod has the `io.kubernetes.cri.untrusted-workload annotation` set as
`"true"`, the CRI plugin will run the pod with the Kata Containers runtime.
```bash
# number of seconds to wait for pod network to become available
$ timeout_dns=420
```bash
# Create untrusted pod configuration
$ cat << EOT | tee nginx-untrusted.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-untrusted
annotations:
io.kubernetes.cri.untrusted-workload: "true"
spec:
containers:
- name: nginx
image: nginx
EOT
$ while [ "$timeout_dns" -gt 0 ]; do
if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
break
fi
sleep 1s
((timeout_dns-=1))
done
```
# Create untrusted pod
$ sudo -E kubectl apply -f nginx-untrusted.yaml
- Check the pod network is running
```bash
$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )
```
# Check pod is running
$ sudo -E kubectl get pods
## Allow pods to run in the master node
# Check qemu is running
$ ps aux | grep qemu
By default, the cluster will not schedule pods in the master node. To enable master node scheduling:
```bash
$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
```
### Delete created pod
## Create an unstrusted pod using Kata Containers
By default, all pods are created with the default runtime configured in CRI containerd plugin.
If a pod has the `io.kubernetes.cri.untrusted-workload` annotation set to `"true"`, the CRI plugin runs the pod with the
[Kata Containers runtime](https://github.com/kata-containers/runtime/blob/master/README.md).
- Create an untrusted pod configuration
```bash
$ cat << EOT | tee nginx-untrusted.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-untrusted
annotations:
io.kubernetes.cri.untrusted-workload: "true"
spec:
containers:
- name: nginx
image: nginx
EOT
```
- Create an untrusted pod
```bash
$ sudo -E kubectl apply -f nginx-untrusted.yaml
```
- Check pod is running
```bash
$ sudo -E kubectl get pods
```
- Check hypervisor is running
```bash
$ ps aux | grep qemu
```
## Delete created pod
```bash
# Delete pod
$ sudo -E kubectl delete -f nginx-untrusted.yaml
$ sudo -E kubectl delete -f nginx-untrusted.yaml
```

0 comments on commit 184a293

Please sign in to comment.