Migration from v1 to v2 #119

ymmt2005 · 2020-10-06T15:25:25Z

This PR elaborates on the migration steps and implements the necessary tools and changes.
Specifically,

The design is updated to describe the complete steps for online migration.
Coil v2 is a bit enhanced to support the migration.
The migration tool called coil-migrator is implemented.

The new design is to reserved AddressBlocks used by Coil v1. The reserved blocks will not be used by Coil v2 so Pods run by Coil v1 can survive during and after migration to v1.

By setting "coil.cybozu.com/reserved=true" label to an AddressBlock, the block won't be used for IP address assignments. This work is to implement a migrator from Coil v1.

it should not have been renamed.

If not, a zero value of such a field would be omitted and the resulting JSON or YAML could create a wrong resource where the field is set to its default value, not zero.

dump subcommand does the following: - Uninstall Coil v1 resources - Annotate namespaces that use a non-default AddressPool - Convert v1 resources into v2 and dump YAML This command is idempotent and can be run multiple times.

This command does the followings: - Remove all the currently running Pods one by one. - After the pod removal, delete all reserved address blocks.

ymmt2005 · 2020-10-07T03:48:41Z

Test steps and results in a dctest environment. Result: SUCCESS

Prepare coil v2

$ cd cybozu-go/coil/coil-migrator
$ go build .
$ cd ../v2
$ make certs

bootstrap dctest and copy coil v2 to boot-0

$ cd cybozu-go/neco/dctest
$ make placemat
$ make test SUITE=bootstrap
$ ./dcscp .../coil/v2/coil-migrator/coil-migrator boot-0:
$ ./dcscp -r .../coil/v2 boot-0:
$ ./dcssh boot-0

Prepare migration

boot-0:~$ ckecli kubernetes issue --ttl=3h > $HOME/.kube/config
boot-0:~$ ckecli etcd root-issue --output=file
cert files:  etcd-ca.crt etcd-root.crt etcd-root.key
boot-0:~$ kubectl run pod1 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl run pod2 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl run pod3 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl run pod4 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl get pods -A -o wide
NAMESPACE         NAME                                READY   STATUS    RESTARTS   AGE     IP            NODE
default           pod1                                1/1     Running   0          19s     10.64.0.128   10.69.0.197
default           pod2                                1/1     Running   0          16s     10.64.0.160   10.69.0.196
default           pod3                                1/1     Running   0          12s     10.64.0.33    10.69.2.68
default           pod4                                1/1     Running   0          8s      10.64.0.1     10.69.0.4
internet-egress   squid-75c5d8bdbf-drtwk              2/2     Running   0          7m17s   172.19.0.0    10.69.0.197
internet-egress   squid-75c5d8bdbf-p8tnv              2/2     Running   0          7m17s   172.19.0.2    10.69.0.4
internet-egress   unbound-559dfc758f-7j2hr            1/1     Running   0          7m17s   172.19.0.3    10.69.0.196
internet-egress   unbound-559dfc758f-jn8dl            1/1     Running   0          7m17s   172.19.0.1    10.69.2.68
kube-system       cluster-dns-89b9b9b49-q7ffn         1/1     Running   0          7m18s   10.64.0.0     10.69.0.4
kube-system       cluster-dns-89b9b9b49-xqvcz         1/1     Running   0          7m18s   10.64.0.32    10.69.2.68
kube-system       coil-controllers-6766cfdcc8-vm4k9   1/1     Running   0          7m16s   10.69.0.196   10.69.0.196
kube-system       coil-node-9jgs7                     1/1     Running   0          7m34s   10.69.0.196   10.69.0.196
kube-system       coil-node-ghlgw                     1/1     Running   0          7m34s   10.69.2.68    10.69.2.68
kube-system       coil-node-h9vv4                     1/1     Running   0          7m34s   10.69.0.197   10.69.0.197
kube-system       coil-node-nbd74                     1/1     Running   0          7m34s   10.69.0.4     10.69.0.4
kube-system       coil-node-qhf62                     1/1     Running   0          7m34s   10.69.0.6     10.69.0.6
kube-system       node-dns-2w2wc                      2/2     Running   0          7m41s   10.69.0.196   10.69.0.196
kube-system       node-dns-6ls89                      2/2     Running   0          7m41s   10.69.0.6     10.69.0.6
kube-system       node-dns-gd4vg                      2/2     Running   0          7m41s   10.69.0.4     10.69.0.4
kube-system       node-dns-mz6ll                      2/2     Running   0          7m41s   10.69.0.197   10.69.0.197
kube-system       node-dns-slgh4                      2/2     Running   0          7m41s   10.69.2.68    10.69.2.68

Remove Coil resources from CKE

boot-0:~$ ckecli resource delete /usr/share/neco/coil-deploy.yml
boot-0:~$ ckecli resource delete /usr/share/neco/coil-rbac.yml
boot-0:~$ ckecli resource list
ClusterRole/psp:restricted
ClusterRoleBinding/default:psp:restricted
ConfigMap/internet-egress/local-unbound
ConfigMap/internet-egress/squid
ConfigMap/internet-egress/unbound
Deployment/internet-egress/squid
Deployment/internet-egress/unbound
Namespace/internet-egress
PodDisruptionBudget/internet-egress/squid-pdb
PodDisruptionBudget/internet-egress/unbound-pdb
PodSecurityPolicy/coil
PodSecurityPolicy/restricted
PodSecurityPolicy/squid
Role/internet-egress/psp:squid
RoleBinding/internet-egress/squid:psp:squid
Service/internet-egress/squid
Service/internet-egress/unbound
ServiceAccount/internet-egress/squid
ServiceAccount/internet-egress/unbound

PodSecurityPolicy/coil was defined separately, so it remains.
Coil v2 names PSP differently, so this will not cause conflicts.

Run coil-migrator dump

boot-0:~$ CPNODE=$(kubectl get nodes | grep master | head -1 | awk '{print $1}')
boot-0:~$ echo $CPNODE
10.69.0.196

boot-0:~$ ./coil-migrator dump --etcd-tls-ca=etcd-ca.crt --etcd-tls-cert=etcd-root.crt --etcd-tls-key=etcd-root.key --etcd-endpoints=$CPNODE:2379 > data.yaml
uninstalling Coil v1 ...
  deleted *v1.DaemonSet coil-node
  deleted *v1.Deployment coil-controllers
  deleted *v1.ConfigMap coil-config
  deleted *v1.ClusterRoleBinding coil-node
  deleted *v1.ClusterRoleBinding coil-controllers
  deleted *v1.ClusterRole coil-node
  deleted *v1.ClusterRole coil-controllers
  deleted *v1.ServiceAccount coil-node
  deleted *v1.ServiceAccount coil-controller
annotated namespace internet-egress

This generated the following YAML:

---
apiVersion: coil.cybozu.com/v2
kind: AddressPool
metadata:
  creationTimestamp: null
  name: internet-egress
spec:
  blockSizeBits: 0
  subnets:
  - ipv4: 172.19.0.0/28
---
apiVersion: coil.cybozu.com/v2
kind: AddressPool
metadata:
  creationTimestamp: null
  name: default
spec:
  blockSizeBits: 5
  subnets:
  - ipv4: 10.64.0.0/14
---
apiVersion: coil.cybozu.com/v2
index: 5
ipv4: 10.64.0.160/27
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.0.196
    coil.cybozu.com/pool: default
    coil.cybozu.com/reserved: "true"
  name: default-5-v1
---
apiVersion: coil.cybozu.com/v2
index: 3
ipv4: 172.19.0.3/32
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.0.196
    coil.cybozu.com/pool: internet-egress
    coil.cybozu.com/reserved: "true"
  name: internet-egress-3-v1
---
apiVersion: coil.cybozu.com/v2
index: 4
ipv4: 10.64.0.128/27
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.0.197
    coil.cybozu.com/pool: default
    coil.cybozu.com/reserved: "true"
  name: default-4-v1
---
apiVersion: coil.cybozu.com/v2
index: 0
ipv4: 172.19.0.0/32
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.0.197
    coil.cybozu.com/pool: internet-egress
    coil.cybozu.com/reserved: "true"
  name: internet-egress-0-v1
---
apiVersion: coil.cybozu.com/v2
index: 0
ipv4: 10.64.0.0/27
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.0.4
    coil.cybozu.com/pool: default
    coil.cybozu.com/reserved: "true"
  name: default-0-v1
---
apiVersion: coil.cybozu.com/v2
index: 2
ipv4: 172.19.0.2/32
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.0.4
    coil.cybozu.com/pool: internet-egress
    coil.cybozu.com/reserved: "true"
  name: internet-egress-2-v1
---
apiVersion: coil.cybozu.com/v2
index: 1
ipv4: 10.64.0.32/27
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.2.68
    coil.cybozu.com/pool: default
    coil.cybozu.com/reserved: "true"
  name: default-1-v1
---
apiVersion: coil.cybozu.com/v2
index: 1
ipv4: 172.19.0.1/32
kind: AddressBlock
metadata:
  creationTimestamp: null
  labels:
    coil.cybozu.com/node: 10.69.2.68
    coil.cybozu.com/pool: internet-egress
    coil.cybozu.com/reserved: "true"
  name: internet-egress-1-v1

Apply Coil v2 CRDs

boot-0:~$ cd v2/config/crd/
boot-0:~/v2/config/crd$ kustomize build . | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/addressblocks.coil.cybozu.com created
customresourcedefinition.apiextensions.k8s.io/addresspools.coil.cybozu.com created
customresourcedefinition.apiextensions.k8s.io/blockrequests.coil.cybozu.com created
customresourcedefinition.apiextensions.k8s.io/egresses.coil.cybozu.com created

boot-0:~/v2/config/crd$ cd

Apply data.yaml

boot-0:~$ kubectl apply -f data.yaml
addresspool.coil.cybozu.com/internet-egress created
addresspool.coil.cybozu.com/default created
addressblock.coil.cybozu.com/default-5-v1 created
addressblock.coil.cybozu.com/internet-egress-3-v1 created
addressblock.coil.cybozu.com/default-4-v1 created
addressblock.coil.cybozu.com/internet-egress-0-v1 created
addressblock.coil.cybozu.com/default-0-v1 created
addressblock.coil.cybozu.com/internet-egress-2-v1 created
addressblock.coil.cybozu.com/default-1-v1 created
addressblock.coil.cybozu.com/internet-egress-1-v1 created

Edit Coil v2 kustomization.yaml and netconf.json, then apply it

Enable PSP, calico compatibility, the image tag, and set MTU for dctest environment.

boot-0:~$ cd v2
boot-0:~/v2$ sed -i -E 's,^#(- config/default/pod_security_policy.yaml),\1, ; s,^#(- config/pod/compat_calico.yaml),\1, ; s,newTag:.*$,newTag: test,' kustomization.yaml

boot-0:~/v2$ cat >netconf.json <<'EOF'
{
  "cniVersion": "0.4.0",
  "name": "k8s-pod-network",
  "plugins": [
    {
      "type": "coil",
      "socket": "/run/coild.sock"
    },
    {
      "type": "tuning",
      "mtu": 1400
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
EOF

boot-0:~/v2$ kustomize build . | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/addressblocks.coil.cybozu.com configured
customresourcedefinition.apiextensions.k8s.io/addresspools.coil.cybozu.com configured
customresourcedefinition.apiextensions.k8s.io/blockrequests.coil.cybozu.com configured
customresourcedefinition.apiextensions.k8s.io/egresses.coil.cybozu.com configured
serviceaccount/coil-controller created
serviceaccount/coil-router created
serviceaccount/coild created
podsecuritypolicy.policy/coil-controller created
podsecuritypolicy.policy/coil-egress created
podsecuritypolicy.policy/coil-router created
podsecuritypolicy.policy/coild created
role.rbac.authorization.k8s.io/coil-leader-election created
clusterrole.rbac.authorization.k8s.io/coil-controller created
clusterrole.rbac.authorization.k8s.io/coil-egress created
clusterrole.rbac.authorization.k8s.io/coil-router created
clusterrole.rbac.authorization.k8s.io/coild created
clusterrole.rbac.authorization.k8s.io/coilv2-addressblock-viewer-role created
clusterrole.rbac.authorization.k8s.io/coilv2-addresspool-viewer-role created
clusterrole.rbac.authorization.k8s.io/coilv2-blockrequest-viewer-role created
clusterrole.rbac.authorization.k8s.io/coilv2-egress-viewer-role created
clusterrole.rbac.authorization.k8s.io/psp-coil-controller created
clusterrole.rbac.authorization.k8s.io/psp-coil-egress created
clusterrole.rbac.authorization.k8s.io/psp-coil-router created
clusterrole.rbac.authorization.k8s.io/psp-coild created
rolebinding.rbac.authorization.k8s.io/coil-leader-election created
clusterrolebinding.rbac.authorization.k8s.io/coil-controller created
clusterrolebinding.rbac.authorization.k8s.io/coil-egress created
clusterrolebinding.rbac.authorization.k8s.io/coil-router created
clusterrolebinding.rbac.authorization.k8s.io/coild created
clusterrolebinding.rbac.authorization.k8s.io/psp-coil-controller created
clusterrolebinding.rbac.authorization.k8s.io/psp-coil-egress created
clusterrolebinding.rbac.authorization.k8s.io/psp-coil-router created
clusterrolebinding.rbac.authorization.k8s.io/psp-coild created
configmap/coil-config-22b825t954 created
secret/coilv2-webhook-server-cert created
service/coilv2-webhook-service created
deployment.apps/coil-controller created
daemonset.apps/coild created
mutatingwebhookconfiguration.admissionregistration.k8s.io/coilv2-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/coilv2-validating-webhook-configuration created

Run a new pod to see Coil v2 is working

boot-0:~/v2$ kubectl run newpod1 --image=quay.io/cybozu/testhttpd:0
boot-0:~/v2$ kubectl get pods -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE
newpod1   1/1     Running   0          5s    10.64.0.64    10.69.2.68
pod1      1/1     Running   0          10m   10.64.0.128   10.69.0.197
pod2      1/1     Running   0          10m   10.64.0.160   10.69.0.196
pod3      1/1     Running   0          10m   10.64.0.33    10.69.2.68
pod4      1/1     Running   0          10m   10.64.0.1     10.69.0.4

Run coil-migrator restart

boot-0:~$ ./coil-migrator replace
DO YOU WANT TO PROCEED? [y/N] y
Deleting pods on node 10.69.0.4...
  deleting default/pod4
  deleting internet-egress/squid-75c5d8bdbf-p8tnv
  deleting kube-system/cluster-dns-89b9b9b49-q7ffn
Deleting pods on node 10.69.2.68...
  deleting default/newpod1
  deleting default/pod3
  deleting internet-egress/unbound-559dfc758f-jn8dl
  deleting kube-system/cluster-dns-89b9b9b49-xqvcz
Deleting pods on node 10.69.0.197...
  deleting default/pod1
  deleting internet-egress/squid-75c5d8bdbf-drtwk
Deleting pods on node 10.69.0.196...
  deleting default/pod2
  deleting internet-egress/unbound-559dfc758f-7j2hr
deleting reserved address blocks

Check remaining address blocks

boot-0:~$ kubectl get addressblocks
NAME                NODE          POOL              IPV4            IPV6
default-2           10.69.0.197   default           10.64.0.64/27
default-3           10.69.0.4     default           10.64.0.96/27
internet-egress-4   10.69.2.68    internet-egress   172.19.0.4/32
internet-egress-5   10.69.0.4     internet-egress   172.19.0.5/32
internet-egress-6   10.69.0.4     internet-egress   172.19.0.6/32
internet-egress-7   10.69.2.68    internet-egress   172.19.0.7/32

morimoto-cybozu

LGTM

Update migration steps from v1 to v2.

f89c732

The new design is to reserved AddressBlocks used by Coil v1. The reserved blocks will not be used by Coil v2 so Pods run by Coil v1 can survive during and after migration to v1.

ymmt2005 self-assigned this Oct 6, 2020

ymmt2005 added this to the v2 milestone Oct 6, 2020

ymmt2005 changed the title ~~Update migration steps from v1 to v2.~~ Migration from v1 to v2 Oct 6, 2020

ymmt2005 force-pushed the migration-from-v1 branch 2 times, most recently from bb33b9d to e8af076 Compare October 6, 2020 17:04

ymmt2005 requested a review from morimoto-cybozu October 6, 2020 17:15

ymmt2005 marked this pull request as ready for review October 6, 2020 17:15

ymmt2005 added 6 commits October 7, 2020 01:00

pkg/ipam: make AddressBlocks reservable

40b0311

By setting "coil.cybozu.com/reserved=true" label to an AddressBlock, the block won't be used for IP address assignments. This work is to implement a migrator from Coil v1.

coild: delete pod network created by Coil v1

d7ebb69

v1: fix module path

453f414

it should not have been renamed.

api/v2: don't omit empty value for fields with default values

b41ad30

If not, a zero value of such a field would be omitted and the resulting JSON or YAML could create a wrong resource where the field is set to its default value, not zero.

coil-migrator: implement dump subcommand

2f60cf3

dump subcommand does the following: - Uninstall Coil v1 resources - Annotate namespaces that use a non-default AddressPool - Convert v1 resources into v2 and dump YAML This command is idempotent and can be run multiple times.

coil-migrator: implement replace subcommand

ecff237

This command does the followings: - Remove all the currently running Pods one by one. - After the pod removal, delete all reserved address blocks.

ymmt2005 force-pushed the migration-from-v1 branch from e8af076 to 35e5420 Compare October 7, 2020 01:00

Fix PodSecurityPolicies

b4b50d5

ymmt2005 force-pushed the migration-from-v1 branch from 35e5420 to b4b50d5 Compare October 7, 2020 03:04

morimoto-cybozu approved these changes Oct 7, 2020

View reviewed changes

morimoto-cybozu merged commit 44def64 into master Oct 7, 2020

morimoto-cybozu deleted the migration-from-v1 branch October 7, 2020 05:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration from v1 to v2 #119

Migration from v1 to v2 #119

ymmt2005 commented Oct 6, 2020 •

edited

Loading

ymmt2005 commented Oct 7, 2020 •

edited

Loading

morimoto-cybozu left a comment

Migration from v1 to v2 #119

Migration from v1 to v2 #119

Conversation

ymmt2005 commented Oct 6, 2020 • edited Loading

ymmt2005 commented Oct 7, 2020 • edited Loading

morimoto-cybozu left a comment

Choose a reason for hiding this comment

ymmt2005 commented Oct 6, 2020 •

edited

Loading

ymmt2005 commented Oct 7, 2020 •

edited

Loading