Move Cilium back to kube-system namespace

[nebril]

Behaviour from kubernetes/kubernetes#51376 is
no longer observed, tested in both COS and Ubuntu.

[tgraf]

@nebril What happened to the RBAC problems involving access to secrets?

[nebril]

@tgraf TBH I was not aware of these, but all secrets seems to be mounting properly in this setup. Maybe it was the same issue which got fixed? I have no idea :/

[tgraf]

Last time you explained it that it depends on a user account with clusterAdmin role which was difficult to take as granted. Are you using the same accounts to set up the clusters?

I also checked the example podSecurity policy:
https://raw.githubusercontent.com/kubernetes/website/81b862e2bcdafbdb0cfc91b6524ef06496f1dc79/content/en/examples/policy/restricted-psp.yaml

[ruigulala]

@nebril What's the reason that cilium has to be in "kube-system" instead of "cilium"?

[nebril]

@ruigulala the main upside of running in kube-system is that Cilium pods will be properly treated as critical pods: https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/

[nebril]

@tgraf I am using clusterAdmin role to set Cilium up if that's what you are asking for

[tgraf]

@nebril So what changed? What are you doing differently from before? We can't just keep switching back and forth.

If we do this, then we also need to document that the user account used for setup needs to have a clusterolebinding to admin via kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole cluster-admin --user [USER_ACCOUNT].

[aanm]

@tgraf @nebril I also need to know what will be the final namespace chosen. Pods can have the priority value that we want, critical pods are pods with a higher priority set by default. If we are in kube-system no changes are need to be performed in our k8s descriptors, if it's to stay on cilium namespace then we need to add a priority class with a higher value for cilium pods.

[tgraf]

@aanm Then we should just add it. It's good to have the flexibility to run outside of the kube-system namespace. People may have arbitrary RBAC restrictions on kube-system.