Low-privilege, self-service deployments in shared clusters

[mkmik]

Currently we assume a cluster admin will be willing to install sealed secrets controller globally.
This assumption doesn't hold for all of our users.

Many people end up with a shared k8s cluster where they get access to one namespace (or a handful of)

We should allow such users to self-service and deploy their own instance of the sealed-secrets controller that will serve their own sealed secrets resources in their own namespace.

It's not yet clear if we should limit this to only one namespace or whether the controller should operate on all sealed secrets resources the RBAC rules allow such a service account to access.

In any case we need to devise a mechanism for multiple controllers to avoid stomping on each others. Since each controller will have a different set of private keys, the effects of stomping on each other will not primarily affect correctness as such, but pollute logs, create spurious k8s events and possibly have unwanted side effects on work queue retries etc.

[jmichalek132]

Hi in my case I don't need avoid multiple controllers however, I cannot install sealed secrets globally.

[chrisob]

Not to add an unnecessary 👍, but IMHO this is a really important issue, as multi-tenant setups are more and more common (ie. OpenShift).

[ThomasVitt]

Same here! My customer has a large multi-tenant Redhat Openshift Cluster, and I won't get ClusterAdmin rights, just for a dozen namespaces. The sealedsecrets controller won't stop complaining about not having cluster-wide view of sealedsecrets.

[mkmik]

See related #501

[mkmik]

@ThomasVitt do you have the rights to install a CRD?

[ThomasVitt]

No, but the CRD has been installed by the cluster admins.
I found the "--all-namespaces" flag, but it doesn't help much, because I want the controller to watch a handful of namespaces which I control, not just one.

[mkmik]

@ThomasVitt for now you have to run N copies if the controller one in each namespace.

It's technically possible for us to add support for watching a specific set of namespaces. This is a common problem with kubernetes controllers/operators.

[ThomasVitt]

Thanks for the quick reply!
OK then I'll deploy the controller N times and wait for the new version containing this feature :-D

[mkmik]

The biggest hassle is that each controller will have its own private keys and thus you need to make sure to use the right public key to encrypt each secret.

One temporarily workaround is to just pick the "main" namespace (the one whose public key you tell your users to seal the secrets with) and just copy its private keys to the other namespaces.

See the backup section/faq of the readme for instructions on how to locate and download/transfer the sealed secrets private key secret.

[childofthewired]

The deployment of the controller seems to have issues in OpenShift Dedicated as well.

With limited administrative rights, users don't inherit the ability to grant access to the CRD API groups:

Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "secrets-unsealer" is forbidden: user "bob" (groups=["dedicated-admins" "system:authenticated:oauth" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["events"], Verbs:["create" "patch"]}
{APIGroups:[""], Resources:["secrets"], Verbs:["get" "create" "update" "delete"]}
{APIGroups:["bitnami.com"], Resources:["sealedsecrets"], Verbs:["get" "list" "watch"]}
{APIGroups:["bitnami.com"], Resources:["sealedsecrets/status"], Verbs:["update"]}
Error from server (NotFound): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "secrets-unsealer" not found

[RichardNixon52]

Any updates?

[varkrish]

Any update on this? I think ExternalSecrets controller is having this kind of a feature.

ExternalSecret manifest allows scoping the access of kubernetes-external-secrets controller. This allows deployment of multiple kubernetes-external-secrets instances at the same cluster and each instance can access a set of ExternalSecrets.