Merging of Creds/secrets into one env var

[duglin]

With CloudFoundry if you bind your app to more than one service the creds from both services will appear under VCAP_SERVICES - yes both of them. While in kube we can populate an env var from a secret, is there a way to populate an env var from multiple secrets? I'm guessing no. However, we may want to think about this since this means that the same app can not be deployed to CF and Kube and w/o code changes w.r.t. how they get the creds. While interop at this level has never been formally stated as a goal for us it might be nice to discuss/brainstorm this a bit to see if there's something we can do to ease the pain here.

[pmorie]

I remember that you had mentioned this quite a while ago, maybe in November at Kubecon. Now that I know a little bit more about the use cases you have for this for bluemix, I see that it has value for people that want to have applications interoperate between different consuming platforms without modification.

I think it should be possible to do this with a webhook admission controller (see kubernetes/community#132). You would run the controller as a pod on kubernetes and configure the API server to call out to it. The admission controller would look at the union of bindings that a pod would consume, and generate a VCAP_SERVICES environment variable matching the semantics of the CF env var.

I would be cool with having that admission controller live in our repo, since it is directly applicable to catalog use cases and would probably be useful in general to people working in the service catalog/broker space.

[duglin]

This feels like it would make the admission controller just a specialized podpreset

[nilebox]

@duglin one of the options I see there is to have a separate object which will produce a new secret from the multiple input secrets, let's call it SecretMerge for example. And the secret produced by SecretMerge controller is the one you want to specify in the PodPreset instead of the secrets produced by Bindings:

apiVersion: cloud-foundry.service-catalog.k8s/v1
kind: SecretMerge
metadata:
  name: cf-secret-merge
spec:
  outputSecretName: vcap-services-secret
  inputSecrets:
  - name: binding1-secret
  - name: binding2-secret

We have a similar problem when we need to inject the secret into an Instance as a merged result of multiple Bindings (e.g. environment variables for EC2 with multiple resource credentials), and going to solve it in similar way on top of Service Catalog.

@pmorie that's where decoupling between PodPreset and Binding pays off by giving such flexibility. Embedding PodPreset bits into Binding could make it less flexible.

[pmorie]

@nilebox

that's where decoupling between PodPreset and Binding pays off by giving such flexibility. Embedding PodPreset bits into Binding could make it less flexible.

I do not see at all what this has to do with that problem

[duglin]

The problem I see with an A.C. is that its global right? Will people be able to get VCAP_SERVICES on just some of their apps but not others?

[duglin]

I guess the A.C. could look for a special annotation/label on the pod as a trigger, and the user would have to set that as an indicator that they want this special sauce, right?

[duglin]

oh man @nilebox, that's one heck of a solution! :-)

I have to admit, I'm not thrilled with creating an A.C. or an entirely new controller to solve this.

[nilebox]

@duglin given that this problem is specific to applications designed for CloudFoundry, having a separate resource and controller seems to be a perfectly viable solution to me.

Also, is there such a requirement to run the same application on both Kube and CloudFoundry platforms without any changes? Especially given that this is not part of OSB.

[duglin]

@nilebox granted its nice that your solution reuses podpreset and secrets but it just feel odd that we have to create a new resource and controller for this. We may have to do something that big (or create an AC), just hoping we can come up with something else.

This isn't a requirement from OSB, its more from a realization that we have customers that run apps in both places. In fact, we even had a customer that hosted their app on CF and Docker but used the Docker version as a back-up for when/if the CF side of things acted up. So them not having to have special logic to get the creds based on where the app was running would have been nice.

[nilebox]

@duglin fair enough. I agree that AC with special annotation could be a better and easier to use solution.

Labels are not designed for this, and I don't see a requirement for searching for all applications with this label, so annotations are preferable.

[pmorie]

I have to admit, I'm not thrilled with creating an A.C. or an entirely new controller to solve this.

As far as I can tell, there is no way to do this without a core admission controller.

[pmorie]

Re: secret composition

Short version: I understand the desire for this; this repo is not the place for it; I suggest sig-auth as engagement point on the problem of composing secrets and suggest looking into the history of templating and secret management in kubernetes as starting points.

Long version: I am not opposed to someone building a new resource like the one suggested earlier in their own API server that lives outside this repository at all. Given my prior experience with secret management in kubernetes, I would caution anyone wanting to create such a resource that it will probably be far more complex than it seems on the surface, and advise a careful security review and upfront design. Example: the controller implementing this resource would have to watch all secrets in all namespaces and be able to get them and create them. That is a very high degree of privilege for a controller to operate on; think carefully before building it.

Such a resource is not in scope for this repository; a possible point of engagement would be SIG auth.

As far as I can tell, there is no way to do this without a core admission controller.

For clarity, I meant an admission controller for the core API server, which can be a remote admission controller that does not have to have code compiled into kubernetes.

granted its nice that your solution reuses podpreset and secrets but it just feel odd that we have to create a new resource and controller for this. We may have to do something that big (or create an AC), just hoping we can come up with something else.

It seems to me like CF must have a component very similar to the proposed AC that fills the same role - it's just that no one has written such a thing for Kubernetes yet. This is the type of use case that the remote admission controller use-case is meant to enable in Kubernetes.

I think the specific experience being asked for is:

1. User makes x number of bindings
2. User's pods are injected automatically with a VCAP services variable for all bindings in the namespace

Is that accurate? If so, the remote AC seems like an option that could deliver this. I am not making a guarantee here; you will need to check out kubernetes/community#132 and do some experimenting.

[pmorie]

Hrm

Example: the controller implementing this resource would have to watch all secrets in all namespaces and be able to get them and create them. That is a very high degree of privilege for a controller to operate on; think carefully before building it.

This is also true of an initializer (remote admission controller). Also, I did some homework: see PR kubernetes/kubernetes#36721 for the initial implementation of initializer.

Despite the fact that the initializer has similar security implications to a new resource, I would recommend the initializer because secret templating is a very deep, hard problem.

You may want to build an initializer in your own repository first; that would let you prototype without having to convince the SIG at every turn. I suspect it would go much faster to build an initializer that does what you want and then demo to the SIG when you had it the way you wanted it.

Another thought I had: I think that this use-case will also be interesting to people migrating applications from cloud foundry to Kubernetes in addition to the cases @duglin has already described.

[MHBauer]

This is an interesting request. Since OSB is all about Platform <-> Broker, is there any expectation that end-user programs can use services without adaptation?

The munging of everything into a json blob is especially troublesome.

[nilebox]

I think we should just close this issue.
It again comes down to the goals/non-goals of the Service Catalog project, and this issue IMO is out of the scope of Service Catalog.

It doesn't prevent anyone from building services on top of Service Catalog that will do any secret transformation they might want. Actually, that's what we already do internally at Atlassian, and we are fine with it.

@duglin your thoughts?

P.S. It would be nice to have a bot like in other Kubernetes repos that would mark issues as stale and eventually (after 90 days of inactivity) just close them. That would force us to revisit abandoned issues if we think they are still valid.

[duglin]

I'd prefer to leave it open until we discuss what we're going to do about podpreset type of stuff. As of now we don't have any mechanism to help people bind their apps to secrets - we stop at creating the secrets. While that leaves them in a state that we could consider just a "normal kube state" and then they can do whatever they normally would with secrets, I'd like a formal discussion (and agreement) that we have no intention (as of now) to do any more. For example, I'd like a formal decision on the idea of PodPreSetBinding type of resources. If the group decides that we have no need for these kinds of things then yea we can probably close this issue since this is just another variant of that idea.

Might be a good topic for the f2f - I'll add it.

[nilebox]

As of now we don't have any mechanism to help people bind their apps to secrets - we stop at creating the secrets.

We do let them bind secrets as envFrom directly to the Pod (or via PodPreset) if the broker returns credentials in a certain simple way (Map<String, String>).

So the question is whether we want to support some other advanced/flexible ways of transforming secrets, or it's out of the scope of Service Catalog.

Discussing at F2F sounds good.

[ash2k]

Actually, that's what we already do internally at Atlassian, and we are fine with it.

For the curious external observers, it is part of Smith. We are planning to use or use already object references and field references to pull in data from Secrets/ConfigMaps/etc to construct parameters for ServiceInstances (or Secrets/ConfigMaps for Deployments) and put sensitive ones into an automatically constructed Secrets (see atlassian/smith#233).

Some of that has been implemented already, some things are in progress.

We have decided that we don't want to and often cannot impose any restrictions on the "shape" of output of an OSB broker and we definitely cannot restrict in what format the user's application wants its parameters. It may be a blob of JSON or distinct environment variables or a config file. There should be some glue that does the transformation. I don't see a viable way how Service Catalog could support this. So we decided to build an external mechanism that does the transformations that are needed for our use cases (not all use cases).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[MHBauer]

/assign @jberkhahn
If you want VCAP_SERVICES output, this is the issue.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten