Support dynamic configuration distribution

[bgrant0607]

A related topic came up on IRC today.

@pwhiteside pointed out envconsul, which can be used to distribute environment variables (or, in theory, arbitrary key-value config data). Something similar could be supported on top of etcd. As with master election (#1542) and endpoint listing, it would be useful to have a canonical solution available in the Kubernetes ecosystem that wasn't tied to a specific key-value store API.

Even if we don't provide explicit support, it would be useful to document this pattern for others.

[trinitronx]

This goes along with the 12-factor app paradigm of storing configuration info (e.g.: db_name, db_host, db_port, redis_host, etc...) in the environment and allows for separation of configuration info from the deployable code in a way that facilitates different settings for different environments (e.g.: test, stage, prod). Probably a good idea to provide an 'adapter' or way for 12-factor apps written to depend on environment variables being set to get this info from etcd.

[bketelsen]

This tool already exists: https://github.com/mattn/etcdenv

[bgrant0607]

@bketelsen Awesome!

[stp-ip]

So we encourage to put all config stuff in ENVs and if need be, instead of distributing these at runtime through docker parameters, we could use etcdenv as first bootstrap step and generate the additional config after that right?
I want to get some best practices going. It's much easier to work with changing environments, when there is at least some sort of guideline.
So I would love to see only ENVs being used for config. Either provided via parameters or later on via etcdenv. Optimal thing would be the possibility to use etcdenv without the docker container knowing. This would make it possible to use any container setup to use ENVs to be handled via etcd later on.

The next step would be to generate config files from these values and using sed etc. is just not clean in my opinion. I would propose augeas.net for that. (I'll write a guideline issue in the coming days, just wanted to get the idea out here already in context)
The missing peace would be file and secret distribution. File distribution could be handled via data only containers or (my preferred way) the newly introduced git volumes #1945 .
For secrets it is a bit more difficult, but while talking on IRC there were a few solutions (I will get into in more detail in my own not hijacked issue) ranging from ldap to key rotation and encrypted data only containers.
The "consent" was that using an ENV for decryption secret and one ENV for encryption true/false would be the starting point. The data only container could then be encrypted on build using moby/moby#8021 for example. This would lead to a non sensitive images being stored in the usual private registry. At runtime it would be decrypted with the secret. In dev environments the encryption flag could be set to false and therefore make it easy to test with dev ssl certs/keys whatever.

That happened to be longer than I thought, but better start the discussion now and get more streamlined thoughts into the actual guideline issues.

[bketelsen]

for an unrelated project, we released crypt [1] this weekend, which uses openpgp keys to encrypt configuration files and store them in etcd or consul. Maybe over kill here, but another option.

[1] https://github.com/xordataexchange/crypt

[stp-ip]

Interesting, not sure if it's perfectly easy to integrate, but encrypted containers and ldap wouldn't be either I assume. Would you favour crypt over encrypted data only containers?
With data only containers you have the luxury to have versioning, with crypt and storing this in etcd you don't have that and you are not as free in your setup as you could be I feel.

[bketelsen]

I am using crypt in my k8s setup, but you're correct, it's not the easiest thing to use. We have it set up to encrpyt in our CI environment. It also requires that you either keep your private key in a docker container or on the docker host. Not the best setting for everyone, but it is very flexible. Using it in CI allows me to have multiple keys (one for dev, production, etc).

[bketelsen]

crypt does have a benefit, it can be used from a shell outside the docker container to inject ENV vars (with some bash scripting) or can be used as a library inside your app.

[stp-ip]

How much overhead does it introduce for decryption in a docker container? Is there any filesize/number limit? Any chance this is something possible to be proposed as secret distribution solution in k8s itself?
The crypt library needs access to etcd right? So without ACL right now in etcd, one would need to basically encrypt most data in etcd so that one container using etcd can't use it maliciousily.
Do you reuse keys or would a one time key be possible? It seems key rotation is still a missing piece.

[bketelsen]

the encryption and decryption is very quick, not noticeable. Any limitations would be imposed by etcd or consul, in terms of maximum key size, or maximum amount of data that could be POSTed.

If kubernetes were to adopt something like this it would make more sense to bake it in completely and have every k8s namespace get its own keypair. I'm happy to help think through the idea.

[bgrant0607]

@stp-ip Good discussion, but please file a new issue for secret distribution, since that has additional requirements.

[stp-ip]

@bketelsen What about library size added to each image decrypting secrets? Any huge overhead there?

@bgrant0607 Was not my intention to completely hijack. Will get a few issues done soon, which will be more suited for general and pinpointed discussion.

[thockin]

ENV vars are such a hack. Can't we do better? Can't we offer managed
volumes that hold config state in files, distribute that config state
somehow, and poke apps when state changes? We have internal systems like
this and I find it much more elegant than rebooting an app just to get a
new env var.

On Wed, Oct 22, 2014 at 8:24 AM, Michael Grosser [email protected]
wrote:

@bketelsen https://github.com/bketelsen What about library size added
to each image decrypting secrets? Any huge overhead there?

@bgrant0607 https://github.com/bgrant0607 Was not my intention to
completely hijack. Will get a few issues done soon, which will be more
suited for general and pinpointed discussion.

Reply to this email directly or view it on GitHub
#1553 (comment)
.

[rboyer]

Would the managed volumes be persisted to disk on the node or dropped into a tmpfs mount? I can see GC benefits to the tmpfs route, but on a hard node restart none of the pods could come back up without phoning home for configs.

[bgrant0607]

@thockin We also have internal systems akin to the env. vars. / etcd approach. There likely won't be just one way to do this. People writing their applications from scratch could even talk directly to etcd. I view env. vars. as a bridge to legacy apps, or at least a familiar mechanism for people who are either new to the approach and/or have simple use cases.

Look on the bright side: it's better than changing command-line flags. :-)

[pmorie]

@bgrant0607 @thockin @stp-ip A couple of thoughts:

1. I think there will be a lot of legacy apps that require env vars
2. It should be possible to consume parameters in many different ways:
3. Env vars
4. Files from shared EmptyDir volume in sidecar container
5. Other volume types
6. There should definitely be a way for pods to be notified when parameters change, either by having kubernetes restart them (especially in the case of env vars), or a notification file becoming present in a volume (personally I like this a lot less than kubernetes just restarting the pod)

[stp-ip]

@pmorie

1. yes
2. yes
3. Totally agreed. Restarting is way more plausible and logical.

[pmorie]

One mechanism for notification might be if the kubelet did an http POST to an endpoint in the pod that would trigger a configuration reload.

[davidopp]

Or have Kubelet send a Unix signal like SIGTERM, which can be caught by apps that are able to re-read their config without restarting, or not caught by apps that want to restart when their config changes.

[pmorie]

@davidopp Also a good idea. Summarizing possible reload mechanisms:

1. Restart pod
2. Create file with unique name in known location
3. Post to an endpoint on the pod
4. Have kubelet send unix signal to containers in pod
5. No notification (have an unspecified other component take action)

I'm not sure I really love the idea of having the kubelet restart the pod on a config change; it seems like restarts should be part of a higher level mechanism. That said my context is undoubtedly colored by working on OpenShift.

Update:

Added 5 above after thinking through my last sentence above. For context, I'm thinking of a use-case for OpenShift, where a user would want update to config to result in a new deployment.

[thockin]

How much of this needs to be done by Kubelet vs a side-car container. I'm
much happier with the latter model.

On Mon, Jan 19, 2015 at 1:10 PM, Paul Morie [email protected]
wrote:

@davidopp https://github.com/davidopp Also a good idea. Summarizing
possible reload mechanisms:

1. Restart pod
2. Create file with unique name in known location
3. Post to an endpoint on the pod
4. Have kubelet send unix signal to containers in pod

I'm not sure I really love the idea of having the kubelet restart the pod
on a config change; it seems like restarts should be part of a higher level
mechanism. That said my context is undoubtedly colored by working on
OpenShift.

Reply to this email directly or view it on GitHub
#1553 (comment)
.

[erictune]

@thockin you have mentioned side-cars a couple of times. Is there a general issue talking about whether we should use them? I'm not sold on them yet, versus more narrowly-tailored mechanisms with more specific error handling, etc.

[stefwalter]

Further work on this done here: #10982

[bgrant0607]

cc @kelseyhightower

[bgrant0607]

Putting a link here so I remember to take a look: https://github.com/kelseyhightower/confd

[bketelsen]

@kelseyhightower and I also wrote crypt[1], which might be interesting in that it uses etcd/consul as a backend but allows configurations to be encrypted.

[1] https://github.com/xordataexchange/crypt

[lavalamp]

Can someone summarize the action items remaining on this issue?

[bgrant0607]

Closing in favor of #18455