Support secret templating

[nimjor]

Is your feature request related to a problem? Please describe.
To really be practical and cover most potential use cases, the VaultStaticSecret and VaultDynamicSecret types should allow templating based on the values retrieved from Vault.

Describe the solution you'd like
We should be able to use Go templating or Consul Template, similar to what can be done using the Vault agent annotations. For example, I want to be able to use the Artifactory Vault dynamic secrets plugin/engine to generate temporary Artifactory credentials that can be injected into a kubernetes.io/dockerconfigjson secret to use for container image pull secrets.

Describe alternatives you've considered
I can already do as described above using external-secrets.io

Additional context
How this is done with external-secrets:

---
apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
metadata:
  name: vault-dynamic
  namespace: demo
spec:
  path: "/artifactory/token/demo"
  method: "GET"
  provider:
    server: "https://vault-dev.tools.demo.com"
    auth:
      kubernetes:
        mountPath: "k8s-platform"
        role: "demo-tools-access-read"
        serviceAccountRef:
          name: "demo"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: image-pull-dynamic
  namespace: demo
spec:
  refreshInterval: "3600s"
  target:
    name: image-pull-dynamic
    template:
      type: kubernetes.io/dockerconfigjson
      engineVersion: v2
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "containers.artifactory-dev.tools.demo.com": {
                "auth": "{{ list .username ":" .access_token | join "" | b64enc }}",
                "email": "[email protected]",
                "password": "{{ .access_token }}",
                "username": "{{ .username }}"
              }
            }
          }
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: VaultDynamicSecret
        name: vault-dynamic

[benashz]

Thanks for your valuable feedback @nimjor! Adding template support is definitely a feature we are considering in the near future.

[TheLonelyGhost]

Whoa! I didn't realize ExternalSecret resources could do that! That's awesome! Seems to add support to the proposal from #120

[linuxbsdfreak]

This feature is already present under https://github.com/ricoberger/vault-secrets-operator#using-templated-secrets

I have been using the above operator and it works really well. Would be nice to have this feature.

[kennedn]

Seconding this. Until this operator has templating, it's use cases are going to be severely narrowed.

[BOsterbuhr]

Hey @benashz, do you have any updates to share? Specifically around VSO supporting .dockerconfigjson secrets in the near future?

[kschoche]

Hi @BOsterbuhr - adding templating support is on our upcoming roadmap, we will have more updates on that coming soon. Right now we support the built-in types that Kubernetes automatically templates.

As far as supporting .dockerconfigjson, I believe this should already work by setting the spec.destination.type field to kubernetes.io/dockerconfigjson?

Here is an example which uses a KV secret to do the same, I've been meaning to put this into a public document.. if it doesnt work feel free to open a separate issue because it should:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: awesomeapps
  name: foo
spec:
  type: kv-v2
  mount: kvv2
  path: docker/config
  # dest k8s secret
  destination:
    name: myregistryKey  # <-- this is your imagePullSecret name
    create: true
    type: "kubernetes.io/dockerconfigjson"
---
apiVersion: v1
kind: Pod
metadata:
  name: foo
  namespace: awesomeapps
spec:
  containers:
    - name: foo
      image: janedoe/awesomeapp:v1
  imagePullSecrets:
   - name: myregistrykey

# Example vault command to load the dockerconfig into a kv:
 vault kv put kvv2/docker/config .dockerconfigjson=`cat ~/.docker/config.json`

[BOsterbuhr]

Hey @kschoche! Our team has verified that this works - thank you so much!
We found that this functionality wasn't immediately clear from the documentation, so we think sharing a public post about it could be really helpful.
Thanks again for your prompt response - we appreciate it!

[AdrienneCohea]

Awesome that we can do it through External Secrets.

But it would be cool if the VSO could natively do it. I also dropped a plug in the Vault CSI Provider that the Vault functions of consul-template should be available in here. Valid use cases would be things like creating a file formatted in a particular way for secrets. For example, pgpass.

[TheLonelyGhost]

# Example vault command to load the dockerconfig into a kv:
 vault kv put kvv2/docker/config .dockerconfigjson=`cat ~/.docker/config.json`

Excellent that there's a workaround for this case, however this wouldn't be a workable solution for me.

I've found it a lot easier to maintain secrets by only including the credentials and not the formatting in the KV store. In other words, just username and password in the KV secret, then use Vault Agent (or VSO, External Secrets, ...) to format it for the specific needs of that use case (e.g., dockerconfig).

This approach is especially applicable if the same credential is needed in multiple formats, such as with Artifactory supporting auth for Maven, Docker, pip, and other protocols. For now, we'll need to stick with our Vault Agent approach but consider this a +1 for templating in VSO.

[TJM]

We are also using vault agent approach, using my "vault-gcp-secrets" helm chart (which is just vault agent plus kubectl). I am going to have to try your template formatting above to maybe remove the need to run "kubectl" to format the dockerconfigjson secret. GCP puts an entire JSON formatted file as the password field, and I didn't want to try to figure out how to get all the proper escaping in. So I just use "template_command" like:

    kubectl create secret docker-registry "{{ .Values.secret.name }}" \
      --docker-server "{{ .Values.secret.dockerServer }}" \
      --docker-username "{{ .Values.secret.dockerUsername }}" \
      --docker-email "{{ .Values.secret.dockerEmail }}" \
      --docker-password "$(cat /dev/shm/key.json)" \
      --dry-run=client -o yaml | kubectl replace -f -

... with all the non-secret values coming from the helm chart

[freimer]

What is the status of this enhancement? Any timeline for when this will be completed?

[raj-saxena]

What is the status of this enhancement? Any timeline for when this will be completed?

@freimer from the open PR, seems like we should have the changes by end of Feb.

[imranzunzani]

Hi @BOsterbuhr - adding templating support is on our upcoming roadmap, we will have more updates on that coming soon. Right now we support the built-in types that Kubernetes automatically templates.

As far as supporting .dockerconfigjson, I believe this should already work by setting the spec.destination.type field to kubernetes.io/dockerconfigjson?

Here is an example which uses a KV secret to do the same, I've been meaning to put this into a public document.. if it doesnt work feel free to open a separate issue because it should:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: awesomeapps
  name: foo
spec:
  type: kv-v2
  mount: kvv2
  path: docker/config
  # dest k8s secret
  destination:
    name: myregistryKey  # <-- this is your imagePullSecret name
    create: true
    type: "kubernetes.io/dockerconfigjson"
---
apiVersion: v1
kind: Pod
metadata:
  name: foo
  namespace: awesomeapps
spec:
  containers:
    - name: foo
      image: janedoe/awesomeapp:v1
  imagePullSecrets:
   - name: myregistrykey

# Example vault command to load the dockerconfig into a kv:
 vault kv put kvv2/docker/config .dockerconfigjson=`cat ~/.docker/config.json`

As we need to rotate the password periodically, this doesn't work out when trying to make things work with https://github.com/hashicorp/vault-action and referencing just the password using the plain kv format. You need to fetch the entire config and go through it. We are thus looking into using https://external-secrets.io/latest/ which has templating support. Could you (@benashz) please let me know, if the templating feature will be getting released anytime soon, so we wait?

[benashz]

Hi @imranzunzani, thank you for your interest in VSO. We are currently planning to release the secret transformation support (templating) in v0.5.0. We are targeting end of February as a release date.

[benashz]

Added some VSO secret transformation examples that would be compatible for the K8s secret type kubernetes.io/dockerconfigjson here: #619 (comment) and here #619 (comment)