Key Rotation breaks appending new values to existing sealed secrets

[chrisharm]

With new new key rotation feature introduced in #143, the ability to append new values to an existing sealed secret is broken. Each time the controller restarts, a new key is generated and set as the active key. It looks like the unseal process expects all of the keys in a SealedSecret object to be encrypted using the same key.

Steps to Reproduce:

1. Encrypt foo=bar:
   kubectl create secret generic mysecret --dry-run --from-literal=foo=bar -o yaml | kubeseal --format yaml

Output:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
spec:
  encryptedData:
    foo: AgDSe+W6X+6Tk5WfwmMPFrlspMNOypWGY7uNX2JoRlC5/lGieYlUm0LTuFmTXZXwC+YzpjixHDjTkLGIgXw2uQj/BFZAae6rTrP78auQKZWA6FvQAdfPrX7PL9vNCJrlHLCcMf+03giJp5N9fDnjJf9Apxof+YFU6n4dy7EBn0AhIBJ0Zk7cJ30BHT4qUdrNf9W0u9ME4CmeaR7kwj7iGBlstNvxvd8yGO9g608jhclPzkugG30BvbkzeKqyuDTH0qUlFf5aGCZtHI6zZhz0GOLLkKLwZEHM/BkOsydm4jYJPXFatU5uFwXpsof+PfqWSRuhH/gqDcfYNjRPXR2CuFC7Tiy7WOdPTqINmw1cu26yHR6wsDRY+6o7kjHdfn9zB9UGhWBBiItU9vMuwhvte4yBkAvMVH67CYUUAgduCHpE870mVupgS1xyze2gVOTxpoG4x3OmGrGThHWQwnSIHcgB8dJKXiFudOT8B+U6PqOjOPpyyGjnC0WeM99Ets56TUNIRpQ7M1m5clp7Zz+8MltqdO4KgHph6H2hNfN0tLwohhcodZE+qhjYSSzDqA/3sjU/xUASBJfQH8in4WhRi0hk9toWjBb0pMex1eeYTNcM/LMvsOjRnFUqqQy/GpBXx4B78HEZ/tNBXwR69Jdw5E2B2B1eUFRu1eYD9lviGcZjhzCWDu1W3Q0NMqImYZqIRPYf6T0=

2. Restart sealed-secret-controller pod:
   kubectl delete pod --namespace kube-system --selector=name=sealed-secrets-controller

3. Generate a new secret for foo2=bar2
   kubectl create secret generic mysecret --dry-run --from-literal=foo2=bar2 -o yaml | kubeseal --format yaml

Output:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
spec:
  encryptedData:
    foo2: AgBuE1IsRoIJwjuLIp0w15Y1016ZU7KMgptdpgObNGJhmvPw8b6zshBhJ8N48eP/M6FHLPFhqPMj9ZR14tgA2aCjufLc6xBfAoQ9OGVZqepMQWI4jkjcJGtSlEZKy0FMpEaR0oF+cr/YCa7aiWBbdhjT+nBQLPxStH7L1RYJ5gQt/9tGiyCEJes8QcDLh5Fa78WLkMnaxcLO2JvhAh84vpdVea8dqA5U3l0bol0cN5pTZe+21xQZw6LXrvNdy+sGoyT0ytoOJx6xZdQQjz+EWUb+LgfLD2ddGY4SSHxUOTgZ9t3UvCpx6R4I2Gfe9nO2X7irvjZ2yEuBYMTxanGrlAoWlGXmVmC5U8t2BtVdEhnJ0BQsT2JbaTmXuiiggWzz6teyOKIne8Cxp5tojW+yNxEFO0Z0nXqnbVWpXoXy3dRExlAyHUYHr1NZ/b9lQpBuO64N+3KP/xWaM0RiB7brA46vAiCdgdKIGIxPcDUEPEuaw2yEN8mTt34nixGsIW6VWCJy47TAYoS3JZt9kjqggVbHdim0/f3O+ghGsg1X5FrXjyD4cfq8SIWXLFmqpiHz6zCng81Dl/atWDkRJJ8J+jn8SNugfikxCPCWuLsA2u5wf+k1U+kFKu3FGMFxjGA74HAP/9P3qbq63NbmCgknP+M5cyWZTCtLhRlIySPe3gxdbBdfysA8x5Pvqjt8oyyAyGlu0juT

4. Append foo2 to original secret:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
spec:
  encryptedData:
    foo: AgDSe+W6X+6Tk5WfwmMPFrlspMNOypWGY7uNX2JoRlC5/lGieYlUm0LTuFmTXZXwC+YzpjixHDjTkLGIgXw2uQj/BFZAae6rTrP78auQKZWA6FvQAdfPrX7PL9vNCJrlHLCcMf+03giJp5N9fDnjJf9Apxof+YFU6n4dy7EBn0AhIBJ0Zk7cJ30BHT4qUdrNf9W0u9ME4CmeaR7kwj7iGBlstNvxvd8yGO9g608jhclPzkugG30BvbkzeKqyuDTH0qUlFf5aGCZtHI6zZhz0GOLLkKLwZEHM/BkOsydm4jYJPXFatU5uFwXpsof+PfqWSRuhH/gqDcfYNjRPXR2CuFC7Tiy7WOdPTqINmw1cu26yHR6wsDRY+6o7kjHdfn9zB9UGhWBBiItU9vMuwhvte4yBkAvMVH67CYUUAgduCHpE870mVupgS1xyze2gVOTxpoG4x3OmGrGThHWQwnSIHcgB8dJKXiFudOT8B+U6PqOjOPpyyGjnC0WeM99Ets56TUNIRpQ7M1m5clp7Zz+8MltqdO4KgHph6H2hNfN0tLwohhcodZE+qhjYSSzDqA/3sjU/xUASBJfQH8in4WhRi0hk9toWjBb0pMex1eeYTNcM/LMvsOjRnFUqqQy/GpBXx4B78HEZ/tNBXwR69Jdw5E2B2B1eUFRu1eYD9lviGcZjhzCWDu1W3Q0NMqImYZqIRPYf6T0=
    foo2: AgBuE1IsRoIJwjuLIp0w15Y1016ZU7KMgptdpgObNGJhmvPw8b6zshBhJ8N48eP/M6FHLPFhqPMj9ZR14tgA2aCjufLc6xBfAoQ9OGVZqepMQWI4jkjcJGtSlEZKy0FMpEaR0oF+cr/YCa7aiWBbdhjT+nBQLPxStH7L1RYJ5gQt/9tGiyCEJes8QcDLh5Fa78WLkMnaxcLO2JvhAh84vpdVea8dqA5U3l0bol0cN5pTZe+21xQZw6LXrvNdy+sGoyT0ytoOJx6xZdQQjz+EWUb+LgfLD2ddGY4SSHxUOTgZ9t3UvCpx6R4I2Gfe9nO2X7irvjZ2yEuBYMTxanGrlAoWlGXmVmC5U8t2BtVdEhnJ0BQsT2JbaTmXuiiggWzz6teyOKIne8Cxp5tojW+yNxEFO0Z0nXqnbVWpXoXy3dRExlAyHUYHr1NZ/b9lQpBuO64N+3KP/xWaM0RiB7brA46vAiCdgdKIGIxPcDUEPEuaw2yEN8mTt34nixGsIW6VWCJy47TAYoS3JZt9kjqggVbHdim0/f3O+ghGsg1X5FrXjyD4cfq8SIWXLFmqpiHz6zCng81Dl/atWDkRJJ8J+jn8SNugfikxCPCWuLsA2u5wf+k1U+kFKu3FGMFxjGA74HAP/9P3qbq63NbmCgknP+M5cyWZTCtLhRlIySPe3gxdbBdfysA8x5Pvqjt8oyyAyGlu0juT

5. Apply the sealed secret to the cluster
   kubectl apply -f /tmp/ss.yaml

Result:

2019/07/16 16:47:23 Error updating default/new-keys-ss, giving up: No key could decrypt secret
E0716 16:47:23.374767       1 controller.go:191] No key could decrypt secret
2019/07/16 16:47:23 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"new-keys-ss", UID:"ce8e2aa5-9887-4581-a796-c9d610be5b96", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"175933", FieldPath:""}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: No key could decrypt secret

[chrisharm]

See #162 and #95

[grossvogel]

I'm sure many are far ahead of me on this, but I think a good solution to this would be to store a unique key id along with each encrypted value in the SealedSecret. That way values encrypted by different keys could co-exist on the same SealedSecret and git diffs would be clean, indicating exactly which data fields are added or updated. I think this is similar to how openid connect handles key rotation.

[mkmik]

I also think that's an efficient approach. I'd just like to make sure it's done in a backward compatible way.

Currently all keys are tried in turn until one succeeds. This already sucks when there are many keys and sucks even more if the whole list has to be iterated for every item of a secret

[mkmik]

Fixed in #216 but in a rather inefficient way.
We'll deal with the efficiency issue in #226.