Vault secret backend

docs/kap_proposals/kap_6_hashicorp_vault.md

@@ -0,0 +1,131 @@
+# Hashicorp Vault  
+
+This feature allows the user to fetch secrets from [Hashicorp Vault](https://www.vaultproject.io/), with the new secret backend keyword 'vaultkv'.  
+
+Author: [@vaibahvk](https://github.com/vaibhavk) [@daminisatya](https://github.com/daminisatya)  
+## Specification  
+
+The following variables need to be exported to the environment(depending on authentication used) where you will run `kapitan secrets --reveal` in order to authenticate to your HashiCorp Vault instance:  
+* VAULT_ADDR: URL for vault  
+* VAULT_SKIP_VERIFY=true: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing  
+* VAULT_TOKEN: token for vault or file (~/.vault-tokens)  
+* VAULT_ROLE_ID: required by approle  
+* VAULT_SECRET_ID: required by approle  
+* VAULT_USERNAME: username to login to vault  
+* VAULT_PASSWORD: password to login to vault  
+* VAULT_CLIENT_KEY: the path to an unencrypted PEM-encoded private key matching the client certificate  
+* VAULT_CLIENT_CERT: the path to a PEM-encoded client certificate for TLS authentication to the Vault server  
+* VAULT_CACERT: the path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate  
+* VAULT_CAPATH: the path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate  
+* VAULT_NAMESPACE: specify the Vault Namespace, if you have one  
+
+Considering a key-value pair like `my_key`:`my_secret` ( in our case let’s store `hello`:`batman` inside the vault ) in the path `secret/foo` in a kv-v2(KV version 2) secret engine on the vault server, to use this as a secret either follow:  
+
+```shell  
+$ echo "foo:hello" > somefile.txt  
+$ kapitan secrets --write vaultkv:path/to/secret_inside_kapitan --file somefile.txt --target dev-sea  
+```  
+or in a single line  
+```shell  
+$ echo "foo:hello"  | kapitan secrets --write vaultkv:path/to/secret_inside_kapitan -t dev-sea -f -  
+```  
+The entire string __"foo:hello"__ is base64 encoded and stored in the secret_inside_kapitan. Now secret_inside_kapitan contains the following  
+
+```yaml    
+data: Zm9vOmhlbGxvCg==  
+encoding: original  
+type: vaultkv  
+vault_params:  
+  auth: token
+```  
+
+Encoding tells the type of data given to kapitan, if it is `original` then after decoding base64 we'll get the original secret and if it is `base64` then after decoding once we still have a base64 encoded secret and have to decode again.  
+Parameters in the secret file are collected from the inventory of the target we gave from CLI `--target dev-sea`. If target isn't provided then kapitan will identify the variables from the environment, but providing `auth` is necessary as a key inside target parameters like the one shown:  
+```yaml  
+parameters:
+  kapitan:
+    secrets:
+      vaultkv:
+        auth: userpass
+        engine: kv-v2
+        mount: team-alpha-secret
+        VAULT_ADDR: http://127.0.0.1:8200
+        VAULT_NAMESPACE: CICD-alpha
+        VAULT_SKIP_VERIFY: false
+        VAULT_CLIENT_KEY: /path/to/key
+        VAULT_CLIENT_CERT: /path/to/cert
+```
+Environment variables that can be defined in kapitan inventory are `VAULT_ADDR`, `VAULT_NAMESPACE`, `VAULT_SKIP_VERIFY`, `VAULT_CLIENT_CERT`, `VAULT_CLIENT_KEY`, `VAULT_CAPATH` & `VAULT_CACERT`.  
+Extra parameters that can be defined in inventory are:  
+* `auth`: specify which authentication method to use like `token`,`userpass`,`ldap`,`github` & `approle`  
+* `mount`: specify the mount point of key's path. e.g if path=`alpha-secret/foo/bar` then `mount: alpha-secret` (default `secret`)  
+* `engine`: secret engine used, either `kv-v2` or `kv` (default `kv-v2`)  
+Environment variables cannot be defined in inventory are `VAULT_TOKEN`,`VAULT_USERNAME`,`VAULT_PASSWORD`,`VAULT_ROLE_ID`,` VAULT_SECRET_ID`.  
+This makes the secret_inside_kapitan file accessible throughout the inventory, where we can use the secret whenever necessary like `?{vaultkv:path/to/secret_inside_kapitan}`  
+
+Following is the example file having a secret and pointing to the vault `?{vaultkv:path/to/secret_inside_kapitan}`  
+
+```yaml
+parameters:
+  releases:
+    cod: latest
+  cod:
+    image: alledm/cod:${cod:release}
+    release: ${releases:cod}
+    replicas: ${replicas}
+    args:
+      - --verbose=${verbose}
+      - --password=?{vaultkv:path/to/secret_inside_kapitan}
+```  
+when `?{vaultkv:path/to/secret_inside_kapitan}` is compiled, it will look same with an 8 character prefix of sha256 hash added at the end like:  
+```yaml  
+kind: Deployment
+metadata:
+  name: cod
+  namespace: dev-sea
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: cod
+    spec:
+      containers:
+        - args:
+            - --verbose=True
+            - --password=?{vaultkv:path/to/secret_inside_kapitan:57d6f9b7}
+          image: alledm/cod:v2.0.0
+          name: cod
+``` 
+
+Only the user with the required tokens/permissions can reveal the secrets. Please note that the roles and permissions will be handled at the Vault level. We need not worry about it within Kapitan. Use the following command to reveal the secrets:  
+
+```shell  
+$ kapitan secrets --reveal -f compile/file/containing/secret 
+```  
+
+Following is the result of the cod-deployment.md file after Kapitan reveal.
+
+```yaml  
+kind: Deployment
+metadata:
+  name: cod
+  namespace: dev-sea
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: cod
+    spec:
+      containers:
+        - args:
+            - --verbose=True
+            - --password=batman
+          image: alledm/cod:v2.0.0
+          name: cod
+```  
+
+## Dependencies  
+
+- [hvac](https://github.com/hvac/hvac) is a python client for Hashicorp Vault

docs/secrets.md

@@ -5,15 +5,15 @@ Kapitan can manage secrets with the following key management services:
 - GPG
 - Google Cloud KMS (beta)
 - AWS KMS (beta)
-- Vault (TBD)
+- Vaultkv (read only support)
 
 If you want to get started with secrets but don't have a GPG or KMS setup, you can also use the secret `ref` type. Note that `ref` is not encrypted and is intended for development purposes only. *Do not use ref secrets if you're storing sensitive information!*
 
 ## Using secrets
 
 The usual flow of creating and using an encrypted secret with kapitan is:
 
-#### 1. Define your GPG recipients or KMS key 
+#### 1. Define your GPG recipients, Vault client parameters or KMS key 
 
 This is done in the inventory under `parameters.kapitan.secrets`. 
 
@@ -34,6 +34,9 @@ parameters:
         key: 'projects/<project>/locations/<location>/keyRings/<keyRing>/cryptoKeys/<key>'
       awskms:
         key: 'alias/nameOfKey'
+      vaultkv:
+        VAULT_ADDR: http://127.0.0.1:8200
+        auth: token
 ```
 
 #### 2. Create your secret
@@ -50,7 +53,7 @@ $ kapitan secrets --write <secret_type>:path/to/secret/file -t <target_name> -f
 - `gpg`: GPG
 - `gkms`: Google Cloud KMS
 - `awskms`: AWS KMS
-- `vault` (TBC)
+- `vaultkv`: Hashicorp Vault with kv/kv-v2 secret engine
 
 Kapitan will inherit the secrets configuration for the specified target, and encrypt and save your secret into `<path/to/secret/file>`.
 
@@ -68,6 +71,7 @@ rsapublic - Derives an RSA public key from a revealed private key i.e. `|reveal:
 ```
 
 *Note*: If you use `|reveal:/path/secret`, when changing the `/path/secret` file make sure you also delete any secrets referencing `/path/secret` so kapitan can regenerate them.
+*Note*: `vaultkv` can't be used to generate secrets automatically for now, manually create the secret using the command line.
 
 #### 3. Reference your secrets in your classes/targets and run `kapitan compile`
 
@@ -135,4 +139,37 @@ $ kapitan secrets --write gpg:components/secrets/mysql_secrets -t prod -f secret
 
 To reference `secret_foo`inside this file, you can specify it in the inventory as follows:
 
-`secret_foo: ${gpg:components/secrets/mysql_secrets@mysql_passwords.secret_foo}`
+`secret_foo: ${gpg:components/secrets/mysql_secrets@mysql_passwords.secret_foo}`
+
+### Vaultkv Secret Backend (Read Only) - Addons
+
+Considering a key-value pair like `my_key`:`my_secret` in the path `secret/foo/bar` in a kv-v2(KV version 2) secret engine on the vault server, to use this as a secret use:  
+
+```shell  
+$ echo "foo/bar:my_key"  | kapitan secrets --write vaultkv:path/to/secret_inside_kapitan -t <target_name> -f -  
+```  
+
+Parameters in the secret file are collected from the inventory of the target we gave from CLI `-t <target_name>`. If target isn't provided then kapitan will identify the variables from the environment when revealing secret.
+
+Environment variables that can be defined in kapitan inventory are `VAULT_ADDR`, `VAULT_NAMESPACE`, `VAULT_SKIP_VERIFY`, `VAULT_CLIENT_CERT`, `VAULT_CLIENT_KEY`, `VAULT_CAPATH` & `VAULT_CACERT`.  
+Extra parameters that can be defined in inventory are:  
+* `auth`: specify which authentication method to use like `token`,`userpass`,`ldap`,`github` & `approle`  
+* `mount`: specify the mount point of key's path. e.g if path=`alpha-secret/foo/bar` then `mount: alpha-secret` (default `secret`)  
+* `engine`: secret engine used, either `kv-v2` or `kv` (default `kv-v2`)  
+Environment variables cannot be defined in inventory are `VAULT_TOKEN`,`VAULT_USERNAME`,`VAULT_PASSWORD`,`VAULT_ROLE_ID`,` VAULT_SECRET_ID`.  
+
+```yaml  
+parameters:
+  kapitan:
+    secrets:
+      vaultkv:
+        auth: userpass
+        engine: kv-v2
+        mount: team-alpha-secret
+        VAULT_ADDR: http://127.0.0.1:8200
+        VAULT_NAMESPACE: CICD-alpha
+        VAULT_SKIP_VERIFY: false
+        VAULT_CLIENT_KEY: /path/to/key
+        VAULT_CLIENT_CERT: /path/to/cert
+```
+

examples/kubernetes/components/mysql/secret.jsonnet

@@ -18,4 +18,4 @@ local inv = kap.inventory();
       "MYSQL_ROOT_PASSWORD_SHA256": inv.parameters.mysql.users.root.password_sha256_subvar
     }
   }
-}
+}

examples/kubernetes/inventory/classes/component/mysql.yml

@@ -27,6 +27,10 @@ parameters:
         - manifests/mysql_service_simple.yml
       kind: service
       version: 1.14.0
+    # For vaultkv secrets it is important to declare auth in parameters
+    secrets:
+      vaultkv:
+        auth: token
   mysql:
     storage: 10G
     storage_class: standard
@@ -44,3 +48,6 @@ parameters:
 
         password_subvar: ?{gpg:targets/${target_name}/mysql/[email protected]}
         password_sha256_subvar: ?{gpg:targets/${target_name}/mysql/[email protected]_sha256}
+
+        # This secret requires a running vault instance & vaultkv parameters either in inventory or environment
+        # password_vaultkv: ?{vaultkv:targets/${target_name}/mysql/vault_secret}

kapitan/cli.py

@@ -34,6 +34,7 @@
 from kapitan.refs.base64 import Base64Ref
 from kapitan.refs.secrets.awskms import AWSKMSSecret
 from kapitan.refs.secrets.gkms import GoogleKMSSecret
+from kapitan.refs.secrets.vaultkv import VaultSecret
 from kapitan.refs.secrets.gpg import GPGSecret, lookup_fingerprints
 from kapitan.resources import (inventory_reclass, resource_callbacks,
                                search_imports)
@@ -201,6 +202,9 @@ def main():
     refs_parser.add_argument('--key', '-K', help='set KMS key',
                              default=from_dot_kapitan('refs', 'key', ''),
                              metavar='KEY')
+    refs_parser.add_argument('--vault-auth', help='set authentication type for vaultkv secrets',
+                             default=from_dot_kapitan('refs', 'vault-auth', ''),
+                             metavar='AUTH')
     refs_parser.add_argument('--refs-path', help='set refs path, default is "./refs"',
                              default=from_dot_kapitan('refs', 'refs-path', './refs'))
     refs_parser.add_argument('--verbose', '-v',
@@ -400,7 +404,7 @@ def ref_write(args, ref_controller):
             recipients = kap_inv_params['secrets']['gpg']['recipients']
         if not recipients:
             raise KapitanError("No GPG recipients specified. Use --recipients or specify them in " +
-                               "parameters.kapitan.secrets.gpg.recipients and use --target")
+                               "parameters.kapitan.secrets.gpg.recipients and use --target-name")
         secret_obj = GPGSecret(data, recipients, encode_base64=args.base64)
         tag = '?{{gpg:{}}}'.format(token_path)
         ref_controller[tag] = secret_obj
@@ -416,7 +420,7 @@ def ref_write(args, ref_controller):
 
             key = kap_inv_params['secrets']['gkms']['key']
         if not key:
-            raise KapitanError("No KMS key specified. Use --key or specify it in parameters.kapitan.secrets.gkms.key and use --target")
+            raise KapitanError("No KMS key specified. Use --key or specify it in parameters.kapitan.secrets.gkms.key and use --target-name")
         secret_obj = GoogleKMSSecret(data, key, encode_base64=args.base64)
         tag = '?{{gkms:{}}}'.format(token_path)
         ref_controller[tag] = secret_obj
@@ -432,7 +436,7 @@ def ref_write(args, ref_controller):
 
             key = kap_inv_params['secrets']['awskms']['key']
         if not key:
-            raise KapitanError("No KMS key specified. Use --key or specify it in parameters.kapitan.secrets.awskms.key and use --target")
+            raise KapitanError("No KMS key specified. Use --key or specify it in parameters.kapitan.secrets.awskms.key and use --target-name")
         secret_obj = AWSKMSSecret(data, key, encode_base64=args.base64)
         tag = '?{{awskms:{}}}'.format(token_path)
         ref_controller[tag] = secret_obj
@@ -449,6 +453,28 @@ def ref_write(args, ref_controller):
         tag = '?{{base64:{}}}'.format(token_path)
         ref_controller[tag] = ref_obj
 
+    elif token_name.startswith("vaultkv:"):
+        type_name, token_path = token_name.split(":")
+        _data = data.encode()
+        vault_params = {}
+        encoding = 'original'
+        if args.target_name:
+            inv = inventory_reclass(args.inventory_path)
+            kap_inv_params = inv['nodes'][args.target_name]['parameters']['kapitan']
+            if 'secrets' not in kap_inv_params:
+                raise KapitanError("parameters.kapitan.secrets not defined in {}".format(args.target_name))
+
+            vault_params = kap_inv_params['secrets']['vaultkv']
+        if args.vault_auth:
+            vault_params['auth'] = args.vault_auth
+        if vault_params.get('auth') is None:
+            raise KapitanError("No Authentication type parameter specified. Specify it"
+                               " in parameters.kapitan.secrets.vaultkv.auth and use --target-name or use --vault-auth")
+
+        secret_obj = VaultSecret(_data, vault_params)
+        tag = '?{{vaultkv:{}}}'.format(token_path)
+        ref_controller[tag] = secret_obj
+
     elif token_name.startswith("plain:"):
         type_name, token_path = token_name.split(":")
         _data = data.encode()
@@ -462,7 +488,7 @@ def ref_write(args, ref_controller):
         ref_controller[tag] = ref_obj
 
     else:
-        fatal_error("Invalid token: {name}. Try using gpg/gkms/awskms/base64/plain:{name}".format(name=token_name))
+        fatal_error("Invalid token: {name}. Try using gpg/gkms/awskms/vaultkv/base64/plain:{name}".format(name=token_name))
 
 
 def secret_update(args, ref_controller):
@@ -570,6 +596,10 @@ def secret_update_validate(args, ref_controller):
             awskey = kap_inv_params['secrets']['awskms']['key']
         except KeyError:
             awskey = None
+        try:
+            vaultkv = kap_inv_params['secrets']['vaultkv']['auth']
+        except KeyError:
+            vaultkv = None
 
         for token_path in token_paths:
             if token_path.startswith("?{gpg:"):
@@ -620,6 +650,12 @@ def secret_update_validate(args, ref_controller):
                         secret_obj.update_key(awskey)
                         ref_controller[token_path] = secret_obj
 
+            elif token_path.startswith("?{vaultkv:"):
+                if not vaultkv:
+                    logger.debug("secret_update_validate: target: %s has no inventory vaultkv parameter 'auth', skipping %s", target_name, token_path)
+                    continue
+                secret_obj = ref_controller[token_path]
+
             else:
                 logger.info("Invalid secret %s, could not get type, skipping", token_path)
                 ret_code = 1

kapitan/refs/base.py

@@ -248,7 +248,7 @@ def _reveal_replace_match(self, match_obj):
             revealed_yaml = yaml.load(plaintext, Loader=YamlLoader)
             if not isinstance(revealed_yaml, dict):
                 raise RefError("revealed secret is not in yaml, cannot access sub-variable"
-                               "at {}".format(subvar_path))
+                               " at {}".format(subvar_path))
             return self._get_value_in_yaml_path(revealed_yaml, subvar_path)
 
     @lru_cache(maxsize=256)
@@ -388,6 +388,9 @@ def _get_backend(self, type_name):
             elif type_name == 'awskms':
                 from kapitan.refs.secrets.awskms import AWSKMSBackend
                 self.register_backend(AWSKMSBackend(self.path))
+            elif type_name == 'vaultkv':
+                from kapitan.refs.secrets.vaultkv import VaultBackend
+                self.register_backend(VaultBackend(self.path))
             else:
                 raise RefBackendError('no backend for ref type: {}'.format(type_name))
         return self.backends[type_name]