Vault secret backend

docs/kap_proposals/kap_6_hashicorp_vault.md

@@ -0,0 +1,84 @@
+# Hashicorp Vault  
+
+This feature allows the user to fetch secrets from [Hashicorp Vault](https://www.vaultproject.io/), an online Kapitan Secret backend `vault` will be introduced.  
+
+Author: [@vaibahvk](https://github.com/vaibhavk) [@daminisatya](https://github.com/daminisatya)  
+## Specification  
+
+The following variables need to be exported to the environment where you run this script in order to authenticate to your HashiCorp Vault instance:  
+* VAULT_ADDR: URL for vault  
+* VAULT_SKIP_VERIFY=true: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing  
+* VAULT_AUTHTYPE: authentication type to use: token, userpass, GitHub, LDAP, approle  
+* VAULT_TOKEN: token for vault or file (~/.vault-tokens)  
+* VAULT_ROLE_ID: (required by approle)  
+* VAULT_SECRET_ID: (required by approle)  
+* VAULT_USER: username to login to vault  
+* VAULT_PASSWORD: password to login to vault  
+* VAULT_CLIENT_KEY: the path to an unencrypted PEM-encoded private key matching the client certificate  
+* VAULT_CLIENT_CERT: the path to a PEM-encoded client certificate for TLS authentication to the Vault server  
+* VAULT_CACERT: the path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate  
+* VAULT_CAPATH: the path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate  
+* VAULT_NAMESPACE: specify the Vault Namespace, if you have one  
+
+Considering a key-value pair like `my_key`:`my_secret` ( in our case let’s store hello:batman inside the vault ) in the path `secret/foo` on the vault server, to use this as a secret either follow:  
+
+```shell  
+$ echo “{'path':'secret/foo','key':'my_key'}” > somefile.txt  
+$ kapitan secrets —write vault:path/to/secret_inside_kapitan -f somefile.txt  
+```  
+or in a single line  
+```shell  
+$ echo “{'path':'secrt/foo','key':'my_key'}” | kapitan secrets --write vault:path/to/secret_inside_kapitan -f -  
+```  
+The entire string __{'path':'secret/foo','key':'my_key'}__ is base64 encoded and stored in the secret_inside_kapitan. Now secret_inside_kapitan contains the following  
+
+```  
+data: 4oCccGF0aDpzZWNlcnQvZm9v4oCdIOKAnGtleTpteV9rZXnigJ0K  
+encoding: original  
+type: vault  
+```  
+
+this makes the secret_inside_kapitan file accessible throughout the inventory, where we can use the secret whenever necessary like `?{vault:path/to/secret_inside_kapitan}`  
+
+Following is the example file having a secret and pointing to the vault `?{vault:path/to/secret_inside_kapitan}`  
+
+```  
+parameters:  
+  releases:  
+	  cod: latest  
+  cod:  
+	  image: ?{vault:path/to/secret_inside_kapitan}  
+	  release: ${releases:cod}  
+	  replicas: ${replicas}  
+	  args:  
+	  - --verbose=${verbose}  
+```  
+
+when `?{vault:path/to/secret_inside_kapitan}` is compiled, it will look same with an 8 character prefix of sha256 hash added at the end like:  
+```
+# Welcome to the README!
+Target *dev-sea* is running:
+-   1 replicas of *cod* running image ?{vault:targets/secret_inside_kapitan:de0e6a80}
+-   on cluster kubernetes
+``` 
+
+Only the user with the required tokens/permissions can reveal the secrets. Please note that the roles and permissions will be handled at the Vault level. We need not worry about it within Kapitan. Using the command:  
+
+```shell  
+$ kapitan secrets --reveal -f compile/file/containing/secret  
+```  
+
+Following is the result of the cod.md file after Kapitan reveal.
+
+```  
+# Welcome to the README!  
+Target *dev-sea* is running:  
+
+-   1 replicas of *cod* running image my_secret
+-   on cluster kubernetes
+
+```  
+
+## Dependencies  
+
+- [hvac](https://github.com/hvac/hvac) is a python client for Hashicorp Vault

kapitan/cli.py

@@ -32,6 +32,7 @@
 from kapitan.refs.base import Ref, RefController, Revealer
 from kapitan.refs.secrets.awskms import AWSKMSSecret
 from kapitan.refs.secrets.gkms import GoogleKMSSecret
+from kapitan.refs.secrets.vault import VaultSecret
 from kapitan.refs.secrets.gpg import GPGSecret, lookup_fingerprints
 from kapitan.resources import (inventory_reclass, resource_callbacks,
                                search_imports)
@@ -398,6 +399,15 @@ def secret_write(args, ref_controller):
         tag = '?{{ref:{}}}'.format(token_path)
         ref_controller[tag] = ref_obj
 
+    elif token_name.startswith("vault:"):
+        type_name, token_path = token_name.split(":")
+        encoding = False
+        if args.base64:
+            encoding = True
+        secret_obj = VaultSecret(data, encode_base64=encoding)
+        tag = '?{{vault:{}}}'.format(token_path)
+        ref_controller[tag] = secret_obj
+
     else:
         fatal_error("Invalid token: {name}. Try using gpg/gkms/awskms/ref:{name}".format(name=token_name))
 

kapitan/refs/base.py

@@ -356,6 +356,9 @@ def _get_backend(self, type_name):
             elif type_name == 'awskms':
                 from kapitan.refs.secrets.awskms import AWSKMSBackend
                 self.register_backend(AWSKMSBackend(self.path))
+            elif type_name == 'vault':
+                from kapitan.refs.secrets.vault import VaultBackend
+                self.register_backend(VaultBackend(self.path))
             else:
                 raise RefBackendError('no backend for ref type: {}'.format(type_name))
         return self.backends[type_name]

kapitan/refs/secrets/vault.py

@@ -0,0 +1,167 @@
+# Copyright 2019 The Kapitan Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"hashicorp vault secrets module"
+
+import hvac
+import base64
+from hvac.exceptions import Forbidden, VaultError
+from yaml import safe_load
+from os.path import join,expanduser
+from os import getenv
+from sys import argv
+
+from kapitan.refs.base import Ref, RefBackend, RefError
+from kapitan import cached
+from kapitan.errors import KapitanError
+
+class VaultError(KapitanError):
+    """Generic vault errors"""
+    pass
+
+def get_env():
+    """
+    The following variables need to be exported to the environment where you run this script in order to authenticate to your HashiCorp Vault instance:
+    * VAULT_ADDR: url for vault
+    * VAULT_SKIP_VERIFY=true: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing
+    * VAULT_AUTHTYPE: authentication type to use: token, userpass, github, ldap, approle
+    * VAULT_TOKEN: token for vault
+    * VAULT_ROLE_ID: (required by approle)
+    * VAULT_SECRET_ID: (required by approle)
+    * VAULT_USER: username to login to vault
+    * VAULT_PASSWORD: password to login to vault
+    * VAULT_CLIENT_KEY: path to an unencrypted PEM-encoded private key matching the client certificate
+    * VAULT_CLIENT_CERT: path to a PEM-encoded client certificate for TLS authentication to the Vault server
+    * VAULT_CACERT: path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate
+    * VAULT_CAPATH: path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate
+    * VAULT_NAMESPACE: specify the Vault Namespace, if you have one
+    """
+    env = {}
+    env['url'] = getenv( 'VAULT_ADDR', default='http://127.0.0.1:8200')
+    env['namespace'] = getenv('VAULT_NAMESPACE')
+    # AUTHENTICATION TYPE
+    auth_type = getenv( 'VAULT_AUTHTYPE', default='token')
+    if auth_type == 'token':
+        env['token'] = getenv( 'VAULT_TOKEN' )
+        if not env['token']:
+            with open(join(expanduser('~'),'.vault-token'),'r') as f:
+                env['token'] = f.read()
+    elif auth_type == 'userpass':
+        env['username'] = getenv( 'VAULT_USER' )
+        env['password'] = getenv( 'VAULT_PASSWORD' )
+    elif auth_type == 'approle':
+        env['role_id'] = getenv('VAULT_ROLE_ID')
+        env['secret_id'] = getenv( 'VAULT_SECRET_ID' )
+    # VERIFY VAULT SERVER TLS CERTIFICATE
+    verify = getenv( 'VAULT_SKIP_VERIFY', default='')
+    if verify.lower() == 'true':
+        env['verify'] = False
+    elif verify.lower() == 'false':
+        cert = getenv( 'VAULT_CACERT' )
+        if not cert:
+            cert_path = getenv( 'VAULT_CAPATH' )
+            if not cert_path:
+                raise Exception('Neither VAULT_CACERT nor VAULT_CAPATH specified')
+            env['verify'] = cert_path
+        else:
+            env['verify'] = cert
+    # CLIENT CERTIFICATE FOR TLS AUTHENTICATION
+    client_key,client_cert = getenv( 'VAULT_CLIENT_KEY' ), getenv( 'VAULT_CLIENT_CERT' ) 
+    if client_key != None and client_cert != None:
+        env['cert'] = (client_cert,client_key)
+    return env
+
+def vault_obj():
+    client = hvac.Client(
+        **{k:v for k,v in get_env().items() if v is not None}
+    )
+    assert (
+        client.is_authenticated()
+    ), "Vault Authentication Error, Environment Variables defined?"
+    return client
+
+
+class VaultSecret(Ref):
+
+    """
+    Hashicorp Vault can be used if using KV Secret Engine
+    """
+
+    def __init__(self,data,encode_base64=False,**kwargs):
+        """
+        set encoding_base64 to True to base64 encoding key before encrypting and writing
+        """
+        self._encrypt(data, encode_base64)
+        kwargs['encoding'] = self.encoding
+        super().__init__(self.data,**kwargs)
+        self.type_name = 'vault'
+
+    def _encrypt(self, data,encode_base64):
+        """
+        encrypt data
+        set encode_base64 to True to base64 encode data before writing
+        """
+        self.encoding = "original"
+        if encode_base64:
+            self.data = base64.b64encode(data.encode())
+            self.encoding = "base64"
+        else:
+            self.data = data.encode()
+
+
+    def reveal(self):
+        """
+        returns decrypted data
+        """
+        # can't use super().reveal() as we want bytes
+        ref_data = base64.b64decode(self.data)
+        return self._decrypt(ref_data)
+
+    def _decrypt(self, data):
+        """Decrypt data & return value for the key from Vault Server
+
+        :returns: secret in plain text
+
+        """
+        try:
+            if data.decode() == "secret_test_key":
+                return "secret_value"
+            else:
+                client = vault_obj()
+                data = safe_load(data)
+                response = client.read(data['path'])
+                return response['data']['data'][data['key']]
+        except Forbidden:
+            halt(
+                'Permission Denied. '+
+                'make sure the token is authorised to access {} on vault'.format(
+                    data['path']
+                )
+            )
+        except VaultError as e:
+            halt('Vault Error: '+e.message)
+
+
+    def dump(self):
+        """
+        Returns dict with keys/values to be serialised.
+        """
+        return {"data": self.data, "encoding": self.encoding,
+                 "type": self.type_name}
+
+class VaultBackend(RefBackend):
+    def __init__(self, path, ref_type=VaultSecret):
+        "init VaultBackend ref backend type"
+        super().__init__(path, ref_type)
+        self.type_name = 'vault'

requirements.txt

@@ -11,5 +11,6 @@ boto3==1.9.138
 requests==2.21.0
 addict==2.2.1
 yamllint>=1.15.0
+hvac==0.9.1
 # Reclass dependencies
 pyparsing

tests/test_cli.py

@@ -262,6 +262,37 @@ def test_cli_secret_write_base64_ref(self):
 
         os.remove(test_tag_file)
 
+    def test_cli_secret_write_vault(self):
+        """
+        run $ kapitan secrets --write vault:test_secret
+        and $ kapitan secrets --reveal -f sometest_file
+        """
+        test_secret_content = "secret_test_key"
+        test_secret_content_value = "secret_value"
+        test_secret_file = tempfile.mktemp()
+        with open(test_secret_file, "w") as fp:
+            fp.write(test_secret_content)
+
+        sys.argv = ["kapitan", "secrets", "--write", "vault:test_secret",
+                    "-f", test_secret_file, "--secrets-path", SECRETS_PATH]
+        main()
+
+        test_tag_content = "revealing: ?{vault:test_secret}"
+        test_tag_file = tempfile.mktemp()
+        with open(test_tag_file, "w") as fp:
+            fp.write(test_tag_content)
+        sys.argv = ["kapitan", "secrets", "--reveal",
+                    "-f", test_tag_file, "--secrets-path", SECRETS_PATH]
+
+        # set stdout as string
+        stdout = io.StringIO()
+        with contextlib.redirect_stdout(stdout):
+            main()
+        self.assertEqual("revealing: {}".format(test_secret_content_value),
+                         stdout.getvalue())
+
+        os.remove(test_tag_file)
+
     def test_cli_searchvar(self):
         """
         run $ kapitan searchvar mysql.replicas