uc-cdis · EliseCastle23 · Mar 11, 2024 · Aug 3, 2023 · Aug 3, 2023 · Aug 3, 2023
diff --git a/.secrets.baseline b/.secrets.baseline
diff --git a/README.md b/README.md
@@ -117,8 +117,19 @@ NOTE: Gen3 helm charts are currently not used in production by CTDS, but we are
 
 For local development you must be connected to a kubernetes cluster. As referenced above in the section `Kubernetes cluster` we recommend using [Rancher Desktop](https://rancherdesktop.io/) as Kubernetes on your local machine, especially on M1 Mac's. You also get ingress and other benefits out of the box.
 
+For MacOS users, [Minikube](https://minikube.sigs.k8s.io/docs/start/) equipped with the ingress addon serves as a viable alternative to Rancher Desktop. On Linux, we've observed that using [Kind](https://kind.sigs.k8s.io/) with an NGINX ingress installed often provides a more seamless experience compared to both Rancher Desktop and Minikube. Essentially, Helm requires access to a Kubernetes cluster with ingress capabilities, facilitating the loading of the portal in your browser for an optimal development workflow.
+
+To install the NGINX ingress: 
+```
+  helm repo add nginx-stable https://helm.nginx.com/stable
+  helm repo update
+  kubectl create ns nginx-ingress
+  helm install nginx-ingress nginx-stable/nginx-ingress --namespace nginx-ingress 
+```
+
 > **Warning**
 > If you are using Rancher Desktop you need to increase the vm.max_map_count as outlined [here](https://docs.rancherdesktop.io/how-to-guides/increasing-open-file-limit/)
+> If you are using Minikube you will need to enabled the ingress addon as outlined [here](https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/)
 
 1. Clone the repository
 2. Navigate to the `gen3-helm/helm/gen3` directory and run `helm dependency update`

diff --git a/docs/PRODUCTION.md b/docs/PRODUCTION.md
@@ -13,3 +13,14 @@ The postgres and helm charts are included as conditionals in the Gen3 [umbrella
   repository: "https://charts.bitnami.com/bitnami"
   condition: global.dev
   ```
+
+### Kubernetes Configmap and Secret Configuration
+For the seamless operation of our services, we utilize Kubernetes secrets. To streamline the integration and management of these secrets, we highly recommend deploying External Secret Manager alongside any existing secret management systems you may already have in place. For a comprehensive guide and best practices on implementing External Secrets within our ecosystem, please consult our dedicated External Secrets Documentation available [here](https://github.com/uc-cdis/gen3-helm/blob/master/docs/external_secrets.md).
+
+Our services also utilize non-secret configuration variables provided via Kubernetes ConfigMaps. For streamlined management, we advise keeping your values.yaml and configuration files files in source control and utilizing ArgoCD for automatic updates and efficient management of your Gen3 Helm chart.
+
+Each service is designed to seamlessly integrate and manage the combination of Kubernetes secrets and ConfigMaps, ensuring the encapsulated information is effectively injected into the underlying application.
+
+Please see the diagram provided that details how External Secrets operates. We also mention the use of Argo CD as our choice option for Helm deployments. 
+![External Secrets Diagram](./images/lucidChart.png  "Helm Secrets Manager")
+
diff --git a/docs/external_secrets.md b/docs/external_secrets.md
@@ -0,0 +1,179 @@
+# External Secrets Operator
+
+ "External Secrets Operator" is a tool that was created by the Kubernetes community to manage external secrets in a Kubernetes cluster. It allows you to fetch and sync external secret values from various external secret management systems into Kubernetes secrets. One of the external secret management systems it can connect to is AWS Secrets Manager. AWS Secrets Manager allows for the secure storing of your secrets as well as the ability to periodically and automatically rotate your secrets.
+
+This document will guide you through setting up the essential resources to access your secrets in AWS Secrets Manager and download the External Secrets Operator Helm chart. This way, you can effectively utilize your stored secrets with Helm.
+
+## Download External Secrets Operator and Create Resources in AWS.
+You can use the following Bash script to apply the External Secrets Operator to your cluster and create the necessary AWS resources. Fill in the variables below to get started:
+
+***Notice:
+The Gen3 Helm chart has various jobs and uses for an IAM user. To enhance code reusability, we've implemented the option for jobs and services to share the same AWS IAM global user. If you would like to use the same IAM user for External Secrets and jobs like ["Fence Usersync"](fence_usersync_job.md) or our "AWS ES Proxy Service", you can follow [THIS](global_IAM_helm_user.md) guide that details how to setup a Helm global user. In case you opt for a global IAM user, please comment out the "create_iam_policy" and "create_iam_user" functions at the end of the script.***
+
+```
+#!/bin/bash
+
+AWS_ACCOUNT="<Your AWS Account ID>" 
+region="us-east-1"
+iam_policy="external_secrets_policy"
+iam_user="external_secrets_user"
+
+helm_install() 
+{
+    echo "# ------------------ Install external-secrets via helm --------------------------#"
+    helm repo add external-secrets https://charts.external-secrets.io
+    helm install external-secrets \
+    external-secrets/external-secrets \
+    -n external-secrets \
+    --create-namespace \
+    --set installCRDs=true
+}
+
+create_iam_policy()
+{
+    echo "# ------------------ create iam policy for aws secrets manager --------------------------#"
+    POLICY_ARN=$(aws iam create-policy --policy-name $iam_policy --policy-document '{
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": [
+                    "secretsmanager:ListSecrets",
+                    "secretsmanager:GetSecretValue"
+                ],
+                "Resource": [
+                    "*"
+                ]
+            }
+        ]
+    }')
+
+    iam_policy_arn=$(aws iam list-policies --query "Policies[?PolicyName=='$iam_policy'].Arn" --output text)
+    echo "Policy Arn: $iam_policy_arn"
+    # return $iam_policy_arn
+}
+
+create_iam_user()
+{
+    echo "# ------------------ create user $iam_user --------------------------#"
+    aws iam create-user --user-name $iam_user
+
+    echo "# ------------------ add iam user $iam_user to policy $iam_policy --------------------------#"
+    aws iam attach-user-policy --user-name $iam_user --policy-arn $iam_policy_arn
+    echo "aws iam attach-user-policy --user-name $iam_user --policy-arn $iam_policy_arn"
+
+    echo "# ------------------ create access key and secret key for external-secrets --------------------------#"
+    aws iam create-access-key --user-name $iam_user > keys.json
+    access_key=$(jq -r .AccessKey.AccessKeyId keys.json)
+    secret_key=$(jq -r .AccessKey.SecretAccessKey keys.json)
+    kubectl create secret generic "$iam_user"-secret --from-literal=access-key=$access_key --from-literal=secret-access-key=$secret_key
+    rm keys.json
+}
+
+helm_install
+#comment out the below if using global iam user.
+create_iam_policy
+create_iam_user
+```
+
+***Please note that Terraform for the creation and population of Gen3 Secrets in AWS Secrets Manager is in development currently. This Terraform will also create the Iam user and policies necessary to access these secrets.***
+
+## Enabling External Secrets in Helm charts
+Our Helm architecture includes a comprehensive [umbrella](https://github.com/uc-cdis/gen3-helm/tree/master/helm/gen) chart designed to streamline the deployment of external secrets across both the umbrella chart itself and its associated subcharts. By configuring the `.Values.global.externalSecrets.deploy` setting within this umbrella chart, users can effortlessly initiate the deployment of all related external secret resources. This includes the external secret resources within the subcharts and the secret store required for their management.
+
+#### Global Deployment of External Secrets
+Upon deployment of the umbrella chart, the `.Values.global.externalSecrets.deploy` setting automatically provisions external secret resources for every subchart. This occurs regardless of the individual external secrets deployment settings within subcharts, even if they are explicitly set to `false`. This feature ensures a unified approach to secret management across all components of the architecture.
+
+#### Selective Secret Management
+For users requiring a more selective application of external secrets — targeting specific secrets while excluding others — the system is designed to accommodate such scenarios with ease.
+
+External secret resources will only attempt to replace Kubernetes secrets when a corresponding secret is successfully located within the Secrets Manager. In instances where a specific secret is not found, the External Secrets resource will indicate a `SecretSyncedError`, signaling the absence of the targeted resource within the Secrets Manager. This error is acceptable and helpful for users who want to enable the use of AWS Secrets Manager for some, but not all the secrets in a specific Helm chart.
+
+However, if you wish to utilize External Secrets for managing non-database secrets while still automating the creation of your database secrets, you can configure this behavior explicitly. Set `.Values.global.externalSecrets.dbCreate` to true alongside `.Values.global.postgres.dbCreate` or `.Values.postgres.dbCreate` to initiate the database creation job. This configuration will result in the creation of the necessary databases with their credentials stored securely within Kubernetes Secrets. Subsequently, you also choose to create Secrets in Secrets manager with the values that were generated from the dbCreate job if you wish to store these credentials long term.
+
+By default, the following services will not create the Helm internal secrets when Secrets Manager is enabled: 
+- Audit
+- Fence
+- Indexd
+- Manifestservice
+- Metadata
+
+This is because CD tools like Argocd will have trouble syncing resources if the K8s secret was generated via Helm and External Secrets continues to override it. You can configure Helm to still create these secrets with External Secrets enabled by setting the appropriate variable to true. 
+
+For example, to ensure the "audit-g3auto" secret is still created by Helm, you would need to set the following in your values.yaml file: 
+```
+audit: 
+    # -- (map) External Secrets settings.
+    externalSecrets:
+    # -- (string) Will create the Helm "audit-g3auto" secret even if Secrets Manager is enabled. This is helpful if you are wanting to use External Secrets for some, but not all secrets.
+    createK8sAuditSecret: true
+```
+
+#### Independent Subchart Deployment
+In scenarios where subcharts are deployed independently, outside the scope of the umbrella chart, it is crucial to set the `.Values.global.externalSecrets.deploy` directive within the `values.yaml` file for each specific service.
+
+Additionally, to facilitate the creation of a Secret Store capable of authenticating with AWS Secrets Manager, the `.Values.global.externalSecrets.separateSecretStore` should be set to true in the relevant charts. This configuration is essential for establishing proper authentication mechanisms for secret retrieval.
+
+#### Configuring Separate Secret Stores
+The `.Values.global.externalSecrets.separateSecretStore` setting can also be applied within the context of the umbrella chart deployment. Utilizing this setting allows for the creation of distinct Secret Stores dedicated to individual services. This approach is particularly beneficial for environments where it is preferable to limit access to Secrets Manager, ensuring that services only have access to the secrets explicitly required for their operation.
+
+## Helm IAM User
+If you are using a separate IAM user for AWS Secrets Manager please follow the below instructions: 
+
+This script Bash script at the beginning of this document should have created a secret titled "NameofIAMuser-user-secret" in your cluster. You will need to retrieve these values to input into your Helm chart for the Secret Store to authenticate with AWS Secrets Manager.
+
+
+Access Key:
+```
+kubectl get secret "your secret name" -o jsonpath="{.data.access-key}" | base64 --decode
+```
+
+
+Secret Access Key:
+```
+kubectl get secret "your secret name" -o jsonpath="{.data.secret-access-key}" | base64 --decode
+```
+
+You can paste the IAM access key and secret access key in the `.Values.secrets.awsAccessKeyId`/`.Values.secrets.awsSecretAccessKey` fields in the values.yaml file for the chart(s) you would like to use external secrets for. 
+
+If you are deploying external secrets with the Gen3 umbrella chart, you can utilize a local secret to avoid pasting credentials in the values.yaml file. Just set `.global.aws.useLocalSecret.enabled` to true and supply your secret name.
+
+Please note that only some Helm charts are compatible with External Secrets currently. We hope to expand this functionality in the future. If a chart is able to use External Secrets, you can see a `.Values.externalSecrets` section in the values.yaml file.
+
+## How External Secrets Works. 
+External Secrets relies on three main resources to function properly. (The below have links to examples of each resource)
+1. [Aws-config](https://github.com/uc-cdis/gen3-helm/blob/master/helm/common/templates/_aws_config.tpl)- Contains Access and Secret Access keys used by the Cluster Secret Store to authenticate with AWS Secrets Manager
+2. [Secret Store](https://github.com/uc-cdis/gen3-helm/blob/master/helm/common/templates/_external_secrets.tpl#L41-L62)- Resource to Authenticate with AWS Secrets Manager
+3. [External Secret](https://github.com/uc-cdis/gen3-helm/blob/master/helm/common/templates/_external_secrets.tpl#L15-L38)- References the Secret Store and is used as a "map" to tell External Secrets Operator what secret to grab from External Secrets and the name of the Kubernetes Secret to create locally.
+
+    Anatomy of an ExternalSecret:
+    ```
+    apiVersion: external-secrets.io/v1beta1
+    kind: ExternalSecret
+    metadata:
+      # Name of the External Secret resource
+      name: audit-g3auto
+    spec:
+      #How often to Sync with AWS Secrets Manager
+      refreshInterval: 5m
+      secretStoreRef:
+        # The name of the Secret Store to use.
+        name: {{include "cluster-secret-store" .}}
+        kind: SecretStore
+      target:
+        # What Kubernetes secret to create from the secret pulled from AWS Secrets Manager.
+        name: audit-g3auto
+        creationPolicy: Owner
+      data:
+      # the key inside the new Kubernetes secret
+      - secretKey: audit-service-config.yaml
+          remoteRef:
+          #name of secret in AWS Secrets Manager
+          key: {{include "audit-g3auto" .}}
+    ```
+
+## Customizing the AWS Secrets Manager Secrets Name.
+When pulling a secret from AWS Secrets Manager, you want to ensure that the External Secret resource is referencing the proper name of the secret in AWS Secrets Manager.
+You can customize the name of the secret to pull from in the `.Values.externalSecrets` section of a Chart. You can see the name for the confiugrable secrets in a chart by looking in this section as well. 
+
+Any string you put in this section will override the name of the secret that is pulled from AWS Secrets Manager NOT the name of the Kubernetes secret that is created from the External Secret resource.
diff --git a/docs/fence_usersync_job.md b/docs/fence_usersync_job.md
@@ -15,7 +15,12 @@ User lists can be synced from three sources:
 # S3 user.yaml Setup {#s3-setup}
 Please see [this](https://github.com/uc-cdis/fence/blob/master/docs/user.yaml_guide.md) documentation that details user.yaml formatting.
 
-You can pull this file from an S3 bucket that is set in the `.Values.usersync.userYamlS3Path` field. Then input the iam credentials for a user that has read access to the specified S3 bucket in the `.Values.usersync.secrets.awsAccessKeyId` and `.Values.usersync.secrets.awsSecretAccessKey` fields. 
+You can pull this file from an S3 bucket that is set in the `.Values.usersync.userYamlS3Path` field. Then input the IAM credentials for a user that has read access to the specified S3 bucket in the `.Values.secrets.awsAccessKeyId` and `.Values.secrets.awsSecretAccessKey` fields. 
+
+You can utilize a local secret to avoid pasting credentials in the values.yaml file. Just set `.global.aws.useLocalSecret.enabled` to true and supply your secret name.
+
+***Notice:
+The Gen3 Helm chart has various jobs and uses for an IAM user. To enhance code reusability, we've implemented the option for jobs and services to share the same AWS IAM global user. If you would like to use the same IAM user for Fence Usersync, External Secrets, etc.- you can follow [THIS](global_iam_helm_user.md) guide that details how to setup a Helm global user.***
 
 As previously mentioned, if the `.Values.usersync.userYamlS3Path` string is set to "none", the user.yaml file from Fence values.yaml will be used. 
 
@@ -59,7 +64,7 @@ For an example of a dbGap auth file (csv), please see [this](https://github.com/
 
 
 
-# Other Customizations
+## Other Customizations
   The `.Values.usersync.schedule` option can be set to customize the cron schedule expression. The default setting is to have the job run once every 30 minutes. 
 
   The `.Values.usersync.custom_image` can be set to override the default "awshelper" image for the init container of the userync cronjob.
diff --git a/docs/global_iam_helm_user.md b/docs/global_iam_helm_user.md
@@ -0,0 +1,55 @@
+# AWS IAM Global User
+
+For Helm code reusability, we have added the functionality to use one IAM user for various jobs/services.
+
+We are currently in the process of integrating this user into our Terraform code. In the meantime, you can manually create a global user by referring to this guide.
+
+## What this user can do in Helm
+- Fence Usersync Job
+- ES Index Restore
+- Restore PGdump
+- External Secrets
+- AWS ES Proxy Service
+
+
+
+Example policy containing all the proper permissions:
+```
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": "s3:GetObject",
+            "Resource": [
+                "arn:aws:s3:::$BUCKET/$ENVIRONMENT/*",  
+                # Fence Usersync Job: Name of the userYamlS3Path containing the user.yaml file
+                "arn:aws:s3:::$BUCKET/$ENVIRONMENT/$VERSION/elasticsearch/*",  
+                # ES Index Restore Job: Name of the dbRestoreBucket with the proper path to the ES dump files.
+                "arn:aws:s3:::$BUCKET/$ENVIRONMENT/$VERSION/pgdumps/*"  
+                # DB PG Dump Restore Job: Name of the dbRestoreBucket with the proper path to the SQL dump files.
+            ]
+        },
+        {
+            "Effect": "Allow",
+            "Action": [
+                "secretsmanager:ListSecrets",
+                "secretsmanager:GetSecretValue"
+            ],
+            "Resource": [
+                "*" 
+                # External Secrets: Leave as is to allow External Secrets access to your secrets in Secrets Manager.
+            ]
+        },
+        {
+            "Effect": "Allow",
+            "Action": "es:*",
+            "Resource": "arn:aws:es:REGION:ACCOUNT_ID:domain/CLUSTER_NAME/*" 
+            # AWS ES Proxy Service: Arn of your Elasticsearch Cluster in AWS.
+        }
+    ]
+}
+```
+
+## After Creating the User
+In order to integrate the user in Helm, paste in the values of your Access and Secret Access key in `.Values.global.aws.awsAccessKeyId` and `.Values.global.aws.awsSecretAccessKey`
diff --git a/docs/images/lucidChart.png b/docs/images/lucidChart.png
diff --git a/helm/ambassador/Chart.yaml b/helm/ambassador/Chart.yaml
@@ -15,7 +15,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.1.9
+version: 0.1.10
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
@@ -25,5 +25,5 @@ appVersion: "1.4.2"
 
 dependencies:
 - name: common
-  version: 0.1.8
+  version: 0.1.9
   repository: file://../common