Add CloudSQL option for accounts-db, ledger-db (#383)

* first draft * updates * Update main README * PR Feedback * test data * Initialize - add setup scripts, Cloud SQL ledger db * Update manifests * Cleanup * Cleanup * Fix indentation in README * gce-proxy:1.19.0-alpine * typo * missing securityContext.runAsNonRoot: true * fix cloud sql creation script error * Remove extra file Co-authored-by: askmeegs <[email protected]> Co-authored-by: Megan O'Keefe <[email protected]> Co-authored-by: Mathieu Benoit <[email protected]>
GoogleCloudPlatform · Nov 30, 2020 · 536f921 · 536f921
1 parent 474d8a1
commit 536f921
Show file tree

Hide file tree

Showing 16 changed files with 1,666 additions and 5 deletions.
diff --git a/README.md b/README.md
@@ -54,10 +54,10 @@ cd bank-of-anthos
 ```
 ZONE=us-central1-b
 gcloud beta container clusters create bank-of-anthos \
-    --project=${PROJECT_ID} --zone=${ZONE} \
-    --machine-type=e2-standard-2 --num-nodes=4 \
-		--enable-stackdriver-kubernetes --subnetwork=default \
-		--tags=bank-of-anthos --labels csm=
+--project=${PROJECT_ID} --zone=${ZONE} \
+--machine-type=e2-standard-2 --num-nodes=4 \
+--enable-stackdriver-kubernetes --subnetwork=default \
+--tags=bank-of-anthos --labels csm=
 ```
 
 4. **Deploy the demo JWT public key** to the cluster as a Secret. This key is used for user account creation and authentication. 
@@ -109,8 +109,9 @@ EXTERNAL-IP
 ## Other Deployment Options 
 
 - **Workload Identity**: [See these instructions.](docs/workload-identity.md)
+- **Cloud SQL**: [See these instructions](extras/cloudsql) to replace the in-cluster databases with hosted Google Cloud SQL.
 - **Istio**: Apply `istio-manifests/` to your cluster to access the frontend through the IngressGateway. 
-- **Anthos Service Mesh**: ASM requires Workload Identity to be enabled in your GKE cluster. [See the workload identity instructions](docs/workload-identity.md) to configure and deploy the app. Then, apply `istio-manifests/` to your cluster to confugure frontend ingress. 
+- **Anthos Service Mesh**: ASM requires Workload Identity to be enabled in your GKE cluster. [See the workload identity instructions](docs/workload-identity.md) to configure and deploy the app. Then, apply `istio-manifests/` to your cluster to configure frontend ingress. 
 - **Java Monolith (VM)**: We provide a version of this app where the three Java microservices are coupled together into one monolithic service, which you can deploy inside a VM (eg. Google Compute Engine). See the [ledgermonolith](src/ledgermonolith) directory.
 
 ## Troubleshooting 

diff --git a/extras/cloudsql/README.md b/extras/cloudsql/README.md
@@ -0,0 +1,72 @@
+
+# Cloud SQL + Bank of Anthos 
+
+This directory contains instructions and Kubernetes manifests for overriding the default in-cluster PostgreSQL databases (`accountsdb` + `ledgerdb`) with Google Cloud SQL. 
+
+![diagram](arch.png)
+
+## How it works 
+
+The setup scripts provided will provision a Cloud SQL instance in your Google Cloud Project. The script will then create two databases - one for the **accounts DB**, one for the **ledger DB**. This replaces the two separate PostgreSQL StatefulSets used in Bank of Anthos by default. 
+
+
+## Setup 
+
+1. **Create a [Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)** if you don't already have one. 
+
+2. **Set environment variables** corresponding to your project, desired GCP region/zone, and the Kubernetes namespace into which you want to deploy Bank of Anthos.
+
+```
+export PROJECT_ID="my-project"
+export REGION="us-east1"
+export ZONE="us-east1-b" 
+export CLUSTER="my-cluster-name"
+export NAMESPACE="default"
+```
+
+3. **Create a GKE cluster** with [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#overview) enabled. Workload Identity lets you use a Kubernetes service account like a Google Cloud service account, giving your pods granular Google Cloud API permissions - in this case, permission for the Bank of Anthos Pods to access Cloud SQL. 
+
+```
+gcloud container clusters create ${CLUSTER} \
+	--project=${PROJECT_ID} --zone=${ZONE} \
+	--machine-type=e2-standard-4 --num-nodes=4 \
+	--workload-pool="${PROJECT_ID}.svc.id.goog"
+```
+
+4. **Run the Workload Identity setup script** for your new cluster. This script creates a Google Service Account (GSA) and Kubernetes Service Account (KSA), associates them together, then grants the service account permission to access Cloud SQL. 
+
+```
+./setup_workload_identity.sh
+```
+
+5. **Run the Cloud SQL instance create script**. This takes a few minutes to complete. 
+
+```
+./create_cloudsql_instance.sh 
+```
+
+6. **Deploy Bank of Anthos** to your cluster. Each backend Deployment (`userservice`, `contacts`, `transactionhistory`, `balancereader`, and `ledgerwriter`) is configured with a [Cloud SQL Proxy](https://cloud.google.com/sql/docs/postgres/sql-proxy#what_the_proxy_provides) sidecar container. Cloud SQL Proxy provides a secure TLS connection between the backend GKE pods and your Cloud SQL instance. 
+
+This command will also deploy two Kubernetes Jobs, to populate the accounts and ledger dbs with Tables and test data. 
+
+
+```
+kubectl apply -n ${NAMESPACE} -f ./kubernetes-manifests 
+```
+
+7. Wait a few minutes for all the pods to be `RUNNING`. (Except for the two `populate-` Jobs. They should be marked `0/3 - Completed` when they finish successfully.) 
+
+```
+NAME                                  READY   STATUS      RESTARTS   AGE
+balancereader-d48c8d84c-j7ph7         2/2     Running     0          2m56s
+contacts-bbfdbb97f-vzxmv              2/2     Running     0          2m55s
+frontend-65c78dd78c-tsq26             1/1     Running     0          2m55s
+ledgerwriter-774b7bf7b9-jpz7l         2/2     Running     0          2m54s
+loadgenerator-f489d8858-q2n46         1/1     Running     0          2m54s
+populate-accounts-db-wrh4m            0/3     Completed   0          2m54s
+populate-ledger-db-422cr              0/3     Completed   0          2m53s
+transactionhistory-747476548c-j2zqx   2/2     Running     0          2m53s
+userservice-7f6df69544-nskdf          2/2     Running     0          2m53s
+```
+
+8. Access the Bank of Anthos frontend at the frontend service `EXTERNAL_IP`, then log in as `test-user` with the pre-populated credentials added to the Cloud SQL-based `accounts-db`. You should see the pre-populated transaction data show up, from the Cloud SQL-based `ledger-db`. You're done! 
diff --git a/extras/cloudsql/arch.png b/extras/cloudsql/arch.png
diff --git a/extras/cloudsql/create_cloudsql_instance.sh b/extras/cloudsql/create_cloudsql_instance.sh
@@ -0,0 +1,47 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# !/bin/bash 
+
+INSTANCE_NAME='bank-of-anthos-db'
+
+echo "☁️ Enabling the Cloud SQL API..."  
+gcloud config set project ${PROJECT_ID}
+gcloud services enable sqladmin.googleapis.com
+
+echo "☁️ Creating Cloud SQL instance: ${INSTANCE_NAME} ..."  
+gcloud sql instances create $INSTANCE_NAME \
+    --database-version=POSTGRES_12 --tier=db-custom-1-3840 \
+    --region=${REGION} --project ${PROJECT_ID}
+
+echo "☁️ All done creating ${INSTANCE_NAME} ..."  
+INSTANCE_CONNECTION_NAME=$(gcloud sql instances describe $INSTANCE_NAME --format='value(connectionName)')
+
+echo "☁️ Creating admin user..."  
+gcloud sql users create admin \
+   --instance=$INSTANCE_NAME --password=admin 
+
+echo "☁️ Creating a K8s Secret with project, connection, and admin user credentials..."  
+kubectl create secret -n ${NAMESPACE} generic cloud-sql-admin \
+ --from-literal=username=admin --from-literal=password=admin \
+ --from-literal=connectionName=${INSTANCE_CONNECTION_NAME}
+
+# Create Accounts DB
+echo "☁️ Creating accounts-db in ${INSTANCE_NAME}..."  
+gcloud sql databases create accounts-db --instance=$INSTANCE_NAME
+
+# Create Ledger DB  
+echo "☁️ Creating ledger-db in ${INSTANCE_NAME}..."  
+gcloud sql databases create ledger-db --instance=$INSTANCE_NAME
+
+echo "⭐️ Done."
diff --git a/extras/cloudsql/kubernetes-manifests/balance-reader.yaml b/extras/cloudsql/kubernetes-manifests/balance-reader.yaml
@@ -0,0 +1,124 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: balancereader
+spec:
+  selector:
+    matchLabels:
+      app: balancereader
+  template:
+    metadata:
+      labels:
+        app: balancereader
+    spec:
+      serviceAccountName: boa-ksa
+      terminationGracePeriodSeconds: 5
+      containers:
+      - name: balancereader
+        image: gcr.io/bank-of-anthos/balancereader:v0.3.1
+        volumeMounts:
+        - name: publickey
+          mountPath: "/root/.ssh"
+          readOnly: true
+        env:
+        - name: VERSION
+          value: "v0.3.1"
+        - name: PORT
+          value: "8080"
+        # toggle Cloud Trace export
+        - name: ENABLE_TRACING
+          value: "true"
+        - name: ENABLE_METRICS
+          value: "true"
+        - name: POLL_MS
+          value: "100"
+        - name: CACHE_SIZE
+          value: "1000000"
+        # tell Java to obey container memory limits
+        - name: JVM_OPTS
+          value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
+        # Valid levels are debug, info, warn, error, fatal.
+        # If no valid level is set, will default to info.
+        - name: LOG_LEVEL
+          value: "info"
+        - name: NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        envFrom:
+        - configMapRef:
+            name: environment-config
+        # add ledger-db credentials from ConfigMap
+        - configMapRef:
+            name: ledger-db-config
+        resources:
+          requests:
+            cpu: 100m
+            memory: 512Mi
+          limits:
+            cpu: 500m
+            memory: 1Gi
+        readinessProbe:
+          httpGet:
+            path: /ready
+            port: 8080
+          initialDelaySeconds: 60
+          periodSeconds: 5
+          timeoutSeconds: 10
+        livenessProbe:
+          httpGet:
+            path: /healthy
+            port: 8080
+          initialDelaySeconds: 120
+          periodSeconds: 5
+          timeoutSeconds: 10
+      - name: cloudsql-proxy
+        resources:
+          limits:
+            cpu: "200m"
+            memory: "100Mi"
+        image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine
+        env:
+        - name: CONNECTION_NAME
+          valueFrom:
+            secretKeyRef:
+              name: cloud-sql-admin
+              key: connectionName
+        command: ["/cloud_sql_proxy",
+                  "-instances=$(CONNECTION_NAME)=tcp:5432"]
+        securityContext:
+          runAsNonRoot: true
+      volumes:
+      - name: publickey
+        secret:
+          secretName: jwt-key
+          items:
+          - key: jwtRS256.key.pub
+            path: publickey
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: balancereader
+spec:
+  type: ClusterIP
+  selector:
+    app: balancereader
+  ports:
+  - name: http
+    port: 8080
+    targetPort: 8080
diff --git a/extras/cloudsql/kubernetes-manifests/config.yaml b/extras/cloudsql/kubernetes-manifests/config.yaml
@@ -0,0 +1,66 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: environment-config
+data:
+  LOCAL_ROUTING_NUM: "883745000"
+  PUB_KEY_PATH: "/root/.ssh/publickey"
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: service-api-config
+data:
+  TRANSACTIONS_API_ADDR: "ledgerwriter:8080"
+  BALANCES_API_ADDR: "balancereader:8080"
+  HISTORY_API_ADDR: "transactionhistory:8080"
+  CONTACTS_API_ADDR: "contacts:8080"
+  USERSERVICE_API_ADDR: "userservice:8080"
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: demo-data-config
+data:
+  USE_DEMO_DATA: "True"
+  DEMO_LOGIN_USERNAME: "testuser"
+  # All demo user accounts are hardcoded to use the login password 'password'
+  DEMO_LOGIN_PASSWORD: "password"
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: accounts-db-config
+  labels:
+    app: accounts-db
+data:
+  # Updated to use CloudSQL Proxy
+  ACCOUNTS_DB_URI: postgresql://admin:[email protected]:5432/accounts-db
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ledger-db-config
+  labels:
+    app: postgres
+data:
+  POSTGRES_DB: postgresdb
+  POSTGRES_USER: admin
+  POSTGRES_PASSWORD: admin
+  # Updated to use CloudSQL Proxy
+  SPRING_DATASOURCE_URL: jdbc:postgresql://127.0.0.1:5432/ledger-db
+  SPRING_DATASOURCE_USERNAME: admin # should match POSTGRES_USER
+  SPRING_DATASOURCE_PASSWORD: admin # should match POSTGRES_PASSWORD