diff --git a/README.md b/README.md index fea7c5769..2a771a44a 100644 --- a/README.md +++ b/README.md @@ -54,10 +54,10 @@ cd bank-of-anthos ``` ZONE=us-central1-b gcloud beta container clusters create bank-of-anthos \ - --project=${PROJECT_ID} --zone=${ZONE} \ - --machine-type=e2-standard-2 --num-nodes=4 \ - --enable-stackdriver-kubernetes --subnetwork=default \ - --tags=bank-of-anthos --labels csm= +--project=${PROJECT_ID} --zone=${ZONE} \ +--machine-type=e2-standard-2 --num-nodes=4 \ +--enable-stackdriver-kubernetes --subnetwork=default \ +--tags=bank-of-anthos --labels csm= ``` 4. **Deploy the demo JWT public key** to the cluster as a Secret. This key is used for user account creation and authentication. @@ -109,8 +109,9 @@ EXTERNAL-IP ## Other Deployment Options - **Workload Identity**: [See these instructions.](docs/workload-identity.md) +- **Cloud SQL**: [See these instructions](extras/cloudsql) to replace the in-cluster databases with hosted Google Cloud SQL. - **Istio**: Apply `istio-manifests/` to your cluster to access the frontend through the IngressGateway. -- **Anthos Service Mesh**: ASM requires Workload Identity to be enabled in your GKE cluster. [See the workload identity instructions](docs/workload-identity.md) to configure and deploy the app. Then, apply `istio-manifests/` to your cluster to confugure frontend ingress. +- **Anthos Service Mesh**: ASM requires Workload Identity to be enabled in your GKE cluster. [See the workload identity instructions](docs/workload-identity.md) to configure and deploy the app. Then, apply `istio-manifests/` to your cluster to configure frontend ingress. - **Java Monolith (VM)**: We provide a version of this app where the three Java microservices are coupled together into one monolithic service, which you can deploy inside a VM (eg. Google Compute Engine). See the [ledgermonolith](src/ledgermonolith) directory. ## Troubleshooting diff --git a/extras/cloudsql/README.md b/extras/cloudsql/README.md new file mode 100644 index 000000000..a548ba7b3 --- /dev/null +++ b/extras/cloudsql/README.md @@ -0,0 +1,72 @@ + +# Cloud SQL + Bank of Anthos + +This directory contains instructions and Kubernetes manifests for overriding the default in-cluster PostgreSQL databases (`accountsdb` + `ledgerdb`) with Google Cloud SQL. + +![diagram](arch.png) + +## How it works + +The setup scripts provided will provision a Cloud SQL instance in your Google Cloud Project. The script will then create two databases - one for the **accounts DB**, one for the **ledger DB**. This replaces the two separate PostgreSQL StatefulSets used in Bank of Anthos by default. + + +## Setup + +1. **Create a [Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)** if you don't already have one. + +2. **Set environment variables** corresponding to your project, desired GCP region/zone, and the Kubernetes namespace into which you want to deploy Bank of Anthos. + +``` +export PROJECT_ID="my-project" +export REGION="us-east1" +export ZONE="us-east1-b" +export CLUSTER="my-cluster-name" +export NAMESPACE="default" +``` + +3. **Create a GKE cluster** with [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#overview) enabled. Workload Identity lets you use a Kubernetes service account like a Google Cloud service account, giving your pods granular Google Cloud API permissions - in this case, permission for the Bank of Anthos Pods to access Cloud SQL. + +``` +gcloud container clusters create ${CLUSTER} \ + --project=${PROJECT_ID} --zone=${ZONE} \ + --machine-type=e2-standard-4 --num-nodes=4 \ + --workload-pool="${PROJECT_ID}.svc.id.goog" +``` + +4. **Run the Workload Identity setup script** for your new cluster. This script creates a Google Service Account (GSA) and Kubernetes Service Account (KSA), associates them together, then grants the service account permission to access Cloud SQL. + +``` +./setup_workload_identity.sh +``` + +5. **Run the Cloud SQL instance create script**. This takes a few minutes to complete. + +``` +./create_cloudsql_instance.sh +``` + +6. **Deploy Bank of Anthos** to your cluster. Each backend Deployment (`userservice`, `contacts`, `transactionhistory`, `balancereader`, and `ledgerwriter`) is configured with a [Cloud SQL Proxy](https://cloud.google.com/sql/docs/postgres/sql-proxy#what_the_proxy_provides) sidecar container. Cloud SQL Proxy provides a secure TLS connection between the backend GKE pods and your Cloud SQL instance. + +This command will also deploy two Kubernetes Jobs, to populate the accounts and ledger dbs with Tables and test data. + + +``` +kubectl apply -n ${NAMESPACE} -f ./kubernetes-manifests +``` + +7. Wait a few minutes for all the pods to be `RUNNING`. (Except for the two `populate-` Jobs. They should be marked `0/3 - Completed` when they finish successfully.) + +``` +NAME READY STATUS RESTARTS AGE +balancereader-d48c8d84c-j7ph7 2/2 Running 0 2m56s +contacts-bbfdbb97f-vzxmv 2/2 Running 0 2m55s +frontend-65c78dd78c-tsq26 1/1 Running 0 2m55s +ledgerwriter-774b7bf7b9-jpz7l 2/2 Running 0 2m54s +loadgenerator-f489d8858-q2n46 1/1 Running 0 2m54s +populate-accounts-db-wrh4m 0/3 Completed 0 2m54s +populate-ledger-db-422cr 0/3 Completed 0 2m53s +transactionhistory-747476548c-j2zqx 2/2 Running 0 2m53s +userservice-7f6df69544-nskdf 2/2 Running 0 2m53s +``` + +8. Access the Bank of Anthos frontend at the frontend service `EXTERNAL_IP`, then log in as `test-user` with the pre-populated credentials added to the Cloud SQL-based `accounts-db`. You should see the pre-populated transaction data show up, from the Cloud SQL-based `ledger-db`. You're done! \ No newline at end of file diff --git a/extras/cloudsql/arch.png b/extras/cloudsql/arch.png new file mode 100644 index 000000000..4cf0f9f74 Binary files /dev/null and b/extras/cloudsql/arch.png differ diff --git a/extras/cloudsql/create_cloudsql_instance.sh b/extras/cloudsql/create_cloudsql_instance.sh new file mode 100755 index 000000000..a64a286e5 --- /dev/null +++ b/extras/cloudsql/create_cloudsql_instance.sh @@ -0,0 +1,47 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# !/bin/bash + +INSTANCE_NAME='bank-of-anthos-db' + +echo "☁️ Enabling the Cloud SQL API..." +gcloud config set project ${PROJECT_ID} +gcloud services enable sqladmin.googleapis.com + +echo "☁️ Creating Cloud SQL instance: ${INSTANCE_NAME} ..." +gcloud sql instances create $INSTANCE_NAME \ + --database-version=POSTGRES_12 --tier=db-custom-1-3840 \ + --region=${REGION} --project ${PROJECT_ID} + +echo "☁️ All done creating ${INSTANCE_NAME} ..." +INSTANCE_CONNECTION_NAME=$(gcloud sql instances describe $INSTANCE_NAME --format='value(connectionName)') + +echo "☁️ Creating admin user..." +gcloud sql users create admin \ + --instance=$INSTANCE_NAME --password=admin + +echo "☁️ Creating a K8s Secret with project, connection, and admin user credentials..." +kubectl create secret -n ${NAMESPACE} generic cloud-sql-admin \ + --from-literal=username=admin --from-literal=password=admin \ + --from-literal=connectionName=${INSTANCE_CONNECTION_NAME} + +# Create Accounts DB +echo "☁️ Creating accounts-db in ${INSTANCE_NAME}..." +gcloud sql databases create accounts-db --instance=$INSTANCE_NAME + +# Create Ledger DB +echo "☁️ Creating ledger-db in ${INSTANCE_NAME}..." +gcloud sql databases create ledger-db --instance=$INSTANCE_NAME + +echo "⭐️ Done." \ No newline at end of file diff --git a/extras/cloudsql/kubernetes-manifests/balance-reader.yaml b/extras/cloudsql/kubernetes-manifests/balance-reader.yaml new file mode 100644 index 000000000..9578a5097 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/balance-reader.yaml @@ -0,0 +1,124 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: balancereader +spec: + selector: + matchLabels: + app: balancereader + template: + metadata: + labels: + app: balancereader + spec: + serviceAccountName: boa-ksa + terminationGracePeriodSeconds: 5 + containers: + - name: balancereader + image: gcr.io/bank-of-anthos/balancereader:v0.3.1 + volumeMounts: + - name: publickey + mountPath: "/root/.ssh" + readOnly: true + env: + - name: VERSION + value: "v0.3.1" + - name: PORT + value: "8080" + # toggle Cloud Trace export + - name: ENABLE_TRACING + value: "true" + - name: ENABLE_METRICS + value: "true" + - name: POLL_MS + value: "100" + - name: CACHE_SIZE + value: "1000000" + # tell Java to obey container memory limits + - name: JVM_OPTS + value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap" + # Valid levels are debug, info, warn, error, fatal. + # If no valid level is set, will default to info. + - name: LOG_LEVEL + value: "info" + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + envFrom: + - configMapRef: + name: environment-config + # add ledger-db credentials from ConfigMap + - configMapRef: + name: ledger-db-config + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + cpu: 500m + memory: 1Gi + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 60 + periodSeconds: 5 + timeoutSeconds: 10 + livenessProbe: + httpGet: + path: /healthy + port: 8080 + initialDelaySeconds: 120 + periodSeconds: 5 + timeoutSeconds: 10 + - name: cloudsql-proxy + resources: + limits: + cpu: "200m" + memory: "100Mi" + image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine + env: + - name: CONNECTION_NAME + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: connectionName + command: ["/cloud_sql_proxy", + "-instances=$(CONNECTION_NAME)=tcp:5432"] + securityContext: + runAsNonRoot: true + volumes: + - name: publickey + secret: + secretName: jwt-key + items: + - key: jwtRS256.key.pub + path: publickey +--- +apiVersion: v1 +kind: Service +metadata: + name: balancereader +spec: + type: ClusterIP + selector: + app: balancereader + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/extras/cloudsql/kubernetes-manifests/config.yaml b/extras/cloudsql/kubernetes-manifests/config.yaml new file mode 100644 index 000000000..e72ca5a37 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/config.yaml @@ -0,0 +1,66 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: v1 +kind: ConfigMap +metadata: + name: environment-config +data: + LOCAL_ROUTING_NUM: "883745000" + PUB_KEY_PATH: "/root/.ssh/publickey" +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: service-api-config +data: + TRANSACTIONS_API_ADDR: "ledgerwriter:8080" + BALANCES_API_ADDR: "balancereader:8080" + HISTORY_API_ADDR: "transactionhistory:8080" + CONTACTS_API_ADDR: "contacts:8080" + USERSERVICE_API_ADDR: "userservice:8080" +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: demo-data-config +data: + USE_DEMO_DATA: "True" + DEMO_LOGIN_USERNAME: "testuser" + # All demo user accounts are hardcoded to use the login password 'password' + DEMO_LOGIN_PASSWORD: "password" +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: accounts-db-config + labels: + app: accounts-db +data: + # Updated to use CloudSQL Proxy + ACCOUNTS_DB_URI: postgresql://admin:admin@127.0.0.1:5432/accounts-db +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: ledger-db-config + labels: + app: postgres +data: + POSTGRES_DB: postgresdb + POSTGRES_USER: admin + POSTGRES_PASSWORD: admin + # Updated to use CloudSQL Proxy + SPRING_DATASOURCE_URL: jdbc:postgresql://127.0.0.1:5432/ledger-db + SPRING_DATASOURCE_USERNAME: admin # should match POSTGRES_USER + SPRING_DATASOURCE_PASSWORD: admin # should match POSTGRES_PASSWORD \ No newline at end of file diff --git a/extras/cloudsql/kubernetes-manifests/contacts.yaml b/extras/cloudsql/kubernetes-manifests/contacts.yaml new file mode 100644 index 000000000..4e289b398 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/contacts.yaml @@ -0,0 +1,102 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: contacts +spec: + selector: + matchLabels: + app: contacts + template: + metadata: + labels: + app: contacts + spec: + serviceAccountName: boa-ksa + terminationGracePeriodSeconds: 5 + containers: + - name: contacts + image: gcr.io/bank-of-anthos/contacts:v0.3.1 + volumeMounts: + - name: publickey + mountPath: "/root/.ssh" + readOnly: true + env: + - name: VERSION + value: "v0.3.0" + - name: PORT + value: "8080" + - name: ENABLE_TRACING + value: "true" + # Valid levels are debug, info, warning, error, critical. + # If no valid level is set, will default to info. + - name: LOG_LEVEL + value: "info" + envFrom: + - configMapRef: + name: environment-config + - configMapRef: + name: accounts-db-config + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 5 + timeoutSeconds: 10 + - name: cloudsql-proxy + resources: + limits: + cpu: "200m" + memory: "100Mi" + image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine + env: + - name: CONNECTION_NAME + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: connectionName + command: ["/cloud_sql_proxy", + "-instances=$(CONNECTION_NAME)=tcp:5432"] + securityContext: + runAsNonRoot: true + volumes: + - name: publickey + secret: + secretName: jwt-key + items: + - key: jwtRS256.key.pub + path: publickey +--- +apiVersion: v1 +kind: Service +metadata: + name: contacts +spec: + type: ClusterIP + selector: + app: contacts + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/extras/cloudsql/kubernetes-manifests/frontend.yaml b/extras/cloudsql/kubernetes-manifests/frontend.yaml new file mode 100644 index 000000000..336fc3073 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/frontend.yaml @@ -0,0 +1,97 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: frontend +spec: + selector: + matchLabels: + app: frontend + template: + metadata: + labels: + app: frontend + spec: + serviceAccountName: boa-ksa + terminationGracePeriodSeconds: 5 + containers: + - name: front + image: gcr.io/bank-of-anthos/frontend:v0.3.1 + volumeMounts: + - name: publickey + mountPath: "/root/.ssh" + readOnly: true + env: + - name: VERSION + value: "v0.3.1" + - name: PORT + value: "8080" + - name: ENABLE_TRACING + value: "true" + - name: SCHEME + value: "http" + # Valid levels are debug, info, warning, error, critical. If no valid level is set, gunicorn will default to info. + - name: LOG_LEVEL + value: "info" + - name: DEFAULT_USERNAME + valueFrom: + configMapKeyRef: + name: demo-data-config + key: DEMO_LOGIN_USERNAME + - name: DEFAULT_PASSWORD + valueFrom: + configMapKeyRef: + name: demo-data-config + key: DEMO_LOGIN_PASSWORD + envFrom: + - configMapRef: + name: environment-config + - configMapRef: + name: service-api-config + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 5 + timeoutSeconds: 10 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + volumes: + - name: publickey + secret: + secretName: jwt-key + items: + - key: jwtRS256.key.pub + path: publickey +--- +apiVersion: v1 +kind: Service +metadata: + name: frontend +spec: + type: LoadBalancer + selector: + app: frontend + ports: + - name: http + port: 80 + targetPort: 8080 diff --git a/extras/cloudsql/kubernetes-manifests/jwt-secret.yaml b/extras/cloudsql/kubernetes-manifests/jwt-secret.yaml new file mode 100644 index 000000000..e885b95e3 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/jwt-secret.yaml @@ -0,0 +1,24 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This secret contains a keypair used to sign and verify JWTs for authentication +# In practice, this should never be checked into version control. It is provided here to simplify deployment +apiVersion: v1 +kind: Secret +metadata: + name: jwt-key +type: Opaque +data: + jwtRS256.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS3dJQkFBS0NBZ0VBNEd2Wk5Ta2o1aWlXYjJEWk0yZkN4NWJHR1lnd2FLNFNNc3lJOEs4b0ZRZXBMMTNZClJPWEt3NVMrbEVMSDlBZXRUZGNUQWRrYkN6eWFwU2RrM1Bxd3hoQ2FmNGtudktlUmk3NmNzRzZtYzBrcngxR0kKdFk2NVgrTnEwcWFSdkxkT09iRlFHbUlIeDc1emtaNjU0K2Z0dm52Q1k1T1BqVjQ0TGt0dTF3bHJCbXIyMjVRUwpGYklhOEE5NkJTcEp6S20wYWhLKzZyOFZ2dk9xMUJIR1pJSUtLR1E3L3pMdXhQT24rYzN3Q0YwVlFZMjBZNFhvCm8rRFZ0LyswTUk4T1p2ZEF1YWIyVVh1YitUaXdBZ0c3Njh5bkpQaDdla2xMdGR0QTFCR2dsWG1vRVI0eHZoaFgKUXNPUnRUaTYwRlpQWDdDM1YxcmxVZStyVTlCWTZDbHJXbGVKdGdCZjFYUi90Qm1MZGZCNFlWbkt0ekI3dkxDbApPS3VsdGFkdVNWbmZ2OEMwdXoxelBCWkl3WG5zVk83WHlTVlBBSkJUOFBWR1Iza2JnOW5FRGsvclZsS1lNSW5EClNPOFNITnlLTytrNnBUaG1hbW9xTUJKQkNwY0MrRzFVUFVxWWV1Szh3NjZaU3BLV2ZQY3RtWWViUHNraUpiS2UKT3JIVlUyRGo3enpkaW1MMmJlaHFTd1NpM3pBa2xXZjRGU0F6aC80enBENFdxM0lDd2NYbFBxYTU5T0IwcXFBQwp4NUFtRUZsdlF1VUkwcTZ3WjZ2cmxBWUsrTWRtNER2RkFVbFhzNW1mbDZPRnZ0YkZTZHF1ckU2SXROVnlSVlFBCmx6WFdtaEM4R3JXQklodDFPSkxXREtPNWx2Wm1KUS9sazdic1VyNCsyK05ObnNzTXBuTUI2QytJejZzQ0F3RUEKQVFLQ0FnRUFoQ0c3ZnpzN3NiVlA3VCsrWWkvRFZMUUc2dDF1Y3hPYUQyS3hkZFFLNjJiWlM5UWJFMldnWGtyagpjbk5uT3R5U1F2UlFjcFFiQWlDcWhIYVBRalF2aEU0ZVNOeStFcXN1WEhMYjJsMHJqMUVUZjRVWDBwMjNzeWM0CnE5SDZtSHZzaUxkSERHR1BaeUZROWZuWVJNRTNEeUhhcjdvZm1RSlNYWjk0SVErZ2ZlVFlkeXZWQWdVZEt4NVIKbjF6b21xR0lZYk9yeUphOC9QS0dMOXBQVHJucHRTSmUyTUJxS0hlMC9MVXRrQjhOOXN5eDJxN3E3Q0hpY3pVUQpLRTd5RWRtZHVERC9jYll2ZlNyTkprRlRIV0FmUVBzejZDbmpjNjBsbkRiTXIzVXphUEM0Sk00OEFra1hMdlc1CnVCUTVJNFB1VG5acjhqUUVVWktuNWZrTkJlSG9aNjlZVVp2eWdOcGJRaVMxWi9CRzYwVDVZN1VPY09wWVJ2bCsKL0dSNUFSNnpZQVJKbTBKNmg3U2NsdGhLSjdaVnFldWV5U3B1UG9mTjg0V241Zm1mbWdsR2F0dlc4KzFGR2ZNRQppWGxHbzhHQ1ZyampzN013a0pMT3Jlc0lRblFWK3ZmWC95Nk9WRExMS21yeGJCT3BPcm5TK01icnFoOXBsRGFICks1N0VOVngzWWxVd2M3bFRHd2Q2cS9nVnhSWDZRc1F3VEl5UHVOOUozMWd0dXRCeS9FTVVGc1hJenJhRmRIQzgKZXVlT1N3V0JQT05XbUkzN1BCVTIycXFyMWE3d1Y3Z0lJOGg3WS8waDNDYjFYWjZsQ2FzOCt0emI3VFZNMGxWQwpINWNYaXRqM2xKS1B1b2VzOXpTNFhEQjQ0V1hwZkw4SFVXQ1NuSWF2ZVJYUHhwbDRJbUVDZ2dFQkFQMjJEZml3CmVaRDBSTzlManJSR2cxODJlNnNSZ052VUQrOTFjUnBRNFBDcnlKQm1tTmNWOFBNZVdoZFlPaHdCWW1JMjkyWkkKcjFyeHVwclBCdE43dTNnWmhKYkVXL1IvSFozc2lPSmNycHlhejBVNUpDSzRTOERSbkdnUUdOQUdJNFlsVUNxNwpLL2JOa0JLcnRpSEVpOWV6RGQ3SjIrRGJGK0FiSXNWdlNnNnNHdEp1TG8rY0VRUzFaNzkycTQ5NUt6dWJyUDhvClVIbWFuc1R1ZEtLL1lZY0dlcU9pS2V0bmgrK0ZUc0hXek1NTEJ3VllySVlZRHZ5RWUvZEFkNWUzM3ZTRSsrbXkKZEVaVkZidVRMQmdzSW4wM0dmZ1pnNE56QkNhSm9NZGJtU3NiaUxGMkRTL1g4eXNKNldLYzdVTFMwMXJwSjBLTgo0VldQNjhiZlZOd1YwbmtDZ2dFQkFPSnlKaVN5aGJrL1hEVjRzT2ZLS1BBQ3RJTkN0ZWk2OWpYVVVOUHNhQndNCjFMNURCOU5tNjhLb3piVTN1Rk9QbmRPZERrMms5cm5WS1JkSlYrTHhPQ2NEdE5aMzlGQ3JtbUs3TGc4TnBUM2cKVksxc29YamFtcGI3NU1LWjZzaGFIUDRFUUMvTVVoeVF1UXkydEtUS3JMYUNFRmdzR1RhRHFaOU9xMVdIZzhWNwoxdUkzak8xYnNzNFRQWjJxbXVOYjE0WEMzQnJaeDE4eWY3MnJZMEk5RTAxdXorUmp4VUdSK0E3dlloSklYVWtMCmtzQTJqV1ludktsVmZRRWJpWlVpcjQ0d2IraEZ4Z3JwcVBvSmNCTzgvdS9UYXZ3MjFqbjFkRys3NWZqZnI5M3UKRjExbUh2VVY2Z0RJenN4NDVicDVDek5oUmdTTHNRZk9INFdpdytIcUNrTUNnZ0VCQU1UZGtFZkpmL3IzWDhvaQo0ZStHeTRlRStqOEtqT2VHekhxdHNYNlBCWXdhYjcyRXJ0SXV4MUdPMnE5RW1ZcURsSGpMVi9zNUtVQXpVKzJ0CitRaisrTCsvWlQrcnpBS1M1RU5YZC92Vm1QUVJ1QVZweWwxYWpnVm5ZS3JxMnFZSUxXWjQ2NVRNdWRkL09HMFUKYW5ZWFViK2t3MzE5T3ExbXRFY0VKMTMxOWd3ZWhZMkZTNEhKd3ZiWllGQ1QzNW1ybHQxZzljTGo0RllMb1drRAo4dU1hQjY1RFc4US9IN0gzR3ViRGxSSVovN2JVaHJVaVVuU1dsSUppdHVKclZxRVdYdDF6bHhtR2pHekt0dHRlCjNwYy9IOS96Y3FZdC9mdHdzdWJJWWtadkVCSUFBbmhaTnZCSjYva1liczFESVdONXZlRE1DOWU3eWY2ZmQzOVEKYWtPQWQxa0NnZ0VCQUpCQ2RraFVydHBJSWg4eWgwanpRa2M5QWZESlZBZ3k2MTE1cUJDS2YveTJzK1dONEhOdQpFdTROQ2hmVHFvc1phRXVDdDdVQlRla3ZnaUVDcVltN2NMRnlMQWVobmJTeXpnVHVDRGF3MDc5cXBhZGlHREJjCjI3VXFQaDgzWFJwTVJrSVJSUzd1TkxWY0FYZTNBYmdtSWdlWExvQnRmNVo2SkZxSURLRE5WMFk5VWJVRi9MTE8KQTBoRGU3SnhHSUdWWmVVaU1ZU3RqdFQzMytkZVF6clVtL0p0a09XSUpnZFN6YnYrQnZWc3dua2hkVmtjcDlJWApxSm9jQ3lua2VDN1pUUXdCa1psZ3NmMEx4SW1kNzVlemhKb1dqd2FraksrWnpwYk1Gb01KUmduVmYyOFIrdERhCktCeGQwVmowQUM1ZXBLTVQ2ajVGRy8vRDhkTDUwVjZmOGU4Q2dnRUJBTWgreklJb3FKZWlXTGgzOUFtTTRQZEkKSFNod0lISXBUNUhGS05jOTFzRnRDZFdXVDIweXVPZHNLZU9TTDZjMjJsVFpjcklFZFlDY0Q5SVhyWm8xVmxXOAorSGFmNDFhcHhlaU1PZVBNR3ZlNEQ2ZEdrUXcyWXBuNXlOQUUwYk1qa1hFU3JuMzMvV3JqSFFtWFdvNmFSdnVZCnkrMktzbGwvK1BzajlwNXd3WVYxRGp0QzFXU0lmdFZNMkJuRlViQVVyMmNHSVNXWlNnWU9yUDU1elZKK3REeloKSXhrMDZ6TUNZU3pEL1pPTUJZSjUvTFFPWnlPUUs0R2xnMytUUUNXUERHNTRtUEo0dElmNUZCZ25wSU5QRGNLQwpLc2RTQlg4S0xNeU83aEVzOCtuZFRWNWlxa0pscGNCdVJ3VXVUSmlFakc0MWZnaXZPVThTR0V5elZPbW1NNG89Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== + jwtRS256.key.pub: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUE0R3ZaTlNrajVpaVdiMkRaTTJmQwp4NWJHR1lnd2FLNFNNc3lJOEs4b0ZRZXBMMTNZUk9YS3c1UytsRUxIOUFldFRkY1RBZGtiQ3p5YXBTZGszUHF3CnhoQ2FmNGtudktlUmk3NmNzRzZtYzBrcngxR0l0WTY1WCtOcTBxYVJ2TGRPT2JGUUdtSUh4NzV6a1o2NTQrZnQKdm52Q1k1T1BqVjQ0TGt0dTF3bHJCbXIyMjVRU0ZiSWE4QTk2QlNwSnpLbTBhaEsrNnI4VnZ2T3ExQkhHWklJSwpLR1E3L3pMdXhQT24rYzN3Q0YwVlFZMjBZNFhvbytEVnQvKzBNSThPWnZkQXVhYjJVWHViK1Rpd0FnRzc2OHluCkpQaDdla2xMdGR0QTFCR2dsWG1vRVI0eHZoaFhRc09SdFRpNjBGWlBYN0MzVjFybFVlK3JVOUJZNkNscldsZUoKdGdCZjFYUi90Qm1MZGZCNFlWbkt0ekI3dkxDbE9LdWx0YWR1U1ZuZnY4QzB1ejF6UEJaSXdYbnNWTzdYeVNWUApBSkJUOFBWR1Iza2JnOW5FRGsvclZsS1lNSW5EU084U0hOeUtPK2s2cFRobWFtb3FNQkpCQ3BjQytHMVVQVXFZCmV1Szh3NjZaU3BLV2ZQY3RtWWViUHNraUpiS2VPckhWVTJEajd6emRpbUwyYmVocVN3U2kzekFrbFdmNEZTQXoKaC80enBENFdxM0lDd2NYbFBxYTU5T0IwcXFBQ3g1QW1FRmx2UXVVSTBxNndaNnZybEFZSytNZG00RHZGQVVsWApzNW1mbDZPRnZ0YkZTZHF1ckU2SXROVnlSVlFBbHpYV21oQzhHcldCSWh0MU9KTFdES081bHZabUpRL2xrN2JzClVyNCsyK05ObnNzTXBuTUI2QytJejZzQ0F3RUFBUT09Ci0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQo= diff --git a/extras/cloudsql/kubernetes-manifests/ledger-writer.yaml b/extras/cloudsql/kubernetes-manifests/ledger-writer.yaml new file mode 100644 index 000000000..5f871a9db --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/ledger-writer.yaml @@ -0,0 +1,113 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ledgerwriter +spec: + selector: + matchLabels: + app: ledgerwriter + template: + metadata: + labels: + app: ledgerwriter + spec: + serviceAccountName: boa-ksa + terminationGracePeriodSeconds: 5 + containers: + - name: ledgerwriter + image: gcr.io/bank-of-anthos/ledgerwriter:v0.3.1 + volumeMounts: + - name: publickey + mountPath: "/root/.ssh" + readOnly: true + env: + - name: VERSION + value: "v0.3.1" + - name: PORT + value: "8080" + - name: ENABLE_TRACING + value: "true" + - name: ENABLE_METRICS + value: "true" + # tell Java to obey container memory limits + - name: JVM_OPTS + value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap" + # service level override of log level + - name: LOG_LEVEL + value: "info" + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + envFrom: + - configMapRef: + name: environment-config + - configMapRef: + name: service-api-config + # add ledger-db credentials from ConfigMap + - configMapRef: + name: ledger-db-config + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + cpu: 500m + memory: 1Gi + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 60 + periodSeconds: 5 + timeoutSeconds: 10 + - name: cloudsql-proxy + resources: + limits: + cpu: "200m" + memory: "100Mi" + image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine + env: + - name: CONNECTION_NAME + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: connectionName + command: ["/cloud_sql_proxy", + "-instances=$(CONNECTION_NAME)=tcp:5432"] + securityContext: + runAsNonRoot: true + volumes: + - name: publickey + secret: + secretName: jwt-key + items: + - key: jwtRS256.key.pub + path: publickey +--- +apiVersion: v1 +kind: Service +metadata: + name: ledgerwriter +spec: + type: ClusterIP + selector: + app: ledgerwriter + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/extras/cloudsql/kubernetes-manifests/loadgenerator.yaml b/extras/cloudsql/kubernetes-manifests/loadgenerator.yaml new file mode 100644 index 000000000..563fcf763 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/loadgenerator.yaml @@ -0,0 +1,49 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: loadgenerator +spec: + selector: + matchLabels: + app: loadgenerator + replicas: 1 + template: + metadata: + labels: + app: loadgenerator + annotations: + sidecar.istio.io/rewriteAppHTTPProbers: "true" + spec: + serviceAccountName: boa-ksa + terminationGracePeriodSeconds: 5 + restartPolicy: Always + containers: + - name: loadgenerator + image: gcr.io/bank-of-anthos/loadgenerator:v0.3.1 + env: + - name: FRONTEND_ADDR + value: "frontend:80" + - name: USERS + value: "5" + - name: LOG_LEVEL + value: "error" + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + cpu: 500m + memory: 1Gi diff --git a/extras/cloudsql/kubernetes-manifests/populate-accounts-db.yaml b/extras/cloudsql/kubernetes-manifests/populate-accounts-db.yaml new file mode 100644 index 000000000..cf54d8040 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/populate-accounts-db.yaml @@ -0,0 +1,337 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: batch/v1 +kind: Job +metadata: + name: populate-accounts-db +spec: + template: + spec: + shareProcessNamespace: true # Important to stop all processes + serviceAccountName: boa-ksa + containers: + - name: sidecar-controller + image: bash + command: ['bash', '-c', '. /scripts/wait-to-complete-sidecar.sh "initialize-database.sh" "cloud_sql_proxy"'] + volumeMounts: + - name: scripts + mountPath: "/scripts" + readOnly: true + resources: + limits: + cpu: "200m" + memory: "100Mi" + - name: populate-accounts-db + image: postgres:13.0 + command: ['bash', '-c','. /scripts/initialize-database.sh 127.0.0.1 5432 accounts-db'] + volumeMounts: + - name: scripts + mountPath: "/scripts" + readOnly: true + env: + - name: PGUSER + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: username + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: password + # /start: Required for testing data + - name: LOCAL_ROUTING_NUM + valueFrom: + configMapKeyRef: + name: environment-config + key: LOCAL_ROUTING_NUM + - name: USE_DEMO_DATA + valueFrom: + configMapKeyRef: + name: demo-data-config + key: USE_DEMO_DATA + - name: POSTGRES_DB + value: "accounts-db" + - name: PGHOSTADDR + value: "127.0.0.1" + - name: POSTGRES_USER + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: username + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: password + # /end: Required for testing data + # CloudSQL Proxy + - name: cloudsql-proxy + resources: + limits: + cpu: "200m" + memory: "100Mi" + image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine + env: + - name: CONNECTION_NAME + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: connectionName + command: ["/cloud_sql_proxy", + "-instances=$(CONNECTION_NAME)=tcp:5432"] + securityContext: + runAsNonRoot: true + volumes: + - name: scripts + configMap: + name: accounts-schema-config + restartPolicy: Never + backoffLimit: 4 + +--- +# kubectl create configmap accounts-schema-config --from-file=src/accounts-db/initdb/0-accounts-schema.sql --dry-run -o yaml (copy and add below) +# kubectl create configmap accounts-schema-config --from-file=src/accounts-db/initdb/1-load-testdata.sh --dry-run -o yaml (copy and add below) +apiVersion: v1 +kind: ConfigMap +metadata: + name: accounts-schema-config +data: + wait-to-complete-sidecar.sh: | + #!/bin/bash + COUNTER=0 + TARGET=$1 + PROCESS=$2 + + echo "Looking for $TARGET" + + sleep 10s # Give 10s to allow processes to start + + # 20 minutes (60 x 10s increments) + while [ $COUNTER -lt 120 ]; do + let COUNTER=$COUNTER+1 + + IS_RUNNING=$(ps -A | grep "scripts/$TARGET" | grep -v grep) + + if [ "$IS_RUNNING" != "" ]; then + echo "Attempt # ${COUNTER}: Process not completed, trying again in 10 seconds -- ${IS_RUNNING}" + sleep 10s + else + echo "'${TARGET}' Process Finished, Stopping '${PROCESS}'" + killall ${PROCESS} + exit 0 + fi + done + + echo "Could not determine if the import finished, killing the proxy" + killall ${PROCESS} + exit 0 + + initialize-database.sh: | + #!/bin/bash + + COUNTER=0 + SLEEP_TIME=60 # 1 minute + DB_READY=0 # false + INIT_SQL_SCRIPT="/scripts/0-accounts-schema.sql" + TEST_SQL_SCRIPT="/scripts/1-load-testdata.sh" + + HOST=${1:-'127.0.0.1'} + PORT=${2:-'5432'} + DB_NAME=${3:-'default'} + + # Initial wait for sql-proxy to catch up + sleep 20 + + while [ $COUNTER -lt 10 ]; do + let COUNTER=$COUNTER+1 + + pg_isready --host=${HOST} --port=${PORT} --dbname=${DB_NAME} + + if [ $? -gt 0 ]; then + echo "Attempt # ${COUNTER}: Database is not ready, trying again in 1 minute" + sleep $SLEEP_TIME + else + echo "Database is ready to connect" + let DB_READY=1 + break + fi + done + + if [ "${DB_READY}" -eq 1 ]; then + echo "Running initialization script" + psql --host=${HOST} --port=${PORT} --dbname=${DB_NAME} -f ${INIT_SQL_SCRIPT} + if [ $? -gt 0 ]; then + echo "Problems running the initialization script" + else + echo "Run Test Data" + . ${TEST_SQL_SCRIPT} + fi + fi + 0-accounts-schema.sql: | + /* + * Copyright 2020, Google LLC. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + CREATE TABLE IF NOT EXISTS users ( + accountid CHAR(10) PRIMARY KEY, + username VARCHAR(64) UNIQUE NOT NULL, + passhash BYTEA NOT NULL, + firstname VARCHAR(64) NOT NULL, + lastname VARCHAR(64) NOT NULL, + birthday DATE NOT NULL, + timezone VARCHAR(8) NOT NULL, + address VARCHAR(64) NOT NULL, + state CHAR(2) NOT NULL, + zip VARCHAR(5) NOT NULL, + ssn CHAR(11) NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_users_accountid ON users (accountid); + CREATE INDEX IF NOT EXISTS idx_users_username ON users (username); + + + + CREATE TABLE IF NOT EXISTS contacts ( + username VARCHAR(64) NOT NULL, + label VARCHAR(128) NOT NULL, + account_num CHAR(10) NOT NULL, + routing_num CHAR(9) NOT NULL, + is_external BOOLEAN NOT NULL, + FOREIGN KEY (username) REFERENCES users(username) + ); + + CREATE INDEX IF NOT EXISTS idx_contacts_username ON contacts (username); + 1-load-testdata.sh: | + #!/bin/bash + # Copyright 2020 Google LLC + # + # Licensed under the Apache License, Version 2.0 (the "License"); + # you may not use this file except in compliance with the License. + # You may obtain a copy of the License at + # + # http://www.apache.org/licenses/LICENSE-2.0 + # + # Unless required by applicable law or agreed to in writing, software + # distributed under the License is distributed on an "AS IS" BASIS, + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + # See the License for the specific language governing permissions and + # limitations under the License. + + # Immediately exit if any error occurs during script execution. + set -o errexit + + # Skip adding data if not enabled + if [ "$USE_DEMO_DATA" != "True" ]; then + echo "no demo users added" + exit 0 + fi + + + # Expected environment variables + readonly ENV_VARS=( + "POSTGRES_DB" + "POSTGRES_USER" + "LOCAL_ROUTING_NUM" + ) + + + add_user() { + # Usage: add_user "ACCOUNTID" "USERNAME" "FIRST_NAME" + echo "adding user: $2" + psql -X -v ON_ERROR_STOP=1 -v account="$1" -v username="$2" -v firstname="$3" -v passhash="$DEFAULT_PASSHASH" --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL + INSERT INTO users VALUES (:'account', :'username', :'passhash', :'firstname', 'User', '2000-01-01', '-5', 'Bowling Green, New York City', 'NY', '10004', '111-22-3333') ON CONFLICT DO NOTHING; + EOSQL + } + + + add_external_account() { + # Usage: add_external_account "OWNER_USERNAME" "LABEL" "ACCOUNT" "ROUTING" + echo "user $1 adding contact: $2" + psql -X -v ON_ERROR_STOP=1 -v username="$1" -v label="$2" -v account="$3" -v routing="$4" --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL + INSERT INTO contacts VALUES (:'username', :'label', :'account', :'routing', 'true') ON CONFLICT DO NOTHING; + EOSQL + } + + + add_contact() { + # Usage: add_contact "OWNER_USERNAME" "CONTACT_LABEL" "CONTACT_ACCOUNT" + echo "user $1 adding external account: $2" + psql -X -v ON_ERROR_STOP=1 -v username="$1" -v label="$2" -v account="$3" -v routing="$LOCAL_ROUTING_NUM" --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL + INSERT INTO contacts VALUES (:'username', :'label', :'account', :'routing', 'false') ON CONFLICT DO NOTHING; + EOSQL + } + + + # Load test data into the database + create_accounts() { + # Add demo users. + add_user "1011226111" "testuser" "Test" + add_user "1033623433" "alice" "Alice" + add_user "1055757655" "bob" "Bob" + add_user "1077441377" "eve" "Eve" + + # Make everyone contacts of each other + add_contact "testuser" "Alice" "1033623433" + add_contact "testuser" "Bob" "1055757655" + add_contact "testuser" "Eve" "1077441377" + add_contact "alice" "Testuser" "1011226111" + add_contact "alice" "Bob" "1055757655" + add_contact "alice" "Eve" "1077441377" + add_contact "bob" "Testuser" "1011226111" + add_contact "bob" "Alice" "1033623433" + add_contact "bob" "Eve" "1077441377" + add_contact "eve" "Testuser" "1011226111" + add_contact "eve" "Alice" "1033623433" + add_contact "eve" "Bob" "1055757655" + + # Add external accounts + add_external_account "testuser" "External Bank" "9099791699" "808889588" + add_external_account "alice" "External Bank" "9099791699" "808889588" + add_external_account "bob" "External Bank" "9099791699" "808889588" + add_external_account "eve" "External Bank" "9099791699" "808889588" + } + + + main() { + # Check environment variables are set + for env_var in ${ENV_VARS[@]}; do + if [[ -z "${!env_var}" ]]; then + echo "Error: environment variable '$env_var' not set. Aborting." + exit 1 + fi + done + + # A password hash + salt for the demo password 'password' + # Via Python3: bycrypt.hashpw('password'.encode('utf-8'), bcrypt.gensalt()) + DEFAULT_PASSHASH='\x243262243132245273764737474f39777562452f4a786a79444a7263756f386568466b762e634e5262356e6867746b474752584c6634437969346643' + + create_accounts + } + + + main + diff --git a/extras/cloudsql/kubernetes-manifests/populate-ledger-db.yaml b/extras/cloudsql/kubernetes-manifests/populate-ledger-db.yaml new file mode 100644 index 000000000..55838244f --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/populate-ledger-db.yaml @@ -0,0 +1,335 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: batch/v1 +kind: Job +metadata: + name: populate-ledger-db +spec: + template: + spec: + shareProcessNamespace: true # Important to stop all processes + serviceAccountName: boa-ksa + containers: + - name: sidecar-controller + image: bash + command: ['bash', '-c', '. /scripts/wait-to-complete-sidecar.sh "initialize-database.sh" "cloud_sql_proxy"'] + volumeMounts: + - name: scripts + mountPath: "/scripts" + readOnly: true + resources: + limits: + cpu: "200m" + memory: "100Mi" + - name: populate-ledger-db + image: postgres:13.0 + command: ['bash', '-c','. /scripts/initialize-database.sh 127.0.0.1 5432 ledger-db'] + volumeMounts: + - name: scripts + mountPath: "/scripts" + readOnly: true + env: + - name: PGUSER + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: username + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: password + # /start: Required for testing data + - name: LOCAL_ROUTING_NUM + valueFrom: + configMapKeyRef: + name: environment-config + key: LOCAL_ROUTING_NUM + - name: USE_DEMO_DATA + valueFrom: + configMapKeyRef: + name: demo-data-config + key: USE_DEMO_DATA + - name: POSTGRES_DB + value: "ledger-db" + - name: PGHOSTADDR + value: "127.0.0.1" + - name: POSTGRES_USER + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: username + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: password + # /end: Required for testing data + # CloudSQL Proxy + - name: cloudsql-proxy + resources: + limits: + cpu: "200m" + memory: "100Mi" + image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine + env: + - name: CONNECTION_NAME + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: connectionName + command: ["/cloud_sql_proxy", + "-instances=$(CONNECTION_NAME)=tcp:5432"] + securityContext: + runAsNonRoot: true + volumes: + - name: scripts + configMap: + name: ledger-schema-config + restartPolicy: Never + backoffLimit: 4 + +--- +# kubectl create configmap ledger-schema-config --from-file=src/ledger-db/initdb/0-ledger-schema.sql --dry-run -o yaml (copy and add below) +# kubectl create configmap ledger-schema-config --from-file=src/ledger-db/initdb/1-load-testdata.sh --dry-run -o yaml (copy and add below) +apiVersion: v1 +kind: ConfigMap +metadata: + name: ledger-schema-config +data: + wait-to-complete-sidecar.sh: | + #!/bin/bash + COUNTER=0 + TARGET=$1 + PROCESS=$2 + + echo "Looking for $TARGET" + + sleep 10s # Give 10s to allow processes to start + + # 20 minutes (60 x 10s increments) + while [ $COUNTER -lt 120 ]; do + let COUNTER=$COUNTER+1 + + IS_RUNNING=$(ps -A | grep "scripts/$TARGET" | grep -v grep) + + if [ "$IS_RUNNING" != "" ]; then + echo "Attempt # ${COUNTER}: Process not completed, trying again in 10 seconds -- ${IS_RUNNING}" + sleep 10s + else + echo "'${TARGET}' Process Finished, Stopping '${PROCESS}'" + killall ${PROCESS} + exit 0 + fi + done + + echo "Could not determine if the import finished, killing the proxy" + killall ${PROCESS} + exit 0 + + initialize-database.sh: | + #!/bin/bash + + COUNTER=0 + SLEEP_TIME=60 # 1 minute + DB_READY=0 # false + INIT_SQL_SCRIPT="/scripts/0-ledger-schema.sql" + TEST_SQL_SCRIPT="/scripts/1-load-testdata.sh" + + HOST=${1:-'127.0.0.1'} + PORT=${2:-'5432'} + DB_NAME=${3:-'default'} + + # Initial wait for sql-proxy to catch up + sleep 20 + + while [ $COUNTER -lt 10 ]; do + let COUNTER=$COUNTER+1 + + pg_isready --host=${HOST} --port=${PORT} --dbname=${DB_NAME} + + if [ $? -gt 0 ]; then + echo "Attempt # ${COUNTER}: Database is not ready, trying again in 1 minute" + sleep $SLEEP_TIME + else + echo "Database is ready to connect" + let DB_READY=1 + break + fi + done + + if [ "${DB_READY}" -eq 1 ]; then + echo "Running initialization script" + psql --host=${HOST} --port=${PORT} --dbname=${DB_NAME} -f ${INIT_SQL_SCRIPT} + if [ $? -gt 0 ]; then + echo "Problems running the initialization script" + else + echo "Run Test Data" + . ${TEST_SQL_SCRIPT} + fi + fi + 0-ledger-schema.sql: | + /* + * Copyright 2020, Google LLC. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + CREATE TABLE TRANSACTIONS ( + TRANSACTION_ID BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + FROM_ACCT CHAR(10) NOT NULL, + TO_ACCT CHAR(10) NOT NULL, + FROM_ROUTE CHAR(9) NOT NULL, + TO_ROUTE CHAR(9) NOT NULL, + AMOUNT INT NOT NULL, + TIMESTAMP TIMESTAMP NOT NULL + ); + -- index account number/routing number pairs + CREATE INDEX ON TRANSACTIONS (FROM_ACCT, FROM_ROUTE, TIMESTAMP); + CREATE INDEX ON TRANSACTIONS (TO_ACCT, TO_ROUTE, TIMESTAMP); + -- append only ledger; prevent updates or deletes + CREATE RULE PREVENT_UPDATE AS + ON UPDATE TO TRANSACTIONS + DO INSTEAD NOTHING; + CREATE RULE PREVENT_DELETE AS + ON DELETE TO TRANSACTIONS + DO INSTEAD NOTHING; + 1-load-testdata.sh: | + #!/bin/bash + # Copyright 2020 Google LLC + # + # Licensed under the Apache License, Version 2.0 (the "License"); + # you may not use this file except in compliance with the License. + # You may obtain a copy of the License at + # + # http://www.apache.org/licenses/LICENSE-2.0 + # + # Unless required by applicable law or agreed to in writing, software + # distributed under the License is distributed on an "AS IS" BASIS, + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + # See the License for the specific language governing permissions and + # limitations under the License. + + + # Create demo transactions in the ledger for the demo user accounts. + # + # Gerenated transactions follow a pattern of biweekly large deposits with + # periodic small payments to randomly choosen accounts. + # + # To run, set environment variable USE_DEMO_DATA="True" + + set -u + + + # skip adding transactions if not enabled + if [ -z "$USE_DEMO_DATA" ] && [ "$USE_DEMO_DATA" != "True" ]; then + echo "\$USE_DEMO_DATA not \"True\"; no demo transactions added" + exit 0 + fi + + + # Expected environment variables + readonly ENV_VARS=( + "POSTGRES_DB" + "POSTGRES_USER" + "LOCAL_ROUTING_NUM" + ) + + + add_transaction() { + DATE=$(date -u +"%Y-%m-%d %H:%M:%S.%3N%z" --date="@$(($6))") + echo "adding demo transaction: $1 -> $2" + psql -X -v ON_ERROR_STOP=1 -v fromacct="$1" -v toacct="$2" -v fromroute="$3" -v toroute="$4" -v amount="$5" --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL + INSERT INTO TRANSACTIONS (FROM_ACCT, TO_ACCT, FROM_ROUTE, TO_ROUTE, AMOUNT, TIMESTAMP) + VALUES (:'fromacct', :'toacct', :'fromroute', :'toroute', :'amount', '$DATE'); + EOSQL + } + + + create_transactions() { + PAY_PERIODS=3 + DAYS_BETWEEN_PAY=14 + SECONDS_IN_PAY_PERIOD=$(( 86400 * $DAYS_BETWEEN_PAY )) + DEPOSIT_AMOUNT=250000 + + # create a UNIX timestamp in seconds since the Epoch + START_TIMESTAMP=$(( $(date +%s) - $(( $(($PAY_PERIODS+1)) * $SECONDS_IN_PAY_PERIOD )) )) + + for i in $(seq 1 $PAY_PERIODS); do + # create deposit transaction for each user + for account in ${USER_ACCOUNTS[@]}; do + add_transaction "$EXTERNAL_ACCOUNT" "$account" "$EXTERNAL_ROUTING" "$LOCAL_ROUTING_NUM" $DEPOSIT_AMOUNT $START_TIMESTAMP + done + + # create 15-20 payments between users + TRANSACTIONS_PER_PERIOD=$(shuf -i 15-20 -n1) + for p in $(seq 1 $TRANSACTIONS_PER_PERIOD); do + # randomly generate an amount between $10-$100 + AMOUNT=$(shuf -i 1000-10000 -n1) + + # randomly select a sender and receiver + SENDER_ACCOUNT=${USER_ACCOUNTS[$RANDOM % ${#USER_ACCOUNTS[@]}]} + RECIPIENT_ACCOUNT=${USER_ACCOUNTS[$RANDOM % ${#USER_ACCOUNTS[@]}]} + # if sender equals receiver, send to a random anonymous account + if [[ "$SENDER_ACCOUNT" == "$RECIPIENT_ACCOUNT" ]]; then + RECIPIENT_ACCOUNT=$(shuf -i 1000000000-9999999999 -n1) + fi + + TIMESTAMP=$(( $START_TIMESTAMP + $(( $SECONDS_IN_PAY_PERIOD * $p / $(($TRANSACTIONS_PER_PERIOD + 1 )) )) )) + + add_transaction "$SENDER_ACCOUNT" "$RECIPIENT_ACCOUNT" "$LOCAL_ROUTING_NUM" "$LOCAL_ROUTING_NUM" $AMOUNT $TIMESTAMP + done + + START_TIMESTAMP=$(( $START_TIMESTAMP + $(( $i * $SECONDS_IN_PAY_PERIOD )) )) + done + } + + + create_ledger() { + # Account numbers for users 'testuser', 'alice', 'bob', and 'eve'. + USER_ACCOUNTS=("1011226111" "1033623433" "1055757655" "1077441377") + # Numbers for external account 'External Bank' + EXTERNAL_ACCOUNT="9099791699" + EXTERNAL_ROUTING="808889588" + + create_transactions + } + + + main() { + # Check environment variables are set + for env_var in ${ENV_VARS[@]}; do + if [[ -z "${env_var}" ]]; then + echo "Error: environment variable '$env_var' not set. Aborting." + exit 1 + fi + done + + create_ledger + } + + + main + + diff --git a/extras/cloudsql/kubernetes-manifests/transaction-history.yaml b/extras/cloudsql/kubernetes-manifests/transaction-history.yaml new file mode 100644 index 000000000..cfdc37803 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/transaction-history.yaml @@ -0,0 +1,129 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: transactionhistory +spec: + selector: + matchLabels: + app: transactionhistory + template: + metadata: + labels: + app: transactionhistory + spec: + serviceAccountName: boa-ksa + terminationGracePeriodSeconds: 5 + containers: + - name: transactionhistory + image: gcr.io/bank-of-anthos/transactionhistory:v0.3.1 + volumeMounts: + - name: publickey + mountPath: "/root/.ssh" + readOnly: true + env: + - name: VERSION + value: "v0.3.1" + - name: PORT + value: "8080" + - name: ENABLE_TRACING + value: "true" + - name: ENABLE_METRICS + value: "true" + - name: POLL_MS + value: "100" + - name: CACHE_SIZE + value: "1000" + - name: CACHE_MINUTES + value: "60" + - name: HISTORY_LIMIT + value: "100" + # tell Java to obey container memory limits + - name: JVM_OPTS + value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap" + #- name: EXTRA_LATENCY_MILLIS + # value: "5000" + # Valid levels are debug, info, warn, error, fatal. + # If no valid level is set, will default to info. + - name: LOG_LEVEL + value: "info" + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + envFrom: + - configMapRef: + name: environment-config + # add ledger-db credentials from ConfigMap + - configMapRef: + name: ledger-db-config + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + cpu: 500m + memory: 1Gi + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 60 + periodSeconds: 5 + timeoutSeconds: 10 + livenessProbe: + httpGet: + path: /healthy + port: 8080 + initialDelaySeconds: 120 + periodSeconds: 5 + timeoutSeconds: 10 + - name: cloudsql-proxy + resources: + limits: + cpu: "200m" + memory: "100Mi" + image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine + env: + - name: CONNECTION_NAME + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: connectionName + command: ["/cloud_sql_proxy", + "-instances=$(CONNECTION_NAME)=tcp:5432"] + securityContext: + runAsNonRoot: true + volumes: + - name: publickey + secret: + secretName: jwt-key + items: + - key: jwtRS256.key.pub + path: publickey +--- +apiVersion: v1 +kind: Service +metadata: + name: transactionhistory +spec: + type: ClusterIP + selector: + app: transactionhistory + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/extras/cloudsql/kubernetes-manifests/userservice.yaml b/extras/cloudsql/kubernetes-manifests/userservice.yaml new file mode 100644 index 000000000..8b82351d7 --- /dev/null +++ b/extras/cloudsql/kubernetes-manifests/userservice.yaml @@ -0,0 +1,112 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: userservice +spec: + selector: + matchLabels: + app: userservice + template: + metadata: + labels: + app: userservice + spec: + serviceAccountName: boa-ksa + terminationGracePeriodSeconds: 5 + containers: + - name: userservice + image: gcr.io/bank-of-anthos/userservice:v0.3.1 + volumeMounts: + - name: keys + mountPath: "/root/.ssh" + readOnly: true + ports: + - name: http-server + containerPort: 8080 + env: + - name: VERSION + value: "v0.3.0" + - name: PORT + value: "8080" + - name: ENABLE_TRACING + value: "true" + - name: TOKEN_EXPIRY_SECONDS + value: "3600" + - name: PRIV_KEY_PATH + value: "/root/.ssh/privatekey" + # Valid levels are debug, info, warning, error, critical. If no valid level is set, gunicorn will default to info. + - name: LOG_LEVEL + value: "info" + envFrom: + - configMapRef: + name: environment-config + - configMapRef: + name: accounts-db-config + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 5 + timeoutSeconds: 10 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + - name: cloudsql-proxy + resources: + limits: + cpu: "200m" + memory: "100Mi" + image: gcr.io/cloudsql-docker/gce-proxy:1.19.0-alpine + env: + - name: CONNECTION_NAME + valueFrom: + secretKeyRef: + name: cloud-sql-admin + key: connectionName + command: ["/cloud_sql_proxy", + "-instances=$(CONNECTION_NAME)=tcp:5432"] + securityContext: + runAsNonRoot: true + volumes: + - name: keys + secret: + secretName: jwt-key + items: + - key: jwtRS256.key + path: privatekey + - key: jwtRS256.key.pub + path: publickey + + +--- +apiVersion: v1 +kind: Service +metadata: + name: userservice +spec: + type: ClusterIP + selector: + app: userservice + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/extras/cloudsql/setup_workload_identity.sh b/extras/cloudsql/setup_workload_identity.sh new file mode 100755 index 000000000..59c1b6c40 --- /dev/null +++ b/extras/cloudsql/setup_workload_identity.sh @@ -0,0 +1,53 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# !/bin/bash + + +export KSA_NAME="boa-ksa" +export GSA_NAME="boa-gsa" + +echo "✅ Creating namespace..." +kubectl create namespace $NAMESPACE + +echo "✅ Creating GCP and K8s service accounts..." +kubectl create serviceaccount --namespace $NAMESPACE $KSA_NAME +gcloud iam service-accounts create $GSA_NAME + + +echo "✅ Annotating service accounts to connect your GSA and KSA..." +gcloud iam service-accounts add-iam-policy-binding \ + --role roles/iam.workloadIdentityUser \ + --member "serviceAccount:${PROJECT_ID}.svc.id.goog[$NAMESPACE/$KSA_NAME]" \ + $GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com + +kubectl annotate serviceaccount \ + --namespace $NAMESPACE \ + $KSA_NAME \ + iam.gke.io/gcp-service-account=$GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com + + +echo "✅ Granting Service account permissions..." +gcloud projects add-iam-policy-binding ${PROJECT_ID} \ + --member "serviceAccount:${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \ + --role roles/cloudtrace.agent + +gcloud projects add-iam-policy-binding ${PROJECT_ID} \ + --member "serviceAccount:${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \ + --role roles/monitoring.metricWriter + +gcloud projects add-iam-policy-binding ${PROJECT_ID} \ + --member "serviceAccount:${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \ + --role roles/cloudsql.client + +echo "⭐️ Done." \ No newline at end of file