Add CloudSQL option for accounts-db, ledger-db

[cloud-pharaoh]

Change summary: Adds a Cloud SQL option for the in-cluster Postgres ledger-db and accounts-db.

Added:

• setup script for the Cloud SQL instance
• setup script for workload identity configuring Cloud SQL access from pods
• two K8s jobs to populate Cloud SQL DBs with test data - source: Google Cloud Solutions team
• a version of the k8s deployments with cloud SQL proxy sidecar containers
• README with setup steps
• link from main README to the cloudsql directory

Follow-up work (filed issues for these).

• Add instructions without Workload Identity
• Deploy one cloud sql proxy to be accessed by both services (suggestion by @bharathkkb)
• Add tests for the Cloud SQL option, tests should run the setup scripts.
• Add kustomize to each deploy option (default, workload identity, cloud sql, ledger monolith) to avoid duplicating manifests within the repo, causing the YAML to get out of sync with the latest releases
• Test with ASM-enabled cluster to verify Cloud SQL proxy -> Cloud SQL connectivity when there is another sidecar container in the pod (Envoy)

[github-actions]

🚀 PR staged at http://35.223.94.0

[askmeegs]

Hi Amina! This looks good, I got the Cloud SQL instructions working on my ASM-enabled WI cluster.
A few small readme comments, otherwise I have only 1 outstanding concern which is -- the in-cluster postgres accounts db loads in test data (here) with a test user and password / prepopulates contacts.

Ideally we have consistency between the in-cluster and Cloud SQL postgres DBs. Not sure how to turn the load_testdata script into a set of raw SQL commands. But ideally we add this data in the step when the user creates the tables manually.

[mathieu-benoit]

Hi @askmeegs and @cloud-pharaoh, I think that's indeed a great idea to have an option to leverage Cloud SQL here.

I may have few questions:

• I think this command gcloud services enable sqladmin.googleapis.com in your instructions is missing otherwise the cloud-sql-proxy containers will complain. At least I needed to do this to make them running properly.
• Any chance to leverage the new and smaller gcr.io/cloudsql-docker/gce-proxy:1.18.0-alpine image instead of gcr.io/cloudsql-docker/gce-proxy:1.17?
• Since recently, the cloudsql-proxy can now run as non root like illustrated here: https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/examples/kubernetes/proxy_with_workload_identity.yaml#L53. Is it possible to include this in your 2 files userservice.yaml and contacts.yaml?
• I'm wondering, is it possible to move the ledger-db in Cloud SQL too? I think that will complete this PR to get rid of the postgresql as containers and have a more robust and cloud native way to deal with postgresql.
• Regarding the initialization of the accounts-db with https://github.com/GoogleCloudPlatform/bank-of-anthos/tree/master/src/accounts-db/initdb, any recommendation/documentation to accomplish that with Cloud SQL? I think this point is related to what @askmeegs just mentioned above.

Just sharing and wondering, thanks!

[github-actions]

🚀 PR staged at http://35.223.94.0

[cloud-pharaoh]

Hi @askmeegs and @cloud-pharaoh, I think that's indeed a great idea to have an option to leverage Cloud SQL here.

Hi Mathieu,
Thanks for these comments and questions :)

I may have few questions:

• I think this command gcloud services enable sqladmin.googleapis.com in your instructions is missing otherwise the cloud-sql-proxy containers will complain. At least I needed to do this to make them running properly.

Good catch. I do that in terraform in my test set up and I didn't realize it wasn't here. Added to the instructions.

• Any chance to leverage the new and smaller gcr.io/cloudsql-docker/gce-proxy:1.18.0-alpine image instead of gcr.io/cloudsql-docker/gce-proxy:1.17?

No reason why we cannot. Just needs to be tested to make sure all is good. @askmeegs

• Since recently, the cloudsql-proxy can now run as non root like illustrated here: https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/examples/kubernetes/proxy_with_workload_identity.yaml#L53. Is it possible to include this in your 2 files userservice.yaml and contacts.yaml?

The comment seems to imply that is the default behavior or am I misunderstanding?

• I'm wondering, is it possible to move the ledger-db in Cloud SQL too? I think that will complete this PR to get rid of the postgresql as containers and have a more robust and cloud native way to deal with postgresql.

Very possible. I just wanted to show an example with one of them. @askmeegs up to you now Megan if you want to do it later or as part of this PR.

• Regarding the initialization of the accounts-db with https://github.com/GoogleCloudPlatform/bank-of-anthos/tree/master/src/accounts-db/initdb, any recommendation/documentation to accomplish that with Cloud SQL? I think this point is related to what @askmeegs just mentioned above.

A k8s Job would be a good way to run this initialization script.

Just sharing and wondering, thanks!

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[github-actions]

🚀 PR staged at http://35.223.94.0

[mathieu-benoit]

LGTM

[github-actions]

🚀 PR staged at http://35.223.94.0