IaC: Terraform and ACM configurations for Multi-Autopilot GKE environment setup with CI/CD, Anthos & CloudSQL #1006

aablsk · 2022-10-10T14:33:24Z

Background

Initially the fork that this PR is based on was created to showcase Google Cloud's CICD story encompassing Cloud Build / Cloud Deploy / kustomize / skaffold. In order to maximize value from the work done for the showcase we decided to automate the setup with terraform utilising Cloud Foundation Toolkit.

Change Summary

Added configuration that includes:

Terraform for all infrastructure (with exception of Github repo connection which is a one-time manual step)
CI per team
- CloudBuild /w skaffold (includes caching)
CD (staging, production) per team
- CloudDeploy /w skaffold profiles per environment
3 Environments:
- Development (skaffold dev only)
  - GKE
  - Autopilot
  - ASM
  - ACM for service accounts, namespaces, rolebindings…
  - Workload identity
  - Namespace: bank-of-anthos-development
- Staging (CD deploy)
  - GKE
  - Autopilot
  - ASM
  - ACM for service accounts, namespaces, rolebindings…
  - Workload identity
  - Namespace: bank-of-anthos-staging
  - CloudSQL
- Production (CD deploy)
  - GKE
  - Autopilot
  - ASM
  - ACM for service accounts, namespaces, rolebindings…
  - Workload Identity
  - Namespace: bank-of-anthos-production
  - CloudSQL
Heavy use of kustomize components to stay DRY
Minimized service account permissions
Uses CloudFoundation Toolkit for Kubernetes (which restricts modularization of environments due to kubernetes provider circular dependency)

Additional Notes

This PR does NOT include the changes to the src/ folder, which include kustomizations for microservices, skaffold profiles and cloudbuild.yaml definitions for individual teams.
The terraform code included in this PR does however reference these files, which leads to this IaC not being immediately compatible with the current application source.

Testing Procedure

Follow steps here using the prepared shell scripts for easiest setup.

Please note that script 4 and 5 will log errors as they require the second PR (which includes changes to src/ folder)

Related PRs or Issues

I stole shamelessly from these PRs:
#881
#792

Changes to src/ can be seen in this fork

snippet-bot · 2022-10-10T14:33:29Z

Here is the summary of changes.

You are about to add 4 region tags.

iac/acm-multienv-cicd-anthos-autopilot/base/config.yaml:15, tag gke_dev_kubernetes_manifests_config_configmap_environment_config
iac/acm-multienv-cicd-anthos-autopilot/base/config.yaml:25, tag gke_dev_kubernetes_manifests_config_configmap_service_api_config
iac/acm-multienv-cicd-anthos-autopilot/base/config.yaml:38, tag gke_dev_kubernetes_manifests_config_configmap_demo_data_config
iac/acm-multienv-cicd-anthos-autopilot/base/jwt-secret.yaml:15, tag gke_jwt_jwt_secret_secret_jwt_key

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

…-anthos into iac-multienv-cicd-anthos-autopilot

aablsk · 2022-10-10T15:53:53Z

@bourgeoisor What's our requirement on empty lines at end of file? (need to fix this still)

…-anthos into iac-multienv-cicd-anthos-autopilot

bourgeoisor

Thank you Alex for the PR, added a handful of comments and questions :)

bourgeoisor · 2022-10-28T17:49:01Z

iac/acm-multienv-cicd-anthos-autopilot/base/config.yaml

@@ -0,0 +1,48 @@
+# Copyright 2020 Google LLC


Are these ACM-managed files "copies" or "moved" of e.g. https://github.com/GoogleCloudPlatform/bank-of-anthos/blob/main/kubernetes-manifests/config.yaml

If copied: how will we make sure they're kept in sync?

If moved: how will non-TF and/or non-ACM deployments deploy the required ConfigMap (for example)?

I propose to 'move' these files instead of 'copying' them. Since this base configuration is a kustomize component, it can be easily added to a skaffold configuration. As discussed I have prepared a skaffold profile ('development') which is compatible with a vanilla k8s cluster for the next PR that will include this.

Source to include these manifests specifically can be found here.

bourgeoisor · 2022-10-28T17:51:49Z

iac/acm-multienv-cicd-anthos-autopilot/base/kustomization.yaml

@@ -0,0 +1,19 @@
+# Copyright 2022 Google LLC


Does ACM work with Kustomize?

Yes it does! ✨

bourgeoisor · 2022-10-28T17:52:27Z

iac/acm-multienv-cicd-anthos-autopilot/overlays/development/kustomization.yaml

+  patch: |-
+    - op: add
+      path: /metadata/annotations/iam.gke.io~1gcp-service-account
+      value: gke-workload-development@bank-of-anthos-aablsk.iam.gserviceaccount.com


I see a leftover "aablsk", is that intentional?

This was previously replaced by the 01-patch-acm-config.sh script. As a proposal I have removed all bash scripts and instead added step-by-step instructions for the initial (one-time) setup to the README.md

bourgeoisor · 2022-10-28T18:09:20Z

iac/acm-multienv-cicd-anthos-autopilot/overlays/production/configmap.yaml

+metadata:
+  name: cloud-sql-admin
+data:
+  connectionName: bank-of-anthos-aablsk:europe-west1:bank-of-anthos-db-production


Hardcoded values?

This was previously replaced by the 01-patch-acm-config.sh script. As a proposal I have removed all bash scripts and instead added step-by-step instructions for the initial (one-time) setup to the README.md

bourgeoisor · 2022-10-28T18:10:01Z

iac/acm-multienv-cicd-anthos-autopilot/overlays/production/kustomization.yaml

+    patch: |-
+      - op: add
+        path: /metadata/annotations/iam.gke.io~1gcp-service-account
+        value: gke-workload-production@bank-of-anthos-aablsk.iam.gserviceaccount.com 


Hardcoded values?

This was previously replaced by the 01-patch-acm-config.sh script. As a proposal I have removed all bash scripts and instead added step-by-step instructions for the initial (one-time) setup to the README.md

bourgeoisor · 2022-10-28T20:06:59Z

iac/tf-multienv-cicd-anthos-autopilot/shell-scripts/01-patch-acm-config.sh

@@ -0,0 +1,28 @@
+# Copyright 2022 Google LLC


I'm not super sure about this script -- I think I'd prefer if we'd hardcode (or default a variable) to this current GCP project / region and not have this script at all. I'd rather git commands (besides git clone) not be involved.

Perhaps instead of a script, it could be part of the README, where in the prereq you state that if you're deploying this TF outside the context of BoA maintainers, you should:

fork the repo

substitute some stuff like the GCP project name and commit those in your now forked repo (perhaps that bit can be a quick script, but it should not be part of the TF family of script) -- though I'd personally lean towards simplifying and not scripting it at all... Just stating it in the README

THEN you can run the TF scripts

I agree. These scripts were mostly created to support me in testing reproducibility of the setup with little effort. For this PR I have therefore removed all bash scripts and instead added step-by-step instructions for the initial (one-time) setup to the README.md.

iac/tf-multienv-cicd-anthos-autopilot/README.md

bourgeoisor · 2022-10-28T20:12:51Z

iac/tf-multienv-cicd-anthos-autopilot/main.tf

+  /* backend "gcs" {
+      bucket = "tf-state-boa-tf"
+      prefix = "bank-of-anthos"
+  } */


Commented out?

I have removed the comment and added instructions for setting this up to the README.md

iac/tf-multienv-cicd-anthos-autopilot/env-staging.tf

bourgeoisor · 2022-10-28T20:17:20Z

iac/tf-multienv-cicd-anthos-autopilot/README.md

+    1. Terraform will set up your GCP infrastructure
+    1. Waits for provisioning of managed ASM to complete
+    1. Deploys jobs to staging and production k8s clusters which initialise the Cloud SQL databases.
+    1. Triggers CICD for accounts, frontend & ledger teams


This means BoA gets fully deployed as well, correct? Does it get these things? https://github.com/GoogleCloudPlatform/bank-of-anthos/tree/main/.github/prod-cluster

Ideally we'd dogfood this entire TF flow for real when we develop on BoA and the prod cluster would be the https://bank-of-anthos.xyz/ cluster :)

Agreed. I was not aware of this configuration, but happy to add it with the second PR. (will include terraform for CloudArmor, IP... and adapted production skaffold profile with a fitting kustomization)

aablsk · 2022-10-30T11:41:37Z

I've responded to most questions and have made some adjustments in line with your comments. I'm also happy make myself available for a synchronous meeting from November 3rd if that would be helpful.

@bourgeoisor Could you please provide me with the following values that you'd like to use for your setup, so I can include them in the configuration:

Google Cloud ProjectId in which you are planning to deploy this
Region in which the resources should be deployed (if applicable)
Name of the GCS bucket in which you'd like to store the shared terraform state

Thanks in advance!

Please note that work on the second PR is still in progress:

upgrading to new skaffold version (which will enable deployment verification)
introducing private clusters for security purposes
including the additional features from https://github.com/GoogleCloudPlatform/bank-of-anthos/tree/main/.github/prod-cluster

… instance

…-anthos into iac-multienv-cicd-anthos-autopilot

NimJay · 2022-11-14T16:55:26Z

iac/tf-multienv-cicd-anthos-autopilot/apis.tf

+    "iam.googleapis.com",
+    "sqladmin.googleapis.com"
+  ]
+}


Praise: I like that we're dedicating a separate apis.tf file for GCP API enablement. This is clean!

🙈 Thanks for the kind words, Nim!

NimJay · 2022-11-14T16:59:35Z

iac/tf-multienv-cicd-anthos-autopilot/artifact-registry.tf

+  ]
+}
+
+# we cannot use a custom service account with autopilot clusters in terraform https://github.com/hashicorp/terraform-provider-google/issues/9505 🤷


Thought: After merging this pull-request, we should consider tracking our desire to use a custom service account here in a GitHub issue (i.e., create a new GitHub Issue to add visibility).

bourgeoisor

LGTM

…ment setup with CI/CD, Anthos & CloudSQL (GoogleCloudPlatform#1006) * add ACM base * add ACM overlays for development, staging, production environments * add tf infrastructure definition * add cicd pipeline custom tf module * add shell scripts for easy setup * add README documenting how to setup multienv-cicd-anthos-autopilot * substitute projectId and region references in ACM config * substitute projectId and region references in ACM config * add configurable sync_branch * remove obsolete file * fix license header * replace hard-coded script names with variable * make EOF consistent empty line * remove commented master_authorized_networks * substitute projectId and region references in ACM config * remove ZONE variable * move targets, teams to tf.locals * remove shell scripts, update README.md to contain manual steps * add instructions for terraform state bucket * fix terraform, update README.md, reintroduce zone for CloudSQL master instance Co-authored-by: Olivier Bourgeois <[email protected]>

aablsk added 9 commits October 6, 2022 14:20

add ACM base

bd26211

add ACM overlays for development, staging, production environments

29dc017

add tf infrastructure definition

51c21bd

add cicd pipeline custom tf module

191153d

add shell scripts for easy setup

0071563

add README documenting how to setup multienv-cicd-anthos-autopilot

aaeaa4c

substitute projectId and region references in ACM config

5379952

substitute projectId and region references in ACM config

3ded8c5

add configurable sync_branch

c80ab64

aablsk requested a review from a team as a code owner October 10, 2022 14:33

aablsk added 4 commits October 10, 2022 14:37

remove obsolete file

ae3e8f3

fix license header

e261b5d

Merge branch 'main' of https://github.com/GoogleCloudPlatform/bank-of…

98ff21b

…-anthos into iac-multienv-cicd-anthos-autopilot

replace hard-coded script names with variable

a8866f2

aablsk and others added 4 commits October 11, 2022 09:08

Merge branch 'main' into iac-multienv-cicd-anthos-autopilot

24593f3

Merge branch 'main' into iac-multienv-cicd-anthos-autopilot

ffb20e6

make EOF consistent empty line

1751de5

Merge branch 'main' of https://github.com/GoogleCloudPlatform/bank-of…

2b6f6f7

…-anthos into iac-multienv-cicd-anthos-autopilot

bourgeoisor self-assigned this Oct 17, 2022

bourgeoisor requested changes Oct 28, 2022

View reviewed changes

aablsk added 6 commits October 30, 2022 10:26

remove commented master_authorized_networks

fb15566

substitute projectId and region references in ACM config

4ff74fe

remove ZONE variable

c1e3ca9

move targets, teams to tf.locals

a319b65

remove shell scripts, update README.md to contain manual steps

8b0ea04

add instructions for terraform state bucket

9d59121

fix terraform, update README.md, reintroduce zone for CloudSQL master…

e3d023a

… instance

Merge branch 'main' of https://github.com/GoogleCloudPlatform/bank-of…

61d432f

…-anthos into iac-multienv-cicd-anthos-autopilot

NimJay reviewed Nov 14, 2022

View reviewed changes

bourgeoisor added 2 commits November 14, 2022 17:04

Merge branch 'main' into iac-multienv-cicd-anthos-autopilot

14c8ea1

Merge branch 'main' into iac-multienv-cicd-anthos-autopilot

22c8a1b

aablsk requested a review from bourgeoisor November 22, 2022 08:46

Merge branch 'main' into iac-multienv-cicd-anthos-autopilot

84d56d2

bourgeoisor approved these changes Nov 22, 2022

View reviewed changes

bourgeoisor merged commit 5ca01c4 into GoogleCloudPlatform:main Nov 23, 2022

aablsk deleted the iac-multienv-cicd-anthos-autopilot branch November 24, 2022 07:22

aablsk mentioned this pull request Jan 16, 2023

[BREAKING CHANGES] Refactor of CI/CD, kustomize & skaffold #1221

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IaC: Terraform and ACM configurations for Multi-Autopilot GKE environment setup with CI/CD, Anthos & CloudSQL #1006

IaC: Terraform and ACM configurations for Multi-Autopilot GKE environment setup with CI/CD, Anthos & CloudSQL #1006

aablsk commented Oct 10, 2022

snippet-bot bot commented Oct 10, 2022 •

edited

Loading

aablsk commented Oct 10, 2022

bourgeoisor left a comment

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

bourgeoisor Oct 28, 2022

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

bourgeoisor Oct 28, 2022

aablsk Oct 30, 2022

aablsk commented Oct 30, 2022 •

edited

Loading

NimJay Nov 14, 2022

aablsk Nov 15, 2022

NimJay Nov 14, 2022

aablsk Nov 15, 2022

bourgeoisor left a comment

IaC: Terraform and ACM configurations for Multi-Autopilot GKE environment setup with CI/CD, Anthos & CloudSQL #1006

IaC: Terraform and ACM configurations for Multi-Autopilot GKE environment setup with CI/CD, Anthos & CloudSQL #1006

Conversation

aablsk commented Oct 10, 2022

Background

Change Summary

Additional Notes

Testing Procedure

Related PRs or Issues

snippet-bot bot commented Oct 10, 2022 • edited Loading

aablsk commented Oct 10, 2022

bourgeoisor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aablsk commented Oct 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bourgeoisor left a comment

Choose a reason for hiding this comment

snippet-bot bot commented Oct 10, 2022 •

edited

Loading

aablsk commented Oct 30, 2022 •

edited

Loading