[BREAKING CHANGES] Refactor of CI/CD, kustomize & skaffold (GoogleClo…

…udPlatform#1221) * add ACM base * add ACM overlays for development, staging, production environments * add tf infrastructure definition * add cicd pipeline custom tf module * add shell scripts for easy setup * add README documenting how to setup multienv-cicd-anthos-autopilot * substitute projectId and region references in ACM config * substitute projectId and region references in ACM config * add configurable sync_branch * remove obsolete file * fix license header * replace hard-coded script names with variable * made development cluster private * make staging and production private clusters as well * make EOF consistent empty line * src split into teams, kustomize overlays, skaffold profiles * clean up * improve teardown process * substitute projectId and region references in ACM config * upgrade cloudbuild.yaml to skaffold v2 * fix access for private clusters * remove commented master_authorized_networks * substitute projectId and region references in ACM config * remove ZONE variable * move targets, teams to tf.locals * remove shell scripts, update README.md to contain manual steps * add instructions for terraform state bucket * fix terraform, update README.md, reintroduce zone for CloudSQL master instance * substitute projectId and region references in ACM config * fix skaffold v2 deploy configuration for nested skaffold.yamls * updated for $PROJECT_ID and $REGION * fix deployment to private anthos cluster * fix deployment to anthos cluster * add endpoint configuration for production environment * fix deployment to anthos cluster through connect gateway * add first cloudbuild definition for PR CI * fix style issues * add logging configuration * fix test execution * add prep step for maven builds * fix addlicense, fix skaffold build * change go install instructions * switch to GOROOT from GOPATH * hopefully fix go install * maybe fix go install * fix go install * try to fix go install * bump go version * fix addlicense * skip tests in build phase * ci fixes * ci fixes * fix ci * add licenses * remove unnecessary env var from ci * add go/ to gitignore for ci * fix contacts skaffold.yaml * add pr cache bucket * install pylint * fix ci * install python-venv * switch to python3-venv * fix arg * possibly fix test step * fix python tests * fix python tests * auto yes apt-get install * separate python environments for all tests * fix style issues * move dependencies from skaffold.yaml into ci-pr * improve naming of python venv * switch test definition to python3 * add balancereader tests * add /workspace to requirements.txt * install pytest * fix python tests * add tests for ledger services * add frontend tests * fix frontend pylint test * removed pytest for frontend * fix pylint for frontend * add tests to build pipeline * generalize init-db jobs into kustomize component * fix tests * fix cloudbuild tests * fix filename * fix env var reference * fix frontend tests in ci * fix test frontend ci * try another update for fixing frontend ci * fix issues, re-enable statuschecks * fix skaffold version * fix skaffold version * patch skaffold version * update cloud deploy image version * cloudbuild yaml skaffold version for cloud deploy * deploy skaffold_preview * rollback tolerateFailuresUntilDeadline * roll back skaffold apiVersion * remove irrelevant files from docker images * upgrade to v4beta1 * upgrade to v4beta1 * dev setup, pr namespace, e2e tests * fix flaky test * fix ci-pr cloudbuild.yaml * add license header * pass MAVEN_USER_HOME to skaffold * update for ci pr * fix ci ledger * fix iam bindings for cloudbuild pr sa * fix sql db initialization * improve ci-pr * fix artifacts indentation * fix tags indentation * fix substitution * fix substitution * fix tags * set skaffold profile correctly * fix verify * fix verify job 🤞 * potential fix for uploading screenshots * add verify to cloud deploy * small fix to trigger deploy * fix verification in clouddeploy * fix e2e tests * fix license header * fix e2e-test config for staging * fix e2e image * fix e2e container image * fix e2e tests * fix ci pr * fix pipeline * fix bucket reference for ci-pr * improve log message for cypress screenshots * fix E2E tests * exit with status code 1 if cypress fails instead of relying on screenshot folder * improve readability in google cloud build * fix e2e tests * fix file upload * remove error for testing artifact upload of e2e test failures * fix e2e tests for production * fix ci-pr * potentially fix e2e tests for accounts, ledger * remove verify from skaffold.yaml for accounts, ledger * fix e2e tests in skaffold * request faster machine for ci pr builds * fix ci-pr workload ksa * upgrade and fix cypress tests * fix license header * fix ci-pr * fix e2e tests * possible fix for ci-pr * fix skaffold deploy * add ci-pr to terraform * fix ci pr * remove cloud build artifacts from ci-pr * substitute projectId and region references in ACM config * fix README * remove projectid and region * remove bielski domain * update README with region information for cloud build * fix region for ci-pr pipeline * reduce ui test flakiness * "substitute $PROJECT_ID and $REGION references in ACM config" * update terraform.tfvars for bank-of-anthos-ci * update README with setup steps * fix region tags * add removed region tag * ci fixes for compatibility with new source structure * add removed region tags in skaffold.yaml * removed helper scripts for reproduction tests * fix terraform provider config * substitute boa-aablsk-delivery-refactor and europe-west1 references in ACM config * refactor pipelines to microservice scope * fix skaffold test execution * fix pylint issue * fix ledger ci * maybe fix PR ci * possible fix for jib maven build * potential fix for jib builds * bump skaffold version for java 17 * fix release names, ledger pipeline * fix pipelines * fix ledger pipeline * fix ledger pipeline * update account ci * remove unnecessary workaround * potentially fix ci-pr * fix release names * potentially fix ci pr * fix ci pr * re-added java build workaround * ci pr * CI PR * possibly fix ci pr * ci pr * possible fix for ci-pr * fix substitutions * potential fix for ci pr * possible fix for ledger builds * improve cloudbuild.yamls * fix deploy dependencies in ci-pr * add e2e tests * fix verify * improve README.md * update README.md * fix domain for prod * update project specific values for bank-of-anthos-ci deployment * set config for new repro * fix e2e tests * fix sql script * fix sql script * fix e2e tests for production * change timeouts and retries * fix load generator * fix screenshot upload to bucket * reduce number of CD targets, make VERIFY production only * remove namespace from loadgenerator base to avoid conflict in ci-pr and skaffold dev * fix delivery pipelines * revert any config values to bank-of-anthos-ci env * bump java to 17 * update documentation --------- Co-authored-by: Shabir Mohamed Abdul Samadh <[email protected]>
big-appled · Jun 9, 2023 · 72bfba4 · 72bfba4
1 parent ecf86ec
commit 72bfba4
Show file tree

Hide file tree

Showing 213 changed files with 4,757 additions and 409 deletions.
diff --git a/.github/cloudbuild/ci-pr.yaml b/.github/cloudbuild/ci-pr.yaml
@@ -0,0 +1,93 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+steps:
+  - name: gcr.io/cloud-builders/go:1.16
+    id: install-license-check-script
+    script: |
+      go install github.com/google/addlicense@latest
+    timeout: 120s
+  - name: gcr.io/cloud-builders/git
+    id: check-licenses
+    script: |
+      #!/bin/bash
+      set -ex
+      if [[ -d ".git" ]]; then
+        rm -rf .git
+      fi
+      git init
+      git add --all
+      git -c user.name="CI Bot" -c user.email="<>" commit -m "initial state"
+      ./go/bin/addlicense ./
+      git status -s
+      if [[ -n $(git status -s) ]]; then
+        exit 1
+      fi
+    timeout: 120s
+  - name: gcr.io/cloud-builders/gsutil # get skaffold build cache
+    id: download-skaffold-cache
+    args: ['cp', $_CACHE_URI, '/workspace/cache']
+    timeout: 120s
+  - name: gcr.io/k8s-skaffold/skaffold:v2.1.0 # build images with skaffold
+    id: build-and-push-images
+    script: | 
+      MAVEN_USER_HOME=$MAVEN_USER_HOME ./mvnw jib:_skaffold-fail-if-jib-out-of-date -Djib.requiredVersion=1.4.0 --projects src/ledger/balancereader --also-make jib:_skaffold-files-v2 --quiet --batch-mode
+      MAVEN_USER_HOME=$MAVEN_USER_HOME skaffold build --file-output=/workspace/artifacts.json --default-repo=$CONTAINER_REGISTRY --cache-file=/workspace/$CACHE --profile=development
+    #args: ['skaffold', 'build', '--file-output=/workspace/artifacts.json', '--default-repo=$_CONTAINER_REGISTRY', '--cache-file=/workspace/$_CACHE', '--profile=development'] # set _CACHE to anything other than "cache" e.g. "no-cache" to reset skaffold cache
+  - name: gcr.io/cloud-builders/gsutil # upload skaffold build cache
+    id: upload-skaffold-cache
+    args: ['cp', '/workspace/$_CACHE',  $_CACHE_URI]
+    timeout: 120s
+  - name: gcr.io/k8s-skaffold/skaffold:v2.1.0 # run tests with skaffold
+    id: run-tests
+    script: |
+      #!/bin/bash
+      apt-get update && apt-get -y install python3-venv && python3 -m venv $HOME/venv-python-tests && . $HOME/venv-python-tests/bin/activate
+      skaffold test --build-artifacts=/workspace/artifacts.json --assume-yes --profile=development
+  - name: gcr.io/cloud-builders/gcloud
+    id: set-policy-binding-for-ksa
+    args: ['iam', 'service-accounts', 'add-iam-policy-binding', 'gke-workload-development@$PROJECT_ID.iam.gserviceaccount.com', '--role=roles/iam.workloadIdentityUser', '--member=serviceAccount:$PROJECT_ID.svc.id.goog[pr$_PR_NUMBER/bank-of-anthos]']
+  - name: gcr.io/k8s-skaffold/skaffold:v2.1.0 # deploy to pr-namespace with skaffold
+    id: deploy-to-pr-namespace
+    script: |
+      #!/bin/bash
+      gcloud container fleet memberships get-credentials development-membership
+      kubectl create namespace $NAMESPACE
+      skaffold deploy --namespace=$NAMESPACE --build-artifacts=/workspace/artifacts.json --assume-yes --profile=development --iterative-status-check=false
+  - name: gcr.io/k8s-skaffold/skaffold:v2.1.0 # run verification/e2e tests with skaffold
+    id: run-e2e-tests
+    script: |
+      #!/bin/bash
+      BASE_URL=http://$(kubectl get service frontend --namespace $NAMESPACE -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+      CYPRESS_baseUrl=$BASE_URL ROLLOUT=$ROLLOUT skaffold verify --build-artifacts=/workspace/artifacts.json --profile=development -m frontend
+    env:
+      - "ROLLOUT=pr$_PR_NUMBER/$SHORT_SHA"
+      - "ARTIFACTS_BUCKET_NAME=$_CACHE_URI"
+  - name: gcr.io/k8s-skaffold/skaffold:v2.1.0 # 
+    id: delete-environment
+    script: |
+      #!/bin/bash
+      skaffold delete --namespace=$NAMESPACE --profile=development
+options:
+  logging: CLOUD_LOGGING_ONLY
+  env:
+    - "NAMESPACE=pr$_PR_NUMBER"
+    - "GOPATH=/workspace/go"
+    - "MAVEN_USER_HOME=/workspace/.m2"
+    - "CONTAINER_REGISTRY=$_CONTAINER_REGISTRY"
+    - "CACHE=$_CACHE"
+  machineType: 'E2_HIGHCPU_8'
+tags:
+  - pr$_PR_NUMBER
+  - ci-pr
diff --git a/.github/workflows/ci-main.yaml b/.github/workflows/ci-main.yaml
@@ -61,8 +61,8 @@ jobs:
         source /etc/profile.d/java.sh
         for SERVICE in "balancereader" "ledgerwriter" "transactionhistory"; do
           echo "checking $SERVICE..."
-          # save current working dir to memory and cd to src/$SERVICE
-          pushd src/$SERVICE
+          # save current working dir to memory and cd to src/ledger/$SERVICE
+          pushd src/ledger/$SERVICE
             mvn jacoco:report
             echo "Coverage for $SERVICE:"
             awk -F, \
@@ -79,8 +79,8 @@ jobs:
         set -x
         for SERVICE in "contacts" "userservice"; do
           echo "testing $SERVICE..."
-          # save current working dir to memory and cd to src/$SERVICE
-          pushd src/$SERVICE
+          # save current working dir to memory and cd to src/accounts/$SERVICE
+          pushd src/accounts/$SERVICE
             python3 -m venv $HOME/venv-$SERVICE
             source $HOME/venv-$SERVICE/bin/activate
             pip install --upgrade pip
@@ -95,8 +95,8 @@ jobs:
       run: |
         for SERVICE in "contacts" "userservice"; do
           echo "testing $SERVICE..."
-          # save current working dir to memory and cd to src/$SERVICE
-          pushd src/$SERVICE
+          # save current working dir to memory and cd to src/accounts/$SERVICE
+          pushd src/accounts/$SERVICE
             python3 -m venv $HOME/venv-$SERVICE
             source $HOME/venv-$SERVICE/bin/activate
             pip install --upgrade pip
@@ -133,7 +133,7 @@ jobs:
         EOF
         echo Deploying application
         skaffold config set --global local-cluster false
-        skaffold run --default-repo=gcr.io/$PROJECT_ID/$GITHUB_REF --tag=$GITHUB_SHA --namespace=$NAMESPACE
+        skaffold run --default-repo=gcr.io/$PROJECT_ID/$GITHUB_REF --tag=$GITHUB_SHA --namespace=$NAMESPACE --profile=development
       env:
         PROJECT_ID: "bank-of-anthos-ci"
         PR_CLUSTER: "bank-of-anthos-prs"

diff --git a/.github/workflows/ci-pr.yaml b/.github/workflows/ci-pr.yaml
@@ -62,8 +62,8 @@ jobs:
         source /etc/profile.d/java.sh
         for SERVICE in "balancereader" "ledgerwriter" "transactionhistory"; do
           echo "checking $SERVICE..."
-          # save current working dir to memory and cd to src/$SERVICE
-          pushd src/$SERVICE
+          # save current working dir to memory and cd to src/ledger/$SERVICE
+          pushd src/ledger/$SERVICE
             mvn jacoco:report
             echo "Coverage for $SERVICE:"
             awk -F, \
@@ -80,8 +80,8 @@ jobs:
         set -x
         for SERVICE in "contacts" "userservice"; do
           echo "testing $SERVICE..."
-          # save current working dir to memory and cd to src/$SERVICE
-          pushd src/$SERVICE
+          # save current working dir to memory and cd to src/accounts/$SERVICE
+          pushd src/accounts/$SERVICE
             python3 -m venv $HOME/venv-$SERVICE
             source $HOME/venv-$SERVICE/bin/activate
             pip install --upgrade pip
@@ -96,8 +96,8 @@ jobs:
       run: |
         for SERVICE in "contacts" "userservice"; do
           echo "testing $SERVICE..."
-          # save current working dir to memory and cd to src/$SERVICE
-          pushd src/$SERVICE
+          # save current working dir to memory and cd to src/accounts/$SERVICE
+          pushd src/accounts/$SERVICE
             python3 -m venv $HOME/venv-$SERVICE
             source $HOME/venv-$SERVICE/bin/activate
             pip install --upgrade pip
@@ -135,7 +135,7 @@ jobs:
         EOF
         echo Deploying application
         skaffold config set --global local-cluster false
-        skaffold run --default-repo=gcr.io/$PROJECT_ID/refs/$PR_NUMBER --tag=$GITHUB_SHA --namespace=$NAMESPACE
+        skaffold run --default-repo=gcr.io/$PROJECT_ID/refs/$PR_NUMBER --tag=$GITHUB_SHA --namespace=$NAMESPACE --profile=development
       env:
         PR_NUMBER: ${{ github.event.pull_request.number }}
         PROJECT_ID: "bank-of-anthos-ci"

diff --git a/.github/workflows/ui-tests/Dockerfile b/.github/workflows/ui-tests/Dockerfile
@@ -0,0 +1,23 @@
+# Copyright 2022 Google LLC
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#      http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM cypress/included:12.3.0
+WORKDIR /e2e
+# install gcloud cli tools to get kubectl context and service/ingress endpoint ip to test
+RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg  add - && apt-get update -y && apt-get install google-cloud-cli -y
+RUN apt-get update
+RUN apt-get install google-cloud-sdk-gke-gcloud-auth-plugin kubectl
+ENV USE_GKE_GCLOUD_AUTH_PLUGIN=True
+# cypress code & config
+COPY . .
+ENV XDG_CONFIG_HOME=/e2e
+# run custom bash script to set CYPRESS_baseUrl and execute tests
+ENTRYPOINT [ "/bin/bash", "-c", "./run_for_env.sh" ]
diff --git a/.github/workflows/ui-tests/cypress.json → .github/workflows/ui-tests/cypress.config.js b/.github/workflows/ui-tests/cypress.json → .github/workflows/ui-tests/cypress.config.js
@@ -1,9 +1,28 @@
-{
+// Copyright 2023 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+const { defineConfig } = require('cypress')
+
+module.exports = defineConfig({
     "CI": true,
-    "retries": 1,
+    "retries": 3,
     "video": false,
-    "pluginsFile": false,
     "fixturesFolder": false,
+    "chromeWebSecurity": false,
+    "defaultCommandTimeout": 6000,
+    "pageLoadTimeout": 10000,
+    "responseTimeout": 10000,
     "env": {
         "messages": {
             "transaction": {
@@ -54,5 +73,9 @@
             ],
             "localRoutingNum": "883745000"
         }
+    },
+    "e2e": {
+        supportFile: "cypress/support/index.js",
+        specPattern: "cypress/integration/**/*.js"
     }
-}
+})
diff --git a/.github/workflows/ui-tests/run_for_env.sh b/.github/workflows/ui-tests/run_for_env.sh
@@ -0,0 +1,42 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+gcloud auth list
+
+# if cypress_baseurl is not set
+if [[ -z "${CYPRESS_baseUrl}" ]]; then
+    # Get credentials for current Anthos cluster (staging/production)
+    export ANTHOS_MEMBERSHIP_SHORT=$(echo $ANTHOS_MEMBERSHIP | cut -d/ -f6)
+    export ARTIFACTS_BUCKET_NAME=gs://delivery-artifacts-$PIPELINE-$PROJECT
+    gcloud container fleet memberships get-credentials $ANTHOS_MEMBERSHIP_SHORT
+    if [[ "$ANTHOS_MEMBERSHIP_SHORT" == "staging-membership" ]]; then
+        export CYPRESS_baseUrl=http://$(kubectl get service frontend --namespace bank-of-anthos-staging -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+    elif [[ "$ANTHOS_MEMBERSHIP_SHORT" == "production-membership" ]]; then
+        export CYPRESS_baseUrl=https://$(kubectl get ingress frontend-ingress --namespace bank-of-anthos-production -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+    else
+        echo ERROR: CYPRESS_baseUrl is not set and cannot be automatically determined. Exiting with status code 1.
+        exit 1
+    fi
+fi
+
+# run tests
+CYPRESS_baseUrl=$CYPRESS_baseUrl NO_COLOR=1 cypress run --reporter json-stream --browser chrome --headed
+
+# if failed, copy screenshots to bucket and exit with status code 1
+if [[ "$?" -ne 0 ]]; then
+    export COPY_DESTINATION=$ARTIFACTS_BUCKET_NAME/$ROLLOUT/e2e/cypress/
+    echo ERROR: Cypress E2E tests have failed. Screenshots will be uploaded to $COPY_DESTINATION screenshots 
+    gcloud storage cp -r /e2e/cypress/screenshots $COPY_DESTINATION
+    exit 1
+fi
diff --git a/.gitignore b/.gitignore
@@ -203,4 +203,10 @@ override.tf.json
 # example: *tfplan*
 
 # tf lock file
-.terraform.lock.hcl
+.terraform.lock.hcl
+
+# python venv
+venv-*
+
+# go for ci
+go/
diff --git a/iac/acm-multienv-cicd-anthos-autopilot/overlays/development/kustomization.yaml b/iac/acm-multienv-cicd-anthos-autopilot/overlays/development/kustomization.yaml
@@ -13,7 +13,6 @@
 # limitations under the License.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-namespace: bank-of-anthos-development
 resources:
   - namespace.yaml
 components:
@@ -24,4 +23,4 @@ patches:
   patch: |-
     - op: add
       path: /metadata/annotations/iam.gke.io~1gcp-service-account
-      value: gke-workload-development@$PROJECT_ID.iam.gserviceaccount.com
+      value: gke-workload-development@bank-of-anthos-ci.iam.gserviceaccount.com
diff --git a/iac/acm-multienv-cicd-anthos-autopilot/overlays/production/configmap.yaml b/iac/acm-multienv-cicd-anthos-autopilot/overlays/production/configmap.yaml
@@ -16,4 +16,4 @@ kind: ConfigMap
 metadata:
   name: cloud-sql-admin
 data:
-  connectionName: $PROJECT_ID:$REGION:bank-of-anthos-db-production
+  connectionName: bank-of-anthos-ci:us-central1:bank-of-anthos-db-production
diff --git a/iac/acm-multienv-cicd-anthos-autopilot/overlays/production/kustomization.yaml b/iac/acm-multienv-cicd-anthos-autopilot/overlays/production/kustomization.yaml
@@ -25,4 +25,4 @@ patches:
     patch: |-
       - op: add
         path: /metadata/annotations/iam.gke.io~1gcp-service-account
-        value: gke-workload-production@$PROJECT_ID.iam.gserviceaccount.com 
+        value: gke-workload-production@bank-of-anthos-ci.iam.gserviceaccount.com 
diff --git a/iac/acm-multienv-cicd-anthos-autopilot/overlays/staging/configmap.yaml b/iac/acm-multienv-cicd-anthos-autopilot/overlays/staging/configmap.yaml
@@ -16,4 +16,4 @@ kind: ConfigMap
 metadata:
   name: cloud-sql-admin
 data:
-  connectionName: $PROJECT_ID:$REGION:bank-of-anthos-db-staging
+  connectionName: bank-of-anthos-ci:us-central1:bank-of-anthos-db-staging
diff --git a/iac/acm-multienv-cicd-anthos-autopilot/overlays/staging/kustomization.yaml b/iac/acm-multienv-cicd-anthos-autopilot/overlays/staging/kustomization.yaml
@@ -25,4 +25,4 @@ patches:
   patch: |-
     - op: add
       path: /metadata/annotations/iam.gke.io~1gcp-service-account
-      value: gke-workload-staging@$PROJECT_ID.iam.gserviceaccount.com
+      value: gke-workload-staging@bank-of-anthos-ci.iam.gserviceaccount.com