Skip deploying services that do not need to be re-run (#739)

* Add dummy commit to test * Try to get from lerna inline * change ref * Commit previous * Test in unit tests for now for speed * Fix typo * Move it to get-branch-name-vars * put it in a script * pass a triggered commit * fix gha syntax * Try not from env * newline * try to get last sha * missing some history * fix fetch depth? * test some changes * fix typo * output only changed services * single line output * Trying to pass to deploy-to-env * pass changed-services * testing, for now remove deps for dev * test a change * use gha func * Test syntax * Test a skip * Test skipping * See if postgres triggers now * Add an initial ts file for workflow * build the scripts and try it * Need to setup the env * Try to just yarn install focus * For now let's commit it, faster dev loop * Move to an action * Add action.yml * It's always the typo * Try node16 for module import * can't avoid setup here in monorepo * Add GITHUB_TOKEN * Test getting all services * Try to run a full run * Output the value for debug * dashes not underlines * Add a few more conditionals * Check whether each service needs a deploy * Fix app-web * Try to skip successful jobs * Don't use local testing things * Run in CI * Actually pass the branch name * pass the stage name correctly * Not just success, but skipped as well * Prisma layer name * Check the migrations step * Always run cypress * Reference the local * Fix a postgres manual delete * Try to install tsc global without env * Pass the appropriate stage name * Use yarn build from scripts * Need the env * Don't run cypress if deploy-app hasn't gone * Fix stage name pass * Add postgres install back * Add in new infra-api service * Fix branch name * Get changed services from lerna * Try to get the sha * Try to only use success * Have promote deploy everything always * Remove the js * Always run finishing prep * check the sha being passed * Larger fetch-depth on the deploy check * Fix naming. Call it what it really is. * Add test commit to see if we deploy the service * Skipping can make jobs skip * Testing another service change * Clean the PR * Update readme to document adding services * Use lerna func to get all services * One more test before we merge it * Put it back before merge * Try to reduce code climate complexity score
Enterprise-CMCS · Jan 7, 2022 · ad9acd9 · ad9acd9
1 parent bb0b87b
commit ad9acd9
Show file tree

Hide file tree

Showing 11 changed files with 349 additions and 23 deletions.
diff --git a/.github/workflows/deploy-app-to-env.yml b/.github/workflows/deploy-app-to-env.yml
@@ -9,6 +9,9 @@ on:
       app_version:
         required: true
         type: string
+      changed_services:
+        required: true
+        type: string
     secrets:
       aws_access_key_id:
         required: true
@@ -29,6 +32,7 @@ on:
 
 jobs:
   app-web:
+    if: ${{ contains(inputs.changed_services, 'app-web') }}
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository
@@ -53,6 +57,7 @@ jobs:
           pushd services/app-web && npx serverless deploy --stage ${{ inputs.stage_name }}
 
   app-api:
+    if: ${{ contains(inputs.changed_services, 'app-api') }}
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository

diff --git a/.github/workflows/deploy-infra-to-env.yml b/.github/workflows/deploy-infra-to-env.yml
@@ -6,6 +6,9 @@ on:
       stage_name:
         required: true
         type: string
+      changed_services:
+        required: true
+        type: string
     secrets:
       aws_access_key_id:
         required: true
@@ -30,6 +33,7 @@ on:
 
 jobs:
   postgres:
+    if: ${{ contains(inputs.changed_services, 'postgres') }}
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository
@@ -50,6 +54,7 @@ jobs:
           pushd services/postgres && npx serverless deploy --stage ${{ inputs.stage_name }}
 
   ui:
+    if: ${{ contains(inputs.changed_services, 'ui') }}
     runs-on: ubuntu-latest
 
     steps:
@@ -72,6 +77,7 @@ jobs:
           pushd services/ui && npx serverless deploy --stage ${{ inputs.stage_name }}
 
   storybook:
+    if: ${{ contains(inputs.changed_services, 'storybook') }}
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository
@@ -94,6 +100,7 @@ jobs:
           pushd services/storybook && npx serverless deploy --stage ${{ inputs.stage_name }}
 
   uploads:
+    if: ${{ contains(inputs.changed_services, 'uploads') }}
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository
@@ -114,6 +121,7 @@ jobs:
           pushd services/uploads && npx serverless deploy --stage ${{ inputs.stage_name }}
 
   infra-api:
+    if: ${{ contains(inputs.changed_services, 'infra-api') }}
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository
@@ -133,6 +141,7 @@ jobs:
 
   ui-auth:
     needs: [infra-api, ui, uploads]
+    if: ${{ contains(inputs.changed_services, 'ui-auth') }}
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository

diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -14,12 +14,14 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       deploy-id: ${{ steps.ghdeployment.outputs.deployment_id }}
-      branch-name: ${{ steps.branch-name.outputs.stage-name-for-branch}}
+      stage-name: ${{ steps.stage-name.outputs.stage-name-for-branch}}
       app-version: ${{ steps.branch-name.outputs.app-version}}
-
+      changed-services: ${{ steps.changed-services-action.outputs.changed-services }}
     steps:
       - name: Check out repository
         uses: actions/checkout@v2
+        with:
+          fetch-depth: 10
 
       - name: Check for secrets
         uses: ./.github/actions/check_secrets
@@ -31,16 +33,43 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: set branch_name
-        id: branch-name
+      - name: set app version
+        id: app-version
+        shell: bash
+        run: echo "::set-output name=app-version::$(scripts/app_version.sh)"
+
+      - name: set stage name
+        id: stage-name
         shell: bash
         run: |
           echo "::set-output name=stage-name-for-branch::$(./scripts/stage_name_for_branch.sh ${GITHUB_REF#refs/heads/})"
 
-      - name: set app version
-        id: app-version
+      - name: set branch name
+        id: branch-name
         shell: bash
-        run: echo "::set-output name=app-version::$(scripts/app_version.sh)"
+        run: echo "::set-output name=branch-name::$(echo ${GITHUB_REF#refs/heads/})"
+
+      - name: Setup env
+        uses: ./.github/actions/setup_env
+
+      - name: build scripts
+        shell: bash
+        run: |
+          lerna run build:ci-scripts
+
+      - name: run get_changed_services
+        id: changed-services-action
+        uses: ./scripts/get-changed-services/
+        with:
+          branchName: ${{ steps.branch-name.outputs.branch-name}}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: test we have changed services output
+        id: test
+        shell: bash
+        run: |
+          echo "${{steps.changed-services-action.outputs.changed-services}}"
 
       - uses: chrnorm/deployment-action@releases/v1
         name: Create GitHub deployment
@@ -155,9 +184,10 @@ jobs:
 
   deploy-infra:
     needs: [begin-deployment]
-    uses: CMSgov/managed-care-review/.github/workflows/deploy-infra-to-env.yml@main
+    uses: CMSgov/managed-care-review/.github/workflows/deploy-infra-to-env.yml@mt-skip-deploys
     with:
-      stage_name: ${{ needs.begin-deployment.outputs.branch-name}}
+      stage_name: ${{ needs.begin-deployment.outputs.stage-name}}
+      changed_services: ${{ needs.begin-deployment.outputs.changed-services }}
     secrets:
       aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -176,10 +206,15 @@ jobs:
         api-unit-tests,
         deploy-infra,
       ]
-    uses: CMSgov/managed-care-review/.github/workflows/deploy-app-to-env.yml@main
+    if: |
+      always() &&
+      (needs.deploy-infra.result == 'success' || needs.deploy-infra.result == 'skipped')
+
+    uses: CMSgov/managed-care-review/.github/workflows/deploy-app-to-env.yml@mt-skip-deploys
     with:
-      stage_name: ${{ needs.begin-deployment.outputs.branch-name }}
+      stage_name: ${{ needs.begin-deployment.outputs.stage-name }}
       app_version: ${{ needs.begin-deployment.outputs.app-version }}
+      changed_services: ${{ needs.begin-deployment.outputs.changed-services }}
     secrets:
       aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -193,6 +228,7 @@ jobs:
   finishing-prep:
     name: Finishing Prep
     needs: [deploy-infra, begin-deployment]
+    if: always()
     runs-on: ubuntu-latest
     outputs:
       application-endpoint: ${{ steps.save-app-endpoint.outputs.app-endpoint }}
@@ -208,23 +244,23 @@ jobs:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
-          BRANCH_NAME: ${{ needs.begin-deployment.outputs.branch-name }}
+          STAGE_NAME: ${{ needs.begin-deployment.outputs.stage-name }}
           TEST_USERS_PASS: ${{ secrets.TEST_USERS_PASS }}
         run: |
           cd scripts
           yarn tsc
-          node ./add_cypress_test_users.js $BRANCH_NAME $TEST_USERS_PASS
+          node ./add_cypress_test_users.js $STAGE_NAME $TEST_USERS_PASS
 
       - name: get application endpoint
         id: save-app-endpoint
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
-          BRANCH_NAME: ${{ needs.begin-deployment.outputs.branch-name }}
+          STAGE_NAME: ${{ needs.begin-deployment.outputs.stage-name }}
         run: |
           cd services
-          endpoint=$(./output.sh ui CloudFrontEndpointUrl $BRANCH_NAME)
+          endpoint=$(./output.sh ui CloudFrontEndpointUrl $STAGE_NAME)
           echo $endpoint
           echo "::set-output name=app-endpoint::$endpoint"
 
@@ -257,6 +293,7 @@ jobs:
   cypress:
     name: cypress-run
     needs: [deploy-app, finishing-prep]
+    if: always()
     runs-on: ubuntu-latest
     container: cypress/browsers:node14.17.0-chrome91-ff89
     strategy:

diff --git a/.github/workflows/promote.yml b/.github/workflows/promote.yml
@@ -11,7 +11,18 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       app-version: ${{ steps.branch-name.outputs.app-version}}
-
+      changed-services: "[
+        'app-api',
+        'app-web',
+        'postgres',
+        'storybook',
+        'ui-auth',
+        'ui',
+        'uploads',
+        'run-migrations',
+        'prisma-layer',
+        'infra-api'
+        ]"
     services:
       postgres:
         image: postgres:13.3
@@ -100,6 +111,7 @@ jobs:
     uses: CMSgov/managed-care-review/.github/workflows/deploy-infra-to-env.yml@main
     with:
       stage_name: main
+      changed_services: ${{ needs.unit-tests.outputs.changed-services}}
     secrets:
       aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -118,6 +130,7 @@ jobs:
     with:
       stage_name: main
       app_version: ${{ needs.unit-tests.outputs.app-version }}
+      changed_services: ${{ needs.unit-tests.outputs.changed-services}}
     secrets:
       aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -183,6 +196,7 @@ jobs:
     uses: CMSgov/managed-care-review/.github/workflows/deploy-infra-to-env.yml@main
     with:
       stage_name: val
+      changed_services: ${{ needs.unit-tests.outputs.changed-services}}
     secrets:
       aws_access_key_id: ${{ secrets.VAL_AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.VAL_AWS_SECRET_ACCESS_KEY }}
@@ -201,6 +215,7 @@ jobs:
     with:
       stage_name: val
       app_version: ${{ needs.unit-tests.outputs.app-version }}
+      changed_services: ${{ needs.unit-tests.outputs.changed-services}}
     secrets:
       aws_access_key_id: ${{ secrets.VAL_AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.VAL_AWS_SECRET_ACCESS_KEY }}
@@ -266,6 +281,7 @@ jobs:
     uses: CMSgov/managed-care-review/.github/workflows/deploy-infra-to-env.yml@main
     with:
       stage_name: prod
+      changed_services: ${{ needs.unit-tests.outputs.changed-services}}
     secrets:
       aws_access_key_id: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }}
@@ -284,6 +300,7 @@ jobs:
     with:
       stage_name: prod
       app_version: ${{ needs.unit-tests.outputs.app-version }}
+      changed_services: ${{ needs.unit-tests.outputs.changed-services}}
     secrets:
       aws_access_key_id: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
       aws_secret_access_key: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }}

diff --git a/.gitignore b/.gitignore
@@ -27,6 +27,7 @@ notes/
 /scripts/add_cypress_test_users.js*
 /scripts/collect_ci_runtime_stats.js*
 /scripts/recent_actions_cache.json
+/scripts/get-changed-services/index.js*
 
 services/app-web/public/env-config.js
 services/app-web/storybook-static

diff --git a/README.md b/README.md
@@ -248,16 +248,29 @@ to Cloudtamer.
 
 To verify serverless (and AWS access) is set up properly with ctkey, run:
 
- ```shell
+```shell
 which serverless # should return something like /managed-care-review/scripts/serverless`
 which sls # should return something like /managed-care-review/scripts/sls`
 ```
 
 These should both point to paths inside the codebase (not to paths in /usr/local/bin).
 
-
 Then verify things are working by running any serverless command , e.g. `cd services/app-api && serverless info --stage main`. This command should print information and not return any Serverless Error around "AWS Credentials".
 
+## Adding a Service
+
+The Serverless framework calls encapsulated units of lambdas + AWS infrastructure a "service", so we've inherited this terminology from that project. All of our services live under the `./services/` directory. If you need to add a new service to the project a few things need to happen:
+
+-   `lerna create ${service-name}`. Follow Lerna's prompts and you'll end up with a directory under `./services/` with a generated `package.json` and `README.md` file.
+-   Add a `serverless.yml` file to the root directory of this new service. You can copy off of an existing config or run the `serverless` command in `./services/${service-name}` to use one of their starter templates.
+-   If this service is going to require js or ts code, you'll want to create a `src` directory as well as copy over the appropriate `tsconfig.json` and `.eslintrc` configs. Refer to one of the existing services to get an idea of how we are currently doing this.
+
+You'll need to add this service to our deployment GitHub Actions workflows:
+
+-   If it is only infrastructure it can be added to `./.github/workflows/deploy-infra-to-env.yml`.
+-   Services that include application code can be added to `./.github/workflows/deploy-app-to-env.yml`.
+-   We have a CI script that skips branch redeploys when possible in `./scripts/get-changed-services/index.ts`. Make sure your service is added to that list.
+
 ## Contributing
 
 We welcome contributions to this project. MC Review is an internal CMS tool for facilitating the review of state Medicaid contracts. It is developed by a federal contracting team under contract with CMS and is deployed internally for that purpose. MC Review is built using agile development processes and accepts both issues and feature requests via GitHub issues on this repository. If you’d like to contribute back any changes to this code base, please create a Pull Request and a team member will review your work. While this repository is dedicated primarily to delivering MC Review to the government, if you find any parts of it useful or find any errors in the code we would love your contributions and feedback. All contributors are required to follow our [Code of Conduct](./CODE_OF_CONDUCT.md)

diff --git a/scripts/get-changed-services/action.yml b/scripts/get-changed-services/action.yml
@@ -0,0 +1,12 @@
+name: 'Get Changed Services'
+description: 'Finds services on a PR branch that need to be deployed'
+inputs:
+  branchName:
+    description: 'The current branch that we are deploying'
+    required: true
+outputs:
+  changed-services:
+    description: 'An array of services that need to be deployed'
+runs:
+  using: 'node16'
+  main: 'index.js'