Deploying a Notify Development Environment
This is an overview of the steps needed to configure and deploy a Notify development environment using the deploy pipeline. The intended audience is developers with existing knowledge of the Notify deploy pipeline, detailed here.
Before building a development environment you should:
- Find a development environment that is not in use by checking the #govuk-notify-infrastructure-team slack channel topic
- Update the #govuk-notify-infrastructure-team slack channel topic to claim the development environment you are going to use.
- Make sure you are on the VPN (you will need this for concourse and set-pipeline-vars)
- Pull the latest version of these three repos:
- Check you have the $PASSWORD_STORE_DIR environment variable set by running echo $PASSWORD_STORE_DIR. This should point to the local directory containing notifications-credentials. If you do not have this set, add this line to your ~/.zshrc file and load a new terminal window:
export PASSWORD_STORE_DIR=/full/path/to/notifications-credentials
- You will need an up-to-date version of:
- GDS-CLI
- Terraform using tfenv (v1.7.5 and above), instructions for macOS.
From your terminal navigate to your notifications-aws-account-wide-terraform repo directory. Run the following command where <env> is formatted like "dev-c":
gds aws notify-<env>-admin -- make <env> apply
This will deploy the latest account-wide terraform to the dev environment updating things like concourse worker permissions. In most cases it is unlikely to make any changes, but is worth checking before proceeding.
From your terminal navigate to your notifications-aws/scripts/upload-credentials repo directory. Run the following command, where <env> is formatted like "dev-c":
gds aws notify-<env>-admin -- ./upload-credentials.sh <env>
This will upload dev environment specific credentials needed for the deployment. These update the parameter store of eu-west-1 (Ireland) in the dev specific account.
In concourse each development environment has a concourse team and within that team a concourse pipeline called deploy-notify. This makes up the deployment pipeline for that whole environment. The pipeline consists of four tabs:
deployoperatorpack-bagdestroy
To deploy an environment you initially run the pack-bag job, which once complete, triggers the start-deploy job in the deploy tab of the deploy-notify pipeline.
To destroy an environment you can run the destroy job. When you do, don't forget to update the #govuk-notify-infrastructure-team slack channel topic to show the dev environment is free for the next person.
To perform pipeline control tasks you can use jobs in the operator tab, for example if you need to forcibly release the pipeline lock if it has not been relinquished by a failed job (care must be taken to ensure no jobs are running that have legitimately taken the lock).
The deploy pipeline is a single pipeline for the whole of Notify. The versions of the different apps which make up a deployment are dictated by the resources that are input to the pack-bag job.
- You should pin the versions of each resource while you are developing a feature, as this ensures that changes to the latest versions of other apps do not interfere with the effects of changes you are working on.
Each resource input to the pack-bag job is sourced from an AWS Parameter Store (SSM) variable, specifically in the notify-deploy AWS account in the London region (eu-west-2). NB other environment specific AWS accounts all reside in the Ireland region (eu-west-1).
To restore all SSM parameters used by the concourse deploy pipeline to their default values you can use the following make target in notifications-aws/concourse/deploy-notify where <env> is formatted like "dev-c":
gds aws notify-deploy-devsecrets -- make <env> set-pipeline-vars
There may be some errors relating to missing items in the password store. This is normal, as these are creds used by other builds.
The way that you set custom pack-bag input resources in dev environments is different depending on whether the resource is a git-repo concourse resource (e.g.
)
or a docker-image concourse resource (e.g.
).
-
Git-reporesources such asnotifications-awscan be modified to use code from a specific branch by altering the value of the corresponding<resource>_branchvariable in SSM Parameter Store. Be aware that resources in concourse appear with dashes "-", but their corresponding resource paths in SSM use underscores "_". -
Docker-imageresources can be modified to use specific images by altering a corresponding<resource>_tag_regexparameter in SSM Parameter Store. Note that docker images are built using the existing pipelines in thenotifyconcourse team, usually in thepull-requeststeam. The tag corresponding to the image you wish to use is defined there.
Log into the deploy-notify console in the London region:
gds aws notify-deploy-devsecrets -l
Navigate to Parameter Store (ensuring the London eu-west-2 region is selected):
Search for the specific dev environment you wish to modify:
Locate the SSM parameter corresponding to the resource you wish to modify to a different source branch or image tag.
- For the branch of a git-repo resource you need to locate the _branch_name parameter and enter the branch name
- For a docker image resource you need to locate the _tag_regex parameter and enter a regex matching the PR you're using to build your dev PR image. The default regex should be
^\d+-\d+-\d+.*, you will modify to a regex to something like^pr-4321-.*).
You can ignore this error, the devsecrets account has limited permissions intentionally:
Return to concourse and select the team for your dev environment. In the pack-bag tab you can now click into that resource and verify that the latest hash of the commit (if a git-repo resource) or hash of the target image in ECR is correct. It may take a few minutes for concourse to detect changes to it’s SSM source variables and retrieve the correct resources:
Once you have set up the required inputs to your pack-bag and pinned the versions of every resource that you are not working on, you can start the pack-bag job to begin a deployment of the dev environment.
If you find that the deployment seems to hang at the start-deploy job, specifically on the step acquiring the pipeline-lock, you may need to forcibly unlock the pipeline using the force-unlock-pipeline job under the operator tab (
As the deployment progresses through the jobs of the deploy pipeline the same deployment-bag is fetched by each job, assuring that the same versions of each app are played together. You must allow the jobs to run in sequence before you can assume that the latest deployment is available (e.g. don't force the tests job to run before the deploy job has completed).
If a job fails for transient reasons you can rerun it by clicking into the job and clicking "trigger a new build". Because of the way the deployment-bag is consumed, it will run idempotently as long as the same deployment-bag is still available (if you have modified variables in SSM since deployment you may find old deployment bags are no longer available).
Important note. The instructions in this sub section do not apply to dev-livemail. In dev-livemail SES is in production mode and can send emails to any address.
The Simple Email Service (SES) in the dev environments (with the exception of dev-livemail) is running in sandbox mode. When SES is in the sandbox mode, you can only send email to verified identities. A verified identity is an email addresses that you've proven that you own. Additionally, when SES is in sandbox mode, there are limits to the volume of email you can send each day, and to the number of messages you can send each second.
Therefore to sign up for a Notify account or send email notifications to yourself via Notify in a dev environment, you must make sure your email address is configured as a verified identity in SES.
SES sandbox identities are not managed via terraform, and should be manually added to each dev environment that you are using. To do this, log in to the dev environment with full admin privileges with the following command:
gds aws notify-dev-<env>-admin -l
From the AWS console, choose Amazon Simple Email Service (SES) using the services search menu. Then click Identities under the Configuration menu on the left hand menu. Here you will see a list of identities set up in SES.
Use the "Create Identify" button to add a new email address (not domain) to the list that SES is allowed to send to. SES will send a verification message with a magic link to the email address that you configured to verify you are able to access the address. Once an identity is verified, the dev environment will be able to send emails to that address.
"Shutting down" currently means destroying. This can be done by going to the destroy tab of the pipeline and triggering a new "start-destroy" job. (Anything clickops'd will be lost). Other jobs need to be finished before running destroy (it shares the pipeline lock).
After the destroy is complete go back to the pack bag and unpin the components (so the environment is ready for the next person to configure).
Don't forget to edit the slack channel topic on #govuk-notify-infrastructure-team to show that the dev environment that has been destroyed is now available for reuse.