Releasing NationalMap and InvestorMap

Here is the process that we use at Data61 to release a new version of NationalMap or Investor Map to Geoscience Australia.

• Steps marked "NM only" apply for NationalMap releases.
• Steps marked "IM only" apply for Investor Map releases.
• Steps marked "NM and IM" apply to both NationalMap and Investor Map releases.

Configure your Amazon Web Services credentials

You need:

• Access to the nationalmap account on AWS.
• Appropriate AWS credentials configured in a nationalmap profile so that you can upload files to S3 and deploy a CloudFormation stack.

If you are using macOS:

• Install gnu-tar with brew install gnu-tar

Release TerriaJS (NM only)

NationalMap releases should always use an official version of TerriaJS released on npm. To publish a new version of TerriaJS to npm, you'll need to be listed as a collaborator here. Existing collaborators can add new ones. Then:

• Checkout and pull the master branch of terriajs, run npm install if necessary.
• Bump the version number in package.json. Follow semver.
• Review and edit CHANGES.md. It can be helpful to diff against the last version to verify that all the changes made it into the right section. Make sure the section name matches the version you set in package.json above.
• Commit and push your changes. Directly to master is fine.
• rm -rf wwwroot/build
• Make sure you don't have any changes in your working directory.
• Install python dependencies for docs (pip install -r doc/requirements.txt) or activate your virtual environment with the dependencies installed.
• gulp lint docs release
• npm publish (npm publish --tag mobx for mobx)
• Publish the updated documentation to terria.io: npm run publish-doc. Verify that the documentation at https://docs.terria.io still works. Note that it may take a few minutes for GitHub Pages to actually publish your update.

The above will publish a new version to npm and also tag that version on GitHub.

Publish the latest catalogue to S3 (NM only)

This step is only necessary if the catalogue changed since last release.

• Checkout and pull the master branch of NationalMap-Catalog, run npm install if ncessary.
• Run gulp. If there any changes in your working directory afterward, inspect them (usually it will just be a new LGA being added or something like that) and then commit and push them.
• Update/verify CHANGES.md.
• Tag the catalogue release: git tag -a 2016-08-08 -m '2016-08-08 release'
• Push the tag: git push origin 2016-08-08
• Copy the catalogue to S3 by running ./publish.sh 2016-08-08 where 2016-08-08 is today's date. If, for some reason, you already have a deployed instance of NationalMap file named after today's date, add b to end of it or something like that.

Prepare NationalMap (NM only)

• Update package.json to refer to the new version of TerriaJS that you published above. Verify that both NationalMap and TerriaJS use the same version of terriajs-cesium and react. Usually you'll update NationalMap to match TerriaJS. You can automate this by running gulp sync-terriajs-dependencies.
• Update CHANGES.md. Include relevant changes from both terriajs and NationalMap-Catalog.
• Copy the existing private client config file from S3: aws s3 --profile nationalmap cp <packagejson.config.awsS3ClientConfigOverridePath> ./wwwroot/privateconfig.json where <packagejson.config.awsS3ClientConfigOverridePath> is the value of the awsS3ClientConfigOverridePath config property in package.json.
• Edit wwwroot/privateconfig.json with HTTP path to the new catalog file on S3. Don't forget to include the /proxy/_60s/ bit on the front. You can probably just change the date on the end.
• Copy the new client config to S3: aws s3 --profile nationalmap cp ./wwwroot/privateconfig.json s3://nationalmap-apps/nationalmap/privateclientconfig-2016-08-08.json where 2016-08-08 is today's date.
• Change the awsS3ClientConfigOverridePath property in package.json to the new path above.
• rm wwwroot/privateconfig.json
• You may want to update awsEc2ImageId in package.json to refer to the latest version, which will make deployments go faster because fewer patches need to be installed. You can obtain the latest AMI ID with: curl -s https://cloud-images.ubuntu.com/query/trusty/server/released.current.txt | grep ap-southeast-2 | grep amd64 | grep hvm | grep ebs-ssd
• Commit and push your changes (directly to master is fine). Verify that your working directory is clean.
• Tag the release: git tag -a 2016-08-08 -m '2016-08-08 release'. Do not push the tag yet.

Prepare Investor Map (formerly Northern Australia Map) (IM only)

git clone https://github.com/TerriaJS/InvestorMap
cd InvestorMap

# Manual steps if it has been more than a week or two since the last release:

# Merge the latest changes from NationalMap into Investor Map. Commit and push these changes.
git remote add nm https://github.com/TerriaJS/NationalMap
git fetch nm
git merge nm/master

# There may be merge conflicts, particularly in package.json. Resolve these by hand, generally
# favouring HEAD (InvestorMap). 
# Then `git add` the files.
git commit
git push -u origin master

# Make sure we have the latest catalog
rm -rf node_modules/nationalmap-catalog && npm install

Publish the Investor Map catalog (IM)

Automatic method:

npm run release-catalog

Or manually:

# Build the catalog
gulp build-catalog

#Publish the new catalog to S3. Update with today's date, like 2018-12-17
DATE=`date "+%Y-%m-%d"`
echo $DATE
gzip --keep wwwroot/init/investormap.json 
aws s3 --profile nationalmap cp --content-encoding gzip --content-type "application/json" wwwroot/init/investormap.json.gz s3://static.nationalmap.nicta.com.au/investormap/init/${DATE}.json
rm wwwroot/init/investormap.json.gz

• Edit wwwroot/config.json to refer to the catalog file published above. Generally you will only need to change the date in initializationUrls.

# Commit and push this change.

git add wwwroot/init/investormap*.json
git add wwwroot/config.json
git commit -m 'Update initialization URL'
git push origin master

# Tag the release. (The most recent annotated tag is used as the subdomain name)
# Do not push the tag yet, in case something goes wrong and you need to tag a different commit and re-release.
git tag -a ${DATE} -m "${DATE} release"

Deploy a CloudFormation stack (NM and IM)

# This will do a full rebuild, then build a CloudFormation stack and upload. It will take awhile. You will need your MFA device.
npm run deploy

• Wait until both the CloudFormation is finished being created and the EC2 instance has finished initializing. You can check this by logging into the AWS console (in the NationalMap role).
• Visit the new site at https://nationalmap-2016-08-08.nm.terria.io or https://investormap-2016-08-08.nm.terria.io where 2016-08-08 is the name of the tag you created above. Wait 10 minutes before trying. Verify everything works and you are happy to commit to this release.
• If something is wrong:
  • delete the stack
  • delete the local tag that you hadn't pushed yet
  • go back and try again. :)
• Point staging.nm.terria.io or inv-staging.nm.terria.io at the new stack using the AWS Route53 UI.
• Remove the previous staging CloudFormation stack.

Test Investor Map at /investormap/ (IM only)

Test Investor Map at https://inv-staging.nm.terria.io/investormap/, especially for changes to help pages or token-enabled layers, to find any relative url issues that otherwise would only show up in GA's preprod environment.

Send the release to Geoscience Australia (NM and IM)

• Push the tags you previously created to GitHub:
  • git push origin ${DATE}
• Get the SHA256 sum of the release package:
  • Windows: sha256sum deploy/packages/*.tar.gz.
  • Mac: shasum -a 256 deploy/packages/*.tar.gz.
• Prepare an email to GA. Use previous emails as a template.
  • If there are any changes to the deployment (e.g. the user-data file), don't forget to include details in the email and to attach the file as well. Usually there aren't any changes.
  • The URL of the release package will be:
    • NM: https://staging.nm.terria.io/<SHA256SUM>/nationalmap-<DATE>.tar.gz
    • IM: https://inv-staging.nm.terria.io/<SHA256SUM>/investormap-<DATE>.tar.gz