Add migration testing to TESTING.md

TESTING.md

@@ -169,8 +169,112 @@ cargo make \
   test
 ```
 
-(`-r` tells testsys to also show the status of resources like the cluster and instances in addition to tests):
-
 ```shell
 cargo make watch-test
 ```
+
+## Migration Testing
+
+Migration testing starts testsys at a starting version, or a provided initial image id and migrates
+instances to the target version.
+In order to accomplish this a few artifacts need to be created:
+* Tuf repos that are publicly accessible
+* A previous release of bottlerocket signed with available keys
+* The ami id from the previous release
+* Build images and repos for current changes
+
+### The setup
+
+#### Prepare `Infra.toml`
+
+A publicly available s3 bucket is needed to complete the next steps, once a bucket is created the
+steps may be continued.
+Infra.toml is used by testsys to determine tuf repo locations, so `metadata_base_url` and
+`targets_base_url` need to be set based on the s3 bucket. The examples below assume
+`metadata_base_url` is the top level directory of the bucket and `targets_base_url` is a directory
+named `targets` in the top level directory. 
+The examples below also assume that the default repo in Infra.toml, but any repo can be used by
+setting the `PUBLISH_REPO` environment variable.
+
+#### Starting bottlerocket images
+
+In this example we will use `v1.8.0` as our starting bottlerocket version, but any tag from
+bottlerocket will work. 
+The following bash will checkout the proper branch from git and create the build images and repos
+for testing.
+
+```shell
+export TESTSYS_STARTING_VERSION="1.8.0"
+git checkout v${TESTSYS_STARTING_VERSION}
+cargo make
+cargo make ami
+cargo make repo
+```
+
+Next, the repos need to be added to the tuf repos (s3 bucket). Make sure to change `S3_BUCKET_NAME`
+to the tuf repo bucket name.
+
+```shell
+S3_BUCKET_NAME=<s3 bucket name>
+aws s3 sync build/repos/default/latest s3://${S3_BUCKET_NAME}/
+```
+
+The last step with the older bottlerocket version is retrieving the ami that was created.
+
+```shell
+BUILDSYS_ARCH="x86_64"
+BUILDSYS_VARIANT="aws-k8s-1.21"
+REGION="us-west-2"
+
+AMIS_JSON="build/images/${BUILDSYS_ARCH}-${BUILDSYS_VARIANT}/latest/bottlerocket-${BUILDSYS_VARIANT}-${BUILDSYS_ARCH}-amis.json"
+export TESTSYS_STARTING_IMAGE_ID=$(jq -r --arg r "${REGION}" ' .[$r].id' "${AMIS_JSON}")
+echo "Starting ami for migration tests '${TESTSYS_STARTING_IMAGE_ID}'"
+```
+
+Make note of the ami that is printed, it will be needed in future steps.
+
+#### Target bottlerocket images
+
+Now, it's time to create the bottlerocket artifacts that need to be upgraded to.
+
+Switch to the working git branch that should be built from.
+
+```shell
+WORKING_BRANCH="develop"
+git checkout ${WORKING_BRANCH}
+```
+
+Now, [Release.toml](Release.toml) may need to be updated.
+The version at the top of the file should be changed to whatever version migrations should be tested
+for.
+In the example using `v1.8.0`, the version should be changed to `1.9.0`.
+
+Next, build bottlerocket images and repos and sync tuf repos.
+
+```shell
+cargo make
+cargo make ami
+cargo make repo
+
+S3_BUCKET_NAME=<s3 bucket name>
+aws s3 sync build/repos/default/latest s3://${S3_BUCKET_NAME}/
+```
+
+This completes the setup and it is time to test migrations!
+
+### Testing Migrations
+
+The previous steps set up the artifacts necessary to perform migration testing using `testsys`.
+Ensure all environment variables are still set and set them if they aren't.
+
+```shell
+echo ${TESTSYS_STARTING_IMAGE_ID}
+echo ${TESTSYS_STARTING_VERSION}
+```
+
+To run the migration test set `TESTSYS_TEST=migrations` in the `cargo make test` call.
+```shell
+cargo make -e TESTSYS_TEST=migration test
+```
+
+To see the state of the tests as they run use `cargo make watch-test`.

tools/testsys/src/run.rs

@@ -218,7 +218,7 @@ pub(crate) enum TestType {
     /// variance this will run sonobuoy in "quick" mode. For ECS variants, this will run a simple
     /// ECS task.
     Quick,
-    /// Run an upgrade downgrade test on a given arch and variant. This requires
+    /// Migration testing ensures that all bottlerocket migrations work as expected. This requires
     /// `migration-starting-version` and `migration-target-version`. To target a specific starting
     /// ami use `starting-image-id`. A migration test starts by creating necessary instances, then
     /// running a quick conformance test. Then the nodes are upgraded to the