diff --git a/TESTING.md b/TESTING.md index b3959ba2eef..c63edb90a45 100644 --- a/TESTING.md +++ b/TESTING.md @@ -169,8 +169,112 @@ cargo make \ test ``` -(`-r` tells testsys to also show the status of resources like the cluster and instances in addition to tests): - ```shell cargo make watch-test ``` + +## Migration Testing + +Migration testing starts testsys at a starting version, or a provided initial image id and migrates +instances to the target version. +In order to accomplish this a few artifacts need to be created: +* Tuf repos that are publicly accessible +* A previous release of bottlerocket signed with available keys +* The ami id from the previous release +* Build images and repos for current changes + +### The setup + +#### Prepare `Infra.toml` + +A publicly available s3 bucket is needed to complete the next steps, once a bucket is created the +steps may be continued. +Infra.toml is used by testsys to determine tuf repo locations, so `metadata_base_url` and +`targets_base_url` need to be set based on the s3 bucket. The examples below assume +`metadata_base_url` is the top level directory of the bucket and `targets_base_url` is a directory +named `targets` in the top level directory. +The examples below also assume that the default repo in Infra.toml, but any repo can be used by +setting the `PUBLISH_REPO` environment variable. + +#### Starting bottlerocket images + +In this example we will use `v1.8.0` as our starting bottlerocket version, but any tag from +bottlerocket will work. +The following bash will checkout the proper branch from git and create the build images and repos +for testing. + +```shell +export TESTSYS_STARTING_VERSION="1.8.0" +git checkout v${TESTSYS_STARTING_VERSION} +cargo make +cargo make ami +cargo make repo +``` + +Next, the repos need to be added to the tuf repos (s3 bucket). Make sure to change `S3_BUCKET_NAME` +to the tuf repo bucket name. + +```shell +S3_BUCKET_NAME= +aws s3 sync build/repos/default/latest s3://${S3_BUCKET_NAME}/ +``` + +The last step with the older bottlerocket version is retrieving the ami that was created. + +```shell +BUILDSYS_ARCH="x86_64" +BUILDSYS_VARIANT="aws-k8s-1.21" +REGION="us-west-2" + +AMIS_JSON="build/images/${BUILDSYS_ARCH}-${BUILDSYS_VARIANT}/latest/bottlerocket-${BUILDSYS_VARIANT}-${BUILDSYS_ARCH}-amis.json" +export TESTSYS_STARTING_IMAGE_ID=$(jq -r --arg r "${REGION}" ' .[$r].id' "${AMIS_JSON}") +echo "Starting ami for migration tests '${TESTSYS_STARTING_IMAGE_ID}'" +``` + +Make note of the ami that is printed, it will be needed in future steps. + +#### Target bottlerocket images + +Now, it's time to create the bottlerocket artifacts that need to be upgraded to. + +Switch to the working git branch that should be built from. + +```shell +WORKING_BRANCH="develop" +git checkout ${WORKING_BRANCH} +``` + +Now, [Release.toml](Release.toml) may need to be updated. +The version at the top of the file should be changed to whatever version migrations should be tested +for. +In the example using `v1.8.0`, the version should be changed to `1.9.0`. + +Next, build bottlerocket images and repos and sync tuf repos. + +```shell +cargo make +cargo make ami +cargo make repo + +S3_BUCKET_NAME= +aws s3 sync build/repos/default/latest s3://${S3_BUCKET_NAME}/ +``` + +This completes the setup and it is time to test migrations! + +### Testing Migrations + +The previous steps set up the artifacts necessary to perform migration testing using `testsys`. +Ensure all environment variables are still set and set them if they aren't. + +```shell +echo ${TESTSYS_STARTING_IMAGE_ID} +echo ${TESTSYS_STARTING_VERSION} +``` + +To run the migration test set `TESTSYS_TEST=migrations` in the `cargo make test` call. +```shell +cargo make -e TESTSYS_TEST=migration test +``` + +To see the state of the tests as they run use `cargo make watch-test`. diff --git a/tools/testsys/src/run.rs b/tools/testsys/src/run.rs index f75e0564b21..5f9e426939e 100644 --- a/tools/testsys/src/run.rs +++ b/tools/testsys/src/run.rs @@ -218,7 +218,7 @@ pub(crate) enum TestType { /// variance this will run sonobuoy in "quick" mode. For ECS variants, this will run a simple /// ECS task. Quick, - /// Run an upgrade downgrade test on a given arch and variant. This requires + /// Migration testing ensures that all bottlerocket migrations work as expected. This requires /// `migration-starting-version` and `migration-target-version`. To target a specific starting /// ami use `starting-image-id`. A migration test starts by creating necessary instances, then /// running a quick conformance test. Then the nodes are upgraded to the