Move cirrus definitions to YAML files

[cvangerpen]

Inputs to the task-batch-compute, task, and workflow submodules are now configured via YAML definitions stored in individual directories and files rather than long lists of HCL objects. This prevents redundant cirrus configuration across deployment environments while still allowing for environment-specific values to be set via template variables in task definitions.

A new cirrus submodule, typed-definitions, was created to facilitate this change by addressing Terraform's difficulty with implicit typecasting of complex objects derived from YAML definitions. This module is used behind the scenes and should not be called by the user.

Related issue(s)

• N/A

Proposed Changes

1. The cirrus module (and all upstream modules) no longer take lists of task, task-batch-compute, and workflow config objects. Instead, it accepts the following:
   • task_batch_compute_definitions_dir - filepath to the directory containing subdirectories with task-batch-compute YAML definitions. Filepath is relative to the Terraform deployment's root directory.
   • task_definitions_dir - filepath to the directory containing subdirectories with task YAML definitions. Filepath is relative to the Terraform deployment's root directory.
   • task_definitions_variables - map of maps to strings. Used for templating task YAML definitions with any environment-specific values, such as data buckets. The outer map keys should be task names, the inner map keys a name that represents the placeholder's purpose (e.g., { copy-assets = { image_tag = "v3.0" }}). See example below for more.
   • workflow_definitions_dir - filepath to the directory containing subdirectories with workflow YAML definitions. Filepath is relative to the Terraform deployment's root directory.
2. At runtime, the cirrus module will glob each of the specified directories above for any definition.yaml files that are one level deep, e.g. <definitions dir>/<namespaced dir>/definition.yaml. Each set of globbed YAML files are decoded into a list of their HCL equivalent objects. task YAML definitions are templated prior to HCL conversion using the task_definitions_variables contents. task-batch-compute and workflow YAML definitions are not templated; contents are the same across all environments (though workflow state machine JSONs are different by necessity - this is handled by the cirrus module).
3. Terraform struggles with implicitly typecasting lists of disparate HCL objects derived from YAML into a list of single-type objects, likely because some will omit optional attributes. This leads to a number of type-related issues during cirrus's submodule executions. Thus, the typed-definitions submodule was added under cirrus to explicitly convert these tuples to their expected type (along with some additional input validation). The resulting list of objects are passed to the task, task-batch-compute, and workflow submodules. The user does not need to use this submodule directly; the cirrus module will use it automatically.
4. The resulting task, task-batch-compute, and workflow resources are created just as before.

Converting to YAML

1. Create the following the directory structure at the root of your repository:

cirrus/
    task-batch-compute/
    tasks/
    workflows/

2. For each element in your cirrus_inputs.task_batch_compute list, perform the following steps:
   1. Create a new directory under cirrus/task-batch-compute. The directory name should match that object's name attribute.
   2. Create a file named definition.yaml under this new subdirectory. Convert the contents of that single HCL element into its YAML equivalent.
   3. (optional) Create a file named README.md with any information about this task-batch-compute item that you deem useful (purpose, etc.).

Updated layout:

cirrus/
    task-batch-compute/
        my-compute/
            definition.yaml
            README.md
        my-other-compute/
            definition.yaml
            README.md
        ...
    tasks/
    workflows/

Example source HCL:

task_batch_compute = [
  {
    name = "my-compute"
    batch_compute_environment = {
      compute_resources = {
        type = "FARGATE_SPOT"
        max_vcpus = 4
      }
    }
  },
  ...
]

Example converted YAML:

name: my-compute
batch_compute_environment:
  compute_resources:
    type: FARGATE_SPOT
    max_vcpus: 4

3. For each element in your cirrus_inputs.tasks list, perform the following steps:
   1. Create a new directory under cirrus/tasks. The directory name should match that object's name attribute.
   2. Create a file named definition.yaml under this new subdirectory. Convert the contents of that single HCL element into its YAML equivalent. If there are any values that might be specific to a given deployment environment, such as a source data bucket, you should template those now:
      • Replace the environment-specific value with a template sequence, e.g.: ${my-first-task.image_tag}
      • In the cirrus_inputs.task_definitions_variables HCL variable, add an entry with a value specific to your target environment, e.g.: { my_first_task = { image_tag = "dev", ...} ...}. This value will be templated into your task definition at runtime prior to HCL conversion.
      • Since the cirrus data bucket will always differ between environments, you may use the builtin template variable of ${CIRRUS_DATA_BUCKET} in your task YAML definition without having to add an entry for it to cirrus_inputs.task_definitions_variables.
   3. (optional) Create a file named README.md with any information about this task item that you deem useful (purpose, source code repo link, etc.).

Updated layout:

cirrus/
    task-batch-compute/
        my-compute/
            definition.yaml
            README.md
        my-other-compute/
            definition.yaml
            README.md
        ...
    tasks/
        my-first-task/
            definition.yaml
            README.md
        my-second-task/
            definition.yaml
            README.md
        ...
    workflows/

Example source HCL for a dev environment:

tasks = [
  {
    name = "my-first-task"
    common_role_statements = [
      {
        sid = "ReadSomeBucketThatsDifferentForEachEnvironment"
        effect = "Allow"
        actions = [
          "s3:ListBucket",
          "s3:GetObject",
          "s3:GetBucketLocation"
        ]
        resources = [
          "arn:aws:s3:::my-data-bucket",
          "arn:aws:s3:::my-data-bucket/*"
        ]
      },
      {
        sid = "WriteCirrusDataBucket"
        effect = "Allow"
        actions = ["s3:PutObject"]
        resources = ["arn:aws:s3:::cirrus-data-bucket/*"]
      }
    ]
    lambda = {
      description = "Cirrus task that does something"
      ecr_image_uri = "<full ECR image URI>:dev"
      vpc_enabled = true
      timeout_seconds = 900
      memory_mb = 128
      env_vars = {
        SOME_VARIABLE_YOUR_CODE_NEEDS = "some-value"
        SOME_VARIABLE_YOUR_CODE_NEEDS_THAT_CHANGES_BY_ENVIRONMENT = "DEV"
      }
    }
  },
  ...
]

Example converted YAML that would be used across all environments:

name: my-first-task
common_role_statements:
  - sid: ReadSomeBucketThatsDifferentForEachEnvironment
    effect: Allow
    actions:
      - s3:ListBucket
      - s3:GetObject
      - s3:GetBucketLocation
    resources:
      - arn:aws:s3:::${my-first-task.ancillary_data_bucket}
      - arn:aws:s3:::${my-first-task.ancillary_data_bucket}/*
  - sid: WriteCirrusDataBucket
    effect: Allow
    actions:
      - s3:PutObject
    resources:
      - arn:aws:s3:::${CIRRUS_DATA_BUCKET}/*
lambda:
  description: Cirrus task that does something
  ecr_image_uri: <full ECR image URI>:${my-first-task.image_tag}
  vpc_enabled: true
  timeout_seconds: 900
  memory_mb: 128
  env_vars:
    SOME_VARIABLE_YOUR_CODE_NEEDS: some-value
    SOME_VARIABLE_YOUR_CODE_NEEDS_THAT_CHANGES_BY_ENVIRONMENT: ${my-first-task.env_variable}

Example cirrus_inputs.task_definitions_variables for the dev environment:

task_definitions_variables = {
  my-first-task = {
    image_tag = "dev"
    ancillary_data_bucket = "my-data-bucket"
    env_variable = "DEV"
  }
}

5. For each element in your cirrus_inputs.workflows list, perform the following steps:
   1. Create a new directory under cirrus/workflows. The directory name should match that object's name attribute.
   2. Create a file named definition.yaml under this new subdirectory. Convert the contents of that single HCL element into its YAML equivalent.
      • Note that template_variables is no longer used. You'll update the the state machine JSON accordingly in the next step.
      • Note that template_filepath is now named state_machine_filepath.
   3. Create a file named state-machine.json under this new subdirectory and fill it with the contents of your state machine JSON. Make sure your definition.yaml's state_machine_filepath points to this file (path is relative to the root of the repo). If you haven't already replaced your usage of template_variables, do so now:
      • Wherever you need to reference a task resource ARN, such as a lambda function, replace your existing ${arbitrary_variable_name} with the explicit lookup value, e.g. ${<your task name>.<task type>.<task attribute>}, such as ${my-first-task.lambda.function_arn}. See here for the valid values.
      • Remove the template_variables config from your definition.yaml if you haven't already.
   4. (optional) Create a file named README.md with any information about this workflow item that you deem useful (purpose, task flow, arch diagram, etc.).

Updated layout:

cirrus/
    task-batch-compute/
        my-compute/
            definition.yaml
            README.md
        my-other-compute/
            definition.yaml
            README.md
        ...
    tasks/
        my-first-task/
            definition.yaml
            README.md
        my-second-task/
            definition.yaml
            README.md
        ...
    workflows/
        my-workflow/
            definition.yaml
            state-machine.json
            README.md
        ...

Example source HCL:

workflows = [
  {
    name = "my-workflow"
    state_machine_filepath = "old-filepath.json"
  },
  ...
]

Example converted YAML:

name: my-workflow
state_machine_filepath: "cirrus/workflows/my-workflow/state-machine.json"

Testing

This change was validated by the following observations:

1. This feature was deployed to an existing FilmDrop environment and, after running through the conversion steps outlined above, the changes were successfully deployed. If everything was configured properly, the old task-batch-compute, task, and workflow resources should be the same as they were before (though the state machine contents may change slightly based on the whims of Terraform's JSON diffing).

Checklist

• I have deployed and validated this change
• Changelog
  • I have added my changes to the changelog
  • No changelog entry is necessary
• README migration
  • I have added any migration steps to the Readme
  • No migration is necessary