the_workspace_run has erratic apply and destroy behaviour

[dejoost]

Terraform Cloud/Enterprise version

Terraform Cloud Plus

Terraform version

1.6.6

Terraform Configuration Files

resource "tfe_workspace_run" "runs" {
  for_each     = var.workspaces
  workspace_id = tfe_workspace.workspace[each.key].id

  dynamic "apply" {
    for_each = each.value.queue_all_runs == false ? toset([]) : toset([1]) # apply if queue_all_runs is null or true
    content {
      manual_confirm    = false
      wait_for_run      = true
      retry_attempts    = 5
      retry_backoff_min = 5
    }
  }

  destroy {
    manual_confirm    = false
    wait_for_run      = true
    retry_attempts    = 3
    retry_backoff_min = 10
  }

}

Debug Output

See screenshot

Expected Behavior

After removing the resource from the configuration the run settings should be removed from the workspace without altering the workspace resources

Actual Behavior

a destroy plan is queued in the workspace and apply is stuck for several minutes and end with an error message, see screenshot.

Additional Context

we applied tfe_workspace_run on around 80 existing workspaces, 2 succeeded (after a long time, +1 hour), 8 got stuck at applying and the other workspaces got queued, pending apply. All plans were successful. The queued and applying runs had to be cancelled through the run api.

When removing the runs on the 2 workspaces where it succeeded we observed this behaviour.

[tmatilai]

To me it sounds like its like working as expected regarding the destruction. The idea of the resource is to schedule a destroy run when the resource is removed, if the destroy block was specified. And this only works for the successfully applied workspace runs, as the failed ones were not stored in the state.

At least now there is a big warning in the docs, too:

NOTE:
Use caution when removing the `tfe_workspace_run` resource from your configuration, as
destroying it with a `destroy` block present will create a destroy run which will destroy the
workspace's underlying managed resources. To avoid this behavior, remove the `destroy`
block first.