First apply fails with "Error putting S3 policy: OperationAborted"

[iainelder]

The full error is

Error: Error putting S3 policy: OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.
	status code: 409, request id: 64594BB0B34560F3, host id: 4AoXq2VVwqUyjqdp5ZiQDh6lKrR1XEbUcYUUnQan3pwIc0Tc3eBy9w9B8c/MRCFWadtG1bz6wWA=

If I plan and apply a second time the apply completes successfully.

I would expect the first apply to be enough.

I'm not sure if this is a problem with the module or with the AWS provider. See similar issue hashicorp/terraform-provider-aws#7628.

Full output:

$ (cd tfb && terraform plan -var profile=saa-tfb -out ~/tmp/tfplan)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # aws_iam_user.terraform will be created
  + resource "aws_iam_user" "terraform" {
      + arn           = (known after apply)
      + force_destroy = false
      + id            = (known after apply)
      + name          = "TerraformUser"
      + path          = "/"
      + unique_id     = (known after apply)
    }

  # aws_iam_user_policy_attachment.remote_state_access will be created
  + resource "aws_iam_user_policy_attachment" "remote_state_access" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + user       = "TerraformUser"
    }

  # module.remote_state.data.aws_iam_policy_document.replica_force_ssl will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "replica_force_ssl"  {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "s3:*",
            ]
          + effect    = "Deny"
          + resources = [
              + (known after apply),
              + (known after apply),
            ]
          + sid       = "AllowSSLRequestsOnly"

          + condition {
              + test     = "Bool"
              + values   = [
                  + "false",
                ]
              + variable = "aws:SecureTransport"
            }

          + principals {
              + identifiers = [
                  + "*",
                ]
              + type        = "*"
            }
        }
    }

  # module.remote_state.data.aws_iam_policy_document.state_force_ssl will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "state_force_ssl"  {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "s3:*",
            ]
          + effect    = "Deny"
          + resources = [
              + (known after apply),
              + (known after apply),
            ]
          + sid       = "AllowSSLRequestsOnly"

          + condition {
              + test     = "Bool"
              + values   = [
                  + "false",
                ]
              + variable = "aws:SecureTransport"
            }

          + principals {
              + identifiers = [
                  + "*",
                ]
              + type        = "*"
            }
        }
    }

  # module.remote_state.aws_dynamodb_table.lock will be created
  + resource "aws_dynamodb_table" "lock" {
      + arn              = (known after apply)
      + billing_mode     = "PAY_PER_REQUEST"
      + hash_key         = "LockID"
      + id               = (known after apply)
      + name             = "tf-remote-state-lock"
      + stream_arn       = (known after apply)
      + stream_label     = (known after apply)
      + stream_view_type = (known after apply)
      + tags             = {
          + "Terraform" = "true"
        }

      + attribute {
          + name = "LockID"
          + type = "S"
        }

      + point_in_time_recovery {
          + enabled = (known after apply)
        }

      + server_side_encryption {
          + enabled     = (known after apply)
          + kms_key_arn = (known after apply)
        }
    }

  # module.remote_state.aws_iam_policy.replication[0] will be created
  + resource "aws_iam_policy" "replication" {
      + arn         = (known after apply)
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "tf-remote-state-replication-policy"
      + path        = "/"
      + policy      = (known after apply)
    }

  # module.remote_state.aws_iam_policy.terraform[0] will be created
  + resource "aws_iam_policy" "terraform" {
      + arn         = (known after apply)
      + id          = (known after apply)
      + name        = (known after apply)
      + name_prefix = "terraform"
      + path        = "/"
      + policy      = (known after apply)
    }

  # module.remote_state.aws_iam_policy_attachment.replication[0] will be created
  + resource "aws_iam_policy_attachment" "replication" {
      + id         = (known after apply)
      + name       = "tf-iam-role-attachment-replication-configuration"
      + policy_arn = (known after apply)
      + roles      = (known after apply)
    }

  # module.remote_state.aws_iam_role.replication[0] will be created
  + resource "aws_iam_role" "replication" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "s3.amazonaws.com"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + max_session_duration  = 3600
      + name                  = (known after apply)
      + name_prefix           = "tf-remote-state-replication-role"
      + path                  = "/"
      + tags                  = {
          + "Terraform" = "true"
        }
      + unique_id             = (known after apply)
    }

  # module.remote_state.aws_kms_key.replica will be created
  + resource "aws_kms_key" "replica" {
      + arn                      = (known after apply)
      + customer_master_key_spec = "SYMMETRIC_DEFAULT"
      + deletion_window_in_days  = 30
      + description              = "The key used to encrypt the remote state bucket."
      + enable_key_rotation      = true
      + id                       = (known after apply)
      + is_enabled               = true
      + key_id                   = (known after apply)
      + key_usage                = "ENCRYPT_DECRYPT"
      + policy                   = (known after apply)
      + tags                     = {
          + "Terraform" = "true"
        }
    }

  # module.remote_state.aws_kms_key.this will be created
  + resource "aws_kms_key" "this" {
      + arn                      = (known after apply)
      + customer_master_key_spec = "SYMMETRIC_DEFAULT"
      + deletion_window_in_days  = 30
      + description              = "The key used to encrypt the remote state bucket."
      + enable_key_rotation      = true
      + id                       = (known after apply)
      + is_enabled               = true
      + key_id                   = (known after apply)
      + key_usage                = "ENCRYPT_DECRYPT"
      + policy                   = (known after apply)
      + tags                     = {
          + "Terraform" = "true"
        }
    }

  # module.remote_state.aws_s3_bucket.replica will be created
  + resource "aws_s3_bucket" "replica" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = (known after apply)
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = "tf-remote-state-replica"
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags                        = {
          + "Terraform" = "true"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + lifecycle_rule {
          + enabled = true
          + id      = (known after apply)

          + noncurrent_version_transition {
              + days          = 7
              + storage_class = "GLACIER"
            }
        }

      + versioning {
          + enabled    = true
          + mfa_delete = false
        }
    }

  # module.remote_state.aws_s3_bucket.state will be created
  + resource "aws_s3_bucket" "state" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = (known after apply)
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = "tf-remote-state"
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags                        = {
          + "Terraform" = "true"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + lifecycle_rule {
          + enabled = true
          + id      = (known after apply)

          + noncurrent_version_transition {
              + days          = 7
              + storage_class = "GLACIER"
            }
        }

      + replication_configuration {
          + role = (known after apply)

          + rules {
              + id     = "replica_configuration"
              + status = "Enabled"

              + destination {
                  + bucket             = (known after apply)
                  + replica_kms_key_id = (known after apply)
                  + storage_class      = "STANDARD"
                }

              + source_selection_criteria {
                  + sse_kms_encrypted_objects {
                      + enabled = true
                    }
                }
            }
        }

      + server_side_encryption_configuration {
          + rule {
              + apply_server_side_encryption_by_default {
                  + kms_master_key_id = (known after apply)
                  + sse_algorithm     = "aws:kms"
                }
            }
        }

      + versioning {
          + enabled    = true
          + mfa_delete = false
        }
    }

  # module.remote_state.aws_s3_bucket_policy.replica_force_ssl will be created
  + resource "aws_s3_bucket_policy" "replica_force_ssl" {
      + bucket = (known after apply)
      + id     = (known after apply)
      + policy = (known after apply)
    }

  # module.remote_state.aws_s3_bucket_policy.state_force_ssl will be created
  + resource "aws_s3_bucket_policy" "state_force_ssl" {
      + bucket = (known after apply)
      + id     = (known after apply)
      + policy = (known after apply)
    }

  # module.remote_state.aws_s3_bucket_public_access_block.replica will be created
  + resource "aws_s3_bucket_public_access_block" "replica" {
      + block_public_acls       = true
      + block_public_policy     = true
      + bucket                  = (known after apply)
      + id                      = (known after apply)
      + ignore_public_acls      = true
      + restrict_public_buckets = true
    }

  # module.remote_state.aws_s3_bucket_public_access_block.state will be created
  + resource "aws_s3_bucket_public_access_block" "state" {
      + block_public_acls       = true
      + block_public_policy     = true
      + bucket                  = (known after apply)
      + id                      = (known after apply)
      + ignore_public_acls      = true
      + restrict_public_buckets = true
    }

Plan: 15 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + kms_key      = (known after apply)
  + state_bucket = (known after apply)

------------------------------------------------------------------------

This plan was saved to: /home/isme/tmp/tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "/home/isme/tmp/tfplan"

$ (cd tfb && terraform apply ~/tmp/tfplan)
module.remote_state.aws_kms_key.this: Creating...
module.remote_state.aws_kms_key.replica: Creating...
aws_iam_user.terraform: Creating...
module.remote_state.aws_iam_role.replication[0]: Creating...
module.remote_state.aws_s3_bucket.replica: Creating...
module.remote_state.aws_dynamodb_table.lock: Creating...
aws_iam_user.terraform: Creation complete after 1s [id=TerraformUser]
module.remote_state.aws_iam_role.replication[0]: Creation complete after 1s [id=tf-remote-state-replication-role20210216094217541000000001]
module.remote_state.aws_dynamodb_table.lock: Creation complete after 5s [id=tf-remote-state-lock]
module.remote_state.aws_s3_bucket.replica: Creation complete after 5s [id=tf-remote-state-replica20210216094217542700000001]
module.remote_state.data.aws_iam_policy_document.replica_force_ssl: Reading...
module.remote_state.aws_s3_bucket_public_access_block.replica: Creating...
module.remote_state.data.aws_iam_policy_document.replica_force_ssl: Read complete after 0s [id=3647925587]
module.remote_state.aws_s3_bucket_policy.replica_force_ssl: Creating...
module.remote_state.aws_s3_bucket_public_access_block.replica: Creation complete after 1s [id=tf-remote-state-replica20210216094217542700000001]
module.remote_state.aws_kms_key.this: Creation complete after 6s [id=a9645197-93df-4398-8f67-f87d3f1f822b]
module.remote_state.aws_kms_key.replica: Creation complete after 6s [id=1ebaeaea-edeb-483b-9d3c-280406dfd291]
module.remote_state.aws_s3_bucket.state: Creating...
module.remote_state.aws_s3_bucket.state: Creation complete after 8s [id=tf-remote-state20210216094224163400000002]
module.remote_state.data.aws_iam_policy_document.state_force_ssl: Reading...
module.remote_state.aws_s3_bucket_public_access_block.state: Creating...
module.remote_state.aws_iam_policy.terraform[0]: Creating...
module.remote_state.aws_iam_policy.replication[0]: Creating...
module.remote_state.data.aws_iam_policy_document.state_force_ssl: Read complete after 0s [id=3354841211]
module.remote_state.aws_s3_bucket_policy.state_force_ssl: Creating...
module.remote_state.aws_s3_bucket_policy.state_force_ssl: Creation complete after 0s [id=tf-remote-state20210216094224163400000002]
module.remote_state.aws_s3_bucket_public_access_block.state: Creation complete after 0s [id=tf-remote-state20210216094224163400000002]
module.remote_state.aws_iam_policy.replication[0]: Creation complete after 1s [id=arn:aws:iam::192985681585:policy/tf-remote-state-replication-policy20210216094231689700000005]
module.remote_state.aws_iam_policy_attachment.replication[0]: Creating...
module.remote_state.aws_iam_policy.terraform[0]: Creation complete after 1s [id=arn:aws:iam::192985681585:policy/terraform20210216094231678800000004]
aws_iam_user_policy_attachment.remote_state_access: Creating...
module.remote_state.aws_iam_policy_attachment.replication[0]: Creation complete after 1s [id=tf-iam-role-attachment-replication-configuration]
aws_iam_user_policy_attachment.remote_state_access: Creation complete after 1s [id=TerraformUser-20210216094233630600000006]

Error: Error putting S3 policy: OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.
	status code: 409, request id: 64594BB0B34560F3, host id: 4AoXq2VVwqUyjqdp5ZiQDh6lKrR1XEbUcYUUnQan3pwIc0Tc3eBy9w9B8c/MRCFWadtG1bz6wWA=


$ (cd tfb && terraform plan -var profile=saa-tfb -out ~/tmp/tfplan)

module.remote_state.aws_kms_key.this: Refreshing state... [id=a9645197-93df-4398-8f67-f87d3f1f822b]
module.remote_state.aws_kms_key.replica: Refreshing state... [id=1ebaeaea-edeb-483b-9d3c-280406dfd291]
module.remote_state.aws_iam_role.replication[0]: Refreshing state... [id=tf-remote-state-replication-role20210216094217541000000001]
module.remote_state.aws_s3_bucket.replica: Refreshing state... [id=tf-remote-state-replica20210216094217542700000001]
aws_iam_user.terraform: Refreshing state... [id=TerraformUser]
module.remote_state.aws_dynamodb_table.lock: Refreshing state... [id=tf-remote-state-lock]
module.remote_state.aws_s3_bucket_public_access_block.replica: Refreshing state... [id=tf-remote-state-replica20210216094217542700000001]
module.remote_state.aws_s3_bucket.state: Refreshing state... [id=tf-remote-state20210216094224163400000002]
module.remote_state.aws_iam_policy.replication[0]: Refreshing state... [id=arn:aws:iam::192985681585:policy/tf-remote-state-replication-policy20210216094231689700000005]
module.remote_state.aws_s3_bucket_public_access_block.state: Refreshing state... [id=tf-remote-state20210216094224163400000002]
module.remote_state.aws_iam_policy.terraform[0]: Refreshing state... [id=arn:aws:iam::192985681585:policy/terraform20210216094231678800000004]
module.remote_state.aws_s3_bucket_policy.state_force_ssl: Refreshing state... [id=tf-remote-state20210216094224163400000002]
aws_iam_user_policy_attachment.remote_state_access: Refreshing state... [id=TerraformUser-20210216094233630600000006]
module.remote_state.aws_iam_policy_attachment.replication[0]: Refreshing state... [id=tf-iam-role-attachment-replication-configuration]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.remote_state.aws_s3_bucket_policy.replica_force_ssl will be created
  + resource "aws_s3_bucket_policy" "replica_force_ssl" {
      + bucket = "tf-remote-state-replica20210216094217542700000001"
      + id     = (known after apply)
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "s3:*"
                      + Condition = {
                          + Bool = {
                              + aws:SecureTransport = [
                                  + "false",
                                ]
                            }
                        }
                      + Effect    = "Deny"
                      + Principal = "*"
                      + Resource  = [
                          + "arn:aws:s3:::tf-remote-state-replica20210216094217542700000001/*",
                          + "arn:aws:s3:::tf-remote-state-replica20210216094217542700000001",
                        ]
                      + Sid       = "AllowSSLRequestsOnly"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: /home/isme/tmp/tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "/home/isme/tmp/tfplan"

$ (cd tfb && terraform apply ~/tmp/tfplan)
module.remote_state.aws_s3_bucket_policy.replica_force_ssl: Creating...
module.remote_state.aws_s3_bucket_policy.replica_force_ssl: Creation complete after 0s [id=tf-remote-state-replica20210216094217542700000001]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

kms_key = "a9645197-93df-4398-8f67-f87d3f1f822b"
state_bucket = "tf-remote-state20210216094224163400000002"
$ (cd tfb && terraform plan -var profile=saa-tfb -out ~/tmp/tfplan)
aws_iam_user.terraform: Refreshing state... [id=TerraformUser]
module.remote_state.aws_kms_key.this: Refreshing state... [id=a9645197-93df-4398-8f67-f87d3f1f822b]
module.remote_state.aws_kms_key.replica: Refreshing state... [id=1ebaeaea-edeb-483b-9d3c-280406dfd291]
module.remote_state.aws_s3_bucket.replica: Refreshing state... [id=tf-remote-state-replica20210216094217542700000001]
module.remote_state.aws_iam_role.replication[0]: Refreshing state... [id=tf-remote-state-replication-role20210216094217541000000001]
module.remote_state.aws_dynamodb_table.lock: Refreshing state... [id=tf-remote-state-lock]
module.remote_state.aws_s3_bucket_public_access_block.replica: Refreshing state... [id=tf-remote-state-replica20210216094217542700000001]
module.remote_state.aws_s3_bucket.state: Refreshing state... [id=tf-remote-state20210216094224163400000002]
module.remote_state.aws_s3_bucket_policy.replica_force_ssl: Refreshing state... [id=tf-remote-state-replica20210216094217542700000001]
module.remote_state.aws_iam_policy.replication[0]: Refreshing state... [id=arn:aws:iam::192985681585:policy/tf-remote-state-replication-policy20210216094231689700000005]
module.remote_state.aws_s3_bucket_public_access_block.state: Refreshing state... [id=tf-remote-state20210216094224163400000002]
module.remote_state.aws_iam_policy.terraform[0]: Refreshing state... [id=arn:aws:iam::192985681585:policy/terraform20210216094231678800000004]
module.remote_state.aws_s3_bucket_policy.state_force_ssl: Refreshing state... [id=tf-remote-state20210216094224163400000002]
aws_iam_user_policy_attachment.remote_state_access: Refreshing state... [id=TerraformUser-20210216094233630600000006]
module.remote_state.aws_iam_policy_attachment.replication[0]: Refreshing state... [id=tf-iam-role-attachment-replication-configuration]

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

[ismailyenigul]

Hi @iainelder
I just tested with latest version code in this repo. I could not reproduce it (using terrraform 0.15.5)
Can you please test again?

[iainelder]

Hi, @ismailyenigul . I'm not in a position to test this just now. I'm not currently working on Terraform projects.

It's no longer a problem for me, so feel free to close this.

[nozaq]

I also didn't see any issue in my environment now.
This issue should be fixed in #33, so I'm closing this for now.
Let's revisit here if another report comes up.