404 error on S3 replication state when refreshing

[ghost]

This issue was originally opened by @hongkongkiwi as hashicorp/terraform#25954. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.13.0
+ provider registry.terraform.io/hashicorp/aws v3.3.0

Terraform Configuration Files

provider "aws" {
  region     = var.wasabi_region
  access_key = var.wasabi_access_key
  secret_key = var.wasabi_secret_key

  endpoints {
    sts = "https://sts.${var.wasabi_region}.wasabisys.com"
    iam = "https://iam.${var.wasabi_region}.wasabisys.com"
    s3 = "https://s3.${var.wasabi_region}.wasabisys.com"
  }

  s3_force_path_style = true
  skip_get_ec2_platforms = true
  skip_requesting_account_id = true
  skip_credentials_validation = true
  skip_metadata_api_check = true
}

resource "aws_s3_bucket" "newbucket" {
  bucket = "test_bucket"
  acl    = "private"
}

Debug Output

2020-08-21T21:42:22.785+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: ---[ REQUEST POST-SIGN ]-----------------------------
2020-08-21T21:42:22.785+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: GET /test_bucket?replication= HTTP/1.1
2020-08-21T21:42:22.785+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Host: s3.us-west-1.wasabisys.com
2020-08-21T21:42:22.785+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: User-Agent: aws-sdk-go/1.34.4 (go1.14.5; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.13.0 (+https://www.terraform.io)
<---snip---->
2020-08-21T21:42:22.785+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: -----------------------------------------------------
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/21 21:42:23 [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetBucketReplication Details:
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: ---[ RESPONSE ]--------------------------------------
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: HTTP/1.1 404 Not Found
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Connection: close
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Transfer-Encoding: chunked
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Content-Type: application/xml
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Date: Fri, 21 Aug 2020 13:42:23 GMT
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Server: WasabiS3/6.2.2626-2020-07-14-cd9e935 (head04)
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: X-Amz-Id-2: dW4+bZh/X85v+FeEEwJ8e+fQXiUNaI7t7YAskvs2l8CXsO1LCpa5cmG5xQAo0XQ4/uL2DJTwqHEZ
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: X-Amz-Request-Id: 56C9F6F83462E331
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: -----------------------------------------------------
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/21 21:42:23 [DEBUG] [aws-sdk-go]
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: FeatureDisabledReplication56C9F6F83462E331dW4+bZh/X85v+FeEEwJ8e+fQXiUNaI7t7YAskvs2l8CXsO1LCpa5cmG5xQAo0XQ4/uL2DJTwqHEZ
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/21 21:42:23 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/GetBucketReplication failed, attempt 0/25, error FeatureDisabled: Replication
2020-08-21T21:42:23.334+0800 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: status code: 404, request id: 56C9F6F83462E331, host id: dW4+bZh/X85v+FeEEwJ8e+fQXiUNaI7t7YAskvs2l8CXsO1LCpa5cmG5xQAo0XQ4/uL2DJTwqHEZ
2020/08/21 21:42:23 [ERROR] eval: *terraform.EvalRefresh, err: error getting S3 Bucket replication: FeatureDisabled: Replication
status code: 404, request id: 56C9F6F83462E331, host id: dW4+bZh/X85v+FeEEwJ8e+fQXiUNaI7t7YAskvs2l8CXsO1LCpa5cmG5xQAo0XQ4/uL2DJTwqHEZ
2020/08/21 21:42:23 [ERROR] eval: *terraform.EvalSequence, err: error getting S3 Bucket replication: FeatureDisabled: Replication
status code: 404, request id: 56C9F6F83462E331, host id: dW4+bZh/X85v+FeEEwJ8e+fQXiUNaI7t7YAskvs2l8CXsO1LCpa5cmG5xQAo0XQ4/uL2DJTwqHEZ
2020/08/21 21:42:23 [TRACE] [walkRefresh] Exiting eval tree: module.wasabi-bucket.aws_s3_bucket.newbucket

Expected Behavior

I expect an option to skip refreshing replication or being able to ignore 404 error and not cancel my plan.

Actual Behavior

Wasabi is an S3 alternative that is compatible with existing AWS APIs. They don't support replication and they return a 404 error. Terraform considers this as an error and stops running the plan.

Steps to Reproduce

terraform init
terraform apply
terraform refresh

[ewbankkit]

Similar (non-AWS S3 provider):

• Acceleration NotImplemented error with third party S3 implementation #13726

[hongkongkiwi]

I would really like an option in terraform core to treat errors as warnings. That way if there’s a problem I know can be safely skipped then I can mark to continue on.

Or as another option I tried to skip this check with ignore changes, but it seems that ignore changes (under lifecycle) did not skip refresh for this item even though I specifically asked it to ignore this item.

My expectation is that when using ignore_changes it should also skip refresh.

[anarsen]

My expectation is that when using ignore_changes it should also skip refresh.

Would that not result in potentially stale values being used in case the attribute in question is referenced in other places?

[hongkongkiwi]

Hmmm I suppose I see your point.

In this case, any refresh causes a 404 error which is considered fatal and stops the plan. There is currently no such option as no_refresh and it seems the providers would need to be rewritten to support this.

I guess the neatest option is a way to not treat 404 as fatal errors when refreshing.

[andrew-womeldorf]

I've also tried using this provider to create the bucket in Wasabi, and to set the replication status to disabled:

resource "aws_s3_bucket" "this" {
  provider = aws.wasabi
  bucket   = "mybucket"

  replication_configuration {
    role = aws_iam_role.this.arn
    rules {
      status = "Disabled"
      destination {
        bucket = ""
      }
    }
  }
}

Unfortunately, the refresh still throws the same error as the original post here.

My workaround solution is to use a data block, rather than a resource block. In order to "keep" the definition around, I

• Create the bucket with terraform
• Remove the bucket from the state, and comment the resource block
• Add the data block

It's obviously not ideal, since I can't make changes to the bucket from terraform, but this way I have a record of what was created.

[ext4]

Could the fix for #13726 be applied to this scenario as well?

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.