Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GetBucketAccelerateConfiguration Details - NotImplemented error with third party S3 implementation #23291

Closed
command-tab opened this issue Feb 19, 2022 · 10 comments · Fixed by #35035
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/s3 Issues and PRs that pertain to the s3 service. technical-debt Addresses areas of the codebase that need refactoring or redesign.
Milestone

Comments

@command-tab
Copy link

Following hot on the heels of the resolution of #14645 (thank you!), the aws 4.2.0 provider is erroring when my NetApp StorageGRID S3 emulation returns 501 Not Implemented with an XNotImplemented error code (in the XML) when the provider performs a GET on mybucket.storagegrid.mycompany.com/?accelerate=. Apparently fetching bucket transfer acceleration is also not implemented, just like fetching a bucket's static website configuration was not supported. Would it be possible to ignore XNotImplemented in this call in a similar way to how it was handled for #14645?

Here's the Terraform log:

[DEBUG] provider.terraform-provider-aws_v4.2.0_x5: [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>XNotImplemented</Code><Message>The request you provided implies functionality that is not implemented.</Message><Resource>/mybucket?accelerate</Resource><RequestId>1645235779552623</RequestId></Error>: timestamp=2022-02-18T18:38:09.378-0800

[DEBUG] provider.terraform-provider-aws_v4.2.0_x5: [aws-sdk-go] DEBUG: Validate Response s3/GetBucketAccelerateConfiguration failed, attempt 0/25, error XNotImplemented: The request you provided implies functionality that is not implemented.
	status code: 501, request id: 1645235779552623, host id: 12445862: timestamp=2022-02-18T18:38:09.378-0800

I'm not sure how many calls like this there are beyond this one. I thought static website configuration would be the only one, but then this cropped up, and I'm not sure if there are others beyond this that will encounter the same issue. I'd be curious what other calls there are that might be problematic.

@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Feb 19, 2022
@command-tab
Copy link
Author

command-tab commented Feb 19, 2022

Looking at resourceBucketRead in internal/service/s3/bucket.go, these other calls may also be not be implemented.

  • GetBucketRequestPayment
  • GetBucketLogging
  • GetBucketLifecycleConfiguration
  • GetBucketReplication
  • GetBucketEncryption
  • GetObjectLockConfiguration
  • GetBucketTagging
  • GetBucketCors

I wonder if there's a quick way to test their availability on this third party S3 implementation without having to open a new issue for each as we work through them. Thoughts?

@lweberru
Copy link

same error here. Would be in general a good Idea, that these additional config options were optionally

@anGie44 anGie44 added service/s3 Issues and PRs that pertain to the s3 service. enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Feb 22, 2022
@anGie44
Copy link
Contributor

anGie44 commented Feb 28, 2022

Hi @command-tab , thank you for creating this issue! This looks like a great place to aggregate all the S3 methods that are facing this issue so it may help create 1 uniform solution.

@jerome-jutteau
Copy link

Hi! We are interested in this improvement too!

Check our last commits on this topic:

@MPV
Copy link

MPV commented Mar 22, 2022

On this topic, would it be an option to improve the documentation here as to include how to use this provider with non-default S3 solutions?

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/custom-service-endpoints#connecting-to-local-aws-compatible-solutions

@command-tab
Copy link
Author

command-tab commented Apr 8, 2022

I was hoping that 4.9.0's inclusion of the aws_s3_bucket_accelerate_configuration resource (and related ones, similar to the list above) would allow me to fix this issue, but that was not the case. I tried:

resource "aws_s3_bucket_accelerate_configuration" "uploads" {
  bucket   = "mybucket"
  status   = "Suspended"
}

resource "aws_s3_bucket" "uploads" {
  bucket   = "mybucket"
}

And still got this error while TF state was being refreshed:

│ Error: error getting S3 Bucket (mybucket) accelerate configuration: XNotImplemented: The request you provided implies functionality that is not implemented.
│ 	status code: 501, request id: 1649458045221114, host id: 12670683
│ 
│   with aws_s3_bucket.uploads,
│   on storagegrid.tf line 21, in resource "aws_s3_bucket" "uploads":
│   21: resource "aws_s3_bucket" "uploads" {

@command-tab
Copy link
Author

Checking in. It seems like the attempted read of bucket accelerate config is still present in version 4.18.0 of this provider and it is not safely ignoring the XNotImplemented response. Using the following Terraform config...

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "REDACTED"
  secret_key                  = "REDACTED"
  skip_credentials_validation = true
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_requesting_account_id  = true
  endpoints {
    s3 = "https://storagegrid.mycompany.com"
  }
}

resource "aws_s3_bucket" "mybucket" {
  bucket = "mybucket"
}

I see (by increasing the TF_LOG level) an HTTP 501 response (from my S3-emulating device) with an XML response body of...

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>XNotImplemented</Code>
    <Message>The request you provided implies functionality that is not implemented.</Message>
    <Resource>/mybucket?accelerate</Resource>
</Error>

...and Terraform reports...

│ Error: error getting S3 Bucket (mybucket) accelerate configuration: XNotImplemented: The request you provided implies functionality that is not implemented.
│ 	status code: 501, request id: 1654786961575377, host id: 12163780
│ 
│   with aws_s3_bucket.mybucket,
│   on storagegrid.tf line 21, in resource "aws_s3_bucket" "mybucket":
│   21: resource "aws_s3_bucket" "mybucket" {

And just to verify:

$ cat .terraform.lock.hcl 
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version = "4.18.0"
...and so on...

@absa-rsuarez
Copy link
Contributor

Adding more support for this...facing a similar issue with another 3rd party S3 storage appliance

resource "aws_s3_bucket" "test_bucket" {
  provider = aws.emc
  bucket   = "${local.common_resource_name}-test-bucket"

  lifecycle_rule {
    enabled = false
  }

  replication_configuration {
    role = ""
    rules {
      status = "Disabled"
      destination {
        bucket = ""
      }
    }
  }

  lifecycle {
    ignore_changes = all
  }
}


 Error: getting S3 Bucket replication: NotImplemented: The requested functionality is not implemented.
│ 	status code: 501, request id: 16f0bec8:18265009baf:151222:1f27, host id: 

Debug Output

2023-01-10T15:10:12.997+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: -----------------------------------------------------
2023-01-10T15:10:13.185+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetBucketReplication Details:
2023-01-10T15:10:13.185+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: ---[ RESPONSE ]--------------------------------------
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: HTTP/1.1 501 
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: Transfer-Encoding: chunked
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: Content-Type: application/xml
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: Date: Tue, 10 Jan 2023 14:10:14 GMT
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: Date: Tue, 10 Jan 2023 14:10:14 GMT
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: Server: ViPR/1.0
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: X-Amz-Id-2: 
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: X-Amz-Request-Id: 16f0bec8:18265009baf:151222:1f27
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: 
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: 
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: -----------------------------------------------------
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: [DEBUG] [aws-sdk-go] <Error><Code>NotImplemented</Code><Message>The requested functionality is not implemented.</Message><RequestId>16f0bec8:18265009baf:151222:1f27</RequestId></Error>
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/GetBucketReplication failed, attempt 0/25, error NotImplemented: The requested functionality is not implemented.
2023-01-10T15:10:13.186+0100 [DEBUG] provider.terraform-provider-aws_v4.49.0_x5: 	status code: 501, request id: 16f0bec8:18265009baf:151222:1f27, host id: 
2023-01-10T15:10:13.186+0100 [ERROR] provider.terraform-provider-aws_v4.49.0_x5: Response contains error diagnostic: @module=sdk.proto diagnostic_detail= diagnostic_severity=ERROR diagnostic_summary="getting S3 Bucket replication: NotImplemented: The requested functionality is not implemented.
	status code: 501, request id: 16f0bec8:18265009baf:151222:1f27, host id: " tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=94e73262-30f1-fd7d-c488-1733232d3cda tf_resource_type=aws_s3_bucket tf_rpc=ReadResource @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:55 timestamp=2023-01-10T15:10:13.186+0100
2023-01-10T15:10:13.188+0100 [ERROR] vertex "aws_s3_bucket.test_bucket" error: getting S3 Bucket replication: NotImplemented: The requested functionality is not implemented.
	status code: 501, request id: 16f0bec8:18265009baf:151222:1f27, host id:
2023-01-10T15:10:13.188+0100 [ERROR] vertex "aws_s3_bucket.test_bucket (expand)" error: getting S3 Bucket replication: NotImplemented: The requested functionality is not implemented.
	status code: 501, request id: 16f0bec8:18265009baf:151222:1f27, host id:

absa-rsuarez added a commit to absa-rsuarez/terraform-provider-aws that referenced this issue Jan 10, 2023
…ation

To better support 3rd party S3 storage systems which may not implement these capabilities (such as EMC ECS). Potentially resolves GitHub Issues: hashicorp#23291 (comment)
jar-b pushed a commit that referenced this issue Feb 10, 2023
…ation

To better support 3rd party S3 storage systems which may not implement these capabilities (such as EMC ECS). Potentially resolves GitHub Issues: #23291 (comment)
jar-b added a commit that referenced this issue Feb 10, 2023
…plication_configuration` reads (#29348)

* Ignore errors for GetBucketLifecycleConfiguration and GetBucketReplication

To better support 3rd party S3 storage systems which may not implement these capabilities (such as EMC ECS). Potentially resolves GitHub Issues: #23291 (comment)

* chore: changelog

---------

Co-authored-by: absa-rsuarez <[email protected]>
@github-actions github-actions bot added this to the v5.32.0 milestone Dec 22, 2023
Copy link

This functionality has been released in v5.32.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/s3 Issues and PRs that pertain to the s3 service. technical-debt Addresses areas of the codebase that need refactoring or redesign.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants