An empty Content-Encoding header is set in object's metadata after uploading the object to s3 with extra argument ChecksumAlgorithm

[ronanliu]

Describe the bug

There's an issue which is similar to #3568. When calling the method upload_file with ChecksumAlgorithm argument, an empty Content-Encoding record will be set in uploaded object's metadata.

Expected Behavior

I would expect no Content-Encoding record is set as object's metadata if I don't pass it as argument when uploading a object.

Current Behavior

Without specifying an extra argument for Content Encoding, the object's metadata still have an empty record for this header.

Reproduction Steps

import boto3

s3 = boto3.resource('s3')
s3_client = s3.meta.client

extra_args = {}

extra_args['ChecksumAlgorithm'] = 'SHA1'
extra_args['ServerSideEncryption'] = 'AES256'
extra_args['ContentType'] = 'text/css'

s3_client.upload_file(
  'FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME',
  extra_args,
)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.26.90

Environment details (OS name and version, etc.)

Linux

[tim-finnigan]

Hi @ronanliu thanks for reporting this issue. I could reproduce the behavior you described. Just to clarify the differences between #3568, the specific issue reported there now appears to be resolved, as the ContentEncoding header will now get set to when specified along with ChecksumAlgorithm. However, only specifying a ChecksumAlgorithm using the code snippet in #3568 also adds an empty ContentEncoding header as you described here. I will mark this issue for further investigation.

[tim-finnigan]

Upon further discussion with a colleague, we concluded that this is likely an issue with the S3 console. The payloads generated by the code snippets referenced above include Content-Encoding': b'aws-chunked' when using a ChecksumAlgorithm. This documentation provides some more context: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.

I created a new issue in our cross-SDK repository (aws/aws-sdk#498) and will reach out to S3 for further review. For updates please refer to that issue going forward. Thanks!

[ronanliu]

Hi @tim-finnigan , thanks for looking into it!. I'll keep an eye on the new issue.

[jmklix]

Re-opening this issue to track updates here. Closing the old tracking issue

P83760338 / D119267049

[ima747]

Since aws/aws-sdk#498 (comment) has been closed in favor of this to track, just bringing back up that this happens from Boto3 as well so is not limited to console uploads as theorized above. There are reports of the same behavior with sdks for other languages as well, implying this is more fundamental.

[alex-keeler]

Since I haven't seen anyone mention this -- it does appear to be documented behavior.
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

"If you specify Content-Encoding in your request as Content-Encoding : aws-chunked, S3 adds an empty value for Content-Encoding and stores the object metadata (Content-Encoding : ) to the resulting object."

The issue is that an empty Content-Encoding header is invalid and causes undefined behavior, i.e. when it conflicts with cloudfront's compression headers.

Just throwing this out there in case it helps locate the root cause.

[ianbotsf]

This issue has now been resolved by S3. Going forward, uploading objects via chunked encoding will no longer result in storing or retrieving Content-Encoding: . The service documentation has also been updated to reflect this:

Amazon S3 stores the resulting object without the aws-chunked value in the content-encoding header. If aws-chunked is the only value that you pass in the content-encoding header, S3 considers the content-encoding header empty and does not return this header when your retrieve the object.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.