Skip to content

Commit

Permalink
Stop chunked bodies causing empty ContentEncoding metadata in S3
Browse files Browse the repository at this point in the history
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
says that chunked requests should set `Content-Encoding: aws-chunked`.
S3 strips any occurrence of `aws-chunked` from that header, and stores
the remainder as the `ContentEncoding` metadata on the object. It
isn't very smart about this: if `aws-chunked` is the only encoding
listed, S3 will store the empty string as the encoding, which breaks
some HTTP clients and CDNs.

Multiple AWS SDKs appear to have hit this issue, and the developers of
`fog` contacted AWS support. Their suggestion was to omit the
`Content-Encoding:` header entirely because S3 is capable of figuring
it out: fog/fog-aws#147
  • Loading branch information
endgame committed Oct 13, 2021
1 parent 5187737 commit 6c1b1c4
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 2 deletions.
2 changes: 2 additions & 0 deletions amazonka/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ Released: **unreleased**, Compare: 2.0 RC1 (TODO: Linkify)
[\#669](https://github.com/brendanhay/amazonka/pull/669)
- Fixed S3 to use vhost endpoints, parse `LocationConstraint`s properly, and ensure `Content-MD5` header correctly set.
[\#673](https://github.com/brendanhay/amazonka/pull/673)
- Fixed S3 to not set empty `Content-Encoding`, and to be able to set `Content-Encoding` without breaking signing
[\#681](https://github.com/brendanhay/amazonka/pull/681)


### Other Changes
Expand Down
10 changes: 8 additions & 2 deletions amazonka/src/Network/AWS/Sign/V4/Chunked.hs
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,16 @@ chunked c rq a r ts = signRequest meta (toRequestBody body) auth
where
(meta, auth) = base (Tag digest) (prepare rq) a r ts

-- Although https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html says to include
-- `Content-Encoding: aws-chunked`, we don't. If it's the only header, S3 will remove
-- `aws-chunked` leaving a blank header, and store `"ContentEncoding": ""` in the object's metadata.
-- This breaks some CDNs and HTTP clients.
--
-- According to https://github.com/fog/fog-aws/pull/147 , AWS support have confirmed that the
-- header is not strictly necessary, and S3 will figure out that it's a chunked body.
prepare =
requestHeaders
<>~ [ (HTTP.hContentEncoding, "aws-chunked"),
(hAMZDecodedContentLength, toBS (_chunkedLength c)),
<>~ [ (hAMZDecodedContentLength, toBS (_chunkedLength c)),
(HTTP.hContentLength, toBS (metadataLength c))
]

Expand Down

0 comments on commit 6c1b1c4

Please sign in to comment.