[lib-storage] Upload fails if application provides checksum for >5 MB file #6742
Comments
trivikr
added
bug
|
This bug can be fixed when S3 allows passing
|
|
While we wait for model to updated to allow passing Workaround 1: Let SDK compute the checksumPass Test code class CustomHandler extends NodeHttpHandler {
constructor() {
super();
}
printChecksumHeaders(prefix, headers) {
for (const [header, value] of Object.entries(headers)) {
if (
header.startsWith("x-amz-checksum-") ||
header.startsWith("x-amz-sdk-checksum-")
) {
console.log(`${prefix}['${header}']: '${value}'`);
}
}
}
async handle(request, options) {
const response = await super.handle(request, options);
console.log();
console.log("------------------");
this.printChecksumHeaders("request", request.headers);
this.printChecksumHeaders("response", response.response.headers);
console.log("------------------");
console.log();
return response;
}
}
const client = new S3({ requestHandler: new CustomHandler() });
const Bucket = "test-flexible-checksums"; // Replace with your test bucket name.
const Body = createReadStream(Key);
const ChecksumAlgorithm = "SHA256";
const upload = new Upload({
client,
params: { Bucket, Key, Body, ChecksumAlgorithm },
});
await upload.done();Note that SDK sends the 6 MB file in two parts, and computes checksums for each part ------------------
request['x-amz-checksum-algorithm']: 'SHA256'
response['x-amz-checksum-algorithm']: 'SHA256'
response['x-amz-checksum-type']: 'COMPOSITE'
------------------
------------------
request['x-amz-sdk-checksum-algorithm']: 'SHA256'
request['x-amz-checksum-sha256']: 'jl28wIE8B/50cPRez5JaVTYNlvVk39FWKapxP8KEu88='
response['x-amz-checksum-sha256']: 'jl28wIE8B/50cPRez5JaVTYNlvVk39FWKapxP8KEu88='
------------------
------------------
request['x-amz-sdk-checksum-algorithm']: 'SHA256'
request['x-amz-checksum-sha256']: 'JU0DrgdnLiihtY/7GHhqvAmJv50Va9RhaNLVwUUu9NU='
response['x-amz-checksum-sha256']: 'JU0DrgdnLiihtY/7GHhqvAmJv50Va9RhaNLVwUUu9NU='
------------------
------------------
------------------Workaround 2: Use PutObject from client-s3 instead of Upload from lib-storageclass CustomHandler extends NodeHttpHandler {
constructor() {
super();
}
printChecksumHeaders(prefix, headers) {
for (const [header, value] of Object.entries(headers)) {
if (
header.startsWith("x-amz-checksum-") ||
header.startsWith("x-amz-sdk-checksum-")
) {
console.log(`${prefix}['${header}']: '${value}'`);
}
}
}
async handle(request, options) {
const response = await super.handle(request, options);
console.log();
console.log("------------------");
this.printChecksumHeaders("request", request.headers);
this.printChecksumHeaders("response", response.response.headers);
console.log("------------------");
console.log();
return response;
}
}
const client = new S3({ requestHandler: new CustomHandler() });
const Bucket = "test-flexible-checksums"; // Replace with your test bucket name.
const Body = createReadStream(Key);
await client.putObject({ Bucket, Key, Body, ChecksumSHA256 });The PutObject call will send the provided checksum when making the call ------------------
request['x-amz-checksum-sha256']: 'ZRRKEcEAxGazUzgqh+rSEecXfI27XNZQ8Uv7aMOX64s='
response['x-amz-checksum-sha256']: 'ZRRKEcEAxGazUzgqh+rSEecXfI27XNZQ8Uv7aMOX64s='
response['x-amz-checksum-type']: 'FULL_OBJECT'
------------------Between the two workarounds, we recommend using Upload without checksum, as |
|
@trivikr I think I've also seen this behavior. I am using So how I can do it if I have to do it? Like when the source (e.g. an IoT device) is sending the data to a backend app and from there we need to upload it to AWS S3. And this file is HUGE. So I need to do this and cannot let the SDK to do it for me. BTW here is a very simplified version of what I am trying to do. I really appreciate it if you could take look and tell me if there is a solution for me. And JFYI I have racked my brain to get this to work, actually you can see my footsteps almost everywhere 😂. |
|
And on another note I've read in the docs that AWS S3 due to technical limitation only support CRC-based algorithms for
But here you've use SHA256, so does that mean that the doc is outdated @trivikr? |
|
Another dirty hack would be: https://github.com/kasir-barati/bugs/blob/1149c73557b939a911ec4ec999c8430aa99124f0/upload1.ts#L36-L56 @trivikr how we can know when should we expect this bug to be fixed in the AWS SDK? |
Checkboxes for prior research
Describe the bug
Upload fails if application provides checksum for >5 MB file
Regression Issue
SDK version number
@aws-sdk/[email protected], @aws-sdk/[email protected]
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
All, verified in v22.11.0
Reproduction Steps
Observed Behavior
When
SIZE_IN_MBis greater than 5, the following error is thrownWhen
SIZE_IN_MBis less than or equal to 5, then no error is thrown.Expected Behavior
No error thrown when application provides Checksum for >5 MB file.
Possible Solution
No response
Additional Information/Context
No response
The text was updated successfully, but these errors were encountered: