Upload utility class API should allow us to have FULL_OBJECT checksum check

[kasir-barati]

Describe the feature

I need to check the uploaded file's integrity. I get an stream of data, so I was also hoping to see a more intuitive API. I mean right now I have to calculate the checksum of the entire file first since I need it when I am instantiating a new instance of this class.

It is somewhat related to this issue: #6742

Use Case

What this means in a real world application is that my client has to read the file twice which does not make sense;

1. I am gonna ask for the checksum (need it here):
2. They will stream the file!

new Upload({
  ...,
  params: {
    ChecksumCRC32: "some checksum",
    Body: passThroughStream
  }
})

• As I've already mentioned it, this code as it is will lead to errors due to how you're just passing all the params in each request:

  • aws-sdk-js-v3/lib/lib-storage/src/Upload.ts

    Line 124 in 1f1905e

    const params = { ...this.params, Body: dataPart.data };

  • aws-sdk-js-v3/lib/lib-storage/src/Upload.ts

    Line 205 in 1f1905e

    const createCommandParams = { ...this.params, Body: undefined };

  • Especially this one, it is basically telling AWS S3 to generate a checksum for each part whereas what I want is a checksum for the whole object.

So it would be great if we could just pass the checksum for a FULL_OBJECT checksum check at the end.

Proposed Solution

new Upload({
  ...,
  params: {
    ChecksumCRC32: "some checksum",
    Body: passThroughStream,
    ChecksumType: "FULL_OBJECT",
  }
})

So now here and other places you can do a simple check, if ChecksumType is FULL_OBJECT you can skip attaching this params to the request.

Same can be applied to the rest of the places you're simply spreading whatever developer passes to the params.

Other Information

I like to contribute but to me your codebase looked like it's been generated from a python code. So I am not entirely sure how this SDK is being developed and maintained.

Note

Please take a look at my repo: https://github.com/kasir-barati/bugs/tree/aws-s3-multipart-completion-error

There you can see how ugly Upload's interface is right now. Especially this example: https://github.com/kasir-barati/bugs/blob/1149c73557b939a911ec4ec999c8430aa99124f0/upload1.ts

And I really like to know when do you think you can address this feature request.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

3.749.0

Environment details (OS name and version, etc.)

Linux, Ubuntu 24.04.01

[kasir-barati]

Workarounds

1. Using middleware: https://github.com/kasir-barati/bugs/blob/db72dbcb8f2b9c832370ea08c063b80d69912d56/upload1.ts

2. This is the modified version which is doing the job but it is dirty and just a POC: kasir-barati/bugs@cdd3342

   And this is how you can use it: https://github.com/kasir-barati/bugs/blob/db72dbcb8f2b9c832370ea08c063b80d69912d56/upload2.ts#L72-L74

[zshzbh]

Thanks for the feedback. I will forward this to the service team.

@kasir-barati Thanks for providing the workaround!