V3 SDK s3 managed uploader fails with older api version - V2 is fine

[marcolarosa]

Describe the bug

Using Upload from @aws-sdk/lib-storage against a fake S3 endpoint (RadosGW running Ceph Nautilus.) fails. The V2 equivalent works fine with the same endpoint.

Your environment

RadosGW running Ceph Nautilus. API Version: '2006-03-01'.

SDK version number

"@aws-sdk/client-s3": "^3.8.1"
"@aws-sdk/lib-storage": "^3.8.1"
"aws-sdk": "^2.865.0"

Is the issue in the browser/Node.js/ReactNative?

Node

Details of the browser/Node.js/ReactNative version

v14.16.0

Steps to reproduce

I've created a test with the v2 and v3 codes. https://github.com/marcolarosa/test-s3-multipart-upload.

Observed behavior

The V3 uploader fails with:

*** MalformedXML: MalformedXML
    at deserializeAws_restXmlCompleteMultipartUploadCommandError (/home/debian/src/test-s3-multipart-upload/node_modules/@aws-sdk/client-s3/dist/cjs/protocols/Aws_restXml.js:4170:41)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /home/debian/src/test-s3-multipart-upload/node_modules/@aws-sdk/middleware-serde/dist/cjs/deserializerMiddleware.js:6:20
    at async /home/debian/src/test-s3-multipart-upload/node_modules/@aws-sdk/middleware-signing/dist/cjs/middleware.js:12:24
    at async StandardRetryStrategy.retry (/home/debian/src/test-s3-multipart-upload/node_modules/@aws-sdk/middleware-retry/dist/cjs/defaultStrategy.js:56:46)
    at async /home/debian/src/test-s3-multipart-upload/node_modules/@aws-sdk/middleware-logger/dist/cjs/loggerMiddleware.js:6:22
    at async Upload.__doMultipartUpload (/home/debian/src/test-s3-multipart-upload/node_modules/@aws-sdk/lib-storage/dist/cjs/Upload.js:104:41)
    at async Upload.done (/home/debian/src/test-s3-multipart-upload/node_modules/@aws-sdk/lib-storage/dist/cjs/Upload.js:45:16)
    at async v3MultipartUpload (/home/debian/src/test-s3-multipart-upload/test-s3-multipart-upload.js:103:17)
    at async /home/debian/src/test-s3-multipart-upload/test-s3-multipart-upload.js:30:2 {
  Code: 'MalformedXML',
  BucketName: '1190_pdsc_2020',
  RequestId: 'tx000000000000000021d31-006051895d-91267cb-unimelb-qh2',
  HostId: '91267cb-unimelb-qh2-unimelb',
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

Expected behavior

Either it should be backwards compatible with responses from older API versions or fail early stating
that the API version is not compatible.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

From the operators of the fake S3 service:

I can verify that v3 upload class errored out. I've tested the upload on a
100GB file with 1910 parts and it looks like the uploader works fine until the end,
where it probably aggregates the parts metadata and raises a MalformedXML.

[alexforsyth]

I have a feeling this is may be an issue with the custom S3 service implementation, but its curious that v2 works and v3 doesn't. Looks like the current version of s3 is 2006-03-01.

In v3 we use service generated models to code generate the s3 client and upload wraps multipart upload from that generated s3 client. This is because the raw multipart upload is a pain to use. I've tested this with an extremely large file and it seems to work just fine for me against amazon's s3. Unfortunately, I'm having trouble reproducing the error against your custom endpoint. Can you let me know what the error looks like if we run an old school multipart upload?

something like:

import {
   CompleteMultipartUploadCommand,
   CreateMultipartUploadCommand,
   S3Client,
   UploadPartCommand,
 } from "@aws-sdk/client-s3";


(async () => {
   const client = new S3Client({
       ...
   });

   const createMultipartUploadResult = await client.send(new CreateMultipartUploadCommand({
       ...
   }));

   const PartNumber = 1;
   const partResult = await client.send(
       new UploadPartCommand({
           UploadId: createMultipartUploadResult.UploadId,
           Body: "this is the first and only part",
           PartNumber,
       })
   );

   const uploadedParts = [];
   uploadedParts.push({
       PartNumber,
       ETag: partResult.ETag,
   });

   const completeMultipartUpload = await client.send(
       new CompleteMultipartUploadCommand({
           UploadId: createMultipartUploadResult.UploadId,
           MultipartUpload: {
           Parts: uploadedParts,
           },
       })
   );
})();

This code may need minor tinkering.

[marcolarosa]

hi @alexforsyth

The upload failed the same way doing it the hard way. The response from the server that is triggering this error is:

HttpResponse {
  statusCode: 400,
  headers: {
    'content-length': '238',
    'x-amz-request-id': 'tx00000000000000040996f-0060528571-82cc3b0-unimelb-qh2',
    'accept-ranges': 'bytes',
    'content-type': 'application/xml',
    date: 'Wed, 17 Mar 2021 22:40:49 GMT',
    'set-cookie': 'SERVERID=rgw1-ceph3-qh2-new.lb.storage.unimelb.edu.au; path=/'
  },
  body: Uint8Array(238) [
     60,  63, 120, 109, 108,  32, 118, 101, 114, 115, 105, 111,
    110,  61,  34,  49,  46,  48,  34,  32, 101, 110,  99, 111,
    100, 105, 110, 103,  61,  34,  85,  84,  70,  45,  56,  34,
     63,  62,  60,  69, 114, 114, 111, 114,  62,  60,  67, 111,
    100, 101,  62,  77,  97, 108, 102, 111, 114, 109, 101, 100,
     88,  77,  76,  60,  47,  67, 111, 100, 101,  62,  60,  66,
    117,  99, 107, 101, 116,  78,  97, 109, 101,  62,  49,  49,
     57,  48,  95, 112, 100, 115,  99,  95,  50,  48,  50,  48,
     60,  47,  66, 117,
    ... 138 more items
  ]
}

The response gets thrown in inside Aws_restXml.js at line 4097 because the status code is 400.

However, my colleagues who run the service pointed me at #1814.

If I manually patch Aws_restXml.js with the following:

 diff Aws_restXml-original.js Aws_restXml.js
8828c8828
<     const bodyNode = new xml_builder_1.XmlNode("CompletedMultipartUpload");
---
>     const bodyNode = new xml_builder_1.XmlNode("CompleteMultipartUpload");

Note the missing d from the xml name?

It works:

HttpResponse {
  statusCode: 200,
  headers: {
    'x-amz-request-id': 'tx00000000000000004a24a-006052886d-91267cb-unimelb-qh2',
    'content-type': 'application/xml',
    'content-length': '313',
    date: 'Wed, 17 Mar 2021 22:53:33 GMT',
    'set-cookie': 'SERVERID=rgw2-ceph3-qh2-new.lb.storage.unimelb.edu.au; path=/'
  },
  body: Uint8Array(313) [
     60,  63, 120, 109, 108,  32, 118, 101, 114, 115, 105, 111,
    110,  61,  34,  49,  46,  48,  34,  32, 101, 110,  99, 111,
    100, 105, 110, 103,  61,  34,  85,  84,  70,  45,  56,  34,
     63,  62,  60,  67, 111, 109, 112, 108, 101, 116, 101,  77,
    117, 108, 116, 105, 112,  97, 114, 116,  85, 112, 108, 111,
     97, 100,  82, 101, 115, 117, 108, 116,  32, 120, 109, 108,
    110, 115,  61,  34, 104, 116, 116, 112,  58,  47,  47, 115,
     51,  46,  97, 109,  97, 122, 111, 110,  97, 119, 115,  46,
     99, 111, 109,  47,
    ... 213 more items
  ]
}

[marcolarosa]

hi devs! any word on this ? @alexforsyth

[github-actions]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.