Getting FailedToFetch error for DynamoDBClient QueryCommand

[renchix]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Fetching data from DynamoDB through QueryCommand ocasionally results in FailedToFetch error. Error persists temporarily and comes and goes periodically. First cases were observed in Oct 31, 2024 and Nov 1, 2024. And again today.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Browser

Details of the browser/Node.js/ReactNative version

Microsoft Edge

Reproduction Steps

        const command = new QueryCommand(input);
        response = await client.send(command);

Observed Behavior

Code has been working for at least a year, no changes have been made recently.
First idea was that it some how related to CORS, but I am not sure if this can somehow be configured for DynamoDBClient.

Expected Behavior

DyanmoDb results should be fetched without raising error.

Possible Solution

No response

Additional Information/Context

No response

[renchix]

It is CORS issue, but the question is how comes it hasn't been before and how can it be resolved:

Access to fetch at 'https://dynamodb.eu-north-1.amazonaws.com/' from origin 'https://XXXX.XXXX.XXX has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

[zshzbh]

Hey @renchix ,

There's some potential root causes:

1. Network condition : Intermittent network issues or latency spikes could cause requests to timeout or fail, which might trigger CORS errors in some browsers.
2. AWS Service Availability: While rare, there could be intermittent issues with AWS services in your region if the DDB service or its associated auth services have brief outages or performance issues.
3. Browser Caching: Browsers can cache CORS preflight responses. If a successful request is cached, subsequent requests might work, while new requests or requests after the cache expires might fail. ref: https://repost.aws/questions/QUt1qUofacSDKawoShV1fgKA/api-gateway-method-cors-error-cloudformation. And there's a stack overflow topic regarding this issue.
   To Implement Retry Logic: Add a retry mechanism for failed requests. This can help mitigate intermittent issues.
4. DNS Resolution
5. Load Balancing: If AWS is using multiple servers or load balancers for the DynamoDB service, some might be configured correctly for CORS while others are not.

To identify the patterns in the failures, you can implement detailed logging to include the timestamps and full request details and response headers.

To help potentially resolve this issue, you can apply retry logic:

const MAX_RETRIES = 3;
const RETRY_DELAY = 1000; 

This can help mitigate intermittent issues.

Thanks!
Maggie

[renchix]

Thanks for the detailed response.

To my understanding default in-built retry mechanism () already makes 3 attempts after failed request, so it seems that this doesn't solve the issue as it should already be working.

Caching errors seems to be a possible cause, however, as we have seen cases where even after trying to fully refresh page for several times error still persists, then this also doesn't feel as the likely cause.

I could hope that this ir really some temporary issue - the fact that 99.9% of time everything works correctly also kind of supports this logic. However, I am still puzzled about the fact that in the last week there have been multiple such cases, without anything similar before.

Also, in those cases where DynamoDBClient keeps failing Lambda Client works without issues (I understand that there the logic is a bit different - but still it goes under aws-sdk which then makes causes #1 and #4 as less likely) - this also make me feel that this is some issue specific to DynamoDBClient.

[zshzbh]

Thanks for the updates! I agree, and also it's hard to troubleshoot to find the root cause as it works well in 99.9% of time. I have asked Dynamo DB team and see if there's any known issue happened last week in your region.

[renchix]

Still getting the same error every day or two:

monitor:1 Access to fetch at 'https://dynamodb.eu-north-1.amazonaws.com/' from origin 'https://www.xxxxxxx.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
dynamodb.eu-north-1.amazonaws.com/:1

    Failed to load resource: net::ERR_FAILED

main.535af7c7.js:2

    Uncaught (in promise) TypeError: Failed to fetch
at Vv.handle (main.535af7c7.js:2:1408941)
at async main.535af7c7.js:2:1317124
at async clientCommand.middlewareStack.addRelativeTo.name (main.535af7c7.js:2:1427841)
at async main.535af7c7.js:2:1318306
at async main.535af7c7.js:2:1362663
at async main.535af7c7.js:2:1314299
at async R (main.535af7c7.js:2:3258575)

Is there some way to force those headers for DynamoDBClient request in our React code?

[renchix]

To me it seems that only solution then is to do all requests through API gateway (not directly to DynamoDB as it is now)
That obviously would mean that there is one extra intermediary step involved as well as additional costs for API gateway use, and a lot of code rewriting. This doesn't sound good.

[renchix]

To add maybe a little bit more context for DynamoDB team - even if QueryItems fails and then problem persists for some time - at the same time GetItem requests are fulfilled.

[renchix]

After some additional investigation I found that preflight request is made and received successfully, however the real request in those failed cases is missing 'Remote address' in the Headers.
This is how failed one looks:

and this is successful request:

Preflight requests for both of them seem to be identical.

@zshzbh any suggestions?

[zshzbh]

Hey @renchix ,

I can't reproduce this error. Could you please provide any pre-condition and minimal code reproduction so we can better troubleshoot this issue?

[renchix]

There is not much of code to provide.
This basically is all that leads to the error:

const client = new DynamoDBClient({region: region, credentials: credentials, maxRetries: 5 });
const command = new QueryCommand(input);
const response = await DynamoDBDocumentClient.from(client).send(command)

As error usually comes by surprise, I cannot reliably reproduce it as well.
When it has happened I have inspected the http requests and missing Remote address field in the Headers, mentioned in my previous post was only thing that seemed out of normal in all of these cases.

[zshzbh]

Hey @renchix,

We are not able to troubleshoot the exact root cause with the existing information. I'd like to suggest -

1. Try to use other browsers, for example Firefox , and see if this also happens on Firefox, or if it's a specific issue on the browser you use.
2. monitor if there’s anything changes when you get the error.
3. try to make the request with lambda.
4. use error handling methods

Use middlewareStack to check the request header and adjust the request - ref.This would be helpful.

[felixschwamm]

I'm also trying to resolve this issue for about 3 weeks now. It also started around Oct 31, and it happens very rarely, but once it happens all requests to DynamoDB fail, and it takes some time (usually a few minutes) until the requests can succeed again.

This image shows the response headers of the failed request on the left side and the response headers of the successful request on the right side. As you can see, the failed request on the left does not include the Access-Control headers in the response and this is also what causes the browser to throw a CORS error.

Another thing: the failed requests always have the following capitalization for the request id header "x-amzn-requestid" and the successful ones "x-amzn-RequestId". I know that the HTTP/1.1 specification states that capitalization should not be relevant for request handling, but this difference suggests that even if both requests have a 200 status (even the failed ones always have a 200 status) and the response is for the most part identical the code that generates this response must be somehow a different one because it sends a slightly different request id header back.

Can you also observe this behavior @renchix ?

Hint: Some browsers have dev tools that mess with the capitalization of the headers to make it more consistent when they display it. In that case you would have to download the HAR file and inspect that, or in chrome there is a checkbox with something like "raw data" that you need to check.

[felixschwamm]

The AWS Console is also affected

[MainzerKaiser]

I can confirm that we struggle with the very same problems for a couple weeks now. Sporadic
"TypeError: Failed to fetch"
and

"Access to fetch at 'https://dynamodb.eu-central-1.amazonaws.com/' from origin 'https://xxxx.xxxx.xxx/ has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled."

Status code is 200 and AWS console confirms CORS problem.

If one user sees the problems, another user has it at the same time.

After a couple minutes everything works smoothly.
Anything we can do to improve the situation?

How is this exactly done, looking from the developer side:

If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled

The only thing that I adapted reading this thread was setting the maxAttempts higher:

import { DynamoDB } from "@aws-sdk/client-dynamodb";

const fetchDataDynamoDB_getItem= async(
    dynamoName: string,
    uuid: string, 
    mycred:Mycred,
    ): Promise<GetItemResult> => {

    const dynamoClient = new DynamoDB({ region: 'eu-central-1', credentials: mycred, maxAttempts: 5 }); 

My dependencies are:
"dependencies": {
"@aws-sdk/client-athena": "^3.379.1",
"@aws-sdk/client-dynamodb": "^3.363.0",
"@aws-sdk/client-s3": "^3.363.0",
"@aws-sdk/lib-dynamodb": "^3.370.0",

[ndbs-tsb]

The AWS Console is also affected

Just encountered this in the browser, but a hard reload solved it for me.

[zshzbh]

Hey guys,

I'm sorry for this experience. But I can't reproduce this issue since it's happening intermittent. I have two questions here -

• Are you all seeing this issue in EU regions? (This might be a region specific issue.)
• What browser you are using while seeing this issue?

It would help a lot if we can narrow down the scope.

Thanks!

[felixschwamm]

I don't think that it's browser specific we observed this with Edge, Chrome, Safari and Firefox. Missing CORS headers will always lead to an error in every common browser with default settings (at least the POST requests that the DynamoDbClient sends).

Our DynamoDB table is in eu-central-1 (the one from @renchix was also in a EU region so it could actually be region specific).

[ndbs-tsb]

My DynamoDB table is also in eu-central-1.

[renchix]

Hi!
First of all I must say that if at the beginning we saw the issue very very rarely then as the weeks went on the issue seems to be happening more often as we received more complaints from our users.

Adding larger maxAttempts parameter didn't seem to help.
As both preflight and fetch requests are returned with status code 200, I am pretty sure it is not really seen as unsuccessful attempt.

@felixschwamm - I checked and indeed the failed fetch request has all lowercase letters for x-amzn-requestId, as well as missing 'Remote address' as I wrote already before.
@zshzbh - and yes we are in eu region only

What we did at the end - we wrote a bunch of API gateway endpoints for the most crucial DynamoDBClient requests and added requests to those APIs as a fallback scenario when DynamoDBClient request returns CORS error.
From the user side it adds a bit more latency, it took a while to set up all the necessary endpoints and add them to the code, and of course additional costs for using API Gateway. But as the problem started to happen more often this was only solution that kept the problem out of sight for the users.

[zshzbh]

Hey,

It seems all issues are in EU regions, it seems that this issue is region specific. I will report this issue to service team

[zshzbh]

Internal ref - V1600427559. I will update this ticket once I got back from service team.

[marcortw]

@zshzbh still getting this on the console in eu-central-1. Using chrome 131.0.6778.109 (Official Build) (64-bit) (cohort: Stable). if it helps, here's a a screenshot of the console output:

[MainzerKaiser]

Also had this Friday 6pm CET. Does it make sense to post exact times for your RCA?

[vishwa98]

This issue occurs intermittently on the AWS Console. My DynamoDB table is in eu-north-1.

[renchix]

@zshzbh - Any updates?

[MainzerKaiser]

I would be also interested to get updates. Had a lot of problems today with that CORS error. CRTL+F5 didn't help, just waiting and retrying, which cannot be the solution.

[zshzbh]

Hey there, I've reached out to the service team but haven't got back from them. I will contact them again and let you know if there's any updates. Thanks!

[zshzbh]

Hey all,

I've got back from service team. They are taking care of this issue now. The team is working on fixing this issue. The fix is expected to resolve the issue for all affected customers.

Thanks!

[zshzbh]

Hey all,

The fix has been released.

Between October 8, 2024 and December 19, 2024, some customers may have experienced intermittent HTTP status code 400 errors from DynamoDB for cross-origin resource sharing requests in the Frankfurt (EU-CENTRAL-1) and Stockholm (EU-NORTH-1) Regions. This issue was mitigated on December 19, 2024. The service is operating normally.

We appreciate everyone's contributions and insights. The information you've provided is crucial for identifying and addressing this issue effectively.

Thanks~!
Maggie

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.