Intermittent InvalidSignatureException/SignatureDoesNotMatch errors using current SDK client versions

[clocked0ne]

Describe the bug

Our org is still using aws-sdk v2 client in the majority of our serverless environments (deployed using Serverless Framework v3), we have identified that some version of the SDK client after version 2.1680.0 has introduced intermittent InvalidSignatureException errors, though we are not sure specifically which version as this is the last known version deployed that doesn't appear to introduce the issue.

This has been confirmed by updating a service to use a newer version of the package, deploying a beta release and observing errors intermittently logged, then deploying a followup beta release with only the aws-sdk rolled back to 2.1680.0; the error then stops appearing.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Requests are processed successfully by Amazon through aws-sdk without 4xx errors.

Current Behavior

Function calls respond with e.g:

{
  "code":"InvalidSignatureException",
  "message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.",
  "[__type]":"See error.__type for details.",
  "time":"2024-12-13T12:29:46.154Z",
  "requestId":"M73GQ0H1N9...",
  "statusCode":400,
  "retryable":false,
  "retryDelay":83.80437991528376,
  "stack":["InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.","    at Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/json.js:80:27)","    at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:106:20)","    at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:78:10)","    at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:686:14)","    at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)","    at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)","    at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10","    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)","    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:688:12)","    at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"]}

{
  "code":"SignatureDoesNotMatch",
  "message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.",
  "time":"2024-12-11T13:53:55.951Z",
  "requestId":"006d4b88...",
  "statusCode":403,
  "retryable":false,
  "retryDelay":118.38055748397065,
  "stack":["SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.","    at Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/query.js:50:29)","    at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:106:20)","    at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:78:10)","    at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:686:14)","    at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)","    at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)","    at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10","    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)","    at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:688:12)","    at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"]}

Reproduction Steps

This issue is not consistently reproducible. This applies to all services and not just requests to DynamoDB for example, but a typical request looks something like the below. I have included the AWS.config.update we apply globally in case it is relevant:

const AWS = require('aws-sdk');
const https = require('https');
const sslAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 50,
  rejectUnauthorized: true
});

sslAgent.setMaxListeners(0);

AWS.config.update({
  httpOptions: {
    agent: sslAgent
  }
});

const dynamo = new AWS.DynamoDB.DocumentClient({ region: 'eu-west-2' });

const params = {
    TableName: process.env.TABLE,
    Key: { pk, sk },
    ConsistentRead: true
  };

const response = await dynamo.get(params).promise();

Possible Solution

I suspect that a vulnerabilty in one of the package's dependencies has been patched or other updates made which have not been picked up by testing because the issue is relatively intermittent.

Additional Information/Context

Please note that we have seen this occurring in all our AWS accounts, from development to production, across all deployment regions (we deploy to four main regions: eu-west-2, us-east-1, ca-central-1, ap-southeast-2). Our Lambdas are all configured to use the NodeJS v20 runtime. We include the aws-sdk in deployment packages as recommended (and since the latest runtimes do not bundle v2).

We also raised an enterprise support ticket with Amazon Support who after much back and forth suggested that we raise the ticket here on the repo for the team to investigate, including confirming that we use execution roles with permissions and do not provide credentials in the requests, confirming the server time has no clock skew (which was odd to request given this is within Lambda context), etc. All the typical reasons for potential signature mismatch were covered:

Kindly understand that code/SDK support is out of scope of AWS Support and after reviewing the information provided, I can confirm that the latest version of AWS SDK v2 is 2.1692.0 [1] and the issue does not occur at 2.1680.0 which verifies the issue around SDK which needs to be reviewed by the SDK team. I reviewed the changelogs [2] on the SDK versions and was not able to pinpoint the exact version where the issue originates.

Given the complexity of this issue and its potential impact, I recommend one of the following course of actions:

1. Open an issue on the AWS SDK GitHub repository [3]. This will directly reach to the SDK internal team, who can investigate the problem in depth.

2. If you prefer, you can provide the relevant code snippets in this case. I can then escalate it to the SDK team internally for analysis and to identify any version-specific changes.

I have notified your Technical Account Manager in this regards as well and can help prioritize this issue with SDK internal team.

We have chosen option 1.

SDK version used

2.1692.0

Environment details (OS name and version, etc.)

AWS Lambda. E.g: Runtime Version nodejs:20.v51 Runtime Version ARN: arn:aws:lambda:eu-west-2::runtime:cb6527bfb6726a080a367eca00e49765ca5abd8cd1a17783fbee683313121ece

[kuhe]

We haven't changed anything in the AWS SDK for JavaScript v2 between v2.1680.0 and the latest version with regards to signing.

In the history of the lib folder, containing the core logic, https://github.com/aws/aws-sdk-js/commits/master/lib, there are only changes relating to endpoint resolution since 2.1680.0.

[clocked0ne]

It may not be a change directly related to signing, it may be a dependency or something like that which has changed and introduced the issue. We have now downgraded only the aws-sdk version in multiple services to our known good version - 2.1680.0 - and they have all stopped reporting these errors, of which we were seeing thousands across our platform's requests.

So if it is not the client itself then it can only be something within the Lambda runtimes or environment.

[aBurmeseDev]

Hi @clocked0ne - thanks for reaching out and appreciate the follow up.

We've had a preliminary discussion within our team regarding the issue you are facing, and we have confirmed that there haven't been any recent changes related to the behavior you're observing. As you may be aware, AWS SDK JavaScript v2 entered maintenance mode on September 8, 2024 and since then, there haven't been any new changes introduced.

Now, the errors you're seeing typically indicate an issue with the SDK being unable to properly sign the requests being made to AWS services. This can be caused by a variety of factors, including outdated SDK versions, misconfigurations, or issues with the underlying credentials or permissions.

I have also attempted to reproduce the error with minimal code across versions and haven't been able to.

Since you had mentioned that the issue is intermittent, it's also worth noting that the InvalidSignatureException and SignatureDoesNotMatch errors can also occur due to intermittent network issues or throttling by services.

Could you investigate any potential differences in the environment setup, such as proxy configurations, network settings, or any other factors that could affect the SDK's ability to communicate with services?

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.