AWS IAM auth method doesn't retry sts:GetCallerIdentity request if it fails

[daveadams]

We are seeing failures like this happen on a small percentage of AWS IAM login attempts:

Code: 400. Errors:
  * error making upstream request: error making request:
    Post https://sts.amazonaws.com//:
    net/http: TLS handshake timeout

An immediate retry is successful, and this is an intermittent issue. Obviously the problem is between Vault (which is running in an EC2 instance) and sts.amazonaws.com. I know that behind the scenes, Vault takes a signed sts:GetCallerIdentity request and submits it to STS on behalf of the user in order to validate that the user has the necessary IAM credentials. So generally we would have expected SDK retries to cover any temporal failures with contacting the AWS APIs.

However, looking at the code (https://github.com/hashicorp/vault/blob/master/builtin/credential/aws/path_login.go#L1558), it appears that the SDK is not being used to make that final request. And I don't see any logic in path_login.go nor in the go-cleanhttp library to implement any retries.

Obviously we can implement retry logic on our end, but this seems like something Vault should be handling since it's talking directly to the AWS APIs. Thanks!

• Vault Server Version: 1.3.4
• Vault CLI Version: 1.3.4
• Server Operating System/Architecture: Linux - Ubuntu 18.04

[tyrannosaurus-becks]

Ah yes, thanks for bringing this to our attention, it does seem like it would be a fairly simple fix to put that call inside a retry loop.