Grafana - Athena datasource access key rotation fails

[MaxZorko]

What happened:
Our workflow was next:

1. We've created an IAM user + access key
2. Used this access key in Athena datasource. The datasource and Grafana dashboard started to work as expected
3. We've tried to update an access key: deleted an old one, generated a new one and updated with this new access key Athena datasource
4. After that immediately we got an error: "error querying the database: error executing query: UnrecognizedClientException: The security token included in the request is invalid. status code: 400, request id: 72b289e4-1eb5-4e57-a502-59930882c1fe"

What you expected to happen:
Grafana should continue to work without any issues after a new access key was applied.

How to reproduce it (as minimally and precisely as possible):

Screenshots

Anything else we need to know?:

Environment:

• Grafana version: 9.3.1
• Plugin version: latest
• OS Grafana is installed on: on premise
• User OS & Browser: Mac OS, Chrome
• Others:

[MaxZorko]

I've found something similar here: grafana/grafana#52420

[MaxZorko]

[idastambuk]

Hi @MaxZorko I think this might have been fixed along with this ticket: #246
Can you update your datasource and check if that did it? Thanks!

[MaxZorko]

idastambuk

Hi, @idastambuk. Thanks for your update.
I can see this fix was implemented in your code with the 2.10.2 plugin version (Fix connection error when changing access and secret key #262).

I'm using the latest 2.11.1 version and still see the same issue.
I've tried once again the workflow I described at the very beginning. The result is the same - error querying the database: error executing query: UnrecognizedClientException: The security token included in the request is invalid. status code: 400, request id: 6e3bfb33-1d55-4ebb-8de7-8b775512f4de

What else to try?

[MatinF]

We see the same issue - in practice we’ve found it necessary to delete and recreate the data source entirely, as if the info is cached - this would be nice to get fixed

[iwysiu]

Hi @MaxZorko @MatinF ! Sorry to hear you're still having issues. I investigated this and I was able to reproduce it. I'll add it to our next column to be picked up when we can.

For whoever picks this up:
From what I can tell, while the other PR fixed how the api's are cached (so the resource handlers, which get the API every time, get the new credentials successfully), it seems like the queries never fetch the api/session after the first query is called for the Athena datasource. This supported by the fact that the new credentials work if the grafana server is restarted.
Also a note for reproing: it takes AWS a little time to kill an access key so wait for it to fail before saving a new one.

[njvrzm]

Edit: that fix worked but was not the most direct approach. This is now being addressed with #314. This will be addressed by grafana/sqlds#111 and grafana/grafana-aws-sdk#128. Once those are released we'll bump the versions here.