Bug: Mimir 2.15.0 unable to access block storage

[eric-engberg]

What is the bug?

I was upgrading from chart version 5.5.0-weekly.304 to 5.6.0 and after upgrading mimir was unable to access the configured s3 buckets. Downgrading fixed the problem so it shouldn't be a permissions issue. The error message doesn't really give any details, just "Access Denied".

All services that use block storage are affected. None of them work. When comparing the manifest changes that would be deployed there's nothing that would cause this. No changes to the config and no changes to the service account which uses IRSA

How to reproduce it?

Deploy chart 5.5.0-weekly.304 using s3. Upgrade to chart 5.6.0

What did you think would happen?

Mimir would work and be able to access block storage

What was your environment?

Kubernetes 1.30, deployed using kustomize and helm via ArgoCD. Mimir version 2.15.0 chart 5.6.0

Any additional context to share?

No response

[narqo]

upgrading from chart version 5.5.0-weekly.304 to 5.6.0

There have been lots of internal changes between these two, but I don't recall anything that was intentional on the mimir's side.

Could you provide more details on how you configure the s3 access — do you maybe have an IAM policy with a strict list of API methods in the allow-list? Under the hood, Mimir uses objstore: I do recall there was some back and worth there about using one s3 API over the other. Maybe there is something to explore there in details.