[Loki+Helm] S3 backend error: incorrect region, the bucket is not in 'eu-west-1' region at endpoint ''

[sklarsa]

Describe the bug
I'm running Loki in EKS using IRSA to use the s3 storage backend, and I'm getting the following errors in the loki-read pods:

│ level=error ts=2022-10-12T21:06:28.774105533Z caller=cached_client.go:65 msg="failed to build cache" err="BucketRegionError: incorrect region, the bucke │
│ t is not in 'eu-west-1' region at endpoint ''\n\tstatus code: 301, request id: , host id: "                                                              │
│ level=error ts=2022-10-12T21:06:28.774123664Z caller=compactor.go:432 msg="failed to run compaction" err="BucketRegionError: incorrect region, the bucke │
│ t is not in 'eu-west-1' region at endpoint ''\n\tstatus code: 301, request id: , host id: "                                                              │
│ level=error ts=2022-10-12T21:06:44.83911778Z caller=reporter.go:203 msg="failed to delete corrupted cluster seed file, deleting it" err="BucketRegionErr │
│ or: incorrect region, the bucket is not in 'eu-west-1' region at endpoint ''\n\tstatus code: 301, request id: , host id: "                               │
│ level=error ts=2022-10-12T21:07:21.961894672Z caller=ruler.go:497 msg="unable to list rules" err="BucketRegionError: incorrect region, the bucket is not │
│  in 'eu-west-1' region at endpoint ''\n\tstatus code: 301, request id: , host id: "

My helm chart values.yaml config is simple:

        serviceAccount:
          name: loki-test
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::012345678901:role/my-role
        loki:
          storage:
            type: s3
            s3:
              s3: s3://eu-west-1/my-bucket

I can also confirm that the service account is configured correctly in a separate pod running the aws cli. I've also tried multiple permutations of the s3: storage config, but I've only gotten different errors around assuming the role, and figured that this was the closest that I got to a working solution.

I've also confirmed the bucket region:

$ aws s3api get-bucket-location --bucket my-bucket
{
    "LocationConstraint": "eu-west-1"
}

To Reproduce
Using the following helm chart & version:

chart: loki
repoURL: https://grafana.github.io/helm-charts
targetRevision: 3.2.1

Expected behavior
Able to connect to s3 bucket

Environment:

• Infrastructure: EKS
• Deployment tool: helm & argocd

[sklarsa]

When I add more granular s3 configs:

                  s3: s3://my-bucket
                  bucketnames: my-bucket
                  endpoint: s3.eu-west-1.amazonaws.com
                  insecure: false
                  region: eu-west-1
                  s3forcepathstyle: true

I get an UnmarshallError:

l=error ts=2022-10-12T22:16:00.023675344Z caller=ruler.go:497 msg="unable to list rules" err="WebIdentityErr: failed to retrieve credentials\ncaused │
│  by: SerializationError: failed to unmarshal error message\n\tstatus code: 405, request id: \ncaused by: UnmarshalError:

and the return value is an xml-formatted HTTP 405 Method Not Allowed (POST)

[jeschkies]

@sklarsa I think you need to define the region explicitly like you've done in your second post. As for the HTTP 405. It seems your permissions for the S3 bucket are not defined properly. Did you allow these?

[sklarsa]

@jeschkies Yes, I allowed s3:* for testing, and can confirm that I can write to the bucket using the same service account that I am using for loki. Is there a way to debug the exact HTTP Request that loki is making? Perhaps I'm pointed to the wrong bucket somehow?

[sklarsa]

Digging into this a little further, it looks like loki expects static credentials to be passed into the config file.

The aws-sdk-go V1 does support additional credentials handlers (ec2roles, sts, etc), but they don't seem to be implemented at this time, unless I'm misunderstanding something. Ideally, I'd like to run loki using AWS IRSA roles, which doesn't require the storage of static AWS credentials to pass into the config.

I think this is still a problem based on this issue, which should be reopened.

I'll take a look at this and see if I can come up with some sort of workable solution

[trallnag]

@sklarsa, you sure this is not already working? Because I am using Loki in Kubernetes with IRSA and Loki is able to access my S3 bucket. Here are the values I provide to my release of the grafana/loki chart:

serviceAccount:
  create: false
  name: ${kubernetes_service_account.this[0].metadata[0].name}

monitoring:
  dashboards:
    enabled: false
  rules:
    enabled: false
  alerts:
    enabled: false
  serviceMonitor:
    enabled: false
  selfMonitoring:
    enabled: false
    lokiCanary:
      enabled: false

loki:
  auth_enabled: false

  storage:
    type: s3
    bucketNames:
      chunks: ${aws_s3_bucket.this[0].id}
      ruler: ${aws_s3_bucket.this[0].id}
      admin: ${aws_s3_bucket.this[0].id}
    s3:
      region: ${local.region}

  storage_config:
    boltdb_shipper:
      cache_ttl: 48h

  compactor:
    retention_enabled: true

  limits_config:
    retention_period: ${var.retention_period}

[sklarsa]

@trallnag thank you, I'll give this config a try! Which versions of loki and the helm chart are you using?

[trallnag]

@sklarsa, I am using chart version 3.2.1 with the default app version.

[sklarsa]

Thanks! I can confirm that the config you sent works. I'll close my PR, since I'm guessing that the golang AWS cli uses the environment's default credential chain if no static credentials are provided 🤦