Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(operator): Configure Loki to use virtual-host-style URLs for S3 AWS endpoints #12469

Merged
merged 16 commits into from
Apr 10, 2024
Merged

fix(operator): Configure Loki to use virtual-host-style URLs for S3 AWS endpoints #12469

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
10032e2
Configure Loki to use virtual-host-style URLs for S3 AWS endpoints
btaani Apr 4, 2024
9113d18
remove comment
btaani Apr 4, 2024
5eb49f2
modify CHANGELOG.md
btaani Apr 4, 2024
6842735
Merge branch 'main' into virtual-host-style-s3
btaani Apr 4, 2024
7461436
remove setting s3forcepathstyle to false
btaani Apr 4, 2024
5b93697
Merge branch 'main' into virtual-host-style-s3
btaani Apr 4, 2024
d3a63b6
Merge branch 'main' into virtual-host-style-s3
btaani Apr 5, 2024
d584cb1
Merge branch 'main' into virtual-host-style-s3
btaani Apr 8, 2024
d22a6a6
Merge branch 'main' into virtual-host-style-s3
btaani Apr 8, 2024
4b277cb
move from go-template
btaani Apr 8, 2024
5e5493d
move from go-template to config options
btaani Apr 9, 2024
293b649
Merge branch 'main' into virtual-host-style-s3
btaani Apr 9, 2024
4c5b0ac
add unit test
btaani Apr 9, 2024
81c4663
Merge branch 'main' into virtual-host-style-s3
btaani Apr 9, 2024
f621cd1
move from config options to secret extraction
btaani Apr 9, 2024
01924e7
Merge branch 'main' into virtual-host-style-s3
btaani Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions operator/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
## Main

- [12469](https://github.com/grafana/loki/pull/12469) **btaani**: Configure Loki to use virtual-host-style URLs for S3 AWS endpoints
- [12370](https://github.com/grafana/loki/pull/12370) **periklis**: Update Loki operand to v2.9.6
- [12333](https://github.com/grafana/loki/pull/12333) **periklis**: Bump max OpenShift version to next release

Expand Down
14 changes: 13 additions & 1 deletion operator/internal/manifests/internal/config/build.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"bytes"
"embed"
"io"
"strings"
"text/template"

"github.com/ViaQ/logerr/v2/kverrors"
Expand All @@ -25,7 +26,9 @@ var (
//go:embed loki-runtime-config.yaml
lokiRuntimeConfigYAMLTmplFile embed.FS

lokiConfigYAMLTmpl = template.Must(template.ParseFS(lokiConfigYAMLTmplFile, "loki-config.yaml"))
lokiConfigYAMLTmpl = template.Must(template.New("loki-config.yaml").Funcs(template.FuncMap{
"isEndpointAWS": isEndpointAWS,
}).ParseFS(lokiConfigYAMLTmplFile, "loki-config.yaml"))

lokiRuntimeConfigYAMLTmpl = template.Must(template.New("loki-runtime-config.yaml").ParseFS(lokiRuntimeConfigYAMLTmplFile, "loki-runtime-config.yaml"))
)
Expand Down Expand Up @@ -54,3 +57,12 @@ func Build(opts Options) ([]byte, []byte, error) {
}
return cfg, rcfg, nil
}

func isEndpointAWS(endpoint string) bool {
awsEndpointSuffix := ".amazonaws.com"
if strings.HasSuffix(endpoint, awsEndpointSuffix) {
return true
} else {
return false
}
}
35 changes: 17 additions & 18 deletions operator/internal/manifests/internal/config/build_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -282,7 +282,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -707,7 +707,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -1065,7 +1065,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -1424,7 +1424,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -1813,7 +1813,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -2151,7 +2151,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -2570,12 +2570,11 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://s3.us-east.amazonaws.com
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
secret_access_key: ${AWS_ACCESS_KEY_SECRET}
s3forcepathstyle: true
compactor_grpc_address: loki-compactor-grpc-lokistack-dev.default.svc.cluster.local:9095
ring:
kvstore:
Expand Down Expand Up @@ -2879,7 +2878,7 @@ overrides:
ObjectStorage: storage.Options{
SharedStore: lokiv1.ObjectStorageSecretS3,
S3: &storage.S3StorageConfig{
Endpoint: "http://test.default.svc.cluster.local.:9000",
Endpoint: "http://s3.us-east.amazonaws.com",
Region: "us-east",
Buckets: "loki",
},
Expand Down Expand Up @@ -2916,7 +2915,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -3412,7 +3411,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -3672,7 +3671,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -3934,7 +3933,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -4194,7 +4193,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -4496,7 +4495,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -4795,7 +4794,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down Expand Up @@ -5284,7 +5283,7 @@ chunk_store_config:
common:
storage:
s3:
s3: http://test.default.svc.cluster.local.:9000
endpoint: http://test.default.svc.cluster.local.:9000
bucketnames: loki
region: us-east
access_key_id: ${AWS_ACCESS_KEY_ID}
Expand Down
4 changes: 3 additions & 1 deletion operator/internal/manifests/internal/config/loki-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,15 @@ common:
region: {{.Region}}
s3forcepathstyle: false
{{- else }}
s3: {{ .Endpoint }}
endpoint: {{ .Endpoint }}
bucketnames: {{ .Buckets }}
region: {{ .Region }}
access_key_id: ${AWS_ACCESS_KEY_ID}
secret_access_key: ${AWS_ACCESS_KEY_SECRET}
{{- if not (isEndpointAWS .Endpoint) }}
s3forcepathstyle: true
{{- end }}
{{- end }}
{{- with .SSE }}
{{- if .Type }}
sse:
Expand Down
Loading