[Tempo-distributed] - Unclear override configuation for tempo-distributed

[Nierhoff]

Describe the bug
I am trying to increase the rate limits in tempo, deployed using helm and the chart tempo-distributed and has run into issues understanding the documentation.

To Reproduce
Steps to reproduce the behavior:
Deployed using command:
helm install tempo-distributed grafana/tempo-distributed --namespace tempo --create-namespace --values distributed.yaml

the value file "distributed.yaml" content:

storage:
  trace:
    backend: s3
    s3:
      bucket: "tempo-traces"
      endpoint: minio.minio.svc.cluster.local:80
      access_key: tempo
      secret_key: REPLACE
      forcepathstyle: true
      insecure: true

global_overrides:
  per_tenant_override_config: /runtime-config/overrides.yaml
overrides: |-
  overrides:
    defaults:
      ingestion:
        burst_size_bytes: 20000000 # 20MB
        rate_limit_bytes: 20000000 # 20MB
        max_traces_per_user: 10000
        ingestion_burst_size_bytes: 20000000 # 20MB
        ingestion_rate_limit_bytes: 20000000 # 20MB
      global:
        max_bytes_per_trace: 20000000 # 20MB
    "*":
      ingestion:
        burst_size_bytes: 20000000 # 20MB
        rate_limit_bytes: 20000000 # 20MB
        max_traces_per_user: 10000
        ingestion_burst_size_bytes: 20000000 # 20MB
        ingestion_rate_limit_bytes: 20000000 # 20MB
      global:
        max_bytes_per_trace: 20000000 # 20MB

ingester:
   replicas: 1

traces:
  otlp:
    http:
      enabled: true
    grpc:
      enabled: true

I have seen multiple errors: the current is: "field ingestion not found in type overrides.LegacyOverrides"

The question is

1. How should "global_overrides" be used ?
2. what is the correct format for "overrides:" ?

I have been trying to use this as a guide https://grafana.com/docs/tempo/latest/configuration/#runtime-overrides

Expected behavior
I am sure this is me not fully understanding the documentation. A single example for a value file for my version (1.9.9) would be much appreciated, where typical rate limit configuation is done.
It might be related to #2802

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm
• helm Chart version: 1.9.9
• Tempo version: 2.4.1

Link to issue in tempo repo:
#3171

[benmathews]

I came here to file the same ticket. By examining the tempo and chart source, I was able to puzzle out the correct configuration. (And it has nothing to do with what is documented.)

Put something like this in your overrides.yaml file.

tempo:
  structuredConfig:
    overrides:
      defaults:
        ingestion:
          rate_limit_bytes: 40000000
          burst_size_bytes: 50000000
          max_traces_per_user: 30000
        global:
          max_bytes_per_trace: 8000000

[vterdunov]

I came here to file the same ticket. By examining the tempo and chart source, I was able to puzzle out the correct configuration. (And it has nothing to do with what is documented.)

Put something like this in your overrides.yaml file.

tempo:
  structuredConfig:
    overrides:
      defaults:
        ingestion:
          rate_limit_bytes: 40000000
          burst_size_bytes: 50000000
          max_traces_per_user: 30000
        global:
          max_bytes_per_trace: 8000000

This fails for me with

failed parsing config: failed to parse configFile /conf/tempo.yaml: yaml: unmarshal errors:
  line 76: field defaults not found in type overrides.legacyConfig

Chart: 1.9.11
App Version: 2.4.2

[benmathews]

Chart

I'm using chart 1.9.10, but don't see anything in the diff that would matter. 🤷

[pantuza]

If you want to workaround this Helm Chart issue, just use as follows:

global_overrides:
  defaults:
    ingestion:
      rate_limit_bytes: 32000000 # 32MB
      burst_size_bytes: 48000000 # 48MB
      max_traces_per_user: 50000

Here I explain a little bit further why the issue is happening.

[buker]

global_overrides:
defaults:
ingestion:
rate_limit_bytes: 32000000 # 32MB
burst_size_bytes: 48000000 # 48MB
max_traces_per_user: 50000

This one not working for me...
field defaults not found in type overrides.legacyConfig

[pantuza]

Hi, @buker . You probably have to figure out why your configs are using the legacyConfig.
The defaults block is from the current Config format.

For the record, check Tempo code for:

• Legacy Config
• Current Config

Hope that helps figuring things out.

[knylander-grafana]

Hi there! I noticed that we had a bunch of issues open for the overrides settings for the tempo-distributed Helm. chart. We have two updates that may address the issue that you're having:

• [tempo-distributed] Clarifies use of global_overrides and overrides objects. #3468
• [DOC] Add overrides info to tempo-distributed doc tempo#4415

Please let me know if this addresses the issue.