[exporter/loki] Auto replace labels with dot to underscore

[rogeriospinaintelipost]

Is your feature request related to a problem? Please describe.

Loki doesn`t accept labels that contains dots, so I have to duplicate my attributes via processors copying the value from attribute with dot to other attribute with underscore.

Ex: host.name copy to -> host_name

      resource:
        attributes:     
          - action: insert
            key: host_name
            from_attribute: host.name
    
          - action: insert
            key: namespace
            from_attribute: k8s.namespace.name
   
          - action: insert
            key: container_name
            from_attribute: k8s.container.name

Describe the solution you'd like

Would be nice if the str replace happens automatically by the lokiexporter

Describe alternatives you've considered

No response

Additional context

@jpkrohling

[github-actions]

Pinging code owners: @gramidt @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[disfluxly]

This is absolutely needed, especially with the change between the Legacy Exporter & the New Exporter and the way hints are now set by the Attributes Processor, whereas previously you could do dot to underscore label mapping in the Legacy Exporter.

[jpkrohling]

@mar4uk, can you keep this on your radar?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[fkalinowski]

Hi, something similar to resource_to_telemetry_conversion feature available in exporter/prometheusremotewriteexporter would also be of great help !

BTW: there is already a translator to convert from OpenTelemetry naming convention into Prometheus (here Loki) naming convention cf. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/translator/prometheus

[jpkrohling]

@kovrus, @mar4uk, I think it would be a good idea to consider this, especially if this exists on the Prometheus remote writer exporter. What do you think?

[kovrus]

+1 We can add it. I guess, we would keep it behind some config parameter and not make it default behavior?

[kago-dk]

If Loki doesn't accept labels that contain dots, in which scenario would you need to send labels with dots? I cannot rule out that I am reading the description correctly.

[jpkrohling]

@kovrus, didn't we have this translation already in the past? Did it get lost?

[disfluxly]

@kago-dk - Loki doesn't accept label index mappings with . in them. So currently, if you want to index service.name, you need to use the attributes processor to effectively duplicate service.name into service_name, as shown in the original description. You then have to add the newly created attribute (service_name) to the loki.attribute/resource.labels attribute/resourceattribute. This is seen better here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md

Adding in the auto-conversion option would effectively remove the necessity of having to relabel all your necessary attributes/labels.

So basically I'm 🙏 over here for this to be added :)

[kovrus]

@kovrus, didn't we have this translation already in the past? Did it get lost?

In the legacy exporter, we used the attributes mapping as part of the exporter config to create loki labels.
In the new exporter with labels hints we use the attributes processor to do the same thing.

+1 We can add it. I guess, we would keep it behind some config parameter and not make it default behavior?

I am actually hesitating now about supporting this feature. It would require a major config change, also I guess, it would make sense to control what loki labels should be created, so exclude/include filters probably would have to be added as well, etc. Also, changing attribute names (replacing . with _) can be achieved with the transform processor.

[kovrus]

Adding in the auto-conversion option would effectively remove the necessity of having to relabel all your necessary attributes/labels.

FYI We also use the hints approach (using the attributes processor) to provide tenant information to the exporter.

[jpkrohling]

I agree, @kovrus. Besides, Loki may end up supporting dots in the future, causing yet another breaking change for this exporter. I'd stick with explicit vs. implicit here.

[gouthamve]

I want to see if we can revisit this. I don't see . support as a priority item on the Loki roadmap and I think it might be months before Loki supports it.

And even if it does support it, we still have to support the existing users of Loki that haven't or cannot upgrade.

Also, changing attribute names (replacing . with _) can be achieved with the transform processor.

Could you give an example? The best I can get to is the following:

Without Transform processor

processors:
  attributes:
    actions:
    - action: insert
      key: event_domain
      from_attribute: event.domain
    - action: insert
      key: loki.attribute.labels
      value: event_domain

  resource:
    attributes:
    - action: insert
      key: service_name
      from_attribute: service.name
    - action: insert
      key: service_namespace
      from_attribute: service.namespace
    - action: insert
      key: loki.resource.labels
      value: service_name, service_namespace

With Transform Processor:

processors:
  transform:
    log_statements:
      - context: resource
        statements:
          - set(attributes["service_name"], attributes["service.name"])
          - set(attributes["service_namespace"], attributes["service.namespace"])
      - context: log
        statements:
          - set(attributes["event_domain"], attributes["event.domain"])

  attributes:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: event_domain

  resource:
    attributes:
    - action: insert
      key: loki.resource.labels
      value: service_name, service_namespace

(I couldn't find how to do "insert" with OTTL). Again, this is from me reading the docs but I couldn't find a simpler way to do it.

While this is smaller, its only by a bit and its just as cumbersome as before. If I want to do 6 labels in Loki, I have to manually escape them and specify the config.

[gouthamve]

Hi, I synced with @kovrus offline and we decided to support this usecase. We will escape . to _ before sending to Loki.

[fkalinowski]

Hi @gouthamve, great news that you will support automatic escape of . to _ !
What about resource_to_telemetry_conversion feature available in exporter/prometheusremotewriteexporter, will you support it also ?

[gouthamve]

That is a more risky thing to support as it could lead to massive cardinality. However, given a resource is more or less stable, I am inclined to consider it.

Could you open a new issue for this so we can have a discussion there?

[kovrus]

@mar4uk would like to take it over, so it is assigned to her now.

[jpkrohling]

This seems to have been done. I'm closing, if anything is still pending let me know and I'll reopen.