Loki exporter: handle multi-tenant use cases

[jean]

Is your feature request related to a problem? Please describe.

The lokiexporter has a tenant_id parameter that can set the X-Scope-OrgID HTTP header, but this sets a single static tenant.
We will have logs from 1000s of orgs coming in, and we were assuming that client devices would be setting X-Scope-OrgID HTTP header, and that the exporter would be passing this through to Loki, resulting in 1000s of separate per-org log streams in S3.

Currently this would require an exporter instance per tenant, which doesn't work.

Describe the solution you'd like

Enable passing OrgID either as the X-Scope-OrgID HTTP header or as a tenant "Attribute" on the spans/logs.

Then within the Loki exporter we can use that attribute for setting the tenant header. The exporter groups logs into Loki LogStreams based on the cardinality of the labels, so the grouping would also need to account for the tenant attribute. It may make sense to combine this solution with the "Group by Attributes" processor in the pipeline. slack

We want to send logs in the form of spans. I've created a separate issue for that: #3166

Describe alternatives you've considered

We don't at this time have an alternative for routing logs to S3 for long-term storage via the Collector.

Additional context

Related issues:

• Access headers in processor (Access headers in processor opentelemetry-collector#2401)
• Multi tenancy support proposal (Multi tenancy support proposal opentelemetry-collector#2495)

[gillg]

This dynamic tenant ID could be based on log attributes ?
It seems a good attributes usage (standard in OTLP), then with a config like tenant_id_from rule pointing on an OTLP attribute name, if could make the job.

[gramidt]

Thank you so much for creating this issue and referencing our Slack conversation, @jean! Much appreciated.

Could we also split this issue into two? One for mutli-tenant based on attribute and the other for adding Span support to the exporter? The Span support is more complex and will likely have some long discussions on the best path for implementation.

[jean]

Hi @gramidt, I've split out span support to #3166, and updated this issue's description accordingly.

[gramidt]

Thank you, @jean!

[politician]

@jean This was posted a year ago, I am curious to know what you ended up doing for your use case?

@gillg The tenant_id_from would be perfect for this use case. But Tempo (OTLP Exporter) also makes use of X-Scope-OrgID header, just like Loki (and maybe other exporters have similar needs).

So, how about a more generic solution that could be implemented in other exporters as well as Loki?

For example:

processors:
  k8sattributes:

exporters:
  loki:
    headers:
      - action: insert
        key: X-Scope-OrgID
        from_attribute: k8s.namespace.name

  otlp/tempo:
    headers:
      - action: insert
        key: X-Scope-OrgID
        from_attribute: k8s.namespace.name

[jean]

@politician we're sending directly to Loki, not using OTel.

[gillg]

So, how about a more generic solution that could be implemented in other exporters as well as Loki?

I like it. To see with other maintainers if it's not too bad to do this kind of transformations outside processors.

But saying it, it doesn't alter data, we just reuse an internal lib to insert/upset/delete data inside an http header, so that seems correct

[jpkrohling]

I think we have almost all of the pieces to get this done, but we need a small change to the loki exporter. I'll add this to my queue, unless someone else is interested.

[otaviocarvalho]

@jpkrohling looking at this one, please assign it to me

[jpkrohling]

As we briefly discussed via Slack, we might not even need any code changes. If you can get it working with the current options already, that would be great. Otherwise, we can make this exporter aware of the recent changes to the client package: if the user has specified that the source of the tenant information is the original request, we could try to get it from the client from the context.

https://github.com/open-telemetry/opentelemetry-collector/blob/9d47500c66ac98f74a72b371a69f3b8d094dd3d1/client/client.go#L110-L112

[politician]

@jpkrohling can you please show an example yaml config of how to use it if there are no code changes?

An example of using a k8s workspace as tenant like I've shown in my snippet would be great since it's kind of a standard use case.

[jpkrohling]

Sorry, my comment was meant to @otaviocarvalho. He'd be the one trying to get a working config, and if he can't, we'll do some code changes.

[itaykat]

Does this solution of exporting headers is available also in Kafka exporter? i.e if I want to set a specific attribute field in the Kafka message header.

[jpkrohling]

@itaykat, if you didn't have an answer yet, it might be better to open a new issue with your question. I'll then tag the Kafka exporter code owners.

@otaviocarvalho, do you have an status on this one?