Access headers in processor

[pavolloffay]

Is your feature request related to a problem? Please describe.

I would like to access HTTP headers in the processor pipeline and add attribute to span/resource - e.g. tenant ID.

Describe the solution you'd like

Read the headers from the context object in a processor. The headers which should be injected into the context could be specified in the receiver config.

Describe alternatives you've considered

A custom implementation of Zipkin/OTLP receiver that does that.

Additional context

Related issue #2101

[pavolloffay]

import "google.golang.org/grpc/metadata"
md, ok := metadata.FromIncomingContext(ctx)

returns the HTTP headers for any gRPC receiver. However the processor cannot be used after the batch processor. The same approach uses https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/master/processor/routingprocessor

However, I also need to access the headers from Jaeger HTTP thrift (14268) receiver.

[pavolloffay]

@bogdandrutu @tigrannajaryan could we add HTTP headers to the context in the HTTP based receivers?

[jcchavezs]

I think if this goes forward there should be a way to access headers (metadata) decoupled from the receiver (e.g. a method to access it instead of the need to call grpc metadata.FromIncomingContext or protocol specific), that is probably the less implementation aware way.

[bogdandrutu]

@pavolloffay we were just discussing this, and I don't know what is the right approach. The problem with adding it to the Context is that if batch span processor is the first processor that will drop the context, so most likely we have to carry this with the data itself and not the Context.

[pavolloffay]

The problem with adding it to the Context is that if batch span processor is the first processor that will drop the context, so most likely we have to carry this with the data itself and not the Context.

Correct, I understand this as a limitation of the current design which is fine for us since the processor that access the headers will run immediately after the receiver.
The https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/master/processor/routingprocessor uses it as well and documents how the processor should be used.

Shall we start thinking on merging contexts in the batch processor or changing the API to bind context to each original tracedata?

[tigrannajaryan]

Related to #2495 which has similar requirements.

[bogdandrutu]

@jpkrohling is this related to the auth extensions pass-through? Can you take this?

[jpkrohling]

This was requested earlier, and some aspects of this might be doable with the auth work. Once the passthrough PR is done, it should be easy to use the same technique to implement this, by storing the headers in the Resource* structs.

[PatrikSteuer]

Are there any thoughts on this? Or ideas how an implementation should look like?

[jpkrohling]

We have the client package which is just being finished to propagate the auth information. This here could use the same facility in the future.

[bogdandrutu]

@jpkrohling I do believe that this does not provide enough information about why this is needed. If somebody random comes and say I need "foo" are you going to implement it?

I am not suggesting that this is not important and we should not do it, but would be good to understand why this is needed.

[jpkrohling]

If somebody random comes and say I need "foo" are you going to implement it?

No, this one has been requested a few times in the past already though. The typical reasoning is using the incoming HTTP headers down the pipeline to make tenancy decisions, either at the processor level (routing processor, for instance), or exporter level (passthrough of API keys).

[bogdandrutu]

@jpkrohling awesome, but because of unnecessary clone cost in some cases I would recommend to do this opt-in. Have a Boolean in the HttpClientSettings to enable this

[jpkrohling]

Sounds reasonable to me.

[jcchavezs]

@jpkrohling @bogdandrutu maybe I am missing something but the first step here is to receive the headers from the receiver and then pass to the processors, how is HTTPClientSettings involved?

[jpkrohling]

I believe he meant the server settings, used by most receivers

[disq]

Updated the PR now cloning headers optionally.

[yashvardhan-zs]

@jpkrohling @bogdandrutu maybe I am missing something but the first step here is to receive the headers from the receiver and then pass to the processors, how is HTTPClientSettings involved?

how can we receive the headers and send same to exporter for multi tennat?

[jpkrohling]

Take a look at the header setter extension for one of the possible solutions: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetter

[yashvardhan-zs]

Take a look at the header setter extension for one of the possible solutions: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetter

I tried to use but then collector stops. also please let me know what from_context means

[jpkrohling]

It's a new extension, I believe it might be available starting from 0.59.0.

[yashvardhan-zs]

It's a new extension, I believe it might be available starting from 0.59.0.

is there any available method to access headers from receiver and use it in exporter as tennat?

[jpkrohling]

What's your concrete use case? Without knowing that, I can only provide general information.

HTTP headers are part of the client metadata. You can access them via the attributes or resources processor, adding that information to the data points. Exporters would then be responsible for accessing this data. The header setter helps in this last part by allowing the headers to be used as values for headers on outgoing HTTP calls.

[yashvardhan-zs]

What's your concrete use case? Without knowing that, I can only provide general information.

HTTP headers are part of the client metadata. You can access them via the attributes or resources processor, adding that information to the data points. Exporters would then be responsible for accessing this data. The header setter helps in this last part by allowing the headers to be used as values for headers on outgoing HTTP calls.

i am using zipkin as receiver and taking headers from application, also metatdata: true and now i want to use this headers in exporter to diffrentiate tennat on the basis of headers.

receivers:
  otlp:
    protocols:
      grpc:
  zipkin:
exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true
    headers:
      X-Scope-OrgID: 1234   //////////////////////////// I want to take this either from receiver or dynamically
service:
  pipelines:
    traces:
      receivers: [otlp, zipkin]
      exporters: [otlp]

[jpkrohling]

Your best bet is indeed the header setter. That's exactly the use-case it's solving.

[yash1539]

What is from context here in headerssetter