OTLP/HTTP Exporter does not respect tls/insecure setting

[js8080]

Describe the bug
The OTLP/HTTP Exporter has a documented configuration option for skipping TLS verification, e.g.

exporters:
  otlphttp:
    endpoint: "https://1.2.3.4:1234"
    tls:
      insecure: true

Unfortunately, it does not seem to be respected in otel/collector:0.44.0

Steps to reproduce
My collector config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"   # default grpc endpoint is 0.0.0.0:4317

processors:
  batch:

exporters:
  otlphttp:
    endpoint: https://hostname/tempo-gateway/otlp
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

I then start the collector using the following docker-compose file, mounting the config above into the container as /etc/otel-config.yaml:

version: '3.4'
services:
  collector:
    image: otel/opentelemetry-collector:0.44.0
    hostname: otel-collector
    ports:
      - "4317:4317"
    command: "--config /etc/otel-config.yaml"
    volumes:
      - ./otel-tocluster-config.yaml:/etc/otel-config.yaml

I then send a trace over to the collector from my application and it outputs the following error:

collector_1  | 2022-02-09T18:46:54.343Z info    exporterhelper/queued_retry.go:215      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"https://hostname/tempo-gateway/otlp/v1/traces\": x509: certificate signed by unknown authority", "interval": "16.969110576s"}

So it is clearly performing TLS verification even though I have tls/insecure: true

What did you expect to see?
I expected the collector to skip TLS verification as documented.

What did you see instead?
The collector is still attempting to verify the server TLS certificate.

What version did you use?
Version: v0.44.0

What config did you use?

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"   # default grpc endpoint is 0.0.0.0:4317

processors:
  batch:

exporters:
  otlphttp:
    endpoint: https://hostname/tempo-gateway/otlp
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Environment
OS: Ubuntu 20.04

[codeboten]

@js8080 is it possible you're looking for insecure_skip_verify instead? insecure disables TLS altogether, see https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/configtls#tls-configuration-settings

[js8080]

@js8080 is it possible you're looking for insecure_skip_verify instead? insecure disables TLS altogether, see https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/configtls#tls-configuration-settings

Hmm, maybe... Although as documented in the OTLP/HTTP Exporter README:

insecure (default = false): when set to true disables verifying the server's certificate chain and host name. The connection is still encrypted but server identity is not verified.

[jpkrohling]

You are specifying https as the protocol in the endpoint parameter, which will cause HTTP clients to hit the port 443 (TLS). Try to use http instead:

    endpoint: http://hostname/tempo-gateway/otlp

[js8080]

You are specifying https as the protocol in the endpoint parameter, which will cause HTTP clients to hit the port 443 (TLS). Try to use http instead:

    endpoint: http://hostname/tempo-gateway/otlp

Yes because I want to use TLS, I just didn't want to verify the server certificate like the OTLP/HTTP Exporter README describes the insecure setting. I think there is either a documentation mistake or coding mistake -- as @codeboten pointed out, there appears to be another setting documented in the TLS Configuration Settings called insecure_skip_verify. So my guess is the documentation on the OTLP/HTTP Exporter just needs to be fixed.

[jpkrohling]

True, sorry, I misunderstood the original problem.

[codeboten]

Ah yes, in this case if insecure_skip_verify supports your usecase, we can fix this with a documentation change. Would you like to submit a fix @js8080, if not I can assign it to myself.

[js8080]

@codeboten I have to go through an approval process from my company to contribute so it's probably more straightforward for you.

[codeboten]

👍 np, thanks for reporting this @js8080!

[anujap18]

For me, the insecure_skip_verify=True also does not work. It says that it is an unexpected keyword argument.

[codeboten]

@anujap18 can you provide collector version information along with a configuration example?

[anujap18]

I am actually trying to simulate collector data in otlp format and push to grpc client. I am using OTLPMetricExporter ([https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlphttpexporter/README.md]) to push data to the grpc client. So, in that if I add this arg insecure_skip_verify=True, it says invalid argument. insecure=True also does not work.

[codeboten]

I just tested the following configuration and it worked w/ the latest collector:

exporters:
    logging:
        loglevel: debug
    otlphttp:
        endpoint: http://127.0.0.1:4318
        tls:
            insecure: true
            insecure_skip_verify: true

[zekiahmetbayar]

I just tested the following configuration and it worked w/ the latest collector:

exporters:
    logging:
        loglevel: debug
    otlphttp:
        endpoint: http://127.0.0.1:4318
        tls:
            insecure: true
            insecure_skip_verify: true

when i use insecure and insecure_skip_verify params, collector throws error

otel-collector-1  | Error: cannot start pipelines: grpc: the credentials require transport level security (use grpc.WithTransportCredentials() to set)
otel-collector-1  | 2024/06/03 07:41:13 collector server run finished with error: cannot start pipelines: grpc: the credentials require transport level security (use grpc.WithTransportCredentials() to set)
otel-collector-1 exited with code 1

[codeboten]

@zekiahmetbayar can you paste your configuration? I just re-tested the config in my previous comment and it worked as expected