Regression: crash when TLS insecure is set on authenticators #6619

Depechie · 2022-11-17T14:48:36Z

Component(s)

extension/oauth2clientauth

What happened?

Description

Trying to have 2 opentelemetry collectors talk to each other using the oauth2client extension ( agent < > server ).

Steps to Reproduce

Docker compose file

services:
  otel-agent:
    image: otel/opentelemetry-collector-contrib:0.64.1
    container_name: otel-agent
    hostname: secure-otel
    restart: unless-stopped
    ports:
      - 5317:4317
    environment:
      - TZ=Europe/Brussels
    volumes:
      - type: bind
        source: /var/lib/docker/volumes/secure-otel/otel-agent.yml
        target: /etc/otel-collector-config.yaml
        read_only: true
    command: ["--config=/etc/otel-collector-config.yaml"]
    depends_on:
      - otel-server
      
  otel-server:
    image: otel/opentelemetry-collector-contrib:0.64.1
    container_name: otel-server
    hostname: secure-otel
    restart: unless-stopped
    ports:
      - 4317:4317
    environment:
      - TZ=Europe/Brussels
    volumes:
      - type: bind
        source: /var/lib/docker/volumes/secure-otel/otel-server.yml
        target: /etc/otel-collector-config.yaml
        read_only: true
    command: ["--config=/etc/otel-collector-config.yaml"]
    
  tempo:
    image: grafana/tempo:1.5.0
    container_name: tempo
    hostname: secure-otel
    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/lib/docker/volumes/secure-otel/tempo.yml:/etc/tempo.yaml
      - /var/lib/docker/volumes/secure-otel/tempo:/tmp/tempo
    restart: unless-stopped
    ports:
      - 3200:3200  # tempo
      - 4007:4317  # otlp grpc
    depends_on:
      - otel-server

With 2 collector configs defined

Expected Result

Correct data transfer

Actual Result

The opentelemetry configured as agent keeps crashing

Collector version

v0.64.1

Environment information

Environment

OS: Windows with WSL2 Ubuntu

OpenTelemetry Collector configuration

-- agent

extensions:
  oauth2client:
    client_id: ***
    client_secret: ***
    token_url: https://login.microsoftonline.com/***/oauth2/v2.0/token
    scopes: ["api://***/.default"]

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  otlp/auth:
    endpoint: otel-server:4317
    tls:      
      insecure: true
    auth:
      authenticator: oauth2client

service:
  extensions:
    - oauth2client
  pipelines:
    traces:
      receivers:
        - otlp
      exporters:
        - otlp/auth

-- server
extensions:
  oidc:
    issuer_url: https://login.microsoftonline.com/***/v2.0
    audience: ***

receivers:
  otlp/auth:
    protocols:
      grpc:
        auth:
          authenticator: oidc

exporters:
  otlp:
    endpoint: tempo:4007
    tls:
      insecure: true

service:
  extensions:
    - oidc
  pipelines:
    traces:
      receivers:
        - otlp/auth
      exporters:
        - otlp

Log output

2022-11-17T14:46:34.893Z	info	service/telemetry.go:110	Setting up own telemetry...
2022-11-17T14:46:34.893Z	info	service/telemetry.go:140	Serving Prometheus metrics	{"address": ":8888", "level": "basic"}
2022-11-17T14:46:34.894Z	info	service/service.go:89	Starting otelcol-contrib...	{"Version": "0.64.1", "NumCPU": 12}
2022-11-17T14:46:34.894Z	info	extensions/extensions.go:41	Starting extensions...
2022-11-17T14:46:34.894Z	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "oauth2client"}
2022-11-17T14:46:34.894Z	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "oauth2client"}
2022-11-17T14:46:34.894Z	info	pipelines/pipelines.go:74	Starting exporters...
2022-11-17T14:46:34.894Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth"}
2022-11-17T14:46:34.894Z	info	service/service.go:115	Starting shutdown...
2022-11-17T14:46:34.894Z	info	pipelines/pipelines.go:118	Stopping receivers...
2022-11-17T14:46:34.894Z	info	pipelines/pipelines.go:125	Stopping processors...
2022-11-17T14:46:34.894Z	info	pipelines/pipelines.go:132	Stopping exporters...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x9c012b]
goroutine 1 [running]:
google.golang.org/grpc.(*ClientConn).Close(0x0)
	google.golang.org/[email protected]/clientconn.go:1016 +0x4b
go.opentelemetry.io/collector/exporter/otlpexporter.(*exporter).shutdown(0xc000997210?, {0x9?, 0x8ce8401?})
	go.opentelemetry.io/collector/exporter/[email protected]/otlp.go:93 +0x1d
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2({0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0x5a
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc000d36050, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/internal/pipelines/pipelines.go:135 +0x36b
go.opentelemetry.io/collector/service.(*service).Shutdown(0xc00039b200, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/service.go:121 +0xd4
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc0013c5a88, {0x74b9640?, 0xc0000bc018?})
	go.opentelemetry.io/[email protected]/service/collector.go:234 +0x36
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc0013c5a88, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/collector.go:155 +0x286
go.opentelemetry.io/collector/service.(*Collector).Run(0xc0013c5a88, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/collector.go:164 +0x46
go.opentelemetry.io/collector/service.NewCommand.func1(0xc00058cf00, {0x680c260?, 0x1?, 0x1?})
	go.opentelemetry.io/[email protected]/service/command.go:53 +0x479
github.com/spf13/cobra.(*Command).execute(0xc00058cf00, {0xc0000b4050, 0x1, 0x1})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc00058cf00)
	github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
main.runInteractive({{0xc000991860, 0xc0009b8a50, 0xc000991c80, 0xc000991500}, {{0x6833f67, 0xf}, {0x68af92d, 0x1f}, {0x6805d6e, 0x6}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x1d8

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2022-11-17T14:48:55Z

Pinging code owners:

extension/oauth2clientauth: @open-telemetry/collector-contrib-approvers @pavankrish123 @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

pavankrish123 · 2022-11-17T16:11:37Z

@Depechie can you please enable debug logging and share the agent collector logs.

I have quickly run 0.64.1 on my machine with custom oauth server it works fine.

Depechie · 2022-11-17T21:53:34Z

Hey @pavankrish123
So I added following section to the otel collector config for the agent:

service:
  telemetry:
    logs:
      level: "debug"

But is that enough, because the logs don't seem to be different?

2022-11-17T21:51:52.805Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
2022-11-17T21:51:52.805Z	info	service/service.go:115	Starting shutdown...
2022-11-17T21:51:52.805Z	info	pipelines/pipelines.go:118	Stopping receivers...
2022-11-17T21:51:52.805Z	info	pipelines/pipelines.go:125	Stopping processors...
2022-11-17T21:51:52.805Z	info	pipelines/pipelines.go:132	Stopping exporters...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x9c012b]
goroutine 1 [running]:
google.golang.org/grpc.(*ClientConn).Close(0x0)
	google.golang.org/[email protected]/clientconn.go:1016 +0x4b
go.opentelemetry.io/collector/exporter/otlpexporter.(*exporter).shutdown(0xc000a133d0?, {0x9?, 0x8ce8401?})
	go.opentelemetry.io/collector/exporter/[email protected]/otlp.go:93 +0x1d
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2({0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0x5a
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc000c165a0, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/internal/pipelines/pipelines.go:135 +0x36b
go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000a4d100, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/service.go:121 +0xd4
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc0013b7a88, {0x74b9640?, 0xc0001a6000?})
	go.opentelemetry.io/[email protected]/service/collector.go:234 +0x36
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc0013b7a88, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/collector.go:155 +0x286
go.opentelemetry.io/collector/service.(*Collector).Run(0xc0013b7a88, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/collector.go:164 +0x46
go.opentelemetry.io/collector/service.NewCommand.func1(0xc000bff200, {0x680c260?, 0x1?, 0x1?})
	go.opentelemetry.io/[email protected]/service/command.go:53 +0x479
github.com/spf13/cobra.(*Command).execute(0xc000bff200, {0xc0001b4010, 0x1, 0x1})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000bff200)
	github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
main.runInteractive({{0xc00097bef0, 0xc000a4f0e0, 0xc000a4e330, 0xc00097bb90}, {{0x6833f67, 0xf}, {0x68af92d, 0x1f}, {0x6805d6e, 0x6}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x1d8
2022/11/17 21:51:54 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/11/17 21:51:54 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-11-17T21:51:55.027Z	info	service/telemetry.go:110	Setting up own telemetry...
2022-11-17T21:51:55.028Z	info	service/telemetry.go:140	Serving Prometheus metrics	{"address": ":8888", "level": "basic"}
2022-11-17T21:51:55.028Z	debug	components/components.go:28	Stable component.	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth", "stability": "stable"}
2022-11-17T21:51:55.028Z	debug	components/components.go:28	Stable component.	{"kind": "receiver", "name": "otlp", "pipeline": "traces", "stability": "stable"}
2022-11-17T21:51:55.028Z	info	service/service.go:89	Starting otelcol-contrib...	{"Version": "0.64.1", "NumCPU": 12}
2022-11-17T21:51:55.028Z	info	extensions/extensions.go:41	Starting extensions...
2022-11-17T21:51:55.028Z	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "oauth2client"}
2022-11-17T21:51:55.028Z	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "oauth2client"}
2022-11-17T21:51:55.028Z	info	pipelines/pipelines.go:74	Starting exporters...
2022-11-17T21:51:55.028Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth"}
2022-11-17T21:51:55.029Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel created	{"grpc_log": true}
2022-11-17T21:51:55.029Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel Connectivity change to SHUTDOWN	{"grpc_log": true}
2022-11-17T21:51:55.029Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
2022-11-17T21:51:55.029Z	info	service/service.go:115	Starting shutdown...
2022-11-17T21:51:55.029Z	info	pipelines/pipelines.go:118	Stopping receivers...
2022-11-17T21:51:55.029Z	info	pipelines/pipelines.go:125	Stopping processors...
2022-11-17T21:51:55.029Z	info	pipelines/pipelines.go:132	Stopping exporters...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x9c012b]
goroutine 1 [running]:
google.golang.org/grpc.(*ClientConn).Close(0x0)
	google.golang.org/[email protected]/clientconn.go:1016 +0x4b
go.opentelemetry.io/collector/exporter/otlpexporter.(*exporter).shutdown(0xc0008af5d0?, {0x9?, 0x8ce8401?})
	go.opentelemetry.io/collector/exporter/[email protected]/otlp.go:93 +0x1d
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2({0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0x5a
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc000c35cc0, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/internal/pipelines/pipelines.go:135 +0x36b
go.opentelemetry.io/collector/service.(*service).Shutdown(0xc0009ad700, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/service.go:121 +0xd4
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc001337a88, {0x74b9640?, 0xc0000bc018?})
	go.opentelemetry.io/[email protected]/service/collector.go:234 +0x36
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc001337a88, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/collector.go:155 +0x286
go.opentelemetry.io/collector/service.(*Collector).Run(0xc001337a88, {0x74b9640, 0xc0000bc018})
	go.opentelemetry.io/[email protected]/service/collector.go:164 +0x46
go.opentelemetry.io/collector/service.NewCommand.func1(0xc00094d200, {0x680c260?, 0x1?, 0x1?})
	go.opentelemetry.io/[email protected]/service/command.go:53 +0x479
github.com/spf13/cobra.(*Command).execute(0xc00094d200, {0xc0000b4050, 0x1, 0x1})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc00094d200)
	github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
main.runInteractive({{0xc00089fb60, 0xc0008d2d50, 0xc00089ff80, 0xc00089f800}, {{0x6833f67, 0xf}, {0x68af92d, 0x1f}, {0x6805d6e, 0x6}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x1d8

Depechie · 2022-11-18T06:08:20Z

@pavankrish123 If it would help, I can also pass all the oauth settings data... it is just a test app registration in Azure.
So can easily delete it when this is fixed/working.

Just let me know if that is needed or not.

pavankrish123 · 2022-11-18T07:39:22Z

Thanks @Depechie. I was looking for these grpc logs

2022-11-17T21:51:55.029Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel created	{"grpc_log": true}
2022-11-17T21:51:55.029Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel Connectivity change to SHUTDOWN	{"grpc_log": true}
2022-11-17T21:51:55.029Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
2022-11-17T21:51:55.029Z	info	service/service.go:115	Starting shutdown...

Is it possible to enable grpc debug logging by setting these environment variables as specified

`GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info`.

Also Quick question, curious any reason why we have inferred that oauth2client extension is causing the crash? logs indicate that shutdown of the exporter is triggering the issue - that said something is causing the shutdown

Depechie · 2022-11-18T07:44:52Z

Well @pavankrish123 if I create 2 ( agent & server ) services in the docker compose file that just leave out the

    auth:
      authenticator: oauth2client

part, everything just works.
It's as soon as I add the configuration for oauth and use the extension, the agent crashes. The server keeps working.
If I leave out the oauth2client part only at agent level, the agent stays active and if I then send trace info, I get following log line in the otel server logs: Unauthorized.
So it seems that the oauth2client setting there does work.

I will try adding the environment variable, this is done at docker container level I suppose?

Depechie · 2022-11-18T07:58:24Z

services:
  otel-agent:
    image: otel/opentelemetry-collector-contrib:0.64.1
    container_name: otel-agent
    hostname: secure-otel
    restart: unless-stopped
    ports:
      - 5317:4317
    environment:
      - TZ=Europe/Brussels
      - GRPC_GO_LOG_VERBOSITY_LEVEL=99
      - GRPC_GO_LOG_SEVERITY_LEVEL=info
    volumes:
      - type: bind
        source: /var/lib/docker/volumes/secure-otel/otel-agent.yml
        target: /etc/otel-collector-config.yaml
        read_only: true
    command: ["--config=/etc/otel-collector-config.yaml"]
    depends_on:
      - otel-server

Logs:

2022-11-18T07:58:09.652Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
2022-11-18T07:58:09.652Z	info	service/service.go:115	Starting shutdown...
2022-11-18T07:58:09.652Z	info	pipelines/pipelines.go:118	Stopping receivers...
2022-11-18T07:58:09.652Z	info	pipelines/pipelines.go:125	Stopping processors...
2022-11-18T07:58:09.652Z	info	pipelines/pipelines.go:132	Stopping exporters...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x9c012b]
goroutine 1 [running]:
google.golang.org/grpc.(*ClientConn).Close(0x0)
	google.golang.org/[email protected]/clientconn.go:1016 +0x4b
go.opentelemetry.io/collector/exporter/otlpexporter.(*exporter).shutdown(0xc000a0a330?, {0x9?, 0x8ce8401?})
	go.opentelemetry.io/collector/exporter/[email protected]/otlp.go:93 +0x1d
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2({0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0x5a
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc0009e7b80, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/internal/pipelines/pipelines.go:135 +0x36b
go.opentelemetry.io/collector/service.(*service).Shutdown(0xc0006aa000, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/service.go:121 +0xd4
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc0015b1a88, {0x74b9640?, 0xc0001a6000?})
	go.opentelemetry.io/[email protected]/service/collector.go:234 +0x36
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc0015b1a88, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/collector.go:155 +0x286
go.opentelemetry.io/collector/service.(*Collector).Run(0xc0015b1a88, {0x74b9640, 0xc0001a6000})
	go.opentelemetry.io/[email protected]/service/collector.go:164 +0x46
go.opentelemetry.io/collector/service.NewCommand.func1(0xc000968300, {0x680c260?, 0x1?, 0x1?})
	go.opentelemetry.io/[email protected]/service/command.go:53 +0x479
github.com/spf13/cobra.(*Command).execute(0xc000968300, {0xc0001b4010, 0x1, 0x1})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000968300)
	github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
main.runInteractive({{0xc00095f500, 0xc00097c6f0, 0xc00095f920, 0xc00095f1a0}, {{0x6833f67, 0xf}, {0x68af92d, 0x1f}, {0x6805d6e, 0x6}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x1d8
2022/11/18 07:58:10 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/11/18 07:58:10 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-11-18T07:58:10.790Z	info	service/telemetry.go:110	Setting up own telemetry...
2022-11-18T07:58:10.791Z	info	service/telemetry.go:140	Serving Prometheus metrics	{"address": ":8888", "level": "basic"}
2022-11-18T07:58:10.791Z	debug	components/components.go:28	Stable component.	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth", "stability": "stable"}
2022-11-18T07:58:10.791Z	debug	components/components.go:28	Stable component.	{"kind": "receiver", "name": "otlp", "pipeline": "traces", "stability": "stable"}
2022-11-18T07:58:10.792Z	info	service/service.go:89	Starting otelcol-contrib...	{"Version": "0.64.1", "NumCPU": 12}
2022-11-18T07:58:10.792Z	info	extensions/extensions.go:41	Starting extensions...
2022-11-18T07:58:10.792Z	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "oauth2client"}
2022-11-18T07:58:10.792Z	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "oauth2client"}
2022-11-18T07:58:10.792Z	info	pipelines/pipelines.go:74	Starting exporters...
2022-11-18T07:58:10.792Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth"}
2022-11-18T07:58:10.792Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel created	{"grpc_log": true}
2022-11-18T07:58:10.792Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel Connectivity change to SHUTDOWN	{"grpc_log": true}
2022-11-18T07:58:10.792Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
2022-11-18T07:58:10.792Z	info	service/service.go:115	Starting shutdown...
2022-11-18T07:58:10.792Z	info	pipelines/pipelines.go:118	Stopping receivers...
2022-11-18T07:58:10.792Z	info	pipelines/pipelines.go:125	Stopping processors...
2022-11-18T07:58:10.792Z	info	pipelines/pipelines.go:132	Stopping exporters...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x9c012b]
goroutine 1 [running]:
google.golang.org/grpc.(*ClientConn).Close(0x0)
	google.golang.org/[email protected]/clientconn.go:1016 +0x4b
go.opentelemetry.io/collector/exporter/otlpexporter.(*exporter).shutdown(0xc000ce9340?, {0x9?, 0x8ce8401?})
	go.opentelemetry.io/collector/exporter/[email protected]/otlp.go:93 +0x1d
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2({0x74b9640, 0xc0000be018})
	go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0x5a
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc00108a460, {0x74b9640, 0xc0000be018})
	go.opentelemetry.io/[email protected]/service/internal/pipelines/pipelines.go:135 +0x36b
go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000128c80, {0x74b9640, 0xc0000be018})
	go.opentelemetry.io/[email protected]/service/service.go:121 +0xd4
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc001347a88, {0x74b9640?, 0xc0000be018?})
	go.opentelemetry.io/[email protected]/service/collector.go:234 +0x36
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc001347a88, {0x74b9640, 0xc0000be018})
	go.opentelemetry.io/[email protected]/service/collector.go:155 +0x286
go.opentelemetry.io/collector/service.(*Collector).Run(0xc001347a88, {0x74b9640, 0xc0000be018})
	go.opentelemetry.io/[email protected]/service/collector.go:164 +0x46
go.opentelemetry.io/collector/service.NewCommand.func1(0xc0001fd500, {0x680c260?, 0x1?, 0x1?})
	go.opentelemetry.io/[email protected]/service/command.go:53 +0x479
github.com/spf13/cobra.(*Command).execute(0xc0001fd500, {0xc0000b4050, 0x1, 0x1})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001fd500)
	github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
main.runInteractive({{0xc000c29f80, 0xc000da5170, 0xc000da43c0, 0xc000c29c20}, {{0x6833f67, 0xf}, {0x68af92d, 0x1f}, {0x6805d6e, 0x6}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x1d8

pavankrish123 · 2022-11-18T10:16:02Z

Thanks @Depechie I think I can reproduce the error, need to get to the bottom of it - I am guessing recent changes to core collector service lifecycle management is causing this issue.

Just FYI, I was able to reproduce this on local machine directly
Steps : downloaded 0.64.1 version of otel-collector and started it with the following collector config yaml

./otelcol-contrib --config otel-agent.yml

extensions:
 oauth2client:
   client_id: agent
   client_secret:  ****** 
   token_url: http://localhost:8080/auth/realms/opentelemetry/protocol/openid-connect/token

receivers:
 otlp:
   protocols:
     grpc:

exporters:
 otlp/auth:
   endpoint: myserver:5000
   tls:
     insecure: true
   auth:
     authenticator: oauth2client

service:
 telemetry:
   logs:
     level: "debug"
 extensions:
   - oauth2client
 pipelines:
   traces:
     receivers:
       - otlp
     exporters:
       - otlp/auth

caused the entire process to crash

./otelcol-contrib --config otel-agent.yml 
2022/11/18 01:51:18 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/11/18 01:51:18 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-11-18T01:51:19.294-0800	info	service/telemetry.go:110	Setting up own telemetry...
2022-11-18T01:51:19.296-0800	info	service/telemetry.go:140	Serving Prometheus metrics	{"address": ":8888", "level": "basic"}
2022-11-18T01:51:19.297-0800	debug	components/components.go:28	Stable component.	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth", "stability": "stable"}
2022-11-18T01:51:19.298-0800	debug	components/components.go:28	Stable component.	{"kind": "receiver", "name": "otlp", "pipeline": "traces", "stability": "stable"}
2022-11-18T01:51:19.299-0800	info	service/service.go:89	Starting otelcol-contrib...	{"Version": "0.64.1", "NumCPU": 12}
2022-11-18T01:51:19.299-0800	info	extensions/extensions.go:41	Starting extensions...
2022-11-18T01:51:19.299-0800	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "oauth2client"}
2022-11-18T01:51:19.299-0800	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "oauth2client"}
2022-11-18T01:51:19.299-0800	info	pipelines/pipelines.go:74	Starting exporters...
2022-11-18T01:51:19.299-0800	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth"}
2022-11-18T01:51:19.299-0800	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel created	{"grpc_log": true}
2022-11-18T01:51:19.300-0800	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel Connectivity change to SHUTDOWN	{"grpc_log": true}
2022-11-18T01:51:19.300-0800	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
2022-11-18T01:51:19.300-0800	info	service/service.go:115	Starting shutdown...
2022-11-18T01:51:19.300-0800	info	pipelines/pipelines.go:118	Stopping receivers...
2022-11-18T01:51:19.300-0800	info	pipelines/pipelines.go:125	Stopping processors...
2022-11-18T01:51:19.300-0800	info	pipelines/pipelines.go:132	Stopping exporters...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x15bc0ab]

goroutine 1 [running]:
google.golang.org/grpc.(*ClientConn).Close(0x0)
	google.golang.org/[email protected]/clientconn.go:1016 +0x4b
go.opentelemetry.io/collector/exporter/otlpexporter.(*exporter).shutdown(0xc000e487d0?, {0x9?, 0x962cd01?})
	go.opentelemetry.io/collector/exporter/[email protected]/otlp.go:93 +0x1d
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2({0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0x5a
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc0000c18b0, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/internal/pipelines/pipelines.go:135 +0x36b
go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000633800, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/service.go:121 +0xd4
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc0015fba88, {0x7e544c0?, 0xc00019e000?})
	go.opentelemetry.io/[email protected]/service/collector.go:234 +0x36
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc0015fba88, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/collector.go:155 +0x286
go.opentelemetry.io/collector/service.(*Collector).Run(0xc0015fba88, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/collector.go:164 +0x46
go.opentelemetry.io/collector/service.NewCommand.func1(0xc00063d200, {0x71e628b?, 0x2?, 0x2?})
	go.opentelemetry.io/[email protected]/service/command.go:53 +0x479
github.com/spf13/cobra.(*Command).execute(0xc00063d200, {0xc00019a190, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc00063d200)
	github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
main.runInteractive({{0xc00107a4e0, 0xc00107b6b0, 0xc00107a900, 0xc0004199e0}, {{0x720c2dc, 0xf}, {0x7283ffd, 0x1f}, {0x71e023a, 0x6}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x1d8

cc: @jpkrohling

pavankrish123 · 2022-11-18T10:42:41Z

As a matter of fact any client auth extensions is causing the crash :(

example 1: enabling basic auth extension is also causing the same crash

extensions:
  # oauth2client:
  #   client_id: agent
  #   client_secret: AkW04qQhMavsbEzGmNofSsr576Ye3IVg
  #   token_url: http://localhost:8080/auth/realms/opentelemetry/protocol/openid-connect/token
  basicauth/client:
    client_auth:
      username: username
      password: password

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  otlp/auth:
    endpoint: otel-server:4317
    tls:
      insecure: true
    auth:
      authenticator: basicauth/client

service:
  telemetry:
    logs:
      level: "debug"
  extensions:
    - basicauth/client
  pipelines:
    traces:
      receivers:
        - otlp
      exporters:
        - otlp/auth

 $ ./otelcol-contrib --config otel-agent.yml
2022/11/18 02:34:13 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/11/18 02:34:13 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-11-18T02:34:13.783-0800	info	service/telemetry.go:110	Setting up own telemetry...
2022-11-18T02:34:13.783-0800	info	service/telemetry.go:140	Serving Prometheus metrics	{"address": ":8888", "level": "basic"}
2022-11-18T02:34:13.784-0800	debug	components/components.go:28	Stable component.	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth", "stability": "stable"}
2022-11-18T02:34:13.784-0800	debug	components/components.go:28	Stable component.	{"kind": "receiver", "name": "otlp", "pipeline": "traces", "stability": "stable"}
2022-11-18T02:34:13.784-0800	info	service/service.go:89	Starting otelcol-contrib...	{"Version": "0.64.1", "NumCPU": 12}
2022-11-18T02:34:13.784-0800	info	extensions/extensions.go:41	Starting extensions...
2022-11-18T02:34:13.784-0800	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "basicauth/client"}
2022-11-18T02:34:13.784-0800	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "basicauth/client"}
2022-11-18T02:34:13.784-0800	info	pipelines/pipelines.go:74	Starting exporters...
2022-11-18T02:34:13.784-0800	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth"}
2022-11-18T02:34:13.784-0800	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel created	{"grpc_log": true}
2022-11-18T02:34:13.784-0800	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel Connectivity change to SHUTDOWN	{"grpc_log": true}
2022-11-18T02:34:13.784-0800	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
2022-11-18T02:34:13.784-0800	info	service/service.go:115	Starting shutdown...
2022-11-18T02:34:13.784-0800	info	pipelines/pipelines.go:118	Stopping receivers...
2022-11-18T02:34:13.784-0800	info	pipelines/pipelines.go:125	Stopping processors...
2022-11-18T02:34:13.784-0800	info	pipelines/pipelines.go:132	Stopping exporters...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x15bc0ab]

goroutine 1 [running]:
google.golang.org/grpc.(*ClientConn).Close(0x0)
	google.golang.org/[email protected]/clientconn.go:1016 +0x4b
go.opentelemetry.io/collector/exporter/otlpexporter.(*exporter).shutdown(0xc0010d0ad0?, {0x9?, 0x962cd01?})
	go.opentelemetry.io/collector/exporter/[email protected]/otlp.go:93 +0x1d
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2({0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0x5a
go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown(...)
	go.opentelemetry.io/[email protected]/component/component.go:91
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc0011f0190, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/internal/pipelines/pipelines.go:135 +0x36b
go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000cad300, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/service.go:121 +0xd4
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc0012bfa88, {0x7e544c0?, 0xc00019e000?})
	go.opentelemetry.io/[email protected]/service/collector.go:234 +0x36
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc0012bfa88, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/collector.go:155 +0x286
go.opentelemetry.io/collector/service.(*Collector).Run(0xc0012bfa88, {0x7e544c0, 0xc00019e000})
	go.opentelemetry.io/[email protected]/service/collector.go:164 +0x46
go.opentelemetry.io/collector/service.NewCommand.func1(0xc0001d6c00, {0x71e628b?, 0x2?, 0x2?})
	go.opentelemetry.io/[email protected]/service/command.go:53 +0x479
github.com/spf13/cobra.(*Command).execute(0xc0001d6c00, {0xc00019a190, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001d6c00)
	github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:968
main.runInteractive({{0xc00085c750, 0xc00085d920, 0xc00085cb70, 0xc00085c3f0}, {{0x720c2dc, 0xf}, {0x7283ffd, 0x1f}, {0x71e023a, 0x6}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x1d8

@Depechie can you please validate this in your end and adjust the bug description accordingly.

Depechie · 2022-11-18T12:13:37Z

@pavankrish123 confirmed

pavankrish123 · 2022-11-18T17:28:06Z

@Depechie an update

If you remove tls.insecure section in exporter.otlp you should not see any issue. (for all the authentication extensions).

So here is what is happening setting tls.insecure on OTEL exporter usually lead to an error on OTEL collector and collector used to die of gracefully.

Logs from 0.62.0 version

otel-agent    | 2022-11-18T17:00:12.673Z	info	service/service.go:112	Starting otelcol-contrib...	{"Version": "0.60.0", "NumCPU": 8}
otel-agent    | 2022-11-18T17:00:12.673Z	info	extensions/extensions.go:42	Starting extensions...
otel-agent    | 2022-11-18T17:00:12.673Z	info	extensions/extensions.go:45	Extension is starting...	{"kind": "extension", "name": "oauth2client"}
otel-agent    | 2022-11-18T17:00:12.673Z	info	extensions/extensions.go:49	Extension started.	{"kind": "extension", "name": "oauth2client"}
otel-agent    | 2022-11-18T17:00:12.673Z	info	pipelines/pipelines.go:74	Starting exporters...
otel-agent    | 2022-11-18T17:00:12.673Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth"}
otel-agent    | 2022-11-18T17:00:12.673Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel created	{"grpc_log": true}
otel-agent    | 2022-11-18T17:00:12.673Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel Connectivity change to SHUTDOWN	{"grpc_log": true}
otel-agent    | 2022-11-18T17:00:12.673Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
otel-agent    | Error: cannot start pipelines: grpc: the credentials require transport level security (use grpc.WithTransportCredentials() to set)
otel-agent    | 2022/11/18 17:00:12 collector server run finished with error: cannot start pipelines: grpc: the credentials require transport level security (use grpc.WithTransportCredentials() to set)

For some reason instead of failing gracefully because exporter must not be insecure as usual, this version of the collector is crashing now.

Please remove exporter insecure and try out.

Depechie · 2022-11-18T17:32:23Z

Well... I would love to try that. But on local docker container install, without insecure the stack does not work ☹️
I do not have a HTTPS cert available.

pavankrish123 · 2022-11-18T17:39:07Z

@Depechie please refer to this blog written by our friend @jpkrohling on how to create some dummy certs is TLS setup section. It's fairly easy with cfssl tool.

You see the collector refuses to connect over plain text channel.

Depechie · 2022-11-18T17:40:09Z

Ok will try that next week :) thx for the details!!

Depechie · 2022-11-21T07:56:10Z

Any guidance on how to get this done with docker compose?
No matter what I try for endpoint setting in the otel agent yaml, I keep getting

Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4317: connect: connection refused"	{"grpc_log": true}

And sometimes

Err: connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

I tried:

endpoint: otel-server:4317
endpoint: 0.0.0.0:4317
endpoint: localhost:4317
endpoint: https://host.docker.internal:4317

jpkrohling · 2022-11-23T17:29:16Z

@pavankrish123, I don't think we should be crashing the collector when the "tls.insecure" option is set, even if the transport requires confidentiality. Given you debugged this already, would you be OK with opening a PR fixing that? I can then review the PR.

Depechie · 2022-11-23T17:53:05Z

@jpkrohling and @pavankrish123 is there any possibility you are able to help out with that local sample I'm trying to work out? In the meantime?
The @jpkrohling example blog post does a local install of OpenTelemetry outside docker it seems?

If not no worries...

pavankrish123 · 2022-11-23T18:53:15Z

Thanks @jpkrohling - will issue a PR soon. Looks like a regression, the fix probably is in the core. Also can we please remove oauth2extension from the labels.

So here is what is happening setting tls.insecure on OTEL exporter usually lead to an error on OTEL collector and collector used to die of gracefully.

Logs from 0.62.0 version

otel-agent    | 2022-11-18T17:00:12.673Z	info	service/service.go:112	Starting otelcol-contrib...	{"Version": "0.60.0", "NumCPU": 8}
otel-agent    | 2022-11-18T17:00:12.673Z	info	extensions/extensions.go:42	Starting extensions...
otel-agent    | 2022-11-18T17:00:12.673Z	info	extensions/extensions.go:45	Extension is starting...	{"kind": "extension", "name": "oauth2client"}
otel-agent    | 2022-11-18T17:00:12.673Z	info	extensions/extensions.go:49	Extension started.	{"kind": "extension", "name": "oauth2client"}
otel-agent    | 2022-11-18T17:00:12.673Z	info	pipelines/pipelines.go:74	Starting exporters...
otel-agent    | 2022-11-18T17:00:12.673Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlp/auth"}
otel-agent    | 2022-11-18T17:00:12.673Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel created	{"grpc_log": true}
otel-agent    | 2022-11-18T17:00:12.673Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel Connectivity change to SHUTDOWN	{"grpc_log": true}
otel-agent    | 2022-11-18T17:00:12.673Z	info	zapgrpc/zapgrpc.go:174	[core] [Channel open-telemetry/opentelemetry-collector-contrib#1] Channel deleted	{"grpc_log": true}
otel-agent    | Error: cannot start pipelines: grpc: the credentials require transport level security (use grpc.WithTransportCredentials() to set)
otel-agent    | 2022/11/18 17:00:12 collector server run finished with error: cannot start pipelines: grpc: the credentials require transport level security (use grpc.WithTransportCredentials() to set)

For some reason instead of failing gracefully because exporter must not be insecure as usual, this version of the collector is crashing now.

pavankrish123 · 2022-11-23T18:57:43Z

@jpkrohling and @pavankrish123 is there any possibility you are able to help out with that local sample I'm trying to work out? In the meantime? The @jpkrohling example blog post does a local install of OpenTelemetry outside docker it seems?

If not no worries...

Will try few things on my end and get back to you soon @Depechie. Been busy last couple of days.

jpkrohling · 2022-11-23T19:39:46Z

is there any possibility you are able to help out with that local sample I'm trying to work out?

I'm not familiar with Docker compose enough to be helpful. If you can reproduce the problem using your local machine instead, I can probably help you.

jpkrohling · 2022-11-23T19:42:09Z

I moved this to the collector core repository. I don't think this warrants a patch release, as this is triggered only on invalid configuration options.

pavankrish123 · 2022-11-29T16:47:23Z

The issue is now fixed. We can close this.

Depechie added the bug Something isn't working label Nov 17, 2022

Depechie changed the title ~~Unable to start agent with oauth2client extension~~ OpenTelemetry Collector keeps crashing when authentication is configured as extension Nov 18, 2022

jpkrohling changed the title ~~OpenTelemetry Collector keeps crashing when authentication is configured as extension~~ Regression: crash when TLS insecure is set on authenticators Nov 23, 2022

jpkrohling transferred this issue from open-telemetry/opentelemetry-collector-contrib Nov 23, 2022

jpkrohling added area:extension auth Authentication related labels Nov 23, 2022

This was referenced Nov 29, 2022

Panic on collector start when using otlp exporter with tls: insecure: true and headers_setter extension open-telemetry/opentelemetry-collector-contrib#16508

Closed

Fix nil panic from otlp exporter #6633

Merged

jpkrohling closed this as completed Nov 29, 2022

frzifus mentioned this issue Jan 12, 2023

Question on basicauth extension usage open-telemetry/opentelemetry-collector-contrib#17532

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression: crash when TLS insecure is set on authenticators #6619

Regression: crash when TLS insecure is set on authenticators #6619

Depechie commented Nov 17, 2022 •

edited by jpkrohling

Loading

github-actions bot commented Nov 17, 2022

pavankrish123 commented Nov 17, 2022 •

edited

Loading

Depechie commented Nov 17, 2022

Depechie commented Nov 18, 2022 •

edited

Loading

pavankrish123 commented Nov 18, 2022

Depechie commented Nov 18, 2022

Depechie commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022

Depechie commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022 •

edited

Loading

Depechie commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022

Depechie commented Nov 18, 2022

Depechie commented Nov 21, 2022 •

edited

Loading

jpkrohling commented Nov 23, 2022

Depechie commented Nov 23, 2022

pavankrish123 commented Nov 23, 2022 •

edited

Loading

pavankrish123 commented Nov 23, 2022

jpkrohling commented Nov 23, 2022

jpkrohling commented Nov 23, 2022

pavankrish123 commented Nov 29, 2022 •

edited

Loading

Regression: crash when TLS insecure is set on authenticators #6619

Regression: crash when TLS insecure is set on authenticators #6619

Comments

Depechie commented Nov 17, 2022 • edited by jpkrohling Loading

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Nov 17, 2022

pavankrish123 commented Nov 17, 2022 • edited Loading

Depechie commented Nov 17, 2022

Depechie commented Nov 18, 2022 • edited Loading

pavankrish123 commented Nov 18, 2022

Depechie commented Nov 18, 2022

Depechie commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022

Depechie commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022 • edited Loading

Depechie commented Nov 18, 2022

pavankrish123 commented Nov 18, 2022

Depechie commented Nov 18, 2022

Depechie commented Nov 21, 2022 • edited Loading

jpkrohling commented Nov 23, 2022

Depechie commented Nov 23, 2022

pavankrish123 commented Nov 23, 2022 • edited Loading

pavankrish123 commented Nov 23, 2022

jpkrohling commented Nov 23, 2022

jpkrohling commented Nov 23, 2022

pavankrish123 commented Nov 29, 2022 • edited Loading

Depechie commented Nov 17, 2022 •

edited by jpkrohling

Loading

pavankrish123 commented Nov 17, 2022 •

edited

Loading

Depechie commented Nov 18, 2022 •

edited

Loading

pavankrish123 commented Nov 18, 2022 •

edited

Loading

Depechie commented Nov 21, 2022 •

edited

Loading

pavankrish123 commented Nov 23, 2022 •

edited

Loading

pavankrish123 commented Nov 29, 2022 •

edited

Loading