[jaeger-v2] Map Collector Specific Flags To Corresponding OTEL Configurations

[mahadzaryab1]

Requirement

• Part of [jaeger-v2] Rethink components configuration to align with OTEL #5229
• We want to start by mapping the collector specific flags in Jaeger V1 to OTEL using the steps outlined in [jaeger-v2] Rethink components configuration to align with OTEL #5229 (comment) in a google document.
• If any flags aren't supported by OTEL, we'll discuss how to handle them in this issue.

[mahadzaryab1]

After running SPAN_STORAGE_TYPE=memory go run ./cmd/collector help, these are the collector specific flags that I'll be mapping to the corresponding OTEL configs.

 --collector.enable-span-size-metrics                        Enables metrics based on processed span size, which are more expensive to calculate.
      --collector.grpc-server.host-port string                    The host:port (e.g. 127.0.0.1:12345 or :12345) of the collector's gRPC server (default ":14250")
      --collector.grpc-server.max-connection-age duration         The maximum amount of time a connection may exist. Set this value to a few seconds or minutes on highly elastic environments, so that clients discover new collector nodes frequently. See https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters (default 0s)
      --collector.grpc-server.max-connection-age-grace duration   The additive period after MaxConnectionAge after which the connection will be forcibly closed. See https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters (default 0s)
      --collector.grpc-server.max-message-size int                The maximum receivable message size for the collector's gRPC server (default 4194304)
      --collector.grpc.tls.cert string                            Path to a TLS Certificate file, used to identify this server to clients
      --collector.grpc.tls.cipher-suites string                   Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
      --collector.grpc.tls.client-ca string                       Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
      --collector.grpc.tls.enabled                                Enable TLS on the server
      --collector.grpc.tls.key string                             Path to a TLS Private Key file, used to identify this server to clients
      --collector.grpc.tls.max-version string                     Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.grpc.tls.min-version string                     Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.http-server.host-port string                    The host:port (e.g. 127.0.0.1:12345 or :12345) of the collector's HTTP server (default ":14268")
      --collector.http-server.idle-timeout duration               See https://pkg.go.dev/net/http#Server (default 0s)
      --collector.http-server.read-header-timeout duration        See https://pkg.go.dev/net/http#Server (default 2s)
      --collector.http-server.read-timeout duration               See https://pkg.go.dev/net/http#Server (default 0s)
      --collector.http.tls.cert string                            Path to a TLS Certificate file, used to identify this server to clients
      --collector.http.tls.cipher-suites string                   Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
      --collector.http.tls.client-ca string                       Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
      --collector.http.tls.enabled                                Enable TLS on the server
      --collector.http.tls.key string                             Path to a TLS Private Key file, used to identify this server to clients
      --collector.http.tls.max-version string                     Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.http.tls.min-version string                     Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.num-workers int                                 The number of workers pulling items from the queue (default 50)
      --collector.otlp.enabled                                    Enables OpenTelemetry OTLP receiver on dedicated HTTP and gRPC ports (default true)
      --collector.otlp.grpc.host-port string                      The host:port (e.g. 127.0.0.1:12345 or :12345) of the collector's gRPC server (default ":4317")
      --collector.otlp.grpc.max-connection-age duration           The maximum amount of time a connection may exist. Set this value to a few seconds or minutes on highly elastic environments, so that clients discover new collector nodes frequently. See https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters (default 0s)
      --collector.otlp.grpc.max-connection-age-grace duration     The additive period after MaxConnectionAge after which the connection will be forcibly closed. See https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters (default 0s)
      --collector.otlp.grpc.max-message-size int                  The maximum receivable message size for the collector's gRPC server (default 4194304)
      --collector.otlp.grpc.tls.cert string                       Path to a TLS Certificate file, used to identify this server to clients
      --collector.otlp.grpc.tls.cipher-suites string              Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
      --collector.otlp.grpc.tls.client-ca string                  Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
      --collector.otlp.grpc.tls.enabled                           Enable TLS on the server
      --collector.otlp.grpc.tls.key string                        Path to a TLS Private Key file, used to identify this server to clients
      --collector.otlp.grpc.tls.max-version string                Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.otlp.grpc.tls.min-version string                Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.otlp.grpc.tls.reload-interval duration          The duration after which the certificate will be reloaded (0s means will not be reloaded) (default 0s)
      --collector.otlp.http.cors.allowed-headers string           Comma-separated CORS allowed headers. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers
      --collector.otlp.http.cors.allowed-origins string           Comma-separated CORS allowed origins. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin
      --collector.otlp.http.host-port string                      The host:port (e.g. 127.0.0.1:12345 or :12345) of the collector's HTTP server (default ":4318")
      --collector.otlp.http.idle-timeout duration                 See https://pkg.go.dev/net/http#Server (default 0s)
      --collector.otlp.http.read-header-timeout duration          See https://pkg.go.dev/net/http#Server (default 2s)
      --collector.otlp.http.read-timeout duration                 See https://pkg.go.dev/net/http#Server (default 0s)
      --collector.otlp.http.tls.cert string                       Path to a TLS Certificate file, used to identify this server to clients
      --collector.otlp.http.tls.cipher-suites string              Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
      --collector.otlp.http.tls.client-ca string                  Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
      --collector.otlp.http.tls.enabled                           Enable TLS on the server
      --collector.otlp.http.tls.key string                        Path to a TLS Private Key file, used to identify this server to clients
      --collector.otlp.http.tls.max-version string                Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.otlp.http.tls.min-version string                Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.otlp.http.tls.reload-interval duration          The duration after which the certificate will be reloaded (0s means will not be reloaded) (default 0s)
      --collector.queue-size int                                  The queue size of the collector (default 2000)
      --collector.queue-size-memory uint                          (experimental) The max memory size in MiB to use for the dynamic queue.
      --collector.tags string                                     One or more tags to be added to the Process tags of all spans passing through this collector. Ex: key1=value1,key2=${envVar:defaultValue}
      --collector.zipkin.cors.allowed-headers string              Comma-separated CORS allowed headers. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers
      --collector.zipkin.cors.allowed-origins string              Comma-separated CORS allowed origins. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin
      --collector.zipkin.host-port string                         The host:port (e.g. 127.0.0.1:9411 or :9411) of the collector's Zipkin server (disabled by default)
      --collector.zipkin.keep-alive                               KeepAlive configures allow Keep-Alive for Zipkin HTTP server (enabled by default) (default true)
      --collector.zipkin.tls.cert string                          Path to a TLS Certificate file, used to identify this server to clients
      --collector.zipkin.tls.cipher-suites string                 Comma-separated list of cipher suites for the server, values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
      --collector.zipkin.tls.client-ca string                     Path to a TLS CA (Certification Authority) file used to verify certificates presented by clients (if unset, all clients are permitted)
      --collector.zipkin.tls.enabled                              Enable TLS on the server
      --collector.zipkin.tls.key string                           Path to a TLS Private Key file, used to identify this server to clients
      --collector.zipkin.tls.max-version string                   Maximum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)
      --collector.zipkin.tls.min-version string                   Minimum TLS version supported (Possible values: 1.0, 1.1, 1.2, 1.3)

[yurishkuro]

Most of these flags are for controlling server endpoints (which we should still document).

The unique ones are these:

• --collector.num-workers
• --collector.queue-size
• --collector.queue-size-memory
• --collector.tags

We should look how they map into various standard components in OTEL, like the batch processor, the attributes processor (which may not be even included in the binary yet).

[mahadzaryab1]

@yurishkuro couple of questions:

• For --collector.queue-size, can we use map it to the send_batch_size in the batchprocessor?
• For --collector.queue-size-memory, can we use the memorylimiterprocessor, specifically the limit_mib field?
• For --collector.tags, can we use the attributesprocessor where each attribute's key and value match would be passed into --collector.tags and action is insert?

[yurishkuro]

• No,
• maybe,
• probably

[yurishkuro]

It's ok to say in the doc if there is no equivalent

[mahadzaryab1]

@yurishkuro Sounds good! What's the reason the first one is a no? And do you know of any equivalent to --collector.queue-size? I haven't been able to find one so far.

[yurishkuro]

because Jaeger queue just keeps items in memory until they are processed, and it does not batch spans. It's actually not a great design for resiliency, it's better to have back pressure to the clients.

[mahadzaryab1]

got it! couple more questions:

• do you know how we should handle --collector.enable-span-size-metrics? is this part of the spanmetricsconnector?
• is there no alternative for --collector.queue-size then?
• how do we handle the zipkin flags?

[yurishkuro]

• it's not. I'd say not supported
• there may be, I think OTEL batch helper can enable a queue
• zipkin receiver should have equivalent flags

There should also be a collector flag for concurrency (number of processors), we should find out what the equivalent for that is in OTEL.

[mahadzaryab1]

@yurishkuro I only see queues in the exporterhelper. Is this what we're looking for?

[mahadzaryab1]

@yurishkuro The migration document is mostly complete. Let me know if you have any feedback. I'm also looking for some guidance on how to proceed with --collector.num-workers, --collector.queue-size, and --collector.queue-size-memory.

[yurishkuro]

I don't think we use exporter helper when exporting to storage - it's worth having a look into what features it would bring.

[yurishkuro]

Re num workers - I actually don't know what threading model OTEL uses, we should find out.

[mahadzaryab1]

@yurishkuro how would you like me to proceed regarding the above? would you like me to add some documentation about the exporter helper to the document? how can I find out what threading model OTEL uses?

[mahadzaryab1]

@yurishkuro any other action items here? looks like we still don't have a mapping for the following flags

--collector.num-workers 
--collector.queue-size
--collector.queue-size-memory

Any thoughts on how to proceed?

[yurishkuro]

• --collector.num-workers - I don't see a direct equivalent. gRPC creates new goroutine for each incoming request, there is no "worker pool" that can control concurrency (there is a MaxConcurrentStreams setting, but it's only for streaming requests, not unary). I think we should leave it as unsupported.
• --collector.queue-size and --collector.queue-size-memory - there is no queue by default. We could consider using exporterhelper in the jaeger_storage_exporter, which will provide these capabilities.

[mahadzaryab1]

@yurishkuro we currently have queuing disabled here. The exporterhelper queue has a queue size field in the configuration (see https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queue_sender.go#L34). Do we want to expose this in the storage exporter. There isn't anything here to configure memory though.

[yurishkuro]

yeah, I think we should enable all those features. Do they already have config structs exposed?

[mahadzaryab1]

@yurishkuro doesn't look like it - see https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/internal/exporters/storageexporter/config.go#L17-L19. Should we embed the QueueConfig struct there? Also, are we not going to support the max memory? There's also the memorylimiterprocessor.

[mahadzaryab1]

@yurishkuro Opened at PR at #6080.

[yurishkuro]

Also, are we not going to support the max memory?

no. It wasn't well supported in v1 either.

[mahadzaryab1]

@yurishkuro Any other items we want to tackle as part of this issue? If not, we can close it out.