Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Target Allocator not able to allocate #23342

Closed
achetronic opened this issue Jun 13, 2023 · 2 comments
Closed

Target Allocator not able to allocate #23342

achetronic opened this issue Jun 13, 2023 · 2 comments
Labels
bug Something isn't working needs triage New item requiring triage receiver/prometheus Prometheus receiver

Comments

@achetronic
Copy link

Component(s)

receiver/prometheus

What happened?

Description

Configured OpenTelemetryCollector CR in Kubernetes, and a ServiceMonitor just for testing. The TargetAllocator is not able to get the config because some empty parameter, but all of them are filled.

The logs from target allocator:

{"level":"info","ts":"2023-06-13T15:53:30Z","logger":"allocator","msg":"Successfully started a collector pod watcher","component":"opentelemetry-targetallocator"}
{"level":"error","ts":"2023-06-13T15:53:48Z","logger":"setup","msg":"Unable to load configuration","error":"empty duration string","stacktrace":"main.main.func13\n\t/app/main.go:198\ngithub.jparrowsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38"}
{"level":"error","ts":"2023-06-13T15:58:48Z","logger":"setup","msg":"Unable to load configuration","error":"empty duration string","stacktrace":"main.main.func13\n\t/app/main.go:198\ngithub.jparrowsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38"}

The configuration for the collector:

    processors:
        memory_limiter:
          check_interval: 1s
          limit_percentage: 70
          spike_limit_percentage: 25

        batch:
          send_batch_size: 500
          send_batch_max_size: 500
          timeout: 1s

      receivers:
        otlp:
          protocols:
            grpc:
            http:
        prometheus:
          config:
            global:
              scrape_interval: 2m
              scrape_timeout: 10s
              evaluation_interval: 30s
            scrape_configs:
              - job_name: dummy
                static_configs:
                  - targets:
                      - 127.0.0.1:8888

      exporters:
        logging:
          verbosity: detailed
  
        prometheus:
          endpoint: "0.0.0.0:9090"
          metric_expiration: 5m
          resource_to_telemetry_conversion:
            enabled: true
  
        prometheusremotewrite:
          endpoint: https://placeholder/api/v1/push
          timeout: 15s
          retry_on_failure:
            enabled: true
            initial_interval: 5s
            max_interval: 10s
            max_elapsed_time: 30s
          external_labels:
            cluster: ${EXTERNAL_LABEL_CLUSTER}
            region: ${EXTERNAL_LABEL_REGION}

      service:
        telemetry:
          logs:
            level: "debug"
        pipelines:
          logs:
            exporters: [otlp/newrelic]
            processors: [memory_limiter, batch]
            receivers: [otlp]
  
          metrics:
            exporters: [prometheusremotewrite]
            processors: [memory_limiter, batch]
            receivers: [otlp, prometheus]
  
          traces:
            exporters: [otlp/newrelic]
            processors: [memory_limiter, batch]
            receivers: [otlp]

with targetAllocator field inside OpenTelemetryCollector CR:

targetAllocator:
    enabled: true
    prometheusCR:
      enabled: true
    serviceAccount: opentelemetry-collector-global

The following piece of config is added to avoid another bug about empty scrape_configs better described here

scrape_configs:
    - job_name: dummy
      static_configs:
        - targets:
            - 127.0.0.1:8888

All the permissions have been granted to the ClusterRole used by the target allocator according to the guide you provided

Expected Result

Expected to have ServiceMonitor resources just scrapped and imported

Actual Result

ServiceMonitor resources are not even get

Collector version

0.79.0

Environment information

Environment

OS: Kubernetes 1.23

OpenTelemetry Collector configuration

provided on top

Log output

{"level":"info","ts":"2023-06-13T15:53:30Z","logger":"allocator","msg":"Successfully started a collector pod watcher","component":"opentelemetry-targetallocator"}
{"level":"error","ts":"2023-06-13T15:53:48Z","logger":"setup","msg":"Unable to load configuration","error":"empty duration string","stacktrace":"main.main.func13\n\t/app/main.go:198\ngithub.jparrowsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38"}
{"level":"error","ts":"2023-06-13T15:58:48Z","logger":"setup","msg":"Unable to load configuration","error":"empty duration string","stacktrace":"main.main.func13\n\t/app/main.go:198\ngithub.jparrowsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38"}
{"level":"error","ts":"2023-06-13T16:03:48Z","logger":"setup","msg":"Unable to load configuration","error":"empty duration string","stacktrace":"main.main.func13\n\t/app/main.go:198\ngithub.jparrowsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38"}
{"level":"info","ts":"2023-06-13T16:08:30Z","logger":"allocator","msg":"Successfully started a collector pod watcher","component":"opentelemetry-targetallocator"}
{"level":"error","ts":"2023-06-13T16:08:48Z","logger":"setup","msg":"Unable to load configuration","error":"empty duration string","stacktrace":"main.main.func13\n\t/app/main.go:198\ngithub.jparrowsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38"}

Additional context

No response

@achetronic achetronic added bug Something isn't working needs triage New item requiring triage labels Jun 13, 2023
@github-actions github-actions bot added the receiver/prometheus Prometheus receiver label Jun 13, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@achetronic
Copy link
Author

Hey, seems to be related to this issue:
open-telemetry/opentelemetry-operator#1811

So let's wait

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage New item requiring triage receiver/prometheus Prometheus receiver
Projects
None yet
Development

No branches or pull requests

1 participant