Inaccurate data collected by prometheus receiver

[GlowingRuby]

Describe the bug
Use prometheus receiver to scrape node_exporter data and expose the data to native prometheus through prometheus exporter. The CPU usage rate obtained on the node exporter dashboard is inaccurate, which is much different from directly using prometheus to scrape node_exporter.

The flat part is to use native prometheus, and the severe jitter part is to use otelcol.
I make sure that my environment is stable and the phenomenon can be reproduced stably.

Steps to reproduce
Use prometheus receiver to scrape node_exporter data and expose the data to native prometheus through prometheus exporter.

What did you expect to see?
Correct cpu usage

What did you see instead?
Wrong cpu usage

What version did you use?
Version: 0.26.0

What config did you use?
otelcol config:

receivers:
  otlp:
    protocols:
      grpc:

  prometheus:
      config:
        global:
          scrape_interval: 15s
        scrape_configs:
          #- job_name: 'prometheus'
          #  static_configs:
          #    - targets: ['192.168.101.209:9090']
          - job_name: 'node'
            static_configs:
              - targets: ['192.168.101.200:9100']
  prometheus:
    endpoint: "192.168.101.209:9111"
    send_timestamps: true
    resource_to_telemetry_conversion:
      enabled: true
  jaeger: 
    endpoint: "192.168.101.209:14250"
    insecure: true
processors: 
  batch:    
extensions:
  health_check:
  pprof:      
    endpoint: :1888
  zpages:
    endpoint: :55679
service: 
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces: 
      receivers: [otlp] 
      processors: [batch]
      exporters: [jaeger]
    metrics/1:
      receivers: [otlp,prometheus]
      processors: []
      exporters: [prometheus]

native prometheus config:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'otel-collector'
    honor_labels: true
    static_configs:
    - targets: ['192.168.101.209:9111']

Environment
OS: CentOS Linux release 7.4.1708 (Core)

[GlowingRuby]

The version of node_exporter is 1.0.1.linux-amd64, and the version of prometheus is 2.25.0.linux-amd64.

In addition, while investigating this problem, I found another interesting problem. Otelcol seems to discard some samples.

Link to similar issue: https://github.com/open-telemetry/opentelemetry-collector/issues/3118

[bogdandrutu]

Can you give a try to head? #3047 may fix your main issue

[GlowingRuby]

Can you give a try to head? #3047 may fix your main issue

@bogdandrutu sure,reply later.

[GlowingRuby]

Are there any configuration item changes? The package I recompiled from the head does not take effect