Configure prometheus receiver garbage collection interval

[courageJ]

Is your feature request related to a problem? Please describe.
For prometheus receivers which disable UseStartTimeMetric to scrape endpoints sampling metrics every 2+ minutes, the timeseries are always reset as the initial, and no metrics will be passed to nextConsumer.

Describe the solution you'd like
Provide an option in prometheus receiver to configure the gc interval

[courageJ]

@dashpole For visibility

[dashpole]

open-telemetry/opentelemetry-collector#3047 might have changed this behavior recently.

[courageJ]

With open-telemetry/opentelemetry-collector#3047, the first timeseries will not be dropped. The remaining issue is for metrics scraped every 2+ minutes, the starttime is always adjusted even if the process is not restarted.

[dashpole]

Should the garbage collection interval be based on the scrape interval?

[rakyll]

If we decide to do anything here, we shouldn't do it in the Prometheus receiver.

@tigrannajaryan @mxiamxia Maybe this is another case for merging the ballast extension + memory limiter. Then, we can add gc tuning options. It seems like we need a global memory management component.

[Aneurysm9]

If we decide to do anything here, we shouldn't do it in the Prometheus receiver.

@tigrannajaryan @mxiamxia Maybe this is another case for merging the ballast extension + memory limiter. Then, we can add gc tuning options. It seems like we need a global memory management component.

I think this is referring to the GC implemented inside the prometheus receiver for the data used to determine start times.

Should the garbage collection interval be based on the scrape interval?

The scrape interval can be set per-job, right? There is only one GC interval that cleans the JobMap for all jobs. Maybe it could be set to the max scrape interval.

[dashpole]

Maybe it could be set to the max scrape interval.

Yeah, thats what I had in mind.

[rakyll]

@Aneurysm9 My bad, thanks for the clarification.