Transparent poller to port mapping service.

[ebusto]

We have an existing Prometheus instance that uses the HTTP service discovery mechanism to retrieve a list of targets from a custom service. This service exposes the data in our asset inventory, NetBox, and includes a rich set of labels in the target configuration.

This works extremely well for platforms that natively run an exporter, as well as platforms that require metrics be collected through an external service, such as power metrics from PDUs exposed over SNMP using the SNMP exporter. In the latter case, a little relabel magic handles fixing up the URL that Prometheus scrapes from, so ultimately it is something like https://prometheus-snmp-exporter.foo.com/snmp?target=<target.fqdn>.

We've been trying to figure out how to integrate Harvest into this approach, and the sticking point is that Harvest doesn't appear to expose a way to retrieve metrics given a poller name.

The obvious approach is to implement a proxy service, which can take the poller name as a parameter, and returns the latest metrics, without the caller having to be aware of the port mapping that needs to occur behind the scenes.

Although we use HTTP service discovery, this approach would be agnostic to the service discovery mechanism.

[cgrinds]

hi @ebusto like many, we were excited when the Prometheus HTTP service discovery PR dropped earlier this year and we've discussed adding support for it. We also have Consul service discovery in the backlog, but HTTP is a better fit for most of our customers since it's simpler and doesn't require more software.

As you've noted, the poller processes are independent and know nothing about each other. They know their name, config, promPort, collectors, etc. but nothing about the other running pollers. That's by design.

We've discussed adding a Harvest management node, harvestctl, which would be one way to address your ask, both from a naming service perspective, but also from a service discovery point of view, since we would implement the HTTP service discovery endpoint in harvestctl.

In general, a management node would make talking to all pollers easier, but comes with more operational and code complexity, is a SPOF, pollers need to handle check-in/retry, etc. All doable of course, but a bigger change.

Let's talk workarounds - I know of at least one customer who achieved what you're asking by extracting the name to port mapping from the harvest.ymlfile and another who extracted the mapping from /proc.

[cgrinds]

hi @ebusto we've added support for Prometheus HTTP SD. When you get a chance, check it out and let us know if this meets your need.

This will be in 21.11 release targeted to ship next month, but of course is runnable in main today

[cgrinds]

Fixed by:

• feat: add a Prometheus HTTP service discovery end-point #575
• feat: http sd should include poller name #622

Please reopen if the addition of HTTP SD doesn't cover your need.