Cannot add custom versions of existing system metrics

[msiebuhr]

I'm writing a proxy that ingests Heroku router/runtime metrics logs and converts it to prometheus-ingestable metrics. (See my fork of lumbermill.)

I thought it'd be nice to output metrics in the same form as the ones used elsewhere in Prometheus, ex http_request_size_bytes with the relevant upstream job, instance and endpoint tags. (Though ideally I'd like to include HTTP Method and response code).

Trying to register a handler for this:

httpResponseSizeBytes = prometheus.NewSummaryVec(prometheus.SummaryOpts{
    Name: "http_request_size_bytes",
    Help: "The HTTP response sizes in bytes.",
}, []string {"job", "instance", "handler"})

func init () {
    prometheus.MustRegister(httpResponseSizeBytes)
}

Conflicts with the existing handler for the very same data:

panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "http_request_size_bytes", help: "The HTTP request sizes in bytes.", constLabels: {handler="prometheus"}, variableLabels: []} has different label names or a different help string

I've tried unregistering the existing SummaryVec, but to no avail:

func drain(...) {
    prometheus.Unregister(prometheus.NewSummary(prometheus.SummaryOpts{
        Name:        "http_response_size_bytes",
        Help: "The HTTP response sizes in bytes.",
        ConstLabels: prometheus.Labels{"handler": "prometheus"},
    }))
    prometheus.MustRegister(httpResponseSizeBytes)
}

Detects that a handler with a different set of labels has been there previously, and more or less gives me the same panic as above.

[brian-brazil]

I'd argue this is correct behaviour in this case. Exporting the same metric name with different semantics and labels will cause confusion for downstream users, so you should prefix your metrics to separate them.

[fabxc]

@beorn7 probably can you give the most elaborate answer on this and how this might be affected/resolved by his planned changed to client_golang.

Part of that includes switching the default HTTP handler instrumentation to histograms. So you might want to use histograms instead of summaries right away.

[beorn7]

@msiebuhr I'm working on a client_golang refactoring that will make everything way more easier to configure, but in this particular example, just use UninstrumentedHandler instead of Handler as the handler to register. In that case, no metric will be registered.

[beorn7]

https://github.com/msiebuhr/lumbermill/blob/prometheus/http.go#L51

That's where you need 'UninstrumentedHandler`.