allow user-defined regular expressions

[rokroskar]

The default simple regex can be too restrictive -- for example, it makes it impossible to import any of the graphite-exported hadoop metrics because they start with a "." and contain standard hostnames.

Here I've modified the metric line parsing to allow for arbitrary user-defined regular expressions on lines that begin with ":userRegExp:" -- all other metric specifications are treated like before.

[brian-brazil]

Can you give an example of a metric that you're having problems with?

[rokroskar]

sure, for example: .yarn.QueueMetrics.Queue=root.Context=yarn.Hostname=node2.myhadoop.org.ActiveApplications

[rokroskar]

Now I process this metric like so:

:userRegExp: \.yarn\.QueueMetrics\.Queue=(.+)\.Context=(.+)\.Hostname=(.+)\.(.+)$
name="hadoop_yarn_queue"
queue="$1"
context="$2"
host="$3"
type="$4"

[brian-brazil]

That looks like it's coming from JMX, have you considered using the JMX exporter?

[rokroskar]

hmm that's a good point -- I have not since using graphite was path of least resistance for the moment (I had other metrics configured to export to graphite before I started using Prometheus)

[rokroskar]

Thanks for the suggestion to try JMX instead -- I've been playing around with it, but it seems to me that the functionality offered by exporting to Graphite is still preferred. For example, using JMX, I would have to configure one instance per node to scrape -- as far as I can tell there are no wildcards allowed in the target specification? Having to specify all the targets by hand for a whole hadoop cluster would be pretty cumbersome. I suppose it could be done using the custom service discovery functionality but then you still have to have an external script run to put all the ports and hostnames into a file.

In any case, having more flexibility in the regex used for the graphite_exporter is still desirable IMO. Why not use a .yml (or json) file like in jmx_exporter?

[brian-brazil]

as far as I can tell there are no wildcards allowed in the target specification?

This is correct, this is the recommended way to use Prometheus exporters as it's much easier to manage (no SPOFs, integrates better with configuration management).

The graphite and collectd exporters are only intended as temporary measures to help you transition over to full Prometheus monitoring. If they did have to become permanent for some reason (not likely in this case), the proper way to run them would be one per target.

I suppose it could be done using the custom service discovery functionality but then you still have to have an external script run to put all the ports and hostnames into a file.

If you don't already have infrastructure in place that's capable of doing this, I'd strongly recommend adding it independent of the rest of this discussion (http://www.robustperception.io/you-look-good-have-you-lost-machines/).

The core problem here is that something is putting metrics into Graphite in a way that doesn't follow the usual Graphite conventions and is also not escaped.

Why not use a .yml (or json) file like in jmx_exporter?

I'd like to keep things simple. Regexes are to some extent the domain of sysadmins, whereas globs are easier to understand for everyone.

[Aleksandr-Chernishev]

how can i map metric from hadoop hdfs like this:
(taken from graphite_exporter output)
\# HELP __dfs_datanode_Context_dfs_Hostname_nnode03_saas_rambler_tech_BlockChecksumOpAvgTime Graphite metric *.dfs.datanode.Context=dfs.Hostname=nnode03.saas.rambler.tech.BlockChecksumOpAvgTime \# TYPE __dfs_datanode_Context_dfs_Hostname_nnode03_saas_rambler_tech_BlockChecksumOpAvgTime gauge __dfs_datanode_Context_dfs_Hostname_nnode03_saas_rambler_tech_BlockChecksumOpAvgTime 0

[brian-brazil]

That also looks like JMX, use the JMX exporter for that.

It seems the use cases presented here are all already covered by the jmx exporter.

[Aleksandr-Chernishev]

i'm going to use many hadoop services on one server. One exporter for hadoop service, on some server will be 4 jmx exporters

[Aleksandr-Chernishev]

If cluster will be more 100 server, one graphite_exporter per node better that 2-4 jmx exporters

[brian-brazil]

The Prometheus architecture is one exporter per instance, so even if you were using the graphite exporter you'd still be running as many.

[Aleksandr-Chernishev]

graphite exporter listen one port, few services write in this port in graphite format. One graphite exporter per server

[rdsubhas]

(removed comment, not relevant to this PR, rather #21 )