metrics expiry

[zerthimon]

It'd be nice if there was a configuration parameter to specify how long the metric should "live" inside the pushgateway and then expie.

[brian-brazil]

This has been discussed previously, we will not be adding this feature. See #19

[zerthimon]

So, I was hoping I could alert on cron-based scripts used to backup stuff off my servers to a remote storage. I made each script send it's exit status to prometheus-pushgateway. It works well - I defined alert rules for a non-zero exit status. Now, how do I alert if the scripts crash in the middle (or don't run at all) and therefore dont send their exit code ? The pushgateway reports the last status forever, so I'll never know there is a problem with the script.

[zerthimon]

P.S. I read the thread #19 Is my solution for this use case an antipattern ? Why ? What's the prometheus-based best practice approach to monitoring cron script results ?

[brian-brazil]

http://www.robustperception.io/monitoring-batch-jobs-in-python/ explains how to do this from Python.

[zerthimon]

Ok, understood, additional metric for time of last success. Thanks @brian-brazil