Make reporting more metrics for tasks possible #3343
Conversation
| @@ -66,6 +66,9 @@ def handle_task_disabled(self, task, config): | |||
| def handle_task_done(self, task): | |||
| pass | |||
|
|
|||
| def handle_task_statistics(self, task, statistics): | |||
| pass | |||
There was a problem hiding this comment.
Is there a common implementation that can be introduced here? Or within the Datadog and/or Prometheus implementations of MetricsCollector?
If not, and this method needs to remain as an optionally-implemented method, i would ask at a minimum for there to be appropriately informative and discoverable documentation written surrounding its intended use. (I don't believe there are currently any docs/ content around metrics collectors).
There was a problem hiding this comment.
This method is entirely optional. I had not provided any documentation and tests yet, because I was hoping to gather some feedback in #3342 as you suggested in my earlier pull request (#3333 (comment)).
I will gladly add tests and documentation if you are convinced this change is small enough.
I would like to collect and monitor more metrics from luigi graphs runs.
Description
I added method for passing tasks' run statistics from TaskContext into scheduler and metrics collector.
Motivation and Context
Example and motivation is in Issue #3342.
Have you tested this? If so, how?
I tested in on example graph and it worked.