Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow reporting timing data for several pipelines at once (from any Detectors or DistributionMaker instance) ... #822

Merged
merged 4 commits into from
Sep 2, 2024

Conversation

thehrh
Copy link
Contributor

@thehrh thehrh commented Aug 30, 2024

consistently propagate value of profile to any contained Pipeline instances and enable customisable number format (making use of the very powerful utils.format.format_num function).

Note that Pipeline.get_outputs(), Pipeline.setup(), and Pipeline.run() can now be separately profiled, so one does not have to manually add up the timings from the individual services within a pipeline and any overhead from computations in addition to Pipeline.run() becomes visible.

Example output from a call to new DistributionMaker.report_profile() with default args:

Pipeline: neutrinos
- setup:        Total time (s): 61.584, n calls: 1
- run:          Total time (s): 3.825, n calls: 2, time/call (s): mean 1.913, max. 3.535, min. 0.291
- get_outputs:  Total time (s): 4.080, n calls: 2, time/call (s): mean 2.040, max. 3.547, min. 0.533
Individual services:
data simple_data_loader
- setup:    Total time (s): 0.010, n calls: 1
- compute:  Total time (s): +0.000, n calls: 1
- apply:    Total time (s): 0.017, n calls: 2, time/call (s): mean 0.008, max. 0.011, min. 0.006

....

Pipeline: muons
- setup:        Total time (s): 0.006, n calls: 1
- run:          Total time (s): 2.621, n calls: 2, time/call (s): mean 1.310, max. 2.619, min. 0.001
- get_outputs:  Total time (s): 2.624, n calls: 2, time/call (s): mean 1.312, max. 2.622, min. 0.002
Individual services:
data simple_data_loader
- setup:    Total time (s): +0.000, n calls: 1
- compute:  Total time (s): +0.000, n calls: 1
- apply:    Total time (s): +0.000, n calls: 2, time/call (s): mean +0.000, max. +0.000, min. +0.000

...

Resolves #815

thehrh added 4 commits August 24, 2024 14:09
…use format_num for customisability of outputs; small test of correct propagation of profile from DistributionMaker to Pipeline
@JanWeldert JanWeldert merged commit 7753dea into icecube:master Sep 2, 2024
2 checks passed
@thehrh thehrh deleted the propagate_profile branch September 4, 2024 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants