Redesign of the association maps, multivector manager, HGCAL Rechits and Validation with significant speedup of Phase-2 workflows #45865
Conversation
|
cms-bot internal usage |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45865/41614
|
|
A new Pull Request was created by @felicepantaleo for master. It involves the following packages:
@Martin-Grunewald, @antoniovagnerini, @civanch, @cmsbuild, @jfernan2, @mandrenguyen, @mdhildreth, @mmusich, @nothingface0, @rvenditti, @srimanob, @subirsarkar, @syuvivida, @tjavaid can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
test parameters:
|
|
@cmsbuild please test |
felicepantaleo
changed the title
|
-1 Failed Tests: ClangBuild Clang BuildI found compilation error while trying to compile with clang. Command used: >> Entering Package RecoLocalCalo/HGCalRecProducers >> Entering Package SimCalorimetry/HGCalAssociatorProducers >> Entering Package SimDataFormats/Associations >> Entering Package Validation/HGCalValidation >> Compile sequence completed for CMSSW CMSSW_14_2_X_2024-09-03-1100 gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1 + eval scram build outputlog '&&' '(python3' /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cms-bot/buildLogAnalyzer.py --logDir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_2_X_2024-09-03-1100/tmp/el8_amd64_gcc12/cache/log/src '||' 'true)' ++ scram build outputlog >> Entering Package CommonTools/RecoAlgos Entering library rule at src/CommonTools/RecoAlgos/plugins >> Compiling edm plugin src/CommonTools/RecoAlgos/plugins/BasicClusterMerger.cc |
cmsbuild
added
tests-pending
code-checks-pending
and removed
tests-rejected
code-checks-approved
labels
|
You can find the full details of the performance comparison here: I'll add this link in the initial message as well. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45865/41617
|
|
enable hlt_p2_timing |
|
@cmsbuild, please test |
|
Based on the already available HLT phase-2 timing results, there is a visible reduction:
from 5437.9 ms/evt (baseline) to 5388.1 ms/evt (this PR) i.e. a -0.915% reduction in timing. |
|
+1 Size: This PR adds an extra 20KB to repository
Comparison SummarySummary:
|
no strong opinion, if that's fine with @cms-sw/reconstruction-l2 it's fine by me. OTOH I would reserve the option of running just the phase-2 HLT menu timing without the rest of the reco profiling. |
Honestly, I'd rather prefer to keep the two tests decoupled for the moment, we are refurbishing the profiling test at present due to igprof problem. Thanks |
|
@cms-sw/dqm-l2 let me know if you have any more comments. |
|
pinging again @cms-sw/dqm-l2 @antoniovagnerini, please review / sign, thank you. |
| std::vector<dqm::reco::MonitorElement*> h_denom_caloparticle_eta[numberOfValidationTypes_]; | ||
| std::vector<dqm::reco::MonitorElement*> h_denom_caloparticle_phi[numberOfValidationTypes_]; | ||
| std::vector<dqm::reco::MonitorElement*> h_denom_caloparticle_en[numberOfValidationTypes_]; | ||
| std::vector<dqm::reco::MonitorElement*> h_denom_caloparticle_pt[numberOfValidationTypes_]; |
There was a problem hiding this comment.
can you confirm that the doubling in numberOfValidationTypes_ from 2 to 4 is behind the expected increase in DQM Histogram size by 2.7MB?
DQMHistoSizes: Histogram memory added: 2745.032 KiB( 44 files compared)
|
+1 |
cmsbuild
added
dqm-approved
fully-signed
and removed
dqm-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2) |
|
+1 |
PR description:
In the context of Next Generation Triggers and HGCAL TICL reconstruction, fast AssociatorMaps between Reco and Sim, Sim to Sim, and Reco to Reco are required in order to develop new reconstruction algorithms and study their performance. They have also been included in the HGCalValidator and in the TICLDumper, but can be extended to many other detectors (MTD for instance, as it is the new main offender).
The HGCAL Rechit producer was optimized by avoiding calling virtual functions for every rechit and to enable support for the MultiVectorManager.
The MultiVectorManager was optimized and together with the HGCAL RecHit map was used to optimize the E/Gamma reconstruction and the associators.
The HGCAL Validation was redesigned to make effective usage of the new associators. The framework was completely redesigned to generate automatically validation plots for any new collection of TICL tracksters against SimTracksters from Simclusters and CaloParticles.
Performance measurements:
Performed with
14_1_0_pre7with TTbar PU200, 10 events, single thread, single stream14_1_0_pre7:
14_1_0_pre7+PR
Performance comparison
Full performance comparison table here:
https://docs.google.com/spreadsheets/d/1vdmDDJJ07tf9ekQ8EC24sMqqvtCtiU2Mh6FawsIRu24/edit?usp=sharing
Notable numbers:
Overall Phase-2 TTbar PU200 Reconstruction+Validation workflows: speedup 313% , allocated memory -38%
Overall E/Gamma Phase-2 Reconstruction: 1.5x speedup
ElectronSeedProducer ecalDrivenElectronSeeds: 3x speedup
HGCalValidator: 30x speedup
New associators: between two orders of magnitude and infinite speedup depending if you want to run on the CaloParticles from pileup as well. In this case the associators in release will crash for the amount of resources needed.
Job Memory usage step4: -22%
@cms-sw/hgcal-dpg-l2 @cms-sw/egamma-pog-l2 @waredjeb @AuroraPerego @rovere