Having Upgrade Matrix Failing for Duplicate Suffixes

[AdrianoDee]

PR description:

This PR proposes to have the upgrade matrix building fail if duplicate suffixes for workflows are found. Currently it fails only if duplicate offsets are found and if, by error, we have two workflows with the same suffix it silently overwrites the first one with the second.

As is the PR should cause a failure just to prove the mechanism is working.

PR validation:

runTheMatrix -n -w upgrade fails. runs.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45715/41355

• There are other open Pull requests which might conflict with changes you have proposed:
  • File Configuration/PyReleaseValidation/python/MatrixUtil.py modified in PR(s): 2024 Data RelVals and InputInfo Events Skimming (actually) Working #45055
  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, CMSSW Integration of LST #45117, TICLv5 : Superclustering DNN #45333, Clean up Phase-2 Geometry D86, D88, D91, D92, D93, D94, D97 #45370, Allowing Jumping BPix2->FPix2 For Pixel Doublets (and Alpaka Pixel Triplets *.406 wf) #45478, Update Pixel GPU DQM online client #45666, Add Phase2 PbPb reco process modifier #45683, Update to relval_gpu.pyfor Alpaka and Pixel Triplets Alpaka Workflows #45694

[cmsbuild]

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@AdrianoDee, @cmsbuild, @kskovpen, @miquork, @srimanob, @subirsarkar, @sunilUIET can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @slomeo this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[AdrianoDee]

please test

[smuzaffar]

@AdrianoDee , currently this breaks the bot code which tries to run the following to get the list of all step1 workflows

> runTheMatrix.py -n -e
processing relval_standard
ERROR importing file  relval_standard Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_highstats
ERROR importing file  relval_highstats Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_pileup
ERROR importing file  relval_pileup Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_generator
ERROR importing file  relval_generator Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_extendedgen
ERROR importing file  relval_extendedgen Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_production
ERROR importing file  relval_production Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_ged
ERROR importing file  relval_ged Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
ignoring relval_upgrade from default matrix
ignoring relval_cleanedupgrade from default matrix
ignoring relval_gpu from default matrix
processing relval_2017
ERROR importing file  relval_2017 Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_2026
ERROR importing file  relval_2026 Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
ignoring relval_identity from default matrix
processing relval_machine
ERROR importing file  relval_machine Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_premix
ERROR importing file  relval_premix Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU
processing relval_nano
ERROR importing file  relval_nano Duplicate special workflow suffixes not allowed: _Patatrack_PixelOnlyCPU

found a total of  0  workflows:


 -------------------------------------------------------------------------------- 

[AdrianoDee]

Perfect that was the idea. Let me remove the bug.

[AdrianoDee]

please abort

[AdrianoDee]

please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45715/41360

• There are other open Pull requests which might conflict with changes you have proposed:
  • File Configuration/PyReleaseValidation/python/MatrixUtil.py modified in PR(s): 2024 Data RelVals and InputInfo Events Skimming (actually) Working #45055
  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Parametrized magnetic field in alpaka #44360, CMSSW Integration of LST #45117, TICLv5 : Superclustering DNN #45333, Clean up Phase-2 Geometry D86, D88, D91, D92, D93, D94, D97 #45370, Allowing Jumping BPix2->FPix2 For Pixel Doublets (and Alpaka Pixel Triplets *.406 wf) #45478, Update Pixel GPU DQM online client #45666, Add Phase2 PbPb reco process modifier #45683, Update to relval_gpu.pyfor Alpaka and Pixel Triplets Alpaka Workflows #45694

[cmsbuild]

Pull request #45715 was updated. @AdrianoDee, @kskovpen, @miquork, @srimanob, @subirsarkar, @sunilUIET can you please check and sign again.

[cmsbuild]

+1

Size: This PR adds an extra 36KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-81d63a/40962/summary.html
COMMIT: 18c29a6
CMSSW: CMSSW_14_1_X_2024-08-15-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/45715/40962/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 1 lines from the logs
• Reco comparison results: 80 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3422510
• DQMHistoTests: Total failures: 4598
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3417892
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
• Checked 196 log files, 165 edm output root files, 45 DQM output files
• TriggerResults: no differences found

[AdrianoDee]

+pdmv

[srimanob]

+Upgrade

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1