[ASNEEDED_X] Multiple RelVals failing with SIGSEGV

[iarspider]

For example Relval 132.0 step 1

Not sure if the error is in Geant4 or Pythia6:

Thread 4 (Thread 0x1544f5f1d700 (LWP 474508) "cmsRun"):
#0  0x000015452848fac1 in poll () from /lib64/libc.so.6
#1  0x00001545231ea857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001545231eaa54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001544ed71f303 in G4PVPlacement::~G4PVPlacement() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#5  0x00001544ed78af98 in G4PhysicalVolumeStore::Clean() [clone .part.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#6  0x00001544ed78b0b5 in G4PhysicalVolumeStore::~G4PhysicalVolumeStore() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#7  0x00001545283adccc in __run_exit_handlers () from /lib64/libc.so.6
#8  0x00001545283ade00 in exit () from /lib64/libc.so.6
#9  0x00001544fb87e675 in _gfortran_stop_string (string=string@entry=0x0, len=len@entry=0, quiet=quiet@entry=false) at ../../../libgfortran/runtime/stop.c:150
#10 0x00001544f3cd8b08 in pystop (mcod=5) at pystop.f:20
#11 0x00001544f3dbdaae in pdfset (parm=..., value=..., _parm=_parm@entry=20) at pdfset.f:22
#12 0x00001544f3c46266 in pyinit (frame=..., beam=..., target=..., win=8000, _frame=3, _beam=1, _target=1) at pyinit.f:115
#13 0x00001544f4460caf in gen::Pythia6Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so

Full stacktrace

Thread 10 (Thread 0x1544e8d7f700 (LWP 474533) "cmsRun"):
#0  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#2  0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#3  0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x1544e9f7f700 (LWP 474532) "cmsRun"):
#0  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#2  0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#3  0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x1544eb17f700 (LWP 474531) "cmsRun"):
#0  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#2  0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#3  0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x1544ec37f700 (LWP 474530) "cmsRun"):
#0  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001544ed3c706b in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#2  0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#3  0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x1544ecddd700 (LWP 474529) "cmsRun"):
#0  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001544ed3ad3c3 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#2  0x000015452882ba73 in std::execute_native_thread_routine (__p=0x1544ef3ec7c0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#3  0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#4  0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x1544f4fff700 (LWP 474509) "cmsRun"):
#0  0x0000154528465098 in nanosleep () from /lib64/libc.so.6
#1  0x0000154528464f9e in sleep () from /lib64/libc.so.6
#2  0x00001545231e71d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00001544ed3c94d3 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#6  0x000015452938f98d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x00001545292b1b40 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x00001545292bbbd9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x00001545294f4b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x154526283b00, waiter=..., this=0x154526339500) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#10 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x154526339500) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#11 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#12 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#13 0x00001545294f6cee in tbb::detail::r1::rml::private_worker::run (this=0x154523b6a000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154523b6a000) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#15 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#16 0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x1544f5f1d700 (LWP 474508) "cmsRun"):
#0  0x000015452848fac1 in poll () from /lib64/libc.so.6
#1  0x00001545231ea857 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001545231eaa54 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00001544ed71f303 in G4PVPlacement::~G4PVPlacement() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#5  0x00001544ed78af98 in G4PhysicalVolumeStore::Clean() [clone .part.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#6  0x00001544ed78b0b5 in G4PhysicalVolumeStore::~G4PhysicalVolumeStore() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#7  0x00001545283adccc in __run_exit_handlers () from /lib64/libc.so.6
#8  0x00001545283ade00 in exit () from /lib64/libc.so.6
#9  0x00001544fb87e675 in _gfortran_stop_string (string=string@entry=0x0, len=len@entry=0, quiet=quiet@entry=false) at ../../../libgfortran/runtime/stop.c:150
#10 0x00001544f3cd8b08 in pystop (mcod=5) at pystop.f:20
#11 0x00001544f3dbdaae in pdfset (parm=..., value=..., _parm=_parm@entry=20) at pdfset.f:22
#12 0x00001544f3c46266 in pyinit (frame=..., beam=..., target=..., win=8000, _frame=3, _beam=1, _target=1) at pyinit.f:115
#13 0x00001544f4460caf in gen::Pythia6Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so
#14 0x00001544f4461ddd in edm::GeneratorFilter<gen::Pythia6Hadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so
#15 0x000015452939fb87 in edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#16 0x000015452938e010 in edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x00001545292cf1ca in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x00001545292d114a in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00001545295a0b35 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#20 0x00001545294f4b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x15452629b600, waiter=..., this=0x154526339400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#21 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x154526339400) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#22 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#23 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#24 0x00001545294f6cee in tbb::detail::r1::rml::private_worker::run (this=0x154523b6a100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#25 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154523b6a100) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#26 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#27 0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x1544f691e700 (LWP 474507) "cmsRun"):
#0  0x0000154528465098 in nanosleep () from /lib64/libc.so.6
#1  0x0000154528464f9e in sleep () from /lib64/libc.so.6
#2  0x00001545231e71d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00001544ed3c94d3 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#6  0x000015452938f98d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x00001545292b1b40 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x00001545292bbbd9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x00001545294f4b3b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x15452628b900, waiter=..., this=0x154526339480) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#10 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x154526339480) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#11 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:137
#12 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/market.cpp:599
#13 0x00001545294f6cee in tbb::detail::r1::rml::private_worker::run (this=0x154523b6a080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#14 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x154523b6a080) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#15 0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#16 0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x154501c6d700 (LWP 474430) "cmsRun"):
#0  0x0000154528464e42 in waitpid () from /lib64/libc.so.6
#1  0x00001545231e7327 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00001545231ea63a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000015452882ba73 in std::execute_native_thread_routine (__p=0x1545048ba440) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000154527bca1ca in start_thread () from /lib64/libpthread.so.0
#5  0x00001545283968d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x154526e52580 (LWP 474343) "cmsRun"):
#0  0x0000154528465098 in nanosleep () from /lib64/libc.so.6
#1  0x0000154528464f9e in sleep () from /lib64/libc.so.6
#2  0x00001545231e71d0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000154527bd048c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00001544ed3c94d3 in OscarMTProducer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/biglib/el8_amd64_gcc12/pluginSimulation.so
#6  0x000015452938f98d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#7  0x00001545292b1b40 in edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x00001545292bbbd9 in tbb::detail::d1::function_task<edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x00001545294fd3e1 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x154526339380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#10 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x154526339380) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#11 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#12 0x00001545292935fb in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#13 0x00001545292a10ef in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#14 0x00001545292a15a1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#15 0x000000000040840c in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#16 0x00001545294e99ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-2391c941213c757dc9a1835b31681235/tbb-v2021.9.0/src/tbb/arena.cpp:688
#17 0x000000000040a0f2 in main::{lambda()#1}::operator()() const ()
#18 0x0000000000405100 in main ()

Current Modules:

Module: Pythia6GeneratorFilter:generator (crashed)

Geant4 also reports this exception:

-------- EEEE ------- G4Exception-START -------- EEEE -------
*** G4Exception : Run0107
      issued by : G4PhysicsListHelper::RegisterProcess
No Ordering Parameter Table 

-------- EEEE -------- G4Exception-END --------- EEEE -------

Some RelVals (e.g. 159.03) are reported to fail with SIGABRT, but log indicates that SIGSEGV also occured:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

pure virtual method called
terminate called without an active exception

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[iarspider]

Two RelVals - Relval 574.0 step 1 and Relval 577.0 step 1 - failed with

 Error: you did not link PDFLIB correctly.
 Dummy routine PDFSET in PYTHIA file called instead.
 Execution stopped!

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

FYI @cms-sw/simulation-l2

[iarspider]

@civanch latest ASNEEDED_X IB: link, example log link

[civanch]

@iarspider , would it be possible to test GEANT4 branch? It uses the most recent Geant4. Even in previous cases OrderingTable should not be opened, in the latest version of G4 both the table and the call to it are removed.

[civanch]

I am not sure but suspect that there are two problems: in generator there is no needed data and in the Geant4 initialisation data race in destruction in circomstances of incomplete initial initialisation. Proposed PR #45986 may be useful for Geant4 part but cannot fix generator problem.

[civanch]

It seems that #45986 removes Geant4 warnings/exceptions but generator problems remain.

[smuzaffar]

Looks like all these workflow require LHAPDF library to be explicitly linked in (see #46665). These workflows call PDFSET which should be loaded from libLHAPDF.so but as this library is not explicitly linked in (linker decided to not link as nothing from this was used) so we ended up calling the dummy implementation of PDFSET from pythia6 ( as seem from the stack trace [a]. Running locally , we also see message like

 Error: you did not link PDFLIB correctly.
 Dummy routine PDFSET in PYTHIA file called instead.
 Execution stopped!

forcefully linking or preloading libLHAPDF.so allows all these workflow to run. We can add some Buildfile flags to indicate the build system to always link the library

#8  0x00001545283ade00 in exit () from /lib64/libc.so.6
#9  0x00001544fb87e675 in _gfortran_stop_string (string=string@entry=0x0, len=len@entry=0, quiet=quiet@entry=false) at ../../../libgfortran/runtime/stop.c:150
#10 0x00001544f3cd8b08 in pystop (mcod=5) at pystop.f:20
#11 0x00001544f3dbdaae in pdfset (parm=..., value=..., _parm=_parm@entry=20) at pdfset.f:22
#12 0x00001544f3c46266 in pyinit (frame=..., beam=..., target=..., win=8000, _frame=3, _beam=1, _target=1) at pyinit.f:115
#13 0x00001544f4460caf in gen::Pythia6Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ASNEEDED_X_2024-08-19-1100/lib/el8_amd64_gcc12/pluginGeneratorInterfacePythia6Filters.so

[smuzaffar]

Looks like lhapdf (which provides pdfset) was not linked as pythia6_pdfdummy (which is part of pythia6 tool definition) was also providing the dummy implemetation of pdfset ( thatis why we were getting Dummy routine PDFSET in PYTHIA file called instead message).

I have split the pythia6 tool and moved pythia6_pdfdummy library in separate tool. #46684 along with cmsdist change cms-sw/cmsdist#9514 should fix these crashes