Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MueLu: unit test for RegionRFactory failing #7832

Closed
mayrmt opened this issue Aug 12, 2020 · 2 comments
Closed

MueLu: unit test for RegionRFactory failing #7832

mayrmt opened this issue Aug 12, 2020 · 2 comments
Assignees
@mayrmt @lucbv
Labels
pkg: MueLu type: bug The primary issue is a bug in Trilinos code or tests

Comments

@mayrmt
Copy link
Member

mayrmt commented Aug 12, 2020

Bug Report

@trilinos/muelu
@lucbv @pohm01 @rstumin

Description

Since the addition of the RegionRFactory in PR #7672, the associated unit tests keeps failing on my machine (though it passes on @lucbv's machine).

Here's the log of the screen output. 
a11bmama:unit_tests$ mpirun -np 4 ./MueLu_UnitTests_Region.exe
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name theseus and rank 1!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name theseus and rank 3!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name theseus and rank 0!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name theseus and rank 2!

***
*** Unit test suite ...
***


Sorting tests by group name then by the order they were added ... (time = 2.86e-06)

Running unit tests ...

0. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_CompositeToRegionMatrix_UnitTest ... [Passed] (0.00296 sec)
1. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_RegionToCompositeMatrix_UnitTest ... [Passed] (0.002 sec)
2. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_MatVec_UnitTest ... [Passed] (0.00129 sec)
3. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_FastMatVec_UnitTest ... [Passed] (0.00159 sec)
4. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_FastMatVec3D_UnitTest ... [Passed] (0.00214 sec)
5. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_FastMatVec2D_Elasticity_UnitTest ... [Passed] (0.00342 sec)
6. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_FastMatVec3D_Elasticity_UnitTest ... [Passed] (0.018 sec)
7. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_Laplace2D_UnitTest ... [Passed] (0.00214 sec)
8. RegionMatrix_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_Laplace3D_UnitTest ... [Passed] (0.00489 sec)
9. RegionRFactory_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_RegionRFactCtor_UnitTest ... [Passed] (2.69e-05 sec)
*** Error in `./MueLu_UnitTests_Region.exe': malloc(): memory corruption: 0x0000000000d1bd70 ***
*** Error in `./MueLu_UnitTests_Region.exe': malloc(): memory corruption: 0x00000000028e5190 ***
[theseus:08498] *** Process received signal ***
[theseus:08498] Signal: Aborted (6)
[theseus:08498] Signal code:  (-6)
[theseus:08501] *** Process received signal ***
[theseus:08501] Signal: Aborted (6)
[theseus:08501] Signal code:  (-6)
10. RegionRFactory_double_int_longlong_Kokkos_Compat_KokkosSerialWrapperNode_RegionRFactLaplace3D_UnitTest ... /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7fbe743317f5]
/lib/x86_64-linux-gnu/libc.so.6(+0x[theseus:08498] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f8e25ac5390]
[theseus:08498] [ 1] ======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fbe7433e1d4]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZNK6Kokkos9HostSpace8allocateEm+0x1e)[0x7fbe7564522e]
/lib/x86_64-linux-gnu/libc.so.6(+0x777f5[theseus:08501] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fce68a3e390]
[theseus:08501] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f8e2571f438]
[theseus:08498] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f8e2572103a]
[theseus:08498] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x777fa)[0x7f8e257617fa]
[theseus:08498] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x8215e)[0x7f61fb23415e]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fce68698438]
[theseus:08501] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fce6869a03a]
[theseus:08501] [ 3] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12/lib/x86_64-linux-gnu/libc.so.6(+0x8215e)[0x7f8e2576c15e]
[theseus:08498] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f8e2576e1d4]
[theseus:08498] [ 6] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZNK6Kokkos9HostSpace8allocateEm+0x1e)[0x7f8e26a7522e]
[theseus:08498] [ 7] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZN6Kokkos4Impl22SharedAllocationRecordINS_9HostSpaceEvEC1ERKS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmPFvPNS1_IvvEEE+0x48)[0x7f8e26a75a48]
[theseus:08498] [ 8] /lib/x86_64-linux-gnu/libc.so.6(+0x777fa)[0x7fce686da7fa]
[theseus:08501] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x8215e)[0x7fce686e515e]
[theseus:08501] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fce686e71d4]
[theseus:08501] [ 6] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZNK6Kokkos9HostSpace8allocateEm+0x1e)[0x7fce699ee22e]
[theseus:08501] [ 7] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZN6Kokkos4Impl22SharedAllocationRecordINS_9HostSpaceEvEC1ERKS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmPFvPNS1_IvvEEE+0x48)[0x7fce699eea48]
[theseus:08501] [ 8] /ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE20createTransposeLocalERKN7Teuchos3RCPINS8_13ParameterListEEE+0x7f5)[0x7f61f97dd2c5]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE20createTransposeLocalERKN7Teuchos3RCPINS8_13ParameterListEEE+0x7f5)[0x7f8e23d152c5]
[theseus:08498] [ 9] /ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15createTransposeERKN7Teuchos3RCPINS8_13ParameterListEEE+0x41)[0x7fbe728e81c1]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE20createTransposeLocalERKN7Teuchos3RCPINS8_13ParameterListEEE+0x7f5)[0x7fce66c8e2c5]
[theseus:08501] [ 9] /ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15createTransposeERKN7Teuchos3RCPINS8_13ParameterListEEE+0x41)[0x7f8e23d181c1]
[theseus:08498] [10] /ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15createTransposeERKN7Teuchos3RCPINS8_13ParameterListEEE+0x41)[0x7f61f97e01c1]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZN5MueLu9UtilitiesIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE9TransposeERN6Xpetra6MatrixIdixS6_EEbRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN7Teuchos3RCPINSK_13ParameterListEEE+0x252)[0x7fbe775bd522]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15createTransposeERKN7Teuchos3RCPINS8_13ParameterListEEE+0x41)[0x7fce66c911c1]
[theseus:08501] [10] /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZN5MueLu9UtilitiesIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE9TransposeERN6Xpetra6MatrixIdixS6_EEbRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN7Teuchos3RCPINSK_13ParameterListEEE+0x252)[0x7f8e289ed522]
[theseus:08498] [11] /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZN5MueLu9UtilitiesIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE9TransposeERN6Xpetra6MatrixIdixS6_EEbRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN7Teuchos3RCPINSK_13ParameterListEEE+0x252)[0x7f61fe4b5522]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZNK5MueLu14RegionRFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5BuildERNS_5LevelES9_+0xbc3)[0x7fbe774f1513]
./MueLu_UnitTests_Region.exe[0x54db4a]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZNK7Teuchos12UnitTestBase11runUnitTestERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x37)[0x7fbe758e35d7]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2c)[0x7fbe758e5d9c]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12/ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZNK5MueLu14RegionRFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5BuildERNS_5LevelES9_+0xbc3)[0x7f8e28921513]
[theseus:08498] [12] /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZN5MueLu9UtilitiesIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE9TransposeERN6Xpetra6MatrixIdixS6_EEbRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN7Teuchos3RCPINSK_13ParameterListEEE+0x252)[0x7fce6b966522]
[theseus:08501] [11] ./MueLu_UnitTests_Region.exe[0x426dec]
./MueLu_UnitTests_Region.exe[0x54db4a]
[theseus:08498] [13] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZNK7Teuchos12UnitTestBase11runUnitTestERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x37)[0x7f8e26d135d7]
[theseus:08498] [14] /lib/x86_64-linux-gnu/libc.so.6/ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2c)[0x7f8e26d15d9c]
[theseus:08498] [15] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository12runUnitTestsERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x15b9)[0x7f8e26d18709]
[theseus:08498] [16] ./MueLu_UnitTests_Region.exe[0x428239]
======= Memory map: ========
/ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository20runUnitTestsFromMainEiPPc+0x112)[0x7f8e26d19ee2]
[theseus:08498] [17] ./MueLu_UnitTests_Region.exe[0x426dec]
[theseus:08498] [18] /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f8e2570a840]
[theseus:08498] [19] ./MueLu_UnitTests_Region.exe[0x428239]
[theseus:08498] *** End of error message ***
./MueLu_UnitTests_Region.exe[0x54db4a]
[theseus:08499] *** Process received signal ***
[theseus:08499] Signal: Aborted (6)
[theseus:08499] Signal code:  (-6)
/ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZNK5MueLu14RegionRFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5BuildERNS_5LevelES9_+0xbc3)[0x7fce6b89a513]
[theseus:08501] [12] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2c)[0x7f61fc7ddd9c]
[theseus:08499] [ 0] ./MueLu_UnitTests_Region.exe[0x54db4a]
[theseus:08501] [13] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository12runUnitTestsERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x15b9)[0x7f61fc7e0709]
/ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZNK7Teuchos12UnitTestBase11runUnitTestERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x37)[0x7fce69c8c5d7]
[theseus:08501] [14] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2c)[0x7fce69c8ed9c]
[theseus:08501] [15] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fbe74695390]
[theseus:08499] [ 1] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository12runUnitTestsERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x15b9)[0x7fce69c91709]
[theseus:08501] [16] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fbe742ef438]
[theseus:08499] [ 2] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository20runUnitTestsFromMainEiPPc+0x112)[0x7fce69c92ee2]
[theseus:08501] [17] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f61fb1d2840]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fbe742f103a]
[theseus:08499] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x777fa)[0x7fbe743317fa]
[theseus:08499] [ 4] ./MueLu_UnitTests_Region.exe[0x426dec]
[theseus:08501] [18] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fce68683840]
[theseus:08501] [19] ./MueLu_UnitTests_Region.exe[0x428239]
[theseus:08501] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x8215e)[0x7fbe7433c15e]
[theseus:08499] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fbe7433e1d4]
[theseus:08499] [ 6] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZNK6Kokkos9HostSpace8allocateEm+0x1e)[0x7fbe7564522e]
[theseus:08499] [ 7] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZN6Kokkos4Impl22SharedAllocationRecordINS_9HostSpaceEvEC1ERKS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmPFvPNS1_IvvEEE+0x48)[0x7fbe75645a48]
[theseus:08499] [ 8] 00400000-00590000 r-xp 00000000 08:01 9577335                            /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/test/unit_tests/MueLu_UnitTests_Region.exe
0078f000-00790000 r--p 0018f000 08:01 9577335                            /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/test/unit_tests/MueLu_UnitTests_Region.exe
00790000-00791000 rw-p 00190000 08:01 9577335                            /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/test/unit_tests/MueLu_UnitTests_Region.exe
00791000-00793000 rw-p 00000000 00:00 0 
02601000-02abb000 rw-p 00000000 00:00 0                                  [heap]
7f61d8000000-7f61d8021000 rw-p 00000000 00:00 0 
7f61d8021000-7f61dc000000 ---p 00000000 00:00 0 
7f61dfe48000-7f61dfe4e000 r-xp 00000000 08:01 5389673                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f61dfe4e000-7f61e004d000 ---p 00006000 08:01 5389673                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f61e004d000-7f61e004e000 r--p 00005000 08:01 5389673                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f61e004e000-7f61e004f000 rw-p 00006000 08:01 5389673                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
7f61e004f000-7f61e8050000 rw-s 00000000 08:01 11406058                   /tmp/openmpi-sessions-a11bmama@theseus_0/27468/1/shared_mem_pool.theseus (deleted)
7f61e8050000-7f61e8451000 rw-s 00000000 08:01 11406055                   /tmp/openmpi-sessions-a11bmama@theseus_0/27468/1/3/vader_segment.theseus.3
7f61e8451000-7f61e8852000 rw-s 00000000 08:01 11406056                   /tmp/openmpi-sessions-a11bmama@theseus_0/27468/1/1/vader_segment.theseus.1
7f61e8852000-7f61e8867000 r-xp 00000000 08:01 5389729                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f61e8867000-7f61e8a66000 ---p 00015000 08:01 5389729                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f61e8a66000-7f61e8a67000 r--p 00014000 08:01 5389729                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f61e8a67000-7f61e8a68000 rw-p 00015000 08:01 5389729                    /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f61e8a68000-7f61e8a6a000 r-xp 00000000 08:01 5389675                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f61e8a6a000-7f61e8c69000 ---p 00002000 08:01 5389675                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f61e8c69000-7f61e8c6a000 r--p 00001000 08:01 5389675                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f61e8c6a000-7f61e8c6b000 rw-p 00002000 08:01 5389675                    /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f61e8c6b000-7f61e8c71000 r-xp 00000000 08:01 5389705                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f61e8c71000-7f61e8e70000 ---p 00006000 08:01 5389705                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f61e8e70000-7f61e8e71000 r--p 00005000 08:01 5389705                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f61e8e71000-7f61e8e72000 rw-p 00006000 08:01 5389705                    /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f61e8e9d000-7f61e8ea2000 r-xp 00000000 08:01 5389680                    /usr/lib/openmpi/lib/openmpi/mca_osc_sm.so
7f61e8ea2000-7f61e90a1000 ---p 00005000 08:01 5389680                    /usr/lib/openmpi/lib/openmpi/mca_osc_sm.so
7f61e90a1000-7f61e90a2000 r--p 00004000 08:01 5389680                    /usr/lib/openmpi/lib/openmpi/mca_osc_sm.so
7f61e90a2000-7f61e90a4000 rw-p 00005000 08:01 5389680                    /usr/lib/openmpi/lib/openmpi/mca_osc_sm.so
7f61e90a7000-7f61e90aa000 r-xp 00000000 08:01 5389694                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f61e90aa000-7f61e92a9000 ---p 00003000 08:01 5389694                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f61e92a9000-7f61e92aa000 r--p 00002000 08:01 5389694                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f61e92aa000-7f61e92ab000 rw-p 00003000 08:01 5389694                    /usr/lib/openmpi/lib/openmpi/mca_pubsub_orte.so
7f61e92ab000-7f61e92c4000 r-xp 00000000 08:01 5389622                    /usr/lib/openmpi/lib/openmpi/mca_coll_libnbc.so
7f61e92c4000-7f61e94c3000 ---p 00019000 08:01 5389622                    /usr/lib/openmpi/lib/openmpi/mca_coll_libnbc.so
7f61e94c3000-7f61e94c4000 r--p 00018000 08:01 5389622                    /usr/lib/openmpi/lib/openmpi/mca_coll_libnbc.so
7f61e94c4000-7f61e94c5000 rw-p 00019000 08:01 5389622                    /usr/lib/openmpi/lib/openmpi/mca_coll_libnbc.so
7f61e94c5000-7f61e94e4000 r-xp 00000000 08:01 5389713                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f61e94e4000-7f61e96e3000 ---p 0001f000 08:01 5389713                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f61e96e3000-7f61e96e4000 r--p 0001e000 08:01 5389713                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f61e96e4000-7f61e96e5000 rw-p 0001f000 08:01 5389713                    /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f61e96e5000-7f61e96f2000 r-xp 00000000 08:01 5389662                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f61e96f2000-7f61e98f2000 ---p 0000d000 08:01 5389662                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f61e98f2000-7f61e98f3000 r--p 0000d000 08:01 5389662                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f61e98f3000-7f61e98f4000 rw-p 0000e000 08:01 5389662                    /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f61e991e000-7f61e9d1f000 rw-s 00000000 08:01 11406062                   /tmp/openmpi-sessions-a11bmama@theseus_0/27468/1/0/vader_segment.theseus.0
7f61e9d1f000-7f61e9d22000 r-xp 00000000 08:01 5389659                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f61e9d22000-7f61e9f22000 ---p 00003000 08:01 5389659                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f61e9f22000-7f61e9f23000 r--p 00003000 08:01 5389659                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f61e9f23000-7f61e9f24000 rw-p 00004000 08:01 5389659                    /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f61e9f24000-7f61e9f3c000 r/ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE20createTransposeLocalERKN7Teuchos3RCPINS8_13ParameterListEEE+0x7f5)[0x7fbe728e52c5]
[theseus:08499] [ 9] -xp 00000000 08:01 5389640                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f61e9f3c000-7f61ea13b000 ---p 00018000 08:01 5389640                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f61ea13b000-7f61ea13c000 r--p 00017000 08:01 5389640                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f61ea13c000-7f61ea13d000 rw-p 00018000 08:01 5389640                    /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so
7f61ea13d000-7f61ea140000 r-xp 00000000 08:01 5389658                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f61ea140000-7f61ea33f000 ---p 00003000 08:01 5389658                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f61ea33f000-7f61ea340000 r--p 00002000 08:01 5389658                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f61ea340000-7f61ea341000 rw-p 00003000 08:01 5389658                    /usr/lib/openmpi/lib/openmpi/mca_btl_self.so
7f61ea341000-7f61ea347000 r-xp 00000000 08:01 5389649                    /usr/lib/openmpi/lib/openmpi/mca_btl_vader.so
7f61ea347000-7f61ea546000 ---p 00006000 08:01 5389649                    /usr/lib/openmpi/lib/openmpi/mca_btl_vader.so
7f61ea546000-7f61ea547000 r--p 00005000 08:01 5389649                    /usr/lib/openmpi/lib/openmpi/mca_btl_vader.so
7f61ea547000-7f61ea549000 rw-p 00006000 08:01 5389649                    /usr/lib/openmpi/lib/openmpi/mca_btl_vader.so
7f61ea549000-7f61ebdff000 r-xp 00000000 08:01 5377018                    /usr/lib/x86_64-linux-gnu/libicudata.so.55.1
7f61ebdff000-7f61ebffe000 ---p 018b6000 08:01 5377018                    /usr/lib/x86_64-linux-gnu/libicudata.so.55.1
7f61ebffe000-7f61ebfff000 r--p 018b5000 08:01 5377018                    /usr/lib/x86_64-linux-gnu/libicudata.so.55.1
7f61ebfff000-7f61ec000000 rw-p 018b6000 08:01 5377018                    /usr/lib/x86_64-linux-gnu/libicudata.so.55.1
7f61ec000000-7f61ec021000 rw-p 00000000 00:00 0 
7f61ec021000-7f61f0000000 ---p 00000000 00:00 0 
7f61f01db000-7f61f01e1000 r-xp 00000000 08:01 5389715                    /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so
7f61f01e1000-7f61f03e0000 ---p 00006000 08:01 5389715                    /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so
7f61f03e0000-7f61f03e1000 r--p 00005000 08:01 5389715                    /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so
7f61f03e1000-7f61f03e2000 rw-p 00006000 08:01 5389715                    /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so
7f61f03e2000-7f61f03ee000 r-xp 00000000 08:01 5389702                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f61f03ee000-7f61f05ed000 ---p 0000c000 08:01 5389702                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f61f05ed000-7f61f05ee000 r--p 0000b000 08:01 5389702                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f61f05ee000-7f61f05ef000 rw-p 0000c000 08:01 5389702                    /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f61f05ef000-7f61f066f000 rw-p 00000000 00:00 0 
7f61f06a1000-7f61f0aa2000 rw-s 00000000 08:01 11406057                   /tmp/openmpi-sessions-a11bmama@theseus_0/27468/1/2/vader_segment.theseus.2
7f61f0aa2000-7f61f0aa5000 r-xp 00000000 08:01 5389664                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f61f0aa5000-7f61f0ca5000 ---p 00003000 08:01 5389664                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f61f0ca5000-7f61f0ca6000 r--p 00003000 08:01 5389664                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f61f0ca6000-7f61f0ca7000 rw-p 00004000 08:01 5389664                    /usr/lib/openmpi/lib/openmpi/mca_bml_r2.so
7f61f0ca7000-7f61f0ca9000 r-xp 00000000 08:01 5389734                    /usr/lib/openmpi/lib/libmca_common_sm.so.4.0.4
7f61f0ca9000-7f61f0ea9000 ---p 00002000 08:01 5389734                    /usr/lib/openmpi/lib/libmca_common_sm.so.4.0.4
7f61f0ea9000-7f61f0eaa000 r--p 00002000 08:01 5389734                    /usr/lib/openmpi/lib/libmca_common_sm.so.4.0.4
7f61f0eaa000-7f61f0eab000 rw-p 00003000 08:01 5389734                    /usr/lib/openmpi/lib/libmca_common_sm.so.4.0.4
7f61f0eab000-7f61f0ead000 r-xp 00000000 08:01 5389719                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f61f0ead000-7f61f10ac000 ---p 00002000 08:01 5389719                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f61f10ac000-7f61f10ad000 r--p 00001000 08:01 5389719                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f61f10ad000-7f61f10ae000 rw-p 00002000 08:01 5389719                    /usr/lib/openmpi/lib/openmpi/mca_mpool_sm.so
7f61f10ae000-7f61f10b1000 r-xp 00000000 08:01 5389696                    /usr/lib/openmpi/lib/openmpi/mca_mpool_grdma.so
7f61f10b1000-7f61f12b0000 ---p 00003000 08:01 5389696                    /usr/lib/openmpi/lib/openmpi/mca_mpool_grdma.so
7f61f12b0000-7f61f12b1000 r--p 00002000 08:01 5389696                    /usr/lib/openmpi/lib/openmpi/mca_mpool_grdma.so
7f61f12b1000-7f61f12b2000 rw-p 00003000 08:01 5389696                    /usr/lib/openmpi/lib/openmpi/mca_mpool_grdma.so
7f61f12b2000-7f61f12b5000 r-xp 00000000 08:01 5389716                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f61f12b5000/ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15createTransposeERKN7Teuchos3RCPINS8_13ParameterListEEE+0x41)[0x7fbe728e81c1]
[theseus:08499] [10] -7f61f14b5000 ---p 00003000 08:01 5389716                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f61f14b5000-7f61f14b6000 r--p 00003000 08:01 5389716                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f61f14b6000-7f61f14b7000 rw-p 00004000 08:01 5389716                    /usr/lib/openmpi/lib/openmpi/mca_rcache_vma.so
7f61f14b7000-7f61f14b9000 r-xp 00000000 08:01 5389652                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f61f14b9000-7f61f16b8000 ---p 00002000 08:01 5389652                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f61f16b8000-7f61f16b9000 r--p 00001000 08:01 5389652                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f61f16b9000-7f61f16ba000 rw-p 00002000 08:01 5389652                    /usr/lib/openmpi/lib/openmpi/mca_allocator_bucket.so
7f61f16ba000-7f61f16bc000 r-xp 00000000 08:01 5389701                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f61f16bc000-7f61f18bb000 ---p 00002000 08:01 5389701                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f61f18bb000-7f61f18bc000 r--p 00001000 08:01 5389701                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f61f18bc000-7f61f18bd000 rw-p 00002000 08:01 5389701                    /usr/lib/openmpi/lib/openmpi/mca_allocator_basic.so
7f61f18bd000-7f61f18de000 r-xp 00000000 08:01 10359297                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f61f18de000-7f61f1add000 ---p 00021000 08:01 10359297                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f61f1add000-7f61f1ade000 r--p 00020000 08:01 10359297                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f61f1ade000-7f61f1adf000 rw-p 00021000 08:01 10359297                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f61f1adf000-7f61f1c5e000 r-xp 00000000 08:01 5377015                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
7f61f1c5e000-7f61f1e5e000 ---p 0017f000 08:01 5377015                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
7f61f1e5e000-7f61f1e6e000 r--p 0017f000 08:01 5377015                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
7f61f1e6e000-7f61f1e6f000 rw-p 0018f000 08:01 5377015                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
7f61f1e6f000-7f61f1e73000 rw-p 00000000 00:00 0 
7f61f1e73000-7f61f2024000 r-xp 00000000 08:01 5374284                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.3
7f61f2024000-7f61f2223000 ---p 001b1000 08:01 5374284                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.3
7f61f2223000-7f61f222b000 r--p 001b0000 08:01 5374284                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.3
7f61f222b000-7f61f222d000 rw-p 001b8000 08:01 5374284                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.3
7f61f222d000-7f61f222e000 rw-p 00000000 00:00 0 
7f61f222e000-7f61f2231000 r-xp 00000000 08:01 6033256                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_xml_libxml.so
7f61f2231000-7f61f2430000 ---p 00003000 08:01 6033256                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_xml_libxml.so
7f61f2430000-7f61f2431000 r--p 00002000 08:01 6033256                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_xml_libxml.so
7f61f2431000-7f61f2432000 rw-p 00003000 08:01 6033256                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_xml_libxml.so
7f61f2432000-7f61f244b000 r-xp 00000000 08:01 10355027                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f61f244b000-7f61f264a000 ---p 00019000 08:01 10355027                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f61f264a000-7f61f264b000 r--p 00018000 08:01 10355027                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f61f264b000-7f61f264c000 rw-p 00019000 08:01 10355027                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f61f264c000-7f61f2654000 r-xp 00000000 08:01 5384505                    /usr/lib/x86_64-linux-gnu/libpciaccess.so.0.11.1
7f61f2654000-7f61f2854000 ---p 00008000 08:01 5384505                    /usr/lib/x86_64-linux-gnu/libpciaccess.so.0.11.1
7f61f2854000-7f61f2855000 r--p 00008000 08:01 5384505                    /usr/lib/x86_64-linux-gnu/libpciaccess.so.0.11.1
7f61f2855000-7f61f2856000 rw-p 00009000 08:01 5384505                    /usr/lib/x86_64-linux-gnu/libpciaccess.so.0.11.1
7f61f2856000-7f61f2858000 r-xp 00000000 08:01 6033258                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_pci.so
7f61f2858000-7f61f2a58000 ---p 00002000 08:01 6033258                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_pci.so
7f61f2a58000-7f61f2a59000 r--p 00002000 08:01 6033258                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_pci.so
7f61f2a59000-7f61f2a5a000 rw-p 00003000 08:01 6033258                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_pci.so
7f61f2a5a000-7f61f2a63000 r-xp 00000000 08:01 5390014                    /usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0
7f61f2a63000-7f61f2c63000 ---p 00009000 08:01 5390014                    /usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0
7f61f2c63000-7f61f2c64000 r--p 00009000 08:01 5390014                    /usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0
7f61f2c64000-7f61f2c65000 rw-p 0000a000 08:0/ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZN5MueLu9UtilitiesIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE9TransposeERN6Xpetra6MatrixIdixS6_EEbRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN7Teuchos3RCPINSK_13ParameterListEEE+0x252)[0x7fbe775bd522]
[theseus:08499] [11] 1 5390014                    /usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0
7f61f2c65000-7f61f2c67000 r-xp 00000000 08:01 6033257                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_opencl.so
7f61f2c67000-7f61f2e66000 ---p 00002000 08:01 6033257                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_opencl.so
7f61f2e66000-7f61f2e67000 r--p 00001000 08:01 6033257                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_opencl.so
7f61f2e67000-7f61f2e68000 rw-p 00002000 08:01 6033257                    /usr/lib/x86_64-linux-gnu/hwloc/hwloc_opencl.so
7f61f2e68000-7f61f2e7e000 r-xp 00000000 08:01 10357561                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f61f2e7e000-7f61f307d000 ---p 00016000 08:01 10357561                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f61f307d000-7f61f307e000 r--p 00015000 08:01 10357561                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f61f307e000-7f61f307f000 rw-p 00016000 08:01 10357561                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f61f307f000-7f61f3081000 rw-p 00000000 00:00 0 
7f61f3081000-7f61f308c000 r-xp 00000000 08:01 10357575                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f61f308c000-7f61f328b000 ---p 0000b000 08:01 10357575                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f61f328b000-7f61f328c000 r--p 0000a000 08:01 10357575                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f61f328c000-7f61f328d000 rw-p 0000b000 08:01 10357575                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f61f328d000-7f61f3298000 r-xp 00000000 08:01 10357571                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f61f3298000-7f61f3497000 ---p 0000b000 08:01 10357571                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f61f3497000-7f61f3498000 r--p 0000a000 08:01 10357571                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f61f3498000-7f61f3499000 rw-p 0000b000 08:01 10357571                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f61f3499000-7f61f349f000 rw-p 00000000 00:00 0 
7f61f349f000-7f61f34a0000 ---p 00000000 00:00 0 
7f61f34a0000-7f61f3ca0000 rw-p 00000000 00:00 0 
7f61f3ca0000-7f61f3ca2000 r-xp 00000000 08:01 5389657                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f61f3ca2000-7f61f3ea1000 ---p 00002000 08:01 5389657                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f61f3ea1000-7f61f3ea2000 r--p 00001000 08:01 5389657                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f61f3ea2000-7f61f3ea3000 rw-p 00002000 08:01 5389657                    /usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so
7f61f3ea3000-7f61f3ea6000 r-xp 00000000 08:01 5389627                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f61f3ea6000-7f61f40a5000 ---p 00003000 08:01 5389627                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f61f40a5000-7f61f40a6000 r--p 00002000 08:01 5389627                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f61f40a6000-7f61f40a7000 rw-p 00003000 08:01 5389627                    /usr/lib/openmpi/lib/openmpi/mca_rml_oob.so
7f61f40a7000-7f61f40ba000 r-xp 00000000 08:01 5389661                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f61f40ba000-7f61f42b9000 ---p 00013000 08:01 5389661                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f61f42b9000-7f61f42ba000 r--p 00012000 08:01 5389661                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f61f42ba000-7f61f42bb000 rw-p 00013000 08:01 5389661                    /usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so
7f61f42bb000-7f61f42bd000 r-xp 00000000 08:01 5389688                    /usr/lib/openmpi/lib/openmpi/mca_errmgr_default_app.so
7f61f42bd000-7f61f44bc000 ---p 00002000 08:01 5389688                    /usr/lib/openmpi/lib/openmpi/mca_errmgr_default_app.so
7f61f44bc000-7f61f44bd000 r--p 00001000 08:01 5389688                    /usr/lib/openmpi/lib/openmpi/mca_errmgr_default_app.so
7f61f44bd000-7f61f44be000 rw-p 00002000 08:01 5389688                    /usr/lib/openmpi/lib/openmpi/mca_errmgr_default_app.so
7f61f44be000-7f61f44c0000 r-xp 00000000 08:01 5389/ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZNK5MueLu14RegionRFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5BuildERNS_5LevelES9_+0xbc3)[0x7fbe774f1513]
[theseus:08499] [12] ./MueLu_UnitTests_Region.exe[0x54db4a]
[theseus:08499] [13] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZNK7Teuchos12UnitTestBase11runUnitTestERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x37)[0x7fbe758e35d7]
[theseus:08499] [14] [theseus:08500] *** Process received signal ***
[theseus:08500] Signal: Aborted (6)
[theseus:08500] Signal code:  (-6)
/ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2c)[0x7fbe758e5d9c]
[theseus:08499] [15] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository12runUnitTestsERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x15b9)[0x7fbe758e8709]
[theseus:08499] [16] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository20runUnitTestsFromMainEiPPc+0x112)[0x7fbe758e9ee2]
[theseus:08499] [17] ./MueLu_UnitTests_Region.exe[0x426dec]
[theseus:08499] [18] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fbe742da840]
[theseus:08499] [19] [theseus:08500] [ 0] ./MueLu_UnitTests_Region.exe[0x428239]
[theseus:08499] *** End of error message ***
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f61fb58d390]
[theseus:08500] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f61fb1e7438]
[theseus:08500] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f61fb1e903a]
[theseus:08500] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x777fa)[0x7f61fb2297fa]
[theseus:08500] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x8215e)[0x7f61fb23415e]
[theseus:08500] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f61fb2361d4]
[theseus:08500] [ 6] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZNK6Kokkos9HostSpace8allocateEm+0x1e)[0x7f61fc53d22e]
[theseus:08500] [ 7] /ssd/codes/mayrmt_trilinos/build_muelu/packages/kokkos/core/src/libkokkoscore.so.12(_ZN6Kokkos4Impl22SharedAllocationRecordINS_9HostSpaceEvEC1ERKS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmPFvPNS1_IvvEEE+0x48)[0x7f61fc53da48]
[theseus:08500] [ 8] /ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE20createTransposeLocalERKN7Teuchos3RCPINS8_13ParameterListEEE+0x7f5)[0x7f61f97dd2c5]
[theseus:08500] [ 9] /ssd/codes/mayrmt_trilinos/build_muelu/packages/tpetra/core/src/libtpetra.so.12(_ZN6Tpetra19RowMatrixTransposerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15createTransposeERKN7Teuchos3RCPINS8_13ParameterListEEE+0x41)[0x7f61f97e01c1]
[theseus:08500] [10] /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZN5MueLu9UtilitiesIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE9TransposeERN6Xpetra6MatrixIdixS6_EEbRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN7Teuchos3RCPINSK_13ParameterListEEE+0x252)[0x7f61fe4b5522]
[theseus:08500] [11] /ssd/codes/mayrmt_trilinos/build_muelu/packages/muelu/src/libmuelu.so.12(_ZNK5MueLu14RegionRFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5BuildERNS_5LevelES9_+0xbc3)[0x7f61fe3e9513]
[theseus:08500] [12] ./MueLu_UnitTests_Region.exe[0x54db4a]
[theseus:08500] [13] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZNK7Teuchos12UnitTestBase11runUnitTestERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x37)[0x7f61fc7db5d7]
[theseus:08500] [14] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2c)[0x7f61fc7ddd9c]
[theseus:08500] [15] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository12runUnitTestsERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x15b9)[0x7f61fc7e0709]
[theseus:08500] [16] /ssd/codes/mayrmt_trilinos/build_muelu/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository20runUnitTestsFromMainEiPPc+0x112)[0x7f61fc7e1ee2]
[theseus:08500] [17] ./MueLu_UnitTests_Region.exe[0x426dec]
[theseus:08500] [18] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f61fb1d2840]
[theseus:08500] [19] ./MueLu_UnitTests_Region.exe[0x428239]
[theseus:08500] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 8501 on node theseus exited on signal 6 (Aborted).
--------------------------------------------------------------------------

Using a bunch of std::cout statements, I could track it down. The line

row_map_type t_offsets
    (view_alloc ("transpose ptr", WithoutInitializing), lclNumCols + 1);

in Tpetra_RowMatrixTransposer_def.hpp fails.

Steps to Reproduce

  1. SHA1: 71d6e40

  2. Configure script: do-configure-trilinos_muelu_tpetra.txt

  3. Configure log: [attach here]

  4. Build log: [attach here]

  5. Input deck: [attach here]

  6. Run ctest -R Region in packages/muelu/test/unit_tests/

  7. Error log: see above.

Additional Information

This becomes more problematic, as we are currently reorganising the regionMG source code. Proper testing is crucial there, so we need to resolve this.

Since this passes on @lucbv's machine, this might also just be related to my configuration (see attachments), though I can't tell for sure.
@mayrmt mayrmt added type: bug The primary issue is a bug in Trilinos code or tests pkg: MueLu labels Aug 12, 2020
@mayrmt mayrmt changed the title MueLu: regionMG unit test for RegionRFactory failing MueLu: unit test for RegionRFactory failing Aug 12, 2020
This was referenced Aug 12, 2020
Merged
Merged
@lucbv
Copy link
Contributor

lucbv commented Aug 12, 2020

@mayrmt I will have a look at this using your configuration to hopefully reproduce the error.
It would be a bit frustrating if the error is indeed stemming from the transposer but it can certainly be fixed non the less.

@lucbv lucbv mentioned this issue Aug 22, 2020
Merged
@mayrmt
Copy link
Member Author

mayrmt commented Aug 24, 2020

Fix has been merged too master via PR #7885. Closing.

@mayrmt mayrmt closed this as completed Aug 24, 2020
@jhux2 jhux2 added this to MueLu Aug 12, 2024
@jhux2 jhux2 moved this to Done in MueLu Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mayrmt mayrmt

@lucbv lucbv

Labels
pkg: MueLu type: bug The primary issue is a bug in Trilinos code or tests
Projects
MueLu
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mayrmt @lucbv