Test TeuchosNumerics_LAPACK_test_MPI_1 fails in all 'debug' builds on power8 'ride'

[bartlettroscoe]

CC: @trilinos/teuchos

Next Action Status:

PR #2447 was merged on 3/23/2018 which disabled the test. PR #4064 which enables the whole test TeuchosNumerics_LAPACK_test_MPI_1 but disables the single unit test for STEQR() merged to 'develop' on 12/18/2018. Next: Watch for test running and passing (minus STEQR() unit test) on 'release-debug' and 'opt' builds on 'white', 'ride', and 'waterman' on 12/19/2018 ...

Description

The test TeuchosNumerics_LAPACK_test_MPI_1 segfaults on the 'debug' builds Trilinos-atdm-white-ride-cuda-debug and Trilinos-atdm-white-ride-gnu-debug-openmp on 'ride' and 'white' but passes in all of the 'opt' builds on these same machines as well as for all of the builds on hansen as shown this morning in:

• https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-17&filtercombine=and&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=TeuchosNumerics_LAPACK_test_MPI_1

The failing tests all show segfaults showing the output:

Teuchos in Trilinos 12.13 (Dev)

GESV test ... passed!
LAPY2 test ... passed!
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 16320 on node white24 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

What is interesting is that this test only failed in all of the Trilinos builds that were done yesterday in the query:

• https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-16&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=TeuchosNumerics_LAPACK_test_MPI_1&field3=status&compare3=62&value3=passed

May this be the same error reported in #1208 that we basically gave up on?

Steps to Reproduce

Following the instructions at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite

one can reproduce this failing test by enabling the Teuchos package for the builds gnu-debug-openmp or cuda-debug and running the failing test.

Related issues

• Related to: Teuchos: Address issus with calling Fortran LAPACK rountines from C++ on Power8 with GCC/gfortran #1208?

[bartlettroscoe]

What is strange about this is that we see only 1 failing Belos test and 2 failing Anasazi tests for the build Trilinos-atdm-white-ride-gnu-debug-openmp shown at:

• https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3373978

and only the one Belos test looks to be segfaulting. But with the corresponding build Trilinos-atdm-white-ride-gnu-opt-openmp shown at:

• https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3373731

there are 8 failing Belos tests and 50 failing Anasazi tests which a lot of them are segfaults but this this test TeuchosNumerics_LAPACK_test_MPI_1 is not failing in that build.

Again, I suspect this might be related to to #1208 that concluded that this compiler on this system is just broken when it comes to producing mixed-language programs.

[bartlettroscoe]

Note that this test actually passes in all of the opt builds as well as on white/ride as well as shown at:

• https://testing-vm.sandia.gov/cdash/testSummary.php?project=1&name=TeuchosNumerics_LAPACK_test_MPI_1&date=2018-03-23

So that compiler defect analyzed in #1208 does not seem to be impacting this test when compiler optimizations are turned on.

[bartlettroscoe]

Since this test is passing in all of the other builds, including all of the opt builds for each compiler on each machine, I think disabling this test for these debug builds on white/ride is not a bad thing.

Do any of you @trilinos/teuchos developers have a problem with that?

I will create a PR that disables this test for just the builds where it fails which are:

• Trilinos-atdm-white-ride-cuda-debug
• Trilinos-atdm-white-ride-gnu-debug-openmp

I will create a PR for this.

[bartlettroscoe]

I created PR #2447 that disables this test for just these builds.

Can some @trilinos/teuchos developer please approve this PR? Otherwise, since this is not really changing Trilinos at all (just the ATDM Trilinos test drivers), it can't possibly break Trilinos so I will merge in a few hours. I just want to see what the auto PR tester does for a PR like this. I suspect that because it touches a *.cmake file that will result in all of Trilinos getting tested (which actually will not be testing anything because that PR does not impact any of the auto PR builds, at least not yet).

[bartlettroscoe]

The PR #2447 has been merged. These failures should go away tomorrow. Putting in review.

[bartlettroscoe]

The test TeuchosNumerics_LAPACK_test_MPI_1 is newly missing from the builds Trilinos-atdm-white-ride-cuda-debug and Trilinos-atdm-white-ride-gnu-debug-openmp as shown at:

• https://testing-vm.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=3389503

and

• https://testing-vm.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=3389489

And this test is shown passing in all of the remaining builds in:

• https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-24&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=TeuchosNumerics_LAPACK_test_MPI_1

Therefore, closing this as complete.

[bartlettroscoe]

Closing as complete for real (I seems to always forget to click that "Close and comment" button).

[bartlettroscoe]

As part the updated process listed at:

• https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/ATDM+Builds+of+Trilinos#ATDMBuildsofTrilinos-Addressingfailures

I assigned the labels "Disabled Tests" and "Stalled" so I am reopening this issue. That way, people can be reminded that this is still a problem (we just avoided it by disabling tests for the ATDM builds).

[bartlettroscoe]

Given the updated NetLIB BLAS and LAPACK being used on 'white' and 'ride' (see #2454) I tried re-enabling the test TeuchosNumerics_LAPACK_test_MPI_1 for the 'debug' builds on 'white' and 'ride'. I tried this with the commit 7958ac7 I made locally and on 'white' I ran:

$ bsub -x -I -q rhel7F -n 16 \
   ./checkin-test-atdm.sh cuda-debug gnu-debug-openmp \
   --enable-packages=Teuchos --local-do-all

that returned:

FAILED (NOT READY TO PUSH): Trilinos: white22

Sat May  5 17:14:03 MDT 2018

Enabled Packages: Teuchos

Build test results:
-------------------
0) MPI_RELEASE_DEBUG_SHARED_PT => Test case MPI_RELEASE_DEBUG_SHARED_PT was not run! => Does not affect push readiness! (-1.00 min)
1) cuda-debug => FAILED: passed=130,notpassed=1 => Not ready to push! (1.49 min)
2) gnu-debug-openmp => FAILED: passed=130,notpassed=1 => Not ready to push! (0.97 min)

The failing tests were:

$ find . -maxdepth 2 -name ctest.out -exec grep -H "[(]Failed[)]" {} \;
./gnu-debug-openmp/ctest.out:   119 - TeuchosNumerics_LAPACK_test_MPI_1 (Failed)
./cuda-debug/ctest.out: 119 - TeuchosNumerics_LAPACK_test_MPI_1 (Failed)

The failing test output for the build gnu-debug-openmp/ showed:

Teuchos in Trilinos 12.13 (Dev)

GESV test ... passed!
LAPY2 test ... passed!
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 44747 on node white22 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

That is identical to the failure output reported above when this Issue was first created.

Also, I verified in the configure output that the new NetLIB BLAS and LAPACK installs are indeed being used as it showed:

Processing enabled TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF)
-- BLAS_LIBRARY_NAMES='blas blas_win32'
-- TPL_BLAS_LIBRARIES='-L/home/projects/pwr8-rhel73-lsf/netlib/3.8.0/gcc/5.4.0/lib;-lblas;-lgfortran;-lgomp;-lm'
Processing enabled TPL: LAPACK (enabled explicitly, disable with -DTPL_ENABLE_LAPACK=OFF)
-- LAPACK_LIBRARY_NAMES='lapack lapack_win32'
-- TPL_LAPACK_LIBRARIES='-L/home/projects/pwr8-rhel73-lsf/netlib/3.8.0/gcc/5.4.0/lib;-llapack;-lgfortran;-lgomp'

So whatever is causing this test to fail did not get fixed by updating BLAS and LAPACK.

Perhaps there is a real defect in the Teuchos LAPACK wrappers and/or the test code for this test?

[nmhamster]

@bartlettroscoe - to re-comment as well, the OpenBLAS tests for the packages we install (fairly limited number) do also all pass and do have C to FORTRAN calling included. That’s what makes me wonder if this might be a deeper issue too.

[fryeguy52]

This test is also failing on both debug builds on waterman, shown here those builds are:

• Trilinos-atdm-waterman-gnu-debug-openmp
• Trilinos-atdm-waterman-cuda-9.2-debug

They fail with similar output:

Teuchos in Trilinos 12.13 (Dev)

GESV test ... passed!
LAPY2 test ... passed!
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node waterman1 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Should we disable TeuchosNumerics_LAPACK_test_MPI_1 for these builds as well?

[hkthorn]

@fryeguy52 @bartlettroscoe Please let me know if the LAPACK test is still failing after the OpenMPI version is changed for the ATDM waterman testing.

[bartlettroscoe]

PR #4064 which enables the whole test TeuchosNumerics_LAPACK_test_MPI_1 but disables the single unit test for STEQR() merged to 'develop' on 12/18/2018. Next: Watch for test running and passing (minus STEQR() unit test) on 'release-debug' and 'opt' builds on 'white', 'ride', and 'waterman' on 12/19/2018 ...

Putting 'in review' awaiting results on CDash ...

[bartlettroscoe]

The test TeuchosNumerics_LAPACK_test_MPI_1 is now running and passing in all of the the 'debug' builds on 'white', 'ride', and 'waterman' as shown today in this query (note the usage of the experimental OR-block filter). For example, we can see this test newly passing in the build Trilinos-atdm-white-ride-cuda-9.2-debug with test results showing:

GESV test ... passed!
LAPY2 test ... passed!
LAMCH test ... [ Double-precision eps = 1.11022e-16, single-precision eps = 5.96046e-08 ] passed!
POTRF test ... passed!
POCON test ... passed!
POTRI test ... passed!
TRTRI test ... passed!
ILAENV test ... passed!
STEQR test ... SKIPPED!
End Result: TEST PASSED

Note STEQR test ... SKIPPED! in the output.

As of right now, all of the 'debug' builds (but not the 'release-debug' builds) on 'white', 'ride', and 'waterman' have already posted to CDash and we can see this test running and passing in 32 of the ATM Trilinos builds in this query.

And there are no more disables for this test in any of the ATDM Trilinos builds as shown on 'develop' just now by the commands:

$ git log-short --name-status -1
b0fd3c1 "Merge Pull Request #4083 from mhoemmen/Trilinos/Fix-4080"
Author: trilinos-autotester <[email protected]>
Date:   Wed Dec 19 06:39:32 2018 -0700 (47 minutes ago)

$ find cmake/std/atdm/ -name "*.cmake" -exec grep -nH TeuchosNumerics_LAPACK_test_MPI_1 {} \;
[empty]

Therefore, I think this is sufficient evidence that this test is running and passing in every ATM Trilinos build, including all of the 'debug' builds on 'white', 'ride', and 'waterman'. The only thing that is not running is the STEQR() unit test and that unit test is only disabled on the IBM Power8 'white/ride'and Power9 'waterman' machines.

I have added the "Disabled Tests" label but I actually think now that we have got Trilinos to stop using this LAPACK STEQR() function that we should actually close this issue. Agree?

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions]

This issue was closed due to inactivity for 395 days.