Enable TeuchosNumerics_LAPACK_test_MPI_1 test in 'debug' Power8 and Power8 builds and disable just the STEQR() test #4064

bartlettroscoe · 2018-12-17T16:21:45Z

CC: @trilinos/teuchos, @fryeguy52

Description

Enable TeuchosNumerics_LAPACK_test_MPI_1 test in 'debug' Power8 and Power8 builds and disable just the STEQR() test.

NOTE: This test was being disabled in some release-debug builds where it did not need to be disabled. In fact, the STEQR() function does not segfault when using optimized compiler options (which are used in a release-debug build).

Motivation and Context

We need to be running LAPACK tests for the functions being used by Trilinos and not disable them all (see #2410).

How Has This Been Tested?

On 'ride' I did:

$ bsub -x -Is -q rhel7F -n 16 \
  ./checkin-test-atdm.sh \
  cuda-9.2-debug gnu-openmp-debug \
  --enable-packages=TeuchosNumerics --local-do-all

which returned:

PASSED (NOT READY TO PUSH): Trilinos: ride10

Mon Dec 17 08:51:43 MST 2018

Enabled Packages: TeuchosNumerics

Build test results:
-------------------
1) cuda-9.2-debug => passed: passed=16,notpassed=0 (1.50 min)
2) gnu-openmp-debug => passed: passed=16,notpassed=0 (0.60 min)

and showed:

$ for build_name in cuda-9.2-debug gnu-openmp-debug ; do grep -nH TeuchosNumerics_LAPACK_test ${build_name}/ctest.out ; done | grep "Pass"

cuda-9.2-debug/ctest.out:33:16/16 Test  #9: TeuchosNumerics_LAPACK_test_MPI_1 ..................   Passed    0.43 sec
gnu-openmp-debug/ctest.out:30:13/16 Test  #9: TeuchosNumerics_LAPACK_test_MPI_1 ..................   Passed    1.88 sec

And I ran this on 'waterman' using:

$ bsub -x -Is -n 20 \
  ./checkin-test-atdm.sh \
  cuda-9.2-debug cuda-9.2-release-debug gnu-openmp-release-debug \
  --enable-packages=TeuchosNumerics --local-do-all

which returned:

PASSED (NOT READY TO PUSH): Trilinos: waterman1

Mon Dec 17 08:56:33 MST 2018

Enabled Packages: TeuchosNumerics

Build test results:
-------------------
1) cuda-9.2-debug => passed: passed=16,notpassed=0 (1.20 min)
2) cuda-9.2-release-debug => passed: passed=16,notpassed=0 (1.34 min)
3) gnu-openmp-release-debug => passed: passed=16,notpassed=0 (1.08 min)

and showed:

$ for build_name in cuda-9.2-debug cuda-9.2-release-debug gnu-openmp-release-debug ; do grep -nH TeuchosNumerics_LAPACK_test ${build_name}/ctest.out ; done | grep "Pass"

cuda-9.2-debug/ctest.out:32:15/16 Test  #9: TeuchosNumerics_LAPACK_test_MPI_1 ..................   Passed    2.20 sec
cuda-9.2-release-debug/ctest.out:29:12/16 Test  #9: TeuchosNumerics_LAPACK_test_MPI_1 ..................   Passed    2.24 sec
gnu-openmp-release-debug/ctest.out:26: 9/16 Test  #9: TeuchosNumerics_LAPACK_test_MPI_1 ..................   Passed    4.10 sec

Checklist

My commit messages mention the appropriate GitHub issue numbers.
All new and existing tests passed.
No new compiler warnings were introduced.

This is the only function that continues to segfault on Power8 and Power9 builds using NETLIB LAPACK in debug builds. All of the Trilinos code downstream has removed a dependence on this LAPACK function. This will allow the rest of the LAPACK tests to be run in Power8 and Power9 'debug' builds.

…EQR() test (trilinos#2410) At least this way we are running the Teuchos LAPACK tests for the LAPACK functions being used downstream in Trilinos.

trilinos-autotester · 2018-12-17T16:34:44Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

hkthorn

Looks good to me. This is a good idea to move forward with improving LAPACK test coverage.

trilinos-autotester · 2018-12-17T18:51:35Z

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job Trilinos_pullrequest_intel_17.0.1 to start: Total Wait = 603

Other jobs have been previously started - We must stop them...

mhoemmen · 2018-12-17T21:48:35Z

Thanks @bartlettroscoe ! :D

trilinos-autotester · 2018-12-17T22:49:20Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

trilinos-autotester · 2018-12-17T23:20:50Z

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job Trilinos_pullrequest_gcc_7.2.0 to start: Total Wait = 603

Other jobs have been previously started - We must stop them...
Terminating Build Tpetra: Don't use atomic ops for unpack unless collisions could happen #1924 of Jenkins job Trilinos_pullrequest_intel_17.0.1 on Jenkins server at https://ascic-jenkins.sandia.gov
Terminating Build Test Stratimikos_Belos_GCRODR_strattest_MPI_4 failed in newly promoted build Trilinos-atdm-white-ride-gnu-debug-openmp #2491 of Jenkins job Trilinos_pullrequest_gcc_4.9.3 on Jenkins server at https://ascic-jenkins.sandia.gov
Terminating Build Tempus: Theta Stepper #2123 of Jenkins job Trilinos_pullrequest_gcc_4.8.4 on Jenkins server at https://ascic-jenkins.sandia.gov
Terminating Build Teuchos: Remove deprecated reduceAllAndScatter #420 of Jenkins job Trilinos_pullrequest_gcc_4.9.3_SERIAL on Jenkins server at https://ascic-jenkins.sandia.gov

trilinos-autotester · 2018-12-18T04:31:02Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

trilinos-autotester · 2018-12-18T04:33:29Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 1933
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 2499
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 2131
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 424
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 38
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)

Pull Request Author: bartlettroscoe

trilinos-autotester · 2018-12-18T09:09:47Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 1933
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 2499
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 2131
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 424
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 38
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4064
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2410-power-disable-lapack-steqr
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`71e9543`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`f94ba63`

CDash Test Results for PR# 4064.

trilinos-autotester · 2018-12-18T09:10:06Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ mhoemmen ]!

trilinos-autotester · 2018-12-18T09:10:13Z

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

trilinos-autotester · 2018-12-18T09:10:17Z

Merge on Pull Request# 4064: IS A SUCCESS - Pull Request successfully merged

bartlettroscoe added 2 commits December 17, 2018 08:53

Enable TeuchosNumerics_LAPACK_test_MPI_1 test and disable just the ST…

71e9543

…EQR() test (trilinos#2410) At least this way we are running the Teuchos LAPACK tests for the LAPACK functions being used downstream in Trilinos.

bartlettroscoe requested review from fryeguy52 and hkthorn December 17, 2018 16:21

bartlettroscoe added the stage: in progress Work on the issue has started label Dec 17, 2018

bartlettroscoe requested a review from mhoemmen December 17, 2018 16:22

bartlettroscoe added this to the Initial cleanup of new ATDM builds of Trilinos milestone Dec 17, 2018

bartlettroscoe added the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Dec 17, 2018

hkthorn reviewed Dec 17, 2018

View reviewed changes

bartlettroscoe mentioned this pull request Dec 17, 2018

Trilinos auto PR tester stability issues #3276

Closed

bartlettroscoe added the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Dec 17, 2018

mhoemmen approved these changes Dec 17, 2018

View reviewed changes

trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Dec 17, 2018

bartlettroscoe added the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Dec 18, 2018

trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Dec 18, 2018

trilinos-autotester merged commit 0079967 into trilinos:develop Dec 18, 2018

trilinos-autotester removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Dec 18, 2018

bartlettroscoe removed the stage: in progress Work on the issue has started label Dec 18, 2018

bartlettroscoe mentioned this pull request Dec 18, 2018

Test TeuchosNumerics_LAPACK_test_MPI_1 fails in all 'debug' builds on power8 'ride' #2410

Closed

bartlettroscoe mentioned this pull request Oct 25, 2019

TeuchosNumerics_LAPACK_test_MPI_1 failing in build Trilinos-atdm-waterman-cuda-9.2-debug starting 2019-10-21 #6166

Closed

trilinos-autotester mentioned this pull request Feb 20, 2020

Tempus fix imex initialization #6864

Merged

Enable TeuchosNumerics_LAPACK_test_MPI_1 test in 'debug' Power8 and Power8 builds and disable just the STEQR() test #4064

Enable TeuchosNumerics_LAPACK_test_MPI_1 test in 'debug' Power8 and Power8 builds and disable just the STEQR() test #4064

Conversation

bartlettroscoe commented Dec 17, 2018

Description

Motivation and Context

How Has This Been Tested?

Checklist

trilinos-autotester commented Dec 17, 2018

hkthorn left a comment

Choose a reason for hiding this comment

trilinos-autotester commented Dec 17, 2018

mhoemmen commented Dec 17, 2018

trilinos-autotester commented Dec 17, 2018

trilinos-autotester commented Dec 17, 2018

trilinos-autotester commented Dec 18, 2018

trilinos-autotester commented Dec 18, 2018

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Jenkins Parameters

Using Repos:

trilinos-autotester commented Dec 18, 2018

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Jenkins Parameters

trilinos-autotester commented Dec 18, 2018

trilinos-autotester commented Dec 18, 2018

trilinos-autotester commented Dec 18, 2018