Disable some individual Kokkos and KokkosKernels tests on a few more full debug builds and disable the Belos_pseudo_stochastic_pcg_hb_[0,1]_MPI_4 tests on a few more platforms #2964

bartlettroscoe · 2018-06-18T14:25:50Z

Description

This PR branch contains commits to disable a few of the individual Kokkos and KokkosKernels unit tests for some full debug builds (see the careful analysis and identification of these individual unit tests in #2827 (comment)) (see #2827).

This PR branch also contains commits to disable the tests Belos_pseudo_stochastic_pcg_hb_[0,1]_MPI_4 in a few other builds on white/ride where then have been seen to be randomly failing with hitting the 100 iteration max (see #2920 (comment) and #2920 (comment)).

I also updated some documentation in the cmake/std/atdm/README.md file about using checkin-test-atdm.sh on hansen/shiller.

Motivation and Context

These tests fairly regularly fail in automated ATDM testing. Going forward we can't tolerate known randomly failing tests. Such tests will destroy automated processes to update Trilinos for that ATDM APP codes.

How Has This Been Tested?

I ran the builds and tests manually on 'shiller' and 'ride' and verified that with these changes all of the Kokkos and KokkosKernels tests in the debug builds far under the 600 sec timeout (but one test was as high as 447 sec).

See details of the testing below.

DETAILED TEST RESULTS: (click to expand)

A) Testing on 'shiller'

$ ./checkin-test-atdm.sh \
  gnu-debug-serial gnu-debug-openmp intel-debug-serial intel-debug-openmp \
  cuda-8.0-debug cuda-9.0-debug \
  --enable-packages=Kokkos,KokkosKernels --configure
$ /usr/bin/srun ./checkin-test-atdm.sh \
  gnu-debug-serial gnu-debug-openmp intel-debug-serial intel-debug-openmp \
  cuda-8.0-debug cuda-9.0-debug \
  --enable-packages=Kokkos,KokkosKernels --build --test

returned

PASSED (NOT READY TO PUSH): Trilinos: shiller02

Sat Jun 16 14:00:58 MDT 2018

Enabled Packages: Kokkos, KokkosKernels

Build test results:
-------------------
0) MPI_RELEASE_DEBUG_SHARED_PT_OPENMP => Test case MPI_RELEASE_DEBUG_SHARED_PT_OPENMP was not run! => Does not affect push readiness! (-1.00 min)
1) gnu-debug-serial => passed: passed=28,notpassed=0 (3.41 min)
2) gnu-debug-openmp => passed: passed=35,notpassed=0 (5.22 min)
3) intel-debug-serial => passed: passed=28,notpassed=0 (4.18 min)
4) intel-debug-openmp => passed: passed=35,notpassed=0 (4.75 min)
5) cuda-8.0-debug => passed: passed=35,notpassed=0 (5.39 min)
6) cuda-9.0-debug => passed: passed=35,notpassed=0 (4.58 min)

Using the script:

$ cat print_expensive_tests.sh
#!/bin/bash

ctest_out_file=$1
n_most_expensive_tests=10

echo
echo "***"
echo "*** $ctest_out_file: $n_most_expensive_tests most expensive tests"
echo "***"
echo

cat $ctest_out_file | grep "Test" | grep "Passed" | sort -rn -k 7 | head -n $n_most_expensive_tests

Looking for expensive tests with:

$for build_name in gnu-debug-serial gnu-debug-openmp intel-debug-serial intel-debug-openm cuda-8.0-debug cuda-9.0-debug ; do ./print_expensive_; done | lessld_name/ctest.out ;

***
*** gnu-debug-serial/ctest.out: 10 most expensive tests
***

28/28 Test #26: KokkosKernels_sparse_serial_MPI_1 ................   Passed  204.28 sec
27/28 Test #23: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  203.75 sec
26/28 Test #27: KokkosKernels_graph_serial_MPI_1 .................   Passed  177.73 sec
25/28 Test #25: KokkosKernels_blas_serial_MPI_1 ..................   Passed  105.65 sec
24/28 Test  #1: KokkosCore_UnitTest_Serial_MPI_1 .................   Passed   77.93 sec
23/28 Test #24: KokkosAlgorithms_UnitTest_MPI_1 ..................   Passed   51.48 sec
16/28 Test  #4: KokkosCore_UnitTest_PushFinalizeHook_terminate ...   Passed   11.30 sec
 7/28 Test #11: KokkosCore_UnitTest_DefaultInit_7_MPI_1 ..........   Passed   10.11 sec
 8/28 Test #10: KokkosCore_UnitTest_DefaultInit_6_MPI_1 ..........   Passed   10.10 sec
 9/28 Test #12: KokkosCore_UnitTest_DefaultInit_8_MPI_1 ..........   Passed   10.09 sec

***
*** gnu-debug-openmp/ctest.out: 10 most expensive tests
***

35/35 Test #33: KokkosKernels_sparse_serial_MPI_1 ................   Passed  313.07 sec
34/35 Test #25: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  236.03 sec
33/35 Test #29: KokkosKernels_sparse_openmp_MPI_1 ................   Passed  225.57 sec
32/35 Test #34: KokkosKernels_graph_serial_MPI_1 .................   Passed  199.32 sec
31/35 Test #26: KokkosContainers_UnitTest_OpenMP_MPI_1 ...........   Passed  139.10 sec
30/35 Test #30: KokkosKernels_graph_openmp_MPI_1 .................   Passed  121.48 sec
29/35 Test #32: KokkosKernels_blas_serial_MPI_1 ..................   Passed  114.06 sec
28/35 Test #27: KokkosAlgorithms_UnitTest_MPI_1 ..................   Passed   98.52 sec
27/35 Test  #1: KokkosCore_UnitTest_Serial_MPI_1 .................   Passed   83.38 sec
26/35 Test  #2: KokkosCore_UnitTest_OpenMP_MPI_1 .................   Passed   73.56 sec

***
*** intel-debug-serial/ctest.out: 10 most expensive tests
***

28/28 Test #26: KokkosKernels_sparse_serial_MPI_1 ................   Passed  250.78 sec
27/28 Test #23: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  226.06 sec
26/28 Test #27: KokkosKernels_graph_serial_MPI_1 .................   Passed  212.57 sec
25/28 Test #25: KokkosKernels_blas_serial_MPI_1 ..................   Passed  114.90 sec
24/28 Test  #1: KokkosCore_UnitTest_Serial_MPI_1 .................   Passed   90.55 sec
23/28 Test #24: KokkosAlgorithms_UnitTest_MPI_1 ..................   Passed   44.57 sec
20/28 Test  #4: KokkosCore_UnitTest_PushFinalizeHook_terminate ...   Passed    4.39 sec
 9/28 Test  #3: KokkosCore_UnitTest_PushFinalizeHook_MPI_1 .......   Passed    2.98 sec
 4/28 Test #15: KokkosCore_UnitTest_DefaultInit_11_MPI_1 .........   Passed    2.43 sec
 3/28 Test #11: KokkosCore_UnitTest_DefaultInit_7_MPI_1 ..........   Passed    2.43 sec

***
*** intel-debug-openm/ctest.out: 10 most expensive tests
***

cat: intel-debug-openm/ctest.out: No such file or directory

***
*** cuda-8.0-debug/ctest.out: 10 most expensive tests
***

35/35 Test #33: KokkosKernels_sparse_serial_MPI_1 ................   Passed  323.10 sec
34/35 Test #25: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  264.12 sec
33/35 Test #34: KokkosKernels_graph_serial_MPI_1 .................   Passed  214.00 sec
32/35 Test #29: KokkosKernels_sparse_cuda_MPI_1 ..................   Passed  192.78 sec
 5/35 Test  #2: KokkosCore_UnitTest_Cuda_MPI_1 ...................   Passed  118.32 sec
 4/35 Test #32: KokkosKernels_blas_serial_MPI_1 ..................   Passed  116.83 sec
 2/35 Test  #1: KokkosCore_UnitTest_Serial_MPI_1 .................   Passed   84.91 sec
 9/35 Test #27: KokkosAlgorithms_UnitTest_MPI_1 ..................   Passed   61.74 sec
 1/35 Test #30: KokkosKernels_graph_cuda_MPI_1 ...................   Passed   61.74 sec
14/35 Test #28: KokkosKernels_blas_cuda_MPI_1 ....................   Passed   31.59 sec

***
*** cuda-9.0-debug/ctest.out: 10 most expensive tests
***

35/35 Test #33: KokkosKernels_sparse_serial_MPI_1 ................   Passed  274.38 sec
34/35 Test #34: KokkosKernels_graph_serial_MPI_1 .................   Passed  268.87 sec
33/35 Test #25: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  238.93 sec
32/35 Test  #2: KokkosCore_UnitTest_Cuda_MPI_1 ...................   Passed  148.07 sec
31/35 Test #29: KokkosKernels_sparse_cuda_MPI_1 ..................   Passed  141.35 sec
 7/35 Test #32: KokkosKernels_blas_serial_MPI_1 ..................   Passed  115.60 sec
 2/35 Test  #1: KokkosCore_UnitTest_Serial_MPI_1 .................   Passed   84.65 sec
 1/35 Test #27: KokkosAlgorithms_UnitTest_MPI_1 ..................   Passed   58.47 sec
 3/35 Test #28: KokkosKernels_blas_cuda_MPI_1 ....................   Passed   44.03 sec
 4/35 Test #30: KokkosKernels_graph_cuda_MPI_1 ...................   Passed   24.78 sec

All of those look well under 600 sec.

**B) Testing on 'ride':

$ bsub -x -Is -q rhel7F -n 16 ctest -j16 \
  ./checkin-test-atdm.sh gnu-debug-openmp cuda-debug \
  --enable-packages=Kokkos,KokkosKernels --local-do-all

returned:

PASSED (NOT READY TO PUSH): Trilinos: ride11

Sat Jun 16 13:43:17 MDT 2018

Enabled Packages: Kokkos, KokkosKernels

Build test results:
-------------------
0) MPI_RELEASE_DEBUG_SHARED_PT_OPENMP => Test case MPI_RELEASE_DEBUG_SHARED_PT_OPENMP was not run! => Does not affect push readiness! (-1.00 min)
1) gnu-debug-openmp => passed: passed=35,notpassed=0 (7.66 min)
2) cuda-debug => passed: passed=35,notpassed=0 (5.94 min)

Looking for expensive tests with:

$ for build_name in gnu-debug-openmp cuda-debug ; do ./print_expensive_tests.sh $build_name/ctest.out ; done

***
*** gnu-debug-openmp/ctest.out: 10 most expensive tests
***

35/35 Test #29: KokkosKernels_sparse_openmp_MPI_1 ................   Passed  447.45 sec
34/35 Test #33: KokkosKernels_sparse_serial_MPI_1 ................   Passed  366.10 sec
33/35 Test #26: KokkosContainers_UnitTest_OpenMP_MPI_1 ...........   Passed  354.53 sec
32/35 Test #30: KokkosKernels_graph_openmp_MPI_1 .................   Passed  347.01 sec
31/35 Test #34: KokkosKernels_graph_serial_MPI_1 .................   Passed  323.47 sec
30/35 Test #25: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  286.63 sec
29/35 Test #32: KokkosKernels_blas_serial_MPI_1 ..................   Passed  274.02 sec
28/35 Test #27: KokkosAlgorithms_UnitTest_MPI_1 ..................   Passed  236.46 sec
27/35 Test  #1: KokkosCore_UnitTest_Serial_MPI_1 .................   Passed  200.46 sec
26/35 Test  #2: KokkosCore_UnitTest_OpenMP_MPI_1 .................   Passed  196.63 sec

***
*** cuda-debug/ctest.out: 10 most expensive tests
***

35/35 Test #33: KokkosKernels_sparse_serial_MPI_1 ................   Passed  322.17 sec
34/35 Test #34: KokkosKernels_graph_serial_MPI_1 .................   Passed  275.03 sec
33/35 Test #25: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  267.83 sec
32/35 Test  #2: KokkosCore_UnitTest_Cuda_MPI_1 ...................   Passed  259.72 sec
31/35 Test #29: KokkosKernels_sparse_cuda_MPI_1 ..................   Passed  240.50 sec
 3/35 Test  #1: KokkosCore_UnitTest_Serial_MPI_1 .................   Passed  156.26 sec
 2/35 Test #27: KokkosAlgorithms_UnitTest_MPI_1 ..................   Passed  136.87 sec
 1/35 Test #32: KokkosKernels_blas_serial_MPI_1 ..................   Passed  136.85 sec
30/35 Test #28: KokkosKernels_blas_cuda_MPI_1 ....................   Passed   49.27 sec
14/35 Test #30: KokkosKernels_graph_cuda_MPI_1 ...................   Passed   39.16 sec

The most expensive test was 447 sec but that is pretty far below 600 sec so hopefully we will be okay.

Checklist

My commit messages mention the appropriate GitHub issue numbers.
All new and existing tests passed.

…some debug builds (trilinos#2827) These very targeted disables should allow these tests to all complete in well under 10 minutes in all of these debug builds on all of these platforms. See the diffs to see exactly what unit tests are disabled in what unit test executables in what builds on what platforms. For details on why these are being disabled, see trilinos#2827.

…PI_4 in gnu-debug-openmp build on white/ride (trilinos#2920) This failed with maxing out at 100 iterations today in the Trilinos-atdm-white-ride-gnu-debug-openmp build where otherwise it converges at 87 iterations. Therefore, we are disabling this test like we did for the cuda-debug build. For more details see trilinos#2920.

…PI_4 in cuda-opt build on white/ride (trilinos#2920) This failed with maxing out at 100 iterations on 6/17/2018 the Trilinos-atdm-white-ride-cupda-top build where otherwise it converges at 87 iterations. Therefore, we are disabling this test like we did for some other builds on white/ride. For more details see trilinos#2920.

…n/shiller (TRIL-211) For some reason, it can't find Ninja when using 'srun' to configure and build on the compute node. But for some crazy reason, you can use 'srun' to configure, build, and run tests on the compute node when using the Jenkins driver script. Very strange.

trilinos-autotester · 2018-06-18T14:44:27Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2018-06-18T15:12:36Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 874
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 591
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 139
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)

Pull Request Author: bartlettroscoe

mhoemmen

At some point we simply won't be running any more tests ;-)

bartlettroscoe · 2018-06-18T17:00:06Z

@mhoemmen said:

At some point we simply won't be running any more tests ;-)

Before we started this effort, zero Belos tests were run on any of these ATDM platforms. We would have to disable a lot of tests before we get anywhere near that.

The alternative is to "Stop the Line" and fix these things as they come up (since the problems with this code has always been there so the initial job was not finished) but it is hard to justify that when there are more urgent things to do. From the ATDM perspective, I think that disabling many of these tests does not loose any testing really at all that is protecting ATDM APP customers. That is my focus right now.

mhoemmen · 2018-06-18T17:36:30Z

@bartlettroscoe wrote:

Before we started this effort, zero Belos tests were run on any of these ATDM platforms. We would have to disable a lot of tests before we get anywhere near that.

I'm just being snarky -- and it's definitely not about you. Belos historically has seen intermittent test failures. The tests (and at least one solver) have tended to use random numbers, and also are a bit sensitive to residual computations.

bartlettroscoe · 2018-06-18T21:17:28Z

@trilinos/framework,

CDash is showing all three of the auto PR builds started over 6 hours ago shown above passing on CDash at:

https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=Pull%20Request&field2=buildstarttime&compare2=84&value2=now&field3=buildname&compare3=65&value3=PR-2964

But for some reason, the auto PR tester has not added a comment to to this PR that testing completed successfully yet (therefore, allowing the merge). From looking at CDash, all of the builds should have completed in under 4 hours and therefore we should have gotten results posted to this PR about 2 hours ago (plus 20 minutes perhaps).

Is there some issue with the auto PR tester?

bartlettroscoe · 2018-06-18T21:31:45Z

To give as shot at merging this PR today (so that it can run in the ATDM Trilinos builds tomorrow morning), I will put on the "AT: RETEST" label and hope it all works this time (and then I can merge at 10 PM ET hopefully).

trilinos-autotester · 2018-06-18T22:16:26Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Resetting Testing Status

trilinos-autotester · 2018-06-18T22:18:51Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 882
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 599
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 147
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)

Pull Request Author: bartlettroscoe

trilinos-autotester · 2018-06-19T02:19:12Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 882
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 599
Status: FAILED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 147
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

CDash Test Results for PR# 2964.

trilinos-autotester · 2018-06-19T06:35:11Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 892
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 609
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 157
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)

Pull Request Author: bartlettroscoe

trilinos-autotester · 2018-06-19T10:17:25Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 892
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 609
Status: FAILED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 157
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

CDash Test Results for PR# 2964.

bartlettroscoe · 2018-06-19T11:58:52Z

@trilinos/framework,

The GCC 4.8.4 build in the PR testing failed twice late yesterday and early this morning. There is no way my changes in this PR can break the build since they only impact ATDM Trilinos builds.

It looks like someone tried to switch over to use the setting for using OpenMP with mpiexec --no-bind none -np <n> <exec_name> ... but they forgot to upgrade to OpenMP 1.10.1 (see #2788). You can see this, for example, at:

https://testing-vm.sandia.gov/cdash/testDetails.php?test=48809072&build=3628668

which shows the command-line:

/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.8.4/openmpi/1.6.5/bin/mpiexec \"--bind-to\" \"none\" \"-np\" \"1\" \"/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/pull_request_test/packages/teuchos/comm/test/Time/TeuchosComm_Time_test.exe\" \"-v\"

How did this change make it into the 'develop' branch? This could not have passed a PR build.

trilinos-autotester · 2018-06-19T14:16:10Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 899
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 616
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 164
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)

Pull Request Author: bartlettroscoe

trilinos-autotester · 2018-06-19T18:22:37Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 899
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 616
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 164
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2964
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	2827-more-kokks-kokkos-kernels-unit-test-disasbles
TRILINOS_SOURCE_REPO	https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA	`a572d07`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`04aeeb3`

CDash Test Results for PR# 2964.

trilinos-autotester · 2018-06-19T18:23:04Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ mhoemmen ]!

trilinos-autotester · 2018-06-19T18:23:23Z

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - Master Automerge is disabled (in .cfg file)

same as what was done in trilinos#2964

bartlettroscoe added 4 commits June 18, 2018 08:03

bartlettroscoe added type: bug The primary issue is a bug in Trilinos code or tests pkg: Kokkos pkg: Belos pkg: KokkosKernels client: ATDM Any issue primarily impacting the ATDM project labels Jun 18, 2018

bartlettroscoe added this to the Keep promoted "ATDM" builds of Trilinos clean milestone Jun 18, 2018

bartlettroscoe requested review from mhoemmen, ibaned and fryeguy52 June 18, 2018 14:25

mhoemmen approved these changes Jun 18, 2018

View reviewed changes

bartlettroscoe added the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Jun 18, 2018

trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Jun 18, 2018

This was referenced Jun 18, 2018

SEACAS tests failing in ATDM builds on mutrino #2815

Closed

Failing MueLu tests on mutrino in ATDM configuration #2839

Closed

This was referenced Jun 19, 2018

Panzer: Fixing integration values side qp construction #2969

Merged

Trilinos GCC 4.8.4 PR build and running of tests is broken #2974

Closed

bartlettroscoe merged commit 23cfe26 into trilinos:develop Jun 19, 2018

bartlettroscoe mentioned this pull request Jun 21, 2018

New Kokkos, KokkosKernels, and Panzer test failures on CUDA 8.0 and CUDA 9.0 builds after Kokkos and KokkosKernels update #2827

Closed

fryeguy52 added a commit to fryeguy52/Trilinos that referenced this pull request Oct 3, 2018

Disable slow running unit test on waterman debug builds (trilinos#2827)

1ee23e1

same as what was done in trilinos#2964

fryeguy52 mentioned this pull request Oct 3, 2018

Atdm waterman disables #3559

Merged

fryeguy52 added a commit to fryeguy52/Trilinos that referenced this pull request Oct 8, 2018

Disable slow running unit test on waterman debug builds (trilinos#2827)

837804d

same as what was done in trilinos#2964

bartlettroscoe mentioned this pull request Oct 16, 2018

Add support for 'release-debug' build for ATDM Trilinos builds and use to avoid timeouts #3633

Closed

tjfulle pushed a commit to tjfulle/Trilinos that referenced this pull request Dec 6, 2018

Disable slow running unit test on waterman debug builds (trilinos#2827)

d513a48

same as what was done in trilinos#2964

trilinos-autotester mentioned this pull request Dec 10, 2020

Framework: Enabling the 'autocloser' github actions script #8445

Merged

Disable some individual Kokkos and KokkosKernels tests on a few more full debug builds and disable the Belos_pseudo_stochastic_pcg_hb_[0,1]_MPI_4 tests on a few more platforms #2964

Disable some individual Kokkos and KokkosKernels tests on a few more full debug builds and disable the Belos_pseudo_stochastic_pcg_hb_[0,1]_MPI_4 tests on a few more platforms #2964

Conversation

bartlettroscoe commented Jun 18, 2018

Description

Motivation and Context

How Has This Been Tested?

Checklist

trilinos-autotester commented Jun 18, 2018

trilinos-autotester commented Jun 18, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Using Repos:

mhoemmen left a comment

Choose a reason for hiding this comment

bartlettroscoe commented Jun 18, 2018

mhoemmen commented Jun 18, 2018

bartlettroscoe commented Jun 18, 2018

bartlettroscoe commented Jun 18, 2018

trilinos-autotester commented Jun 18, 2018

trilinos-autotester commented Jun 18, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Using Repos:

trilinos-autotester commented Jun 19, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

trilinos-autotester commented Jun 19, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Using Repos:

trilinos-autotester commented Jun 19, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

bartlettroscoe commented Jun 19, 2018

trilinos-autotester commented Jun 19, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters