Disable randomly failing Tpetra test (#10847), remove FORCE from all test disables (#10870) #10871

bartlettroscoe · 2022-08-13T02:01:17Z

CC: @trilinos/tpetra

Internal Issues

TRILINOSHD-168

Description

This PR disables the randomly failing tests TpetraTSQR_SequentialTsqr_[non]contiguousCacheBlocks_MPI_1 that is taking out all kinds of PR iterations (see #10847). The PR also removes 'FORCE' from all test disables (see #10870).

How was this tested?

Using the scripts:

load-env-and-cmake-frag-file.sh:

export TRILINOS_DIR=/fgs/rabartl/Trilinos.base/Trilinos
FULL_BUILD_NAME=rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables

if [[ -e GenConfigSettings.cmake ]] ; then
  echo "Remvoing existing file GenConfigSettings.cmake ..."
  rm GenConfigSettings.cmake
fi

export WORKSPACE=$TRILINOS_DIR/..

source $TRILINOS_DIR/packages/framework/GenConfig/gen-config.sh \
--cmake-fragment GenConfigSettings.cmake \
$FULL_BUILD_NAME \
--force \
$TRILINOS_DIR

unset FULL_BUILD_NAME
unset WORKSPACE

# NOTE Above, it is an undocumented requirment that the env vars TRILINOS_DIR
# and WORKSPACE be set!

and:

do-configure:

if [[ -e CMakeCache.txt ]] ; then
  echo "Removing CMakeCache.txt ..."
  rm CMakeCache.txt
fi
if [[ -d CMakeFiles ]] ; then
  echo "Removing CMakeFiles ..."
  rm -r CMakeFiles
fi
cmake \
-G Ninja \
-C GenConfigSettings.cmake \
-D Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES=OFF \
-D Trilinos_ENABLE_TESTS=ON \
-D Trilinos_TRACE_ADD_TEST=ON \
"$@" \
$TRILINOS_DIR

I configured on 'ascicgpu17' with:

$ ssh ascicgpu17

$ cd /fgs/rabartl/Trilinos.base/BUILDS/PR/rhel7_sems-cuda-11.4.2/

$ . load-env-and-cmake-frag-file.sh

$ time ./do-configure -DTrilinos_ENABLE_Tpetra=ON &> configure.out

real    0m55.633s
user    0m22.293s
sys     0m19.367s

and the configure output showed:

$ grep TpetraTSQR_SequentialTsqr_ configure.out 
-- TpetraTSQR_SequentialTsqr_contiguousCacheBlocks_MPI_1: NOT added test because TpetraTSQR_SequentialTsqr_contiguousCacheBlocks_MPI_1_DISABLE='ON'!
-- TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1: NOT added test because TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1_DISABLE='ON'!

You *NEVER* want to force disable a test. If you force disable it, the developer has no way to manually enable that test in order to fix it.

… just on ascicgpu PR build (#10847)

bartlettroscoe · 2022-08-13T02:35:09Z

FYI: I merged this branch into the topic branch for PR #10808 so we can see how this works in practice.

trilinos-autotester · 2022-08-13T06:42:05Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2022-08-13T06:44:05Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Build Num: 860
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Build Num: 861
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

Build Num: 859
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_intel-17.0.1

Build Num: 860
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_clang-10.0.0

Build Num: 864
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

Build Num: 876
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	vortex
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_python3

Build Num: 688
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	ascic
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

Build Num: 396
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	GPU
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: bartlettroscoe

trilinos-autotester · 2022-08-13T08:55:31Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Build Num: 860
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Build Num: 861
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

Build Num: 859
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_intel-17.0.1

Build Num: 860
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_clang-10.0.0

Build Num: 864
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

Build Num: 876
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	vortex
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_python3

Build Num: 688
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	ascic
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

Build Num: 396
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	GPU
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`6e31532`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

CDash Test Results for PR# 10871.

trilinos-autotester · 2022-08-13T08:55:48Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

trilinos-autotester · 2022-08-13T08:55:54Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

bartlettroscoe · 2022-08-13T16:43:01Z

FYI: I merged this branch into the branch in PR #10808 so this will get tested the next time that PR build runs.

NOTE: Trilinos is not testing updates to these files in PR testing. That is not good!

bartlettroscoe · 2022-08-14T01:47:40Z

FYI: I merged this branch into the branch in PR #10808 so this will get tested the next time that PR build runs.

NOTE: Trilinos is not testing updates to these files in PR testing. That is not good!

NOTE: A PR test iteration for PR #10808 on 'ascicgpu050` has started and we see in the configure output here for the build PR-10808-test-rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables-407 showing:

- TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1: NOT added test because TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1_DISABLE='ON'!

So that test will not run in that CUDA-11 build.

But looking at the configure output [here](https://trilinos-cdash.sandia.gov/build/378206/configure for the CUDA build on 'vortex' in the build PR-10808-test-ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables-887 we see:

-- TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1: Added test (BASIC, NUM_MPI_PROCS=1, PROCESSORS=1)!

So that is proof that test TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1 is only being disabled in that CUDA-11 builds where it is randomly failing but not the other builds.

bartlettroscoe · 2022-08-14T01:50:06Z

Hello @jhux2, @csiefer2, and @cgcgcg, can one of you please approve this PR so that it can merge? This will remove this test from taking out PR builds due to its random failing for the CUDA-11 build on the ascicgpu machines but keeps the test running in all other builds, including the CUDA build vortex (see above).

bartlettroscoe · 2022-08-14T03:08:17Z

The impossible seems to have occurred. The test TpetraTSQR_SequentialTsqr_contiguousCacheBlocks_MPI_1 was reported to have run and filed in the build PR-10808-test-rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables-407 but that test is shown disabled in the configure output showing:

-- TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1: NOT added test because TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1_DISABLE='ON'!

I will try to reproduce this on the machine and see what happens.

bartlettroscoe · 2022-08-14T03:29:55Z

I will try to reproduce this on the machine and see what happens.

For the Trilinos version from PR #10808

$ ssh ascicgpu17

$ cd /fgs/rabartl/Trilinos.base/Trilinos/

$ git log-short --name-status -1
3dac90f5fda "Merge remote-tracking branch 'github/10865-ats2-disable-shylu' into 10807-kokkos-kernels-cublas (#10865)"
Author: Roscoe A. Bartlett <[email protected]>
Date:   Fri Aug 12 20:06:05 2022 -0600 (25 hours ago)

using the files:

load-env-and-cmake-frag-file.sh

export TRILINOS_DIR=/fgs/rabartl/Trilinos.base/Trilinos
FULL_BUILD_NAME=rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables

if [[ -e GenConfigSettings.cmake ]] ; then
  echo "Remvoing existing file GenConfigSettings.cmake ..."
  rm GenConfigSettings.cmake
fi

export WORKSPACE=$TRILINOS_DIR/..

source $TRILINOS_DIR/packages/framework/GenConfig/gen-config.sh \
--cmake-fragment GenConfigSettings.cmake \
$FULL_BUILD_NAME \
--force \
$TRILINOS_DIR

unset FULL_BUILD_NAME
unset WORKSPACE

# NOTE Above, it is an undocumented requirment that the env vars TRILINOS_DIR
# and WORKSPACE be set!

do-configure

rm -rf CMake*
cmake \
-G Ninja \
-C GenConfigSettings.cmake \
-D Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES=OFF \
-D Trilinos_ENABLE_TESTS=ON \
-D Trilinos_TRACE_ADD_TEST=ON \
"$@" \
$TRILINOS_DIR

I did the following configuration and build on 'ascicgp17' just now:

$ time ./do-configure -DTrilinos_ENABLE_Tpetra=ON  &> configure.out && time ninja -j32 &> make.out && time ctest -j32 &> ctest.out

real    1m1.765s
user    0m22.275s
sys     0m19.939s

real    9m5.558s
user    95m29.468s
sys     21m59.136s

real    2m39.617s
user    7m47.328s
sys     7m58.314s

$ grep -A 50 "failed out of" ctest.out 
100% tests passed, 0 tests failed out of 280

Subproject Time Summary:
Tpetra    = 4210.91 sec*proc (280 tests)

Total Test time (real) = 159.50 sec

In other words, there has to be something wrong with the PR tester to be posting test results including the test TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1 in the build PR-10808-test-rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables-407.

Looking at the Jenkins logs for this build at:

https://do.sandia.gov/trilinos-ci/job/Trilinos_PR_cuda-11.4.2-uvm-off/407/

the file:

https://do.sandia.gov/trilinos-ci/job/Trilinos_PR_cuda-11.4.2-uvm-off/407/artifact/pull_request_test/Testing/Temporary/LastTest_20220814-0113.log

when downloaded does not even show that test:

$ grep TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1 junk/LastTest_20220814-0113.log
[empty]

What in the check is going on here?

trilinos-autotester · 2022-08-14T06:53:10Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO REVIEWS HAVE BEEN PERFORMED ON THIS PULL REQUEST!

trilinos-autotester · 2022-08-14T06:53:16Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

bartlettroscoe · 2022-08-14T15:56:21Z

Wait, I missed that the failing test described above is TpetraTSQR_SequentialTsqr_contiguousCacheBlocks_MPI_1 which is NOT the same test as TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1 (contiguous vs noncontiguous).

I will update issue issue #10847 to see what other tests are showing this same error "an illegal instruction was encountered".

…ust on ascicgpu PR build (#10847)

bartlettroscoe · 2022-08-14T16:33:04Z

FYI: I put in the disable of the test TpetraTSQR_SequentialTsqr_contiguousCacheBlocks_MPI_1 as well in the commit 81a885d and I merged it to the topic branch for PR #10808 in merge commit 5c08a58.

Now we should see those errors clear up.

bartlettroscoe · 2022-08-14T16:47:45Z

Hello @jhux2, @csiefer2, @cgcgcg, @ccober6, can one of you please approve this PR so that it can merge? These tests are taking out about 30% of the PR iterations where Tpetra tests are getting run (see updated Description on #10847).

trilinos-autotester · 2022-08-15T05:33:05Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Build Num: 883
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Build Num: 884
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

Build Num: 882
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_intel-17.0.1

Build Num: 883
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_clang-10.0.0

Build Num: 887
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

Build Num: 899
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	vortex
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_python3

Build Num: 710
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	ascic
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

Build Num: 419
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	GPU
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

CDash Test Results for PR# 10871.

trilinos-autotester · 2022-08-15T05:33:23Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO REVIEWS HAVE BEEN PERFORMED ON THIS PULL REQUEST!

trilinos-autotester · 2022-08-15T05:33:29Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

trilinos-autotester · 2022-08-15T15:32:15Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

trilinos-autotester · 2022-08-15T15:34:12Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Build Num: 886
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Build Num: 887
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

Build Num: 885
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_intel-17.0.1

Build Num: 886
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_clang-10.0.0

Build Num: 890
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

Build Num: 902
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	vortex
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_python3

Build Num: 713
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	ascic
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

Build Num: 422
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	GPU
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: bartlettroscoe

trilinos-autotester · 2022-08-15T17:42:17Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Build Num: 886
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Build Num: 887
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

Build Num: 885
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_intel-17.0.1

Build Num: 886
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_clang-10.0.0

Build Num: 890
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	trilinos-any
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

Build Num: 902
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	vortex
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_python3

Build Num: 713
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	ascic
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

Build Num: 422
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Tpetra;Framework tasks;AT: RETEST;AT: AUTOMERGE;PA: Framework
PULLREQUESTNUM	10871
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	GPU
TRILINOS_SOURCE_BRANCH	10847-ascicgpu-disable-tpetra-test
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`81a885d`
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e061ffc`

CDash Test Results for PR# 10871.

trilinos-autotester · 2022-08-15T17:42:35Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

jhux2

lgtm

e10harvey

Please use the FORCE option since this will be required during incremental rebuilds and is what cmake -DVAR=VAL does.

bartlettroscoe · 2022-08-15T21:26:58Z

Please use the FORCE option since this will be required during incremental rebuilds and is what cmake -DVAR=VAL does.

Let's discuss this.

bartlettroscoe · 2022-08-15T22:05:32Z

Please use the FORCE option since this will be required during incremental rebuilds and is what cmake -DVAR=VAL does.

Having the FORCE options on these cache vars breaks the workflow documented in:

https://github.com/trilinos/Trilinos/wiki/Addressing-test-failures

in that if the developer passed in -D<FullTestName>_DISABLE=OFF, that will be ignored and the test will still be disabled.

@e10harvey, please document the use cases where you see a problem needing to be solved by using the FORCE option for these cache vars. I promise you, that if you blow away the cache before a reconfigure (in PR builds, in local builds, etc.), there is no issue here. (And you can never keep and existing CMakeCache.txt file in place and expect that rebuilds will be 100% correct.)

e10harvey

Since this just affects cuda UVM off builds, let's make a note of the FORCE option removal here and see if people start running into problems.

bartlettroscoe · 2022-08-15T23:00:35Z

Since this just affects cuda UVM off builds, let's make a note of the FORCE option removal here and see if people start running into problems.

Thanks @e10harvey! I promise that as long as the PR tester and users run rm -rf CMake* before they configure with cmake ... to (re)build, then there is no need to set FORCE on these vars.

trilinos-autotester · 2022-08-16T13:49:59Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ jhux2 e10harvey ]!

trilinos-autotester · 2022-08-16T13:50:06Z

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

trilinos-autotester · 2022-08-16T13:50:09Z

Merge on Pull Request# 10871: IS A SUCCESS - Pull Request successfully merged

bartlettroscoe added 2 commits August 12, 2022 19:34

Remove FORCE from all test disables (#10870)

35dd8bf

You *NEVER* want to force disable a test. If you force disable it, the developer has no way to manually enable that test in order to fix it.

Disable test TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1…

6e31532

… just on ascicgpu PR build (#10847)

bartlettroscoe added pkg: Tpetra Framework tasks Framework tasks (used internally by Framework team) PA: Framework Issues that fall under the Trilinos Framework Product Area labels Aug 13, 2022

bartlettroscoe self-assigned this Aug 13, 2022

bartlettroscoe mentioned this pull request Aug 13, 2022

TpetraTSQR_SequentialTsqr_[non]contiguousCacheBlocks_MPI_1 tests failing randomly with 'illegal instruction was encountered' in PR rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release builds starting 2022-07-20 #10847

Closed

bartlettroscoe mentioned this pull request Aug 13, 2022

KokkosKernels: Add missing dependency on CUBLAS (#10807) #10808

Merged

bartlettroscoe requested review from jhux2, csiefer2 and cgcgcg August 14, 2022 01:47

bartlettroscoe added the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Aug 14, 2022

Disable test TpetraTSQR_SequentialTsqr_contiguousCacheBlocks_MPI_1 j…

81a885d

…ust on ascicgpu PR build (#10847)

bartlettroscoe added the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Aug 15, 2022

trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Aug 15, 2022

jhux2 approved these changes Aug 15, 2022

View reviewed changes

This was referenced Aug 15, 2022

Document process for addressing failures in PR testing #10879

Closed

Make it so that changes under packages/framework/ triggers enable of all Trilinos packages #10881

Closed

e10harvey requested changes Aug 15, 2022

View reviewed changes

e10harvey approved these changes Aug 15, 2022

View reviewed changes

bartlettroscoe added AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed and removed AT: WIP Causes the PR autotester to not test the PR. (Remove to allow testing to occur.) labels Aug 15, 2022

trilinos-autotester merged commit 67db3e9 into develop Aug 16, 2022

trilinos-autotester removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Aug 16, 2022

jhux2 deleted the 10847-ascicgpu-disable-tpetra-test branch August 17, 2022 00:51

Disable randomly failing Tpetra test (#10847), remove FORCE from all test disables (#10870) #10871

Disable randomly failing Tpetra test (#10847), remove FORCE from all test disables (#10870) #10871

Conversation

bartlettroscoe commented Aug 13, 2022 • edited Loading

Internal Issues

Description

How was this tested?

bartlettroscoe commented Aug 13, 2022

trilinos-autotester commented Aug 13, 2022

trilinos-autotester commented Aug 13, 2022

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_intel-17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_clang-10.0.0

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_python3

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

Jenkins Parameters

Using Repos:

trilinos-autotester commented Aug 13, 2022

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_intel-17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_clang-10.0.0

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_python3

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

Jenkins Parameters

trilinos-autotester commented Aug 13, 2022

trilinos-autotester commented Aug 13, 2022

bartlettroscoe commented Aug 13, 2022

bartlettroscoe commented Aug 14, 2022

bartlettroscoe commented Aug 14, 2022

bartlettroscoe commented Aug 14, 2022

bartlettroscoe commented Aug 14, 2022

trilinos-autotester commented Aug 14, 2022

trilinos-autotester commented Aug 14, 2022

bartlettroscoe commented Aug 14, 2022

bartlettroscoe commented Aug 14, 2022

bartlettroscoe commented Aug 14, 2022

trilinos-autotester commented Aug 15, 2022

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

Jenkins Parameters

Build Information

bartlettroscoe commented Aug 13, 2022 •

edited

Loading