Ifpack2: Multiple test failures with intel/2021.4, intel/2023.2.0 (icpc) and Serial, OpenMP backends

[ndellingwood]

Bug Report

Testing builds on Blake (SKX arch) with intel/2021.4 (icpc, intel classic compiler) report multiple test failures
@trilinos/ifpack2

  303 - Ifpack2_Diagonal_tfqmr_belos_MPI_1 (Failed)
  304 - Ifpack2_Diagonal_bicgstab_belos_MPI_1 (Failed)
  305 - Ifpack2_Diagonal_gcrodr_belos_MPI_1 (Failed)
  307 - Ifpack2_Jacobi_belos_MPI_1 (Failed)
  308 - Ifpack2_DatabaseSchwarz_belos_MPI_1 (Failed)
  321 - Ifpack2_ILUT_5w_2_MPI_1 (Failed)
  322 - Ifpack2_ILUT_5w_no_diag_MPI_1 (Failed)
  323 - Ifpack2_Cheby_belos_MPI_1 (Failed)
  324 - Ifpack2_Cheby_belos_nospectralradius_MPI_1 (Failed)
  325 - Ifpack2_GS_belos_MPI_1 (Failed)
  326 - Ifpack2_SGS_belos_MPI_1 (Failed)
  327 - Ifpack2_MTGS_belos_MPI_1 (Failed)
  328 - Ifpack2_MTSGS_belos_MPI_1 (Failed)
  329 - Ifpack2_GS_belos_muelu_MPI_4 (Failed)
  330 - Ifpack2_MTGS_belos_muelu_MPI_4 (Failed)
  331 - Ifpack2_SGS_belos_muelu_MPI_4 (Failed)
  332 - Ifpack2_MTSGS_belos_muelu_MPI_4 (Failed)
  333 - Ifpack2_ILUT_belos_MPI_1 (Failed)
  334 - Ifpack2_Bordered_blockcg_belos_MPI_1 (Failed)
  335 - Ifpack2_Bordered_minres_belos_MPI_1 (Failed)
  341 - Ifpack2_AdditiveSchwarz_MPI_4 (Failed)
  347 - Ifpack2_RILUKSingleProcessUnitTests_MPI_1 (Failed)
  348 - Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 (Failed)

Steps to Reproduce

1. SHA1: 7e0c759
2. Reproducer (Blake testbed):

module load intel/oneAPI/hpc-toolkit/2021.4.0 intel/oneAPI/base-toolkit/2021.4.0 openmpi/4.0.5/intel-oneapi/2021.4.0 cmake/3.25.2 git
module swap gcc/7.2.0 gcc/10.2.0
module load openblas/0.3.21/gcc/10.2.0
module load boost/1.75.0/intel-oneapi/2021.2.0
module load hdf5/1.10.7/openmpi/4.0.5/intel-oneapi/2021.2.0 netcdf-c/4.7.4/openmpi/4.0.5/intel-oneapi/2021.2.0 zlib/1.2.11
export OMPI_CXX="icpc"
export OMPI_CC="icc"
export OMPI_FC="ifort"
export OMPI_F77="ifort"
export OMPI_F90="ifort"

cmake \
 -D CMAKE_CXX_COMPILER="`which mpicxx`" \
 -D CMAKE_C_COMPILER="`which mpicc`" \
 -D CMAKE_CXX_STANDARD="17" \
 -D CMAKE_CXX_FLAGS="-g -no-ip" \
 -D CMAKE_Fortran_COMPILER="mpif77" \
 -D CMAKE_INSTALL_PREFIX="${TRILINOS_INSTALL_DIR}" \
 -D CMAKE_BUILD_TYPE=RELEASE \
\
 -D TPL_ENABLE_MPI=ON \
  -D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="-bind-to;socket;-map-by;socket" \
\
 -D TPL_ENABLE_BLAS:STRING=ON \
  -D BLAS_LIBRARY_DIRS:FILEPATH=${BLAS_ROOT}/lib \
  -D BLAS_LIBRARY_NAMES:STRING="openblas" \
 -D TPL_ENABLE_LAPACK:STRING=ON \
  -D LAPACK_INCLUDE_DIRS:FILEPATH="${LAPACK_ROOT}/include" \
  -D LAPACK_LIBRARY_DIRS:FILEPATH=${LAPACK_ROOT}/lib \
  -D LAPACK_LIBRARY_NAMES:STRING="openblas" \
-D TPL_ENABLE_Boost=ON \
   -D Boost_INCLUDE_DIRS:PATH="${BOOST_ROOT}/include" \
   -D Boost_LIBRARY_DIRS:PATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_BoostLib=ON \
   -D BoostLib_INCLUDE_DIRS:PATH="${BOOST_ROOT}/include" \
   -D BoostLib_LIBRARY_DIRS:PATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_Netcdf=ON \
   -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF_ROOT}/include" \
   -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF_ROOT}/lib64" \
  -D TPL_Netcdf_LIBRARIES:PATH="${NETCDF_ROOT}/lib64/libnetcdf.a;${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a" \
  -D TPL_Netcdf_PARALLEL:BOOL=OFF \
-D TPL_ENABLE_HDF5=ON \
  -D HDF5_INCLUDE_DIRS:PATH="${HDF5_ROOT}/include" \
  -D TPL_HDF5_LIBRARIES:PATH="${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a" \
-D TPL_ENABLE_Zlib=ON \
  -D Zlib_INCLUDE_DIRS:PATH="${ZLIB_ROOT}/include" \
  -D TPL_Zlib_LIBRARIES:PATH="${ZLIB_ROOT}/lib/libz.a" \
-D TPL_ENABLE_DLlib=ON \
-D TPL_ENABLE_Matio=OFF \
-D TPL_ENABLE_X11=OFF \
\
 -D Trilinos_ENABLE_TESTS=OFF \
 -D Trilinos_ENABLE_EXAMPLES=OFF \
 -D Trilinos_ENABLE_COMPLEX=ON \
 -D Trilinos_ENABLE_OpenMP=ON \
\
  -D Trilinos_ENABLE_Amesos=ON \
   -D Amesos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_SERIAL=ON \
  -D Kokkos_ENABLE_OPENMP=ON \
  -D Kokkos_ARCH_SKX=ON \
  -D Trilinos_ENABLE_Intrepid=ON \
   -D Intrepid_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_ROL=ON \
   -D ROL_ENABLE_TESTS=OFF \
 \
  -D Trilinos_ENABLE_Ifpack2=ON \
   -D Ifpack2_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Amesos2=ON \
   -D Amesos2_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_SERIAL=ON \
  -D Kokkos_ARCH_SKX=ON \
   -D Kokkos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_KokkosKernels=ON \
   -D KokkosKernels_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Tpetra=ON \
   -D Tpetra_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Sacado=ON \
   -D Sacado_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Stokhos=ON \
   -D Stokhos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Zoltan2=ON \
   -D Zoltan2_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Intrepid2=OFF \
   -D Intrepid2_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Belos=ON \
   -D Belos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Anasazi=ON \
   -D Anasazi_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Teuchos=ON \
   -D Teuchos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_MueLu=ON \
   -D MueLu_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Panzer=ON \
   -D Panzer_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Phalanx=ON \
   -D Phalanx_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_STKMesh:BOOL=ON \
  -D Trilinos_ENABLE_STKSimd:BOOL=ON \
  -D Trilinos_ENABLE_STKTransfer:BOOL=ON \
  -D Trilinos_ENABLE_STKSearch:BOOL=ON \
  -D Trilinos_ENABLE_STKUtil:BOOL=ON \
  -D Trilinos_ENABLE_STKTopology:BOOL=ON \
  -D Trilinos_ENABLE_STKIO:BOOL=OFF \
\
  -D Trilinos_ENABLE_SEACAS=OFF \
$TRILINOS_DIR

[github-actions]

Automatic mention of the @trilinos/ifpack2 team

[github-actions]

Automatic mention of the @trilinos/ifpack2 team

[ndellingwood]

Just updating the issue, I'm seeing similar failures for Serial and OpenMP builds with the intel-oneapi-compilers/2023.2.0 and mkl modules on Blake when using icpc (intel classic compiler). I'm not sure why the file read/write is causing issues when using this compiler, but just noting here for reference

Common failure output is something like:

4: Test timeout computed to be: 1500
4: Every proc reading parameters from xml_file: test_Diagonal_tfqmr_calore1_mm.xml
4: Matrix Market file for sparse matrix A: calore1.mtx
4: Matrix Market file for right-hand-side(s) B: calore1_rhs.mtx
4:
4: p=0: *** Caught standard std::exception of type 'std::runtime_error' :
4:
4:  /home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/inout/MatrixMarket_Tpetra.hpp:4708:
4:
4:  Throw number = 2
4:
4:  Throw test that evaluated to true: globalReadDataSuccess == 0
4:
4:  Failed to read the multivector's data: /home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/inout/MatrixMarket_Tpetra.hpp:4526:

Potentially intertwined with #11968

# Blake all queue - non-mpi build

# Environment
module load cmake intel-oneapi-compilers/2023.2.0 intel-oneapi-mkl/2023.2.0
module list

export TRILINOS_DIR=<path-to-source>

export BLAS_LIBRARIES="-mkl;${MKLROOT}/lib/intel64/libmkl_intel_lp64.a;${MKLROOT}/lib/intel64/libmkl_intel_thread.a;${MKLROOT}/lib/intel64/libmkl_core.a"
export LAPACK_LIBRARIES=${BLAS_LIBRARIES}

# Configure Trilinos
cmake \
  -D CMAKE_INSTALL_PREFIX="${PWD}/install" \
  -D CMAKE_CXX_COMPILER="`which icpc`" \
  -D CMAKE_C_COMPILER="`which icc`" \
  -D CMAKE_Fortran_COMPILER="`which ifort`" \
  -D CMAKE_CXX_FLAGS="-g -no-ip" \
  -D CMAKE_C_FLAGS="-g -no-ip" \
  -DTPL_ENABLE_MPI=OFF \
  -DTPL_ENABLE_BLAS:BOOL=ON \
  -DTPL_BLAS_LIBRARIES:PATH="${BLAS_LIBRARIES}" \
  -DTPL_LAPACK_LIBRARIES:PATH="${LAPACK_LIBRARIES}" \
  -DTPL_ENABLE_LAPACK:BOOL=ON \
  -DTrilinos_ENABLE_ALL_PACKAGES=OFF \
  -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
  -DTrilinos_ENABLE_TESTS=OFF \
  -DTrilinos_MUST_FIND_ALL_TPL_LIBS=TRUE \
  -DTrilinos_ENABLE_COMPLEX=ON \
  -DTrilinos_ENABLE_OpenMP=OFF \
  -DTrilinos_ENABLE_Kokkos=ON \
   -D Kokkos_ENABLE_SERIAL=ON \
   -D Kokkos_ARCH_SKX=ON \
  -DTrilinos_ENABLE_KokkosKernels=ON \
  -DTrilinos_ENABLE_Tpetra=ON \
   -D Tpetra_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Ifpack2=ON \
   -D Ifpack2_ENABLE_TESTS=ON \
\
  -DTPL_ENABLE_Matio=OFF \
\
  -DTrilinos_ENABLE_INSTALLATION_TESTING=OFF \
$TRILINOS_DIR

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.