MueLu: Multiple test failures with intel/2021.4 (icpc) and OpenMP backend

[ndellingwood]

Bug Report

Testing builds on Blake (SKX arch) with intel/2021.4 (icpc, intel classic compiler) report multiple test failures
@trilinos/muelu

  370 - MueLu_UnitTestsTpetra_MPI_1 (Failed)
  371 - MueLu_UnitTestsTpetra_MPI_4 (Timeout)
  383 - MueLu_GeneralBlockSmoothing_MPI_4 (Failed)
  384 - MueLu_SerialGeneralBlockSmoothing_MPI_1 (Failed)
  385 - MueLu_ZeroAggTest_MPI_2 (Failed)
  386 - MueLu_SerialZeroAggTest_MPI_1 (Failed)
  387 - MueLu_BlockSmoothingWithAverages_MPI_4 (Failed)
  388 - MueLu_SchwarzSmoothingWithAverages_MPI_4 (Failed)
  394 - MueLu_DriverDiagonalModifications_MPI_1 (Failed)
  417 - MueLu_ReitzingerPFactory_MPI_4 (Failed)
  418 - MueLu_Maxwell3D-Tpetra_0_MPI_4 (Failed)
  419 - MueLu_Maxwell3D-Tpetra_1_MPI_4 (Failed)
  420 - MueLu_Maxwell3D-Tpetra_2_MPI_4 (Failed)
  421 - MueLu_Maxwell3D-Tpetra_3_MPI_4 (Failed)
  422 - MueLu_Maxwell3D-GMHD_MPI_1 (Failed)
  423 - MueLu_Maxwell3D-GMHD_MPI_2 (Failed)
  424 - MueLu_MeshTyingBlocked_SimpleSmoother_MPI_4 (Failed)
  425 - MueLu_MeshTyingBlocked_SimpleSmoother_2dof_small_MPI_1 (Failed)
  426 - MueLu_MeshTyingBlocked_SimpleSmoother_2dof_medium_MPI_1 (Failed)
  427 - MueLu_MeshTyingBlocked_SimpleSmoother_2dof_medium_MPI_4 (Failed)
  428 - MueLu_MeshTyingBlocked_SimpleSmoother_2dof_medium_spai_MPI_1 (Failed)
  462 - MueLu_VarDofDriver_MPI_1 (Failed)
  463 - MueLu_VarDofDriverEmin_MPI_1 (Failed)
  464 - MueLu_VarDofDriver_MPI_2 (Failed)

Steps to Reproduce

1. SHA1: 7e0c759
2. Reproducer (Blake testbed):

module load intel/oneAPI/hpc-toolkit/2021.4.0 intel/oneAPI/base-toolkit/2021.4.0 openmpi/4.0.5/intel-oneapi/2021.4.0 cmake/3.25.2 git
module swap gcc/7.2.0 gcc/10.2.0
module load openblas/0.3.21/gcc/10.2.0
module load boost/1.75.0/intel-oneapi/2021.2.0
module load hdf5/1.10.7/openmpi/4.0.5/intel-oneapi/2021.2.0 netcdf-c/4.7.4/openmpi/4.0.5/intel-oneapi/2021.2.0 zlib/1.2.11
export OMPI_CXX="icpc"
export OMPI_CC="icc"
export OMPI_FC="ifort"
export OMPI_F77="ifort"
export OMPI_F90="ifort"

cmake \
 -D CMAKE_CXX_COMPILER="`which mpicxx`" \
 -D CMAKE_C_COMPILER="`which mpicc`" \
 -D CMAKE_CXX_STANDARD="17" \
 -D CMAKE_CXX_FLAGS="-g -no-ip" \
 -D CMAKE_Fortran_COMPILER="mpif77" \
 -D CMAKE_INSTALL_PREFIX="${TRILINOS_INSTALL_DIR}" \
 -D CMAKE_BUILD_TYPE=RELEASE \
\
 -D TPL_ENABLE_MPI=ON \
  -D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="-bind-to;socket;-map-by;socket" \
\
 -D TPL_ENABLE_BLAS:STRING=ON \
  -D BLAS_LIBRARY_DIRS:FILEPATH=${BLAS_ROOT}/lib \
  -D BLAS_LIBRARY_NAMES:STRING="openblas" \
 -D TPL_ENABLE_LAPACK:STRING=ON \
  -D LAPACK_INCLUDE_DIRS:FILEPATH="${LAPACK_ROOT}/include" \
  -D LAPACK_LIBRARY_DIRS:FILEPATH=${LAPACK_ROOT}/lib \
  -D LAPACK_LIBRARY_NAMES:STRING="openblas" \
-D TPL_ENABLE_Boost=ON \
   -D Boost_INCLUDE_DIRS:PATH="${BOOST_ROOT}/include" \
   -D Boost_LIBRARY_DIRS:PATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_BoostLib=ON \
   -D BoostLib_INCLUDE_DIRS:PATH="${BOOST_ROOT}/include" \
   -D BoostLib_LIBRARY_DIRS:PATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_Netcdf=ON \
   -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF_ROOT}/include" \
   -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF_ROOT}/lib64" \
  -D TPL_Netcdf_LIBRARIES:PATH="${NETCDF_ROOT}/lib64/libnetcdf.a;${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a" \
  -D TPL_Netcdf_PARALLEL:BOOL=OFF \
-D TPL_ENABLE_HDF5=ON \
  -D HDF5_INCLUDE_DIRS:PATH="${HDF5_ROOT}/include" \
  -D TPL_HDF5_LIBRARIES:PATH="${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a" \
-D TPL_ENABLE_Zlib=ON \
  -D Zlib_INCLUDE_DIRS:PATH="${ZLIB_ROOT}/include" \
  -D TPL_Zlib_LIBRARIES:PATH="${ZLIB_ROOT}/lib/libz.a" \
-D TPL_ENABLE_DLlib=ON \
-D TPL_ENABLE_Matio=OFF \
-D TPL_ENABLE_X11=OFF \
\
 -D Trilinos_ENABLE_TESTS=OFF \
 -D Trilinos_ENABLE_EXAMPLES=OFF \
 -D Trilinos_ENABLE_COMPLEX=ON \
 -D Trilinos_ENABLE_OpenMP=ON \
\
  -D Trilinos_ENABLE_Amesos=ON \
   -D Amesos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_SERIAL=ON \
  -D Kokkos_ENABLE_OPENMP=ON \
  -D Kokkos_ARCH_SKX=ON \
  -D Trilinos_ENABLE_Intrepid=ON \
   -D Intrepid_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_ROL=ON \
   -D ROL_ENABLE_TESTS=OFF \
 \
  -D Trilinos_ENABLE_Ifpack2=ON \
   -D Ifpack2_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Amesos2=ON \
   -D Amesos2_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_SERIAL=ON \
  -D Kokkos_ARCH_SKX=ON \
   -D Kokkos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_KokkosKernels=ON \
   -D KokkosKernels_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Tpetra=ON \
   -D Tpetra_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Sacado=ON \
   -D Sacado_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Stokhos=ON \
   -D Stokhos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Zoltan2=ON \
   -D Zoltan2_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Intrepid2=OFF \
   -D Intrepid2_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Belos=ON \
   -D Belos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Anasazi=ON \
   -D Anasazi_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_Teuchos=ON \
   -D Teuchos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_MueLu=ON \
   -D MueLu_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Panzer=ON \
   -D Panzer_ENABLE_TESTS=ON \
  -D Trilinos_ENABLE_Phalanx=ON \
   -D Phalanx_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_STKMesh:BOOL=ON \
  -D Trilinos_ENABLE_STKSimd:BOOL=ON \
  -D Trilinos_ENABLE_STKTransfer:BOOL=ON \
  -D Trilinos_ENABLE_STKSearch:BOOL=ON \
  -D Trilinos_ENABLE_STKUtil:BOOL=ON \
  -D Trilinos_ENABLE_STKTopology:BOOL=ON \
  -D Trilinos_ENABLE_STKIO:BOOL=OFF \
\
  -D Trilinos_ENABLE_SEACAS=OFF \
$TRILINOS_DIR

[GrahamBenHarper]

@ndellingwood since the Tpetra issue you posted in #11968 reports errors with

  189 - TpetraCore_MatrixMarket_Tpetra_Map_InOutTest_MPI_4 (Failed)
  190 - TpetraCore_Bug5800_MPI_1 (Failed)
  191 - TpetraCore_Bug6288_MPI_4 (Failed)
  218 - TpetraCore_MatrixMatrix_UnitTests_MPI_4 (Failed)

and we use MatrixMarket maps to input the data for most of the tests you highlighted, I wouldn't be surprised if this is related to whatever Tpetra functionality was broken; however, I have no idea why this happens.

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[ndellingwood]

This was still problematic when I tested with intel/2021.4 awhile back, though it seems related to #11968 like mentioned above