Batched Gemm Failure

[mndevec]

@kyungjoo-kim , I enabled Ifpack2 RBILUK code that uses batched GEMM. Some configureations are failing
as in https://testing.sandia.gov/cdash/viewBuildError.php?buildid=3088682
Could you have a look at it?

In file included from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Impl.hpp:7:0,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/ifpack2/src/Ifpack2_Experimental_RBILUK_def.hpp:57,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/ifpack2/src/Ifpack2_Experimental_RBILUK_Serial.cpp:50:
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Util.hpp:31:17: note: #pragma message: KOKKOS_DISABLE_PRAGMA_UNROLL
 #pragma message "KOKKOS_DISABLE_PRAGMA_UNROLL"
                 ^
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Util.hpp:37:17: note: #pragma message: KOKKOS_BATCHED_DISABLE_AVX256
 #pragma message "KOKKOS_BATCHED_DISABLE_AVX256"
                 ^
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Util.hpp:43:17: note: #pragma message: KOKKOS_BATCHED_DISABLE_AVX512
 #pragma message "KOKKOS_BATCHED_DISABLE_AVX512"
                 ^
In file included from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Impl.hpp:8:0,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/ifpack2/src/Ifpack2_Experimental_RBILUK_def.hpp:57,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/ifpack2/src/Ifpack2_Experimental_RBILUK_Serial.cpp:50:
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Internal.hpp: In instantiation of ‘static int KokkosBatched::Experimental::SerialGemmInternal<ArgAlgo>::invoke(int, int, int, ScalarType, const ValueType*, int, int, const ValueType*, int, int, ScalarType, ValueType*, int, int) [with ScalarType = std::complex<double>; ValueType = Kokkos::complex<double>; ArgAlgo = KokkosBatched::Experimental::Algo::Level3::Blocked]’:
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Impl.hpp:128:52:   required from ‘static int KokkosBatched::Experimental::SerialGemm<ArgTransA, ArgTransB, ArgAlgo>::invoke(ScalarType, const AViewType&, const BViewType&, ScalarType, const CViewType&) [with ScalarType = std::complex<double>; AViewType = Kokkos::View<Kokkos::complex<double>**, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >; BViewType = Kokkos::View<Kokkos::complex<double>**, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >; CViewType = Kokkos::View<Kokkos::complex<double>**, Kokkos::LayoutRight, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >; ArgTransA = KokkosBatched::Experimental::Trans::NoTranspose; ArgTransB = KokkosBatched::Experimental::Trans::NoTranspose; ArgAlgo = KokkosBatched::Experimental::Algo::Level3::Blocked]’
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/ifpack2/src/Ifpack2_Experimental_RBILUK_def.hpp:610:70:   required from ‘void Ifpack2::Experimental::RBILUK<MatrixType>::compute() [with MatrixType = Tpetra::Experimental::BlockCrsMatrix<std::complex<double>, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]’
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/ifpack2/src/Ifpack2_Experimental_RBILUK_Serial.cpp:61:1:   required from here
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Internal.hpp:100:21: error: no match for ‘operator==’ (operand types are ‘const std::complex<double>’ and ‘int’)
       if      (beta == 0) SerialSetInternal  ::invoke(m, n, 0,    C, cs0, cs1);
                     ^
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Internal.hpp:100:21: note: candidates are:
In file included from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/tpetra/core/src/Tpetra_Map.hpp:1:0,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_decl.hpp:53,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/tpetra/core/src/Tpetra_DistObject.hpp:1,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/tpetra/core/src/Tpetra_MultiVector_decl.hpp:54,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/tpetra/core/src/Tpetra_Vector_decl.hpp:54,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/tpetra/core/src/Tpetra_RowMatrix_decl.hpp:49,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp:54,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/tpetra/core/src/Tpetra_CrsMatrix.hpp:1,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/ifpack2/src/Ifpack2_ETIHelperMacros.h:6,
                 from /home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/BUILD/packages/ifpack2/src/Ifpack2_Experimental_RBILUK_Serial.cpp:48:
/home/tawiesn/nightlyTesting/SERIAL_DEBUG_DEV_MueLu_ExtraTypes/Trilinos/packages/tpetra/core/src/Tpetra_Map_decl.hpp:1607:6: note: template<class LocalOrdinal, class GlobalOrdinal, class Node> bool operator==(const Tpetra::Map<LocalOrdinal, GlobalOrdinal, Node>&, const Tpetra::Map<LocalOrdinal, GlobalOrdinal, Node>&)
 bool operator== (const Tpetra::Map<LocalOrdinal,GlobalOrdinal,Node> &map1,

[kyungjoo-kim]

The current code in trilinos seems to be outdated. Replace the directory with kokkoskernels develop or you can wait until a new snap shot happens.

[mndevec]

Ok thanks, I was not sure if this is the same one that you addressed already.

[aprokop]

Can we increase the priority on this, or revert the patch? Currently this breaks Ifpack2 and thus most of MueLu builds.

[mndevec]

I will push a commit to use Tpetra GEMM instead of KokkosKernels GEMM until kokkoskernels is snapshotted into Trilinos.

[kyungjoo-kim]

I think that this issue is done after we snapshot to trilinos lately. Do we still have this problem ?

[mndevec]

Yes, I think it should be done. I will test it in trilinos and close the issue