Update testall

[ndellingwood]

No description provided.

[ndellingwood]

@lucbv @brian-kelley this adds caraway to the cm_test_all_sandia script in BUILD_ONLY mode (and also updates kokkos-dev* with sems-cuda/11.1 compilers). I wasn't able to fully test because the with-gpu queue was full, but hopefully this is sufficient to help with build testing

[lucbv]

OK, that looks fine to me, did you test it on caraway04?

[ndellingwood]

@lucbv no, I think the default compute node I was granted was caraway01 (which at least had vim); I was able to try the dry-run, but hipcc was unavailable to cross-compile so I'm not fully confident in the changes

[brian-kelley]

@ndellingwood Even though both nodes (caraway03/caraway04) in the with-gpu queue are "down", you can still ssh directly to them and run stuff

[lucbv]

I am on caraway04 right now so I can test this

[ndellingwood]

@lucbv I just launched a test on caraway04 (I didn't see anyone on when I ran top), let me know if I should kill the job

[brian-kelley]

They have 96 threads so I don't think there'll be any issues :)

[lucbv]

No you should be fine, building is not taking that long : )

[ndellingwood]

@lucbv @brian-kelley seeing some compile errors in perf_tests, I'm assuming this shouldn't be a blocker on this PR though?

In file included from /ascldap/users/ndellin/kokkos-kernels/perf_test/blas/blas/KokkosBlas_perf_test.cpp:45:
/ascldap/users/ndellin/kokkos-kernels/perf_test/blas/blas/KokkosBlas_trtri_perf_test.hpp:227:17: error: reference to __host__ function 'trtri<Kokkos::View<double **, Kokkos::LayoutStride, Kokkos::Device<Kokkos::Experimental::HIP, Kokkos::Experimental::HIPSpace>, Kokkos::MemoryTraits<0>>>' in __host__ __device__ function
    KokkosBlas::trtri(&trtri_args_.uplo, &trtri_args_.diag, svA);
                ^
/ascldap/users/ndellin/kokkos-kernels/testing/cm-Caraway/TestAll_2020-10-29_15.27.37/rocm/3.8.0/Hip_Serial-release/kokkos-install/include/HIP/Kokkos_HIP_Parallel_Range.hpp:82:5: note: called by 'exec_range<void>'
    m_functor(i);
    ^
/ascldap/users/ndellin/kokkos-kernels/testing/cm-Caraway/TestAll_2020-10-29_15.27.37/rocm/3.8.0/Hip_Serial-release/kokkos-install/include/HIP/Kokkos_HIP_Parallel_Range.hpp:104:22: note: called by 'operator()'
      this->template exec_range<WorkTag>(iwork);
                     ^
/ascldap/users/ndellin/kokkos-kernels/testing/cm-Caraway/TestAll_2020-10-29_15.27.37/rocm/3.8.0/Hip_Serial-release/kokkos-install/include/HIP/Kokkos_HIP_KernelLaunch.hpp:104:11: note: called by 'hip_parallel_launch_local_memory<Kokkos::Impl::ParallelFor<parallel_blas_trtri<Kokkos::Experimental::HIP>, Kokkos::RangePolicy<Kokkos::Experimental::HIP>, Kokkos::Experimental::HIP>, 1024, 1>'
  driver->operator()();
          ^
/ascldap/users/ndellin/kokkos-kernels/src/blas/KokkosBlas_trtri.hpp:75:1: note: 'trtri<Kokkos::View<double **, Kokkos::LayoutStride, Kokkos::Device<Kokkos::Experimental::HIP, Kokkos::Experimental::HIPSpace>, Kokkos::MemoryTraits<0>>>' declared here
trtri (const char uplo[],
^
1 error generated when compiling for gfx900.

In file included from /ascldap/users/ndellin/kokkos-kernels/perf_test/blas/blas3/KokkosBlas3_perf_test.cpp:45:
/ascldap/users/ndellin/kokkos-kernels/perf_test/blas/blas3/KokkosBlas3_trmm_perf_test.hpp:307:17: error: reference to __host__ function 'trmm<Kokkos::View<double **, Kokkos::LayoutStride, Kokkos::Device<Kokkos::Experimental::HIP, Kokkos::Experimental::HIPSpace>, Kokkos::MemoryTraits<0>>, Kokkos::View<double **, Kokkos::LayoutStride, Kokkos::Device<Kokkos::Experimental::HIP, Kokkos::Experimental::HIPSpace>, Kokkos::MemoryTraits<0>>>' in __host__ __device__ function
    KokkosBlas::trmm(&trmm_args_.side, &trmm_args_.uplo, &trmm_args_.trans,
                ^
/ascldap/users/ndellin/kokkos-kernels/testing/cm-Caraway/TestAll_2020-10-29_15.27.37/rocm/3.8.0/Hip_Serial-release/kokkos-install/include/HIP/Kokkos_HIP_Parallel_Range.hpp:82:5: note: called by 'exec_range<void>'
    m_functor(i);
    ^
/ascldap/users/ndellin/kokkos-kernels/testing/cm-Caraway/TestAll_2020-10-29_15.27.37/rocm/3.8.0/Hip_Serial-release/kokkos-install/include/HIP/Kokkos_HIP_Parallel_Range.hpp:104:22: note: called by 'operator()'
      this->template exec_range<WorkTag>(iwork);
                     ^
/ascldap/users/ndellin/kokkos-kernels/testing/cm-Caraway/TestAll_2020-10-29_15.27.37/rocm/3.8.0/Hip_Serial-release/kokkos-install/include/HIP/Kokkos_HIP_KernelLaunch.hpp:104:11: note: called by 'hip_parallel_launch_local_memory<Kokkos::Impl::ParallelFor<parallel_blas_trmm<Kokkos::Experimental::HIP>, Kokkos::RangePolicy<Kokkos::Experimental::HIP>, Kokkos::Experimental::HIP>, 1024, 1>'
  driver->operator()();
          ^
/ascldap/users/ndellin/kokkos-kernels/src/blas/KokkosBlas3_trmm.hpp:85:1: note: 'trmm<Kokkos::View<double **, Kokkos::LayoutStride, Kokkos::Device<Kokkos::Experimental::HIP, Kokkos::Experimental::HIPSpace>, Kokkos::MemoryTraits<0>>, Kokkos::View<double **, Kokkos::LayoutStride, Kokkos::Device<Kokkos::Experimental::HIP, Kokkos::Experimental::HIPSpace>, Kokkos::MemoryTraits<0>>>' declared here
trmm (const char side[],
^
1 error generated when compiling for gfx900.

[ndellingwood]

@lucbv @brian-kelley looks like there are a couple spots guarded with #if !defined(KOKKOS_ENABLE_CUDA), I can change to
#if !defined(KOKKOS_ENABLE_CUDA) && !defined(KOKKOS_ENABLE_HIP) and put in a PR if this isn't part of WIP already

[brian-kelley]

@ndellingwood Those changes are in Luc's pending PR #820. You might as well keep them to get the build working I guess.

[ndellingwood]

@brian-kelley @lucbv

Those changes are in Luc's pending PR #820.

Sounds good, I rebuilt with those changes locally and build completed successfully. Safe to merge these cm_test_all_sandia updates in?

[lucbv]

Yes I would merge this in.