Intrepid2 CUDA 9 build issue

[ibaned]

Moving issue kokkos/kokkos#1183 here.

@bathmatt wrote:

I'm getting kokkos errors I've never seen compile intrepid2 code with cuda 9. I'm not sure if this is a known issue and not sure where to document this..

It is dying on a panzer file compile. Has anyone seen this? I can't get cuda-gdb to work with panzer so I updated to 9 and now I can't compile trilinos

/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:130: error: expansion pattern ‘Kokkos::LayoutStride’ contains no argument packs/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:148: error: expansion pattern ‘ExecSpaceType’ contains no argument packs
/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:151: error: template argument 2 is invalid
/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:151: error: template argument 3 is invalid

is part of the actual error stream

@mperego I'm not sure if this is still happening though...

[mperego]

@ibaned I still have to looked into this. I held off until now because they were sorting out issues in Kokkos related to Cuda9. I'll try to look into this today or tomorrow.

[ndellingwood]

Reproduced similar errors with Cuda 9 as compiler and Kokkos develop branch in Trilinos.
Some changes in Kokkos needed for Cuda 9 are only on the Kokkos develop branch until the next integration testing, so I used a suggestion from @ibaned to use the develop Kokkos in my Trilinos configuration:

Create a symbolic link to my clone of Kokkos (with develop branch checked out) in Trilinos base directory.
Add the following to my Trilinos configuration script:
-D Kokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \

Not sure yet if the issue is in Intrepid2 or another package (with errors exposed during Intrepid2 compilation) which may require similar/further updates like those made for Kokkos.

[ndellingwood]

Here is the Kokkos issue regarding setting up support and nightly testing of Cuda 9:
kokkos/#1174

[mhoemmen]

POKE

[ibaned]

I think it will help to do reinvestigate this after the imminent Kokkos promotion, because prior to that Kokkos emits tons of warnings under CUDA 9.

[mhoemmen]

@ibaned Lots of warnings, but hopefully we can at least get through the build and make sure the tests pass :-D

[ndellingwood]

For update see #1976 and kokkos/kokkos#1183

[ndellingwood]

@kyungjoo-kim created a reproducer outside of Trilinos that requires only Kokkos. That code fails to compile with the same error as Intrepid2 when using Kokkos' master branch, but passes with Kokkos' develop branch.

[ndellingwood]

Intrepid2 will compile with Cuda 9 and gcc/4.9.3 when using the current Kokkos develop branch (sha f27d189a4ecc28b81d461b05629ef22d2a4eba87) along with some modifications to Sacado to remove DynRankView-related items from the Experimental namespace, reflecting similar updates to Kokkos tracked by kokkos/kokkos#1293. Quite a few warning outputs.

A patch of the Sacado changes is attached (created using git diff from the Trilinos base directory, using git apply command with the patch should apply them).

The Kokkos develop branch was tested by doing the following:

1. Add a symbolic link to the kokkos repo (with develop branch checked out) in Trilinos' base directory
2. Add the following line to configure file to override the kokkos package in Trilinos:
   -D Kokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
3. Apply the attached patch to remove the Experimental namespace from Sacado's DynRankView codes.
   sacado_rm_exp_drv.patch.txt

[ndellingwood]

All Intrepid2 unit tests pass with Cuda 9.0 and gcc/4.9.3

100% tests passed, 0 tests failed out of 247

Label Time Summary:
Intrepid2    = 2204.17 sec (247 tests)

Total Test time (real) = 2205.71 sec

[ndellingwood]

Cross-reference issue #2179

[ndellingwood]

Closing, this was addressed back in March by #2351