[Intrepid2 + Cuda9]

[kyungjoo-kim]

@bathmatt @ndellingwood @mperego

There is another CUDA 9 issue. After @ndellingwood and I confirmed that the previous issue mentioned in #1976 is resolved. This error comes up with FAD. Could we check what modification is made after the last time we confirmed that Trilinos was compiled ?

The following is the email conversation between me and @bathmatt

Looks like it goes further or less far, not sure. But now I see this

[ 82%] Building CXX object packages/panzer/disc-fe/src/CMakeFiles/panzer-disc-fe.dir/Panzer_AssemblyEngine.cpp.o
 
/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(51): error: DynRankDimTraits is not a template
 
/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(97): error: ViewCtorProp is not a template
 
/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(170): error: ViewCtorProp is not a template
 
/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(203): error: ViewCtorProp is not a template
 
/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(489): error: namespace "Kokkos::Experimental::Impl" has no member "DynRankSubviewTag"
lots and lots more
 
Probably kokkos developed moved on?
 
From: "Kim, Kyungjoo (-EXP)" <[email protected]>
Date: Monday, January 22, 2018 at 9:55 AM
To: "Bettencourt, Matthew" <[email protected]>, "Pawlowski, Roger P" <[email protected]>
Subject: Re: cuda9 and panzer/intrepid
 
Instead of using Trilinos Kokkos,  the "develop" branch from the Kokkos repo.
 
1. Put a symbolic link (name "kokkos") to kokkos repo (with develop branch checked out) in Trilinos base directory.
2. Add configure -D Kokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos 
 
Configure and build it. It will use kokkos from the base directory not the one in the packages directory.
 
Best
Kyungjoo
 
 
From: "Bettencourt, Matthew" <[email protected]>
Date: Monday, January 22, 2018 at 9:34 AM
To: Kyungjoo Kim <[email protected]>, "Pawlowski, Roger P" <[email protected]>
Subject: cuda9 and panzer/intrepid
 
I heard a rumor that this was working, My machine is getting RHEL7 on it, and I tried to compile using cuda9.1, is there any magic that works for this?  I’m seeing this, looking for ideas on this.
 
 
[ 72%] Building CXX object packages/panzer/dof-mgr/src/CMakeFiles/panzer-dof-mgr.dir/Panzer_IntrepidFieldPattern.cpp.o
….
/remote/mbetten/Trilinos/Trilinos/packages/intrepid2/src/Discretization/Basis/Intrepid2_Basis.hpp:325:130: error: expansion pattern ‘Kokkos::LayoutStride’ contains no argument packs
/remote/mbetten/Trilinos/Trilinos/packages/intrepid2/src/Discretization/Basis/Intrepid2_Basis.hpp:325:148: error: expansion pattern ‘ExecSpaceType’ contains no argument packs
/remote/mbetten/Trilinos/Trilinos/packages/intrepid2/src/Discretization/Basis/Intrepid2_Basis.hpp:325:151: error: template argument 2 is invalid
/remote/mbetten/Trilinos/Trilinos/packages/intrepid2/src/Discretization/Basis/Intrepid2_Basis.hpp:325:151: error: template argument 3 is invalid

@trilinos/

Expectations

Current Behavior

Motivation and Context

Definition of Done

Possible Solution

Steps to Reproduce

Your Environment

• Relevant repo SHA1s:
• Relevant configure flags or configure script:
• Operating system and version:
• Compiler and TPL versions:

Related Issues

• Blocks
• Is blocked by
• Follows
• Precedes
• Related to
• Part of
• Composed of

Additional Information

[kyungjoo-kim]

Also add @etphipp . Do we have any changes that do not allow to use kokkos latest develop branch ?

[bathmatt]

Here is the full output log.
log.txt.gz

Here is the first errors, it is sacado, @etphipp

/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(51): error: DynRankDimTraits is not a template

/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(97): error: ViewCtorProp is not a template

/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(170): error: ViewCtorProp is not a template

/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(203): error: ViewCtorProp is not a template

/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(489): error: namespace "Kokkos::Experimental::Impl" has no m
ember "DynRankSubviewTag"

/remote/mbetten/Trilinos/Trilinos/packages/sacado/src/Kokkos_DynRankView_Fad.hpp(563): error: namespace "Kokkos::Experimental" has no member 
"DynRankView"

[ndellingwood]

@bathmatt @kyungjoo-kim: In the current Kokkos develop branch there was some cleanup of the namespaces, relevant here was removing the DynRankView from the Experimental namespace. The corresponding changes needed to be made in Sacado and I created the attached Sacado patch when testing this for issue #1928 and #1976 (to verify Cuda 9 issues resolved); the patch matches the namespace cleanup.

@bathmatt if you are not using Stokhos, the attached patch should help, otherwise you may want to try the kokkos-develop branch of Trilinos which includes the changes from the Sacado patch and other changes (e.g. similar updates to Stokhos) that are needed for the next Kokkos promotion.

In either case, the current Kokkos develop branch should be used. This is how I do so:

• Create a symbolic link to my clone of Kokkos (with develop branch checked out) in Trilinos base directory.
• Add the following to my Trilinos configuration script:
  -D Kokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \

sacado_rm_exp_drv.patch.txt

[kyungjoo-kim]

@ndellingwood Sorry that I did not read carefully the instruction you already mentioned in the other githiub issue. Thanks.

[bathmatt]

@ndellingwood just to be clean, you are saying that the sacado patch provided will resolve this issue. AND, this patch won't make it into the main branch until kokkos is merged into develop?

[ndellingwood]

@bathmatt:

you are saying that the sacado patch provided will resolve this issue

Yes, unless you are also compiling with Stokhos. If you are also compiling Stokhos (it requires similar namespace updates as the Sacado patch but I did not catch them), instead you may need to checkout the kokkos-develop branch of Trilinos - it is a temporary branch accumulating changes needed in Trilinos for the next snapshot of Kokkos (which should happen around mid-Feb), and it includes the needed changes to Sacado, as well as Stokhos (and possibly other packages) by Dan.

this patch won't make it into the main branch until kokkos is merged into develop

That's right, next merge of Kokkos develop into Trilinos should be sometime mid-Feb.

[ndellingwood]

Closing, this was addressed back in March by #2351