Tpetra,Stokhos: Fix #4870 (prefer running MV norms on device) #4899
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@trilinos/tpetra @trilinos/stokhos
1. Make Tpetra::MultiVector prefer computing the local (to each MPI
process) parts of norms (norm1, norm2, normInf) on device.
2. Factor out the implementation of Tpetra::MultiVector::norm* into
a separate function, Tpetra::Details::normImpl. Do ETI for that
function, when ETI is enabled.
3. Make Stokhos do ETI for Tpetra::Details::normImpl, for its
UQ::PCE and MP::Vector Scalar types.
mhoemmen
requested review from
william76,
tjfulle,
cgcgcg,
kyungjoo-kim,
csiefer2,
etphipp,
kddevin,
GeoffDanielson,
jhux2 and
lucbv
mhoemmen
added
the
AT: AUTOMERGE
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Using Repos:
Pull Request Author: mhoemmen |
etphipp
reviewed
Apr 15, 2019
packages/stokhos/src/sacado/kokkos/vector/tpetra/Tpetra_Details_normImpl_MP_Vector_Cuda.cpp
Outdated
Show resolved
Hide resolved
mhoemmen
removed
the
AT: AUTOMERGE
Also removes the device-specific tmpl files in favor of the ones generated by CMake.
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.8.4 # 3252 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_intel_17.0.1 # 3081 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.9.3_SERIAL # 1528 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_7.2.0 # 1266 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_cuda_9.2 # 943 (click to expand) |
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Using Repos:
Pull Request Author: mhoemmen |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.8.4 # 3255 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_intel_17.0.1 # 3084 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.9.3_SERIAL # 1531 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_7.2.0 # 1269 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_cuda_9.2 # 946 (click to expand) |
mhoemmen
commented
Apr 16, 2019
There was a problem hiding this comment.
@etphipp The thing I suspected would happen actually did happen:
packages/stokhos/src/libstokhos_tpetra_mp_16_cuda.a(Tpetra_MultiVector_MP_Vector_16_Cuda.cpp.o): In function `void (anonymous namespace)::multiVectorNormImpl<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Cuda> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >(Tpetra::MultiVector<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Cuda> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::mag_type*, Tpetra::MultiVector<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Cuda> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >&, Tpetra::Details::EWhichNorm)':
/home/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_cuda_9.2@3/Trilinos/packages/tpetra/core/src/Tpetra_MultiVector_def.hpp:374: undefined reference to `void Tpetra::Details::normImpl<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Cuda> >, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Cuda> > >(Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Cuda> >*, Kokkos::View<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Cuda> > const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > const&, Tpetra::Details::EWhichNorm, Teuchos::ArrayView<unsigned long const> const&, bool, bool, Teuchos::Comm<int> const*)'
The issue is that Tpetra::Details::normImpl needs instantiation for the host mirror device space too. How should I handle this? Do I need to change something in Tpetra or can you handle this in Stokhos?
packages/stokhos/src/sacado/kokkos/pce/tpetra/Stokhos_Tpetra_ETI_Helpers_UQ_PCE_DEVICE.hpp
Show resolved
Hide resolved
packages/stokhos/src/sacado/kokkos/pce/tpetra/Stokhos_Tpetra_ETI_Helpers_UQ_PCE_DEVICE.hpp
Show resolved
Hide resolved
...rc/sacado/kokkos/vector/tpetra/Stokhos_Tpetra_ETI_Helpers_MP_Vector_ENSEMBLE_SIZE_DEVICE.hpp
Show resolved
Hide resolved
packages/stokhos/src/sacado/kokkos/pce/tpetra/Tpetra_Details_normImpl_UQ_PCE_Cuda.cpp
Outdated
Show resolved
Hide resolved
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Using Repos:
Pull Request Author: mhoemmen |
|
NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester... |
|
Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job Trilinos_pullrequest_intel_17.0.1 to start: Total Wait = 603
|
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Using Repos:
Pull Request Author: mhoemmen |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.8.4 # 3267 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_intel_17.0.1 # 3095 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.9.3_SERIAL # 1540 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_7.2.0 # 1278 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_cuda_9.2 # 955 (click to expand) |
mhoemmen
added
the
AT: WIP
|
@etphipp I'm marking this as WIP to avoid clogging the PR testing pipeline. Please feel free to unmark once you've pushed a commit. Thanks! |
etphipp
added
AT: AUTOMERGE
|
I believe I fixed the HostMirror instantiation issues, so I removed the WIP label. |
mhoemmen
removed
the
AT: AUTOMERGE
mhoemmen
added
the
AT: AUTOMERGE
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Using Repos:
Pull Request Author: mhoemmen |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
|
|
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
You’re right, I can’t review it. Can someone else? |
|
Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ lucbv ]! |
|
Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged |
|
Merge on Pull Request# 4899: IS A SUCCESS - Pull Request successfully merged |
trilinos-autotester
removed
the
AT: AUTOMERGE
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@trilinos/tpetra @trilinos/stokhos @vbrunini @etphipp
Supersedes PR #4881, PR #4871, and PR #4885.
Description
Tpetra::MultiVectorprefer computing the local (to each MPI process) parts of norms (norm1,norm2,normInf) on device. Do so even if the MultiVector is currently sync'd to host, unless the MultiVector has too few local rows.Tpetra::MultiVector::norm*into a separate function,Tpetra::Details::normImpl. Do ETI for that function, when ETI is enabled.Tpetra::Details::normImpl, for itsUQ::PCEandMP::VectorScalar types.To elaborate the last point: Tpetra's ETI macro for
Tpetra::Details::normImplonly takes 2 template parameters:ScalarandNode. This meant that I couldn't use Stokhos' ETI system directly (seeTPETRA_ETI_CLASSESinstokhos/src/CMakeLists.txt). I had to take different approaches for PCE types vs.MP::Vectortypes. My solution for PCE types looks more permanent (read "is less of a hack") than my solution forMP::Vectortypes; it would be great if @etphipp could comment.Motivation and Context
Related Issues