MueLu: Add MagmaSparse support to matvec driver #4719

jjellio · 2019-03-25T23:31:11Z

@trilinos/muelu

Description

This PR adds a MAGMA Sparse TPL to Trilinos and optionally enables that functionality in MueLu.
The MatvevDriver provides testing for correctness, but this testing is not enabled by default, since this TPL's use depends on how it was built (e.g., Cuda vs MIC).

The test currently provides Cuda functionality + testing. TODO is to ensure it works seamlessly with many core.

Notes:
This moves the matvec driver out of the EXPERIMENTAL guard (for evaluations we really don't want to be enabling Experimental IMO).

Motivation and Context

Allow testing of SpMV using various underlying implementations. This driver intentionally works at the Xpetra/Tpetra level, so that matrices are constructed in a manner similar to how MueLu does.

How Has This Been Tested?

The code has correctness testing, which is defined by comparing against SpMVs performed by Tpetra.

Checklist

Closes #4718

My commit messages mention the appropriate GitHub issue numbers.
My code follows the code style of the affected package(s).
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the code contribution guidelines for this project.
I have added tests to cover my changes.
All new and existing tests passed.
No new compiler warnings were introduced.
These changes break backwards compatibility.

Fix C++ casting Fix missing reference in KK kernel Fix specialization to avoid Scalar test Add Magma Sparse as TPL in Trilinos Add Magma Sparse to Matvec driver Provide test for Magma Sparse cuda SpMV in driver

jhux2 · 2019-03-26T00:26:37Z

packages/muelu/cmake/Dependencies.cmake, line 9 at r1 (raw file):

SET(TEST_OPTIONAL_DEP_PACKAGES AztecOO Galeri Belos Pamgen)
SET(LIB_REQUIRED_DEP_TPLS BLAS LAPACK)
SET(LIB_OPTIONAL_DEP_TPLS Boost MATLAB AmgX CGAL ViennaCL MKL Avatar CUSPARSE MAGMASparse)

Shouldn't this be an optional test dependency?

jhux2 · 2019-03-26T00:28:58Z

packages/muelu/cmake/Dependencies.cmake, line 9 at r1 (raw file):

Previously, jhux2 (Jonathan Hu) wrote…

Shouldn't this be an optional test dependency?

Also, we've not done a good job of this, but could you also add this to the table in MueLu user's guide on page 19?

jhux2

Reviewed 6 of 6 files at r1.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @cgcgcg, @csiefer2, and @jjellio)

jjellio · 2019-03-26T02:08:44Z

@jhu2 I didn't specify a test, because it wasn't obvious how to filter out tests by enabled execution spaces. Currently, this is tested with Cuda. Without out Cuda, it should build the original KK v Tpetra code. I haven't done the stuff to figure out Magma (on non-cuda). The code could function as a test though... For Magma and CuSparse TPL usage.

As for documentation: What would I document? That you can access the TPL if you want? The same would be true for CuSPARSE. I haven't provided any TPL specific functionality in the library.

mhoemmen · 2019-03-26T02:58:32Z

cmake/TPLs/FindTPLMAGMASparse.cmake

+
+TRIBITS_TPL_FIND_INCLUDE_DIRS_AND_LIBRARIES( MAGMASparse
+  REQUIRED_HEADERS magmasparse.h
+  REQUIRED_LIBS_NAMES magma_sparse magma


Is this a TPL dependency? That is, does MAGMASparse depend on MAGMA, or is MAGMASparse stand-alone? Would it be possible to build with both MAGMASparse and MAGMA enabled, supposing that we had a MAGMA TPL?

Actually, I started with MagmaSparse only... but they do depend. MagmaSparse requires Magma.

This TPL is a WIP. We could make a MAGMA TPL, that provides an identifier that states whether it provides sparse. I would need to look at NetCDF or HDF5, since those packages provide various attributes along with libraries/headers..

We should talk about my scheme for supporting multiple linear algebra things.

This TPL is a WIP.

Shall we mark it as such then? or do you mean "WIP" in the sense that you plan future work?

We should talk about my scheme for supporting multiple linear algebra things.

Let's do that at some point.

WIP = work in progress = meaning the details of in CMake for the TPL need to be worked on - but that is in the direction of better multi-blas support.

packages/muelu/test/scaling/CMakeLists.txt

mhoemmen · 2019-03-26T03:01:03Z

packages/muelu/test/scaling/MatvecKernelDriver.cpp

+  MagmaSparse_SpmV_Pack (const crs_matrix_type& A,
+                      const vector_type& X,
+                            vector_type& Y)
+  {}


It would be nicer for this class not to compile at all if Scalar is not one of the supported types.

I'm not sure. This gets complicated, because we instantiate the driver itself for multiple ETI types. The general idea is that the code will compile, but that it simply won't provide the runtime option.

What we really need, is a better way to express the different instantiations (it would speed up build times).

Christian's way of doing it in kokkos-kernels makes sense. You have a struct with a static method that says "try using ${TPL} to solve the problem." If ${TPL} can't do it -- e.g., because it doesn't support the Scalar type -- the method returns false, and the caller has to try something else.

packages/muelu/test/scaling/MatvecKernelDriver.cpp

mhoemmen · 2019-03-26T03:01:44Z

packages/muelu/test/scaling/MatvecKernelDriver.cpp

+#include <magmasparse.h>
+template<typename Scalar, typename LocalOrdinal, typename GlobalOrdinal, typename Node>
+class MagmaSparse_SpmV_Pack {
+  typedef Tpetra::CrsMatrix<Scalar,LocalOrdinal,GlobalOrdinal,Node> crs_matrix_type;


Prefer C++11 using crs_matrix_type = Tpetra::CrsMatrix<...>;.

jhux2 · 2019-03-26T16:59:40Z

@jhu2 I didn't specify a test, because it wasn't obvious how to filter out tests by enabled execution spaces. Currently, this is tested with Cuda. Without out Cuda, it should build the original KK v Tpetra code. I haven't done the stuff to figure out Magma (on non-cuda). The code could function as a test though... For Magma and CuSparse TPL usage.

No, I mean something different. I believe Magma should be in TEST_OPTIONAL_DEP_PACKAGES. As it is now, doesn't cmake think this is an optional library dependency?

As for documentation: What would I document? That you can access the TPL if you want? The same would be true for CuSPARSE. I haven't provided any TPL specific functionality in the library.

Fair point. I guess the only thing one might document would be that one can do comparisons with Magma.

jjellio · 2019-03-26T20:07:09Z

No, I mean something different. I believe Magma should be in TEST_OPTIONAL_DEP_PACKAGES. As it is now, doesn't cmake think this is an optional library dependency?

I think, I maybe perhaps can possibly understand what you mean.

WithLIB_OPTIONAL_DEP_TPLS, this assumes the library depends on the TPL. With TEST_OPTIONAL_DEP_PACKAGES That would mean a 'test' depends on the TPL, but not the library itself? I can fiddle with it and see if it works. If that is the case, then CUSPARSE needs to get moved there as well.

jhux2 · 2019-03-26T22:51:30Z

No, I mean something different. I believe Magma should be in TEST_OPTIONAL_DEP_PACKAGES. As it is now, doesn't cmake think this is an optional library dependency?

I think, I maybe perhaps can possibly understand what you mean.

WithLIB_OPTIONAL_DEP_TPLS, this assumes the library depends on the TPL. With TEST_OPTIONAL_DEP_PACKAGES That would mean a 'test' depends on the TPL, but not the library itself? I can fiddle with it and see if it works. If that is the case, then CUSPARSE needs to get moved there as well.

Yes, this is what I mean.

jjellio · 2019-03-26T22:52:59Z

You have increased my TRIBITS knowledge by 1!

trilinos-autotester · 2019-03-27T08:40:19Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2019-03-27T08:40:52Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 3011
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 2840
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 1290
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 974
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_cuda_9.2

Build Num: 708
Status: STARTED

Jenkins Parameters

Parameter Name	Value
JENKINS_JOB_TYPE	Experimental
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Using Repos:

Repo: TRILINOS (jjellio/Trilinos)

Pull Request Author: jjellio

trilinos-autotester · 2019-03-27T12:17:34Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 3011
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 2840
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 1290
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 974
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

Build Information

Test Name: Trilinos_pullrequest_cuda_9.2

Build Num: 708
Status: PASSED

Jenkins Parameters

Parameter Name	Value
JENKINS_JOB_TYPE	Experimental
PULLREQUESTNUM	4719
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	jje/muelu-matvecdriver-magma
TRILINOS_SOURCE_REPO	https://github.com/jjellio/Trilinos
TRILINOS_SOURCE_SHA	`1b2ab3e`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`8c69ea0`

CDash Test Results for PR# 4719.

trilinos-autotester · 2019-03-27T12:17:52Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ csiefer2 ]!

trilinos-autotester · 2019-03-27T12:18:00Z

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

trilinos-autotester · 2019-03-27T12:18:03Z

Merge on Pull Request# 4719: IS A SUCCESS - Pull Request successfully merged

Cleanup code

b80ba13

Fix C++ casting Fix missing reference in KK kernel Fix specialization to avoid Scalar test Add Magma Sparse as TPL in Trilinos Add Magma Sparse to Matvec driver Provide test for Magma Sparse cuda SpMV in driver

jjellio requested review from cgcgcg, csiefer2 and jhux2 March 25, 2019 23:31

jjellio requested a review from a team as a code owner March 25, 2019 23:31

jjellio self-assigned this Mar 25, 2019

jjellio added pkg: MueLu AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed labels Mar 25, 2019

Removed compiler warning

1b2ab3e

jhux2 reviewed Mar 26, 2019

View reviewed changes

csiefer2 approved these changes Mar 26, 2019

View reviewed changes

mhoemmen reviewed Mar 26, 2019

View reviewed changes

trilinos-autotester merged commit c044193 into trilinos:develop Mar 27, 2019

trilinos-autotester removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Mar 27, 2019

jhux2 mentioned this pull request Aug 18, 2021

MueLu: Add MagmaSparse support to Matvec driver #4718

Closed

trilinos-autotester mentioned this pull request Feb 18, 2022

Implementing fix to resolve compilation error in Amesos2 with SuperLU 5 and clang 12 #10151

Merged

trilinos-autotester mentioned this pull request Apr 9, 2022

Intrepid2: Support Full Exact Sequence in Structured Integration #10419

Merged

MueLu: Add MagmaSparse support to matvec driver #4719

MueLu: Add MagmaSparse support to matvec driver #4719

Conversation

jjellio commented Mar 25, 2019 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Checklist

jhux2 commented Mar 26, 2019

jhux2 commented Mar 26, 2019

jhux2 left a comment

Choose a reason for hiding this comment

jjellio commented Mar 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhux2 commented Mar 26, 2019

jjellio commented Mar 26, 2019

jhux2 commented Mar 26, 2019

jjellio commented Mar 26, 2019

trilinos-autotester commented Mar 27, 2019

trilinos-autotester commented Mar 27, 2019

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_cuda_9.2

Jenkins Parameters

Using Repos:

trilinos-autotester commented Mar 27, 2019

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_cuda_9.2

Jenkins Parameters

trilinos-autotester commented Mar 27, 2019

trilinos-autotester commented Mar 27, 2019

trilinos-autotester commented Mar 27, 2019

jjellio commented Mar 25, 2019 •

edited

Loading