KokkosKernels,Tpetra: Plug in TPLs (MKL, cuSPARSE, ...) for sparse matrix-vector multiply

[mhoemmen]

Please refer to the three attached patches. Their file names should end in .patch, but I had to rename them to .txt so Github would take them.

NOTE: If you want to use MKL for sparse matrix-vector multiply in Tpetra, you must do the following. First, apply the attached patches in order. Second, enable the MKL TPL in Trilinos. This is not the same thing as building against MKL for BLAS and LAPACK. Here is an example of the required CMake options, assuming that ${MKLROOT} is the path of the MKL installation:

-D TPL_ENABLE_MKL:BOOL=ON
-D MKL_LIBRARY_DIRS:FILEPATH="${MKLROOT}/lib/intel64"
-D MKL_LIBRARY_NAMES:STRING="mkl_rt"
-D MKL_INCLUDE_DIRS:FILEPATH="${MKLROOT}/include"

Third, set the TPETRA_USE_MKL_SPMV environment variable to 1. If you set it to 0, Tpetra will /not/ use MKL for sparse matrix-vector multiply. The environment variable lets you compare performance of Trilinos' implementation and MKL without needing to recompile.

0001-TpetraKernels-Add-comments-static_assert-to-single-v.txt
0002-TpetraKernels-Add-MKL-TPL.txt
0003-Tpetra-Call-MKL-for-single-vec-double-int-SpMV.txt

[mhoemmen]

@aprokop @crtrott @mndevec

[mhoemmen]

These patches are sufficiently general that they could be integrated, except for three issues:

1. One would need to fix a build warning (signed/unsigned comparison).
2. We might want to change the default setting so that Tpetra uses MKL by default. Right now, Tpetra uses its native implementation by default; you have to do something to turn on MKL for sparse mat-vec, even if you built with the MKL TPL enabled.
3. Consider changing from an environment variable to a Tpetra::CrsMatrix::fillComplete option.

[mhoemmen]

@aprokop pointed out an issue with the above patches: Since they don't change the default offset type in KokkosSparse::StaticCrsGraph, Xpetra and therefore MueLu do not get the correct offset type. The patch attached to this comment, which should be applied after the above patches, fixes that.

0001-Kokkos-Tpetra-Change-default-graph-matrix-offset-typ.txt

[mhoemmen]

This depends on #151 for a clean implementation.

[nmhamster]

Can we get a fix for #101 so we can a decent MKL enable that works for your issue, BLAS and LAPACK. I guess what I'm asking is that we enable MKL and we use it everywhere that's possible to bump performance? @bartlettroscoe @bmpersc

[bartlettroscoe]

You can use the MKL for BLAS and LAPACK if you want to. You just need to list out:

  -D BLAS_LIBRARY_DIR=<libdir> \
  -D BLAS_LIBRARY_NAMES="n1;n2;..." \

We have used the MLK on many machines like this. Unfortunately, we had to remove the default FIND_PACKAGE(BLAS) (see TriBITSPub/TriBITS/issues#89) so you will be stuck with manually listing out the libraries in the right order. Past experience is that it is not always trivial to figure out the right set of libs or the right order but with a little trial and error you can get it.

[mhoemmen]

Past experience is that it is not always trivial to figure out the right set of libs or the right order but with a little trial and error you can get it.

I think Si's point is that this is a usability bug.

[mhoemmen]

btw the patches above compile, but apparently don't get called. I'm revising them.

[bartlettroscoe]

I think Si's point is that this is a usability bug.

Then someone who cares enough about this needs to take on TriBITSPub/TriBITS#89. The challenge is "fixing this" without breaking something else.

[mhoemmen]

#153 is closed. Here is my plan: If the row offsets type we use does not match the row offsets type the TPL (e.g., MKL) uses, then we make a deep copy of the row offsets and stash them in the TPL handle (that Tpetra will keep).

We only need to copy the row offsets at first fillComplete. The graph isn't allowed to change on subsequent fillComplete calls. Total extra storage is half that of a double-precision vector that stores the result of a sparse mat-vec (assuming MKL_INT == int). Also, at copy time, we can test whether the matrix is too big, and call our sparse mat-vec implementation instead of MKL if it is.

[mhoemmen]

This is a duplicate of #430.