Tacho - cuSolver interface #6695
Conversation
Tacho is templated with a scheduler type which uses exec space as its template parameter. As it requires UVM for tasking, default exec and memory space ie Cuda and CudaUVM. However, I do not want to use UVM for data buffer. So, Tacho uses internally Kokkos::Device<Cuda,CudaSpace>. This is templated in Tacho_Util.hpp (UseThisDevice struct).
cuSolver uses int for rowptr while trilinos uses unsigned long (at least tacho uses it) So, it needs to be converted when it is interfaced to cuSolver
kyungjoo-kim
added
the
AT: WIP
kyungjoo-kim
added
AT: RETEST
dual view shows a weird alloc error.
kyungjoo-kim
added
AT: AUTOMERGE
|
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_3
Jenkins Parameters
Using Repos:
Pull Request Author: kyungjoo-kim |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_3
Jenkins Parameters
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.8.4 # 5562 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_intel_17.0.1 # 5387 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.9.3_SERIAL # 3816 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_7.2.0 # 3664 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_cuda_9.2 # 3212 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_python_2 # 1491 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_python_3 # 1490 (click to expand) |
trilinos-autotester
removed
the
AT: RETEST
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_3
Jenkins Parameters
Using Repos:
Pull Request Author: kyungjoo-kim |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_3
Jenkins Parameters
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.8.4 # 5567 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_intel_17.0.1 # 5392 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.9.3_SERIAL # 3821 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_gcc_7.2.0 # 3669 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_cuda_9.2 # 3217 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_python_2 # 1496 (click to expand)Console Output (last 100 lines) : Trilinos_pullrequest_python_3 # 1495 (click to expand) |
|
Fixed error by warning. |
kyungjoo-kim
added
the
AT: RETEST
|
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_3
Jenkins Parameters
Using Repos:
Pull Request Author: kyungjoo-kim |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: Trilinos_pullrequest_gcc_4.8.4
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_intel_17.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_gcc_7.2.0
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_cuda_9.2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_2
Jenkins Parameters
Build InformationTest Name: Trilinos_pullrequest_python_3
Jenkins Parameters
|
trilinos-autotester
removed
the
AT: RETEST
|
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
@ndellingwood would you re-approve this PR ? |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
1 similar comment
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
| host_space::print_configuration(std::cout, false); | ||
|
|
||
| typedef UseThisDevice<Kokkos::DefaultHostExecutionSpace> host_device_type; | ||
| printExecSpaceConfiguration<typename host_device_type::execution_space>("HostDevice", false); | ||
|
|
||
| int r_val = 0; | ||
| #if defined (__INTEL_MKL__) |
There was a problem hiding this comment.
Should ShyLU instead use a TPL macro? Trilinos already has an MKL TPL.
There was a problem hiding this comment.
I don't think that makes a difference. INTEL_MKL is defined in mkl.h and mkl.h is guarded by "TACHO_HAVE_MKL" which is determined from TPL_ENABLE_MKL.
| #if defined (KOKKOS_ENABLE_CUDA) | ||
| #include "Tacho_Util.hpp" | ||
| //#include "cusparse.h" | ||
| #include "cusparse_v2.h" |
There was a problem hiding this comment.
I really really wish we wouldn't expose TPL header includes in Trilinos header files. This v1 vs. v2 thing reminds me unpleasantly of a Sierra issue from a while back, where Trilinos was exposing the v1 cuBLAS header and Sierra was using a v2 cuBLAS header.
There was a problem hiding this comment.
CuSolver and PARDISO are for performance benchmark. These files are not included in the main header (Tacho.h) unless users intend to use it.
There was a problem hiding this comment.
@kyungjoo-kim : Should we move this to test instead of src ?
There was a problem hiding this comment.
No, not this moment. I am thinking some overhaul later and I will clean up that time.
| //#include "cusparse.h" | ||
| #include "cusparse_v2.h" | ||
| #include "cusolverSp.h" | ||
| #include "cusolverSp_LOWLEVEL_PREVIEW.h" |
There was a problem hiding this comment.
This looks like it will only work with a particular version of cuSOLVER. If so, then we should make it an optional feature and protect it with a macro.
|
Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ ndellingwood ]! |
|
Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged |
|
Merge on Pull Request# 6695: IS A SUCCESS - Pull Request successfully merged |
trilinos-autotester
removed
the
AT: AUTOMERGE
Motivation
This PR include cuSolver interface to evaluate the solver. Additionally, it includes changes from using exec space to using device type. In particular, when cuda is enabled, the default memory space is uvm space. Tacho uses device<Cuda,CudaSpace> as its default device when cuda is enabled.