MPI.has_cuda() fails on Spectrum MPI

[vchuravy]

using the test script

using MPI
MPI.Init()

@show MPI.has_cuda()

fails with

ERROR: ERROR: ERROR: ERROR: LoadError: LoadError: LoadError: LoadError: ccall: could not find function MPIX_Query_cuda_support in library /opt/ibm/spectrum_mpi/lib/libmpi_ibm.soccall: could not find function MPIX_Query_cuda_support in library
/opt/ibm/spectrum_mpi/lib/libmpi_ibm.soccall: could not find function MPIX_Query_cuda_support in library /opt/ibm/spectrum_mpi/lib/libmpi_ibm.so
Stacktrace:
Stacktrace:
Stacktrace:ccall: could not find function MPIX_Query_cuda_support in library /opt/ibm/spectrum_mpi/lib/libmpi_ibm.so
Stacktrace:
 [1]

[simonbyrne]

What does MPI.MPI_LIBRARY_VERSION give?

[simonbyrne]

Do you need to rebuild MPI.jl?

[vchuravy]

MPI.MPI_LIBRARY_VERSION = "Open MPI v10.3.0.00rtm0, package: IBM Spectrum MPI, ident: IBM Spectrum MPI, repo rev: IBM_SPECTRUM_MPI_10.03.00.00_2019.04.04_RTM0, Unreleased developer copy\0"

[vchuravy]

I just found this entertaining tibit (from https://hpc.llnl.gov/training/tutorials/using-lcs-sierra-system):

Caveat: Do NOT use the MPIX_Query_cuda_support() routine or the preprocessor constant MPIX_CUDA_AWARE_SUPPORT to determine if Spectrum MPI is CUDA-aware. This routine has either been removed from the IBM implementation, or will always return false (older versions).

and this: https://github.com/lammps/lammps/pull/1582/files#diff-3335adca21d9025c15f1c0ba57812a56R43

[simonbyrne]

Also, what on earth is Open MPI v10.3.0.00rtm0? Their most recent version is 4.0.2