Install MPI.jl on HPC

[denisbertini]

Hi
I tried to install MPI.jl on our cluster following the documenation.
I export all necessary Environment variables to let the MPI.jl know it should use the pre-installed system
library.
then i used julia to install MPI.jl doingulia --project -e 'ENV["JULIA_MPI_BINARY"]="system"; using Pkg; Pkg.build("MPI"; verbose=true)'
Building MPI → /lustre/rz/dbertini/julia/1.6.2_packages/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/e4549a5ced642a73bd8ba2dd1d3b1d30b3530d94/build.log
[ Info: using system MPI ] 0/1
ERROR: LoadError: libmpi could not be found
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] top-level scope
@ /lustre/rz/dbertini/julia/1.6.2_packages/packages/MPI/2iRWa/deps/build.jl:76
[3] include(fname::String)
@ Base.MainInclude ./client.jl:444
[4] top-level scope
@ none:5
in expression starting at /lustre/rz/dbertini/julia/1.6.2_packages/packages/MPI/2iRWa/deps/build.jl:64
ERROR: Error building MPI:

Do you have any idea what went wrong?

Using `add MPI` from package Manager works but running the `helloworld.jl` example gives wrong results

mpiexec -n 2 julia test_mpi.jl
┌ Warning: You appear to have run julia under a different mpiexec than the one used by MPI.jl.
│ See the documentation for details.
└ @ MPI /lustre/rz/dbertini/julia/1.6.2_packages/packages/MPI/2iRWa/src/environment.jl:38
┌ Warning: You appear to have run julia under a different mpiexec than the one used by MPI.jl.
│ See the documentation for details.
└ @ MPI /lustre/rz/dbertini/julia/1.6.2_packages/packages/MPI/2iRWa/src/environment.jl:38
Hello world, I am rank 0 of 1
Hello world, I am rank 0 of 1

Thanks for your help

[denisbertini]

But using mpiexecjl works out of the box !

./mpiexecjl -n 10 julia ../../test_mpi.jl
Hello world, I am rank 0 of 10
Hello world, I am rank 9 of 10
Hello world, I am rank 1 of 10
Hello world, I am rank 3 of 10
Hello world, I am rank 5 of 10
Hello world, I am rank 4 of 10
Hello world, I am rank 7 of 10
Hello world, I am rank 2 of 10
Hello world, I am rank 8 of 10
Hello world, I am rank 6 of 10

what is then the difference between mpiexejl and mpirun ?
I really need to use mpirun or better srun on our cluster ...

[simonbyrne]

Is libmpi in your library path? Can you do:

using Libdl
dlopen("libmpi")

?

[denisbertini]

I can load the library via libmpi and all the needed PATHs are set properly.
I am using openMPI3.16.

julia> using Libdl

julia> dlopen("libmpi")
Ptr{Nothing} @0x0000000001332860

[simonbyrne]

Hmm. Something isn't set up correctly. What's in the file at joinpath(first(DEPOT_PATH), "prefs", "MPI.toml")?

[denisbertini]

i got:

library = "/cvmfs/vae.gsi.de/centos7/opt/linux-centos7-zen/gcc-8.1.0/openmpi-3.1.6-ooeaiywmb66kd7flr6p6ug5erexhgqmg/lib"
binary = "system"
path = "/cvmfs/vae.gsi.de/centos7/opt/linux-centos7-zen/gcc-8.1.0/openmpi-3.1.6-ooeaiywmb66kd7flr6p6ug5erexhgqmg"

this lengthly paths come from the installation via spack on our cluster.
Is this correct?

[simonbyrne]

library isn't set correctly (it should be either the library name, or the full path + name). Try doing:

ENV["JULIA_MPI_LIBRARY"]=""
Pkg.build("MPI"; verbose=true)

[denisbertini]

Seems to work:

julia> ENV["JULIA_MPI_LIBRARY"]=""
""

julia> using Pkg

julia> Pkg.build("MPI"; verbose=true)
    Building MPI → `/lustre/rz/dbertini/julia/1.6.2_packages/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/e4549a5ced642a73bd8ba2dd1d3b1d30b3530d94/build.log`
┌ Error: curl_easy_setopt: 48                        ]  0/1
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
[ Info: using system MPI
┌ Info: Using implementation
│   libmpi = "/cvmfs/vae.gsi.de/centos7/opt/linux-centos7-zen/gcc-8.1.0/openmpi-3.1.6-ooeaiywmb66kd7flr6p6ug5erexhgqmg/lib/libmpi"
│   mpiexec_cmd = `/cvmfs/vae.gsi.de/centos7/opt/linux-centos7-zen/gcc-8.1.0/openmpi-3.1.6-ooeaiywmb66kd7flr6p6ug5erexhgqmg/bin/mpiexec`
└   MPI_LIBRARY_VERSION_STRING = "Open MPI v3.1.6, package: Open MPI kresan@lxbuild07 Distribution, ident: 3.1.6, repo rev: v3.1.6, Mar 18, 2020\0"
┌ Info: MPI implementation detected
│   impl = OpenMPI::MPIImpl = 2
│   version = v"3.1.6"
└   abi = "OpenMPI"
Precompiling project...
  1 dependency successfully precompiled in 87 seconds (330 already precompiled, 6 skipped during auto due to previous errors)

[denisbertini]

Hey that works now !

bertini@lxbk0596:/lustre/rz/dbertini/julia > mpirun -n 10 julia test_mpi.jl 
Hello world, I am rank 7 of 10
Hello world, I am rank 0 of 10
Hello world, I am rank 4 of 10
Hello world, I am rank 5 of 10
Hello world, I am rank 6 of 10
Hello world, I am rank 3 of 10
Hello world, I am rank 8 of 10
Hello world, I am rank 2 of 10
Hello world, I am rank 9 of 10
Hello world, I am rank 1 of 10

thanks you for the help !

[simonbyrne]

no problem!