Silent crashes with system OpenMPI

[vchuravy]

@luraess, @christophernhill and I have been noticing silent crashes with system OpenMPI on Satori, and locally.

during build.jl Julia would simply exit. We have to do something like:

julia --project -e 'using Libdl; p=dlopen("libmpi", RTLD_LAZY; throw_error=false); p=dlopen("libmpi", RTLD_LAZY; throw_error=false); using Pkg; Pkg.add("MPI"); Pkg.build("MPI"; verbose=true)'

@christophernhill found https://www.open-mpi.org/faq/?category=troubleshooting#missing-symbols and IIRC we never got past

MPI.jl/deps/build.jl

Line 83 in d15a515

include(joinpath("..","src","implementations.jl"))

and

MPI.jl/src/implementations.jl

Line 50 in d15a515

const MPI_LIBRARY_VERSION_STRING = Get_library_version()

Haven't had a chance to dig into this more.

[simonbyrne]

Ah: we do this at __init__()

MPI.jl/src/MPI.jl

Lines 62 to 67 in 9b69d79

	@static if Sys.isunix()
	# need to open libmpi with RTLD_GLOBAL flag for Linux, before
	# any ccall cannot use RTLD_DEEPBIND; this leads to segfaults
	# at least on Ubuntu 15.10
	Libdl.dlopen(libmpi, Libdl.RTLD_LAZY | Libdl.RTLD_GLOBAL)
	end

but not by deps/build.jl.

Previous issues/changes:

• Don't use RTLD_DEEPBIND #109
• 8c06c8c
  • which links to this https://www.open-mpi.org/community/lists/users/2010/04/12803.php

[simonbyrne]

Actually we now do this:
74f31f2

Are you using the latest version of MPI.jl?

[luraess]

The issue occurred with v0.18.2. However, with fresh install today it did no longer occur with v0.18.2, nor using v0.19.1. Don't know what changed so far to prevent the bug.

[simonbyrne]

Okay, will close for now, but please reopen if you see it again.