Best Practices for Building/Running MPI Under Flux

[SteVwonder]

With most vendor-specific MPIs being hard to support and experiencing frequent breaks under Flux, we should document the best practices for both building MPIs that are maximally compatible with Flux and for running different MPI variants under Flux.

[SteVwonder]

The details for running Spectrum MPI on the CORAL systems are already documented in our RTD: https://flux-framework.readthedocs.io/en/latest/coral.html#launching-spectrum-mpi-within-flux

The documentation does not include any details on alternate MPIs that could potentially run on the CORAL systems, and how to build/run them.

[SteVwonder]

My experiences with running various MPIs on NERSC's Cori:

Intel MPI

Lots of errors unless you use the shared memory "fabric", but the multi-node does work:

> module load impi/2020
> srun --pty -N 2  /global/common/software/flux/sles15_x86_64/master-2020-07-23/bin/boot-flux.sh flux mini run -N 2 -n 4 ./mpi-hello-intelmpi-2019.6.166.exe
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Hello World from rank 1
Hello World from rank 3
Hello World from rank 2
Hello World from rank 0

> I_MPI_FABRICS=shm srun --pty -N 1  /global/common/software/flux/sles15_x86_64/master-2020-07-23/bin/boot-flux.sh flux mini run -N 1 -n 4 ./mpi-hello-intelmpi-2019.6.166.exe
Hello World from rank 3
Hello World from rank 0
Hello World from rank 2
Hello World from rank 1

MPICH

Works without warnings or errors, but the only system-provided MPICH is a debug build:

> module load mpich/3.3.1-debug
> srun -N 2 /global/common/software/flux/sles15_x86_64/master-2020-07-23/bin/boot-flux.sh flux mini run -N 2 -n 4 ./mpi-hello-mpich-3.3.1-debug.exe
Hello World from Node 3
Hello World from Node 2
Hello World from Node 1
Hello World from Node 0

OpenMPI

Works without warnings or errors, but requires environment variables to be set:

> OMPI_MCA_schizo=flux OMPI_MCA_pmix=flux srun -N 2 /global/common/software/flux/sles15_x86_64/master-2020-07-23/bin/boot-flux.sh flux mini run -N 2 -n 4 ./mpi-hello-openmpi-4.0.2.exe
Hello World from rank 1
Hello World from rank 0
Hello World from rank 2
Hello World from rank 3

Applying the following patch to the master branch allows openmpi to boostrap under Flux without manually setting the environment variables.

diff --git a/src/shell/lua.d/openmpi.lua b/src/shell/lua.d/openmpi.lua
index ff833e411..38a1d4385 100644
--- a/src/shell/lua.d/openmpi.lua
+++ b/src/shell/lua.d/openmpi.lua
@@ -11,3 +11,5 @@
 local f = require 'flux'.new ()
 local rundir  = f:getattr ('broker.rundir')
 shell.setenv ("OMPI_MCA_orte_tmpdir_base", rundir)
+shell.setenv ("OMPI_MCA_pmix", "flux")
+shell.setenv ("OMPI_MCA_schizo", "flux")