OpenFOAM MPI issues

[bedroge]

Saw this in the output of the OpenFOAM test:

mpirun: Error: unknown option "-ppn"
Type 'mpirun --help' for usage.

Triggered by:

mpirun -np $NP -ppn $PPN -hostfile hostlist snappyHexMesh -parallel -overwrite 2>&1 | tee log.snappyHexMesh

The script does just continue.

[bedroge]

Something is wrong with that command anyway, just using mpirun -np 2 snappyHexMesh -overwrite -parallel does not work either because of this -parallel flag:

--> FOAM FATAL ERROR: 
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

[bedroge]

Okay, this is weird. Without the -parallel flag, using mpirun doesn't make sense (you just get duplicate output). But looking at the file that runs the original motorBike tutorial (/cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/OpenFOAM-8/tutorials/incompressible/simpleFoam/motorBike/Allrun), they do use runParallel for snappyHexMesh:

#!/bin/sh
cd ${0%/*} || exit 1    # Run from this directory

# Source tutorial run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions

# Copy motorbike surface from resources directory
cp $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/
runApplication surfaceFeatures

runApplication blockMesh

runApplication decomposePar -copyZero
runParallel snappyHexMesh -overwrite

runParallel patchSummary
runParallel potentialFoam
runParallel $(getApplication)

runApplication reconstructParMesh -constant
runApplication reconstructPar -latestTime

[bedroge]

@boegel
I looked into this a bit more. The Pstream library can be found in both the mpi and dummy folders:

$ ls /cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/OpenFOAM-8/platforms/linux64GccDPInt32Opt/lib/mpi/
libPstream.so  libptscotchDecomp.so

$ ls /cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/OpenFOAM-8/platforms/linux64GccDPInt32Opt/lib/dummy/

According to some issues I've found while searching for the error, this could be related to the order of the directories in the $LD_LIBRARY_PATH.

$ echo $LD_LIBRARY_PATH
/cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/OpenFOAM-8/platforms/linux64GccDPInt32Opt/lib/mpi:/cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/ThirdParty-8/platforms/linux64GccDPInt32/lib/mpi:/home/bob/OpenFOAM/bob-8/platforms/linux64GccDPInt32Opt/lib:/cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/site/8/platforms/linux64GccDPInt32Opt/lib:/cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/OpenFOAM-8/platforms/linux64GccDPInt32Opt/lib:/cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/ThirdParty-8/platforms/linux64GccDPInt32/lib:/cvmfs/pilot.eessi-hpc.org/2020.09/software/x86_64/intel/broadwell/software/OpenFOAM/8-foss-2020a/OpenFOAM-8/platforms/linux64GccDPInt32Opt/lib/dummy

The first item is the mpi folder... I don't get it...

[bedroge]

Ah, maybe it's the RPATH in the executable which is messing things up? I do see the dummy folder in there, but not the mpi folder.

[bedroge]

Yep, that's the issue: I've used patchelf to remove this dummy folder from the executable's RPATH, and now it works fine... So now the question is: how does this directory end up in the RPATH / why doesn't the mpi folder get added to the RPATH?

[bedroge]

Some other installation framework makes symlinks in the lib dir to all files in the lib/mpi dir, see:
https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openfoam/common/spack-Allwmake#L33

Since lib appears before lib/dummy in the RPATH, this should fix the issue. I'm now testing this with a modified easyblock...

[bedroge]

This has been fixed in the easyblock (see easybuilders/easybuild-easyblocks#2196). For the current installation, I've created these symlinks manually.