Running GRMHD module of Athena++ in Pleiades after upgradation to TOSS4

[prasundhang]

Hi, I am running a GRMHD simulation of an accretion disk using Athena++ in Pleiades Hasswll nodes in the NASA HECC facility. After the upgrade to TOSS4, I found weird artefacts in my GRMHD simulations along the mesh blocks. The same set-up is running fine in other clusters. Has anyone also encountered this kind of issue in Pleiades? Any suggestions will be really helpful.

[c-white]

Can you say (1) what exact environment you are loading, (2) what configure commands you are using, and (3) how you are calling the executable in the job script?

(I've never actually used Pleiades, so I can only be of indirect assistance here.)

[c-white]

Also, this sounds like an MPI issue. I assume you are following the recommendation here.

[prasundhang]

Hi Chris,

Thanks! Please note that the same set up was giving 'good' results before the upgrade.

Here are the details:

Version used: Athena++ downloaded in 2019
compilation command: python3 configure.py -gb --prob gr_thin_disk_j_isco --coord kerr-schild --nghost 4 --flux hlle --cxx icc -mpi -hdf5

Here I modified configure.py to use avx accelerator and mpi-hpe.
in line no. 482:
if args['cxx'] == 'icpc':
# ICC is C++11 feature-complete since v15.0 (2014-08-26)
definitions['COMPILER_CHOICE'] = 'icpc'
definitions['COMPILER_COMMAND'] = makefile_options['COMPILER_COMMAND'] = 'icpc'
makefile_options['PREPROCESSOR_FLAGS'] = ''
makefile_options['COMPILER_FLAGS'] = (
'-O3 -std=c++11 -lmpi++ -lmpi -ipo -xCORE-AVX2 -inline-forceinline -qopenmp-simd -qopt-prefetch=4 '
'-qoverride-limits'

in line 637:
if (args['cxx'] == 'icpc' or args['cxx'] == 'icpc-debug' or args['cxx'] == 'icpc-phi'):
definitions['COMPILER_COMMAND'] = makefile_options['COMPILER_COMMAND'] = 'icpc'

Modules used (recommended by the NAS support in private communication):

1. comp-intel/2022.1.0 3) mpi-hpe/mpt 5) hdf5/1.12.0_mpt
2. mpi-hpe/mpt.2.28_25Apr23_rhel87 4) szip/2.1.1