Fix MPI synchronization in real.exe #1268
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@honnorat It looks like every test that uses the ARW real program with serial or OpenMP is failing. Perhaps an ifdef around an MPI call is missing? |
|
@honnorat
Nope, I looked at your mods - that is not the trouble. |
|
Maybe I have no time today to fix the fix. Maybe tomorrow 🤞 |
|
@honnorat |
|
@honnorat |
|
Yes, off course |
davegill
reviewed
Aug 27, 2020
|
@honnorat
When I include the whitespace modifications, there are about 150 locations of changes, in two files, one with whitespace changes only. Given the actual small change that you implemented, would you describe what that single MPI command does and why that single command is needed. |
|
As I explained in #1267, I had never seen this bug #1267 before, it has only happend on one machine (an Intel-based cluster using Intel-MPI and ifort). The current fix makes sure that all processes are well synced before proceeding with As you mentioned, the first version of the fix was badly designed and we finally got a one-liner. My code editor removed automatically all trailing spaces in the modified files. |
|
@honnorat
Perhaps the easiest thing to do is to close this PR and re-open a new one. The only mod to the base branch release-v4.2.2 would be the new MPI barrier. You can grab the commit message from this PR for the new PR. |
honnorat
pushed a commit
to EXWEXs/WRF
that referenced
this pull request
Oct 12, 2020
Simplification of PR wrf-model#1268 Fixes wrf-model#1267
davegill
pushed a commit
that referenced
this pull request
Oct 23, 2020
TYPE: bug fix KEYWORDS: mpi, real, bug SOURCE: Marc Honnorat (EXWEXs) DESCRIPTION OF CHANGES: Problem: When running real.exe on multiple processes with MPI, one or more process occasionally crashes in setup_physics_suite (in share/module_check_a_mundo.F). This has been linked to wrf_dm_initialize non-blocking MPI. The real.exe occasionally crashes in setup_physics_suite (in share/module_check_a_mundo.F#L2640) because the latter uses model_config_rec % physics_suite, which on some machines is not initialized. The behavior is as if the broadcast of model_config_rec performed just before in main/real_em.F#L124 had not been received by all processes. I had never seen this bug before, it has only happened on one machine (an Intel-based cluster using Intel-MPI and ifort). The current fix makes sure that all processes are well synced before proceeding with setup_physics_suite. It solves the issue on my machine. Since this is immediately after reading in the namelist, no performance issues are expected as this read and broadcast of the namelist occurs only once per real / WRF / ndown run. Solution: An MPI barrier is added at the end of wrf_dm_initialize to force all of the processes to be synchronized before checking the namelist consistency. This is a simplification of PR #1268, which had extra white space. ISSUE: Fixes #1267 LIST OF MODIFIED FILES: M external/RSL_LITE/module_dm.F TESTS CONDUCTED: 1. On the only machine were I have seen the bug occur, this change fixes the problem. No other test was conducted since I couldn't reproduce the bug on another setup. 2. Jenkins testing is all PASS. RELEASE NOTES: When running real.exe on multiple processes with MPI, one or more processes occasionally crash in setup_physics_suite (in share/module_check_a_mundo.F). This has been traced to the fact that wrf_dm_initialize is non-blocking from an MPI point of view. The problem is intermittent and has only happened on one machine (an Intel-based cluster using Intel-MPI and ifort). An MPI barrier has been added at the end of wrf_dm_initialize to force all processes to be synchronized before checking namelist consistency.
honnorat
pushed a commit
to EXWEXs/WRF
that referenced
this pull request
Jun 17, 2021
honnorat
pushed a commit
to EXWEXs/WRF
that referenced
this pull request
Dec 15, 2021
(cherry picked from commit 183600b)
davegill
pushed a commit
that referenced
this pull request
Dec 16, 2021
TYPE: [bug fix] KEYWORDS: real.exe, MPI, bug fix SOURCE: Marc Honnorat (EXWEXs) DESCRIPTION OF CHANGES: Problem: The communicator `mpi_comm_allcompute`, created by subroutine `split_communicator` called by `init_modules(1)`, is not explicitly activated for the call to `wrf_dm_bcast_bytes( configbuf, nbytes )` in real.exe. On some platforms, this may prevent broadcast of namelist configuration (put in `configbuf` after the call to `get_config_as_buffer()`) across the MPI processes _before_ the call to `setup_physics_suite()`. An example of a problematic platform: a cluster of Intel Xeon E5-2650 v4 running on CentOS Linux release 7.6.1810, with Intel Parallel Studio XE (various versions, including 2018u3 and 2020u4) and Intel MPI Library (same version). Solution: The initialization step used in the WRF executable never triggers a failure as described in issue #1267. This PR reuses the temporary MPI context switch from WRF code. ISSUE: Fixes #1267 LIST OF MODIFIED FILES: M main/real_em.F TESTS CONDUCTED: 1. The modification systematically solves the problem on the noted cluster. 2. Jenkins tests are all passing. RELEASE NOTE: A fix for an MPI synchronization bug related to (not used) split communicators in the real program provides a solution to issue #1267. For users that have had no troubles with the real program running MPI, this will have no impact.
vlakshmanan-scala
pushed a commit
to scala-computing/WRF
that referenced
this pull request
Apr 4, 2024
TYPE: bug fix KEYWORDS: mpi, real, bug SOURCE: Marc Honnorat (EXWEXs) DESCRIPTION OF CHANGES: Problem: When running real.exe on multiple processes with MPI, one or more process occasionally crashes in setup_physics_suite (in share/module_check_a_mundo.F). This has been linked to wrf_dm_initialize non-blocking MPI. The real.exe occasionally crashes in setup_physics_suite (in share/module_check_a_mundo.F#L2640) because the latter uses model_config_rec % physics_suite, which on some machines is not initialized. The behavior is as if the broadcast of model_config_rec performed just before in main/real_em.F#L124 had not been received by all processes. I had never seen this bug before, it has only happened on one machine (an Intel-based cluster using Intel-MPI and ifort). The current fix makes sure that all processes are well synced before proceeding with setup_physics_suite. It solves the issue on my machine. Since this is immediately after reading in the namelist, no performance issues are expected as this read and broadcast of the namelist occurs only once per real / WRF / ndown run. Solution: An MPI barrier is added at the end of wrf_dm_initialize to force all of the processes to be synchronized before checking the namelist consistency. This is a simplification of PR wrf-model#1268, which had extra white space. ISSUE: Fixes wrf-model#1267 LIST OF MODIFIED FILES: M external/RSL_LITE/module_dm.F TESTS CONDUCTED: 1. On the only machine were I have seen the bug occur, this change fixes the problem. No other test was conducted since I couldn't reproduce the bug on another setup. 2. Jenkins testing is all PASS. RELEASE NOTES: When running real.exe on multiple processes with MPI, one or more processes occasionally crash in setup_physics_suite (in share/module_check_a_mundo.F). This has been traced to the fact that wrf_dm_initialize is non-blocking from an MPI point of view. The problem is intermittent and has only happened on one machine (an Intel-based cluster using Intel-MPI and ifort). An MPI barrier has been added at the end of wrf_dm_initialize to force all processes to be synchronized before checking namelist consistency.
vlakshmanan-scala
pushed a commit
to scala-computing/WRF
that referenced
this pull request
Apr 4, 2024
…l#1600) TYPE: [bug fix] KEYWORDS: real.exe, MPI, bug fix SOURCE: Marc Honnorat (EXWEXs) DESCRIPTION OF CHANGES: Problem: The communicator `mpi_comm_allcompute`, created by subroutine `split_communicator` called by `init_modules(1)`, is not explicitly activated for the call to `wrf_dm_bcast_bytes( configbuf, nbytes )` in real.exe. On some platforms, this may prevent broadcast of namelist configuration (put in `configbuf` after the call to `get_config_as_buffer()`) across the MPI processes _before_ the call to `setup_physics_suite()`. An example of a problematic platform: a cluster of Intel Xeon E5-2650 v4 running on CentOS Linux release 7.6.1810, with Intel Parallel Studio XE (various versions, including 2018u3 and 2020u4) and Intel MPI Library (same version). Solution: The initialization step used in the WRF executable never triggers a failure as described in issue wrf-model#1267. This PR reuses the temporary MPI context switch from WRF code. ISSUE: Fixes wrf-model#1267 LIST OF MODIFIED FILES: M main/real_em.F TESTS CONDUCTED: 1. The modification systematically solves the problem on the noted cluster. 2. Jenkins tests are all passing. RELEASE NOTE: A fix for an MPI synchronization bug related to (not used) split communicators in the real program provides a solution to issue wrf-model#1267. For users that have had no troubles with the real program running MPI, this will have no impact.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes MPI synchronization bug in real.exe #1267
TYPE: [bug fix]
KEYWORDS: mpi, real, bug
SOURCE: "Marc Honnorat (EXWEXs)"
DESCRIPTION OF CHANGES:
Problem:
When running real.exe on multiple processes with MPI, one or more process occasionnaly crashes in
setup_physics_suite(inshare/module_check_a_mundo.F). This may be due to the fact thatwrf_dm_initializeis apparently non-blocking from an MPI point of view. See #1267 for an example.Solution:
An MPI barrier is added just after the call to
wrf_dm_initializeto force all processes be synced before checking namelist consistancy.ISSUE:
Fixes #1267
LIST OF MODIFIED FILES:
TESTS CONDUCTED:
On the only machine were I have seen the bug occur, this change fixes the problem. No other test was conducted since I couldn't reproduce the bug on another setup.