ESMF managed threading job submission issue with large number of cores on wcoss2

[junwang-noaa]

Description

It is found that when submitting large number of cores (>20000), the C768 coupled test with ESMF managed threading failed to submit. While the error message does not show up when running with the same number of cores with single thread using traditional threading.

To Reproduce:

1. run C768 coupled test with layout 32x32 using 4 threads and ocean & ice with 1 thread.
2. following error message shows up: "start RPC: RPC timeout after 30s, received 38 of 129 responses", the job can't run.
3. The same test with atm layout 64x64, atm & ocean &ice all have 1 thread, the job can start to run.

Additional context

Output

[junwang-noaa]

There are updates in ESMF 8.5.0. We will test UFS WM with ESMF 8.5.0 on C5.

[junwang-noaa]

Since we are starting to use traditional threading for high resolution runs, I will close the issue until a solution is provided in ESMF library for ESMF managed threading