cpld_control_p8 and cpld_debug_p8 frequently freeze on hera.gnu

[SamuelTrahanNOAA]

Description

When running regression tests, the cpld_control_p8 and cpld_debug_p8 frequently freeze at this point on hera.gnu:

==> /scratch1/NCEPDEV/stmp2/Samuel.Trahan/FV3_RT/rt_21887/cpld_control_p8.err <==
150: WARNING from PE     0: Unused line in INPUT/MOM_input : ODA_INCUPD_NHOURS = 6
150:
  0: Using non-aerosol-aware version of Thompson microphysics
  0:    calling table_ccnAct routine
  0:    creating qc collision eff tables
  0:    creating rain evap table
  0:    creating ice converting to snow table
  0:    creating rain collecting graupel table
  0:    creating freezing of water drops table
  0:   ... DONE microphysical lookup tables

==> /scratch1/NCEPDEV/stmp2/Samuel.Trahan/FV3_RT/rt_21887/cpld_control_p8.out <==
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0:  in radiation_clouds_prop=           8 F           4 F F           2           1
  0: PASS: fcstRUN phase 1, n_atmsteps =                0 time is        12.378961
  0: UFS Aerosols: Advancing from 2021-03-22T06:00:00 to 2021-03-22T06:12:00

The gdb debugger has yielded little results because it doesn't understand the stack of the running fv3.exe process.

To Reproduce:

Run the cpld_control_p8 or cpld_debug_p8 test a few times with hera.gnu.

Additional context

Tested with hash b004ce4

[junwang-noaa]

@SamuelTrahanNOAA Is this issue fixed in #1645?

[SamuelTrahanNOAA]

No, it was not fixed. During my testing of that PR, it froze several times.

[junwang-noaa]

@jkbk2004 Will your team look into this issue?

[zach1221]

Given the results with testing cpld_control_p8_gnu, control_p8_gnu and cpld_debug_p8_gnu on Hera, regarding ufs-wm issue #1440 , this issue should be closed as well once ufs-wm PR #1912 is merged. Cpld_debug_p8_gnu was run several times on hera with standard wallclock and there was no more freezing experienced.