WW3 not reproducing for different number of pets

[JessicaMeixner-NOAA]

Description

If you change the number of pets the WW3 model is using, the results do not reproduce the same answers as before. For the wav-atm cases, just the restart file is different however, for the fully coupled case the output from the wave model, and the restarts are different. This was discovered when trying to make a PR to put in the fix for the thread-safe issue, which is why that PR has not yet been submitted. Details of the fixing this issue, since they are in WW3 will be in the WW3 issue that is related to this: NOAA-EMC/WW3#317

To Reproduce:

Change the number of pets in a test with waves, the results will be different than before.

[JessicaMeixner-NOAA]

So the fv3_ccpp_gfdlmprad_atmwav and fv3_ccpp_gfdlmprad tests reproduce themselves even with different number of MPI tasks, however, there are no restart checks in these tests.

For the cpld_control_wave test, everything but the wave restart file is the same when using different number of MPI tasks for waves.

For the cpld_bmarkfrac_wave and cpld_bmarkfrac_wave_v16 tests (and likely their non-frac counter parts), the tests does not pass when using different number of MPI tasks and even the atm files along with wave gridded output and restarts are different in this case.

[JessicaMeixner-NOAA]

For the cpld_bmarkfrac_wave_v16 test here is an example of the magnitude of differences after 12 hours:

[junwang-noaa]

@JessicaMeixner-NOAA May I ask if this issue is resolved in WW3?

[JessicaMeixner-NOAA]

Not yet, but @MatthewMasarik-NOAA has been looking into what might be causing this just from the WW3 standalone side and has an idea of what one issue there is. Hopefully solving that problem will also solve this issue.

[JessicaMeixner-NOAA]

I have added tests for this issue here: https://github.com/jessicameixner-noaa/ufs-weather-model/tree/feature/addwavetests note the C96 does not have this issue, but C384 does. Updates on progress on the underlying WW3 issue are being updated here: NOAA-EMC/WW3#317 still no solution, but the issue has been narrowed down to the fact that when ice is updated during a wave forecasts, the restart files do not reproduce in VA after that, but it's still not clear how to fix this issue.