regtest: OMP, number of tasks and restart reproducibility

[JessicaMeixner-NOAA]

Add regtests to test that different number of OMP threads and different number of tasks produce the same answer. Also add a restart reproducibility test (ie. if you write a restart half way through a run, and start from that restart you should get the same answer).

[aliabdolali]

Hi @JessicaMeixner-NOAA
I have a setup for restart file reproducibility for the tp2.17 (unstructured) which might need revisiting. For example, can we add commands in matrix and copy restart001.ww3 to another directory? Also we need two ww3_shel.inp for two consecutive runs.
if yes, it is very simple to set it.
i.e. cp ww3_tp2.17/work_a/restart002.ww3 ww3_tp2.17/work_b/restart.ww3
run_test ...
or the structured grids, you pick one and tell me the name, I can do the same.
AA

[JessicaMeixner-NOAA]

@aliabdolali awesome. We might do a couple of the structured grid tests? Something that looks more like GEFSv12 or GFSv16 grid set-up with a multi run and then also a single grid? (Although single grid will likely be reproducible if a multi-grid case is). I'll have to look and see which one would be best.

[MatthewMasarik-NOAA]

@JessicaMeixner-NOAA or @aliabdolali, I believe we have run_tests to check each of these: OMP, tasks, and restart reproducibility. Do any of those need more attention, or could this issue be closed?