Use restart md5 hashes for reproducibility tests

[aekiss]

Following from #266 (comment), I think comparing the md5 hashes in manifests/restart.yaml makes an ideal test for detecting reproducibility failures, because they are

• sensitive and thorough: these md5 hashes will detect bitwise changes in model state arising from any part of any component
• already generated by payu https://payu.readthedocs.io/en/latest/manifests.html#manifest-contents
• low-maintenance: CI will not need to be updated to handle model config changes, because the restarts (by definition) cover all the prognostic variables for all components

However, there are some wrinkles:

• Irrelevant restart metadata differences such as internally-stored timestamps would also alter the md5 hash, but there were no problems of this sort in ACCESS-OM2 non-reproducible runs access-om2#266 and I think this was also true of ACCESS-OM3 (not bitwise reproducible #40) - is that still the case? (note, we need to use md5 rather than binhash, as IIRC binhash includes the file modification date)

• manifests/restart.yaml tracks the initial rather than final state, so we'd need to update the manifests after the final run before comparing them (does payu setup do that?)

• While the restarts would be excellent for initial detection of repro failures, they are less useful for diagnosing their cause, due to a lack of granularity in time or component subroutines. So there's still a place for checking other component-specific files for diagnosis e.g. ocean.stats.

[aekiss]

Related: #41

[anton-seaice]

Also captured in ACCESS-NRI/model-config-tests#86

[aekiss]

Ah thanks, I thought this had already been suggested somewhere!

[dougiesquire]

There's also some related discussion in this issue where some potential problems with using the md5 restart hashes are raised.

MOM6 conveniently includes checksums for each restart variable in the metadata of the restart files, so using these seems like the best of all worlds.