WOMBATlite tracers are not reproducible

[dougiesquire]

Historical repro checks on the dev-1deg_jra55do_ryf_wombatlite branch fail because WOMBATlite tracers are not bitwise reproducible - e.g. see ACCESS-NRI/access-om3-configs#199 (comment). I think this will take some digging to resolve, so I'm opening this issue to keep track of things.

[MartinDix]

What about WOMBATlite in ESM1.6? Though CABLE isn't reproducible in some versions so would need to be careful what was tested.

[dougiesquire]

To try and determine whether the issue is in the generic tracer shared code or specifically in the WOMBATlite module, I set up a 1 deg ACCESS-OM3 + BLING config in this branch: 289-1deg_jra55do_ryf_bling.

This configuration also does not have historical repro of BLING tracers:

diff --git a/control/om3_1deg_jra55do_ryf_bling-test_bit_repro_historical/testing/checksum/historical-3hr-checksum.json b/checksum/historical-3hr-checksum.json
index c673c39..78dd7c8 100644
--- a/control/om3_1deg_jra55do_ryf_bling-test_bit_repro_historical/testing/checksum/historical-3hr-checksum.json
+++ b/checksum/historical-3hr-checksum.json
@@ -44,25 +44,25 @@
       "B64AD181B58D3F4B"
     ],
     "alk": [
-      "1BE0A1F18199262"
+      "1F4C091BFFC81F3E"
     ],
     "ave_ssh": [
       "40E5A21D6F0D1F87"
     ],
     "biomass_p": [
-      "E6C8E6FA73781EE1"
+      "321423192F3052EA"
     ],
     "cased": [
-      "385E1D16040802FD"
+      "1916F3449A3B4B1A"
     ],
     "chl": [
-      "3CFCDA0157D051EF"
+      "4DC58E47DE015488"
     ],
     "co3_ion": [
-      "5E249562C7AC5725"
+      "92A52D91592F8461"
     ],
     "dic": [
-      "4BB5268ACCFD468E"
+      "F1C1D4A3E01616FB"
     ],
     "diffu": [
       "749F723BA63A013C"
@@ -71,10 +71,10 @@
       "CD7614674E679DA8"
     ],
     "dop": [
-      "3EA7A97240B9A2EB"
+      "6893C9AE6145B4D2"
     ],
     "fed": [
-      "57675B5F0ED845D5"
+      "1CC7FE18C07E404B"
     ],
     "frazil": [
       "5FE9C9F905239FC4"
@@ -86,19 +86,19 @@
       "A815BF54039C2D82"
     ],
     "htotal": [
-      "D0E323115D157D09"
+      "FBD8FE65155581DE"
     ],
     "irr_mem": [
-      "61642F0F4BCBE233"
+      "5F2163824369219"
     ],
     "o2": [
-      "4596FA109C8FF43E"
+      "36CBB6A2BEF40FCA"
     ],
     "p_surf_EOS": [
       "161DADB852A006F1"
     ],
     "po4": [
-      "5A45C4D691BF13D2"
+      "3911BB7A6860093D"
     ],
     "sfc": [
       "68FCD5E9DE45D1DF"

So this check didn't really narrow things down at all. The issue could be coming from:

• the generic tracer shared code
• our modifications to the MOM6 NUOPC cap to allow generic tracers
• both the BLING and WOMBATlite modules
• any combination of the above

[dougiesquire]

What about WOMBATlite in ESM1.6?

@MartinDix WOMBATlite tracers in ESM1.6 are bitwise reproducible. I checked this dev-preindustrial+concentrations configuration using model-config-tests from this PR. After first updating the reference checksums because they were out of date:

$ model-config-tests -m checksum
================================================= test session starts ==================================================
platform linux -- Python 3.10.0, pytest-8.3.4, pluggy-1.5.0
rootdir: /g/data/tm70/ds0092/model/config/model-config-tests
configfile: pyproject.toml
plugins: cov-6.0.0
collected 56 items / 55 deselected / 1 selected

../model-config-tests/src/model_config_tests/test_bit_reproducibility.py .                                       [100%]

===================================== 1 passed, 55 deselected in 517.51s (0:08:37) =====================================

So it's something about our use with MOM6 that causes the issue. Our changes to the MOM6 NUOPC cap seem probable.

ADDED: ACCESS-ESM1.6 isn't restart reproducible (2x1month vs 1x2month runs differ) but that is not due to WOMBATlite