Forcing dates out of sync - repeat 10-year spin-up

[mauricehuguenin]

Hi all,
we would like to spin-up the 1deg_jra55_iaf configuration with 1958-1967 repeat year forcing (10 years) for about 300 years. We start from the WOA.

After a ten year run until 1967, I set the forcing_end_date in accessom2.nml to:

&date_manager_nml
    forcing_start_date = '1958-01-01T00:00:00'
    forcing_end_date = '1968-01-01T00:00:00'

Running another ten years gives me the following error:

ERROR in accessom2_progress_date: forcing and experiment dates are out of sync.
forcing date: 1958-01-01T21:00:00.000
experiment date: 1968-01-02T00:00:00.000

Using use_restart_time = .false. in ice/cice_in.nml gives the same error.

Do you think this might be related to issue #22 we had before in libaccessom2?

[aekiss]

Yes it looks like that issue COSIMA/libaccessom2#22 again.

Also are you simply continuing the run for each 10-year segment, or starting a new run with an initial condition from end of the previous cycle?
Not sure if it would fix your problem, but you should be doing the latter, so the forcing and run dates are always the same - see #149

Also see https://github.com/COSIMA/access-om2/wiki/Tutorials#starting-a-new-experiment-using-restarts-from-a-previous-experiment

[rmholmes]

Thanks @aekiss. If possible it would be great if @mauricehuguenin did not have to do anything in between 10-year blocks, so that we can run 300 years with payu run -n 30 (doesn't that work with the core protocol looping over the full 50 years or so?). I think what you are suggesting involves modifying the restart's after each 10-year block? Perhaps this is not possible with the current setup.

[aekiss]

#149 is probably a side-issue. It's an unfortunate workaround to deal with leap years, but it's how we plan to run the next IAF spinup. It's a damn pain the Earth's orbital period is not divisible by its rotational period. Somebody should put in a bug report.

Anyway I'm not sure how they managed it with CORE, but if you use a standard Gregorian calendar on a 10-year cycle you can gain/lose days in run decades which have a different number of leap years from your forcing decade. But perhaps that's not an issue for your experiment?

We weren't aware of this issue in the IAF runs that went into the GMD paper, so they are affected by this offset.

[rmholmes]

We really don't care about gaining or losing days - the spin-up method is artificial enough as it is. We basically just want to do a RDF (repeat-decade-forcing) spinup rather than an RYF spinup.

Restarting each 10-year block as a new run seems unreasonable to me. We could potentially automate this process by running a post-script (from payu) that automatically changes the dates in the restart files after each run. However, it seems to me there should be a better away (but I'm open to ideas?).

Ahhh... while writing this I have seen that @nichannah has added a new namelist option allow_forcing_and_exp_date_mismatch that turns this out of sync error into a warning. This may be what we are looking for. @mauricehuguenin can you try adding allow_forcing_and_exp_date_mismatch = .true. to the date_manager.nml section of accessom2.nml? That should stop the error - but you'll still have to check whether the forcing dates are correct for the second 10-year cycle.

[aekiss]

Well spotted, that sounds like a simple fix.

[mauricehuguenin]

I added Ryan's suggestion to accessom2.nml and that seems to have worked. The model runs successfully for another 10 years. In the access-om2.err file I have the warnings that Ryan also mentioned above.

Payu seems to run into some difficulties at the end of the run while collating output. In access-om2.err I can see

...
WARNING in accessom2_progress_date: forcing and experiment dates are out of sync.
forcing date: 1967-12-30T22:30:00.000
experiment date: 1977-12-31T00:00:00.000
 ice: Error creating restart ncfile ./RESTART/iced.1968-01-01-00000.nc
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 217 in communicator MPI_COMM_WORLD
with errorcode 1.

that it has problems creating the restart files.

I made my logs publicly readable in /home/561/mv7494/access-om2/1deg_jra55v131_iaf

[mauricehuguenin]

I just did another 10-year run starting from 1968-01-01 and forcing the model with JRA iaf from 1958-01-01 to 1968-01-01. In accessom2.nml I have

&date_manager_nml
    forcing_start_date = '1958-01-01T00:00:00'
    forcing_end_date = '1968-01-01T00:00:00'

and I changed ice/cice_in.nml to

runtype = 'continue'
restart = .true.

as suggested by a comment in the cosima slack channel from the 18th of December 2019.

The run gives me the following warning I have not had before
WARNING from PE 0: diag_util_mod::opening_file: one axis has auxiliary but the corresponding field is NOT found in file ocean_month
and I get the error
ice: Error creating restart ncfile ./RESTART/iced.1968-01-01-00000.nc

Ryan assumes it might be related to this issue: COSIMA/cice5#38

Any idea why the restart files cannot be created @aekiss @nichannah?

[rmholmes]

Maurice's run is here - /home/561/mv7494/access-om2/1deg_jra55v131_iaf

Note that we are not neccessarily tied to that exact decade (1958-1968). If the problem ends up being with leap years we could potentially shift the decade (e.g. 1962-1972).

[nichannah]

I have fixes COSIMA/cice5#38 and COSIMA/libaccessom2#22 . I have also added better error reporting - hopefully we will get an error code when the restart write fails.

[aekiss]

@nichannah those links seem unrelated to this issue. Did you mean COSIMA/libaccessom2#22 and COSIMA/cice5#38?

[mauricehuguenin]

This issues has been solved in version v1.3.1 of 1deg_jra55_iaf.

With this new version I did not have to change ice/cice_in.nml or accessom2.nml as suggested above.