Retaining files in gdas ROTDIR needed for Fit2Obs

[CatherineThomas-NOAA]

Initially reported in Fit2Obs Issue #3, failures were experienced on Hera when the arch step would scrub the gdas directories before Fit2Obs had finished.

A possible solution is to modify arch.sh to retain the needed files for Fit2Obs for longer than the default RMOLDEND. This is already done for sflux and RESTART files for the GLDAS step.

@KateFriedman-NOAA @RussTreadon-NOAA @jack-woollen

[RussTreadon-NOAA]

WCOSS_D tests

Locally modify arch.sh used by Mars v16rt2c to test two options as possible solutions to this issue:

1. retain gdas files used by fit2obs
2. copy gdas files used by fit2obs to vrfyarch

With option 1 there is no need to modify fit2obs. Instead, arch.sh is modified such that it retains gdas files used by fit2obs while removing other files from $ROTDIR/gdas.$PDY/$cyc/atmos which are not used. This approach is similar to how the gdas files required by GLDAS are handled. The GLDAS gdas files are retained while other files in the gdas directory are removed. Since the process for retaining GLDAS and fit2obs gdas files is similar, existing logic in arch.sh for GLDAS gdas file retention was expanded to include fit2obs gdas files. This change has been cycled in the v16rt2c and is working. Both GLDAS and fit2obs gdas files are retained in gdas directories while other files are removed.

Option 2 requires changes to fit2obs. Variable COM_INA in fit2obs subfits_dell_nems needs to be changed to point at $ROTDIR/vrfyarch/gdas.$PDY/$cyc instead of $ROTDIR/gdas.$PDY/$cyc/atmos. Concurrent with this change is a modification to arch.sh to copy gdas fit2obs files to $ROTDIR/vrfyarch/gdas.$PDY/$cyc. Logic already exists in arch.sh to copy the gfs forecast files required by fit2obs to $ROTDIR/vrfyarch/gfs.$PDY/$cyc. This logic is expanded to include copies of gdas fit2obs files. The modified arch.sh has been cycled in the v16rt2c and is working. Directory $ROTDIR/vrfyarch is populated with gdas.$PDY/$cyc directories containing gdas fit2obs files.

The second part of testing option 2 is to modify subfits_dell_nems. This was done in an offline mode. COM_INA was modified to point at $ROTDIR/vrfyarch/gdas.$PDY/$cyc and fit2obs rerun. The resulting fit2obs files were not b4b identical with those created via the existing subfits_dell_nems and current definition of COM_INA. This is unexpected. Changing the location from which b4b identical input gdas files are copied should not change fit2obs results.

This test was run on Mars during the 4/12 parallel production test. The output from this test was saved and the modified subfits_dell_nems run again. Output from the second run is b4b identical with output from the first run. It is possible the changes being testing during the Mars parallel production test yield non-b4b results with respect to the pre-test environment? This test will be rerun for a different v16rt2c case after Mars is returned to developers after the parallel production test finishes.

[RussTreadon-NOAA]

WCOSS_D tests - more cases

Reran fit2obs for 2021041106, 2021041112, 2021041118, and 2021041200 following return of Mars to developers. Rerun used gdas files from $ROTDIR/vrfyarch/gdas.$PDY/$cyc. For each date the rerun generated fit2obs files b4b identical with those generated when using gdas files from $ROTDIR/gdas.$PDY/$cyc/atmos. This is expected. The directory from which the gdas files are copied should not impact fit2obs results.

What remains now is to decide whether to adopt solution 1 or 2. Option 1 does not require any changes to fit2obs. Option 2 requires changing the path for COM_INA in fit2obs subfits_$machine_nems, where machine={dell, hera, orion}.

Depending upon which solution, 1 or 2, is adopted, scripting for the the option not chosen should be removed from arch.sh. Fit2obs and workflow code managers can discuss and decide.

The modified arch.sh with both options present is on Mars in /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/global-workflow/release_gfsv16.1/jobs/rocoto

[jack-woollen]

I would prefer not changing fit2obs for this since option 1 seems to work fine and I don't see any additional advantage in using option 2, unless I'm missing something. Plus option 1 is already implemented regarding other files and for similar reasons.

[RussTreadon-NOAA]

As per Jack's request option 2 has been removed from the locally modified version of arch.sh used by the v16rt2c. Pending the modified arch.sh successfully running overnight, I would like to commit the modified arch.sh, which includes option 1, to release/gfsv16.1.

[RussTreadon-NOAA]

Details regarding 88c54d9

Option 1 discussed in this issue has been committed to workflow branch release/gfsv16.1 at 88c54d9. Users and developers should be aware of two details in the implementation of this option.

Detail 1
Retention of gdas files required by Fit2Obs is embedded in the section of arch.sh which retains GLDAS files. One of the dates used in the logic is GLDAS_DATE. GLDAS needs to retain files in $ROTDIR/gdas.$PDY/$cyc/atmos longer than Fit2Obs. This is fine. It just means gdas Fit2Obs files will be retained longer than actually needed. Developers could modify the script logic to be more precise with regards to Fit2Obs gdas file retention.

Detail 2
Scripting in arch.sh removes $ROTDIR/vrfyarc/$CDUMP.$rPDY. The removal date is FHMAX_FITS+36 hours before CDATE. The +36 is potentially problematic.

config.vrfy sets VBACKUP_FITS. This variable is used to determine the date for which vrfy.sh submits Fit2Obs. Most workflow branches set VBACKUP_FITS=24. The scripting in arch.sh that cleans up the Fit2Obs gfs archive does not take VBACKUP_FITS into consideration. Since +36 is greater than the common VBACKUP_FITS setting of 24, required gfs Fit2Obs files are not removed. Problems would arise if VBACKUP_FITS is greater than 36. It would be good to take VRFYBACKUP_FITS into consideration when determining which gfs Fit2Obs archive directory to remove.

arch.sh does not source config.vrfy so it does not know what value is used for VRFYBACKUP_FITS. Variable VRFYBACKUP_FITS could be added to config.arch. Alternatively, arch.sh could source config.vrfy.

[KateFriedman-NOAA]

The same changes made by @RussTreadon-NOAA @ 88c54d9 also made their way into the develop branch and are currently being used to retain the needed amount of inputs for the Fit2Obs package. The Fit2Obs package (when run via global-workflow) now looks for its inputs in the /vrfyarch folder instead of the $CDUMP.$PDY/$cyc/$COMPONENT folders that may have already been scrubbed.

See these blocks of the newer scripts/exglobal_archive.sh script:

https://github.com/NOAA-EMC/global-workflow/blob/develop/scripts/exglobal_archive.sh#L86:L102
https://github.com/NOAA-EMC/global-workflow/blob/develop/scripts/exglobal_archive.sh#L387:L407

Closing this issue as completed.

[KateFriedman-NOAA]

Will also close matching NOAA-EMC/Fit2Obs#3.