Missing or Unable to Read gen_ens_prod.txt Input File

[ShannonShields-NOAA]

I submitted a Pull Request last week and Shelley has been testing my scripts on Cactus. When she ran jevs_cfs_subseasonal_grid2grid_stats.ecf, I checked the output and it was missing the monthly-averaged values for sea ice concentration. I know this ran fine for me, so this seemed strange. When I looked at line 103147 in /lfs/h2/emc/stmp/shelley.melchior/evs_test/prod/tmp/52826/grid2grid_stats/METplus_output/logs/reformat_data_GridStat_fcstcfs_NetCDF_grid2grid_stats_seaice_Monthly_Concentration_for20230207_runon20230303193658.log, it said:
ERROR : is_bufr_file() -> unable to read header from input file "/lfs/h2/emc/stmp/shelley.melchior/evs_test/prod/tmp/52826/grid2grid_stats/METplus_output/stage/file_lists.20230303193658/20230108000000_708_gen_ens_prod.txt"
I also noticed that the Days 32-33 average did not run either due to another error with the stage file list. On line 141786 in reformat_data_GridStat_fcstcfs_NetCDF_grid2grid_stats_sst_Daily_SST_for20230207_runon20230303193658.log:
ERROR : grd_file_type() -> file does not exist "/lfs/h2/emc/stmp/shelley.melchior/evs_test/prod/tmp/52826/grid2grid_stats/METplus_output/stage/file_lists.20230303193658/20230105000000_792_gen_ens_prod.txt"
These errors seem to be METplus-related (as I ran this step fine before so it should not be my scripts). I am not understanding what is happening though as other forecast hours run fine and METplus seems to be adding the CFS forecast files correctly in a list:
DEBUG: Wrote list of filenames to /lfs/h2/emc/stmp/shelley.melchior/evs_test/prod/tmp/52826/grid2grid_stats/METplus_output/stage/file_lists.20230303193658/20230105000000_792_gen_ens_prod.txt
Unfortunately, it seems the stage directory is removed after the run (not sure why) so I can't check the file lists.
My initial thoughts are that maybe this is related to some looping error because some of the CFS data files are the same for the daily, weekly, and monthly periods that I am running. For example, file_lists.20230303193658/20230108000000_708_gen_ens_prod.txt will be generated for daily as well as monthly.
In one of my scripts I am setting: "working_dir_list.append(os.path.join(working_output_base_dir, 'stage',
'file_lists'))"
so I'm not sure if this is having an effect on the directory errors.

[georgemccabe]

Hi @ShannonShields-NOAA,

Starting in METplus v5.0.0, the stage directory is scrubbed by default at the end of a METplus run. See the Major Changes section of the METplus Version 5.0.0 Release Notes. To retain these files after a run, you can set SCRUB_STAGING_DIR = False in a METplus config file. These files are removed at the very end of the METplus run, so subsequent wrappers in the process list should be able to access the files for a given run. If the file list files are read output of the METplus run, then disabling the scrubbing of the directories should make them available.

Another thing to note is the file lists directory name now includes the timestamp of the run, e.g. file_lists.20230303193658, so setting working_dir_list.append(os.path.join(working_output_base_dir, 'stage', 'file_lists')) will not append the correct path. You can override the value of FILE_LISTS_DIR in a METplus config file to change this directory name.

Default value:
FILE_LISTS_DIR = {STAGING_DIR}/file_lists.{LOG_TIMESTAMP}
New value:
FILE_LISTS_DIR = {STAGING_DIR}/file_lists

If you continue to have issues, please share a METplus log file and corresponding final configuration file (logs/metplus.log.YYYYMMDDHHMMSS and metplus_final.conf.YYYYMMDDHHMMSS with matching timestamps) so we can take a closer look.

[ShannonShields-NOAA]

Ah ok, thank you for the info George! I will try setting SCRUB_STAGING_DIR = False in my METplus config file and conduct some test runs.
Thank you for your assistance and I'll keep you informed.

[ShannonShields-NOAA]

I figured out what was happening with the stage/file_lists errors. I had to set FILE_LISTS_DIR = {STAGING_DIR}/file_lists_{ENV[VERIF_TYPE]}_{ENV[job_name]}.{LOG_TIMESTAMP}. The SST and sea ice directories are now split such as:
/lfs/h2/emc/stmp/shannon.shields/evs_test/prod/tmp/126906_new_dirfix/grid2grid_stats/METplus_output/stage/file_lists_seaice_Concentration.20230307232600
/lfs/h2/emc/stmp/shannon.shields/evs_test/prod/tmp/126906_new_dirfix/grid2grid_stats/METplus_output/stage/file_lists_sst_SST.20230307232600
Thank you for your help!