Plot_allvars task goes DEAD due to case mismatch

[gspetro-NOAA]

Expected behavior

The plotting scripts should run successfully as part of the workflow. The plotting tasks are based on the original functional plotting scripts, and since the addition of the plotting scripts to the workflow, it seems that no one else has encountered issues.

Current behavior

On Hera, I get an error when trying to run the plotting tasks on the "out-of-the-box" case (25km CONUS grid, FV3_GFS_v16 physics, DATE_FIRST_CYCL: 2019061518, FCST_LEN_HRS: 12). exregional_plot_allvars.py cannot open the files in postprd due to a case mismatch. The files in postprd are named like rrfs.t18z.prslev.f***.RRFS_CONUS_25km.grib2, but the plot_allvars task is trying to open a file named rrfs.t18z.prslev.f***.rrfs_conus_25km.grib2. (See image below.)

Machines affected

Hera/intel
Possibly others, but I have only tested on Hera.

Steps To Reproduce

Here is my config.yaml file:

I generated the workflow and waited for it to terminate as usual, but the task went DEAD. The error message from the log is in the section above.

Detailed Description of Fix

Removing .lower() from line 345 of exregional_plot_allvars.py and line 355 of exregional_plot_allvars_diff.py fixed the issue for me. (These are the lines that set POST_OUTPUT_DOMAIN_NAME.) After removing .lower() in those two places, I successfully ran the experiment using the same config file (except for the EXPT_SUBDIR field).

However, I am surprised that this hasn't caused problems sooner since the .lower() syntax was in the original plotting script. I am hoping others can verify whether the current code has this issue (and whether the fix works) on other systems/with other experiments.

[MichaelLueken]

@gspetro-NOAA Have you tried running the config.grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional_plot.yaml test as well? @danielabdi-noaa added this test when he added the plotting scripts to the workflow. It would be interesting to see if you encounter the same issue while running this test.

[MichaelLueken]

I have just finished running the grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional_plot test on Cheyenne and the plot_allvars task was DEAD. I know that this test successfully passed when @danielabdi-noaa added it. I'll try wand see what might be happening.

[gspetro-NOAA]

@MichaelLueken Thanks!

[danielabdi-noaa]

I think I found what the issue is. PR #505 has changed the way POST_OUTPUT_DOMAIN_NAME to var_defns.sh file.
It keeps the capitalization of the domain name while the old one uses lowercases, e.g.

POST_OUTPUT_DOMAIN_NAME='custom_esggrid'

This causes an inconsistency in the plotting scripts because the script does a .lower() itself and will be confused when fed a capitalized domain name. I will fix this once i decide whether it is better to fix the plotting script or modify setup.py to write a lower-cased domain name in var_defns.sh