Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Add GDAS and GEFS datasets to the workflow #526

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
9ae8a96
added GEFS and GDAS logic
Dec 22, 2022
c6bd39b
added GEFS and GDAS logic, and function to merge GEFS files
Dec 22, 2022
1d070ed
added logic to handle ensemble datasets
Dec 22, 2022
710e4ee
removed unneeded var
Dec 22, 2022
a853888
added logic to handle GEFS list filenames
Dec 22, 2022
26946d1
removed print statement
Dec 22, 2022
cfc8cd8
added logic to handle empty member arg
Dec 23, 2022
a4e322a
fixed typo
Dec 27, 2022
f6e0d3b
updated output paths
Jan 5, 2023
4af4c96
update file path
Jan 5, 2023
85ab1d7
troubleshooting path
Jan 5, 2023
1117866
removed print statements
Jan 5, 2023
8b5d428
Added GEFS and GDAS vars to the extrn ics and lbcs doc
Jan 9, 2023
44385cf
removed incorrect GDAS logic
Jan 11, 2023
6952c47
removed incorrect GDAS logic
Jan 11, 2023
ec969cb
fixed var names and how filenames are found in the GEFS function
Jan 18, 2023
39a9bcd
updated GDAS extrn model name
Jan 18, 2023
cef51f0
fixed logic around file path and redid the write summary function
Jan 18, 2023
d5cb371
added sfc filename for GDAS fcst
Jan 18, 2023
6d96e2c
removed unneeded py package
Jan 18, 2023
d68e685
updated file path to match what the retrieve data uses
Jan 18, 2023
87aaa9c
added more defined file path
Jan 18, 2023
49552c4
cleaned up script
Jan 18, 2023
c66236a
update gefs merge function
Feb 1, 2023
420080d
Add logic to remove hpss source from GEFS test
Feb 6, 2023
aa15def
Merge branch 'ufs-community:develop' into feature/rrfs-ensemble-bc
EdwardSnyder-NOAA Feb 7, 2023
57df940
Fixed cla.members bug
Feb 7, 2023
517e660
Added two new WE2E tests - get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_…
MichaelLueken Feb 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/UsersGuide/source/ConfigWorkflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -717,7 +717,7 @@ For each workflow task, certain parameter values must be passed to the job sched
Maximum number of times to attempt the task.

``EXTRN_MDL_NAME_ICS``: (Default: "FV3GFS")
The name of the external model that will provide fields from which initial condition (IC) files, surface files, and 0-th hour boundary condition files will be generated for input into the forecast model. Valid values: ``"GSMGFS"`` | ``"FV3GFS"`` | ``"RAP"`` | ``"HRRR"`` | ``"NAM"``
The name of the external model that will provide fields from which initial condition (IC) files, surface files, and 0-th hour boundary condition files will be generated for input into the forecast model. Valid values: ``"GSMGFS"`` | ``"FV3GFS"`` | ``"GEFS"`` | ``"GDAS"`` | ``"RAP"`` | ``"HRRR"`` | ``"NAM"``

``EXTRN_MDL_ICS_OFFSET_HRS``: (Default: 0)
Users may wish to start a forecast using forecast data from a previous cycle of an external model. This variable indicates how many hours earlier the external model started than the FV3 forecast configured here. For example, if the forecast should start from a 6-hour forecast of the GFS, then ``EXTRN_MDL_ICS_OFFSET_HRS: "6"``.
Expand Down Expand Up @@ -799,7 +799,7 @@ For each workflow task, certain parameter values must be passed to the job sched
Maximum number of times to attempt the task.

``EXTRN_MDL_NAME_LBCS``: (Default: "FV3GFS")
The name of the external model that will provide fields from which lateral boundary condition (LBC) files (except for the 0-th hour LBC file) will be generated for input into the forecast model. Valid values: ``"GSMGFS"`` | ``"FV3GFS"`` | ``"RAP"`` | ``"HRRR"`` | ``"NAM"``
The name of the external model that will provide fields from which lateral boundary condition (LBC) files (except for the 0-th hour LBC file) will be generated for input into the forecast model. Valid values: ``"GSMGFS"`` | ``"FV3GFS"`` | ``"GEFS"`` | ``"GDAS"`` | ``"RAP"`` | ``"HRRR"`` | ``"NAM"``

``LBC_SPEC_INTVL_HRS``: (Default: "6")
The interval (in integer hours) at which LBC files will be generated. This is also referred to as the *boundary update interval*. Note that the model selected in ``EXTRN_MDL_NAME_LBCS`` must have data available at a frequency greater than or equal to that implied by ``LBC_SPEC_INTVL_HRS``. For example, if ``LBC_SPEC_INTVL_HRS`` is set to "6", then the model must have data available at least every 6 hours. It is up to the user to ensure that this is the case.
Expand Down
15 changes: 12 additions & 3 deletions docs/UsersGuide/source/InputOutputFiles.rst
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ The environment variables ``FIXgsm``, ``TOPO_DIR``, and ``SFC_CLIMO_INPUT_DIR``

Initial Condition/Lateral Boundary Condition File Formats and Source
-----------------------------------------------------------------------
The SRW Application currently supports raw initial and lateral boundary conditions from numerous models (i.e., FV3GFS, NAM, RAP, HRRR). The data can be provided in three formats: :term:`NEMSIO`, :term:`netCDF`, or :term:`GRIB2`.
The SRW Application currently supports raw initial and lateral boundary conditions from numerous models (i.e., FV3GFS, GEFS, GDAS, NAM, RAP, HRRR). The data can be provided in three formats: :term:`NEMSIO`, :term:`netCDF`, or :term:`GRIB2`.

To download the model input data for the 12-hour "out-of-the-box" experiment configuration in ``config.community.yaml`` file, run:

Expand Down Expand Up @@ -286,7 +286,7 @@ The paths to ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBC
EXTRN_MDL_SOURCE_BASEDIR_LBCS: <path/to/ufs-srweather-app/input_model_data/FV3GFS/grib2/YYYYMMDDHH>
EXTRN_MDL_DATA_STORES: disk

The two ``EXTRN_MDL_SOURCE_BASEDIR_*CS`` variables describe where the :term:`IC <ICs>` and :term:`LBC <LBCs>` file directories are located, respectively. For ease of reusing ``config.yaml`` across experiments, it is recommended that users set up the raw :term:`IC/LBC <IC/LBCs>` file paths to include the model name (e.g., FV3GFS, NAM, RAP, HRRR), data format (e.g., grib2, nemsio), and date (in ``YYYYMMDDHH`` format). For example: ``/path-to/input_model_data/FV3GFS/grib2/2019061518/``. While there is flexibility to modify these settings, this structure will provide the most reusability for multiple dates when using the SRW Application workflow.
The two ``EXTRN_MDL_SOURCE_BASEDIR_*CS`` variables describe where the :term:`IC <ICs>` and :term:`LBC <LBCs>` file directories are located, respectively. For ease of reusing ``config.yaml`` across experiments, it is recommended that users set up the raw :term:`IC/LBC <IC/LBCs>` file paths to include the model name (e.g., FV3GFS, GEFS, GDAS, NAM, RAP, HRRR), data format (e.g., grib2, nemsio), and date (in ``YYYYMMDDHH`` format). For example: ``/path-to/input_model_data/FV3GFS/grib2/2019061518/``. While there is flexibility to modify these settings, this structure will provide the most reusability for multiple dates when using the SRW Application workflow.

When files are pulled from NOAA :term:`HPSS` (rather than downloaded from the data bucket), the naming convention looks something like:

Expand All @@ -296,13 +296,18 @@ When files are pulled from NOAA :term:`HPSS` (rather than downloaded from the da
* ICs: ``gfs.t{cycle}z.atmanl.nemsio`` and ``gfs.t{cycle}z.sfcanl.nemsio``;
* LBCs: ``gfs.t{cycle}z.atmf{fhr}.nemsio``

* GDAS (NETCDF):

* ICs: ``gdas.t{cycle}z.atmf{fhr}.nc`` and ``gdas.t{cycle}z.sfcf{fhr}.nc``;
* LBCs: ``gdas.t{cycle}z.atmf{fhr}.nc``

* RAP (GRIB2): ``rap.t{cycle}z.wrfprsf{fhr}.grib2``
* HRRR (GRIB2): ``hrrr.t{cycle}z.wrfprsf{fhr}.grib2``

where:

* ``{cycle}`` corresponds to the 2-digit hour of the day when the forecast cycle starts, and
* ``{fhr}`` corresponds to the 2- or 3-digit nth hour of the forecast (3-digits for FV3GFS data and 2 digits for RAP/HRRR data).
* ``{fhr}`` corresponds to the 2- or 3-digit nth hour of the forecast (3-digits for FV3GFS/GDAS data and 2 digits for RAP/HRRR data).

For example, a forecast using FV3GFS GRIB2 data that starts at 18h00 UTC would have a {cycle} value of 18, which is the 000th forecast hour. The LBCS file for 21h00 UTC would be named ``gfs.t18z.pgrb2.0p25.f003``.

Expand Down Expand Up @@ -345,6 +350,8 @@ NOMADS: https://nomads.ncep.noaa.gov/pub/data/nccf/com/{model}/prod, where model

* GFS (GRIB2 or NEMSIO) - available for the last 10 days
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/
* GDAS (NETCDF) sfc files - available for the last 2 days
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod
* NAM - available for the last 8 days
https://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/
* RAP - available for the last 2 days
Expand All @@ -355,6 +362,8 @@ NOMADS: https://nomads.ncep.noaa.gov/pub/data/nccf/com/{model}/prod, where model
AWS S3 Data Buckets:

* GFS: https://registry.opendata.aws/noaa-gfs-bdp-pds/
* GEFS: https://registry.opendata.aws/noaa-gefs/
* GDAS: https://registry.opendata.aws/noaa-gfs-bdp-pds/
* HRRR: https://registry.opendata.aws/noaa-hrrr-pds/ (necessary fields for initializing available for dates 2015 and newer)

Google Cloud:
Expand Down
1 change: 1 addition & 0 deletions parm/data_locations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ GDAS:
- gdas.t{hh}z.sfcf{fcst_hr:03d}.nc
fcst:
- gdas.t{hh}z.atmf{fcst_hr:03d}.nc
- gdas.t{hh}z.sfcf{fcst_hr:03d}.nc
nomads:
protocol: download
url: https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/enkfgdas.{yyyymmdd}/{hh}/atmos/mem{mem:03d}
Expand Down
90 changes: 87 additions & 3 deletions scripts/exregional_get_extrn_mdl_files.sh
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ if [ "${ICS_OR_LBCS}" = "ICS" ]; then
fi
fcst_hrs=${TIME_OFFSET_HRS}
file_names=${EXTRN_MDL_FILES_ICS[@]}
if [ ${EXTRN_MDL_NAME} = FV3GFS ] ; then
if [ ${EXTRN_MDL_NAME} = FV3GFS ] || [ "${EXTRN_MDL_NAME}" == "GDAS" ] ; then
file_type=$FV3GFS_FILE_FMT_ICS
fi
input_file_path=${EXTRN_MDL_SOURCE_BASEDIR_ICS:-$EXTRN_MDL_SYSBASEDIR_ICS}
Expand All @@ -74,7 +74,7 @@ elif [ "${ICS_OR_LBCS}" = "LBCS" ]; then
last_time=$((TIME_OFFSET_HRS + FCST_LEN_HRS))
fcst_hrs="${first_time} ${last_time} ${LBC_SPEC_INTVL_HRS}"
file_names=${EXTRN_MDL_FILES_LBCS[@]}
if [ ${EXTRN_MDL_NAME} = FV3GFS ] ; then
if [ ${EXTRN_MDL_NAME} = FV3GFS ] || [ "${EXTRN_MDL_NAME}" == "GDAS" ] ; then
file_type=$FV3GFS_FILE_FMT_LBCS
fi
input_file_path=${EXTRN_MDL_SOURCE_BASEDIR_LBCS:-$EXTRN_MDL_SYSBASEDIR_LBCS}
Expand All @@ -93,6 +93,10 @@ hh=${yyyymmddhh:8:2}
# Set to use the pre-defined data paths in the machine file (ush/machine/).
PDYext=${yyyymmdd}
cycext=${hh}

# Set an empty members directory
mem_dir=""

#
#-----------------------------------------------------------------------
#
Expand Down Expand Up @@ -136,6 +140,13 @@ if [ $SYMLINK_FIX_FILES = "TRUE" ]; then
additional_flags="$additional_flags \
--symlink"
fi

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic doesn't appear to work correctly for ensemble cases. When I ran a test ensemble case (community_ensemble_008mems), the script fails with the following error:

Traceback (most recent call last):
  File "/mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/ufs-srweather-app/ush/retrieve_data.py", line 1020, in <module>
    main(sys.argv[1:])
  File "/mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/ufs-srweather-app/ush/retrieve_data.py", line 808, in main
    unavailable = get_requested_files(
  File "/mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/ufs-srweather-app/ush/retrieve_data.py", line 366, in get_requested_files
    target_path = fill_template(cla.output_path, cla.cycle_date, mem=mem)
  File "/mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/ufs-srweather-app/ush/retrieve_data.py", line 226, in fill_template
    return template_str.format(**format_values)
ValueError: Unknown format code 'd' for object of type 'str'
+ print_err_msg_exit 'Call to retrieve_data.py failed with a non-zero exit status.

The command was:

python3 -u /mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/ufs-srweather-app/ush/retrieve_data.py   --debug   --anl_or_fcst anl   --config /mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/ufs-srweather-app/parm/data_locations.yml   --cycle_date 2019070100   --data_stores disk disk   --external_model FV3GFS   --fcst_hrs 0   --ics_or_lbcs ICS   --output_path /mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/expt_dirs/community_ensemble_008mems/2019070100/FV3GFS/for_ICS/mem{mem:03d}   --summary_file extrn_mdl_var_defns.sh      --file_type nemsio   --input_file_path /mnt/lfs4/BMC/wrfruc/UFS_SRW_App/develop/input_model_data/FV3GFS/nemsio/2019070100   --symlink   --members 1 8
'
FATAL ERROR:
ERROR:
  From script:  "exregional_get_extrn_mdl_files.sh"
  Full path to script:  "/mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/ufs-srweather-app/scripts/exregional_get_extrn_mdl_files.sh"
Call to retrieve_data.py failed with a non-zero exit status.

Full experiment on Jet here: /mnt/lfs4/HFIP/dtc-hurr/kavulich/workdir/PR_526/expt_dirs/community_ensemble_008mems

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Mike, no worries! I updated my fork branch to the latest develop branch. I also found a bug in the code that was causing this error (line 347 needed members to be defined as cla.members. It should run through now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it appears to be working now!

if [ $DO_ENSEMBLE == "TRUE" ] ; then
mem_dir="/mem{mem:03d}"
member_list=(1 ${NUM_ENS_MEMBERS})
additional_flags="$additional_flags \
--members ${member_list[@]}"
fi
#
#-----------------------------------------------------------------------
#
Expand All @@ -158,7 +169,7 @@ python3 -u ${USHdir}/retrieve_data.py \
--external_model ${EXTRN_MDL_NAME} \
--fcst_hrs ${fcst_hrs[@]} \
--ics_or_lbcs ${ICS_OR_LBCS} \
--output_path ${EXTRN_MDL_STAGING_DIR} \
--output_path ${EXTRN_MDL_STAGING_DIR}${mem_dir} \
--summary_file ${EXTRN_DEFNS} \
$additional_flags"

Expand All @@ -171,6 +182,79 @@ ${cmd}
#
#-----------------------------------------------------------------------
#
# Merge GEFS files
#
#-----------------------------------------------------------------------
#
if [ "${EXTRN_MDL_NAME}" = "GEFS" ]; then

# This block of code sets the forecast hour range based on ICS/LBCS
if [ "${ICS_OR_LBCS}" = "LBCS" ]; then
fcst_hrs_tmp=( $fcst_hrs )
all_fcst_hrs_array=( $(seq ${fcst_hrs_tmp[0]} ${fcst_hrs_tmp[2]} ${fcst_hrs_tmp[1]}) )
else
all_fcst_hrs_array=( ${fcst_hrs} )
fi

# Loop through ensemble member numbers and forecast hours
for num in $(seq -f "%02g" ${NUM_ENS_MEMBERS}); do
sorted_fn=( )
for fcst_hr in "${all_fcst_hrs_array[@]}"; do
# Read in filenames from $EXTRN_MDL_FNS and sort them
base_path="${EXTRN_MDL_STAGING_DIR}/mem`printf %03d $num`"
filenames_array=`awk -F= '/EXTRN_MDL_FNS/{print $2}' $base_path/${EXTRN_DEFNS}`
for filename in ${filenames_array[@]}; do
IFS='.' read -ra split_fn <<< "$filename"
if [ `echo -n $filename | tail -c 2` == `printf %02d $fcst_hr` ] && [ "${split_fn[1]}" == "t${hh}z" ] ; then
if [ "${split_fn[2]}" == 'pgrb2a' ] ; then
sorted_fn+=( "$filename" )
elif [ "${split_fn[2]}" == 'pgrb2b' ] ; then
sorted_fn+=( "$filename" )
elif [ "${split_fn[2]}" == "pgrb2af`printf %02d $fcst_hr`" ] ; then
sorted_fn+=( "$filename" )
elif [ "${split_fn[2]}" == "pgrb2bf`printf %02d $fcst_hr`" ] ; then
sorted_fn+=( "$filename" )
elif [ "${split_fn[2]}" == "pgrb2af`printf %03d $fcst_hr`" ] ; then
sorted_fn+=( "$filename" )
elif [ "${split_fn[2]}" == "pgrb2bf`printf %03d $fcst_hr`" ] ; then
sorted_fn+=( "$filename" )
fi
fi
done

# Define filename lists used to check if files exist
fn_list_1=( ${sorted_fn[0]} ${sorted_fn[1]}
"gep$num.t${hh}z.pgrb2.0p50.f`printf %03d $fcst_hr`" )
fn_list_2=( ${sorted_fn[2]} ${sorted_fn[3]}
"gep$num.t${hh}z.pgrb2`printf %02d $fcst_hr`" )
fn_list_3=( ${sorted_fn[4]} ${sorted_fn[5]}
"gep$num.t${hh}z.pgrb2`printf %03d $fcst_hr`" )
echo ${fn_list_1[@]}
fn_lists=( "fn_list_1" "fn_list_2" "fn_list_3" )

# Look for filenames, if they exist, merge files together
printf "Looking for files in $base_path\n"
for fn in "${fn_lists[@]}"; do
fn_str="$fn[@]"
fn_array=( "${!fn_str}" )
if [ -f "$base_path/${fn_array[0]}" ] && [ -f "$base_path/${fn_array[1]}" ]; then
printf "Found files: ${fn_array[0]} and ${fn_array[1]} \nCreating new file: ${fn_array[2]}\n"
cat $base_path/${fn_array[0]} $base_path/${fn_array[1]} > $base_path/${fn_array[2]}
merged_fn+=( "${fn_array[2]}" )
fi
done
done
# If merge files exist, update the extrn_defn file
merged_fn_str="( ${merged_fn[@]} )"
printf "Merged files are: ${merged_fn_str} \nUpdating ${EXTRN_DEFNS}\n\n"
echo "$(awk -F= -v val="${merged_fn_str}" '/EXTRN_MDL_FNS/ {$2=val} {print}' OFS== $base_path/${EXTRN_DEFNS})" > $base_path/${EXTRN_DEFNS}
merged_fn=()
mod_fn_list=()
done
fi
#
#-----------------------------------------------------------------------
#
# Restore the shell options saved at the beginning of this script/function.
#
#-----------------------------------------------------------------------
Expand Down
35 changes: 33 additions & 2 deletions scripts/exregional_make_ics.sh
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,10 @@ fi
#-----------------------------------------------------------------------
#
if [ $RUN_ENVIR = "nco" ]; then
extrn_mdl_staging_dir="${COMINext}"
extrn_mdl_staging_dir="${COMINext}${SLASH_ENSMEM_SUBDIR}"
extrn_mdl_var_defns_fp="${extrn_mdl_staging_dir}/${NET}.${cycle}.${EXTRN_MDL_NAME_ICS}.ICS.${EXTRN_MDL_VAR_DEFNS_FN}.sh"
else
extrn_mdl_staging_dir="${COMIN}/${EXTRN_MDL_NAME_ICS}/for_ICS"
extrn_mdl_staging_dir="${COMIN}/${EXTRN_MDL_NAME_ICS}/for_ICS${SLASH_ENSMEM_SUBDIR}"
extrn_mdl_var_defns_fp="${extrn_mdl_staging_dir}/${EXTRN_MDL_VAR_DEFNS_FN}.sh"
fi
. ${extrn_mdl_var_defns_fp}
Expand Down Expand Up @@ -131,6 +131,8 @@ case "${CCPP_PHYS_SUITE}" in
varmap_file="GSDphys_var_map.txt"
elif [ "${EXTRN_MDL_NAME_ICS}" = "NAM" ] || \
[ "${EXTRN_MDL_NAME_ICS}" = "FV3GFS" ] || \
[ "${EXTRN_MDL_NAME_ICS}" = "GEFS" ] || \
[ "${EXTRN_MDL_NAME_ICS}" = "GDAS" ] || \
[ "${EXTRN_MDL_NAME_ICS}" = "GSMGFS" ]; then
varmap_file="GFSphys_var_map.txt"
fi
Expand Down Expand Up @@ -389,6 +391,35 @@ case "${EXTRN_MDL_NAME_ICS}" in
tg3_from_soil=False
;;

"GDAS")
tracers_input="[\"spfh\",\"clwmr\",\"o3mr\",\"icmr\",\"rwmr\",\"snmr\",\"grle\"]"
tracers="[\"sphum\",\"liq_wat\",\"o3mr\",\"ice_wat\",\"rainwat\",\"snowwat\",\"graupel\"]"
external_model="GFS"
input_type="gaussian_netcdf"
convert_nst=False
fn_atm="${EXTRN_MDL_FNS[0]}"
fn_sfc="${EXTRN_MDL_FNS[1]}"
vgtyp_from_climo=True
sotyp_from_climo=True
vgfrc_from_climo=True
minmax_vgfrc_from_climo=True
lai_from_climo=True
tg3_from_soil=True
;;

"GEFS")
external_model="GFS"
fn_grib2="${EXTRN_MDL_FNS[0]}"
input_type="grib2"
convert_nst=False
vgtyp_from_climo=True
sotyp_from_climo=True
vgfrc_from_climo=True
minmax_vgfrc_from_climo=True
lai_from_climo=True
tg3_from_soil=False
;;

"HRRR")
external_model="HRRR"
fn_grib2="${EXTRN_MDL_FNS[0]}"
Expand Down
26 changes: 24 additions & 2 deletions scripts/exregional_make_lbcs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,10 @@ fi
#-----------------------------------------------------------------------
#
if [ $RUN_ENVIR = "nco" ]; then
extrn_mdl_staging_dir="${COMINext}"
extrn_mdl_staging_dir="${COMINext}${SLASH_ENSMEM_SUBDIR}"
extrn_mdl_var_defns_fp="${extrn_mdl_staging_dir}/${NET}.${cycle}.${EXTRN_MDL_NAME_LBCS}.LBCS.${EXTRN_MDL_VAR_DEFNS_FN}.sh"
else
extrn_mdl_staging_dir="${COMIN}/${EXTRN_MDL_NAME_LBCS}/for_LBCS"
extrn_mdl_staging_dir="${COMIN}/${EXTRN_MDL_NAME_LBCS}/for_LBCS${SLASH_ENSMEM_SUBDIR}"
extrn_mdl_var_defns_fp="${extrn_mdl_staging_dir}/${EXTRN_MDL_VAR_DEFNS_FN}.sh"
fi
. ${extrn_mdl_var_defns_fp}
Expand Down Expand Up @@ -129,6 +129,8 @@ case "${CCPP_PHYS_SUITE}" in
varmap_file="GSDphys_var_map.txt"
elif [ "${EXTRN_MDL_NAME_LBCS}" = "NAM" ] || \
[ "${EXTRN_MDL_NAME_LBCS}" = "FV3GFS" ] || \
[ "${EXTRN_MDL_NAME_LBCS}" = "GEFS" ] || \
[ "${EXTRN_MDL_NAME_LBCS}" = "GDAS" ] || \
[ "${EXTRN_MDL_NAME_LBCS}" = "GSMGFS" ]; then
varmap_file="GFSphys_var_map.txt"
fi
Expand Down Expand Up @@ -291,6 +293,20 @@ case "${EXTRN_MDL_NAME_LBCS}" in
fi
;;

"GDAS")
tracers_input="[\"spfh\",\"clwmr\",\"o3mr\",\"icmr\",\"rwmr\",\"snmr\",\"grle\"]"
tracers="[\"sphum\",\"liq_wat\",\"o3mr\",\"ice_wat\",\"rainwat\",\"snowwat\",\"graupel\"]"
external_model="GFS"
input_type="gaussian_netcdf"
fn_atm="${EXTRN_MDL_FNS[0]}"
;;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we may need additional logic to allow LBCS to be nemsio format given the namelist entries in RRFS_dev1.

That could be a part of a future PR, though.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only see a 'gaussian_netcdf' as the input type for GDAS in the RRFS_dev1 file. Maybe I'm not looking at the right location??

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but there are 2 entries present at the file at that link for fn_atm_nemsio and fn_sfc_nemsio to support nemsio input files from GDAS.

Those entries are also present in the case statement below, too.

Copy link
Collaborator Author

@EdwardSnyder-NOAA EdwardSnyder-NOAA Feb 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I gotcha. In the develop branch, fn_atm_nemsio was replaced with fn_atm variable that accepts both nemsio and netcdf format. But to handle for a nemsio case, I'd have to modify the code block to look something like this:

"GDAS")
    if [ "${FV3GFS_FILE_FMT_LBCS}" = "nemsio" ]; then
        fn_atm="${EXTRN_MDL_FNS[$i]}"
    elif [ "${FV3GFS_FILE_FMT_LBCS}" = "netcdf" ]; then
        fn_atm="${EXTRN_MDL_FNS[$i]}"

Is this what you are looking for? My apologies for not fully understanding this request.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, sorry, I should have been more explicit. In the RRFS_dev1 version, under GDAS on line 337, there are these entries:

"GDASENKF")
  tracers_input="[\"spfh\",\"clwmr\",\"o3mr\",\"icmr\",\"rwmr\",\"snmr\",\"grle\"]"
  tracers="[\"sphum\",\"liq_wat\",\"o3mr\",\"ice_wat\",\"rainwat\",\"snowwat\",\"graupel\"]"
  external_model="GFS"
  input_type="gaussian_netcdf"
  fn_atm_nemsio="${EXTRN_MDL_FNS[0]}"
  fn_sfc_nemsio="${EXTRN_MDL_FNS[1]}"
  ;;

In your addition of GDAS here, you've added a single fn_atm entry where the RRFS version has two lines fn_atm_nemsio and fn_sfc_nemsio. This means we can't use nemsio files for GDAS, we'd have to use netcdf. We need an option to use nemsio.


"GEFS")
external_model="GFS"
fn_grib2="${EXTRN_MDL_FNS[0]}"
input_type="grib2"
;;

"RAP")
external_model="RAP"
input_type="grib2"
Expand Down Expand Up @@ -364,6 +380,12 @@ for (( i=0; i<${num_fhrs}; i++ )); do
fn_atm="${EXTRN_MDL_FNS[$i]}"
fi
;;
"GDAS")
fn_atm="${EXTRN_MDL_FNS[0][$i]}"
;;
"GEFS")
fn_grib2="${EXTRN_MDL_FNS[$i]}"
;;
"RAP")
fn_grib2="${EXTRN_MDL_FNS[$i]}"
;;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
metadata:
description: |-
This test checks the capability of the workflow to retrieve from NOAA
AWS grib2-formatted output files generated by GEFS.
user:
RUN_ENVIR: community
platform:
EXTRN_MDL_DATA_STORES: aws
workflow:
CCPP_PHYS_SUITE: FV3_HRRR
PREDEF_GRID_NAME: RRFS_CONUS_3km
DATE_FIRST_CYCL: '2022040400'
DATE_LAST_CYCL: '2022040400'
FCST_LEN_HRS: 6
PREEXISTING_DIR_METHOD: rename
task_get_extrn_ics:
EXTRN_MDL_NAME_ICS: GEFS
EXTRN_MDL_ICS_OFFSET_HRS: 6
FV3GFS_FILE_FMT_ICS: grib2
task_get_extrn_lbcs:
EXTRN_MDL_NAME_LBCS: GEFS
LBC_SPEC_INTVL_HRS: 6
EXTRN_MDL_LBCS_OFFSET_HRS: 0
FV3GFS_FILE_FMT_LBCS: grib2
global:
DO_ENSEMBLE: true
NUM_ENS_MEMBERS: 2
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
metadata:
description: |-
This test checks the capability of the workflow to retrieve from NOAA
HPSS netcdf-formatted output files generated by GDAS.
user:
RUN_ENVIR: community
platform:
EXTRN_MDL_DATA_STORES: hpss
workflow:
CCPP_PHYS_SUITE: FV3_HRRR
PREDEF_GRID_NAME: RRFS_CONUS_3km
DATE_FIRST_CYCL: '2022040400'
DATE_LAST_CYCL: '2022040400'
FCST_LEN_HRS: 6
PREEXISTING_DIR_METHOD: rename
task_get_extrn_ics:
EXTRN_MDL_NAME_ICS: GDAS
EXTRN_MDL_ICS_OFFSET_HRS: 6
FV3GFS_FILE_FMT_ICS: netcdf
task_get_extrn_lbcs:
EXTRN_MDL_NAME_LBCS: GDAS
LBC_SPEC_INTVL_HRS: 6
EXTRN_MDL_LBCS_OFFSET_HRS: 0
FV3GFS_FILE_FMT_LBCS: netcdf
global:
DO_ENSEMBLE: true
NUM_ENS_MEMBERS: 2
Loading