Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable ESMF threading in the ufs-weather-model forecast #1371

Merged
merged 15 commits into from
Mar 11, 2023
Merged

Enable ESMF threading in the ufs-weather-model forecast #1371

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0453d5d
add threads specs
aerorahul Mar 3, 2023
d904cc9
update nems.configure.*.IN
aerorahul Mar 3, 2023
0ade235
remove references to NTHREADS_FV3 as they no longer are needed
aerorahul Mar 4, 2023
5c1e48b
update for CDUMP=gdas|gfs and WCOSS2.env
aerorahul Mar 4, 2023
9ba87a8
fix some shellcheck errors
aerorahul Mar 4, 2023
67b2760
Apply suggestions from code review
aerorahul Mar 6, 2023
435879b
take suggestions from review. echo details of calculating pets
aerorahul Mar 6, 2023
27bc311
commit WCOSS2 update
aerorahul Mar 6, 2023
6c5ef9f
resolve srun issue w/ help from @junwang-noaa
aerorahul Mar 7, 2023
e6865cf
bugfix on wcoss2 launcher for fcst/efcs
aerorahul Mar 8, 2023
c796911
use full nodes on all machines for the forecast task. load ufs-weathe…
aerorahul Mar 8, 2023
2a2b4a5
reduce long forecast length to 120 hours for it to fit in the allocat…
aerorahul Mar 9, 2023
a2874b2
resolve TODO
aerorahul Mar 9, 2023
3690332
Merge branch 'develop' into feature/esmf
aerorahul Mar 10, 2023
7b483c2
detect machine and load cray-pals on wcoss2 in the fcst and efcs jobs
aerorahul Mar 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 2 additions & 36 deletions env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,43 +188,9 @@ elif [[ "${step}" = "eupd" ]]; then
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "fcst" ]]; then
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_efcs}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
export APRUN_UFS="${launcher}"

elif [[ "${step}" = "post" ]]; then

Expand Down
40 changes: 3 additions & 37 deletions env/JET.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/ksh -x
#! /usr/bin/env bash

if [[ $# -ne 1 ]]; then

Expand Down Expand Up @@ -116,43 +116,9 @@ elif [[ "${step}" = "eupd" ]]; then
[[ ${NTHREADS_ENKF} -gt ${nth_max} ]] && export NTHREADS_ENKF=${nth_max}
export APRUN_ENKF="${launcher} ${npe_enkf:-${npe_eupd:-${PBS_NP}}}"

elif [[ "${step}" = "fcst" ]]; then
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} ${npe_remap:-${npe_fcst:-${PBS_NP}}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} ${npe_fv3:-${npe_efcs:-${PBS_NP}}}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} ${LEVS}"
export APRUN_UFS="${launcher}"

elif [[ "${step}" = "post" ]]; then

Expand Down
40 changes: 3 additions & 37 deletions env/ORION.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,43 +186,9 @@ elif [[ "${step}" = "eupd" ]]; then
[[ ${NTHREADS_ENKF} -gt ${nth_max} ]] && export NTHREADS_ENKF=${nth_max}
export APRUN_ENKF="${launcher} -n ${npe_enkf:-${npe_eupd}}"

elif [[ "${step}" = "fcst" ]]; then
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_efcs}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
export APRUN_UFS="${launcher}"

elif [[ "${step}" = "post" ]]; then

Expand Down Expand Up @@ -306,7 +272,7 @@ elif [[ "${step}" = "gempak" ]]; then
npe_gempak=${npe_gempak_gfs}
npe_node_gempak=${npe_node_gempak_gfs}
fi

nth_max=$((npe_node_max / npe_node_gempak))

export NTHREADS_GEMPAK=${nth_gempak:-1}
Expand Down
38 changes: 2 additions & 36 deletions env/S4.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,43 +172,9 @@ elif [[ "${step}" = "eupd" ]]; then
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "fcst" ]]; then
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_efcs}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
export APRUN_UFS="${launcher}"

elif [[ "${step}" = "post" ]]; then

Expand Down
67 changes: 23 additions & 44 deletions env/WCOSS2.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,53 +170,32 @@ elif [[ "${step}" = "eupd" ]]; then
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="${launcher} -np \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "fcst" ]]; then

export OMP_PLACES=cores
export OMP_STACKSIZE=2048M
export FI_OFI_RXM_RX_SIZE=40000
export FI_OFI_RXM_TX_SIZE=40000

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

if [[ "${CDUMP}" =~ "gfs" ]]; then
nprocs="npe_${step}_gfs"
ppn="npe_node_${step}_gfs" || ppn="npe_node_${step}"
else
nprocs="npe_${step}"
ppn="npe_node_${step}"
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst} -ppn ${npe_node_fcst} --cpu-bind depth --depth ${NTHREADS_FV3}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

export MPICH_MPIIO_HINTS="*:romio_cb_write=disable"
export FI_OFI_RXM_SAR_LIMIT=3145728
(( nnodes = (${!nprocs}+${!ppn}-1)/${!ppn} ))
(( ntasks = ${nnodes}*${!ppn} ))
export APRUN_UFS="${launcher} -n ${ntasks} -ppn ${!ppn} --cpu-bind depth --depth 1"
unset nprocs ppn nnodes ntasks

# TODO: Why are fcst and efcs so different on WCOSS2?
# TODO: Compare these with the ufs-weather-model regression test job card at:
# https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/fv3_conf/fv3_qsub.IN_wcoss2
export FI_OFI_RXM_RX_SIZE=40000
export FI_OFI_RXM_TX_SIZE=40000

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fv3:-${npe_efcs}} -ppn ${npe_node_efcs} --cpu-bind depth --depth ${NTHREADS_FV3}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
if [[ "${step}" = "fcst" ]]; then
export OMP_PLACES=cores
export OMP_STACKSIZE=2048M
elif [[ "${step}" = "efcs" ]]; then
export MPICH_MPIIO_HINTS="*:romio_cb_write=disable"
export FI_OFI_RXM_SAR_LIMIT=3145728
fi

elif [[ "${step}" = "post" ]]; then

Expand Down
7 changes: 3 additions & 4 deletions parm/config/config.defaults.s2sw
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ min_seaice="1.0e-6"
use_cice_alb=".true."

# config.ufs # TODO: This is hard-wired for P8 and needs to be refactored. For now, use case C384
# TODO: Q. for @jessicameixner-noaa: can we make these defaults in config.ufs for C384?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, this is mostly for the commented out:
#layout_x_gfs=24
#layout_y_gfs=16
#WRTTASK_PER_GROUP_GFS=86
However, with the move to 32bit this might be a little bit less. Plus we can probably show people how to update this in config.ufs instead, so yes?

Copy link
Contributor

@junwang-noaa junwang-noaa Mar 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please follow: layout_x < layout_y. Also I am not sure how the WRTTASK_PER_GROUP_GFS=86 is coming out, it may be related to taking whole nodes, but it is really machine dependent, the number may not be good for wcoss2.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JessicaMeixner-NOAA
The uncommented ones are hidden.

    DELTIM=300
    layout_x_gfs=8
    layout_y_gfs=8
    WRITE_GROUP_GFS=1
    WRTTASK_PER_GROUP_GFS=24

I am inclined to remove these as they are not wildly different than the ones in config.ufs and the tests ran w/ the ones in config.ufs.

case "${CASE}" in
"C384")
DELTIM=300
Expand All @@ -26,11 +27,9 @@ case "${CASE}" in
#layout_x_gfs=24
#layout_y_gfs=16
#WRTTASK_PER_GROUP_GFS=86
WRTIOBUF="32M" # TODO: This value is for P8 w/ C384. Why not update/set this as default in config.fv3 C384?
MEDPETS=300 # TODO: Use 300 instead of ATMPETS = layout_x * layout_y * 6
;;
"C768")
MEDPETS=300
*) # All other ${CASE}
echo "config.defaults.s2sw: Using default settings for CASE=${CASE}"
;;
esac

Expand Down
Loading