Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable ESMF threading in the ufs-weather-model forecast #1371

Merged
merged 15 commits into from
Mar 11, 2023
Merged

Enable ESMF threading in the ufs-weather-model forecast #1371

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0453d5d
add threads specs
aerorahul Mar 3, 2023
d904cc9
update nems.configure.*.IN
aerorahul Mar 3, 2023
0ade235
remove references to NTHREADS_FV3 as they no longer are needed
aerorahul Mar 4, 2023
5c1e48b
update for CDUMP=gdas|gfs and WCOSS2.env
aerorahul Mar 4, 2023
9ba87a8
fix some shellcheck errors
aerorahul Mar 4, 2023
67b2760
Apply suggestions from code review
aerorahul Mar 6, 2023
435879b
take suggestions from review. echo details of calculating pets
aerorahul Mar 6, 2023
27bc311
commit WCOSS2 update
aerorahul Mar 6, 2023
6c5ef9f
resolve srun issue w/ help from @junwang-noaa
aerorahul Mar 7, 2023
e6865cf
bugfix on wcoss2 launcher for fcst/efcs
aerorahul Mar 8, 2023
c796911
use full nodes on all machines for the forecast task. load ufs-weathe…
aerorahul Mar 8, 2023
2a2b4a5
reduce long forecast length to 120 hours for it to fit in the allocat…
aerorahul Mar 9, 2023
a2874b2
resolve TODO
aerorahul Mar 9, 2023
3690332
Merge branch 'develop' into feature/esmf
aerorahul Mar 10, 2023
7b483c2
detect machine and load cray-pals on wcoss2 in the fcst and efcs jobs
aerorahul Mar 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 13 additions & 36 deletions env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,43 +188,20 @@ elif [[ "${step}" = "eupd" ]]; then
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "fcst" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

if [[ "${CDUMP}" =~ "gfs" ]]; then
nprocs="npe_${step}_gfs"
ppn="npe_node_${step}_gfs" || ppn="npe_node_${step}"
else
nprocs="npe_${step}"
ppn="npe_node_${step}"
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_efcs}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
(( nnodes = (${!nprocs}+${!ppn}-1)/${!ppn} ))
(( ntasks = nnodes*${!ppn} ))
# With ESMF threading, the model wants to use the full node
export APRUN_UFS="${launcher} -n ${ntasks}"
unset nprocs ppn nnodes ntasks

elif [[ "${step}" = "post" ]]; then

Expand Down
49 changes: 13 additions & 36 deletions env/JET.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/ksh -x
#! /usr/bin/env bash

if [[ $# -ne 1 ]]; then

Expand Down Expand Up @@ -116,43 +116,20 @@ elif [[ "${step}" = "eupd" ]]; then
[[ ${NTHREADS_ENKF} -gt ${nth_max} ]] && export NTHREADS_ENKF=${nth_max}
export APRUN_ENKF="${launcher} ${npe_enkf:-${npe_eupd:-${PBS_NP}}}"

elif [[ "${step}" = "fcst" ]]; then
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
if [[ "${CDUMP}" =~ "gfs" ]]; then
nprocs="npe_${step}_gfs"
ppn="npe_node_${step}_gfs" || ppn="npe_node_${step}"
else
nprocs="npe_${step}"
ppn="npe_node_${step}"
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} ${npe_remap:-${npe_fcst:-${PBS_NP}}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} ${npe_fv3:-${npe_efcs:-${PBS_NP}}}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} ${LEVS}"
(( nnodes = (${!nprocs}+${!ppn}-1)/${!ppn} ))
(( ntasks = nnodes*${!ppn} ))
# With ESMF threading, the model wants to use the full node
export APRUN_UFS="${launcher} -n ${ntasks}"
unset nprocs ppn nnodes ntasks

elif [[ "${step}" = "post" ]]; then

Expand Down
52 changes: 15 additions & 37 deletions env/ORION.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,43 +186,21 @@ elif [[ "${step}" = "eupd" ]]; then
[[ ${NTHREADS_ENKF} -gt ${nth_max} ]] && export NTHREADS_ENKF=${nth_max}
export APRUN_ENKF="${launcher} -n ${npe_enkf:-${npe_eupd}}"

elif [[ "${step}" = "fcst" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

export OMP_STACKSIZE=512M
if [[ "${CDUMP}" =~ "gfs" ]]; then
nprocs="npe_${step}_gfs"
ppn="npe_node_${step}_gfs" || ppn="npe_node_${step}"
else
nprocs="npe_${step}"
ppn="npe_node_${step}"
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_efcs}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
(( nnodes = (${!nprocs}+${!ppn}-1)/${!ppn} ))
(( ntasks = nnodes*${!ppn} ))
# With ESMF threading, the model wants to use the full node
export APRUN_UFS="${launcher} -n ${ntasks}"
unset nprocs ppn nnodes ntasks

elif [[ "${step}" = "post" ]]; then

Expand Down Expand Up @@ -306,7 +284,7 @@ elif [[ "${step}" = "gempak" ]]; then
npe_gempak=${npe_gempak_gfs}
npe_node_gempak=${npe_node_gempak_gfs}
fi

nth_max=$((npe_node_max / npe_node_gempak))

export NTHREADS_GEMPAK=${nth_gempak:-1}
Expand Down
49 changes: 13 additions & 36 deletions env/S4.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,43 +172,20 @@ elif [[ "${step}" = "eupd" ]]; then
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "fcst" ]]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

if [[ "${CDUMP}" =~ "gfs" ]]; then
nprocs="npe_${step}_gfs"
ppn="npe_node_${step}_gfs" || ppn="npe_node_${step}"
else
nprocs="npe_${step}"
ppn="npe_node_${step}"
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_efcs}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
(( nnodes = (${!nprocs}+${!ppn}-1)/${!ppn} ))
(( ntasks = nnodes*${!ppn} ))
# With ESMF threading, the model wants to use the full node
export APRUN_UFS="${launcher} -n ${ntasks}"
unset nprocs ppn nnodes ntasks

elif [[ "${step}" = "post" ]]; then

Expand Down
68 changes: 24 additions & 44 deletions env/WCOSS2.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,53 +170,33 @@ elif [[ "${step}" = "eupd" ]]; then
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="${launcher} -np \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "fcst" ]]; then

export OMP_PLACES=cores
export OMP_STACKSIZE=2048M
export FI_OFI_RXM_RX_SIZE=40000
export FI_OFI_RXM_TX_SIZE=40000

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
if [[ ${CDUMP} == "gfs" ]]; then
npe_fcst=${npe_fcst_gfs}
npe_node_fcst=${npe_node_fcst_gfs}
nth_fv3=${nth_fv3_gfs}
elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

if [[ "${CDUMP}" =~ "gfs" ]]; then
nprocs="npe_${step}_gfs"
ppn="npe_node_${step}_gfs" || ppn="npe_node_${step}"
else
nprocs="npe_${step}"
ppn="npe_node_${step}"
fi

nth_max=$((npe_node_max / npe_node_fcst))

export NTHREADS_FV3=${nth_fv3:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fcst} -ppn ${npe_node_fcst} --cpu-bind depth --depth ${NTHREADS_FV3}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"

export NTHREADS_REMAP=${nth_remap:-2}
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
export APRUN_REMAP="${launcher} -n ${npe_remap:-${npe_fcst}}"
export I_MPI_DAPL_UD="enable"

elif [[ "${step}" = "efcs" ]]; then

export MPICH_MPIIO_HINTS="*:romio_cb_write=disable"
export FI_OFI_RXM_SAR_LIMIT=3145728
(( nnodes = (${!nprocs}+${!ppn}-1)/${!ppn} ))
(( ntasks = nnodes*${!ppn} ))
# With ESMF threading, the model wants to use the full node
export APRUN_UFS="${launcher} -n ${ntasks} -ppn ${!ppn} --cpu-bind depth --depth 1"
unset nprocs ppn nnodes ntasks

# TODO: Why are fcst and efcs so different on WCOSS2?
# TODO: Compare these with the ufs-weather-model regression test job card at:
# https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/fv3_conf/fv3_qsub.IN_wcoss2
export FI_OFI_RXM_RX_SIZE=40000
export FI_OFI_RXM_TX_SIZE=40000

nth_max=$((npe_node_max / npe_node_efcs))

export NTHREADS_FV3=${nth_efcs:-${nth_max}}
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
export cores_per_node=${npe_node_max}
export APRUN_FV3="${launcher} -n ${npe_fv3:-${npe_efcs}} -ppn ${npe_node_efcs} --cpu-bind depth --depth ${NTHREADS_FV3}"

export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
export APRUN_REGRID_NEMSIO="${launcher} -n ${LEVS}"
if [[ "${step}" = "fcst" ]]; then
export OMP_PLACES=cores
export OMP_STACKSIZE=2048M
elif [[ "${step}" = "efcs" ]]; then
export MPICH_MPIIO_HINTS="*:romio_cb_write=disable"
export FI_OFI_RXM_SAR_LIMIT=3145728
fi

elif [[ "${step}" = "post" ]]; then

Expand Down
14 changes: 11 additions & 3 deletions jobs/rocoto/efcs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,17 @@ source "${HOMEgfs}/ush/preamble.sh"

###############################################################
# Source FV3GFS workflow modules
. ${HOMEgfs}/ush/load_fv3gfs_modules.sh
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
#. ${HOMEgfs}/ush/load_fv3gfs_modules.sh
#status=$?
#[[ ${status} -ne 0 ]] && exit ${status}

set +x
module use "${HOMEgfs}/sorc/ufs_model.fd/tests"
module load modules.ufs_model.lua
Comment on lines +14 to +15
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a long-term solution, I think we're going to need to copy these modulefiles to our top level in link_workflow.sh to satisfy NCO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a long term solution, but is needed ATM.
We can no longer delay the job-specific loading of the modules. The JEDI team has its own load_ufsda_modules.sh as a workaround, but IMO, a better more sound solution needs to be developed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be good to give each job/task flexibility to load its own modules. This will allow smooth transition to new libraries to all the workflow components during development stage. Maybe we can have a unified version for opn implementation tags when the code is frozen.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears that just loading the modules from ufs-weather-model modulefile is not enough for wcoss2.
wcoss2 also requires cray-pals that loads the mpiexec.
This module is loaded in the run script of the regression test
https://github.com/ufs-community/ufs-weather-model/blob/495c5fd106ff3facb5b4aedbf94f8bba6c1da091/tests/fv3_conf/fv3_qsub.IN_wcoss2#LL20-L20C22
Pity!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@junwang-noaa Can we load cray-pals in https://github.com/ufs-community/ufs-weather-model/blob/develop/modulefiles/ufs_wcoss2.intel.lua?
It will save us a mountain of trouble till we have a proper solution?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can. This will require model version change, I will submit a PR to see if we can get it in as soon as possible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @junwang-noaa. We will proceed w/ the hack we have for now.
Let us know when there is an update in the model hash.

# Workflow needs utilities from prod_util (setPDY.sh, ndate, etc.)
module load prod_util
module list
set_trace

export job="efcs"
export jobid="${job}.$$"
Expand Down
13 changes: 10 additions & 3 deletions jobs/rocoto/fcst.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,16 @@ source "${HOMEgfs}/ush/preamble.sh"

###############################################################
# Source FV3GFS workflow modules
. ${HOMEgfs}/ush/load_fv3gfs_modules.sh
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
#. ${HOMEgfs}/ush/load_fv3gfs_modules.sh
#status=$?
#[[ ${status} -ne 0 ]] && exit ${status}

set +x
module use "${HOMEgfs}/sorc/ufs_model.fd/tests"
module load modules.ufs_model.lua
module load prod_util
module list
set_trace

export job="fcst"
export jobid="${job}.$$"
Expand Down
7 changes: 3 additions & 4 deletions parm/config/config.defaults.s2sw
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ min_seaice="1.0e-6"
use_cice_alb=".true."

# config.ufs # TODO: This is hard-wired for P8 and needs to be refactored. For now, use case C384
# TODO: Q. for @jessicameixner-noaa: can we make these defaults in config.ufs for C384?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, this is mostly for the commented out:
#layout_x_gfs=24
#layout_y_gfs=16
#WRTTASK_PER_GROUP_GFS=86
However, with the move to 32bit this might be a little bit less. Plus we can probably show people how to update this in config.ufs instead, so yes?

Copy link
Contributor

@junwang-noaa junwang-noaa Mar 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please follow: layout_x < layout_y. Also I am not sure how the WRTTASK_PER_GROUP_GFS=86 is coming out, it may be related to taking whole nodes, but it is really machine dependent, the number may not be good for wcoss2.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JessicaMeixner-NOAA
The uncommented ones are hidden.

    DELTIM=300
    layout_x_gfs=8
    layout_y_gfs=8
    WRITE_GROUP_GFS=1
    WRTTASK_PER_GROUP_GFS=24

I am inclined to remove these as they are not wildly different than the ones in config.ufs and the tests ran w/ the ones in config.ufs.

case "${CASE}" in
"C384")
DELTIM=300
Expand All @@ -26,11 +27,9 @@ case "${CASE}" in
#layout_x_gfs=24
#layout_y_gfs=16
#WRTTASK_PER_GROUP_GFS=86
WRTIOBUF="32M" # TODO: This value is for P8 w/ C384. Why not update/set this as default in config.fv3 C384?
MEDPETS=300 # TODO: Use 300 instead of ATMPETS = layout_x * layout_y * 6
;;
"C768")
MEDPETS=300
*) # All other ${CASE}
echo "config.defaults.s2sw: Using default settings for CASE=${CASE}"
;;
esac

Expand Down
Loading