Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add changes to fix CI tests #6

Closed
wants to merge 3 commits into from
+18 −37
Closed

Add changes to fix CI tests #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
de7b148
Add changes to fix CI tests
MinsukJi-NOAA Jan 21, 2021
1747b9a
Put sudo for docker back in ci.sh
MinsukJi-NOAA Jan 21, 2021
0b5aa35
Modify ccpp_control_run.IN for unit test restart case
MinsukJi-NOAA Jan 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/ci/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
From noaaemc/ubuntu-hpc:v1
From noaaemc/ubuntu-hpc:v1.1

ENV HOME=/home/builder
COPY --chown=builder:builder . $HOME/ufs-weather-model
Expand Down
4 changes: 2 additions & 2 deletions tests/ci/ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ if [ $BUILD = "true" ]; then

elif [ $RUN == "true" ]; then

sudo docker run -d --rm -v DataVolume:/tmp minsukjinoaa/fv3-input-data:develop-20200713
sudo docker run -d -e test_case=${TEST_CASE} -v DataVolume:/home/builder/data/NEMSfv3gfs/develop-20201118 --name my-container ${IMG_NAME}
sudo docker run -d --rm -v DataVolume:/tmp minsukjinoaa/fv3-input-data:input-data-20210115
sudo docker run -d -e test_case=${TEST_CASE} -v DataVolume:/home/builder/data/NEMSfv3gfs/input-data-20210115 --name my-container ${IMG_NAME}

echo 'cache,rss,shmem' >memory_stat
sleep 3
Expand Down
15 changes: 10 additions & 5 deletions tests/fv3_conf/ccpp_control_run.IN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,25 +11,30 @@ fi
echo "inputdir=$inputdir,NPX=$NPX"

UNIT_TEST=${UNIT_TEST:-false}
SUFFIX=${RT_SUFFIX}
if [ $WARM_START = .F. ]; then
cp -r @[INPUTDATA_ROOT]/${inputdir}/INPUT .
mkdir RESTART
else
mkdir INPUT RESTART

rsync -arv ../fv3_ccpp_control${RT_SUFFIX}/RESTART/ INPUT/
if [[ ${UNIT_TEST} == true ]]; then
SUFFIX=${BL_SUFFIX}
fi

rsync -arv ../fv3_ccpp_control${SUFFIX}/RESTART/ INPUT/
rm -f INPUT/fv_core.res.*
rm -f INPUT/fv_srf_wnd.res.*
rm -f INPUT/fv_tracer.res.*
rm -f INPUT/phy_data.*
rm -f INPUT/sfc_data.*
cd INPUT
rename 20161004.000000. '' 20161004.000000.*
cp ../../fv3_ccpp_control${RT_SUFFIX}/INPUT/grid_spec.nc .
cp ../../fv3_ccpp_control${RT_SUFFIX}/INPUT/*_grid.tile*.nc .
cp ../../fv3_ccpp_control${RT_SUFFIX}/INPUT/oro_data.tile*.nc .
cp ../../fv3_ccpp_control${SUFFIX}/INPUT/grid_spec.nc .
cp ../../fv3_ccpp_control${SUFFIX}/INPUT/*_grid.tile*.nc .
cp ../../fv3_ccpp_control${SUFFIX}/INPUT/oro_data.tile*.nc .
if [[ $IAU_INC_FILES != '' ]] || [[ $READ_INCREMENT = '.T.' ]]; then
cp ../../fv3_ccpp_control${RT_SUFFIX}/INPUT/fv3_increment.nc .
cp ../../fv3_ccpp_control${SUFFIX}/INPUT/fv3_increment.nc .
fi
cd ..

Expand Down
3 changes: 3 additions & 0 deletions tests/tests/fv3_ccpp_control
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,9 @@ export LIST_FILES="atmos_4xdaily.tile1.nc \

export_fv3
export FHMAX=48
export NFHOUT=12
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean model will output fields every 12 hours and there is no high frequency output? If CI test requires, we can change to the previous setting of running fv3_control for 24 hours with high frequency output to 12 hours and then every 3 hours to fh=24h. I will update the global restart tests accordingly. Maybe we can do this in the next commit after Dom's ufs-community#386.

Copy link
Author

@MinsukJi-NOAA MinsukJi-NOAA Jan 21, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works for me, Jun. I think that 48 hour runs may be a little too costly (in terms of speed and disk space) for the CI test.

export NFHMAX_HF=-1
export NFHOUT_HF=-1
export RESTART_INTERVAL=24

export FV3_RUN=ccpp_control_run.IN
Expand Down
31 changes: 2 additions & 29 deletions tests/utest
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,6 @@ run_utests() {
case $rc in
std_base)
CREATE_BASELINE=true
RTPWD=$baseline_location
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_std_base
cat <<-EOF > ${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
export UNIT_TEST=true
Expand All @@ -147,8 +145,6 @@ run_utests() {
;;
std)
CREATE_BASELINE=false
RTPWD=$NEW_BASELINE
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_std_base
cat <<-EOF > ${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
export UNIT_TEST=true
Expand All @@ -157,8 +153,6 @@ run_utests() {
;;
thr)
CREATE_BASELINE=false
RTPWD=$NEW_BASELINE
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_std_base
THRD=2
# INPES is sometimes odd, so use JNPES. Make sure JNPES is divisible by THRD
Expand All @@ -178,8 +172,6 @@ run_utests() {
;;
mpi)
CREATE_BASELINE=false
RTPWD=$NEW_BASELINE
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_std_base
JNPES=$(( JNPES/2 ))
WRITE_GROUP=2
Expand All @@ -198,8 +190,6 @@ run_utests() {
;;
dcp)
CREATE_BASELINE=false
RTPWD=$NEW_BASELINE
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_std_base
temp=$INPES
INPES=$JNPES
Expand All @@ -213,8 +203,6 @@ run_utests() {
;;
rst) # this is not going to work for regional model
CREATE_BASELINE=false
RTPWD=$NEW_BASELINE
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_std_base
# Set up date and time of restart files for restart run
RESTART_FILE_PREFIX="${SYEAR}${SMONTH}${SDAY}.$(printf "%02d" $(( SHOUR + FHMAX/2 )))0000"
Expand Down Expand Up @@ -277,8 +265,6 @@ run_utests() {
;;
bit_base)
CREATE_BASELINE=true
RTPWD=$baseline_location
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_bit_base
comp_nm=bit
cat <<-EOF >${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
Expand All @@ -288,8 +274,6 @@ run_utests() {
;;
bit)
CREATE_BASELINE=false
RTPWD=$NEW_BASELINE
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_bit_base
comp_nm=bit
cat <<-EOF >${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
Expand All @@ -299,8 +283,6 @@ run_utests() {
;;
dbg_base)
CREATE_BASELINE=true
RTPWD=$baseline_location
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_dbg_base
comp_nm=dbg
WLCLK=30
Expand All @@ -312,8 +294,6 @@ run_utests() {
;;
dbg)
CREATE_BASELINE=false
RTPWD=$NEW_BASELINE
INPUTDATA_ROOT=$RTPWD
BL_SUFFIX=_dbg_base
comp_nm=dbg
WLCLK=30
Expand All @@ -324,7 +304,6 @@ run_utests() {
EOF
;;
esac
INPUTDATA_ROOT_WW3=${INPUTDATA_ROOT}/WW3_input_data_20201220/

cat <<- EOF > ${RUNDIR_ROOT}/run_test${RT_SUFFIX}.env
export MACHINE_ID=${MACHINE_ID}
Expand Down Expand Up @@ -723,18 +702,12 @@ fi

mkdir -p ${STMP}/${USER}
NEW_BASELINE=${STMP}/${USER}/FV3_UT/UNIT_TEST
if [[ $MACHINE_ID == hera.* || $MACHINE_ID == orion.* ]]; then
baseline_location=$DISKNM/NEMSfv3gfs/develop-20201118/${RT_COMPILER^^}
else
baseline_location=$DISKNM/NEMSfv3gfs/develop-20201118
fi
RTPWD=$baseline_location
INPUTDATA_ROOT=$RTPWD
RTPWD=${NEW_BASELINE}
INPUTDATA_ROOT=${INPUTDATA_ROOT:-$DISKNM/NEMSfv3gfs/input-data-20210115}
INPUTDATA_ROOT_WW3=${INPUTDATA_ROOT}/WW3_input_data_20201220/

rm -rf $NEW_BASELINE
mkdir -p $NEW_BASELINE
ln -s ${RTPWD}/FV3_input_data ${NEW_BASELINE}

rm -f fail_unit_test ${unittest_log}

Expand Down