Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WAM molecular diffusion #120

Merged
merged 40 commits into from
Oct 26, 2021
Merged

Add WAM molecular diffusion #120

merged 40 commits into from
Oct 26, 2021

Conversation

XiaqiongZhou-NOAA
Copy link
Contributor

@XiaqiongZhou-NOAA XiaqiongZhou-NOAA commented Jul 27, 2021

Description

Molecular diffusion developed by Henry Juang is added as a compiling option to replace Rayleigh Damping to configure FV3WAM with the model top reaching several hundreds km.

Fixes # (issue)

How Has This Been Tested?
It was tested on Hera with Intel compiler and will tested on other machines following UFS regression test rules.

Checklist:

Please check all whether they apply or not

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • [X ] I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published in downstream modules

** Dependences:
NCAR/ccpp-physics#712
NOAA-EMC/fv3atm#354
ufs-community/ufs-weather-model#714

akubaryk and others added 25 commits October 2, 2019 16:39
resolve conflicts.
Conflicts:
	driver/fvGFS/atmosphere.F90
	model/dyn_core.F90
	model/fv_arrays.F90
	model/fv_control.F90
	model/fv_dynamics.F90
	model/fv_mapz.F90
	model/nh_utils.F90
	model/sw_core.F90
	model/tp_core.F90
	tools/fv_diagnostics.F90
@junwang-noaa
Copy link
Collaborator

@bensonr @lharris4 Would you please review this PR? Henry will retire after this week, so if you have any comments/suggestions, would you please let him/Kate know. Thanks

@XiaqiongZhou-NOAA
Copy link
Contributor Author

XiaqiongZhou-NOAA commented Oct 6, 2021 via email

@XiaqiongZhou-NOAA
Copy link
Contributor Author

@lharris4 @bensonr
I have merged the latest GFDL dev/emc to my branch and tested in UFS. It works fine.

@lharris4
Copy link
Contributor

lharris4 commented Oct 7, 2021

OK. Glad to see that it works.

@bensonr
Copy link
Contributor

bensonr commented Oct 7, 2021

@XiaqiongZhou-NOAA - thanks for the update. Let us know when the PRs are in for fv3atm and ufs-weather-model

@bensonr
Copy link
Contributor

bensonr commented Oct 7, 2021

@junwang-noaa - can you please review this.

Copy link
Collaborator

@junwang-noaa junwang-noaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the code. I am not sure if I completely understand the updates. But based on my knowledge, I do not see any issue.

@XiaqiongZhou-NOAA
Copy link
Contributor Author

@bensonr The PRs in fv3atm and ufs are planned to merge Tuesday (10/12)

@XiaqiongZhou-NOAA
Copy link
Contributor Author

@bensonr @junwang-noaa
I have some issues to run regression tests on Hera with GNU related to WAM dycore changes. I need fix it.

Copy link
Contributor

@bensonr bensonr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why the changes for the ranges are needed in fv_diagnostics? I ask because it will cause merge issues in the future with what's in the master branch.

@lharris4
Copy link
Contributor

lharris4 commented Oct 14, 2021 via email

@XiaqiongZhou-NOAA
Copy link
Contributor Author

XiaqiongZhou-NOAA commented Oct 14, 2021

@bensonr @lharris4
Lucas is right.
Also I am afraid I have to change the way to assign values to these arrays. There is a memory issue when running compiling jobs on DELL. This revision can fix it (even though I do not understand why).

The error message on DELL is as the following:
/usrx/local/prod/intel/2018UP01/compilers_and_libraries/linux/mpi/bin64/mpiifort: line 750: 31660 Bus error (core dumped) ifort -DENABLE_QUAD_PRECISION -DGFS_PHYS -DGFS_TYPES -DINTERNAL_FILE_NML -DMOIST_CAPPA -DOPENMP -DOVERLOAD_R4 -DOVERLOAD_R8 -DSPMD -DUSE_COND -DUSE_GFSL63 -Duse_WRTCOMP -I/gpfs/dell2/emc/retros/noscrub/Xiaqiong.Zhou/myufs/md5/FV3/atmos_cubed_sphere -I/gpfs/dell2/emc/retros/noscrub/Xiaqiong.Zhou/myufs/md5/FV3/atmos_cubed_sphere/tools -I/gpfs/dell2/ptmp/Xiaqiong.Zhou/FV3_RT/rt_8237/compile_003/build_fv3_003/FV3/ccpp/mod -I/gpfs/dell2/ptmp/Xiaqiong.Zhou/FV3_RT/rt_8237/compile_003/build_fv3_003/FV3/ccpp/framework/src -I/gpfs/dell2/ptmp/Xiaqiong.Zhou/FV3_RT/rt_8237/compile_003/build_fv3_003/FV3/ccpp/physics -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/impi-18.0.1/fms/2021.03/include_r4 -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/impi-18.0.1/netcdf/4.7.4/include -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/w3nco/2.4.1/include_d -I/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/impi-18.0.1/esmf/8_2_0_beta_snapshot_14/mod -I/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/impi-18.0.1/esmf/8_2_0_beta_snapshot_14/include -g '-traceback' '-fpp' '-fno-alias' '-auto' '-safe-cray-ptr' '-ftz' '-assume' 'byterecl' '-nowarn' '-sox' '-align' 'array64byte' '-qno-opt-dynamic-align' '-O2' '-debug' 'minimal' '-fp-model' 'source' '-qoverride-limits' '-qopt-prefetch=3' '-march=core-avx2' -g '-traceback' '-fpp' '-fno-alias' '-auto' '-safe-cray-ptr' '-ftz' '-assume' 'byterecl' '-nowarn' '-sox' '-align' 'array64byte' '-qno-opt-dynamic-align' '-real-size' 32 '-march=core-avx2' '-O2' '-debug' 'minimal' '-fp-model' 'source' '-qoverride-limits' '-qopt-prefetch=3' '-module' 'include/fv3' '-fPIC' '-qopenmp' -c /gpfs/dell2/emc/retros/noscrub/Xiaqiong.Zhou/myufs/md5/FV3/atmos_cubed_sphere/tools/fv_diagnostics.F90 -o 'CMakeFiles/fv3.dir/tools/fv_diagnostics.F90.o' -I/usrx/local/prod/intel/2018UP01/compilers_and_libraries/linux/mpi/intel64/include -I/usrx/local/prod/intel/2018UP01/compilers_and_libraries/linux/mpi/intel64/include
make[2]: *** [FV3/atmos_cubed_sphere/CMakeFiles/fv3.dir/tools/fv_diagnostics.F90.o] Error 135
make[2]: *** Deleting file `FV3/atmos_cubed_sphere/CMakeFiles/fv3.dir/tools/fv_diagnostics.F90.o'
make[1]: *** [FV3/atmos_cubed_sphere/CMakeFiles/fv3.dir/all] Error 2
make: *** [all] Error 2

@bensonr
Copy link
Contributor

bensonr commented Oct 14, 2021 via email

@bensonr
Copy link
Contributor

bensonr commented Oct 14, 2021

@XiaqiongZhou-NOAA - can you check a compile with a different version of gnu (preferably newer) to see if this is a bug in only one version of something that will be persistent. If it doesn't happen on the hera/gnu compiles, how much longer do we need to support the wcoss p3 dells?

@XiaqiongZhou-NOAA
Copy link
Contributor Author

@bensonr This is not a bug for gnu/9.2.0 on Hera. All RTs can be compiled OK there.

@bensonr
Copy link
Contributor

bensonr commented Oct 14, 2021

@XiaqiongZhou-NOAA - I am not a fan of a fix only needed for a specific version of gnu that seems to indicate a bug. But that being said, I'll let @lharris4 make the call on whether he wants to keep this mismatch or have the change brought back into all of the GFDL versions.

@junwang-noaa
Copy link
Collaborator

junwang-noaa commented Oct 14, 2021 via email

@lharris4
Copy link
Contributor

lharris4 commented Oct 14, 2021 via email

@XiaqiongZhou-NOAA
Copy link
Contributor Author

@junwang-noaa
Please advice how to solve this issue. With the original definition with some vrange = (/xxx, yyy/), I got "bus error" when running compiling jobs on DELL_P3.
I do not know if there are other methods to solve instead of using vrange(1) = xxx ; vrange(2) = yyy.

@lharris4
Copy link
Contributor

lharris4 commented Oct 18, 2021 via email

@XiaqiongZhou-NOAA
Copy link
Contributor Author

I will revise the range definition back and keep the original form. Another method will be used to fix the compiling issue on DELL: compiling fv_diagnostics.F90 with -O0 instead of -O2.

@junwang-noaa
Copy link
Collaborator

@bensonr @laurenchilutti The ufs-weather-model RT passed on all the platforms except orion, where the jobs stayed in the queue the whole day and never got submitted. We decided to start committing the code. Would you please merge this PR? Thanks

@bensonr bensonr merged commit 8a3a759 into NOAA-GFDL:dev/emc Oct 26, 2021
@XiaqiongZhou-NOAA XiaqiongZhou-NOAA deleted the md branch March 29, 2023 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants