Feature/post 2d decomp #1077

JesseMeng-NOAA · 2022-03-03T21:22:41Z

The community upp is currently decomposed on latitudes (y-direction) only. As the ufs-weather-model is making 2D decomposition in both latitudes and longitudes (Y and X), one of the upp refactoring tasks is to support ufs-weather-model inline post 2D decomposition functionality.

A ufs-weather-model feature post_2d_decomp is created for this test with inline post turned on in the control_2dwrtdecomp and regional_control_2dwrtdecomp tests,
https://github.com/JesseMeng-NOAA/ufs-weather-model/tree/feature/post_2d_decomp

A fv3atm feature post_2d_decomp is created with modified io drivers to call upp/post_2d_decomp,
https://github.com/JesseMeng-NOAA/fv3atm/tree/feature/post_2d_decomp

The upp/post_2d_decomp branch can be found at
https://github.com/WenMeng-NOAA/UPP/tree/post_2d_decomp

PR Checklist

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.
This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR
An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
are specified below.
Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.
New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Description

To support ufs-weather-model inline post 2D decomposition functionality.

Issue(s) addressed

#1078

Testing

How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)

Dependencies

The upp/post_2d_decomp branch,
https://github.com/WenMeng-NOAA/UPP/tree/post_2d_decomp

…nto HEAD

junwang-noaa · 2022-03-17T18:55:36Z

@JesseMeng-NOAA @WenMeng-NOAA Please let me know when the issues with inline vs standalone post for global and regional test cases are resolved, we can start running RT.

WenMeng-NOAA · 2022-03-17T20:08:00Z

@junwang-noaa From my tests, I saw the grib2 results from RT control & control_2dwrtdecomp, and RT regional_control & regional_control_2dwrtdecomp are not consistent. You might run your own RT testing.

JesseMeng-NOAA · 2022-03-17T21:11:42Z

@junwang-noaa From my tests, I saw the grib2 results from RT control & control_2dwrtdecomp, and RT regional_control & regional_control_2dwrtdecomp are not consistent. You might run your own RT testing.

Trying to figure this out. This did not happen in my earlier tests. I am tracking all the recent changes.

…te upp post_2d_decomp branch

JesseMeng-NOAA · 2022-03-23T18:20:44Z

@junwang-noaa From my tests, I saw the grib2 results from RT control & control_2dwrtdecomp, and RT regional_control & regional_control_2dwrtdecomp are not consistent. You might run your own RT testing.

@junwang-noaa @WenMeng-NOAA

https://github.com/JesseMeng-NOAA/ufs-weather-model/tree/feature/post_2d_decomp
Submodules of fv3 and upp are updated. Regression test for global control_2dwrtdecomp was done. Netcdf files and grib2 files are checked. The only thing "looks" not ok is GFSPRS.GrbF24 in which 4 records are all undefined in both baseline and control_2dwrtdecomp. This branch is updated with ufs-weather-model/develop. Ready for review and tests. I will be working on regional. However I believe at the moment we can confirm global first.

baseline dir = /gpfs/dell2/stmp/Jesse.Meng/FV3_RT/REGRESSION_TEST/control
working dir = /gpfs/dell2/ptmp/Jesse.Meng/FV3_RT/rt_29954/control_2dwrtdecomp
Checking test 001 control_2dwrtdecomp results ....
Comparing sfcf000.nc .........OK
Comparing sfcf024.nc .........OK
Comparing atmf000.nc .........OK
Comparing atmf024.nc .........OK
Comparing GFSFLX.GrbF00 .........OK
Comparing GFSFLX.GrbF24 .........OK
Comparing GFSPRS.GrbF00 .........OK
Comparing GFSPRS.GrbF24 .........NOT OK

GFSPRS.GrbF24
897:52457348:MCDC:middle cloud layer:rpn_corr=-nan:rpn_rms=undefined
899:52533898:HCDC:high cloud layer:rpn_corr=-nan:rpn_rms=undefined
901:52622649:TCDC:entire atmosphere (considered as a single layer):rpn_corr=-nan:rpn_rms=undefined
903:52716460:HGT:cloud ceiling:rpn_corr=-nan:rpn_rms=undefined

junwang-noaa · 2022-03-28T18:09:16Z

@JesseMeng-NOAA From the message you showed, the RT failed. If GFSPRS.GrbF24 has issue, please fix it. If the field has undefined value, then both 1D and 2D need to have same results.

JesseMeng-NOAA · 2022-03-28T18:56:05Z

@JesseMeng-NOAA From the message you showed, the RT failed. If GFSPRS.GrbF24 has issue, please fix it. If the field has undefined value, then both 1D and 2D need to have same results.

The two files have been checked with wgrib and confirmed the baseline were reproduced.

junwang-noaa · 2022-03-28T19:29:33Z

@JesseMeng-NOAA From the message you showed, the RT failed. If GFSPRS.GrbF24 has issue, please fix it. If the field has undefined value, then both 1D and 2D need to have same results.

The two files have been checked with wgrib and confirmed the baseline were reproduced.

The field cwatclm is different.

JesseMeng-NOAA · 2022-03-29T02:57:51Z

@JesseMeng-NOAA From the message you showed, the RT failed. If GFSPRS.GrbF24 has issue, please fix it. If the field has undefined value, then both 1D and 2D need to have same results.

The two files have been checked with wgrib and confirmed the baseline were reproduced.

The field cwatclm is different.

Fixed and regression test passed.
baseline dir = /gpfs/dell2/stmp/Jesse.Meng/FV3_RT/REGRESSION_TEST/control
working dir = /gpfs/dell2/ptmp/Jesse.Meng/FV3_RT/rt_21862/control_2dwrtdecomp
Checking test 001 control_2dwrtdecomp results ....
Comparing sfcf000.nc .........OK
Comparing sfcf024.nc .........OK
Comparing atmf000.nc .........OK
Comparing atmf024.nc .........OK
Comparing GFSFLX.GrbF00 .........OK
Comparing GFSFLX.GrbF24 .........OK
Comparing GFSPRS.GrbF00 .........OK
Comparing GFSPRS.GrbF24 .........OK

Branch updated
https://github.com/JesseMeng-NOAA/ufs-weather-model/tree/feature/post_2d_decomp

junwang-noaa · 2022-03-29T12:11:52Z

@JesseMeng-NOAA Thanks for resolving the issue with control_2dwrtdecomp. How about the regional_control_2dwrtdecomp test, the regional control test with 2D post decomposition? Currently the inline post is turned off in this test. If turned on, it should generate same POST results as regional_control test. Thanks

JesseMeng-NOAA · 2022-03-29T15:03:06Z

@JesseMeng-NOAA Thanks for resolving the issue with control_2dwrtdecomp. How about the regional_control_2dwrtdecomp test, the regional control test with 2D post decomposition? Currently the inline post is turned off in this test. If turned on, it should generate same POST results as regional_control test. Thanks

Just to confirm the work for global fv3 inline post is completed.
Regional is ongoing.

…est case

JesseMeng-NOAA · 2022-04-01T18:38:11Z

@JesseMeng-NOAA Thanks for resolving the issue with control_2dwrtdecomp. How about the regional_control_2dwrtdecomp test, the regional control test with 2D post decomposition? Currently the inline post is turned off in this test. If turned on, it should generate same POST results as regional_control test. Thanks

@junwang-noaa @WenMeng-NOAA
Resolved. Both global and regional regression tests passed.
Branch updated
https://github.com/JesseMeng-NOAA/ufs-weather-model/tree/feature/post_2d_decomp

Fixed a bug in ufs-weather-model/FV3/io/post_regional.F90
To run regression tests, the new base (control and regional_control) can be generated either using my branch,
or using the ufs-weather-model develop branch and manually copy
/u/Jesse.Meng/noscrub/post_refactor_2022q2/ufs-weather-model.dev/FV3/io/post_regional.F90
Either way produces identical results.
All nc files and grib files are identical between the new base and 2d decomp runs.

baseline dir = /gpfs/dell2/stmp/Jesse.Meng/FV3_RT/REGRESSION_TEST/control
working dir = /gpfs/dell2/ptmp/Jesse.Meng/FV3_RT/rt_24823/control_2dwrtdecomp
Checking test 001 control_2dwrtdecomp results ....
Comparing sfcf000.nc .........OK
Comparing sfcf024.nc .........OK
Comparing atmf000.nc .........OK
Comparing atmf024.nc .........OK
Comparing GFSFLX.GrbF00 .........OK
Comparing GFSFLX.GrbF24 .........OK
Comparing GFSPRS.GrbF00 .........OK
Comparing GFSPRS.GrbF24 .........OK

[0] The total amount of wall time = 150.152431
[0] The maximum resident set size (KB) = 461736

Test 001 control_2dwrtdecomp PASS

baseline dir = /gpfs/dell2/stmp/Jesse.Meng/FV3_RT/REGRESSION_TEST/fv3_regional_control
working dir = /gpfs/dell2/ptmp/Jesse.Meng/FV3_RT/rt_24823/regional_control_2dwrtdecomp
Checking test 002 regional_control_2dwrtdecomp results ....
Comparing dynf000.nc .........OK
Comparing dynf024.nc .........OK
Comparing phyf000.nc .........OK
Comparing phyf024.nc .........OK
Comparing PRSLEV.GrbF00 .........OK
Comparing PRSLEV.GrbF24 .........OK
Comparing NATLEV.GrbF00 .........OK
Comparing NATLEV.GrbF24 .........OK

[0] The total amount of wall time = 366.543488
[0] The maximum resident set size (KB) = 574436

Test 002 regional_control_2dwrtdecomp PASS

JesseMeng-NOAA · 2022-04-08T17:37:28Z

Sorry. I was trying to delete an under-drafting comment but did not know somehow hitting that "Delete" ended up close the PR.

JesseMeng-NOAA · 2022-05-12T16:04:54Z

This PR is closed. There have been many updates of upp and other submodules since this PR was opened but unmerged.
A new PR
#1211
is opened to support the same features of this PR.

wx22mj added 5 commits March 3, 2022 16:42

20220303 Jesse Meng create feature/post_2d_decomp

2233ad9

20220303 Jesse Meng use FV3 feature/post_2d_decomp branch

7d9b079

20220311 Jesse Meng update fv3atm and upp branch

6d18f5c

20220311 Jesse Meng update fv3 and upp branch

ce8b60a

20220311 Jesse Meng Merge remote-tracking branch 'upstream/develop' i…

94e74bf

…nto HEAD

JesseMeng-NOAA mentioned this pull request Mar 11, 2022

ufs-weather-model/fv3atm/upp 2D decomposed inline post #1078

Closed

wx22mj added 3 commits March 23, 2022 13:12

20220323 Jesse Meng update FV3 with merging upstream/develop and upda…

42d6665

…te upp post_2d_decomp branch

20220323 Jesse Meng update submodules path FV3 and upp

7392c68

Merge remote-tracking branch 'upstream/develop' into HEAD

926965e

20220328 Jesse Meng update FV3 and upp

446cc8f

wx22mj and others added 2 commits March 30, 2022 19:41

20220330 Jesse Meng update FV3/upp and regional_control_2dwrtdecomp t…

127148b

…est case

20220401 Jesse Meng update upp and FV3/io for 2d decomp

3a65b98

JesseMeng-NOAA closed this Apr 8, 2022

JesseMeng-NOAA reopened this Apr 8, 2022

JesseMeng-NOAA closed this May 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/post 2d decomp #1077

Feature/post 2d decomp #1077

JesseMeng-NOAA commented Mar 3, 2022 •

edited

Loading

junwang-noaa commented Mar 17, 2022

WenMeng-NOAA commented Mar 17, 2022

JesseMeng-NOAA commented Mar 17, 2022

JesseMeng-NOAA commented Mar 23, 2022

junwang-noaa commented Mar 28, 2022

JesseMeng-NOAA commented Mar 28, 2022

junwang-noaa commented Mar 28, 2022

JesseMeng-NOAA commented Mar 29, 2022

junwang-noaa commented Mar 29, 2022

JesseMeng-NOAA commented Mar 29, 2022

JesseMeng-NOAA commented Apr 1, 2022 •

edited

Loading

JesseMeng-NOAA commented Apr 8, 2022

JesseMeng-NOAA commented May 12, 2022

Feature/post 2d decomp #1077

Feature/post 2d decomp #1077

Conversation

JesseMeng-NOAA commented Mar 3, 2022 • edited Loading

PR Checklist

Description

Issue(s) addressed

Testing

Dependencies

junwang-noaa commented Mar 17, 2022

WenMeng-NOAA commented Mar 17, 2022

JesseMeng-NOAA commented Mar 17, 2022

JesseMeng-NOAA commented Mar 23, 2022

junwang-noaa commented Mar 28, 2022

JesseMeng-NOAA commented Mar 28, 2022

junwang-noaa commented Mar 28, 2022

JesseMeng-NOAA commented Mar 29, 2022

junwang-noaa commented Mar 29, 2022

JesseMeng-NOAA commented Mar 29, 2022

JesseMeng-NOAA commented Apr 1, 2022 • edited Loading

JesseMeng-NOAA commented Apr 8, 2022

JesseMeng-NOAA commented May 12, 2022

JesseMeng-NOAA commented Mar 3, 2022 •

edited

Loading

JesseMeng-NOAA commented Apr 1, 2022 •

edited

Loading