Bring exchange grid support to UFS

[uturuncoglu]

PR Checklist

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.

• New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsiblity to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

The PR aims to bring exchange grid capability, which is provided by CMEPS mediator, to the UFS weather model. The PR includes following major changes in the model;

• CMEPS is modified to act as a CCPP host model to perform atmosphere-ocean flux computation on exchange grid under mediator. This is required to move atmosphere-ocean flux calculation to mediator. The following options are added to the nems.configure and needs to be placed MED_attributes:: section.
  • Two new coupling mode is included;
    • nems_frac_aoflux, It allows to pass mediator calculated atmosphere-ocean fluxes to FV3
    • nems_frac_aoflux_sbs, This mode is just for side-by-side comparisons of fluxes calculated on FV3/CCPP and CMEPS/CCPP. In this mode, the fluxes are calculated under mediator but not sent to FV3. So, the results of fully coupled model has no answer change since the fluxes does not received by FV3 but the mediator history files will include also fluxes calculated under mediator.
  • CCPP/Physics specific options: Some of those options is not used by atmosphere-ocean flux calculation but required to satisfy running simplified CCPP suite (*). Maybe there is no need to make all of them configurable through the use of nems.configure and default values works but it might be required in the near future when external land component brings in.
    • ccpp_phy_lseaspray: Include sea-spray effect to atmosphere-ocean flux calculation. The default value is true.
    • ccpp_phy_ivegsrc (*): Flag for vegetation dataset. The default value is 1.
    • ccpp_phy_redrag: Flag for reduced drag coeff. over sea. The default value is true.
    • ccpp_phy_lsm (*): Flag for land surface model. The default value is 2.
    • ccpp_phy_frac_grid: Flag for fractional grid. The default value is true.
    • ccpp_phy_restart: flag whether this is a coldstart (false) or a warmstart/restart (true). The default value is true.
    • ccpp_phy_cplice (*): Flag for activating ice coupling under CCPP. The default value is true.
    • ccpp_phy_cplflx: Flag for activating ocean coupling under CCPP. The default value is true.
    • ccpp_phy_lheatstrg (*): Flag for canopy heat storage parameterization. The default value is true.
    • ccpp_restart_interval: The time interval (in sec.) to write CCPP restart file. The default value is 3600 sec. and the negative numbers can be given to disable writing restart files.
    • ccpp_restart_file: The name of the restart file that will be read by CCPP Host model. If it is not set and the run is restart run (start_type = 'continue'), then CCPP host model uses internally calculated file name.
    • ccpp_ini_file_prefix: The name of the initial condition file that will be read by CCPP Host model. The default value is INPUT/sfc_data.tile. Note that there is no need to specify tile no.
    • ccpp_ini_mosaic_file: The name of mosaic grid file that will be used to read tiled files.
    • ccpp_input_dir: The directory that will contain the tiled grid files pointed by ccpp_ini_mosaic_file. The default value is INPUT/
    • ccpp_ini_layout: The layout for creating FMS domain to read initial condition files. By default, the code internally calculates the layout pair but user could also provide the coma separated list like 3,8.
    • ccpp_ini_read: Flag for reading initial conditions. The default value is false (CMEPS CCPP host will not read the initial conditions).
• The CCPP/physics that works under CMEPS uses simplified version of the suite files to calculate only atmosphere-ocean fluxes.
• The CCPP/framework is also modified to support multiple CCPP models under same modeling system and process suite files correctly.
• The connection between FV3 and CCPP is also modified to receive atmosphere-ocean fluxes from the CMEPS mediator. A new input.nml option (use_med_flux) is added to gfs_physics_nml group. The default value is .false. and the FV3/CCPP Physics will no accept any flux from external sources such as CMEPS. This allow to preserve the existing functionality in the FV3 side.
• A new regression test (cpld_control_c96_noaero_p8_xgrid) is also implemented to test the exchange grid functionality. The baseline needs to be placed into the common input and baseline directory. There is no need to have separate input file for this test since it is just a derivative of cpld_control_c96_noaero_p8.

Issue(s) addressed

Link the issues to be closed with this PR, whether in this repository, or in another repository.
(Remember, issues must always be created before starting work on a PR branch!)

• fixes Bringing exchange grid capability to UFS #1153

Testing

How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss_cray
• wcoss_dell_p3
• opnReqTest for newly added/changed feature
• CI

The model is tested on NCAR's Cheyenne platform with intel compiler. All regression tests are passing at this point.

Dependencies

If testing this branch requires non-default branches in other repositories, list them. Those branches should have matching names (ideally).

Do PRs in upstream repositories need to be merged first?
If so add the "waiting for other repos" label and list the upstream PRs

• waiting on Enable CCPP host model under CMEPS and updates for UFS exchange grid capability ESCOMP/CMEPS#282 (already merged)
• waiting on Update CMEPS to its ESCOMP/master branch 20220607 commit NOAA-EMC/CMEPS#67
• waiting on Enable receiving atmosphere-ocean fluxes from external component NOAA-EMC/fv3atm#517
• waiting on Update sfc_ocean to accept atmosphere-ocean fluxes from external component. NCAR/ccpp-physics#894
• waiting on ccpp_prebuild: ccpp stub and various bugfixes NCAR/ccpp-framework#436

[jkbk2004]

@uturuncoglu cmeps branch merged. Can you create fv3atm PR to NOAA-EMC/fv3atm develop to merge in your feature branch?

[jkbk2004]

@uturuncoglu Oh! we have fv3atm PR already. #517. Sounds like ccpp PRs should be merged in first.

[jkbk2004]

@uturuncoglu can you coordinate with @grantfirl to merge in your ccpp PRs?

[uturuncoglu]

@jkbk2004 I add comment to CCPP PR about it. Thanks for your help.

[uturuncoglu]

@jkbk2004 I updated FV3 PR CCPP submodule pointers after merging CCPP physics and framework PRs. I think FV3 PR is ready to merge.

[BrianCurtis-NOAA]

FV3 merged: Hash 5117434

[jkbk2004]

@DusanJovic-NOAA @junwang-noaa This PR is ready to merge in. Can you leave the review comments?

[climbfuji]

Also, submodule pointers may still need to be updated.

[BrianCurtis-NOAA]

@DusanJovic-NOAA @junwang-noaa This PR is ready to merge in. Can you leave the review comments?

@jkbk2004 Make sure @uturuncoglu updates this PR with the new hash of FV3 and updates .gitmodules before you merge it into develop branch.

[jkbk2004]

@uturuncoglu Can you pull in hash update and revert gitmodule pointers?

[junwang-noaa]

I think previous cpld_control_noaero_p8 does not have wave, but it was updated to include wave in the latest develop branch. Maybe we can consider changing the test name or enabling this exchange grid test to include wave and aerosol later.

[uturuncoglu]

@junwang-noaa I did not know it. It could be nice to update it to be consistent with the existing configurations. We could rename the test but that will require creating baselines again or at least changing folder names. Let me know what do you think? If you want to rename the test, do you have any name in your mind. Thanks.

[uturuncoglu]

@junwang-noaa @climbfuji @jkbk2004 Both FV3 and CMEPS submodules are updated to point NOAA/EMC repos.

[junwang-noaa]

@uturuncoglu Please see my comments, do you think the exchange grid can work with cpld_control_p8 test with wave and aerosol? Maybe in the future, we want to change the cpld_control_noaero_p8_agrid to cpld_control_p8. Thanks

[uturuncoglu]

@junwang-noaa yes. it will work with cpld_control_p8 without any issue.

[uturuncoglu]

@junwang-noaa @climbfuji @jkbk2004 @MinsukJi-NOAA @DusanJovic-NOAA @DeniseWorthen Thanks all of you for your help.

[climbfuji]

@junwang-noaa @climbfuji @jkbk2004 @MinsukJi-NOAA @DusanJovic-NOAA @DeniseWorthen Thanks all of you for your help.

Champagne time!!!

[rsdunlapiv]

Awesome work everyone getting this one in. This required a lot of collaboration and a lot of software components. @uturuncoglu congratulations on delivering this substantial new capability into the UFS!