Update NCAR dtc/develop from EMC master, apply missing updates from CCN/IN commit

[climbfuji]

This PR contains:

• options for IN/CNN and MERRA2 climo of microphysics (ufs-weather-model) ufs-community/ufs-weather-model#34, "Options for IN/CNN and MERRA2 climo of microphysics (ufs-weather-model)"
• remove most tests from rt_ccpp_dtc.conf that are covered in rt.conf already, update the DTC regression test baseline date tag to 20200413

Associated PRs:
NCAR/ccpp-physics#434
NCAR/fv3atm#42
#40

For regression testing, see #40.

[llpcarson]

Are you still running reg-tests for this one?

[climbfuji]

Are you still running reg-tests for this one?

Yes, I just wanted to give you a chance to look at this so that I can merge it relatively quickly. Will be adding information on regression testing below. Thanks for reviewing!

[llpcarson]

Are you still running reg-tests for this one?

Yes, I just wanted to give you a chance to look at this so that I can merge it relatively quickly. Will be adding information on regression testing below. Thanks for reviewing!

Thanks, the early look is very helpful, will approved once the RTs are done.

[climbfuji]

Regression testing on hera.intel using rt.conf: slightly different approach than the last times.

1. rsync the EMC baseline to the DTC location with the new date tag

2. rsync the missing (= new from DTC) RRTMGP input data and regression test results from the old DTC date tag to the new date tag

3. run the full regression tests; then, the following jobs fail:

fv3_ccpp_gfsv16_csawmg_prod
fv3_ccpp_gfsv16_csawmgt_prod
fv3_ccpp_gocart_clm_prod

The differences are in the phyf*.nemsio files only, pure diagnostics (text continues below):

bash-4.2$ /scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/dnemsio/dnemsio -diff /scratch1/BMC/gmtb/ufs-weather-model/RT/NEMSfv3gfs/dtc-develop-20200413/INTEL/fv3_gocart_clm_ccpp/phyf000.nemsio /scratch1/BMC/gmtb/Dom.Heinzeller/FV3_RT/rt_281216/fv3_ccpp_gocart_clm_prod/phyf000.nemsio
 FAST_BYTESWAP ALGORITHM HAS BEEN USED AND DATA ALIGNMENT IS CORRECT FOR          4  )
  meta1
  -------------------------
 gtype        NEMSIO   OK
 gdatatype    bin4     OK
 modelname    FV3GFS   OK
 version      200809 OK
 nmeta          8 OK
 lmeta        120 OK
  meta2
  -------------------------
 nrec           151 OK
 idate         2016   10    3    0    0    0    1 OK
 dimx         384 OK
    1 dlwrf_ave        sfc                 1 OK
    2 dlwrf            sfc                 1 OK
    3 ulwrf_ave        sfc                 1 OK
    4 ulwrf            sfc                 1 OK
    5 dswrf_ave        sfc                 1 OK
    6 dswrf            sfc                 1 OK
    7 uswrf_ave        sfc                 1 OK
    8 uswrf            sfc                 1 OK
    9 duvb_ave         sfc                 1 OK
   10 cduvb_ave        sfc                 1 OK
   11 vbdsf_ave        sfc                 1 OK
   12 vddsf_ave        sfc                 1 OK
   13 nbdsf_ave        sfc                 1 OK
   14 nddsf_ave        sfc                 1 OK
   15 csulf            nom. top            1 OK
   16 csusf            nom. top            1 OK
   17 csdlf            sfc                 1 OK
   18 csusf            sfc                 1 OK
   19 csdsf            sfc                 1 OK
   20 csulf            sfc                 1 OK
   21 dswrf_ave        nom. top            1 OK
   22 uswrf_ave        nom. top            1 OK
   23 ulwrf_ave        nom. top            1 OK
   24 aod550           sfc                 1 min(diff)= 0.4275901E+02 max(diff)= 0.3551510E+04 rms1= 0.5116525E+03 rms2= 0.1421257E+00 digits=  -3 minloc=  68036 maxloc=  25085
   25 du_aod550        sfc                 1 min(diff)= 0.1986461E+01 max(diff)= 0.1761732E+04 rms1= 0.1637526E+03 rms2= 0.4548679E-01 digits=  -3 minloc=  35115 maxloc=  29585
   26 bc_aod550        sfc                 1 min(diff)= 0.4352293E+01 max(diff)= 0.2580979E+03 rms1= 0.3588449E+02 rms2= 0.9967915E-02 digits=  -3 minloc=  68424 maxloc=  24304
   27 oc_aod550        sfc                 1 min(diff)= 0.1017268E+02 max(diff)= 0.2330234E+04 rms1= 0.1665326E+03 rms2= 0.4625908E-01 digits=  -3 minloc=  30128 maxloc=  44582
   28 su_aod550        sfc                 1 min(diff)= 0.1498288E+02 max(diff)= 0.2707299E+04 rms1= 0.1981074E+03 rms2= 0.5502983E-01 digits=  -3 minloc=  65064 maxloc=  25085
   29 ss_aod550        sfc                 1 min(diff)= 0.5865656E+00 max(diff)= 0.4635436E+03 rms1= 0.1189960E+03 rms2= 0.3305444E-01 digits=  -3 minloc=  23915 maxloc=  39705
   30 tmin_min         2 m above gnd       1 OK
   31 tmax_max         2 m above gnd       1 OK
...

The reason for this are the changes in GFS_rrtmg_post.F90 in commit NCAR/ccpp-physics@7726128, which were made in IPD but forgotten in CCPP in the original PR NCAR/ccpp-physics#381

4. Create a new baseline for these three tests only and merge into the new DTC date tag - successful

rt_hera_intel_newbaseline.log

5. Rerun all tests using ./rt.sh -f -e 2>&1 | tee rt_hera_intel.log; this took two attempts, because in the first attempt several tests failed because of ongoing filesystem issues on hera. Rerunning those tests worked, in the end all tests passed.

rt_hera_intel.log
rt_hera_intel_fail_test.log
rt_hera_intel_rerun.log

[climbfuji]

Regression testing on cheyenne.gnu against new = old baseline (none of the tests in rt_gnu.conf are affected by these PRs): all tests pass.

rt_cheyenne_gnu.log

[climbfuji]

Regression testing on hera.intel using rt_ccpp_dtc.conf (because of the large number of changes in there): first, create temporary baseline, then verify against it. All tests pass.

rt_ccpp_dtc_create.log
rt_ccpp_dtc_verify.log

[climbfuji]

Regression testing on cheyenne.intel:

First, create new baseline using rt_intel.conf: all tests complete successfully, baseline created.

rt_cheyenne_intel_create.log

Second, verify against this baseline: all tests pass.

rt_cheyenne_intel_verify.log

[climbfuji]

These PRs are ready to merge.

[climbfuji]

Based on Laurie's comment above I will merge this, since the tests all passed.