Add multilayer UCMs to YSU PBL parameterization

[eahendricks]

TYPE: Enhancement

KEYWORDS: BEP, BEP+BEM, YSU PBL scheme, vertical diffusion equations

SOURCE: Eric A. Hendricks (NCAR/RAL/NSAP)

DESCRIPTION OF CHANGES:
The multilayer BEP (Building Effects Parameterization) and BEP+BEM (BEP with the integration of a Building Energy Model) urban canopy models (UCMs) are added to the YSU planetary boundary layer (PBL) scheme. The implicit (A) and explicit (B) BEP and BEP+BEM source terms are added as forcing to the YSU implicit/explicit vertical diffusion equation solvers. Forcing is applied for zonal momentum, meridional momentum, potential temperature, and water vapor mixing ratio. The A sources are added to the tridiagonal matrix and the B sources are added to the explicit term vectors. Additionally, the finite differencing is modified using the BEP volume fraction and surface fraction not occupied by buildings. The BEP and BEP+BEM TKE tendency terms are added to the diagnostic TKE in YSU when topographic drag is turned on.

References:

• Hendricks, E. A., J. C. Knievel, and Y. Wang, 2020: Addition of multilayer urban canopy models to a nonlocal planetary boundary layer parameterization and evaluation using ideal and real cases, J. Appl. Met. Clim., 59, 1369-1392.
• Martilli et al. (2009), Description of the modifications made in WRF.3.1 and short user's manual of BEP, NCAR technical note.

Hendricks et al. (2020) demonstrated that the scheme works properly and produces expected behaviors. Results are qualitatively similar to MYJ and BOULAC with BEP and BEP+BEM. Minor differences are due to the different treatment of vertical mixing among the schemes.

Note: This is a new pull request to replace pull request #1196. Pull request #1196 had the changes implemented into a new urban YSU routine (module_bl_ysuurb.F). This pull request adds the changes to the original YSU PBL routine (module_bl_ysu.F) instead of a new urban routine. After discussions with the WRF model and YSU scheme experts, it was determined that it would be best to implement the changes into the original YSU routine.

LIST OF MODIFIED FILES:
phys/module_bl_ysu.F
phys/module_pbl_driver.F
phys/module_physics_init.F
run/README.namelist

TESTS CONDUCTED:

1. Real-case 48-h simulations of Houston urban area were conducted (10/05/2017-10/07/2017) with an innermost nest grid spacing of 1 km. The following test suite was set up:

Baseline tests (version before physics enhancement):
a. MYJ BEP
b. MYJ BEP+BEM
c. SU BULK
d. SU SLUCM

New code tests (new version with enhancement):
e. YSU BULK
f. YSU SLUCM
g. YSU BEP
h. YSU BEP+BEM
i. MYJ BEP
j. MYJ BEP+BEM

• Tests (e) and (f) were identical to tests (c) and (d), demonstrating that the new code did not affect the BULK and SLUCM existing code with the YSU PBL parameterization.
• Tests (g) and (h) were shown to produce the same behaviors as Hendricks et al. (2020).
• Tests (i) and (j) were identical to tests (a) and (b), demonstrating that the new code did not affect any existing BEP and BEP+BEM code with the MYJ PBL parameterization.

2. The real-case example test of the NMM dynamical core was also run for the YSU scheme without BEP and BEP+BEM prior to code modifications, and after the code modifications. The output was examined and shown to be the same. Therefore, the new YSU scheme with BEP and BEP+BEM functions properly in the NMM dynamical core when BEP and BEP+BEM are not active (note that the NMM dynamical core does not support BEP and BEP+BEM).

3. Since YSU is now functional with BEP and BEP+BEM, this line was added to replace the old lines in module_physics_init.F for all other PBL routines:

IF ((SF_URBAN_PHYSICS.eq.2).OR.(SF_URBAN_PHYSICS.EQ.3)) CALL wrf_error_fatal &
            ( 'module_physics_init: use ysu (option 1), myj (option 2), or boulac (option 8) with BEP/BEM urban scheme' )

This printout was shown to occur when attempting to run with PBL schemes that do not support the BEP and BEP+BEM functionality.

4. The restart capability was demonstrated to function properly with the new YSU BEP and YSU BEP+BEM code.
5. Jenkins testing is all PASS.

RELEASE NOTE: The multilayer BEP (Building Effects Parameterization) and BEP+BEM (BEP with the Building Energy Model) urban canopy models (UCMs) are added to the Yonsei University (YSU) planetary boundary layer parameterization. Reference: Hendricks, E. A., J. C. Knievel, and Y. Wang, 2020: Addition of multilayer urban canopy models to a nonlocal planetary boundary layer parameterization and evaluation using ideal and real cases, J. Appl. Met. Clim., 59, 1369-1392.

[davegill]

jenkins is OK

Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: fb1dcdad5f9544e598a12e3d31f2fdc9eedfd11b, requested by: eahendricks for PR: https://github.com/wrf-model/WRF/pull/1305. For any query please send e-mail to David Gill.

    Test Type              | Expected  | Received |  Failed
    = = = = = = = = = = = = = = = = = = = = = = = =  = = = =
    Number of Tests        : 19           18
    Number of Builds       : 48           46
    Number of Simulations  : 166           164        0
    Number of Comparisons  : 105           104        0

    Failed Simulations are: 
    None
    Which comparisons are not bit-for-bit: 
    None

[dudhia]

Check if mods are needed in README.namelist.

…

On Mon, Oct 26, 2020 at 7:33 PM Dave Gill ***@***.***> wrote: jenkins is OK Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: fb1dcda, requested by: eahendricks for PR: #1305. For any query please send e-mail to David Gill. Test Type | Expected | Received | Failed = = = = = = = = = = = = = = = = = = = = = = = = = = = = Number of Tests : 19 18 Number of Builds : 48 46 Number of Simulations : 166 164 0 Number of Comparisons : 105 104 0 Failed Simulations are: None Which comparisons are not bit-for-bit: None — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77E53YEI4XS3OS5N3L3SMYPN5ANCNFSM4TABBX5Q> .

[eahendricks]

Thanks for letting me know about that. Yes, the README.namelist should be changed to the bolded type.

sf_urban_physics(max_dom) = 0, ! activate urban canopy model (in Noah and Noah-MP LSMs only)
= 0: no
= 1: Single-layer, UCM
= 2: Multi-layer, Building Environment Parameterization (BEP) scheme
(works only with YSU, MYJ, and BouLac PBL)
= 3: Multi-layer, Building Environment Model (BEM) scheme
(works only with YSU, MYJ, and BouLac PBL)

[weiwangncar]

I agree with the point raised by Jimy...

[dudhia]

It looks like these are packaged with the urban options, so they should not be accessed outside anything that checks whether those options are used.

…

On Tue, Jan 19, 2021 at 1:04 PM Dave Gill ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In phys/module_bl_ysu.F <#1305 (comment)>: > + a_u(:,:,:)=0.0 + a_v(:,:,:)=0.0 + a_t(:,:,:)=0.0 + a_q(:,:,:)=0.0 + a_e(:,:,:)=0.0 + b_u(:,:,:)=0.0 + b_v(:,:,:)=0.0 + b_t(:,:,:)=0.0 + b_q(:,:,:)=0.0 + b_e(:,:,:)=0.0 + sfk(:,:,:)=1.0 + vlk(:,:,:)=1.0 + dl_u(:,:,:)=0.0 + dlg(:,:,:)=0.0 + frcurb(:,:)=0.0 + Are these initializations necessary? 20 lines down, if the "flag_bep" is true, then they are all initialized. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1305 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77EIVE2SCUC6MTN7FCLS2XQWRANCNFSM4TABBX5Q> .

[dudhia]

Note that being in a package, these arrays will not be allocated space when the urban option is 0, so this code should not be needed. Given this, any code using these arrays has to be protected to avoid out-of-bounds usage of unallocated arrays.

…

On Tue, Jan 19, 2021 at 1:37 PM eahendricks ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In phys/module_bl_ysu.F <#1305 (comment)>: > + a_u(:,:,:)=0.0 + a_v(:,:,:)=0.0 + a_t(:,:,:)=0.0 + a_q(:,:,:)=0.0 + a_e(:,:,:)=0.0 + b_u(:,:,:)=0.0 + b_v(:,:,:)=0.0 + b_t(:,:,:)=0.0 + b_q(:,:,:)=0.0 + b_e(:,:,:)=0.0 + sfk(:,:,:)=1.0 + vlk(:,:,:)=1.0 + dl_u(:,:,:)=0.0 + dlg(:,:,:)=0.0 + frcurb(:,:)=0.0 + Yes, they are necessary for the initialization when "flag_bep" is false. I wanted to have a default initialization for the way most users will run (without BEP). — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77GTMCBQDOT34WOIYULS2XUPVANCNFSM4TABBX5Q> .

[weiwangncar]

@eahendricks Can you address the package issue raised by Jimy?

[eahendricks]

These are not packaged only with the urban options. They need to be allocated and have the default values for urban option 0, at least in the current way I wrote the code (see lines 1260-1265, for example, or lines 1230-1231). I'd be happy to modify this code as you all see fit, but right now I am unsure what you want me to change.

[weiwangncar]

@eahendricks If you receive the results from Jenkins test, can you please post it?

[davegill]

@weiwangncar @eahendricks
Wei,
The last jenkins I have is 5 days ago, associated with the latest mods. All pass.

Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: f1d9d7f862593b35e0f1e6ebfdf9e8d4903e9ee8, requested by: eahendricks for PR: https://github.com/wrf-model/WRF/pull/1305. For any query please send e-mail to David Gill.

    Test Type              | Expected  | Received |  Failed
    = = = = = = = = = = = = = = = = = = = = = = = =  = = = =
    Number of Tests        : 19           18
    Number of Builds       : 48           46
    Number of Simulations  : 163           161        0
    Number of Comparisons  : 103           102        0

    Failed Simulations are: 
    None
    Which comparisons are not bit-for-bit: 
    None

[eahendricks]

Thanks for the posting the jenkins @davegill. I also ran the new code through my test suite, and the physical results look the same as before the tile-size correction. I should note that all physical tests (making sure results are sensible) were done using INTEL (ifort/icc): SGI MPT configure=24 option (dmpar) on Cheyenne.

[weiwangncar]

@eahendricks May I ask for one more test? This is a test for OpenMP. You can compile this code using OpenMP (or smpar) only, and run the model twice (using 4 or more threads) and use diffwrf (built in external/io_netcdf/ directory to compare the results. If they are identical, then it is likely mean that this code is thread safe. I ask this because YSU+urban is not tested using Jenkins. If you would like to do more, a more formal test would be to compile the code with configure -d for both dmpar and smpar separately, and compare the model output using diffwrf (10 steps are we do in Jenkins tests). Thanks.

[eahendricks]

@weiwangncar Sure, no problem. I configured on Cheyenne with option 23 (smpar). But I have never run WRF in this configuration before. Do you have an example smpar/OpenMP run script you can pass me (any special directives and the command)? (assuming mpiexec_mpt will not work). I tried mpirun and it did not work either. I tried googling all of this but no luck so thought in the interest of time I would just ask.

[weiwangncar]

@eahendricks Understood. I just learnt that myself. Here are the two lines that are important in a script:
#PBS -l select=1:ncpus=4:mpiprocs=4
./wrf.exe
If you'd like to see a full script, try /glade/scratch/weiwang/runomp.csh. It uses 4 threads in this example.

[eahendricks]

@weiwangncar Thanks. I did a run with 36 threads because of larger domain. With sf_urban_physics=0 it ran ok. With sf_urban_physics=3 (BEP+BEM), it crashed. So it looks like there is a problem with BEP+BEM and also likely BEP. I'll look into this more tomorrow. These are the last lines in "test_job.o6531165":

CALL rrtmg_sw
call surface_driver
in SFCLAY
in NOAH DRV
wrf_abort

The directory of the run is here: /glade/scratch/erichend/WRF/ysu4smpar/wrf/. The error appears to be in "module_surface_driver.F". But it is strange because I did not modify this file. If you have any suggestions on what may be causing this error, please let me know. Glad you suggested this extra test.

[weiwangncar]

@eahendricks For the no urban run, did you run the model twice and compare the results?

[weiwangncar]

@eahendricks Please don't turn on debug_levels, it's not very helpful. Good that you're using a single domain for the test. There may be a memory issue. Could you try to use 16 threads, and definitely comment out 'setenv OMP_NUM_THREADS 36' - this is not needed on Cheyenne according its documentation. I would also suggest that you split output files to one per file by setting frames_per_outfile = 1 and history_interval_s = 50 (instead of history_interval). This will make comparing files easier - at least it helps me. If you didn't compile the code with 'configure -d' (debug mode), it may be helpful to do so.

[eahendricks]

@weiwangncar No I did not compare the model twice for the no urban run. I thought Jenkins already did that, and you said YSU+urban is the one we needed to test with OpenMP? I just tried with 16 threads. Segmentation fault. Would you recommend I try a much smaller number of grid points? I was using 298x298x60. I reduced the vertical levels to 40, and same crash. I could try something like 50x50x20 (and rerun WPS).

[eahendricks]

Just ran with MYJ BEP, and it crashed with the same segmentation fault. So it is a BEP/BEP+BEM issue, and not related specifically to YSU BEP/BEP+BEM.

[weiwangncar]

@eahendricks You are right. The no-urban case is probably safe because of the Jenkins test.

[weiwangncar]

@eahendricks Try the case we use in Jenkins tests. I have wrfbdy_d01 and wrfinput_d01 created for this case in /glade/scratch/weiwang/v42/WRF/urban/ and you can try it for your code. It is a small case, easy to run. I used 4 threads for the tests.

[eahendricks]

@weiwangncar Thanks. It ran fine with my executable and YSU BEP+BEM (/glade/scratch/erichend/WRF/ysu4smparsmall/wrf/). So I think we are good to go. It must have been some issue with my WPS/WRF setup that OMP did not like.

[weiwangncar]

@eahendricks Good to know it is successful.
@davegill @dudhia Any other comments?

[eahendricks]

I ran two diffwrf tests. One with MYJ BEP+BEM and one with YSU BEP+BEM. You can find the differences from the last Jenkins file here: /glade/scratch/erichend/WRF/ysu4smparsmall/wrf/. MYJ: "DIFFWRFMYJBEPBEM.out", YSU: "DIFFWRFYSUBEPBEM.out". So neither is reproducible. This tells me maybe it is a BEP/BEM issue and not likely related to any issue with the implementation in YSU.

[weiwangncar]

@eahendricks Are the comparisons between two OpenMP runs or between MPI and OpenMP runs? Did you compile the code with 'configure -d'? In other words, is the code compiled without optimization?

[eahendricks]

@weiwangncar Two OpenMP runs. I just re-compiled with 'configure -d'. Still differences between runs for both MYJ and YSU with BEP+BEM. Have you verified previously that BEP/BEP+BEM is reproducible using OpenMP (say either with MYJ or BouLac)? From my tests, it appears not, but might be good to have another sanity check.

[weiwangncar]

@eahendricks Using 'configure -d', two OpenMP runs (run for 30 min) are identical for MYJ PBL, but different only at 30 min for BouLac (output at 3 min interval). This is for BEP+BEM, and no radiation calls at 30 min.
When compared to MPI runs, all options show differences in TSK field after first time step.
After checking our current Jenkins tests, it looks like we are NOT testing urban + Noah, and we only test urban + NoahMP, and we only run the model for a 9 min forecast (this info comes from the namelist tar file: Namelists/weekly/em_real/MPI)
@dudhia @davegill Do you have any comments?

[weiwangncar]

@eahendricks After talking to Dave, we find a namelist that we do test Noah + urban physics 2 with BouLac PBL, with nesting and 9 min of forecast time. Could you see if your code can pass this test with MPI and OpenMP? If you don't have a nest input, you can just do it for a single domain. Thanks.

[eahendricks]

@weiwangncar OpenMP results below. I tested with YSU also in addition to BouLac. Will post another update once I finish the MPI runs. Looks like the issue is BEM. Tests pass with BEP, but not with BEP+BEM.

1. Noah + urban physics 2 with BouLac PBL, one nest. PASS: diffwrf has no differences between two OpenMP runs at 9 mins.
2. Noah + urban physics 2 with YSU PBL, one nest. PASS: diffwrf has no differences between two OpenMP runs at 9 mins.
3. Noah + urban physics 3 with BouLac PBL, one nest. FAIL: diffwrf has differences between two OpenMP runs at 9 mins.
4. Noah + urban physics 3 with YSU PBL, one nest. FAIL: diffwrf has differences between two OpenMP runs at 9 mins.

[davegill]

@eahendricks @weiwangncar

OpenMP results below. I tested with YSU also in addition to BouLac. Will post another update once I finish the MPI runs. Looks like the issue is BEM. Tests pass with BEP, but not with BEP+BEM.

Folks,
We know that Chemistry is not OpenMP safe. We can just put a check_a_mundo test for urban=3 with OpenMP. And we make sure not to add an OpenMP test for jenkins. We actually already have that suite of tests that are not OpenMP friendly.

There is no need to fix an existing issue with the WRF code with this PR, or to derail this PR due to an existing WRF OpenMP issue.

[weiwangncar]

@eahendricks Thanks for doing the tests. I think this PR is ready to be merged.
@davegill Can we worry about the check after the branch cut-off?

[davegill]

@weiwangncar

Can we worry about the check after the branch cut-off?

Wei,
Yes, the clean up of "if this is OpenMP and if we are selecting urban==3; then STOP" can come later.

I am with this PR

[davegill]

@weiwangncar
Jimy and I have approved these UCM + YSU PBL changes. You may merge.

[davegill]

@weiwangncar @eahendricks

Before merging, there is an outstanding request to update the README.namelist:

Thanks for letting me know about that. Yes, the README.namelist should be changed to the bolded type.

sf_urban_physics(max_dom) = 0, ! activate urban canopy model (in Noah and Noah-MP LSMs only)
= 0: no
= 1: Single-layer, UCM
= 2: Multi-layer, Building Environment Parameterization (BEP) scheme
(works only with YSU, MYJ, and BouLac PBL)
= 3: Multi-layer, Building Environment Model (BEM) scheme
(works only with YSU, MYJ, and BouLac PBL)

[davegill]

@weiwangncar

Before merging, there is an outstanding request to update the README.namelist:

OK, the file was modified, just not mentioned in the PR. I updated the PR message. OK to merge.