GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. #149

jderber-NOAA · 2021-04-13T14:18:25Z

This change includes the use of high resolution raob data into the GSI. The data is read directly from file uprair without going into the prepbufr file. Note that because of the many levels >2000 for some profiles, the standard bufrlib could not be used and the dynamic allocation version is used. The default values for bufrlib are updated in glbsoi. After all types of data are read, the memory for the bufrlib is released using exitbufr.

Some values that are defined in the creation of the prepbufr file are not defined for this data, so default values are used. The observation errors are specified, but not much work has been done on them since a later project is already planned to do this. The new observations are given types 119 (for q,t, and ps) and 219 and 218 for uv. The observations given type 218 are those that do not have a pressure for the wind obs and only have heights (like pibals).

When there are both observations from the prepbufr file and from the high resolution data for the same station, the prepbufr data is moved to a monitored status and only the high resolution data is used. Of course if the high resolution data is monitored, the prepbufr data will be used.

Two additional input files are modified (errtable and global_convinfo.txt). They are located in /scratch1/NCEPDEV/da/John.Derber/git/gsi on hera.

To use the high definition data, the input file uprair should be pointed at the appropriate data (see Chris Hill) and the lines similar to:
uprair ps null ps 0.0 0 0
uprair t null t 0.0 0 0
uprair q null q 0.0 0 0 and/or
uprair uv null uv 0.0 0 0

should be added to the OBS_INPUT namelist.

One main additional routine was added (hdraobmod.f90). Modifications were made to setupt,setupw,setupps, setupq, read_obs.F90 and glbsoi.f90. In addition, modifications were made to many of the read routines to eliminate warning messages from the dynamic allocation version of bufrlib when files are closed when not open. There was no such warning message for the non-dynamic allocation version.

regression/regression_var.sh

modulefiles/modulefile.ProdGSI.hera

jderber-NOAA · 2021-04-30T19:18:31Z

Made a few changes that fixed 2 issues that Cathy pointed out.
Fixed issue with observation time.
Fixed issue in setupq.f90

Added qsat(i,j,k)=max(qmin,qsat(i,j,k)) to genqsat.
Fixes issue with convergence in some cases.

src/gsi/cplr_gfs_ensmod.f90

jderber-NOAA · 2021-05-06T14:09:51Z

Thanks! I added a presentation to my issue which has revised figures including descent data.

…

On Thu, May 6, 2021 at 10:02 AM CatherineThomas-NOAA < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/gsi/cplr_gfs_ensmod.f90 <#149 (comment)>: > @@ -822,6 +822,10 @@ subroutine parallel_read_nemsio_state_(en_full,m_cvars2d,m_cvars3d,nlon,nlat,nsi if (k3u==0.or.k3v==0.or.k3t==0.or.k3q==0.or.k3oz==0) & write(6,'(" WARNING, problem with one of k3-")') + do k=1,nsig @jderber-NOAA <https://github.com/jderber-NOAA> Thanks, good to know. I'm fine with keeping it in this PR as long as it's here intentionally. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#149 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASD2M5SYTQ2MSG32EWVRYQDTMKOQJANCNFSM423OC7DQ> .

src/gsi/setupq.f90

CatherineThomas-NOAA

Thanks for addressing my comments. The changes look good to me. Thanks for including me as these changes will be important for our new Hollings scholar.

emilyhcliu · 2021-05-07T15:38:56Z

@jderber-NOAA I remembered you told us in a recent obs meeting that you have made substantial changes in the NetCDF IO. I did not see changes in NetCDF IO in this review. The changes in NetCDF IO you referred to had already been merged into develop. Is this correct? We are about to add precipitation components in NetCDF IO. We would like to add on top of the changes you made.

I went through the code changes you made in this review. They look good to me.

jderber-NOAA · 2021-05-07T16:02:02Z

Emily,I made significant changes to the input of the ensembles. These changes are already in the trunk.Best,Johnp.s. I have my talk ready for Tuesday. Do you want me to send it to you so you can attach to meeting?Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: emilyhcliu ***@***.***> Date: 5/7/21 11:39 AM (GMT-05:00) To: NOAA-EMC/GSI ***@***.***> Cc: jderber-NOAA ***@***.***>, Mention ***@***.***> Subject: Re: [NOAA-EMC/GSI] GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. (#149) @jderber-NOAA I remembered you told us in a recent obs meeting that you have made substantial changes in the NetCDF IO. I did not see changes in NetCDF IO in this review. The changes in NetCDF IO you referred to had already been merged into develop. Is this correct? We are about to add precipitation components in NetCDF IO. We would like to add on top of the changes you made. I went through the code changes you made in this review. They look good to me. —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

MichaelLueken

Hi John,

I have completed my review of your changes. There are some missing precisions for real values in hdraobmod.f90, questions related to commented out lines of code (if these lines are no longer necessary, please remove them), and unused variables. The regression tests were run and all passed (or ran through to completion for the debug tests). Once these issues have been addressed, I will be able to submit this work to the review committe.

Thanks,

Mike

MichaelLueken · 2021-05-07T15:43:40Z

src/gsi/cplr_gfs_ensmod.f90

@@ -822,6 +822,7 @@ subroutine parallel_read_nemsio_state_(en_full,m_cvars2d,m_cvars3d,nlon,nlat,nsi
   if (k3u==0.or.k3v==0.or.k3t==0.or.k3q==0.or.k3oz==0) &  
      write(6,'(" WARNING, problem with one of k3-")')

+


Please remove blank added line in cplr_gfs_ensmod.f90

MichaelLueken · 2021-05-07T15:44:45Z

src/gsi/getcount_bufr.f90

-   call closbf(lunit)
+!  call closbf(lunit)


Will closbf(lunit) be used in the future? If it is, then commenting line 36 is fine. If it isn't, then please remove line 36.

MichaelLueken · 2021-05-07T15:50:38Z

src/gsi/hdraobmod.f90

+  use gsi_4dvar, only: time_4dvar,winlen
+  use qcmod, only: errormod,errormod_hdraob,njqc
+  use mpimod, only: mype
+! use blacklist, only : blacklist_read,blacklist_destroy


If blacklist's subroutines aren't going to be used in the future, please remove line 109. Otherwise, please leave it as commented out.

MichaelLueken · 2021-05-07T15:51:56Z

src/gsi/hdraobmod.f90

+!  integer(i_kind),parameter:: mxtb=5000000
+!  integer(i_kind),parameter:: nmsgmax=100000 ! max message count


If mxtb and nmsgmax aren't going to be used in the future, please remove commented out lines 132 and 133. Otherwise, these lines are fine.

MichaelLueken · 2021-05-07T15:53:28Z

src/gsi/hdraobmod.f90

+  !* for match loction station and time
+!       character(7*2000) cstn_idtime,cstn_idtime2
+!       character(7) stn_idtime(2000),stn_idtime2(2000)
+!       equivalence (stn_idtime(1),cstn_idtime)
+!       equivalence (stn_idtime2(1),cstn_idtime2)
+!       integer :: ii1,atmp,btmp,mytimeyy,ibyte
+!       character(4) stid
+!       real(8) :: rval
+!       character(len=8) :: cval
+!       equivalence (rval,cval)
+!       character(7) flnm


If lines 218 through 228 won't be used in the future, please remove these lines. Otherwise, these changes are fine.

MichaelLueken · 2021-05-07T17:41:13Z

src/gsi/hdraobmod.f90

+  integer(i_kind) minobs,minan
+  integer(i_kind) ntb,ntmatch,ncx,cat
+  integer(i_kind) nmsg                ! message index
+  integer(i_kind) jj,start,next,ncount_ps,ncount_q,ncount_uv,ncount_t


While compiling in debug mode, variables jj, start, and next were flagged as being unused. Please remove unused variables

MichaelLueken · 2021-05-07T17:42:11Z

src/gsi/hdraobmod.f90

+  real(r_kind) qoe,qobcon,dlnpob,ppb,poe,obheight
+  real(r_kind) toe,woe,errout,oelev,dlat,dlon,dlat_earth,dlon_earth
+  real(r_kind) dlat_earth_deg,dlon_earth_deg
+  real(r_kind) elev,stnelev


While compiling in debug mode, variable elev was flagged as being unused. Please remove unused variable

MichaelLueken · 2021-05-07T17:43:07Z

src/gsi/hdraobmod.f90

+  real(r_kind) toe,woe,errout,oelev,dlat,dlon,dlat_earth,dlon_earth
+  real(r_kind) dlat_earth_deg,dlon_earth_deg
+  real(r_kind) elev,stnelev
+  real(r_kind) cdist,disterr,disterrmax,rlon00,rlat00


While compiling in debug mode, variables cdist, rlon00 and rlat00 were flagged as being unused. Please remove unused variables

MichaelLueken · 2021-05-07T17:43:46Z

src/gsi/hdraobmod.f90

+  real(r_kind) del,terrmin,werrmin,perrmin,qerrmin,del_ps,del_q,del_t,del_uv
+  real(r_kind) tsavg,ff10,sfcr,zz,deltat
+  real(r_kind) time_correction,fact
+  real(r_kind),dimension(nsig):: presl,hgtl,lnpresl


While compiling in debug mode, variable lnpresl was flagged as being unused. Please remove unused variable

MichaelLueken · 2021-05-07T17:44:29Z

src/gsi/hdraobmod.f90

+  real(r_kind),dimension(nsig-1):: dpres
+  real(r_kind),dimension(maxlevs)::plevs
+  real(r_kind),allocatable,dimension(:,:):: cdata_all,cdata_out
+  real(r_kind) :: zob,tref


While compiling in debug mode, both variables zob and tref were flagged as being unused. Please remove unused variables

Mike,

Sorry there were so many issues. I will try to do better next time.

I made changes to address all of your issues (I hope). You noted that twindin and ctwind were not used. This indicated that I had not included a check to make sure it was inside the time window. When I added that some obs were rejected. This was because for some observation the reported seconds in the input bufr data is sometimes missing. I put in a check for this and if missing use 0. This changed the values in output a bit (because the time of the observation changed for the better).

I also made a change in format statements in statsconv.f90 and statslight.f90 because the number of observations was exceeding what the format statement allowed. I6 to I7.

MichaelLueken · 2021-05-18T14:06:07Z

The due date for the review committee to provide feedback has past, so I will give final approval to these changes and merge them to the NOAA-EMC/GSI master.

commit 67705a1 Merge: 5e53aaf 18afa96 Author: AndrewEichmann-NOAA <[email protected]> Date: Fri Jun 11 10:55:08 2021 -0400 Merge branch 'NOAA-EMC:master' into master commit 18afa96 Author: michael.lueken <[email protected]> Date: Thu Jun 10 11:57:29 2021 +0000 master: Updated fix/rev2 use flags for CrIS NPP due to LW band anomalies (global_satinfo.txt and added gfsv16_historical/global_satinfo.txt.2021052118 commit 51a6564 Merge: 820ee80 d199ed5 Author: MichaelLueken-NOAA <[email protected]> Date: Thu Jun 10 07:48:55 2021 -0400 Merge pull request #169 from RussTreadon-NOAA/master GitHub Issue NOAA-EMC/GSI#120. Merge GFS v16.1.1 DA changes into master commit d199ed5 Author: russ.treadon <[email protected]> Date: Thu May 27 19:08:39 2021 +0000 GitHub Issue NOAA-EMC/GSI#120. merge GFS v16.1.1 DA changes into master commit 820ee80 Merge: 6585234 86d4071 Author: MichaelLueken-NOAA <[email protected]> Date: Fri May 28 11:34:46 2021 -0400 Merge pull request #155 from AndrewEichmann-NOAA/EXP-efso_fv3_PR_util_EFSOI_src GitHub Issue NOAA-EMC/GSI#118. EFSOI additions to util/EFSOI-Utilities/src directory. commit 86d4071 Author: michael.lueken <[email protected]> Date: Fri May 28 15:31:10 2021 +0000 GitHub Issue #118. Implement EFSOI process in FV3 GFS workflow - EFSOI-specific util source code. commit 6585234 Merge: 6cec754 efaa7cf Author: MichaelLueken-NOAA <[email protected]> Date: Fri May 28 11:20:54 2021 -0400 Merge pull request #166 from comgsi/prMaster GitHub Issue NOAA-EMC/GSI#167. Change GSIWRFIO_LIB to WRF_IO_LIB and update build.comgsi accordingly. commit efaa7cf Author: Guoqing.Ge <[email protected]> Date: Wed May 26 15:51:51 2021 -0600 change GSIWRFIO_LIB to WRF_IO_LIB and update build.comgsi accordingly commit 5e53aaf Merge: e54683c 6cec754 Author: AndrewEichmann-NOAA <[email protected]> Date: Mon May 24 14:34:33 2021 -0400 Merge branch 'NOAA-EMC:master' into master commit 6cec754 Merge: 6a6ac4e 3cc26ec Author: MichaelLueken-NOAA <[email protected]> Date: Fri May 21 13:48:26 2021 -0400 Merge pull request #147 from AndrewEichmann-NOAA/EXP-efso_fv3_PR_util_EFSOI_scripts GitHub Issue NOAA-EMC/GSI#118. EFSOI additions to util/EFSOI-Utilities/scripts directory. commit 6a6ac4e Merge: 1162d92 62c884c Author: MichaelLueken-NOAA <[email protected]> Date: Thu May 20 11:03:36 2021 -0400 Merge pull request #146 from AndrewEichmann-NOAA/EXP-efso_fv3_PR_jobs GitHub Issue NOAA-EMC/GSI#118. EFSOI additions to jobs directory for issue. commit 1162d92 Merge: 6e38b98 77a729e Author: MichaelLueken-NOAA <[email protected]> Date: Wed May 19 11:02:40 2021 -0400 Merge pull request #157 from comgsi/tmpdtc GitHub Issue NOAA-EMC/GSI#158. Updates for community users to adopt the NOAA hpc-stack structure. commit 6e38b98 Merge: b511ba0 00eb5da Author: MichaelLueken-NOAA <[email protected]> Date: Wed May 19 07:27:08 2021 -0400 Merge pull request #164 from EdwardSafford-NOAA/master GitHub Issue NOAA-EMC/GSI#160. Correct data file location problem. commit 00eb5da Author: Edward.Safford <[email protected]> Date: Tue May 18 16:31:34 2021 +0000 GitHub Issue NOAA-EMC/GSI#160. Correct data file location problem. commit b511ba0 Author: michael.lueken <[email protected]> Date: Tue May 18 14:38:15 2021 +0000 GitHub Issue NOAA-EMC/GSI#126. Update global_convinfo.txt and prepobs_errtable.global for high resolution Raob data. commit 66db434 Merge: dcb735a 23ec8ef Author: MichaelLueken-NOAA <[email protected]> Date: Tue May 18 10:06:16 2021 -0400 Merge pull request #149 from jderber-NOAA/master GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. commit 77a729e Author: Guoqing.Ge <[email protected]> Date: Fri May 14 11:15:32 2021 -0600 GitHub Issue #158. Updates for community users to adopt the NOAA hpc-stack structure. commit e54683c Merge: f69c3c8 b31b851 Author: andrew.eichmann <[email protected]> Date: Thu May 13 14:39:23 2021 +0000 Merge branch 'master' of github.com:AndrewEichmann-NOAA/GSI commit f69c3c8 Merge: 1dc5f83 dcb735a Author: andrew.eichmann <[email protected]> Date: Thu May 13 14:37:10 2021 +0000 Merge branch 'master' of github.com:NOAA-EMC/GSI commit 3cc26ec Author: andrew.eichmann <[email protected]> Date: Thu May 13 14:28:03 2021 +0000 GitHub Issue #118. Implement EFSOI process in FV3 GFS workflow - EFSOI additions to util/EFSOI-Utilities/scripts directory. commit 62c884c Author: Michael Lueken <[email protected]> Date: Tue May 11 14:31:14 2021 +0000 GitHub Issue NOAA-EMC/GSI#118. Implement EFSOI process in FV3 GFS workflow - EFSOI-specific additions to jobs. commit 23ec8ef Author: john.derber <[email protected]> Date: Mon Apr 26 19:29:18 2021 +0000 GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. commit dcb735a Merge: c502c92 7939d47 Author: MichaelLueken-NOAA <[email protected]> Date: Mon May 3 12:57:43 2021 -0400 Merge pull request #153 from lbi2018/master GitHub Issue NOAA-EMC/GSI#152. commit b31b851 Author: andrew.eichmann <[email protected]> Date: Fri Apr 30 15:34:59 2021 +0000 Should not have gone into master Revert "EFSOI-specific fortran source from branch EXP-efso_fv3 - GSI issue #118" This reverts commit c9b8221. commit c9b8221 Author: andrew.eichmann <[email protected]> Date: Fri Apr 30 15:30:59 2021 +0000 EFSOI-specific fortran source from branch EXP-efso_fv3 - GSI issue #118 commit 7939d47 Author: [email protected] <[email protected]> Date: Wed Apr 28 13:15:30 2021 +0000 GitHub Issue NOAA-EMC/GSI#152. Update of modulefile.ProdGSI.wcoss_c and get_gefs_for_regional.f90 bug fixes. commit c502c92 Merge: 39ed5fa 5d77565 Author: MichaelLueken-NOAA <[email protected]> Date: Mon Apr 26 10:22:12 2021 -0400 Merge pull request #151 from EdwardSafford-NOAA/master GitHub Issue NOAA-EMC/GSI#150. Correct OznMon read diag issue. commit 5d77565 Author: edward.safford <[email protected]> Date: Fri Apr 23 19:42:54 2021 +0000 GitHub Issue NOAA-EMC/GSI#150. Correct OznMon read diag issue. commit 39ed5fa Merge: 02d5ce6 1e6fd61 Author: MichaelLueken-NOAA <[email protected]> Date: Tue Apr 13 09:37:05 2021 -0400 Merge pull request #144 from EdwardSafford-NOAA/master GitHub Issue NOAA-EMC/GSI#123. OznMon update and bugzilla fixes. commit 1e6fd61 Author: Edward.Safford <[email protected]> Date: Fri Apr 9 14:40:01 2021 +0000 GitHub Issue NOAA-EMC/GSI#123. OznMon update and bugzilla fixes. commit 1dc5f83 Author: andrew.eichmann <[email protected]> Date: Thu Apr 8 21:15:51 2021 +0000 removing EFSOI source files until they are merged into GSI master upstream

MichaelLueken requested review from MichaelLueken, CatherineThomas-NOAA and dtkleist April 13, 2021 14:37

MichaelLueken linked an issue Apr 13, 2021 that may be closed by this pull request

Add use of high resolution Raob data #126

Closed

MichaelLueken requested a review from emilyhcliu April 13, 2021 14:47

CatherineThomas-NOAA reviewed Apr 15, 2021

View reviewed changes

regression/regression_var.sh Outdated Show resolved Hide resolved

CatherineThomas-NOAA reviewed Apr 16, 2021

View reviewed changes

modulefiles/modulefile.ProdGSI.hera Outdated Show resolved Hide resolved

jderber-NOAA force-pushed the master branch 4 times, most recently from 530a4b5 to a5d32a1 Compare April 30, 2021 19:15

RussTreadon-NOAA mentioned this pull request Apr 30, 2021

GFS v16 GSI minimization issues #154

Closed

CatherineThomas-NOAA reviewed May 4, 2021

View reviewed changes

src/gsi/cplr_gfs_ensmod.f90 Outdated Show resolved Hide resolved

CatherineThomas-NOAA reviewed May 6, 2021

View reviewed changes

src/gsi/setupq.f90 Outdated Show resolved Hide resolved

jderber-NOAA force-pushed the master branch from a5d32a1 to be2bba8 Compare May 7, 2021 12:44

CatherineThomas-NOAA approved these changes May 7, 2021

View reviewed changes

emilyhcliu approved these changes May 7, 2021

View reviewed changes

MichaelLueken reviewed May 7, 2021

View reviewed changes

jderber-NOAA force-pushed the master branch 2 times, most recently from ed4e09c to 28d8e12 Compare May 8, 2021 20:41

GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data.

23ec8ef

jderber-NOAA force-pushed the master branch from 28d8e12 to 23ec8ef Compare May 10, 2021 16:00

MichaelLueken merged commit 66db434 into NOAA-EMC:master May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. #149

GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. #149

jderber-NOAA commented Apr 13, 2021

jderber-NOAA commented Apr 30, 2021

jderber-NOAA commented May 6, 2021 via email

CatherineThomas-NOAA left a comment

emilyhcliu commented May 7, 2021

jderber-NOAA commented May 7, 2021 via email

MichaelLueken left a comment

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

MichaelLueken May 7, 2021

jderber-NOAA May 8, 2021

MichaelLueken commented May 18, 2021

		@@ -822,6 +822,7 @@ subroutine parallel_read_nemsio_state_(en_full,m_cvars2d,m_cvars3d,nlon,nlat,nsi
		if (k3u==0.or.k3v==0.or.k3t==0.or.k3q==0.or.k3oz==0) &
		write(6,'(" WARNING, problem with one of k3-")')

		! integer(i_kind),parameter:: mxtb=5000000
		! integer(i_kind),parameter:: nmsgmax=100000 ! max message count

GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. #149

GitHub Issue NOAA-EMC/GSI#126. Add use of high resolution Raob data. #149

Conversation

jderber-NOAA commented Apr 13, 2021

jderber-NOAA commented Apr 30, 2021

jderber-NOAA commented May 6, 2021 via email

CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

emilyhcliu commented May 7, 2021

jderber-NOAA commented May 7, 2021 via email

MichaelLueken left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelLueken commented May 18, 2021