GFSv16.3.15 - Update to obsproc.v1.2 #2291

KateFriedman-NOAA · 2024-02-06T15:36:43Z

What new functionality do you need?

Update to obsproc.v1.2 when it is introduced in operations. Also make related GSI change to turn GMI back on.

What are the requirements for the new functionality?

Update obsproc_ver in version files in dev/gfs.v16
Update obsproc_ver in version files in develop
Install obsproc.v1.2 on R&D machines
Uncomment GMI line scripts/exglobal_atmos_analysis.sh (commented out in GFSv16.3.5[6] GSI updates #1404).
Update ops version in RTD

Acceptance Criteria

New version works everywhere.

Suggest a solution (optional)

No response

The text was updated successfully, but these errors were encountered:

KateFriedman-NOAA · 2024-05-02T12:10:20Z

Update from @ilianagenkova - implementation date next Wednesday May 8th during the 12z cycle:

Hello users,

Following a successful PPT in early April, and should there be no CWD or anything else unexpected, 
NCO is targeting to implement obsporc v1.2  on May 8th 12z .

This update will:
- add high-vertical resolution radiosonde profiles (up to 255 pressure levels) to the global and RAP prepbufr files;
- add high-vertical resolution radiosonde profiles (full profiles) to the newly established global and RAP *uprair* bufr dump files;
- add high-spatial and temporal resolution hurricane tailored winds for the HAFS model in the *satwhr* buff dump files;
- increase the amount of msonet observations across all regional models 

Expect larger prepbufr  and msonet bufr dump files and allow for longer run times reading/processing these.

Also, more obs data will be shared via NOMADS -
  - GFS gfs.tCCz.TYPE.tm00.bufr_d

Where TYPE is BUFR formatted files *ahicsr*, *crisf4*, *crsfdb*, *gsrasr*, *gsrcsr*, *omi*, *ompslp*, *ompsn8*, *ompst8*, *satwhr*, *sevasr*, *uprair*

  - GDAS gdas.tCCz.TYPE.tm00.bufr_d

Where TYPE is BUFR formatted files *ahicsr*, *avcsam*, *avcspm*, *crsfdb*, *gsrasr*, *gsrcsr*, *omi*, *ompslp*, *ompsn8*, *ompst8*, *satwhr*, *sevasr*, *uprair*

 - RAP rap.tCCz.TYPE.tm00.bufr_d

Where TYPE is BUFR formatted files *gsrasr*, *gsrcsr*, and *uprair*

See the [SCN ](https://docs.google.com/document/d/1qRUw5xO4zKJSedNMNp555tpjfw1h87XOK2-wdTEHlP8/edit)for details.

Feel free to forward this to anyone who might benefit from the heads up.

Thanks, 
iliana

- Update the obsproc_ver to v1.2 - Update the gfs_ver to v16.3.15 Refs NOAA-EMC#2291

Refs NOAA-EMC#2291

Updates for the upstream dependency upgrade to obsproc/v1.2. Version numbers are increased and release notes are created. Refs #2291

KateFriedman-NOAA · 2024-05-06T12:46:11Z

From May 3rd RFC memo:

RFC 12604 - On WCOSS2, implement the OBSPROC v1.2.0 upgrade along
with downstream models' updates. This upgrade will:

● Expand the global observation datasets (gfs and gdas) with high vertical
resolution radiosonde profiles, and high spatial and temporal resolution
satellite winds tailored for hurricane events
● Expand the rap observation dataset with high vertical resolution
radiosonde profiles. It will also increase the volume of msonet
observations across all regional networks and expand the capacity of
SDMEDIT list entries per observation type.
● Several internal BUFR-lib related arrays were increased to accommodate
increasing obs data volumes.
● A number of obs data sets were added to NOMADS (see SCN).
● Turn off the obsproc_air_now job since the EPA AirNow data was
stopped.

To be implemented on May 8, 1100Z to 1800Z and May 9, 1400Z to 1800Z.

ilianagenkova · 2024-05-08T18:35:06Z

SPA update: Due to ECE/CWD, the OBSPROC upgrade is postponed to May 09 2024 12z .

KateFriedman-NOAA · 2024-05-09T12:52:17Z

Update from @ilianagenkova and NCO: Due to Enhanced Caution remaining in effect until 1200Z Fri May 10 2024 SPAs will implement obsproc v1.2 on Monday 5/13 12z.

KateFriedman-NOAA · 2024-05-13T12:29:33Z

Delayed to May 16th due to continued EC. From last Friday's RFC memo (date now changed):

RFC 12604 - On WCOSS2, implement the OBSPROC v1.2.0 upgrade along
with downstream models' updates. This upgrade will:
● Expand the global observation datasets (gfs and gdas) with high vertical
resolution radiosonde profiles, and high spatial and temporal resolution
satellite winds tailored for hurricane events
● Expand the rap observation dataset with high vertical resolution
radiosonde profiles. It will also increase the volume of msonet
observations across all regional networks and expand the capacity of
SDMEDIT list entries per observation type.
● Several internal BUFR-lib related arrays were increased to accommodate
increasing obs data volumes.
● A number of obs data sets were added to NOMADS (see SCN).
● Turn off the obsproc_air_now job since the EPA AirNow data was
stopped.
Was to be implemented on May 8, 1100Z to 1800Z and May 9, 1400Z to 1800Z;
postponed due to ECE/CWD; new dates are May 13 (1100Z to 1800Z) and May
14 (1500Z to 1800Z.

Updates for the upstream dependency upgrade to obsproc/v1.2. Implemented into operations May 22nd during the 12z cycle. Refs #2291

KateFriedman-NOAA · 2024-05-23T14:32:20Z

Release created and announced to users: https://github.com/NOAA-EMC/global-workflow/releases/tag/gfs.v16.3.15

- Update to obsproc/v1.2.0 and prepobs/v1.1.0 - Revert back to glopara installs on Orion/Hercules - Remove default version for obsproc in config.base Refs NOAA-EMC#2291 Refs NOAA-EMC#2840 Refs NOAA-EMC#2841

KateFriedman-NOAA · 2024-08-26T16:30:27Z

@WalterKolczynski-NOAA Please install the new obsproc/v1.2.0 on Orion and Hercules:

git clone -b obsproc.v1.2.0-rd-gfsv17 https://github.com/NOAA-EMC/obsproc.git ./v1.2.0
cd v1.2.0
INSTALL_TARGET=orion[hercules] INSTALL_PREFIX=path_to_install ./ush/build.sh

WalterKolczynski-NOAA · 2024-08-26T17:04:53Z

Installed in /work/noaa/global/glopara/git_rocky9/obsproc/v1.2.0. Since Orion is now also Rocky 9, should we just switch Orion to use that location as well?

KateFriedman-NOAA · 2024-08-26T17:39:25Z

Installed in /work/noaa/global/glopara/git_rocky9/obsproc/v1.2.0. Since Orion is now also Rocky 9, should we just switch Orion to use that location as well?

Yeah, good idea. That would mean changing BASE_GIT in workflow/hosts/orion.yaml. I can do that in my branch that updates the obsproc/prepobs versions. Need to make sure all other external packages are installed and good-to-go in the git_rocky9 folder. I'll have a request for you to install a new prepobs version on MSU soon, so that can just go to git_rocky9 as well.

Change Orion BASE_GIT to the same as Hercules (git_rocky9). Refs NOAA-EMC#2291 Refs NOAA-EMC#2840

KateFriedman-NOAA · 2024-08-26T17:42:36Z

Updated BASE_GIT in my branch @ KateFriedman-NOAA@f36c329

Add ":-" to obsproc_run_ver variable setting for HOMEobsproc; the run.spack.ver version file is not currently sourced during forecast jobs and thus obsproc_run_ver is not set; it's not needed for forecast jobs however so allow it to be unset Refs NOAA-EMC#2291

KateFriedman-NOAA · 2024-09-17T15:33:43Z

Uncomment GMI line scripts/exglobal_atmos_analysis.sh (commented out in #1404

@ADCollard Are we able to uncomment the GMI lines in scripts/exglobal_atmos_analysis.sh yet or need to wait still? I see it is still commented out in ops:

kate.friedman@clogin03:/lfs/h1/ops/prod/packages/gfs.v16.3.18> grep GMI scripts/exglobal_atmos_analysis.sh
#GMI1CRBF=${GMI1CRBF:-${COMIN_OBS}/${OPREFIX}gmi1cr.tm00.bufr_d${OSUFFIX}} # GMI temporarily disabled due to array overflow.
#$NLN $GMI1CRBF         gmibufr   # GMI temporarily disabled due to array overflow.

ADCollard · 2024-09-17T15:37:17Z

Uncomment GMI line scripts/exglobal_atmos_analysis.sh (commented out in #1404

@ADCollard Are we able to uncomment the GMI lines in scripts/exglobal_atmos_analysis.sh yet or need to wait still? I see it is still commented out in ops:
kate.friedman@clogin03:/lfs/h1/ops/prod/packages/gfs.v16.3.18> grep GMI scripts/exglobal_atmos_analysis.sh
#GMI1CRBF=${GMI1CRBF:-${COMIN_OBS}/${OPREFIX}gmi1cr.tm00.bufr_d${OSUFFIX}} # GMI temporarily disabled due to array overflow.
#$NLN $GMI1CRBF         gmibufr   # GMI temporarily disabled due to array overflow.

@KateFriedman-NOAA This can now be uncommented.

KateFriedman-NOAA · 2024-09-17T15:38:39Z

@KateFriedman-NOAA This can now be uncommented.

Great, thanks for confirming @ADCollard ! I will open a separate issue/PR to take care of that. I'll ask you to be a reviewer.

This PR updates the `develop` branch to use the newer operational `obsproc/v1.2.0` and `prepobs/v1.1.0`. The obsproc/prepobs installs in glopara space on supported platforms use tags cut from the `dev/gfsv17` branches in the respective repos. The installation of `prepobs/v1.1.0` on WCOSS2 is called "gfsv17_v1.1.0" to help avoid GFSv16 users using it instead of the operational module. Also, the `HOMEobsproc` path is updated to set an empty default for `obsproc_run_ver`. This both removes the need to set a default (and constantly update it, which is duplication) and avoid the unset variable error when the fcst jobs use their own load module script that does not know `obsproc_run_ver`: ``` export HOMEobsproc="${BASE_GIT:-}/obsproc/v${obsproc_run_ver:-}" ``` This PR also reverts the prepobs and fit2obs installs on MSU back to the glopara space from the temporary `/work/noaa/global/kfriedma/glopara` space installs. Lastly, this PR also includes updates to complete issue NOAA-EMC#2844 (merge `build.spack.ver` and `run.spack.ver`). Resolves NOAA-EMC#2291 Resolves NOAA-EMC#2840 Resolves NOAA-EMC#2844

KateFriedman-NOAA added feature New feature or request production update Processing update in production labels Feb 6, 2024

KateFriedman-NOAA self-assigned this Feb 6, 2024

KateFriedman-NOAA mentioned this issue Feb 6, 2024

Add GFS v16.3.11[12] changes to develop #1918

Closed

9 tasks

KateFriedman-NOAA changed the title ~~Update to obsproc.v1.2~~ GFSv16.3.15 - Update to obsproc.v1.2 Apr 5, 2024

KateFriedman-NOAA mentioned this issue Apr 5, 2024

GFSv16.3.16 - Add WDQMS processing job #2389

Closed

14 tasks

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 3, 2024

Update versions for obsproc.v1.2 upgrade

63ec896

- Update the obsproc_ver to v1.2 - Update the gfs_ver to v16.3.15 Refs NOAA-EMC#2291

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 3, 2024

Create release notes for GFSv16.3.15

1b0e817

Refs NOAA-EMC#2291

KateFriedman-NOAA added a commit that referenced this issue May 3, 2024

Updates for obsproc/v1.2 into release branch (#2569)

73ac407

Updates for the upstream dependency upgrade to obsproc/v1.2. Version numbers are increased and release notes are created. Refs #2291

KateFriedman-NOAA mentioned this issue May 22, 2024

Updates for obsproc/v1.2 in operations #2616

Merged

KateFriedman-NOAA added a commit that referenced this issue May 23, 2024

Updates for obsproc/v1.2 in operations (#2616)

1a7234b

Updates for the upstream dependency upgrade to obsproc/v1.2. Implemented into operations May 22nd during the 12z cycle. Refs #2291

KateFriedman-NOAA mentioned this issue Aug 16, 2024

Update to prepobs/v1.1.0 #2840

Closed

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 26, 2024

Update Orion BASE_GIT

f36c329

Change Orion BASE_GIT to the same as Hercules (git_rocky9). Refs NOAA-EMC#2291 Refs NOAA-EMC#2840

KateFriedman-NOAA mentioned this issue Sep 11, 2024

Update to obsproc/v1.2.0 and prepobs/v1.1.0 #2903

Merged

KateFriedman-NOAA mentioned this issue Sep 17, 2024

Uncomment GMI line scripts/exglobal_atmos_analysis.sh #2934

Open

WalterKolczynski-NOAA closed this as completed in #2903 Sep 18, 2024

WalterKolczynski-NOAA closed this as completed in 7588d2b Sep 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GFSv16.3.15 - Update to obsproc.v1.2 #2291

GFSv16.3.15 - Update to obsproc.v1.2 #2291

KateFriedman-NOAA commented Feb 6, 2024 •

edited

Loading

KateFriedman-NOAA commented May 2, 2024

KateFriedman-NOAA commented May 6, 2024 •

edited

Loading

ilianagenkova commented May 8, 2024

KateFriedman-NOAA commented May 9, 2024

KateFriedman-NOAA commented May 13, 2024

KateFriedman-NOAA commented May 23, 2024

KateFriedman-NOAA commented Aug 26, 2024

WalterKolczynski-NOAA commented Aug 26, 2024

KateFriedman-NOAA commented Aug 26, 2024

KateFriedman-NOAA commented Aug 26, 2024

KateFriedman-NOAA commented Sep 17, 2024 •

edited

Loading

ADCollard commented Sep 17, 2024

KateFriedman-NOAA commented Sep 17, 2024

GFSv16.3.15 - Update to obsproc.v1.2 #2291

GFSv16.3.15 - Update to obsproc.v1.2 #2291

Comments

KateFriedman-NOAA commented Feb 6, 2024 • edited Loading

What new functionality do you need?

What are the requirements for the new functionality?

Acceptance Criteria

Suggest a solution (optional)

KateFriedman-NOAA commented May 2, 2024

KateFriedman-NOAA commented May 6, 2024 • edited Loading

ilianagenkova commented May 8, 2024

KateFriedman-NOAA commented May 9, 2024

KateFriedman-NOAA commented May 13, 2024

KateFriedman-NOAA commented May 23, 2024

KateFriedman-NOAA commented Aug 26, 2024

WalterKolczynski-NOAA commented Aug 26, 2024

KateFriedman-NOAA commented Aug 26, 2024

KateFriedman-NOAA commented Aug 26, 2024

KateFriedman-NOAA commented Sep 17, 2024 • edited Loading

ADCollard commented Sep 17, 2024

KateFriedman-NOAA commented Sep 17, 2024

KateFriedman-NOAA commented Feb 6, 2024 •

edited

Loading

KateFriedman-NOAA commented May 6, 2024 •

edited

Loading

KateFriedman-NOAA commented Sep 17, 2024 •

edited

Loading