Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update GDAS initialization scripts for GFS v16 #102

Closed
GeorgeGayno-NOAA opened this issue May 8, 2020 · 13 comments · Fixed by #531
Closed

Update GDAS initialization scripts for GFS v16 #102

GeorgeGayno-NOAA opened this issue May 8, 2020 · 13 comments · Fixed by #531
Assignees
@KateFriedman-NOAA @GeorgeGayno-NOAA
Labels
enhancement New feature or request

Comments

@GeorgeGayno-NOAA
Copy link
Collaborator

The GDAS initialization scripts must be updated to support v16 cycled experiments - with and without the Incremental Analysis Update (IAU).
@GeorgeGayno-NOAA GeorgeGayno-NOAA added the enhancement New feature or request label May 8, 2020
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue May 14, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16 This commit references ufs-community#102.
0a5e257 
Baseline initial set of v16 based scripts.  Since there
are not sufficient enkf files stored to hpss, the
scripts use the gdas data as input for both hires and enkf
members.  The output from chgres for the enkf member is
copied 80 times.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jun 16, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16 This commit references ufs-community#102.
8f18239 
Merge branch 'develop' into feature/gdas_init.v16
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jul 20, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16 This commit references ufs-community#102.
689f8a9 
Merge branch 'develop' into feature/gdas_init.v16
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Aug 12, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16: Some bug fixes.
8a5f703 
References ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Sep 15, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16: Merge branch 'develop' into feature/gdas_init.v16
e83c9bb 
Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 9, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
91c666e 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 9, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
c08014f 
Add 'atmos' subdirectory to v16 gdas init scripts.

Issue ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 20, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
616b6c2 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 23, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
2a2bbad 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Oct 27, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
8d60bd2 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 4, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
060b96e 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 16, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
dc236f0 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 20, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
6a478eb 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 20, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16 Update v16 data pull script to
f7c6f16 
point at real-time parallel.

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Nov 23, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
490fd47 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Dec 1, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
f28a0bf 
Update Cray driver script for v16 data.

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Dec 1, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
924c71a 
Update Dell driver script to work with v16 data (thanks, Russ).

After 2020082018, the v16 tarballs include an 'atmos' subdirectory.
Account for this in 'run_v16.chgres.sh'.

Issue ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Dec 2, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
0add68c 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Dec 15, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
9353b5f 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Dec 28, 2020
@GeorgeGayno-NOAA
feature/gdas_init.v16
e07b318 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 8, 2021
@GeorgeGayno-NOAA
feature/gdas_init.v16
33da00c 
Merge branch 'develop' into feature/gdas_init.v16

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 8, 2021
@GeorgeGayno-NOAA
feature/gdas_init.v16
528b85c 
Add new v16 data script based on Eric's hpss repo.

Issue ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 11, 2021
@GeorgeGayno-NOAA
feature/gdas_init.v16
fa5c3aa 
Update to pull the v16 radstat file.

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 12, 2021
@GeorgeGayno-NOAA
feature/gdas_init.v16
8c4a985 
New script run_v16.chgres2.sh to run chgres using v16
data from Eric's parallel.

Issue ufs-community#102.
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jan 12, 2021
@GeorgeGayno-NOAA
feature/gdas_init.v16
edb894f 
Add CDUMP variable to config file.  When set to 'gfs'
only process for a gfs free forecast.  When set to 'gdas',
process gdas and enkf members.  Update v16 scripts
accordingly.

Issue ufs-community#102.
@GeorgeGayno-NOAA
Copy link
Collaborator Author

Some new requirements from @KateFriedman-NOAA to help the gdas_init utility work with the global workflow.

  • Add option to support GFS free forecast mode: read data from the GFS tarball, don't run chgres for the enkf members, and don't save the 'abias' or 'radstat' files. Place data under a directory with the name - gfs.YYYYMMDD.
  • Some of the chgres run scripts contain variable CINP, which sets the resolution of the input data. Update the logic so it can be overridden if the user decides.

@edwardhartnett
Copy link
Collaborator

Is there a test for these changes?

@KateFriedman-NOAA
Copy link
Collaborator

Ok. So you want me to keep the option to use v16 pre-implementation data?

Yes please, thanks!

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 1, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
f2314fc 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 1, 2021
@GeorgeGayno-NOAA
feature/gdas_init.v16: Add warning about using v16 retrospective
98e9730 
parallel data.

Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 5, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
e97691c 
Fixes ufs-community#102
@KateFriedman-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA I've retested your feature/gdas_init.v16 branch on Hera for all supported GFS versions (v12, v13, v14, v15, v16 retro, v16 ops). I had no issues running chgres_cube via a global-workflow job. Tests:

v12:
/scratch1/NCEPDEV/global/Kate.Friedman/expdir/ffv12
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/ffv12

v13:
/scratch1/NCEPDEV/global/Kate.Friedman/expdir/ffv13
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/ffv13

v14:
/scratch1/NCEPDEV/global/Kate.Friedman/expdir/ffv14
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/ffv14

v15:
/scratch1/NCEPDEV/global/Kate.Friedman/expdir/ffv15
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/ffv15

v16 retro:
/scratch1/NCEPDEV/global/Kate.Friedman/expdir/ffv16r
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/ffv16r

v16 ops:
/scratch1/NCEPDEV/global/Kate.Friedman/expdir/ffv16o
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/ffv16o

Going to run the same tests on WCOSS-Dell to confirm no issues there. Then will be ready for feature/gdas_init.v16 to go into UFS_UTILS develop. Thanks!

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 12, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
49394bf 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 14, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
0659c15 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 16, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
dc7b912 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 21, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
9255146 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 21, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
3a56775 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 23, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
de25344 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 26, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
85dc4f9 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 27, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
9dbab77 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Apr 29, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
ff60134 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue May 3, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
b960aa8 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue May 13, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
3f082ee 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue May 19, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
e7438e7 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue May 26, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
977b49f 
Fixes ufs-community#102
@KateFriedman-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA I'm ready for this work to move into develop. I'm wrapping up testing my branch for this work on the workflow side and will be ready for a new hash or tag from UFS_UTILS that includes these updates. Thanks!

@GeorgeGayno-NOAA
Copy link
Collaborator Author

@GeorgeGayno-NOAA I'm ready for this work to move into develop. I'm wrapping up testing my branch for this work on the workflow side and will be ready for a new hash or tag from UFS_UTILS that includes these updates. Thanks!

Ok, Kate. Glad to finally get this merged. Will create a release for you.

@KateFriedman-NOAA
Copy link
Collaborator

Will create a release for you.

Wonderful, thanks @GeorgeGayno-NOAA !

GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jun 1, 2021
@GeorgeGayno-NOAA
Merge branch 'develop' into feature/gdas_init.v16
57fc550 
Fixes ufs-community#102
@GeorgeGayno-NOAA GeorgeGayno-NOAA mentioned this issue Jun 1, 2021
Merged
GeorgeGayno-NOAA added a commit to GeorgeGayno-NOAA/UFS_UTILS that referenced this issue Jun 1, 2021
@GeorgeGayno-NOAA
Remove hardwired USER from ./gdas_init/config
c527f9c 
Fixes ufs-community#102
GeorgeGayno-NOAA added a commit that referenced this issue Jun 3, 2021
@GeorgeGayno-NOAA
Update the GDAS initialization scripts to process GFS v16 data. (#531)
f09acb9 
Update to process OPS GFS v16 data or GFS v16 parallel data.

For all input data sources, add option to process the GFS or the GDAS 
CDUMP.  In other words, process for a GFS free forecast or process 
for a GDAS/ENKF cycle

Fixes #102
@GeorgeGayno-NOAA GeorgeGayno-NOAA mentioned this issue Jun 3, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KateFriedman-NOAA KateFriedman-NOAA

@GeorgeGayno-NOAA GeorgeGayno-NOAA

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update the GDAS initialization scripts to process GFS v16 data. GeorgeGayno-NOAA/UFS_UTILS
3 participants
@KateFriedman-NOAA @edwardhartnett @GeorgeGayno-NOAA