Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding data ingest for global ensembles. (ufs-community#805)
Adds functionality to ush/retrieve_data.py that supports ingesting global ensemble members in the ways that have been used for RRFS. It also adds GEFS and GDAS known data locations to ush/templates/data_locations.yml. This means that it will pull data from AWS and/or HPSS for: - GEFS ICS and LBCS grib2 a & b files (it does not merge them, as that is outside the scope of this tool's functionality). - GDAS ICS and LBCS netcdf files. For ICS, it needs atm and sfc files. For LBCS, it only needs atm files. The tool is set up to pull all members as a single job, or it can be configured to do a subset of members if the user prefers to pull subsets of members. Just like for other datasets, the ICS and LBCS should be handled in separate calls to the tool, and the tool only handles a single start time since that is consistent with the behavior of the workflow. Additional functionality now includes the ability to template the output directory to include the ensemble member so each member can be stored in its own separate directory when the workflow runs. I have added functional tests to pull data from AWS and HPSS, and have hooked the AWS tests up to GitHub Actions to run when the relevant files have changed. No user intervention is needed to have those tests run -- they should run automatically upon opening a PR.
- Loading branch information