Skip to content

Commit

Permalink
Adding data ingest for global ensembles. (ufs-community#805)
Browse files Browse the repository at this point in the history
Adds functionality to ush/retrieve_data.py that supports ingesting global ensemble members in the ways that have been used for RRFS. It also adds GEFS and GDAS known data locations to ush/templates/data_locations.yml. This means that it will pull data from AWS and/or HPSS for:

- GEFS ICS and LBCS grib2 a & b files (it does not merge them, as that is outside the scope of this tool's functionality).
- GDAS ICS and LBCS netcdf files. For ICS, it needs atm and sfc files. For LBCS, it only needs atm files.

The tool is set up to pull all members as a single job, or it can be configured to do a subset of members if the user prefers to pull subsets of members. Just like for other datasets, the ICS and LBCS should be handled in separate calls to the tool, and the tool only handles a single start time since that is consistent with the behavior of the workflow.

Additional functionality now includes the ability to template the output directory to include the ensemble member so each member can be stored in its own separate directory when the workflow runs.

I have added functional tests to pull data from AWS and HPSS, and have hooked the AWS tests up to GitHub Actions to run when the relevant files have changed. No user intervention is needed to have those tests run -- they should run automatically upon opening a PR.
  • Loading branch information
christinaholtNOAA authored Aug 12, 2022
1 parent 3d624df commit 9870430
Show file tree
Hide file tree
Showing 4 changed files with 841 additions and 203 deletions.
33 changes: 33 additions & 0 deletions .github/workflows/python_func_tests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Python unittests
on:
push:
paths:
- ush/*retrieve_data.py
- ush/templates/data_locations.yml
pull_request:

env:
CI: true

jobs:
python_functests:
name: Python Functional Tests
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v2

# Install dependencies
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install python3 python3-pip netcdf-bin
sudo pip3 install pyyaml jinja2 f90nml
sudo pip3 install numpy matplotlib basemap
# Run python functional tests
- name: Run python functional tests
run: |
cd ush
python3 -m unittest -b test_retrieve_data.FunctionalTesting
Loading

0 comments on commit 9870430

Please sign in to comment.