Update #4
Merged
Update #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…onf. (#552) Co-authored-by: Linlin.Pan <[email protected]>
Currently the domain to plot is hard-coded to conus in the plotting scripts. This PR adds the capability to choose either conus or regional or both.
…nvalid entries (#559) This is actually two fixes/enhancements, one required by the other: 1) Fix error message that appears when user specifies an invalid key in their config.yaml file. The current version references undefined variables and appears to be a copy/paste error from some other exception. 2) In order to achieve the above neatly, I had to change the behavior of the python_utils function check_structure_dict(). Rather than simply printing the invalid key/value pair, it now returns a dictionary of invalid key/value pairs (that is empty if all keys are valid). I also fixed a minor bug here: even though this function claimed to detect all invalid entries, it actually only printed the first before returning. Now all invalid entries are returned. --------- Co-authored-by: Zachary Moon <[email protected]>
…ion) (#562) This PR restores the previous behavior of the variable EXPT_BASEDIR (Item 2 is the behavior that was previously broken), which has the following effect on the experiment directory EXPTDIR: 1) If EXPT_BASEDIR is not set or set to a null value, the default value (${HOMEdir}/../expt_dirs) will be used 2) If EXPT_BASEDIR is set to a relative path (i.e. the first character is not /), the user-specified path will be appended to the default value ${HOMEdir}/../expt_dirs (for example if the user specifies EXPT_BASEDIR=some/relative/path in their config.yaml, it will be updated to EXPT_BASEDIR=${HOMEdir}/../expt_dirs/some/relative/path in the workflow 3) If EXPT_BASEDIR is set to an absolute path, that path will be used as entered After the above logic is applied, EXPTDIR will be created by joining the paths EXPT_BASEDIR and EXPT_SUBDIR as usual.
This PR introduces two new scripts to the repository: run_WE2E_tests.py and monitor_jobs.py. The purpose of these scripts is to eventually provide a pythonic replacement for the current workflow end-to-end test submission script. Additionally, the monitor_jobs function gives the capability to monitor and submit jobs automatically via the command line or a batch job, rather than relying on crontab entries.
…ecast files (#566) This PR enables running of only the SRW App's deterministic verification (vx) tasks on staged forecast files from previous runs of the App. It partially resolves Issue #565 (it resolves the issue for deterministic vx but not ensemble vx). Specific changes: * Update lua module file for vx tasks to suppress "Logging error" messages in vx task log files. * Rename experiment variable MODEL to VX_FCST_MODEL_NAME to clarify that this is the name of the forecast model in the context of verification (and which will be used in the vx output files). This requires updates to most (all?) of the * METplus configuration files and the verification ex-scripts. * Create the new variable VX_FCST_INPUT_BASEDIR to allow the user to specify a directory in which to look for staged forecast output (instead of running a forecast). * Modify the rocoto template xml (FV3LAM_wflow.xml) to make dependencies of vx tasks on post-processing tasks appear only when the post tasks are enabled. * Add a new WE2E test category subdirectory named verification in which to group all vx tests (since more vx tasks will be coming in future PRs). Move the two existing tests MET_verification and MET_ensemble_verification from wflow_features to verification, and add a new test named MET_verification_only_vxto test the capability that this PR introduces. Note: The new WE2E test MET_verification_only_vx requires new data, specifically post-processed forecast output from the SRW App. This data needs to be staged on each platform; currently, it is located in a personal directory on Hera.
PR #566 changed the variable "MODEL" to a more descriptive name, but failed to make this change in config.community.yaml. The unit tests for generate_FV3LAM_wflow.py make use of this file as an input config.yaml, so they are now failing due to this incorrect variable name. This wasn't caught because prior to #558 the unit tests were broken for a different reason. This change simply makes the appropriate rename, which should fix the failing unit test. Also created an f-string that was missed in a setup.py error message.
* Add the CCPP physics suite FV3_GFS_v17_p8 to the UFS SRW App. * Add a new WE2E test for the new suite. * Remove Hera from the Jenkins pipeline (no account for EPIC currently on this machine). --------- Co-authored-by: Michael Lueken <[email protected]> Co-authored-by: chan-hoo <[email protected]>
To be consistent with the naming convention used for variables that specify per-task properties (e.g. PPN_RUN_FCST, WTIME_RUN_FCST, MAXTRIES_RUN_FCST), rename variables that store the task names (e.g. RUN_FCST_TN) so that the "TN" part is at the beginning, e.g. TN_RUN_FCST.
This PR adds a Tutorial chapter to the SRW Documentation. There are descriptions for 5 severe weather events and a full tutorial for the first one (2019061518). This PR also includes: * Updates to the Glossary * An introduction to SSH and scp data transfer, which will help users to download tutorial plots from HPC systems to their local system for viewing. * Minor fixes/updates in other chapters for spelling/grammar/accuracy. --------- Co-authored-by: gspetro <[email protected]> Co-authored-by: Michael Kavulich <[email protected]>
…d script and module files (#549) Currently, the Online-CMAQ for air quality modeling is separately managed in the 'online-cmaq' branch of the authoritative UFS SRW App repository. The main target of the current 'online-cmaq' branch is the implementation of AQM ver.7 and delivery to NCO in late March. Therefore, it has been developed and tested on WCOSS2 and Hera. The Online-CMAQ checks out the same ufs weather model as the 'develop' branch. However, since it couples another component 'AQM', the compile process is different from that of the 'develop' branch. To compile Online-CMAQ: For Hera - ./devbuild.sh -p=hera -a=ATMAQ For WCOSS2 - ./devbuild.sh -p=wcoss2 -a=ATMAQ --------- Co-authored-by: chan-hoo <[email protected]> Co-authored-by: chan-hoo <[email protected]> Co-authored-by: chan-hoo jeon <[email protected]> Co-authored-by: Michael Lueken <[email protected]>
Adding correct logic for 3-digit forecast hours in UPP. The filenames coming out of UPP are unfortunately hard-coded in that all forecast hours below 100 will have a 2-digit forecast hour in the filename, and beyond that will switch to a 3-digit forecast hour. The current workflow does not attempt to resolve this difference, resulting in UPP tasks failing after (despite the executable completing successfully). This change adds the correct logic to allow UPP tasks to succeed beyond 99 hours.
RRFS requires certain observation files and data products to be retrieved & staged before the data processing and assimilation tasks are run. This work introduces some small changes to the retrieve_data.py script to allow retrieval/staging of local (machine-hosted; e.g.: Jet, Hera, noaacloud) and general, remotely located (extending beyond nomads/aws) files, primarily in the relaxation of/introduction of new parse_args function arguments. Minor updates to the data_locations.yaml are included here (more will follow), including the addition of RAP bufr files and GSI-fix files from NOAA AWS S3 buckets. FV3GFS & RAP bufr data stream locations on Jet and Hera have been added to the respective ush/machine YAMLS. --------- Co-authored-by: ulmononian <[email protected]>
This PR adds the ensemble (GDAS and GEFS) datasets as options for the SRW App and is part of a bigger effort to merge in RRFS capabilities into the develop branch. Currently, the workflow ensemble tasks all use the same initial conditions. Because of the addition of the ensemble datasets, each ensemble task can now have its own initial conditions with the global ensemble number mapping to the SRW App ensemble number (i.e. GDAS mem002 corresponds to SRW App mem002). The appropriate logic was added to these files so that the GEFS and GDAS datasets can run with the workflow without breaking existing functionality. A number of changes were made in the retrieve_data.py script to handle the --members argument and GEFS filename list. A GEFS function was added to the exregional_get_extrn_mdl_files.sh file to merge the GEFS files and update the summary extrn_mdl_var_defns.sh script. --------- Co-authored-by: Edward Snyder <[email protected]> Co-authored-by: Michael Lueken <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update develop to head at ufs-community
Use this template to give a detailed message describing the change you want to make to the code.
You may delete any sections labeled "optional" and any instructions within .
If you are unclear on what should be written here, see https://github.com/wrf-model/WRF/wiki/Making-a-good-pull-request-message for some guidance and review the Code Contributor's Guide at https://github.com/ufs-community/ufs-srweather-app/wiki/Code-Manager's-Guide.
Code reviewers will assess the PR based on the criteria laid out in the Code Reviewer's Guide (https://github.com/ufs-community/ufs-srweather-app/wiki/Code-Manager's-Guide).
The title of this pull request should be a brief summary (ideally less than 100 characters) of the changes included in this PR. Please also include the branch to which this PR is being issued (e.g., "[develop]: Updated UFS_UTILS hash").
Use the "Preview" tab to see what your PR will look like when you hit "Create pull request"
--- Delete this line and those above before hitting "Create pull request" ---
DESCRIPTION OF CHANGES:
Type of change
TESTS CONDUCTED:
DEPENDENCIES:
DOCUMENTATION:
ISSUE:
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):