Updates feature/METplus_templates to current develop #1
Conversation
…nity#676) Refactors the creation of a Rocoto XML to use a very generic Jinja2 template that is flexible enough to meet the needs of various workflow configurations supported by SRW. Specifically, it allows for a completely arbitrary workflow to be created under SRW, which includes the addition of completely arbitrary tasks on top of the predefined ones. --------- Co-authored-by: Michael Kavulich <[email protected]>
…fs-community#690) When pipeline files are archived to s3 bucket, retrieving the file via a browser attempts to render/display files of known extensions. A browser doesn't generally understand what to do with a .log extension (e.g. build.log). For ease of use in the CI Dashboard, which is a static HTML page, the s3 archived build log needs a .txt extension (e.g. build.txt).
…templates Manually resolved conflicts; still need to run tests
…CORE_ENSMEM_NAME_OR_NULL" variable
…ormat - Fix f-string for utils.py error message
- Fix copy-paste errors in parm/workflow yamls - Update corrected variables for new names in exscripts
…er of tasks that succeeded and failed for failed experiments
…ication_only_vx test
…ve attempt to simplify things was the root of all my problems!
…n tests now pass on Hera!
| # forecast output on which vx will be run may be staged or generated by | ||
| # running an ensemble of forecasts with the weather model as part of the | ||
| # current SRW-App-generated experiment. | ||
| # |
There was a problem hiding this comment.
@mkavulich Since these switches are removed, how do we now turn deterministic and ensemble vx on/off?
There was a problem hiding this comment.
@mkavulich Looks like these are now determined by the taskgroups entry in the configuration file, right?
| VX_FCST_MODEL_NAME: '{{ nco.NET }}.{{ task_run_post.POST_OUTPUT_DOMAIN_NAME }}' | ||
| VX_FIELDS: [ "APCP", "REFC", "RETOP", "SFC", "UPA" ] |
There was a problem hiding this comment.
@mkavulich If I want to run vx on just a subset of these fields, how is that now done?
| VX_FCST_MODEL_NAME: '{{ nco.NET }}.{{ task_run_post.POST_OUTPUT_DOMAIN_NAME }}' | ||
| VX_FIELDS: [ "APCP", "REFC", "RETOP", "SFC", "UPA" ] | ||
| VX_APCP_ACCUMS_HH: [ "01", "03", "06", "24" ] |
There was a problem hiding this comment.
@mkavulich Similar question -- If I want to run APCP vx only for 01 hr and 24 hr, how do I do that?
| # | ||
| # VX_FCST_INPUT_BASEDIR: |
There was a problem hiding this comment.
@mkavulich Why remove "BASE" from the name? Like other variable names, BASEDIR is used to indicate that the forecast files are not directly under this but somewhere further down the directory tree.
| # | ||
| # VX_OUTPUT_BASEDIR: | ||
| # Top-level directory in which METplus will place its output. | ||
| # | ||
| VX_FCST_INPUT_BASEDIR: '{{ "$COMOUT/../.." if user.RUN_ENVIR == "nco" else ( workflow.EXPTDIR if ((workflow_switches.RUN_TASK_RUN_FCST and task_run_fcst.WRITE_DOPOST) or workflow_switches.RUN_TASK_RUN_POST) ) }}' |
There was a problem hiding this comment.
@mkavulich The if-statement that's removed here was to make sure that this variable gets set to EXPTDIR only if UPP is run to generate forecast files. Otherwise, it shouldn't be set to EXPTDIR. I guess the task switches aren't available here any more, so is this now done somewhere else, e.g. setup.py?
| # | ||
| # File name and path templates used in the verification tasks. | ||
| # | ||
| FCST_SUBDIR_TEMPLATE: '{{ "${NET}.{init?fmt=%Y%m%d?shift=-${time_lag}}/{init?fmt=%H?shift=-${time_lag}}" if user.RUN_ENVIR == "nco" else "{init?fmt=%Y%m%d%H?shift=-${time_lag}}${SLASH_ENSMEM_SUBDIR_OR_NULL}/postprd" }}{% raw %}{% endraw %}' | ||
| FCST_FN_TEMPLATE: '{{ "${NET}.t{init?fmt=%H?shift=-${time_lag}}z" }}{{ "${DOT_ENSMEM_OR_NULL}" if user.RUN_ENVIR == "nco" else "" }}{{ ".prslev.f{lead?fmt=%HHH?shift=${time_lag}}.${POST_OUTPUT_DOMAIN_NAME}.grib2" }}{% raw %}{% endraw %}' |
There was a problem hiding this comment.
@mkavulich In the definition of FCST_FN_TEMPLATE, why remove the if-statement that inserts the ensemble member number? It is needed for NCO mode (unless the NCO output file naming convention changed and I'm not aware of it).
| # | ||
| # File name and path templates used in the verification tasks. | ||
| # | ||
| FCST_SUBDIR_TEMPLATE: '{{ "${NET}.{init?fmt=%Y%m%d?shift=-${time_lag}}/{init?fmt=%H?shift=-${time_lag}}" if user.RUN_ENVIR == "nco" else "{init?fmt=%Y%m%d%H?shift=-${time_lag}}${SLASH_ENSMEM_SUBDIR_OR_NULL}/postprd" }}{% raw %}{% endraw %}' | ||
| FCST_FN_TEMPLATE: '{{ "${NET}.t{init?fmt=%H?shift=-${time_lag}}z" }}{{ "${DOT_ENSMEM_OR_NULL}" if user.RUN_ENVIR == "nco" else "" }}{{ ".prslev.f{lead?fmt=%HHH?shift=${time_lag}}.${POST_OUTPUT_DOMAIN_NAME}.grib2" }}{% raw %}{% endraw %}' | ||
| FCST_FN_METPROC_TEMPLATE: '{{ "${NET}.t{init?fmt=%H?}z" }}{{ "${DOT_ENSMEM_OR_NULL}" if user.RUN_ENVIR == "nco" else "" }}{{ ".prslev.f{lead?fmt=%HHH?}.${POST_OUTPUT_DOMAIN_NAME}_a${ACCUM_HH}h.nc" }}{% raw %}{% endraw %}' |
There was a problem hiding this comment.
@mkavulich Same question for FCST_FN_METPROC_TEMPLATE as for FCST_FN_TEMPLATE above: Why remove if-statement for NCO mode?
The changes merge in the changes from develop up to ufs-community@da34aac. I don't promise these changes are all correct, I may have missed some things and haven't gotten all tests to run successfully yet. As I mentioned on Slack, the "only_vx" tests need some strange-looking settings to disable some empty metatask dependencies due to a rocoto limitation, let me know if you have any questions about that.
The deterministic tasks in this merged branch work successfully. For ensemble tests, they mostly work correctly but run into a snag at the EnsembleStat step. The problem here seems to be that some tasks require the forecast files for a single ensemble member, while Ensemblestat need forecast files for all members which I did not account for when I made my initial merge changes. I am sure there is a relatively simple way to solve this I just haven't gotten my head fully wrapped around it yet.
Depending on how this all works out this may require a change to the staged data for "VX_ONLY" tests (although now that I think about it, maybe not, given that forecast files are still in their previous places, it's just a matter of getting MET to look in the right place).
Again, it is entirely possible I've made some incorrect merge conflict resolutions, since I'm still not 100% familiar with all the MET capabilities. Let me know if you have questions.