diff --git a/docs/user_guide/config_options.md b/docs/user_guide/config_options.md index b0d43a31..bbee64a1 100644 --- a/docs/user_guide/config_options.md +++ b/docs/user_guide/config_options.md @@ -1,9 +1,39 @@ # config.yaml options -!!! note +!!! note "Required options" All keys listed here are required unless stated otherwise. -## Technical details +The different running modes of `benchcab` are solely dependent on the options used in `config.yaml`. The following gives some typical ways to configure `benchcab` for each mode, but the tool is not restricted to these choices of options: + +=== "Regression test" + + For this test, you want to: + + * specify the details of 2 branches of CABLE + * do not specify a `patch` + * use the default set of science options, i.e. do not specify science options in `config.yaml` + * choose the `experiment` suitable for your stage of development. A run with the `forty-two-site-test` will be required for submissions of new development to CABLE. + +=== "New feature test" + + For this test, you want to: + + * specify the details of 2 branches of CABLE + * specify a `patch` for **one** of the branches + * use the default set of science options, i.e. do not specify science options in `config.yaml` + * choose the `experiment` suitable for your stage of development. A run with the `forty-two-site-test` will be required for submissions of new development to CABLE. + + +=== "Ensemble mode" + + This running mode is quite open to customisations: + + * specify the number of CABLE's branches you need + * use `patch` on branches as required + * specify the science configurations you want to run. `patch` will be applied on top of the science configurations listed. + + +## Technical options `user` @@ -17,11 +47,11 @@ : NCI modules to use for compiling CABLE -## Simulations details +## Simulations options `realisations` -: Entries for each of the two CABLE branches to use (specified by keys `0` and `1`). Each entry contains the following keys: +: Entries for each CABLE branch to use. Each entry is a dictionary, {}, that contains the following keys: `name` : The base name of the branch on SVN, i.e. relative to: @@ -30,16 +60,19 @@ - `https://trac.nci.org.au/svn/cable/branches/Share` for a shared branch - `https://trac.nci.org.au/svn/cable/branches/Users/{user_id}` for a user branch - `revision` - : The revision number to use for the branch. - : This key is **optional** and can be omitted from the config file. By default `revision` is set to `-1` which indicates the HEAD of the branch to be used. The user may also explicitly specify `-1` to use the HEAD of the branch. - `trunk` : Boolean value set to `True` if this branch is the trunk for CABLE. Else set to `False`. `share_branch` : Boolean value set to `True` if this branch is under `branches/Share` in the CABLE SVN repository. Else set to `False`. + `build_script` + : This key is **optional**. The path to a custom script to build the code in that branch, relative to the name of the branch. E.g: "offline/build.sh" to specify a build script under /offline/. The script specified with this option will run as is, ignoring the entries in the `modules` key of `config.yaml` file. + + `revision` + : The revision number to use for the branch. + : This key is **optional** and can be omitted from the config file. By default `revision` is set to `-1` which indicates the HEAD of the branch to be used. The user may also explicitly specify `-1` to use the HEAD of the branch. + `patch` : Branch-specific namelist settings for `cable.nml`. Settings specified in `patch` get "patched" to the base namelist settings used for both branches. Any namelist settings specified here will overwrite settings defined in the default namelist file and in the science configurations. This means these settings will be set as stipulated in the `patch` for this branch for all science configurations run by `benchcab`. : The `patch` key must be a dictionary like data structure that is compliant with the [`f90nml`][f90nml-github] python package. @@ -47,14 +80,14 @@ Example: ```yaml - realisations: { - 0: { # head of the trunk + realisations: [ + { # head of the trunk name: "trunk", revision: -1, trunk: True, share_branch: False, }, - 1: { # some development branch + { # some development branch name: "test-branch", revision: -1, trunk: False, @@ -67,7 +100,7 @@ } } } - } + ] ``` `experiment` @@ -84,7 +117,7 @@ `science_configurations` -: User defined science configurations. This key is **optional** and can be omitted from the config file. Science configurations that are specified here will replace the default science configurations. +: User defined science configurations. This key is **optional** and can be omitted from the config file. Science configurations that are specified here will replace [the default science configurations](default_science_configurations.md). : Example: ```yaml science_configurations: { @@ -107,48 +140,6 @@ science_configurations: { } ``` -: Currently, the default science configurations are defined internally by the following data structure: -```python -DEFAULT_SCIENCE_CONFIGURATIONS = { - "sci0": {"cable": {"cable_user": {"GS_SWITCH": "medlyn"}}}, - "sci1": {"cable": {"cable_user": {"GS_SWITCH": "leuning"}}}, - "sci2": {"cable": {"cable_user": {"FWSOIL_SWITCH": "Haverd2013"}}}, - "sci3": {"cable": {"cable_user": {"FWSOIL_SWITCH": "standard"}}}, - "sci4": { - "cable": { - "cable_user": { - "GS_SWITCH": "medlyn", - "FWSOIL_SWITCH": "Haverd2013", - } - } - }, - "sci5": { - "cable": { - "cable_user": { - "GS_SWITCH": "leuning", - "FWSOIL_SWITCH": "Haverd2013", - } - } - }, - "sci6": { - "cable": { - "cable_user": { - "GS_SWITCH": "medlyn", - "FWSOIL_SWITCH": "standard", - } - } - }, - "sci7": { - "cable": { - "cable_user": { - "GS_SWITCH": "leuning", - "FWSOIL_SWITCH": "standard", - } - } - }, -} -``` - [meorg]: https://modelevaluation.org/ [forty-two-me]: https://modelevaluation.org/experiment/display/urTKSXEsojdvEPwdR [five-me]: https://modelevaluation.org/experiment/display/xNZx2hSvn4PMKAa9R diff --git a/docs/user_guide/default_science_configurations.md b/docs/user_guide/default_science_configurations.md new file mode 100644 index 00000000..8c7ece1a --- /dev/null +++ b/docs/user_guide/default_science_configurations.md @@ -0,0 +1,47 @@ +# Default science configurations + +A set of science configurations is defined in `benchcab`. This allows for running standardised testing of different version of CABLE. Test results using this default set are required for submission of new code development in CABLE. Occasionally, some code developments might also require test results using a different set of configurations to document their effect on the model results. + +The science configurations are given as patches to apply to a default namelist file. You can find [the default namelist file](https://github.com/CABLE-LSM/bench_example/blob/dev/namelists/cable.nml) in the `bench_example` repository. + +Currently, the default science configurations are defined internally by the following data structure: +```python +DEFAULT_SCIENCE_CONFIGURATIONS = { + "sci0": {"cable": {"cable_user": {"GS_SWITCH": "medlyn"}}}, + "sci1": {"cable": {"cable_user": {"GS_SWITCH": "leuning"}}}, + "sci2": {"cable": {"cable_user": {"FWSOIL_SWITCH": "Haverd2013"}}}, + "sci3": {"cable": {"cable_user": {"FWSOIL_SWITCH": "standard"}}}, + "sci4": { + "cable": { + "cable_user": { + "GS_SWITCH": "medlyn", + "FWSOIL_SWITCH": "Haverd2013", + } + } + }, + "sci5": { + "cable": { + "cable_user": { + "GS_SWITCH": "leuning", + "FWSOIL_SWITCH": "Haverd2013", + } + } + }, + "sci6": { + "cable": { + "cable_user": { + "GS_SWITCH": "medlyn", + "FWSOIL_SWITCH": "standard", + } + } + }, + "sci7": { + "cable": { + "cable_user": { + "GS_SWITCH": "leuning", + "FWSOIL_SWITCH": "standard", + } + } + }, +} +``` \ No newline at end of file diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md index e41e132f..a8e0d68e 100644 --- a/docs/user_guide/index.md +++ b/docs/user_guide/index.md @@ -8,6 +8,11 @@ In this guide, we will describe: `benchcab` has been designed to work on NCI machine exclusively. It might be extended later on to other systems. +!!! warning "Limitations" + Currently, + + * `benchcab` can only run simulations at flux sites. + * spin-up for CASA simulations are not supported. ## Pre-requisites To use `benchcab`, you need to join the following projects at NCI: @@ -32,7 +37,13 @@ You need to load the module on each new session at NCI on login or compute nodes ## Usage -`benchcab` will run the exact same configurations on two CABLE branches specified by the user, e.g. a user branch (with personal changes) against the head of the trunk. The results should be attached with all new [tickets](https://trac.nci.org.au/trac/cable/report/1). +`benchcab` allows you to run an ensemble of configurations for CABLE using any number of code versions. `benchcab` can be used along 3 major modes: + +* *regression test:* running 2 versions of CABLE wih the same standard set of science configurations. +* *new feature:* running 2 versions of CABLE with the same standard set of science configurations except one version is patched to use a new feature. +* *ensemble run:* running any number of versions of CABLE with the same set of customised science configurations. + +The regression and new feature run modes should be used as necessary when evaluating new development in CABLE. The code will: (i) check out and (ii) build the code branches. Then it will run each executable across N standard science configurations for a given number of sites. It is possible to produce some plots locally from the output produced. But [the modelevaluation website](https://modelevaluation.org/) can be used for further benchmarking and evaluation. @@ -55,6 +66,7 @@ git clone git@github.com:CABLE-LSM/bench_example.git Once the work directory is cloned, you will need to adapt the `config.yaml` file to your case. Refer to [the description of the options][config_options] for this file. + ## Run the simulations Change directory into the cloned example work directory @@ -63,20 +75,20 @@ cd bench_example ``` !!! warning - `benchcab` will yell at you if it cannot find files in the current working directory. + `benchcab` will stop if it is not run within a work directory with the proper structure. Currently, `benchcab` can only run CABLE for flux sites. To run the flux site tests, run ```bash -benchcab -f +benchcab run ``` -The benchmarking will follow the steps: +The tool will follow the steps: -1. Checkout both branches. The codes will be stored under `src/` directory in your work directory. The sub-directories are created automatically. -2. Compile the source code from both branches -3. Setup and launch a PBS job to run the simulations in parallel. When `benchcab` launches the PBS job, it will print out the job ID to the terminal. You can check the status of the job with `qstat`. +1. Checkout the code branches. The codes will be stored under `src/` directory in your work directory. The sub-directories are created automatically. +2. Compile the source code from all branches +3. Setup and launch a PBS job to run the simulations in parallel. When `benchcab` launches the PBS job, it will print out the job ID to the terminal. You can check the status of the job with `qstat`. `benchcab` will not warn you when the simulations are over. For help on the available options for `benchcab`: @@ -84,9 +96,12 @@ For help on the available options for `benchcab`: benchcab -h ``` +!!! Tip + It is possible to run each step of the workflow separately using sub-commands for `benchcab`. Refer to the help message to learn more. + ## Directory structure and files -The following files and directories are created when `benchcab -f` executes successfully: +The following files and directories are created when `benchcab run` executes successfully: ``` . ├── benchmark_cable_qsub.sh @@ -125,7 +140,7 @@ The `runs/site` directory contains the log files, output files, and tasks for ru ``` where `met_file_base_name` is the base file name of the meteorological forcing file in the FLUXNET dataset, `realisation_key` is the branch key specified in the config file, and `science_config_key` identifies the science configuration used. -The `runs/site/tasks/` directory contains the executable and input files for each task. +The `runs/site/tasks/` directory contains the executable, the input files for each task and the recorded standard output from the CABLE model run. The output files and log files for all tasks are stored in the `runs/site/outputs` and `runs/site/logs` directories respectively. diff --git a/mkdocs.yml b/mkdocs.yml index 6a89f49a..c776e3ad 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,6 +1,6 @@ site_name: benchcab -# Needed to generate the canonical_url for each pages that are +# Needed to generate the canonical_url for each pages that are # then used to create the sitemap for search engine indexing site_url: https://cable-lsm.github.io/benchcab/ # Git repository @@ -57,16 +57,15 @@ markdown_extensions: emoji_index: !!python/name:materialx.emoji.twemoji emoji_generator: !!python/name:materialx.emoji.to_svg - extra_javascript: - javascripts/mathjax.js - https://polyfill.io/v3/polyfill.min.js?features=es6 - https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js - + # Navigation nav: - About: index.md - User Guide: - - user_guide/index.md - - user_guide/config_options.md - \ No newline at end of file + - user_guide/index.md + - user_guide/config_options.md + - user_guide/default_science_configurations.md