diff --git a/docs/user_guide/config_options.md b/docs/user_guide/config_options.md index b0d43a31..80364ce1 100644 --- a/docs/user_guide/config_options.md +++ b/docs/user_guide/config_options.md @@ -1,76 +1,116 @@ # config.yaml options -!!! note +!!! note "Required options" All keys listed here are required unless stated otherwise. -## Technical details +The different running modes of `benchcab` are solely dependent on the options used in `config.yaml`. The following gives some typical ways to configure `benchcab` for each mode, but the tool is not restricted to these choices of options: -`user` +=== "Regression test" + + For this test, you want to: + + * Specify the details of two branches of CABLE + * Do not specify a [`patch`](#`patch`) + * Use the default set of science options, i.e. do not specify [`science_configurations`](#`science_configurations`) in `config.yaml` + * Choose the [`experiment`](#`experiment`) suitable for your stage of development. A run with the `forty-two-site-test` will be required for submissions of new development to CABLE. + +=== "New feature test" + + For this test, you want to: + + * Specify the details of two branches of CABLE + * Specify a [`patch`](#`patch`) for **one** of the branches + * Use the default set of science options, i.e. do not specify [`science_configurations`](#`science_configurations`) in `config.yaml` + * Choose the [`experiment`](#`experiment`) suitable for your stage of development. A run with the `forty-two-site-test` will be required for submissions of new development to CABLE. + + +=== "Ensemble mode" + + This running mode is quite open to customisations: + + * Specify the number of CABLE's branches you need + * Use [`patch`](#`patch`) on branches as required + * Specify the [science configurations](#`science_configurations`) you want to run. [`patch`](#`patch`) will be applied on top of the science configurations listed. + + +## Technical options + +### `user` : NCI user ID to find the CABLE branch on SVN and run the simulations. -`project` +### `project` : NCI project ID to charge the simulations to. -`modules` +### `modules` : NCI modules to use for compiling CABLE -## Simulations details - -`realisations` - -: Entries for each of the two CABLE branches to use (specified by keys `0` and `1`). Each entry contains the following keys: - - `name` - : The base name of the branch on SVN, i.e. relative to: - - - `https://trac.nci.org.au/svn/cable` for the trunk - - `https://trac.nci.org.au/svn/cable/branches/Share` for a shared branch - - `https://trac.nci.org.au/svn/cable/branches/Users/{user_id}` for a user branch - - `revision` - : The revision number to use for the branch. - : This key is **optional** and can be omitted from the config file. By default `revision` is set to `-1` which indicates the HEAD of the branch to be used. The user may also explicitly specify `-1` to use the HEAD of the branch. - - `trunk` - : Boolean value set to `True` if this branch is the trunk for CABLE. Else set to `False`. - - `share_branch` - : Boolean value set to `True` if this branch is under `branches/Share` in the CABLE SVN repository. Else set to `False`. - - `patch` - : Branch-specific namelist settings for `cable.nml`. Settings specified in `patch` get "patched" to the base namelist settings used for both branches. Any namelist settings specified here will overwrite settings defined in the default namelist file and in the science configurations. This means these settings will be set as stipulated in the `patch` for this branch for all science configurations run by `benchcab`. - : The `patch` key must be a dictionary like data structure that is compliant with the [`f90nml`][f90nml-github] python package. - : This key is **optional** and can be omitted from the config file. By default `patch` is empty and does not modify the namelist file. - - Example: - ```yaml - realisations: { - 0: { # head of the trunk - name: "trunk", - revision: -1, - trunk: True, - share_branch: False, - }, - 1: { # some development branch - name: "test-branch", - revision: -1, - trunk: False, - share_branch: False, - patch: { - cable: { - cable_user: { - FWSOIL_SWITCH: "Lai and Ktaul 2000" - } - } +## Simulations options + +### `realisations` + +: Entries for each CABLE branch to use. Each entry is a dictionary, `{}`, that contains the following keys: + +#### `name` + +: The base name of the branch on SVN, i.e. relative to: + + - `https://trac.nci.org.au/svn/cable` for the trunk + - `https://trac.nci.org.au/svn/cable/branches/Share` for a shared branch + - `https://trac.nci.org.au/svn/cable/branches/Users/{user_id}` for a user branch + +#### `trunk` + +: Boolean value set to `True` if this branch is the trunk for CABLE. Else set to `False`. + +#### `share_branch` + +: Boolean value set to `True` if this branch is under `branches/Share` in the CABLE SVN repository. Else set to `False`. + +#### `build_script` + +: This key is **optional**. The path to a custom script to build the code in that branch, relative to the name of the branch. The script specified with this option will run as is, ignoring the entries in the `modules` key of `config.yaml` file. +: Example: `build_script: offline/build.sh` to specify a build script under `/offline/`. + +#### `revision` + +: The revision number to use for the branch. +: This key is **optional** and can be omitted from the config file. By default `revision` is set to `-1` which indicates the HEAD of the branch to be used. The user may also explicitly specify `-1` to use the HEAD of the branch. + +#### `patch` + +: Branch-specific namelist settings for `cable.nml`. Settings specified in `patch` get "patched" to the base namelist settings used for both branches. Any namelist settings specified here will overwrite settings defined in the default namelist file and in the science configurations. This means these settings will be set as stipulated in the `patch` for this branch for all science configurations run by `benchcab`. +: The `patch` key must be a dictionary like data structure that is compliant with the [`f90nml`][f90nml-github] python package. +: This key is **optional** and can be omitted from the config file. By default `patch` is empty and does not modify the namelist file. + +Example: +```yaml +realisations: [ + { # head of the trunk + name: "trunk", + revision: -1, + trunk: True, + share_branch: False, + }, + { # some development branch + name: "test-branch", + revision: -1, + trunk: False, + share_branch: False, + patch: { + cable: { + cable_user: { + FWSOIL_SWITCH: "Lai and Ktaul 2000" } } } - ``` + } +] +``` -`experiment` +### `experiment` : Type of experiment to run. Experiments are defined in the **NRI Land testing** workspace on [modelevaluation.org][meorg]. This key specifies the met forcing files used in the test suite. To choose from: @@ -82,9 +122,9 @@ - `US-Var`: to run simulations at the Vaira Ranch-Ione (US) site - `US-Whs`: to run simulations at the Walnut Gulch Lucky Hills Shrub (US) site -`science_configurations` +### `science_configurations` -: User defined science configurations. This key is **optional** and can be omitted from the config file. Science configurations that are specified here will replace the default science configurations. +: User defined science configurations. This key is **optional** and can be omitted from the config file. Science configurations that are specified here will replace [the default science configurations](default_science_configurations.md). : Example: ```yaml science_configurations: { @@ -107,48 +147,6 @@ science_configurations: { } ``` -: Currently, the default science configurations are defined internally by the following data structure: -```python -DEFAULT_SCIENCE_CONFIGURATIONS = { - "sci0": {"cable": {"cable_user": {"GS_SWITCH": "medlyn"}}}, - "sci1": {"cable": {"cable_user": {"GS_SWITCH": "leuning"}}}, - "sci2": {"cable": {"cable_user": {"FWSOIL_SWITCH": "Haverd2013"}}}, - "sci3": {"cable": {"cable_user": {"FWSOIL_SWITCH": "standard"}}}, - "sci4": { - "cable": { - "cable_user": { - "GS_SWITCH": "medlyn", - "FWSOIL_SWITCH": "Haverd2013", - } - } - }, - "sci5": { - "cable": { - "cable_user": { - "GS_SWITCH": "leuning", - "FWSOIL_SWITCH": "Haverd2013", - } - } - }, - "sci6": { - "cable": { - "cable_user": { - "GS_SWITCH": "medlyn", - "FWSOIL_SWITCH": "standard", - } - } - }, - "sci7": { - "cable": { - "cable_user": { - "GS_SWITCH": "leuning", - "FWSOIL_SWITCH": "standard", - } - } - }, -} -``` - [meorg]: https://modelevaluation.org/ [forty-two-me]: https://modelevaluation.org/experiment/display/urTKSXEsojdvEPwdR [five-me]: https://modelevaluation.org/experiment/display/xNZx2hSvn4PMKAa9R diff --git a/docs/user_guide/default_science_configurations.md b/docs/user_guide/default_science_configurations.md new file mode 100644 index 00000000..8c7ece1a --- /dev/null +++ b/docs/user_guide/default_science_configurations.md @@ -0,0 +1,47 @@ +# Default science configurations + +A set of science configurations is defined in `benchcab`. This allows for running standardised testing of different version of CABLE. Test results using this default set are required for submission of new code development in CABLE. Occasionally, some code developments might also require test results using a different set of configurations to document their effect on the model results. + +The science configurations are given as patches to apply to a default namelist file. You can find [the default namelist file](https://github.com/CABLE-LSM/bench_example/blob/dev/namelists/cable.nml) in the `bench_example` repository. + +Currently, the default science configurations are defined internally by the following data structure: +```python +DEFAULT_SCIENCE_CONFIGURATIONS = { + "sci0": {"cable": {"cable_user": {"GS_SWITCH": "medlyn"}}}, + "sci1": {"cable": {"cable_user": {"GS_SWITCH": "leuning"}}}, + "sci2": {"cable": {"cable_user": {"FWSOIL_SWITCH": "Haverd2013"}}}, + "sci3": {"cable": {"cable_user": {"FWSOIL_SWITCH": "standard"}}}, + "sci4": { + "cable": { + "cable_user": { + "GS_SWITCH": "medlyn", + "FWSOIL_SWITCH": "Haverd2013", + } + } + }, + "sci5": { + "cable": { + "cable_user": { + "GS_SWITCH": "leuning", + "FWSOIL_SWITCH": "Haverd2013", + } + } + }, + "sci6": { + "cable": { + "cable_user": { + "GS_SWITCH": "medlyn", + "FWSOIL_SWITCH": "standard", + } + } + }, + "sci7": { + "cable": { + "cable_user": { + "GS_SWITCH": "leuning", + "FWSOIL_SWITCH": "standard", + } + } + }, +} +``` \ No newline at end of file diff --git a/docs/user_guide/expected_output.md b/docs/user_guide/expected_output.md new file mode 100644 index 00000000..069fb8b1 --- /dev/null +++ b/docs/user_guide/expected_output.md @@ -0,0 +1,37 @@ +# Expected output from benchcab + +Below you will find examples of the expected output printed by `benchcab` to the screen when running the full workflow, with `benchcab run`. + +Other sub-commands should print out part of this output. + +``` +$ benchcab run +Creating src directory: /scratch/tm70/sb8430/bench_example/src +Checking out repositories... +Successfully checked out trunk at revision 9550 +Successfully checked out test-branch at revision 9550 +Successfully checked out CABLE-AUX at revision 9550 +Writing revision number info to rev_number-1.log +Compiling CABLE serially for realisation trunk... +Successfully compiled CABLE for realisation trunk +Compiling CABLE serially for realisation test-branch... +Successfully compiled CABLE for realisation test-branch +Setting up run directory tree for FLUXNET tests... +Creating runs/site/logs directory: /scratch/tm70/sb8430/bench_example/runs/site/logs +Creating runs/site/outputs directory: /scratch/tm70/sb8430/bench_example/runs/site/outputs +Creating runs/site/tasks directory: /scratch/tm70/sb8430/bench_example/runs/site/tasks +Creating task directories... +Setting up tasks... +Successfully setup FLUXNET tasks +Creating PBS job script to run FLUXNET tasks on compute nodes: benchmark_cable_qsub.sh +PBS job submitted: 82479088.gadi-pbs +The CABLE log file for each task is written to runs/site/logs/_log.txt +The CABLE standard output for each task is written to runs/site/tasks//out.txt +The NetCDF output for each task is written to runs/site/outputs/_out.nc +``` + +The PBS schedule job should print out the following to the job log file: +``` +Running FLUXNET tasks... +Successfully ran FLUXNET tasks +``` diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md index e41e132f..e4773cf5 100644 --- a/docs/user_guide/index.md +++ b/docs/user_guide/index.md @@ -8,6 +8,11 @@ In this guide, we will describe: `benchcab` has been designed to work on NCI machine exclusively. It might be extended later on to other systems. +!!! warning "Limitations" + Currently, + + * `benchcab` can only run simulations at flux sites. + * spin-up for CASA simulations are not supported. ## Pre-requisites To use `benchcab`, you need to join the following projects at NCI: @@ -32,7 +37,13 @@ You need to load the module on each new session at NCI on login or compute nodes ## Usage -`benchcab` will run the exact same configurations on two CABLE branches specified by the user, e.g. a user branch (with personal changes) against the head of the trunk. The results should be attached with all new [tickets](https://trac.nci.org.au/trac/cable/report/1). +`benchcab` allows you to run an ensemble of configurations for CABLE using any number of code versions. `benchcab` can be used along 3 major modes: + +* *Regression test:* running two versions of CABLE with the same standard set of science configurations. +* *New feature:* running two versions of CABLE with the same standard set of science configurations except one version is patched to use a new feature. +* *Ensemble run:* running any number of versions of CABLE with the same set of customised science configurations. + +The regression and new feature run modes should be used as necessary when evaluating new development in CABLE. The code will: (i) check out and (ii) build the code branches. Then it will run each executable across N standard science configurations for a given number of sites. It is possible to produce some plots locally from the output produced. But [the modelevaluation website](https://modelevaluation.org/) can be used for further benchmarking and evaluation. @@ -55,6 +66,7 @@ git clone git@github.com:CABLE-LSM/bench_example.git Once the work directory is cloned, you will need to adapt the `config.yaml` file to your case. Refer to [the description of the options][config_options] for this file. + ## Run the simulations Change directory into the cloned example work directory @@ -63,30 +75,37 @@ cd bench_example ``` !!! warning - `benchcab` will yell at you if it cannot find files in the current working directory. - + `benchcab` will stop if it is not run within a work directory with the proper structure. -Currently, `benchcab` can only run CABLE for flux sites. To run the flux site tests, run +Currently, `benchcab` can only run CABLE for flux sites. **To run the whole workflow**, run ```bash -benchcab -f +benchcab run ``` -The benchmarking will follow the steps: +The tool will follow the steps: -1. Checkout both branches. The codes will be stored under `src/` directory in your work directory. The sub-directories are created automatically. -2. Compile the source code from both branches -3. Setup and launch a PBS job to run the simulations in parallel. When `benchcab` launches the PBS job, it will print out the job ID to the terminal. You can check the status of the job with `qstat`. +1. Checkout the code branches. The codes will be stored under `src/` directory in your work directory. The sub-directories are created automatically. +2. Compile the source code from all branches +3. Setup and launch a PBS job to run the simulations in parallel. When `benchcab` launches the PBS job, it will print out the job ID to the terminal. You can check the status of the job with `qstat`. `benchcab` will not warn you when the simulations are over. -For help on the available options for `benchcab`: +!!! tip "Expected output" + + You can see [an example of the expected output](expected_output.md) printed out to the screen by `benchcab run` to check if the tool has worked as expected. + +For help on the **available options** for `benchcab`: ```bash benchcab -h +benchcab -h ``` +!!! Tip "Running parts of the workflow" + It is possible to run each step of the workflow separately using sub-commands for `benchcab`. Refer to the help message to learn more. + ## Directory structure and files -The following files and directories are created when `benchcab -f` executes successfully: +The following files and directories are created when `benchcab run` executes successfully: ``` . ├── benchmark_cable_qsub.sh @@ -112,22 +131,41 @@ The following files and directories are created when `benchcab -f` executes succ ├── └── ``` +`benchmark_cable_qsub.sh` + +: the job script submitted to run the test suite and `benchmark_cable_qsub.sh.o` contains the job's standard output/error stream. -The `benchmark_cable_qsub.sh` file is the job script submitted to run the test suite and `benchmark_cable_qsub.sh.o` contains the job's standard output/error stream. +`rev_number-*.log` -The `rev_number-*.log` file keeps a record of the revision numbers used for each realisation specified in the config file. +: file to keep a record of the revision numbers used for each realisation specified in the config file. -The `src` directory contains the source code checked out from SVN for each branch specified in the config file (labelled `realisation-*` above) and the CABLE-AUX branch. +`src/` -The `runs/site` directory contains the log files, output files, and tasks for running CABLE. CABLE runs are organised into tasks where a task consists of a branch (realisation), a meteorological forcing, and a science configuration. In the above directory structure, `` uses the following naming convention: +: directory that contains the source code checked out from SVN for each branch specified in the config file (labelled `realisation-*` above) and the CABLE-AUX branch. + +`runs/site/` + +: directory that contains the log files, output files, and tasks for running CABLE. + +`tasks` + +: CABLE runs are organised into tasks where a task consists of a branch (realisation), a meteorological forcing, and a science configuration. In the above directory structure, `` uses the following naming convention: ``` _R_S ``` -where `met_file_base_name` is the base file name of the meteorological forcing file in the FLUXNET dataset, `realisation_key` is the branch key specified in the config file, and `science_config_key` identifies the science configuration used. +: where `met_file_base_name` is the base file name of the meteorological forcing file in the FLUXNET dataset, `realisation_key` is the branch key specified in the config file, and `science_config_key` identifies the science configuration used. + +`runs/site/tasks//` + +: directory that contains the executable, the input files for each task and the recorded standard output from the CABLE model run. + +`runs/site/outputs/` + +: directory that contains the netCDF output files for all tasks -The `runs/site/tasks/` directory contains the executable and input files for each task. +`runs/site/logs/` -The output files and log files for all tasks are stored in the `runs/site/outputs` and `runs/site/logs` directories respectively. +: directory that contains the log files produced by all tasks !!! warning "Re-running `benchcab` multiple times in the same working directory" We recommend the user to manually delete the generated files when re-running `benchcab`. Re-running `benchcab` multiple times in the same working directory is currently not yet supported (see issue [CABLE-LSM/benchcab#20](https://github.com/CABLE-LSM/benchcab/issues/20)). To clean the current working directory, run the following command in the working directory diff --git a/mkdocs.yml b/mkdocs.yml index 6a89f49a..99916f2a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,6 +1,6 @@ site_name: benchcab -# Needed to generate the canonical_url for each pages that are +# Needed to generate the canonical_url for each pages that are # then used to create the sitemap for search engine indexing site_url: https://cable-lsm.github.io/benchcab/ # Git repository @@ -52,6 +52,9 @@ markdown_extensions: - pymdownx.tabbed: alternate_style: true - attr_list + - toc: + permalink: true + permalink_title: Link to this section for reference - md_in_html - pymdownx.emoji: emoji_index: !!python/name:materialx.emoji.twemoji @@ -62,11 +65,12 @@ extra_javascript: - javascripts/mathjax.js - https://polyfill.io/v3/polyfill.min.js?features=es6 - https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js - + # Navigation nav: - About: index.md - User Guide: - - user_guide/index.md - - user_guide/config_options.md - \ No newline at end of file + - user_guide/index.md + - user_guide/config_options.md + - user_guide/default_science_configurations.md + - user_guide/expected_output.md