CMORizer Australian Gridded Climate Data(AGCD) precipitation (#3445)

Co-authored-by: Valeriu Predoi <[email protected]> Co-authored-by: Romain Beucher <[email protected]> Co-authored-by: Axel Lauer <[email protected]>
ESMValGroup · Feb 20, 2024 · 4a659bb · 4a659bb
1 parent 2cc1560
commit 4a659bb
Show file tree

Hide file tree

Showing 7 changed files with 200 additions and 1 deletion.
diff --git a/doc/sphinx/source/input.rst b/doc/sphinx/source/input.rst
@@ -232,6 +232,8 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | Dataset                      | Variables (MIP)                                                                                      | Tier | Script language |
 +==============================+======================================================================================================+======+=================+
+| AGCD                         | pr (Amon)                                                                                            |   2  | Python          |
++------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | APHRO-MA                     | pr, tas (day), pr, tas (Amon)                                                                        |   3  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | AURA-TES                     | tro3 (Amon)                                                                                          |   3  | NCL             |

diff --git a/esmvaltool/cmorizers/data/cmor_config/AGCD.yml b/esmvaltool/cmorizers/data/cmor_config/AGCD.yml
@@ -0,0 +1,36 @@
+---
+# filename: 'agcd_v1-0-1_precip_total_r005_monthly_.*.nc'
+filename: 'agcd_{version}_{variable}_{raw_calc}_r005_{freq}_.*.nc'
+
+attributes:
+  project_id: OBS6
+  dataset_id: AGCD
+  version: 'v2-0-1'
+  tier: 2
+  modeling_realm: ground
+  resolution: '005'  # '001' available for v2
+  source: 'https://dx.doi.org/10.25914/rses-zh67 Australian Bureau of Meteorology (2023),
+           \Australian Gridded Climate Data ( AGCD ) ; v2.0.1 Snapshot (1900-01-01 to 2022-12-31)'
+  reference: 'agcd-v201'
+  comment: 'hosted on NCI (National Computing Infrastructure Australia)'
+
+variables:
+  pr:
+    mip: Amon
+    raw_long: Lwe Thickness Of Precipitation Amount (mm)
+    raw_calc: total
+    freq: monthly  # convert daily as well, v1s only
+    raw: precip
+
+## variables in AGCD v1
+  # tasmax:
+  #   mip: Amon
+  #   raw_long: Daily maximum air temperature, degrees_Celsius, monthly, mean
+  #   raw_calc: mean
+  #   freq: monthly
+  #   raw: tmax
+  # tasmin:
+  #   mip: Amon
+  #   raw_calc: mean
+  #   freq: monthly
+  #   raw: tmin
diff --git a/esmvaltool/cmorizers/data/datasets.yml b/esmvaltool/cmorizers/data/datasets.yml
@@ -1,6 +1,25 @@
 # Dataset information
 ---
 datasets:
+  AGCD:
+    tier: 2
+    source: "http://dx.doi.org/10.25914/6009600786063"
+    last_access: 2023-11-21
+    info: |
+      Australian Gridded Climate Data (AGCD) version 2 is the Bureau of Meteorology's official dataset for climate
+      analyses covering analysis of monthly rainfall. The dataset provides consistent temporal and spatial analyses
+      across Australia for each observed data variable. This accounts for spatial and temporal gaps in observations.
+      Where possible, the gridded analysis techniques provide useful estimates in data-sparse regions
+      such as central Australia. 
+      
+      Time coverage: Site-based data are used to provide gridded climate data at the monthly timescale for rainfall (1900+).
+      Reference: Evans, A., Jones, D.A., Smalley, R., and Lellyett, S. 2020. An enhanced gridded rainfall analysis scheme
+      for Australia. Bureau of Meteorology Research Report. No. 41.
+      National Computational Infrastructure (NCI) - Catalogue Record: http://dx.doi.org/10.25914/6009600786063.
+      Data from NCI (National Computing Infrastructure Australia https://nci.org.au/), 
+      requires an NCI account and access to Gadi(Supercomputer in Canberra) and the project found in catalogue record.
+      Access can be requested through NCI. NCI is an ESGF node (https://esgf.nci.org.au/projects/esgf-nci/)
+
   APHRO-MA:
     tier: 3
     source: "http://aphrodite.st.hirosaki-u.ac.jp/download/"

diff --git a/esmvaltool/cmorizers/data/formatters/datasets/agcd.py b/esmvaltool/cmorizers/data/formatters/datasets/agcd.py
@@ -0,0 +1,122 @@
+"""ESMValTool CMORizer for AGCD data.
+
+Tier
+    Tier 2: other freely available dataset.
+
+Source
+    https://dx.doi.org/10.25914/rses-zh67
+
+Last access
+    20231121
+
+Download and processing instructions
+    Data from NCI (National Computing Infrastructure Australia)
+    https://nci.org.au/,
+    requiring an NCI account and access to Gadi(Supercomputer in Australia)
+      and the dataset project found in
+      catalogue record https://dx.doi.org/10.25914/rses-zh67.
+    Access can be requested through NCI.
+    NCI is an ESGF node: (https://esgf.nci.org.au/projects/esgf-nci/)
+    Processing is done on Gadi.
+
+"""
+import logging
+import os
+import re
+
+import iris
+
+from esmvalcore.cmor._fixes.shared import get_time_bounds
+from esmvaltool.cmorizers.data import utilities as utils
+
+logger = logging.getLogger(__name__)
+
+
+def _get_filepaths(in_dir, basename):
+    """Find correct name of file (extend basename with timestamp)."""
+    regex = re.compile(basename)
+    return_files = []
+    for root, _, files in os.walk(in_dir, followlinks=True):
+
+        for filename in files:
+            if regex.match(filename):
+                return_files.append(os.path.join(root, filename))
+
+    return return_files
+
+
+def fix_data_var(cube, var):
+    """Convert units in cube for the variable."""
+    monthdays = {1: 31, 2: 28, 3: 31, 4: 30, 5: 31, 6: 30,
+                 7: 31, 8: 31, 9: 30, 10: 31, 11: 30, 12: 31}
+    if var == 'pr':
+        newcubels = []
+        for i, m_cube in enumerate(cube.slices(['latitude', 'longitude'])):
+            m_cube = m_cube / (monthdays[i + 1] * 86400)  # days in month
+            newcubels.append(m_cube)
+
+        cube = iris.cube.CubeList(newcubels).merge()[0]
+        cube.units = 'kg m-2 s-1'
+
+    elif var in ['tas', 'tasmin', 'tasmax']:  # other variables in v1
+        cube = cube + 273.15
+        cube.units = 'K'
+        utils.add_height2m(cube)
+
+    else:
+        logger.info("Variable %s not converted", var)
+
+    return cube
+
+
+def _extract_variable(cmor_info, attrs, filepath, out_dir):
+    """Extract variable."""
+    var = cmor_info.short_name
+    logger.info("Var is %s", var)
+    cubes = iris.load(filepath)
+    for cube in cubes:
+
+        cube = fix_data_var(cube, var)
+
+        utils.fix_var_metadata(cube, cmor_info)
+
+        utils.fix_coords(cube)
+        bounds = get_time_bounds(cube.coords('time')[0], 'mon')
+        cube.coords('time')[0].bounds = bounds
+        utils.set_global_atts(cube, attrs)
+
+        logger.info("Saving file")
+        utils.save_variable(cube,
+                            var,
+                            out_dir,
+                            attrs,
+                            unlimited_dimensions=['time'])
+
+
+def cmorization(in_dir, out_dir, cfg, cfg_user, start_date, end_date):
+    """Cmorization func call."""
+    glob_attrs = cfg['attributes']
+    cmor_table = cfg['cmor_table']
+
+    ver = cfg['attributes']['version']
+
+    # Run the cmorization #multiple variables
+    for (var, var_info) in cfg['variables'].items():
+
+        glob_attrs['mip'] = var_info['mip']
+        logger.info("CMORizing variable '%s', %s", var, var_info['mip'])
+
+        raw_filename = cfg['filename'].format(version=ver,
+                                              variable=var_info['raw'],
+                                              raw_calc=var_info['raw_calc'],
+                                              freq=var_info['freq'])
+        filepaths = _get_filepaths(in_dir, raw_filename)
+
+        if not filepaths:
+            logger.info("no files for %s. pattern:%s", var, raw_filename)
+            logger.info("directory:%s", in_dir)
+        for inputfile in filepaths:
+            logger.info("Found input file '%s'", inputfile)
+
+            cmor_info = cmor_table.get_variable(var_info['mip'], var)
+            _extract_variable(cmor_info, glob_attrs, inputfile, out_dir)
diff --git a/esmvaltool/recipes/examples/recipe_check_obs.yml b/esmvaltool/recipes/examples/recipe_check_obs.yml
@@ -18,6 +18,16 @@ diagnostics:
 
   ### TIER 2 ##################################################################
 
+  AGCD:
+    description:
+    variables:
+      pr:
+    additional_datasets:
+      - {project: OBS6, dataset: AGCD, mip: Amon, tier: 3, 
+        type: ground, version: v2-0-1}
+    scripts: null
+
+
   BerkeleyEarth:
     description: BerkeleyEarth check
     variables:
@@ -1931,4 +1941,4 @@ diagnostics:
     additional_datasets:
       - {dataset: UWisc, project: OBS, mip: Amon, tier: 3,
          type: sat, version: v2, start_year: 1988, end_year: 2007}
-    scripts: null
+    scripts: null
diff --git a/esmvaltool/references/agcd-v201.bibtex b/esmvaltool/references/agcd-v201.bibtex
@@ -0,0 +1,9 @@
+@misc{agcd-v201,
+  doi = {10.25914/RSES-ZH67},
+  url = {https://pid.nci.org.au/doi/f5999_1125_5714_7440},
+  author = {{Bureau Of Meteorology}},
+  language = {en},
+  title = {Australian Gridded Climate Data (AGCD) v2.0.1},
+  publisher = {NCI Australia},
+  year = {2023}
+}
diff --git a/tests/unit/cmorizers/test_cmorization_interface.py b/tests/unit/cmorizers/test_cmorization_interface.py
@@ -29,6 +29,7 @@ def test_formatters_have_required_interface():
         except AssertionError:
             print(f'Bad args in {os.path.join(formatters_folder, formatter)}: '
                   f'{spec.args}')
+            print(f"Expected {arg_names}.")
             error = True
     assert not error