Add TropFlux CMORiser (#3863)

Co-authored-by: Valeriu Predoi <[email protected]>
Co-authored-by: Felicity Chun <[email protected]>

Author: 3 people

doc/sphinx/source/input.rst

@@ -463,6 +463,10 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | TCOM-N2O                     | n2o (Amon [#note3]_)                                                                                 |   2  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
+| TROPFLUX                     | ts, rlut, rsds, tauu, tauv, hfds (Amon)                                                              |   2  | Python          |
+|                              | tos (Omon)                                                                                           |      |                 |
+|                              | hfls (Lmon, Amon)                                                                                    |      |                 |
++------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | UWisc* [#t3]_                | clwvi, lwpStderr (Amon)                                                                              |   3  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | WFDE5                        | tas, pr (Amon, day)                                                                                  |   2  | Python          |

esmvaltool/cmorizers/data/cmor_config/TROPFLUX.yml

@@ -0,0 +1,67 @@
+---
+# Global attributes of NetCDF file
+attributes:
+  dataset_id: TROPFLUX
+  project_id: OBS6
+  tier: 2
+  version: "v1"
+  modeling_realm: reanaly
+  source: 'https://incois.gov.in/tropflux/'
+  reference: 'tropflux'
+  comment: |
+    ''
+
+# Variables to CMORise
+variables:
+  # monthly frequency
+  ts_month:
+    short_name: ts
+    mip: Amon
+    raw: t2m
+    file: '/t2m_tropflux_1m_\d{4}\.nc$'
+
+  tos_month:
+    short_name: tos
+    mip: Omon
+    raw: sst
+    file: '/sst_tropflux_1m_\d{4}\.nc$'
+
+  hfls_month_Lmon:
+    short_name: hfls
+    mip: Lmon
+    raw: lhf
+    file: '/lhf_tropflux_1m_\d{4}\.nc$'
+
+  hfls_month_Amon:
+    short_name: hfls
+    mip: Amon
+    raw: lhf
+    file: '/lhf_tropflux_1m_\d{4}\.nc$'
+  hfds_month:
+    short_name: hfds
+    mip: Omon
+    raw: netflux
+    file: '/netflux_tropflux_1m_\d{4}\.nc$'
+
+  rlus_month:
+    short_name: rlus
+    mip: Amon
+    raw: lwr
+    file: '/lwr_tropflux_1m_\d{4}\.nc$'
+
+  tauv_month:
+    short_name: tauv
+    mip: Amon
+    raw: tauy
+    file: '/tauy_tropflux_1m_\d{4}\.nc$'
+  tauu_month:
+    short_name: tauu
+    mip: Amon
+    raw: taux
+    file: '/taux_tropflux_1m_\d{4}\.nc$'
+
+  rsds_month:
+    short_name: rsds
+    mip: Amon
+    raw: swr
+    file: '/swr_tropflux_1m_\d{4}\.nc$'

esmvaltool/cmorizers/data/datasets.yml

@@ -1254,6 +1254,14 @@ datasets:
     last_access: 2023-01-17
     info: |
       Download the file zmn2o_TCOM_plev_T2Dz_1991_2021.nc.
+  
+  TROPFLUX:
+    tier: 2
+    source: "https://incois.gov.in/tropflux"
+    last_access: 2025-01-15
+    info: |
+      The TropFlux project aims at providing daily, timely, accurate air-sea heat and momentum
+      flux data for the entire 30°N-30°S region.
 
   UWisc:
     tier: 3

esmvaltool/cmorizers/data/formatters/datasets/tropflux.py

@@ -0,0 +1,147 @@
+# pylint: disable=unused-argument
+# pylint: disable=too-many-arguments
+# pylint: disable=too-many-function-args
+# pylint: disable=R0917
+# pylint: disable=E1121
+# flake8: noqa
+"""ESMValTool CMORizer for TROPFLUX data.
+
+Tier
+    Tier 2: other freely-available dataset.
+
+Source
+    https://incois.gov.in/tropflux
+
+Last access
+    20250115
+
+Download and processing instructions
+    Register and sign-in here:
+
+    https://incois.gov.in/tropflux/DataHome.jsp
+
+    Data is available per year from 1979 to 2018
+    at daily and monthly frequency.
+
+    Products availables are:
+
+    - Air Temperature at 2m
+    - Latent Heat Flux
+    - Long Wave Radiation
+    - Meridional Wind Stress
+    - Net Surface Heat Flux
+    - Sea Surface Temperature
+    - Sensible Heat Flux
+    - Shortwave Radiation
+    - Specific Humidity at 2m
+    - Wind Speed at 10m
+    - Wind Stress magnitude
+    - Zonal Wind Stress
+
+Caveats
+"""
+
+import logging
+import re
+from copy import deepcopy
+from pathlib import Path
+
+import cf_units
+import iris
+from esmvalcore.cmor.table import CMOR_TABLES
+from iris import NameConstraint
+from iris.util import equalise_attributes
+
+from esmvaltool.cmorizers.data import utilities as utils
+
+logger = logging.getLogger(__name__)
+
+# scitools-iris.readthedocs.io/en/stable/generated/api/iris.html#iris.Future
+try:
+    iris.FUTURE.date_microseconds = True
+    iris.FUTURE.save_split_attrs = True
+except AttributeError as e:
+    # Handle cases where FUTURE or the attributes don't exist
+    logger.warning("AttributeError: %s", e)
+except (TypeError, ValueError) as e:
+    # Handle specific errors if these might occur
+    logger.warning("TypeError or ValueError: %s", e)
+except BaseException as e:
+    # Fallback for rare or unknown issues, but avoid catching Exception
+    logger.warning("An unexpected error occurred: %s", e)
+
+
+def _fix_coordinates(cube, definition):
+    cube = utils.fix_coords(cube)
+
+    for coord_def in definition.coordinates.values():
+        axis = coord_def.axis
+        coord = cube.coord(axis=axis)
+        if axis == 'Z':
+            coord.convert_units(coord_def.units)
+        coord.standard_name = coord_def.standard_name
+        coord.var_name = coord_def.out_name
+        coord.long_name = coord_def.long_name
+        coord.points = coord.core_points().astype('float64')
+        if coord.var_name == 'plev':
+            coord.attributes['positive'] = 'down'
+
+    return cube
+
+
+def _extract_variable(short_name, var, cfg, raw_filepaths, out_dir):
+    attributes = deepcopy(cfg['attributes'])
+    attributes['mip'] = var['mip']
+    cmor_table = CMOR_TABLES[attributes['project_id']]
+    definition = cmor_table.get_variable(var['mip'], short_name)
+    cmor_info = cfg['cmor_table'].get_variable(var['mip'], short_name)
+
+    if cmor_info.positive != '':
+        attributes['positive'] = cmor_info.positive
+
+    # load data
+    raw_var = var.get('raw', short_name)
+    cubes = iris.load(raw_filepaths, NameConstraint(var_name=raw_var))
+    equalise_attributes(cubes)
+    for cube in cubes:
+        cube.rename(cubes[0].name())
+    cube = cubes.concatenate_cube()
+    utils.set_global_atts(cube, attributes)
+
+    # Set correct names
+    cube.var_name = definition.short_name
+    if definition.standard_name:
+        cube.standard_name = definition.standard_name
+    cube.long_name = definition.long_name
+
+    utils.fix_var_metadata(cube, cmor_info)
+
+    # fix time units
+    cube.coord('time').convert_units(
+        cf_units.Unit('days since 1950-1-1 00:00:00', calendar='gregorian'))
+
+    cube = _fix_coordinates(cube, definition)
+
+    if raw_var == 'taux':
+        cube.data = -1 * cube.data
+
+    utils.save_variable(
+        cube,
+        short_name,
+        out_dir,
+        attributes,
+        unlimited_dimensions=['time'],
+    )
+
+def cmorization(in_dir, out_dir, cfg, cfg_user, start_date, end_date):
+    """Run CMORizer for TROPFLUX."""
+    for (short_name, var) in cfg['variables'].items():
+        logger.info("CMORizing variable '%s'", short_name)
+        short_name = var['short_name']
+        raw_filenames = Path(in_dir).rglob("*.nc")
+        filenames = []
+        for raw_filename in raw_filenames:
+            if re.search(var["file"], str(raw_filename)):
+                filenames.append(raw_filename)
+
+        _extract_variable(short_name, var, cfg, filenames, out_dir)

esmvaltool/recipes/examples/recipe_check_obs.yml

@@ -2024,6 +2024,30 @@ diagnostics:
         type: reanaly, version: 4, start_year: 1979, end_year: 2022}
     scripts: null
 
+  TROPFLUX:
+    description: TROPFLUX
+    variables:
+      ts:
+        mip: Amon
+      tos:
+        mip: Omon
+      hfls:
+        mip: Lmon
+      hfls:
+        mip: Amon
+      rlut:
+        mip: Amon
+      tauu:
+        mip: Amon
+      tauv:
+        mip: Amon
+      rsds:
+        mip: Amon
+    additional_datasets:
+      - {dataset: TROPFLUX, project: OBS6, tier: 2,
+        type: reanaly, version: v1, start_year: 1979, end_year: 2018}
+    scripts: null
+
   UWisc:
     description: UWisc check
     variables:

esmvaltool/references/tropflux.bibtex

@@ -0,0 +1,29 @@
+@article{PraveenKumar2012,
+  author    = {Praveen Kumar, B. and Vialard, J. and Lengaigne, M. and Murty, V. S. N. and McPhaden, M. J.},
+  title     = {TropFlux: air-sea fluxes for the global tropical oceans—description and evaluation},
+  journal   = {Climate Dynamics},
+  year      = {2012},
+  volume    = {38},
+  number    = {7},
+  pages     = {1521--1543},
+  month     = {Apr},
+  abstract  = {In this paper, we evaluate several timely, daily air-sea heat flux products (NCEP, NCEP2, ERA-Interim and OAFlux/ISCCP) against observations and present the newly developed TropFlux product. This new product uses bias-corrected ERA-interim and ISCCP data as input parameters to compute air-sea fluxes from the COARE v3.0 algorithm. Wind speed is corrected for mesoscale gustiness. Surface net shortwave radiation is based on corrected ISCCP data. We extend the shortwave radiation time series by using “near real-time” SWR estimated from outgoing longwave radiation. All products reproduce consistent intraseasonal surface net heat flux variations associated with the Madden-Julian Oscillation in the Indian Ocean, but display more disparate interannual heat flux variations associated with El Niño in the eastern Pacific. They also exhibit marked differences in mean values and seasonal cycle. Comparison with global tropical moored buoy array data, I-COADS and fully independent mooring data sets shows that the two NCEP products display lowest correlation to mooring turbulent fluxes and significant biases. ERA-interim data captures well temporal variability, but with significant biases. OAFlux and TropFlux perform best. All products have issues in reproducing observed longwave radiation. Shortwave flux is much better captured by ISCCP data than by any of the re-analyses. Our “near real-time” shortwave radiation performs better than most re-analyses, but tends to underestimate variability over the cold tongues of the Atlantic and Pacific. Compared to independent mooring data, NCEP and NCEP2 net heat fluxes display ~0.78 correlation and >65 W m−2 rms-difference, ERA-I performs better (~0.86 correlation and ~48 W m−2) while OAFlux and TropFlux perform best (~0.9 correlation and ~43 W m−2). TropFlux hence provides a useful option for studying flux variability associated with ocean–atmosphere interactions, oceanic heat budgets and climate fluctuations in the tropics.},
+  issn      = {1432-0894},
+  url       = {https://doi.org/10.1007/s00382-011-1115-0},
+  doi       = {10.1007/s00382-011-1115-0}
+}
+
+@article{PraveenKumar2013,
+  author    = {Praveen Kumar, B. and Vialard, J. and Lengaigne, M. and Murty, V. S. N. and McPhaden, M. J. and Cronin, M. F. and Pinsard, F. and Gopala Reddy, K.},
+  title     = {TropFlux wind stresses over the tropical oceans: evaluation and comparison with other products},
+  journal   = {Climate Dynamics},
+  year      = {2013},
+  volume    = {40},
+  number    = {7},
+  pages     = {2049--2071},
+  month     = {Apr},
+  abstract  = {In this paper, we present TropFlux wind stresses and evaluate them against observations along with other widely used daily air-sea momentum flux products (NCEP, NCEP2, ERA-I and QuikSCAT). TropFlux wind stresses are computed from the COARE v3.0 algorithm, using bias and amplitude corrected ERA-I input data and an additional climatological gustiness correction. The wind stress products are evaluated against dependent data from the TAO/TRITON, PIRATA and RAMA arrays and independent data from the OceanSITES mooring networks. Wind stress products are more consistent amongst each other than surface heat fluxes, suggesting that 10 m-winds are better constrained than near-surface thermodynamical parameters (2 m-humidity and temperature) and surface downward radiative fluxes. QuikSCAT overestimates wind stresses away from the equator, while NCEP and NCEP2 underestimate wind stresses, especially in the equatorial Pacific. QuikSCAT wind stress quality is strongly affected by rain under the Inter Tropical Convergence Zones. ERA-I and TropFlux display the best agreement with in situ data, with correlations >0.93 and rms-differences <0.012 Nm−2. TropFlux wind stresses exhibit a small, but consistent improvement (at all timescales and most locations) over ERA-I, with an overall 17% reduction in root mean square error. ERA-I and TropFlux agree best with long-term mean zonal wind stress observations at equatorial latitudes. All products tend to underestimate the zonal wind stress seasonal cycle by ~20% in the western and central equatorial Pacific. TropFlux and ERA-I equatorial zonal wind stresses have clearly the best phase agreement with mooring data at intraseasonal and interannual timescales (correlation of ~0.9 versus ~0.8 at best for any other product), with TropFlux correcting the ~13% underestimation of ERA-I variance at both timescales. For example, TropFlux was the best at reproducing westerly wind bursts that played a key role in the 1997–1998 El Niño onset. Hence, we recommend the use of TropFlux for studies of equatorial ocean dynamics.},
+  issn      = {1432-0894},
+  url       = {https://doi.org/10.1007/s00382-012-1455-4},
+  doi       = {10.1007/s00382-012-1455-4}
+}