Skip to content

Commit

Permalink
Merge pull request #87 from ESMValGroup/dev_season_max
Browse files Browse the repository at this point in the history
Refactor time operations
  • Loading branch information
mattiarighi authored Sep 9, 2019
2 parents ecff56c + 84cc65f commit 7934111
Show file tree
Hide file tree
Showing 7 changed files with 862 additions and 178 deletions.
197 changes: 171 additions & 26 deletions doc/esmvalcore/preprocessor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -595,16 +595,39 @@ Time manipulation
=================
The ``_time.py`` module contains the following preprocessor functions:

* ``extract_time``: Extract a time range from an Iris ``cube``.
* ``extract_season``: Extract only the times that occur within a specific
season.
* ``extract_month``: Extract only the times that occur within a specific month.
* ``time_average``: Take the weighted average over the time dimension.
* ``seasonal_mean``: Produces a mean for each season (DJF, MAM, JJA, SON)
* ``annual_mean``: Produces an annual or decadal mean.
* ``regrid_time``: Aligns the time axis of each dataset to have common time
* extract_time_: Extract a time range from a cube.
* extract_season_: Extract only the times that occur within a specific season.
* extract_month_: Extract only the times that occur within a specific month.
* daily_statistics_: Compute statistics for each day
* monthly_statistics_: Compute statistics for each month
* seasonal_statistics_: Compute statistics for each season
* annual_statistics_: Compute statistics for each year
* decadal_statistics_: Compute statistics for each decade
* climate_statistics_: Compute statistics for the full period
* anomalies_: Compute anomalies
* regrid_time_: Aligns the time axis of each dataset to have common time
points and calendars.

Statistics functions are applied by default in the order they appear in the
list. For example, the following example applied to hourly data will retrieve
the minimum values for the full period (by season) of the monthly mean of the
daily maximum of any given variable.

.. code-block:: yaml
daily_statistics:
operator: max
monthly_statistics:
operator: mean
climate_statistics:
operator: min
period: season
.. _extract_time:

``extract_time``
----------------

Expand All @@ -625,6 +648,8 @@ will not be accepted.

See also :func:`esmvalcore.preprocessor.extract_time`.

.. _extract_season:

``extract_season``
------------------

Expand All @@ -642,6 +667,8 @@ the seasonal_mean function, below.

See also :func:`esmvalcore.preprocessor.extract_season`.

.. _extract_month:

``extract_month``
-----------------

Expand All @@ -651,37 +678,155 @@ between 1 and 12 as the named month string will not be accepted.

See also :func:`esmvalcore.preprocessor.extract_month`.

.. _time_average:
.. _daily_statistics:

``time_average``
----------------
``daily_statistics``
--------------------

This function takes the weighted average over the time dimension. This
function requires no arguments and removes the time dimension of the cube.
This function produces statistics for each day in the dataset.

See also :func:`esmvalcore.preprocessor.time_average`.
Parameters:
* operator: operation to apply. Accepted values are 'mean',
'median', 'std_dev', 'min' and 'max'. Default is 'mean'

``seasonal_mean``
-----------------
See also :func:`esmvalcore.preprocessor.daily_statistics`.

.. _monthly_statistics:

``monthly_statistics``
----------------------

This function produces a seasonal mean for each season (DJF, MAM, JJA, SON).
Note that this function will not check for missing time points. For instance,
if you are looking at the DJF field, but your datasets starts on January 1st,
the first DJF field will only contain data from January and February.
This function produces statistics for each month in the dataset.

Parameters:
* operator: operation to apply. Accepted values are 'mean',
'median', 'std_dev', 'min' and 'max'. Default is 'mean'

See also :func:`esmvalcore.preprocessor.monthly_statistics`.

.. _seasonal_statistics:

``seasonal_statistics``
-----------------------

This function produces statistics for each season (DJF, MAM, JJA, SON) in the
dataset. Note that this function will not check for missing time points.
For instance, if you are looking at the DJF field, but your datasets
starts on January 1st, the first DJF field will only contain data
from January and February.

We recommend using the extract_time to start the dataset from the following
December and remove such biased initial datapoints.

Parameters:
* operator: operation to apply. Accepted values are 'mean',
'median', 'std_dev', 'min' and 'max'. Default is 'mean'

See also :func:`esmvalcore.preprocessor.seasonal_mean`.

``annual_mean``
---------------
.. _annual_statistics:

``annual_statistics``
---------------------

This function produces statistics for each year.

Parameters:
* operator: operation to apply. Accepted values are 'mean',
'median', 'std_dev', 'min' and 'max'. Default is 'mean'

See also :func:`esmvalcore.preprocessor.annual_statistics`.

.. _decadal_statistics:

``decadal_statistics``
----------------------

This function produces statistics for each decade.

Parameters:
* operator: operation to apply. Accepted values are 'mean',
'median', 'std_dev', 'min' and 'max'. Default is 'mean'

See also :func:`esmvalcore.preprocessor.decadal_statistics`.

.. _climate_statistics:

``climate_statistics``
----------------------

This function produces statistics for the whole dataset. It can produce scalars
(if the full period is chosen) or daily, monthly or seasonal statics.

Parameters:
* operator: operation to apply. Accepted values are 'mean', 'median', 'std_dev',
'min' and 'max'. Default is 'mean'

* period: define the granularity of the statistics: get values for the
full period, for each month or day of year.
Available periods: 'full', 'season', 'seasonal', 'monthly', 'month',
'mon', 'daily', 'day'. Default is 'full'

Examples:
* Monthly climatology:

.. code-block:: yaml
climate_statistics:
operator: mean
period: month
* Daily maximum for the full period:

.. code-block:: yaml
climate_statistics:
operator: max
period: day
* Minimum value in the period:

.. code-block:: yaml
climate_statistics:
operator: min
period: full
See also :func:`esmvalcore.preprocessor.climate_statistics`.

.. _anomalies:

``anomalies``
----------------------

This function computes the anomalies for the whole dataset. It can compute
anomalies from the full, seasonal, monthly and daily climatologies.

Parameters:
* period: define the granularity of the climatology to use:
full period, seasonal, monthly or daily.
Available periods: 'full', 'season', 'seasonal', 'monthly', 'month',
'mon', 'daily', 'day'. Default is 'full'

Examples:

* Anomalies from the monthly climatology:

.. code-block:: yaml
anomalies:
period: month
* Anomalies from the full period climatology:

.. code-block:: yaml
anomalies:
See also :func:`esmvalcore.preprocessor.anomalies`.

This function produces an annual or a decadal mean. The only argument is the
decadal boolean switch. When this switch is set to true, this function
will output the decadal averages.

See also :func:`esmvalcore.preprocessor.annual_mean`.
.. _regrid_time:

``regrid_time``
---------------
Expand Down
3 changes: 2 additions & 1 deletion doc/esmvalcore/recipe.rst
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,8 @@ arguments):
regrid:
target_grid: 1x1
scheme: linear
time_average:
climate_statistics:
operator: mean
multi_model_statistics:
span: overlap
statistics: [mean ]
Expand Down
17 changes: 12 additions & 5 deletions esmvalcore/preprocessor/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,11 @@
from ._reformat import (cmor_check_data, cmor_check_metadata, fix_data,
fix_file, fix_metadata)
from ._regrid import extract_levels, regrid
from ._time import (annual_mean, extract_month, extract_season, extract_time,
regrid_time, seasonal_mean, time_average)
from ._time import (extract_month, extract_season, extract_time, regrid_time,
daily_statistics, monthly_statistics, seasonal_statistics,
annual_statistics, decadal_statistics,
climate_statistics, anomalies)

from ._units import convert_units
from ._volume import (volume_statistics, depth_integration, extract_trajectory,
extract_transect, extract_volume)
Expand Down Expand Up @@ -80,9 +83,13 @@
# 'annual_cycle': annual_cycle,
# 'diurnal_cycle': diurnal_cycle,
'zonal_means',
'annual_mean',
'seasonal_mean',
'time_average',
'daily_statistics',
'monthly_statistics',
'seasonal_statistics',
'annual_statistics',
'decadal_statistics',
'climate_statistics',
'anomalies',
'regrid_time',
'cmor_check_data',
'convert_units',
Expand Down
44 changes: 2 additions & 42 deletions esmvalcore/preprocessor/_area.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,11 @@

import iris
from dask import array as da
from ._shared import guess_bounds, get_iris_analysis_operation

logger = logging.getLogger(__name__)


# guess bounds tool
def _guess_bounds(cube, coords):
"""Guess bounds of a cube, or not."""
# check for bounds just in case
for coord in coords:
if not cube.coord(coord).has_bounds():
cube.coord(coord).guess_bounds()
return cube


# slice cube over a restricted area (box)
def extract_region(cube, start_longitude, end_longitude, start_latitude,
end_latitude):
Expand Down Expand Up @@ -72,37 +63,6 @@ def extract_region(cube, start_longitude, end_longitude, start_latitude,
return cube.copy(data)


def get_iris_analysis_operation(operator):
"""
Determine the iris analysis operator from a string.
Map string to functional operator.
Parameters
----------
operator: str
A named operator.
Returns
-------
function: A function from iris.analysis
Raises
------
ValueError
operator not in allowed operators list.
allowed operators: mean, median, std_dev, variance, min, max
"""
operators = ['mean', 'median', 'std_dev', 'variance', 'min', 'max']
operator = operator.lower()
if operator not in operators:
raise ValueError("operator {} not recognised. "
"Accepted values are: {}."
"".format(operator, ', '.join(operators)))
operation = getattr(iris.analysis, operator.upper())
return operation


def zonal_means(cube, coordinate, mean_type):
"""
Get zonal means.
Expand Down Expand Up @@ -230,7 +190,7 @@ def area_statistics(cube, operator, fx_files=None):

coord_names = ['longitude', 'latitude']
if grid_areas is None or not grid_areas.any():
cube = _guess_bounds(cube, coord_names)
cube = guess_bounds(cube, coord_names)
grid_areas = iris.analysis.cartography.area_weights(cube)
logger.info('Calculated grid area shape: %s', grid_areas.shape)

Expand Down
Loading

0 comments on commit 7934111

Please sign in to comment.