Feat/metrics quantiles (unit8co#2530)

Author: dennisbader

CHANGELOG.md

@@ -11,10 +11,14 @@ but cannot always guarantee backwards compatibility. Changes that may **break co
 
 **Improved**
 
-- Added `IQRDetector`, that allows to detect anomalies using the interquartile range algorithm. [#2441] by [Igor Urbanik](https://github.com/u8-igor).
+- Added `IQRDetector`, that allows to detect anomalies using the interquartile range algorithm. [#2441](https://github.com/unit8co/darts/issues/2441) by [Igor Urbanik](https://github.com/u8-igor).
 - Added hyperparameters controlling the hidden layer sizes for the feature encoders in `TiDEModel`. [#2408](https://github.com/unit8co/darts/issues/2408) by [eschibli](https://github.com/eschibli).
 - Added hyperparameter `activation` to `BlockRNNModel` to specify the activation function in case of a multi-layer output network. [#2408](https://github.com/unit8co/darts/issues/2408) by [eschibli](https://github.com/eschibli).
 - Added support for broadcasting to TimeSeries on component and sample level. [#2476](https://https://github.com/unit8co/darts/pull/2476) by [Joel L.](https://github.com/Joelius300).
+- Added support for computing metrics, backtest, and residuals on one or multiple quantiles `q`, either from probabilistic predictions or predicted quantiles. [#2530](https://github.com/unit8co/darts/issues/2530) by [Dennis Bader](https://github.com/dennisbader).
+- Added quantile interval metrics: `miw` (Mean Interval Width, time aggregated) and `iw` (Interval Width, per time step / non-aggregated) which compute the width of quantile intervals `q_intervals` (expected to be a tuple or sequence of tuples with (lower quantile, upper quantile). [#2530](https://github.com/unit8co/darts/issues/2530) by [Dennis Bader](https://github.com/dennisbader).
+- Added property `TimeSeries.shape` to get the shape of the time series. [#2530](https://github.com/unit8co/darts/issues/2530) by [Dennis Bader](https://github.com/dennisbader).
+- Added support for parameters `enable_optimization` and `predict_likelihood_parameters` to the forecasting models' `backtest()` and `residuals()` methods. [#2530](https://github.com/unit8co/darts/issues/2530) by [Dennis Bader](https://github.com/dennisbader).
 - Helper function `darts.utils.utils.generate_index()` now accepts datetime strings as `start` and `end` parameters to generate the pandas DatetimeIndex. [#2522](https://github.com/unit8co/darts/pull/2522) by [Dennis Bader](https://github.com/dennisbader).
 - Various improvements in the documentation:
   - Made README's forecasting model support table more colorblind-friendly. [#2433](https://github.com/unit8co/darts/pull/2433)

darts/ad/anomaly_model/forecasting_am.py

@@ -132,6 +132,7 @@ def fit(
             `train_length`.
         enable_optimization
             Whether to use the optimized version of historical_forecasts when supported and available.
+            Default: ``True``.
         model_fit_kwargs
             Parameters to be passed on to the forecast model `fit()` method.
 
@@ -212,6 +213,7 @@ def score(
             `train_length`.
         enable_optimization
             Whether to use the optimized version of historical_forecasts when supported and available.
+            Default: ``True``.
         return_model_prediction
             Whether to return the forecasting model prediction along with the anomaly scores.
 
@@ -299,6 +301,7 @@ def predict_series(
             `train_length`.
         enable_optimization
             Whether to use the optimized version of historical_forecasts when supported and available.
+            Default: ``True``.
 
         Returns
         -------
@@ -394,6 +397,7 @@ def eval_metric(
             `train_length`.
         enable_optimization
             Whether to use the optimized version of historical_forecasts when supported and available.
+            Default: ``True``.
         metric
             The name of the metric function to use. Must be one of "AUC_ROC" (Area Under the
             Receiver Operating Characteristic Curve) and "AUC_PR" (Average Precision from scores).
@@ -499,6 +503,7 @@ def show_anomalies(
             `train_length`.
         enable_optimization
             Whether to use the optimized version of historical_forecasts when supported and available.
+            Default: ``True``.
         anomalies
             The ground truth of the anomalies (1 if it is an anomaly and 0 if not).
         names_of_scorers

darts/metrics/__init__.py

@@ -2,7 +2,10 @@
 Metrics
 -------
 
-For deterministic forecasts (point predictions with `num_samples == 1`):
+For deterministic forecasts (point predictions with `num_samples == 1`), probabilistic forecasts (`num_samples > 1`),
+and quantile forecasts. For probablistic and quantile forecasts, use parameter `q` to define the quantile(s) to
+compute the deterministic metrics on:
+
     - Aggregated over time:
         Absolute metrics:
             - :func:`MERR <darts.metrics.metrics.merr>`: Mean Error
@@ -42,8 +45,10 @@
     - Aggregated over time:
         - :func:`MQL <darts.metrics.metrics.mql>`: Mean Quantile Loss
         - :func:`QR <darts.metrics.metrics.qr>`: Quantile Risk
+        - :func:`MIW <darts.metrics.metrics.miw>`: Mean Interval Width
     - Per time step:
         - :func:`QL <darts.metrics.metrics.ql>`: Quantile Loss
+        - :func:`IW <darts.metrics.metrics.iw>`: Interval Width
 
 For Dynamic Time Warping (DTW) (aggregated over time):
     - :func:`DTW <darts.metrics.metrics.dtw_metric>`: Dynamic Time Warping Metric
@@ -57,11 +62,13 @@
     coefficient_of_variation,
     dtw_metric,
     err,
+    iw,
     mae,
     mape,
     marre,
     mase,
     merr,
+    miw,
     mql,
     mse,
     msse,
@@ -90,6 +97,7 @@
     se,
     sle,
     sse,
+    iw,
 }
 
 __all__ = [
@@ -120,4 +128,6 @@
     "sle",
     "smape",
     "sse",
+    "iw",
+    "miw",
 ]