Feat/probabilistic ensemble #1692

madtoinou · 2023-04-04T14:08:42Z

Fixes #1682, fixes #1600.

Summary

The num_sample argument is properly passed to regression_model.predict() in RegressionEnsembleModel.ensemble().

In order to obtain probabilistic predictions, all the models of the RegressionEnsembleModel need to be probabilistic, including its regression_model.

It's however still possible to obtain deterministic predictions if the regression_model is not probabilistic but the others are or by using num_sample=1.

Other Information

The definition of a probabilistic EnsembleModel should be better defined; at the moment, it requires that all its model are probabilistic however for a RegressionEnsembleModel, we might also want the regression model itself to be probabilistic? Or on the opposite, ss a probabilistic regression_model enough to make a RegressionEnsembleModel probabilistic even if all the other models are deterministic?

…m_samples to the ensemble() method

codecov-commenter · 2023-04-04T14:45:39Z

Codecov Report

Patch coverage: 97.36% and project coverage change: -0.12 ⚠️

Comparison is base (1f17580) 94.27% compared to head (d0dd636) 94.16%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1692      +/-   ##
==========================================
- Coverage   94.27%   94.16%   -0.12%     
==========================================
  Files         125      125              
  Lines       11607    11622      +15     
==========================================
+ Hits        10943    10944       +1     
- Misses        664      678      +14

Impacted Files	Coverage Δ
darts/models/forecasting/ensemble_model.py	`96.58% <96.96%> (-0.01%)`	⬇️
darts/models/forecasting/baselines.py	`100.00% <100.00%> (ø)`
...ts/models/forecasting/regression_ensemble_model.py	`100.00% <100.00%> (ø)`

... and 12 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

madtoinou · 2023-05-03T12:19:13Z

Simplified the definition of probabilistic RegressionEnsembleModel : its regression model (the "last/aggregation layer") must be probabilistic, regardless of the others models characteristics.

Having probabilistic models is allowed, they will be trained as such but their prediction will contain only one sample when passed to the regression model at fitting and prediction time (the first one, due to the way RegressionModel._create_lagged_data() get rid of the n_samples dimension).

Attaching two codes snippets to demonstrate how to get probabilistic forecast with EnsembleModel:

from darts.models import LinearRegressionModel, LightGBMModel, RegressionEnsembleModel
from darts.utils.timeseries_generation import sine_timeseries, gaussian_timeseries
import matplotlib.pyplot as plt

# creating synthetic data
start = 10
end = 400
sin_series = sine_timeseries(start=start, end=end, value_amplitude=10)
gaus_series = gaussian_timeseries(mean=10, start=start, end=end)

tmp = sin_series + gaus_series
train, val = tmp.split_after(0.8)

quantiles = [0.25, 0.5, 0.75]

# probabilistic ensembling model
ensemble_lin_reg = LinearRegressionModel(quantiles=quantiles,
                                        lags_future_covariates=[0],
                                        likelihood="quantile")

# probabilistic models
lgbm_model = LightGBMModel(quantiles=quantiles, lags=4, likelihood="quantile")
linreg_model = LinearRegressionModel(quantiles=quantiles, lags=4, likelihood="quantile")

ensemble = RegressionEnsembleModel([lgbm_model, linreg_model],
                                   regression_train_n_points=140,
                                   regression_model=ensemble_lin_reg
                                   )

ensemble.fit(train)
pred = ensemble.predict(len(val), num_samples=1000)

val.plot(label="test")
pred.plot(label="prediction")
plt.show()

# deterministic models
lgbm_model = LightGBMModel(lags=4)
linreg_model = LinearRegressionModel(lags=4)

ensemble = RegressionEnsembleModel([lgbm_model, linreg_model],
                                   regression_train_n_points=140,
                                   regression_model=ensemble_lin_reg
                                   )

ensemble.fit(train)
pred = ensemble.predict(len(val), num_samples=1000)

val.plot(label="test")
pred.plot(label="prediction")
plt.show()

…ut probabilistic ensembles

review-notebook-app · 2023-05-03T13:00:13Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…sembleModel

…mbleModel

…with probabilistic forecasting models, NaiveEnsembleModel also properly ensemble such probabilistic models (takes into account n_samples).

…babilistic

…in ensemble tests

madtoinou · 2023-05-05T15:18:01Z

Extended the feature: if the forecasting_models are probabilistic, they can be sampled to generate the training dataset for the regression model (instead of just taking the first sample). This is controlled with the regression_train_num_samples argument.

These samples are then reduced component-wise (one for each forecasting model) using either the mean or a quantile (using the regression_train_samples_reduction argument) before being used to train the regression model.

Also, NaiveEnsemble can generate probabilistic forecasts if all its models are probabilistic; samples are averaged across the models/components and the n_samples dimension remains untouched.

…8co/darts into feat/probabilistic_ensemble

…el, updated the tests accordingly

review-notebook-app · 2023-05-19T14:46:01Z

View / edit / reply to this conversation on ReviewNB

dennisbader commented on 2023-05-19T14:46:01Z
----------------------------------------------------------------

To make the RegressionEnsembleModel probabilistic, we simply have to use a probabilistic regression model:

dennisbader

Very nice @madtoinou, thanks a lot 🚀 👍

There are one or two points where we could simplify a bit.

Regarding the non-reduction of the samples, I thought more about stacking the points vertically to generate more rows, instead of adding the dimensions as columns.

Maybe we can discuss this again?

darts/models/forecasting/regression_ensemble_model.py

darts/models/forecasting/baselines.py

darts/tests/models/forecasting/test_ensemble_models.py

darts/tests/models/forecasting/test_regression_ensemble_model.py

darts/models/forecasting/regression_ensemble_model.py

Co-authored-by: Dennis Bader <[email protected]>

… regression cannot generate probabilistic forecast

…tacking

dennisbader

Nice, thanks a lot ! :)
We're close! Mainly added a couple of minor suggestions. I think we could allow a mix of deterministic and probabilistic forecasting_models for RegressionEnsembleModel as internally we reduce the probabilistic ones anyway to a deterministic forecast. WDYT?

dennisbader · 2023-05-26T14:24:49Z

darts/models/forecasting/ensemble_model.py

@@ -26,10 +27,24 @@ class EnsembleModel(GlobalForecastingModel):
    ----------
    models
        List of forecasting models whose predictions to ensemble
+
+        .. note::
+                if all the models are probabilistic, the `EnsembleModel` will also be probabilistic.


This is true for naive ensemble but not for RegressionEnsembleModel, right?

Correct, the docstring is different in RegressionEnsembleModel. This note could probably be removed since EnsembleModel cannot be instantiated anyway.

darts/models/forecasting/ensemble_model.py

darts/models/forecasting/regression_ensemble_model.py

dennisbader · 2023-05-26T14:36:41Z

darts/models/forecasting/regression_ensemble_model.py

@@ -69,6 +92,52 @@ def __init__(
            f"{regression_model.lags}",
        )

+        raise_if(


Can we move all those tests to the base class?

Sure, there will be a small discrepancies since the name of the argument/attributes are slightly different between the two classes (regression_ is used as a prefix in RegressionEnsembleModel)

darts/models/forecasting/regression_ensemble_model.py

Co-authored-by: Dennis Bader <[email protected]>

dennisbader

Great work @madtoinou, thanks a lot! 🚀

One more test about the mixed proba model and we're good to go!

dennisbader · 2023-06-02T07:56:55Z

darts/tests/models/forecasting/test_regression_ensemble_model.py

+        # forecasting models are a mix of probabilistic and deterministic, probabilistic regressor
+        ensemble_mixproba = RegressionEnsembleModel(
+            forecasting_models=[
+                self.get_probabilistic_global_model([-1, -3], quantiles),


can you check also mixed proba model with train_num_samples > 1 and a reduction?

had to move the reduction to _make_multiple_prediction() so that they could be stacked properly

…r training

dennisbader

Perfect 💯 Great work @madtoinou!

madtoinou added 2 commits April 4, 2023 15:45

feat: allow for probabilistic regression ensemble model by passing nu…

406b4d0

…m_samples to the ensemble() method

fix: propagating the new argument to the baseline

b3a4d52

madtoinou and others added 2 commits May 2, 2023 10:44

Merge branch 'master' into feat/probabilistic_ensemble

a840eb5

feat: improved the definition of probabilistic EnsembleModel

e3dbed5

madtoinou marked this pull request as ready for review May 3, 2023 11:50

madtoinou requested review from hrzn and dennisbader as code owners May 3, 2023 11:50

doc: updating the docstring and quickstart to include information abo…

de5c096

…ut probabilistic ensembles

madtoinou added 11 commits May 3, 2023 15:21

doc: improved docstring for regression_model argument of RegressionEn…

b8ac4ae

…sembleModel

feat: adding unittests

0cc9b3b

fix: updated EnsembleModel unittest that were covering RegressionEnse…

b0d46b2

…mbleModel

feat: added 2 new parameters to control training of regression model …

e61c89f

…with probabilistic forecasting models, NaiveEnsembleModel also properly ensemble such probabilistic models (takes into account n_samples).

feat: simplify the ensembling method of NaiveEnsemble

0116f83

doc: changed phrasing of the note about how to make EnsembleModel pro…

36fd3a9

…babilistic

feat: improved the tests for stochastic naive ensemble

f085dbd

doc: added comments in regression ensemble tests, fixed a small typo …

396d852

…in ensemble tests

fix: bug in samples reduction prior to regression model training

6a16737

feat: improving the tests for regression ensemble model

2ed3928

fix: simplifiying tests synthax

d9b0121

madtoinou and others added 7 commits May 5, 2023 17:20

Update CHANGELOG.md

d5f0a39

fix: removed useless if else

c4a27ae

Merge branch 'feat/probabilistic_ensemble' of https://github.com/unit…

cb43cf8

…8co/darts into feat/probabilistic_ensemble

Merge branch 'master' into feat/probabilistic_ensemble

53ed564

feat: possible to not reduce the prediction for RegressionEnsembleMod…

5238632

…el, updated the tests accordingly

Merge branch 'master' into feat/probabilistic_ensemble

5baa581

Merge branch 'master' into feat/probabilistic_ensemble

09e0087

dennisbader requested changes May 19, 2023

View reviewed changes

madtoinou and others added 5 commits May 24, 2023 10:42

Apply suggestions from code review

5bceba8

Co-authored-by: Dennis Bader <[email protected]>

fix: adressing reviewer comments

69a7455

fix: ensemble with probabilistic forecasting models but deterministic…

f565778

… regression cannot generate probabilistic forecast

fix: moving predict back to base ensemble class, removed horizontal s…

3beee65

…tacking

Merge branch 'master' into feat/probabilistic_ensemble

f07f416

dennisbader requested changes May 26, 2023

View reviewed changes

madtoinou and others added 3 commits May 30, 2023 08:29

Apply suggestions from code review

02dab52

Co-authored-by: Dennis Bader <[email protected]>

fix: addressing reviewer comments

23e399a

merged with master

cc71a6d

dennisbader approved these changes Jun 2, 2023

View reviewed changes

fix: mixed (proba and deter) forecasting models sampling for regresso…

d0dd636

…r training

dennisbader approved these changes Jun 2, 2023

View reviewed changes

dennisbader merged commit 80c0e5f into master Jun 2, 2023

dennisbader deleted the feat/probabilistic_ensemble branch June 2, 2023 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/probabilistic ensemble #1692

Feat/probabilistic ensemble #1692

madtoinou commented Apr 4, 2023 •

edited

Loading

codecov-commenter commented Apr 4, 2023 •

edited

Loading

madtoinou commented May 3, 2023

review-notebook-app bot commented May 3, 2023

madtoinou commented May 5, 2023

review-notebook-app bot commented May 19, 2023 •

edited

Loading

dennisbader left a comment

dennisbader left a comment

dennisbader May 26, 2023

madtoinou May 30, 2023

dennisbader May 26, 2023

madtoinou May 30, 2023

dennisbader left a comment

dennisbader Jun 2, 2023

madtoinou Jun 2, 2023

dennisbader left a comment

Feat/probabilistic ensemble #1692

Feat/probabilistic ensemble #1692

Conversation

madtoinou commented Apr 4, 2023 • edited Loading

Summary

Other Information

codecov-commenter commented Apr 4, 2023 • edited Loading

Codecov Report

madtoinou commented May 3, 2023

review-notebook-app bot commented May 3, 2023

madtoinou commented May 5, 2023

review-notebook-app bot commented May 19, 2023 • edited Loading

dennisbader left a comment

Choose a reason for hiding this comment

dennisbader left a comment

Choose a reason for hiding this comment

dennisbader May 26, 2023

Choose a reason for hiding this comment

madtoinou May 30, 2023

Choose a reason for hiding this comment

dennisbader May 26, 2023

Choose a reason for hiding this comment

madtoinou May 30, 2023

Choose a reason for hiding this comment

dennisbader left a comment

Choose a reason for hiding this comment

dennisbader Jun 2, 2023

Choose a reason for hiding this comment

madtoinou Jun 2, 2023

Choose a reason for hiding this comment

dennisbader left a comment

Choose a reason for hiding this comment

madtoinou commented Apr 4, 2023 •

edited

Loading

codecov-commenter commented Apr 4, 2023 •

edited

Loading

review-notebook-app bot commented May 19, 2023 •

edited

Loading