Improvement/statsforecastets: make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta #1476

Beerstabr · 2023-01-09T13:25:33Z

This is the first time I make contribution to a library like this, so please forgive me if I forgot to do something obvious!

Main changes:

StatsForecastETS is now probabilistic.
StatsForecasETS now uses future_covariates through first performing OLS and then perform ETS on the in-sample residuals.
Moved StatsForecastAutoARIMA and StatsForecastETS to the same file and extracted some (2) functions.
Added AutoTheta from the StatsForecast library.
Removed sf_auto_arima.py and sf_ets.py since the code is now already included in sf_models.py.

Let me know if you'd like something implemented in a different way. Or if you'd like me to comment a bit more on the changes I've made.

I could also have included AutoCES from StatsForecast. Let me know if you want me to include that one too.

…tAutoARIMA

StatsForecastETS now is probabilistic in the same way as StatsForecas…

…the uncertainty because it doesn't take into account the uncertainty of the coef esimation of the OLS

…ons.

…ded in sf_models.py.

hrzn · 2023-01-10T13:38:24Z

Thanks @Beerstabr, that looks very good :) will have a look in detail soon.
Regarding AutoCES, it would be a welcome addition to Darts, but could also be part of another PR later.

codecov-commenter · 2023-01-20T11:21:07Z

Codecov Report

Base: 94.08% // Head: 94.04% // Decreases project coverage by -0.05% ⚠️

Coverage data is based on head (a0a322b) compared to base (6887ed5).
Patch coverage: 90.38% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1476      +/-   ##
==========================================
- Coverage   94.08%   94.04%   -0.05%     
==========================================
  Files         122      125       +3     
  Lines       10981    11056      +75     
==========================================
+ Hits        10332    10398      +66     
- Misses        649      658       +9

Impacted Files	Coverage Δ
darts/models/__init__.py	`57.14% <60.00%> (+1.14%)`	⬆️
darts/models/components/statsforecast_utils.py	`70.00% <70.00%> (ø)`
darts/models/forecasting/sf_auto_arima.py	`96.87% <75.00%> (+5.69%)`	⬆️
darts/models/forecasting/sf_auto_theta.py	`93.54% <93.54%> (ø)`
darts/models/forecasting/sf_auto_ces.py	`96.29% <96.29%> (ø)`
darts/models/forecasting/sf_auto_ets.py	`97.91% <96.29%> (ø)`
darts/utils/data/tabularization.py	`99.27% <0.00%> (-0.73%)`	⬇️
darts/timeseries.py	`92.14% <0.00%> (-0.23%)`	⬇️
darts/ad/anomaly_model/filtering_am.py	`91.93% <0.00%> (-0.13%)`	⬇️
...arts/models/forecasting/torch_forecasting_model.py	`89.52% <0.00%> (-0.05%)`	⬇️
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

hrzn

Many thanks @Beerstabr , that looks really good and we will definitely be able to include that in the next release of Darts :) I like the way future covariates are now handled.

I just have two fairly important things to ask:

Could you please add unit tests for anything new that you've added (1-2 tests to make sure future covariates work as expected with ETS, and adding AutoTheta to the suite of tests)
Could you split the models in separate files (I know i've initially suggested to regroup... but I have second thoughts because of the impact on the documentation...)

I also have a few other very minor comments, but nothing important.

darts/models/__init__.py

darts/models/forecasting/sf_models.py

Beerstabr · 2023-01-26T14:13:52Z

@hrzn thanks for having a look!

Will work on your request and comments. They all seem totally reasonable.

Co-authored-by: Julien Herzen <[email protected]>

…ents explaining the handling of future covariates by StatsForecastETS. Included StatsForecastTheta in the tests. Moved the utility functions that the statsforecast models share to a singly .py file. Added the CES model which is supposed to be probabilistic, but that doesn't work yet eventhough it is supposed to be included in statsforecast 1.4.0. Trying to figure out why it isn't working. Removed sf_models.py.

hrzn

Looks really good, only a few minor comments/questions left. Thanks @Beerstabr !

darts/models/forecasting/sf_ets.py

darts/tests/models/forecasting/test_local_forecasting_models.py

hrzn · 2023-01-27T15:35:05Z

requirements/core.txt

@@ -13,7 +13,7 @@ requests>=2.22.0
 scikit-learn>=1.0.1
 scipy>=1.3.2
 shap>=0.40.0
-statsforecast>=1.0.0
+statsforecast>=1.4.0


Are you sure this is the lowest possible version that supports everything?
I'm double checking because the higher our lower bounds, the more stringent a condition it puts on users who may have constraints on their Pypi settings.

I'm going to look into it for a bit more. I think it might depend on if we want AutoCES to be probabilistic as well. But let me check it and I'll come back to you with the options.

@hrzn, I'm sure now that it is as follows:

ETS (from statsforecast 1.3.2 onwards is called AutoETS) is available from 1.0.0, but only made probabilistic in 1.4.0

AutoTheta became available in 1.3.0 and was probabilistic from the start

AutoCES became available in 1.1.0 and is not yet probabilistic but will be -soon- in the next pip release (1.4.1/1.5.0?)

So if we don't choose 1.4.0 as a requirement, then we will have to drop the probabilistic nature of StatsForecastETS.
And we might want to rename StatsForecastETS to StatsForecastAutoETS, because it is also renamed by statsforecast.

What would you like me to put in there as a requirement? And do you want me to rename the ETS model?

OK thanks a lot for investigating @Beerstabr .
We can keep 1.4.0 as the lower bound (and even update it once AutoCES is made probabilistic). I think that's OK because anyway all releases of statsforecast are quite recent.
I also think we should rename StatsForecastETS to StatsForecastAutoETS if they renamed it on their side.
Thanks!

…in statsforecast 1.4.0 - changed AutoETS to SFAutoETS - added models to the base tests - wrote two units tests for future covariates use for sf_ets

hrzn · 2023-02-07T15:24:26Z

@Beerstabr the only last thing left is to renamed the model to StatsForecastAutoETS. Let me know if you're fine with still doing this. Thanks again!

Beerstabr · 2023-02-07T15:57:22Z

@hrzn yes, sorry forgot about it. Will change the name tonight!

…babilistic' into improvement/statsforecastets_probabilistic

hrzn

Thanks a lot @Beerstabr ! This will go in the next release.

…covariate support for sf_ets + add AutoTheta (unit8co#1476) * StatsForecastETS now is probabilistic in the same way as StatsForecastAutoARIMA * include future covariates in sf_ets * sf_ets with future_covariates works.. probably it is underestimating the uncertainty because it doesn't take into account the uncertainty of the coef esimation of the OLS * Create separate file for StatsForecast models and extract some functions. * Added AutoTheta from the StatsForecast package. * Deleted sf_auto_arima.py and sf_ets.py, because the code is now included in sf_models.py. * Update darts/models/forecasting/sf_models.py Co-authored-by: Julien Herzen <[email protected]> * Update darts/models/forecasting/sf_models.py Co-authored-by: Julien Herzen <[email protected]> * Moved all statsforecast models to their own .py file. Added some comments explaining the handling of future covariates by StatsForecastETS. Included StatsForecastTheta in the tests. Moved the utility functions that the statsforecast models share to a singly .py file. Added the CES model which is supposed to be probabilistic, but that doesn't work yet eventhough it is supposed to be included in statsforecast 1.4.0. Trying to figure out why it isn't working. Removed sf_models.py. * Beginning of test for fit on residuals for statsforecast ets. * - AutoCES not probablisitc anymore, because that is not yet released in statsforecast 1.4.0 - changed AutoETS to SFAutoETS - added models to the base tests - wrote two units tests for future covariates use for sf_ets * - AutoCES not probablisitc anymore, because that is not yet released in statsforecast 1.4.0 - changed AutoETS to SFAutoETS - added models to the base tests - wrote two units tests for future covariates use for sf_ets * Changed StatsForecastETS to StatsForecastAutoETS. --------- Co-authored-by: Julien Herzen <[email protected]> Co-authored-by: Julien Herzen <[email protected]>

Beerstabr and others added 7 commits January 5, 2023 14:45

StatsForecastETS now is probabilistic in the same way as StatsForecas…

3e571c3

…tAutoARIMA

Merge pull request #1 from Beerstabr/master

0a2bd18

StatsForecastETS now is probabilistic in the same way as StatsForecas…

include future covariates in sf_ets

60253a5

sf_ets with future_covariates works.. probably it is underestimating …

c7f20a3

…the uncertainty because it doesn't take into account the uncertainty of the coef esimation of the OLS

Create separate file for StatsForecast models and extract some functi…

ece2bad

…ons.

Added AutoTheta from the StatsForecast package.

4131cb0

Deleted sf_auto_arima.py and sf_ets.py, because the code is now inclu…

2921d52

…ded in sf_models.py.

Beerstabr requested review from hrzn and dennisbader as code owners January 9, 2023 13:25

Merge branch 'master' into improvement/statsforecastets_probabilistic

049970c

Beerstabr changed the title ~~Improvement/statsforecastets probabilistic~~ Improvement/statsforecastets make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta Jan 9, 2023

Beerstabr changed the title ~~Improvement/statsforecastets make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta~~ Improvement/statsforecastets: make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta Jan 9, 2023

Merge branch 'master' into improvement/statsforecastets_probabilistic

30aff7d

Merge branch 'master' into improvement/statsforecastets_probabilistic

1326065

hrzn reviewed Jan 26, 2023

View reviewed changes

Beerstabr and others added 4 commits January 26, 2023 16:31

Update darts/models/forecasting/sf_models.py

6a514d9

Co-authored-by: Julien Herzen <[email protected]>

Update darts/models/forecasting/sf_models.py

b81fddb

Co-authored-by: Julien Herzen <[email protected]>

Merge branch 'master' into improvement/statsforecastets_probabilistic

2e07d3f

hrzn reviewed Jan 27, 2023

View reviewed changes

Beerstabr added 3 commits January 28, 2023 10:28

Beginning of test for fit on residuals for statsforecast ets.

6a2f73f

- AutoCES not probablisitc anymore, because that is not yet released …

6ea16b4

…in statsforecast 1.4.0 - changed AutoETS to SFAutoETS - added models to the base tests - wrote two units tests for future covariates use for sf_ets

- AutoCES not probablisitc anymore, because that is not yet released …

b74c49a

…in statsforecast 1.4.0 - changed AutoETS to SFAutoETS - added models to the base tests - wrote two units tests for future covariates use for sf_ets

Merge branch 'master' into improvement/statsforecastets_probabilistic

91bbe59

Changed StatsForecastETS to StatsForecastAutoETS.

83f08da

Beerstabr and others added 2 commits February 7, 2023 17:53

Merge remote-tracking branch 'origin/improvement/statsforecastets_pro…

a0a322b

…babilistic' into improvement/statsforecastets_probabilistic

Merge branch 'master' into improvement/statsforecastets_probabilistic

fc20e77

hrzn approved these changes Feb 10, 2023

View reviewed changes

hrzn merged commit e1c8d34 into unit8co:master Feb 10, 2023

dennisbader mentioned this pull request Apr 18, 2023

Support for Future Covariates & Probabilistic Forecasting in AutoCES #1712

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvement/statsforecastets: make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta #1476

Improvement/statsforecastets: make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta #1476

Beerstabr commented Jan 9, 2023 •

edited

Loading

hrzn commented Jan 10, 2023

codecov-commenter commented Jan 20, 2023 •

edited

Loading

hrzn left a comment

Beerstabr commented Jan 26, 2023

hrzn left a comment

hrzn Jan 27, 2023

Beerstabr Jan 28, 2023

Beerstabr Jan 31, 2023

hrzn Feb 1, 2023

hrzn commented Feb 7, 2023

Beerstabr commented Feb 7, 2023

hrzn left a comment

Improvement/statsforecastets: make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta #1476

Improvement/statsforecastets: make sf_ets probabilistic + add future_covariate support for sf_ets + add AutoTheta #1476

Conversation

Beerstabr commented Jan 9, 2023 • edited Loading

hrzn commented Jan 10, 2023

codecov-commenter commented Jan 20, 2023 • edited Loading

Codecov Report

hrzn left a comment

Choose a reason for hiding this comment

Beerstabr commented Jan 26, 2023

hrzn left a comment

Choose a reason for hiding this comment

hrzn Jan 27, 2023

Choose a reason for hiding this comment

Beerstabr Jan 28, 2023

Choose a reason for hiding this comment

Beerstabr Jan 31, 2023

Choose a reason for hiding this comment

hrzn Feb 1, 2023

Choose a reason for hiding this comment

hrzn commented Feb 7, 2023

Beerstabr commented Feb 7, 2023

hrzn left a comment

Choose a reason for hiding this comment

Beerstabr commented Jan 9, 2023 •

edited

Loading

codecov-commenter commented Jan 20, 2023 •

edited

Loading