make prophet, lightgbm and catboost optional (#1589)

* make prophet optional again, adapt install readme

* adapt u8darts setup as well

* remove prophet, lightgbm, catboost for core requirements

* simplify the module importing

* update tests with new optional dependencies

* update gradle for including optional dependencies in unittest_all

* update checkExamples to show all installed dependencies before test

* apply suggestions from PR review

* update install guide

* update shap unit tests

* apply suggestions from PR review

* update changelog

---------

Co-authored-by: dennisbader <[email protected]>

Author: hrzn

CHANGELOG.md

@@ -10,6 +10,13 @@ but cannot always guarantee backwards compatibility. Changes that may **break co
 
 ### For users of the library:
 
+**Installation**
+- 🔴 Removed Prophet, LightGBM, and CatBoost dependencies from PyPI packages (`darts`, `u8darts`, `u8darts[torch]`), and conda-forge packages (`u8darts`, `u8darts-torch`)  to avoid installation issues that some users were facing (installation on Apple M1/M2 devices, ...). [#1589](https://github.com/unit8co/darts/pull/1589) by [Julien Herzen](https://github.com/hrzn) and [Dennis Bader](https://github.com/dennisbader).
+  - The models are still supported by installing the required packages as described in our [installation guide](https://github.com/unit8co/darts/blob/master/INSTALL.md#enabling-optional-dependencies).
+  - PyPi package `u8darts[all]` and conda-forge package `u8darts-all` are now equivalent to the old `darts` package (all dependencies). 
+  - Added new PyPI flavor `u8darts[notorch]`, and conda-forge flavor `u8darts-notorch` which are equivalent to the old `u8darts` installation.
+- 🔴 Removed support for Python 3.7 [#1864](https://github.com/unit8co/darts/pull/#1864) by [Dennis Bader](https://github.com/dennisbader).
+
 **Improved**
 - General model improvements:
   - Added support for `PathLike` to the `save()` and `load()` functions of all non-deep learning based models. [#1754](https://github.com/unit8co/darts/pull/1754) by [Simon Sudrich](https://github.com/sudrich).
@@ -42,9 +49,6 @@ but cannot always guarantee backwards compatibility. Changes that may **break co
 - Fixed an issue when calling `ARIMA.predict()` and `num_samples > 1` (probabilistic forecasting), where the start point of the simulation was not anchored to the end of the target series. [#1893](https://github.com/unit8co/darts/pull/1893) by [Dennis Bader](https://github.com/dennisbader).
 - Fixed an issue when using `TFTModel.predict()` with `full_attention=True` where the attention mask was not applied properly. [#1392](https://github.com/unit8co/darts/issues/1392) by [Dennis Bader](https://github.com/dennisbader).
 
-**Removed**
-- Removed support for Python 3.7 [#1864](https://github.com/unit8co/darts/pull/#1864) by [Dennis Bader](https://github.com/dennisbader).
-
 ### For developers of the library:
 
 **Improvements**

INSTALL.md

@@ -2,111 +2,93 @@
 
 Below, we detail how to install Darts using either `conda` or `pip`.
 
+## From PyPI
+Install darts with all models expect the ones from optional dependencies (Prophet, LightGBM, CatBoost, see more on that [here](#enabling-optional-dependencies)): `pip install darts`.
+
+If this fails on your platform, please follow the official installation 
+guide for [PyTorch](https://pytorch.org/get-started/locally/), then try installing Darts again.
+
+As some dependencies are relatively big or involve non-Python dependencies,
+we also maintain the `u8darts` package, which provides the following alternate lighter install options:
+
+* Install darts with all available models: `pip install u8darts[all]`
+* Install core only (without neural networks, Prophet, LightGBM and Catboost): `pip install u8darts`
+* Install core + Prophet + LightGBM + CatBoost: `pip install "u8darts[notorch]"`
+* Install core + neural networks (PyTorch): `pip install "u8darts[torch]"` (equivalent to `pip install darts`)
+
 ## From conda-forge
 Currently only the x86_64 architecture with Python 3.8-3.10
 is fully supported with conda; consider using PyPI if you are running into troubles.
 
-To create a conda environment for Python 3.9
+Create a conda environment (e.g., for Python 3.10):
 (after installing [conda](https://docs.conda.io/en/latest/miniconda.html)):
 
-    conda create --name <env-name> python=3.9
+    conda create --name <env-name> python=3.10
 
-Don't forget to activate your virtual environment
+Activate the environment
 
     conda activate <env-name>
 
-As some models have relatively heavy dependencies, we provide two conda-forge packages:
+As some models have relatively heavy dependencies, we provide four conda-forge packages:
 
-* Install darts with all available models (recommended): `conda install -c conda-forge -c pytorch u8darts-all`.
+* Install darts with all available models: `conda install -c conda-forge -c pytorch u8darts-all`
+* Install core only (without neural networks, Prophet, LightGBM and Catboost): `conda install -c conda-forge u8darts`
+* Install core + Prophet + LightGBM + CatBoost: `pip install "u8darts-notorch"`
 * Install core + neural networks (PyTorch): `conda install -c conda-forge -c pytorch u8darts-torch`
-* Install core only (without neural networks or AutoARIMA): `conda install -c conda-forge u8darts`
 
-For GPU support, please follow the instructions to install CUDA in the [PyTorch installation guide](https://pytorch.org/get-started/locally/).
 
+## Other Information
 
-## From PyPI
-Install darts with all available models: `pip install darts`.
+### Enabling Optional Dependencies
+By default, as of 0.25.0, `darts` does not have Prophet, CatBoost, and LightGBM as dependencies anymore, because their 
+build processes were too often causing issues. If you want to use any of Darts' `Prophet`, `CatBoostModel`, and 
+`LightGBMModel`, you will need to manually install the corresponding packages.  
 
-If this fails on your platform, please follow the official installation 
-guide for [PyTorch](https://pytorch.org/get-started/locally/), then try installing Darts again.
+#### Prophet
+Install the `prophet` package (version 1.1.1 or more recent) using the [Prophet install guide](https://facebook.github.io/prophet/docs/installation.html#python)
 
-As some dependencies are relatively big or involve non-Python dependencies,
-we also maintain the `u8darts` package, which provides the following alternate lighter install options:
+#### CatBoostModel
+Install the `catboost` package (version 1.0.6 or more recent) using the [CatBoost install guide](https://catboost.ai/en/docs/concepts/python-installation)
 
-* Install core only (without neural networks, Prophet or AutoARIMA): `pip install u8darts`
-* Install core + neural networks (PyTorch): `pip install "u8darts[torch]"`
-* Install core + AutoARIMA: `pip install "u8darts[pmdarima]"`
+#### LightGBMModel
+Install the `lightgbm` package (version 3.2.0 or more recent) using the [LightGBM install guide](https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html)
 
-### Enabling Support for LightGBM
-
-To enable support for LightGBM in Darts, please follow the
-[installation instructions](https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html) for your OS.
-
-#### MacOS Issues with LightGBM
-At the time of writing, there is an issue with ``libomp`` 12.0.1 that results in
-[segmentation fault on Mac OS Big Sur](https://github.com/microsoft/LightGBM/issues/4229).
-Here's the procedure to downgrade the ``libomp`` library (from the
-[original Github issue](https://github.com/microsoft/LightGBM/issues/4229#issue-867528353)):
-* [Install brew](https://brew.sh/) if you don't already have it.
-* Install `wget` if you don't already have it : `brew install wget`.
-* Run the commands below:
-```
-wget https://raw.githubusercontent.com/Homebrew/homebrew-core/fb8323f2b170bd4ae97e1bac9bf3e2983af3fdb0/Formula/libomp.rb
-brew unlink libomp
-brew install libomp.rb
-```
+### Enabling GPU support
+Darts relies on PyTorch for the neural network models.
+For GPU support, please follow the instructions to install CUDA in the [PyTorch installation guide](https://pytorch.org/get-started/locally/).
 
-#### Test environment Apple M1 processor
+### Using an emulated x64 environment on Apple Silicon.
+The installation of `darts` has been tested to work on Apple silicon (M1) (Python 3.10, OSX Ventura 13.2.1).
 
-We currently recommend to run Darts in an x_64 emulated environment on Mac computers with the Silicon M1 processor,
-instead of trying to install directly with native arm64 packages, many of the dependent packages still have compatibility 
-issues. The following is a proposed procedure, if you tested other procedures on similar hardware and they worked, 
-please let us know about them by opening an issue or by updating this file and opening a PR. 
+If you run into issues, you can always use rosetta to run in an (intel) emulated x64 environment:
 
 Before you start make sure that you have rosetta2 installed by running: 
-```
+
+```bash
 pgrep oahd
 ``` 
-If you see some process id you are ready to go, as internally rosetta is known as oah.
+
+If you see a process id you are ready to go, as internally rosetta is known as oah.
 
 If pgrep doesn't return any id then install rosetta2:
-```
+
+```bash
 softwareupdate --install-rosetta
 ```
 
 Below are the necessary instructions to create and configure the environment:
-- Start by installing conda (e.g., with miniforge : `brew install miniforge`).
-- Create the x_64 environment : `CONDA_SUBDIR=osx-64 conda create -n env_name python=3.9 pip`
-- Activate the created environment: `conda activate env_name`
+- Install conda if you haven't done so (e.g., with miniforge : `brew install miniforge`).
+- Create the x_64 environment : `CONDA_SUBDIR=osx-64 conda create -n env_name python=3.10 pip`
+- Activate the environment: `conda activate env_name`
 - Configure the environment : `conda env config vars set CONDA_SUBDIR=osx-64`
 - Deactivate and reactivate the environment:
-  ```
+  ```bash
   conda deactivate
   conda activate env_name
   ```
 - Install darts: `pip install darts`
-  - With this method of installation, lightgbm might still have issues finding the libomp library.
-  The following procedure is to guarantee that the correct libomp (11.1.0) library is linked.
-    - Unlink the existing libomp, from terminal : `brew unlink libomp`
-    - Setup a homebrew installer that is compatible with x_64 packages (follow this [blog](https://medium.com/mkdir-awesome/how-to-install-x86-64-homebrew-packages-on-apple-m1-macbook-54ba295230f) 
-    post):
-    ```
-    cd ~/Downloads
-    mkdir homebrew
-    curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew
-    sudo mv homebrew /usr/local/homebrew
-    export PATH=$HOME/bin:/usr/local/bin:$PATH
-    ```
-    - At this point, we have a new brew command located at /usr/local/homebrew/bin/brew
-    - In the following code bits we download version 11.1.0 of libomp, install it as a x_64 compatible package and link to it so that lightgbm can find it:
-    ```
-    wget https://raw.githubusercontent.com/Homebrew/homebrew-core/fb8323f2b170bd4ae97e1bac9bf3e2983af3fdb0/Formula/libomp.rb
-    arch -x86_64 /usr/local/homebrew/bin/brew install libomp.rb
-    sudo ln -s /usr/local/homebrew/Cellar/libomp/11.1.0/lib /usr/local/opt/libomp/lib
-    ```
-    - Verify that your lightgbm works by importing lightgbm from your python env. It should not give library loading errors. 
-
-## Running the examples only, without installing:
+
+### Running the examples only, without installing:
 
 If the conda setup is causing too many problems, we also provide a Docker image with everything set up for you and ready-to-use Python notebooks with demo examples.
 To run the example notebooks without installing our libraries natively on your machine, you can use our Docker image:

build.gradle

@@ -64,7 +64,7 @@ task installPipLatest {
             commandLine "pip", "install", "pip-tools"
         }
         exec {
-            commandLine "pip-compile", "-o", "requirements-latest.txt"
+            commandLine "pip-compile", "requirements/core.txt", "requirements/notorch.txt", "requirements/torch.txt", "-o", "requirements-latest.txt"
         }
         exec {
             commandLine "pip", "install", "-r", "requirements-latest.txt"
@@ -80,7 +80,7 @@ void createPipInstallTask(String flavour) {
    }
 }
 
-String[] flavours = ["core", "dev", "torch", "release"];
+String[] flavours = ["core", "dev", "notorch", "torch", "release"];
 
 for(String flavour : flavours) {
     createPipInstallTask(flavour);
@@ -94,7 +94,7 @@ task pipInstall() {
     doFirst {
         setupPip
     }
-    dependsOn pip_core, pip_dev, pip_torch, pip_release
+    dependsOn pip_core, pip_dev, pip_notorch, pip_torch, pip_release
 }
 
 task lint_black(type: Exec) {
@@ -158,6 +158,14 @@ def exampleName=project.properties["exampleName"] ?: ""
 task checkExample(type: Exec) {
     dependsOn pipInstall, installLocally
     workingDir "./examples"
+    doFirst {
+        exec {
+            commandLine "echo", "Installed packages"
+        }
+        exec {
+            commandLine "pip", "list"
+        }
+    }
     // exampleName must be passed with -PexampleName=FFT-examples.ipynb
     commandLine "papermill", exampleName, exampleName
 }

darts/models/__init__.py

@@ -26,6 +26,7 @@
 from darts.models.forecasting.tbats_model import BATS, TBATS
 from darts.models.forecasting.theta import FourTheta, Theta
 from darts.models.forecasting.varima import VARIMA
+from darts.models.utils import NotImportedModule
 
 try:
     from darts.models.forecasting.block_rnn_model import BlockRNNModel
@@ -50,48 +51,17 @@
 try:
     from darts.models.forecasting.lgbm import LightGBMModel
 except ModuleNotFoundError:
-    logger.warning(
-        "Support for LightGBM not available. "
-        "To enable LightGBM support in Darts, follow the detailed "
-        "install instructions for LightGBM in the README: "
-        "https://github.com/unit8co/darts/blob/master/INSTALL.md"
-    )
-    # TODO: simpler option would be to write LightGBMModel=None
-    # Prevent a second ImportError that would interrupt the import
-    class NotImportedLightGBM:
-        usable = False
-
-    LightGBMModel = NotImportedLightGBM()
+    LightGBMModel = NotImportedModule(module_name="LightGBM")
 
 try:
     from darts.models.forecasting.prophet_model import Prophet
 except ImportError:
-    logger.warning(
-        "The prophet module could not be imported. "
-        "To enable support for Prophet model, follow "
-        "the instruction in the README: "
-        "https://github.com/unit8co/darts/blob/master/INSTALL.md"
-    )
-
-    class NotImportedProphet:
-        usable = False
-
-    Prophet = NotImportedProphet()
+    Prophet = NotImportedModule(module_name="Prophet")
 
 try:
     from darts.models.forecasting.catboost_model import CatBoostModel
 except ModuleNotFoundError:
-    logger.warning(
-        "The catboost module could not be imported. "
-        "To enable support for CatBoostModel model, "
-        "follow the instruction in the README: "
-        "https://github.com/unit8co/darts/blob/master/INSTALL.md"
-    )
-
-    class NotImportedCatBoostModel:
-        usable = False
-
-    CatBoostModel = NotImportedCatBoostModel()
+    CatBoostModel = NotImportedModule(module_name="CatBoost")
 
 try:
     from darts.models.forecasting.croston import Croston
@@ -102,39 +72,21 @@ class NotImportedCatBoostModel:
 
 except ImportError:
     logger.warning(
-        "The statsforecast module could not be imported. "
+        "The StatsForecast module could not be imported. "
         "To enable support for the StatsForecastAutoARIMA, "
         "StatsForecastAutoETS and Croston models, please consider "
         "installing it."
     )
-
-    class NotImportedStatsForecastAutoARIMA:
-        usable = False
-
-    StatsForecastAutoARIMA = NotImportedStatsForecastAutoARIMA()
-
-    class NotImportedStatsForecastAutoETS:
-        usable = False
-
-    StatsForecastAutoETS = NotImportedStatsForecastAutoETS()
-
-    class NotImportedCroston:
-        usable = False
-
-    Croston = NotImportedCroston()
+    Croston = NotImportedModule(module_name="StatsForecast", warn=False)
+    StatsForecastAutoARIMA = NotImportedModule(module_name="StatsForecast", warn=False)
+    StatsForecastAutoCES = NotImportedModule(module_name="StatsForecast", warn=False)
+    StatsForecastAutoETS = NotImportedModule(module_name="StatsForecast", warn=False)
+    StatsForecastAutoTheta = NotImportedModule(module_name="StatsForecast", warn=False)
 
 try:
     from darts.models.forecasting.xgboost import XGBModel
 except ImportError:
-    logger.warning(
-        "The xgboost module could not be imported. "
-        "To enable support for XGBoost model, install the xgboost package."
-    )
-
-    class NotImportedXGBModel:
-        usable = False
-
-    XGBModel = NotImportedXGBModel()
+    XGBModel = NotImportedModule(module_name="XGBoost")
 
 from darts.models.filtering.gaussian_process_filter import GaussianProcessFilter
 from darts.models.filtering.kalman_filter import KalmanFilter

darts/models/forecasting/regression_model.py

@@ -55,16 +55,6 @@
 
 logger = get_logger(__name__)
 
-try:
-    from catboost import CatBoostRegressor
-except ModuleNotFoundError:
-    logger.warning(
-        "The catboost module could not be imported. "
-        "To enable support for CatBoostRegressor, "
-        "follow the instruction in the README: "
-        "https://github.com/unit8co/darts/blob/master/INSTALL.md"
-    )
-
 
 class RegressionModel(GlobalForecastingModel):
     def __init__(
@@ -533,7 +523,7 @@ def fit(
                     self.model = MultiOutputRegressor(
                         self.model, n_jobs=n_jobs_multioutput_wrapper
                     )
-                elif isinstance(self.model, CatBoostRegressor):
+                elif self.model.__class__.__name__ == "CatBoostRegressor":
                     if (
                         self.model.get_params()["loss_function"]
                         == "RMSEWithUncertainty"

darts/models/utils.py

@@ -0,0 +1,22 @@
+from darts.logging import get_logger, raise_log
+
+logger = get_logger(__name__)
+
+
+class NotImportedModule:
+    """Helper class for handling import errors of optional dependencies."""
+
+    usable = False
+
+    def __init__(self, module_name: str, warn: bool = True):
+        self.error_message = (
+            f"The `{module_name}` module could not be imported. "
+            f"To enable {module_name} support in Darts, follow the detailed "
+            f"instructions in the installation guide: "
+            f"https://github.com/unit8co/darts/blob/master/INSTALL.md"
+        )
+        if warn:
+            logger.warning(self.error_message)
+
+    def __call__(self, *args, **kwargs):
+        raise_log(ImportError(self.error_message), logger=logger)