Multimodal documentation

docs/source/basics/multi_modal_tasks.rst

@@ -1,2 +1,89 @@
 Multi-Modal Tasks
 =================
+
+**FEDOT** can solve not only classical tabular data problems, but also problems with multimodal data. In this section, we will consider the main features of the framework for solving such problems.
+
+**Multimodal data** is data that has a different nature (tables, text, images, time series). Humans perceive the world in a multimodal way, so using this approach in machine learning can also work. Indeed, the sharing of several types of data improves the quality of the model at the expense of information that may be contained in one modality and absent in another.
+
+FEDOT's API supports multimodal data from the box. The only thing you need is to load data using ``MultiModalData`` class:
+
+.. code-block:: python
+
+    from fedot.api.main import Fedot
+    from fedot.core.data.data_split import train_test_data_setup
+    from fedot.core.data.multi_modal import MultiModalData
+
+    data = MultiModalData.from_csv(file_path='multimodal_dataset.csv', task='classification', target_columns='target_column',
+                                   text_columns=['text_col1', 'text_col2'], columns_to_drop=['col_to_drop1', 'col_to_drop2'], index_col=None)
+    fit_data, predict_data = train_test_data_setup(data, shuffle_flag=True, split_ratio=0.7)
+
+Using ``from_csv()`` method, you should define the task type, and target columns. FEDOT can find text columns automatically, but you can set them manually. You can also select columns which will be dropped from the original dataset. By default, FEDOT reads the first column of every dataset as an index column. If there is no index columns in the dataset, you should set ``index_col=None``.
+Initialize the FEDOT object and define the type of modeling problem.
+
+.. code-block:: python
+
+    model = Fedot(problem='classification', timeout=10)
+
+.. note::
+
+    Class ``Fedot.__init__()`` has more than two params, e.g. ``preset`` for choosing the set of models or
+    ``n_jobs`` for parallelization. For more details, see the :doc:`FEDOT API </api/api>` section in our documentation.
+
+The ``fit()`` method begins the optimization and returns the resulting composite pipeline.
+
+.. code-block:: python
+
+    model.fit(features=fit_data, target=fit_data.target)
+
+After the fitting is completed, you can look at the structure of the resulting pipeline.
+
+In text format:
+
+.. code-block:: python
+
+    model.current_pipeline.print_structure()
+
+Output:
+
+.. code-block:: text
+
+    Pipeline structure:
+    {'depth': 3, 'length': 4, 'nodes': [rf, data_source_table, tfidf, data_source_text/description]}
+    rf - {'n_jobs': -1, 'bootstrap': False, 'criterion': 'gini', 'max_features': 0.09622420420481334, 'min_samples_leaf': 1, 'min_samples_split': 8}
+    data_source_table - {}
+    tfidf - {'min_df': 0.026548403557843454, 'max_df': 0.9547108243944858, 'ngram_range': (1, 2)}
+    data_source_text/description - {}
+
+And in plot format:
+
+.. code-block:: python
+
+    model.current_pipeline.show()
+
+|pipeline_structure|
+
+.. |pipeline_structure| image:: img_utilities/multimodal_pipeline_example.png
+   :width: 80%
+
+
+The ``predict()`` method, which uses an already fitted pipeline, returns values for the target.
+
+.. code-block:: python
+
+    prediction = model.predict(predict_data)
+
+The ``get_metrics()`` method estimates the quality of predictions according the selected metrics.
+
+.. code-block:: python
+
+    prediction = model.get_metrics()
+
+Example of using FEDOT for multimodal data classification on Wine Reviews dataset:
+
+.. automodule:: examples.advanced.multimodal_text_num_example
+   :members:
+   :no-undoc-members:
+
+.. hint::
+
+    `Tutorial on using FEDOT for multimodal data classification <https://github.com/ITMO-NSS-team/fedot-examples/blob/main/notebooks/latest/7_multimodal_data.ipynb>`_

docs/source/examples/notebooks.rst

@@ -22,10 +22,13 @@ This repository contains following notebooks:
 * `Using FEDOT for hybrid modeling with custom model tuning <https://github.com/ITMO-NSS-team/fedot-examples/blob/main/notebooks/latest/6_hybrid_modelling.ipynb>`_
     Notebook contains examples of custom model specification for Time Series Forecasting problem.
 
+* `Using FEDOT for multimodal data classification <https://github.com/ITMO-NSS-team/fedot-examples/blob/main/notebooks/latest/7_multimodal_data.ipynb>`_
+    A guide to FEDOT's functionality for multimodal data.
+
 FEDOT can be used in cloud services, e.g. `VK Cloud's ML Platform <https://mcs.mail.ru/>`_. This `repository <https://github.com/stockblog/webinar_automl_fedot>`_
 contains examples of FEDOT usage with JupyterLab and MLflow:
 
 * `Amazon Employee Access Challenge - classification problem <https://github.com/stockblog/webinar_automl_fedot/blob/main/FEDOT%20Tutorial%20-%20Classification.ipynb>`_
 * `Molhack AIRI - Regression problem <https://github.com/stockblog/webinar_automl_fedot/blob/main/FEDOT%20Tutorial%20-%20Regression.ipynb>`_
-* `Wine Reviews - Multimodal classification problem <https://github.com/stockblog/webinar_automl_fedot/blob/main/FEDOT%20Tutorial%20-%20Timeseries%20Forecasting.ipynb>`_
-* `Sea level daily - Time Series Forecasting problem <https://github.com/stockblog/webinar_automl_fedot/blob/main/FEDOT%20Tutorial%20-%20Multimodal%20Data.ipynb>`_
+* `Wine Reviews - Multimodal classification problem <https://github.com/stockblog/webinar_automl_fedot/blob/main/FEDOT%20Tutorial%20-%20Multimodal%20Data.ipynb>`_
+* `Sea level daily - Time Series Forecasting problem <https://github.com/stockblog/webinar_automl_fedot/blob/main/FEDOT%20Tutorial%20-%20Timeseries%20Forecasting.ipynb>`_

examples/advanced/multimodal_text_num_example.py

@@ -8,18 +8,19 @@
 
 def run_multi_modal_example(file_path: str, visualization=False, with_tuning=True) -> float:
     """
-    This is an example of FEDOT use on multimodal data.
-    The data is taken and adapted from Wine Reviews dataset (winemag-data_first150k):
-    https://www.kaggle.com/datasets/zynicide/wine-reviews
-    and contains information about wine country, region, price, etc.
-    Column that contains text features is 'description'.
-    Other columns contain numerical and categorical features.
-    The aim is to predict wine variety, so it's a classification task.
-
-    :param file_path: path to the file with multimodal data
-    :param visualization: if True, then final pipeline will be visualised
-
-    :return: F1 metrics of the model
+    Runs FEDOT on multimodal data from the `Wine Reviews dataset
+    <https://www.kaggle.com/datasets/zynicide/wine-reviews>`_.
+    The dataset contains information about wine country, region, price, etc.
+    with text features in the ``description`` column and other columns containing
+    numerical and categorical features. It is a classification task for wine variety prediction.
+
+    Args:
+        file_path: path to the file with multimodal data.
+        visualization: if True, then final pipeline will be visualised.
+        with_tuning: if True, then pipeline will be tuned.
+
+    Returns:
+        F1 metrics of the model.
     """
     task = 'classification'
     path = Path(fedot_project_root(), file_path)