Adaptation of API for multimodal data #663

andreygetmanov · 2022-04-29T15:19:51Z

Global purpose of PR - to make Fedot API work with multimodal data

Changes:

now multimodal datasets can be run via Fedot API as easy as unimodal datasets
now initial pipeline is based on the type of data (for example, text data always precedes a vectorizing node)
slightly changed the logics of operation filter. Now operations are chosen not only by task but by data type too (each data source will have its own operations list in next PRs)
MultiModalStrategy now works more effectively and stable
AssumptionsBuilder now supports multimodal data and creates subbuilder for each data source
added pretrained embeddings as vectorization models. The type of model can be chosen automatically during pipeline tuning
other parameters of text models now are tunable too
added various tests on new functionality
added new example based on text+table dataset
deleted deprecated imdb case
fixed issue Fix composer and tuner work in the multimodal case Fix composer and tuner work in the multimodal case #630
fixed issue Adapt multimodal example to run by API Adapt multimodal example to run by API #626
fixed issue Fix CNN work in the multimodal case Fix CNN work in the multimodal case #627

pep8speaks · 2022-04-29T15:19:54Z

Hello @andreygetmanov! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-07-15 09:40:51 UTC

fedot/core/data/data_preprocessing.py

examples/advanced/multi_modal_pipeline.py

fedot/api/api_utils/assumptions/preprocessing_builder.py

fedot/core/data/data_preprocessing.py

fedot/core/operations/evaluation/operation_implementations/implementation_interfaces.py

examples/advanced/multi_modal_pipeline.py

Dreamlone · 2022-04-29T15:44:17Z

Напиши плиз сообщение в наш чат с code review - может ребята захотят поревьюить. Кто-то один точно должен захотеть

И ещё от меня просьба, - раз запрошено моё code review - пожалуйста, дай мне время его посмотреть (желательно конечно, чтобы это время было предусмотрено не поздно вечером и в выходные, как это получилось с прошлым PR'м, когда все правки по сути были сделаны в период когда я ехал с работы домой и по быстрому все влито в мастер). Не вливай PR сразу как получил аппрув от одного ревьюера - стоит критически оценивать все изменения и не торопиться

examples/advanced/multi_modal_pipeline.py

fedot/api/api_utils/assumptions/preprocessing_builder.py

fedot/core/data/data_preprocessing.py

Dreamlone

Стоит не забыть поправить класс MulitmodalStrategy, так как именно в нём определяются data source узлы, но нынешняя логика сильно усеченная и не использует тип данных, надо его до туда прокинуть и использовать
См. строчку https://github.com/nccr-itmo/FEDOT/blob/master/fedot/api/api_utils/data_definition.py#L137

examples/advanced/multi_modal_pipeline.py

fedot/api/api_utils/assumptions/preprocessing_builder.py

fedot/core/data/data_preprocessing.py

test/unit/preprocessing/test_preprocessing_though_api.py

fedot/api/api_utils/assumptions/preprocessing_builder.py

Dreamlone

Попробуй ещё плиз в рамках этого PR подумать над примером examples/simple/multitask_classification_regression_api.py, возможно уже сейчас стоит предусмотреть механизм, когда пользователь хочет решать одновременно две разных задачи (классификацию и регрессию, как в этом кейсе например)

fedot/api/api_utils/data_definition.py

fedot/core/data/data_preprocessing.py

test/unit/preprocessing/test_preprocessing_though_api.py

codecov · 2022-05-20T21:18:26Z

Codecov Report

Merging #663 (0adccc4) into master (e08d2e4) will increase coverage by 0.44%.
The diff coverage is 88.25%.

@@            Coverage Diff             @@
##           master     #663      +/-   ##
==========================================
+ Coverage   87.71%   88.16%   +0.44%     
==========================================
  Files         172      181       +9     
  Lines       12098    12467     +369     
==========================================
+ Hits        10612    10991     +379     
+ Misses       1486     1476      -10

Impacted Files	Coverage Δ
fedot/core/composer/composer.py	`89.74% <ø> (ø)`
...ot/core/composer/gp_composer/specific_operators.py	`93.93% <ø> (ø)`
.../core/optimisers/archive/individuals_containers.py	`89.39% <ø> (-0.32%)`	⬇️
...ore/optimisers/gp_comp/operators/regularization.py	`48.57% <0.00%> (+2.62%)`	⬆️
fedot/core/pipelines/tuning/hyperparams.py	`91.83% <0.00%> (+0.34%)`	⬆️
fedot/core/pipelines/tuning/search_space.py	`100.00% <ø> (ø)`
fedot/core/pipelines/tuning/tuner_interface.py	`92.36% <ø> (+0.05%)`	⬆️
fedot/core/serializers/coders/any_serialization.py	`100.00% <ø> (ø)`
fedot/core/serializers/serializer.py	`96.42% <ø> (ø)`
fedot/core/optimisers/adapters.py	`88.77% <33.33%> (ø)`
... and 106 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 703f7b3...0adccc4. Read the comment docs.

- fixed multimodal data preprocessing bug - removed duplicated categorical encoding from preprocessor

- updated CNN tests

- added some docstrings - removed multi_modal_genre_prediction.py case

- added search_space params for word2vec_pretrained and tfidf

- modified data_has_text_features function in preprocessing builder

- text vectorizer params initialisation is moved to init part of class

fedot/api/api_utils/assumptions/assumptions_builder.py

andreygetmanov requested review from Dreamlone, gkirgizov and nicl-nno April 29, 2022 15:19

andreygetmanov commented Apr 29, 2022

View reviewed changes

fedot/core/data/data_preprocessing.py Outdated Show resolved Hide resolved

andreygetmanov commented Apr 29, 2022

View reviewed changes

examples/advanced/multi_modal_pipeline.py Outdated Show resolved Hide resolved

andreygetmanov force-pushed the multimodal_API branch from 505ff69 to cb8bc5c Compare April 29, 2022 15:27

nicl-nno requested changes Apr 29, 2022

View reviewed changes

andreygetmanov force-pushed the multimodal_API branch 2 times, most recently from fedee4c to e69054b Compare April 29, 2022 15:41

andreygetmanov removed the request for review from Dreamlone April 29, 2022 18:27

valer1435 requested changes Apr 30, 2022

View reviewed changes

Dreamlone requested changes May 1, 2022

View reviewed changes

examples/advanced/multi_modal_pipeline.py Outdated Show resolved Hide resolved

fedot/api/api_utils/assumptions/preprocessing_builder.py Outdated Show resolved Hide resolved

andreygetmanov force-pushed the multimodal_API branch from 0fc6a1f to 5edacb7 Compare May 4, 2022 11:03

gkirgizov requested changes May 4, 2022

View reviewed changes

fedot/core/data/data_preprocessing.py Outdated Show resolved Hide resolved

test/unit/preprocessing/test_preprocessing_though_api.py Outdated Show resolved Hide resolved

fedot/api/api_utils/assumptions/preprocessing_builder.py Show resolved Hide resolved

andreygetmanov force-pushed the multimodal_API branch 5 times, most recently from 0f77f29 to a96a60f Compare May 12, 2022 14:22

andreygetmanov requested a review from Dreamlone May 12, 2022 14:44

Dreamlone requested changes May 12, 2022

View reviewed changes

andreygetmanov force-pushed the multimodal_API branch 2 times, most recently from bcc992c to 22b2971 Compare May 20, 2022 13:48

nicl-nno approved these changes May 20, 2022

View reviewed changes

andreygetmanov force-pushed the multimodal_API branch 2 times, most recently from 54fa8b4 to d510824 Compare June 9, 2022 15:07

andreygetmanov force-pushed the multimodal_API branch from d510824 to 9b8085b Compare June 17, 2022 15:16

andreygetmanov added 15 commits July 13, 2022 16:30

- added text+data multimodal example with dataset

a5de7c5

- fixed multimodal data preprocessing bug - removed duplicated categorical encoding from preprocessor

- fixed CNN initial assumption bug

20035ca

- updated CNN tests

- added text vectorization operation by word2vec pretrained models

80da6f2

- added test for data with empty text fields

c3c3e19

- change of multimodal example and case to be run by API

5a4c3db

- CNN node now is added by processing_builder, not by preprocessing

058a44c

- added test for new functionality of MultimodalStrategy data definition

722509c

- modified test_text_features_processed_correctly

ff9580c

- modified test_correct_api_dataset_with_text_preprocessing

3b7f57e

- added test for check of DataDefiner work on multimodal data

e69d8e8

- added some docstrings - removed multi_modal_genre_prediction.py case

- added default_operation_params for word2vec_pretrained and tfidf

748e3c2

- added search_space params for word2vec_pretrained and tfidf

- added tests for various text data

ed3d6e4

- modified data_has_text_features function in preprocessing builder

- changed method of phrase vectorizing from sum to average

3f665c3

- text vectorizer params initialisation is moved to init part of class

- embeddings download info now writes into logger

724cf8a

- fixed bug with non-working tuner on pipeline with keras CNN

7281b0e

andreygetmanov force-pushed the multimodal_API branch 2 times, most recently from 8145bed to 0a22b97 Compare July 13, 2022 14:48

andreygetmanov requested review from valer1435 and gkirgizov July 13, 2022 14:57

valer1435 reviewed Jul 14, 2022

View reviewed changes

fedot/api/api_utils/assumptions/assumptions_builder.py Outdated Show resolved Hide resolved

andreygetmanov force-pushed the multimodal_API branch from 0a22b97 to dd00c9c Compare July 14, 2022 14:47

- MultiModalAssumptionsBuilder refactoring

7ddf0c0

andreygetmanov force-pushed the multimodal_API branch from dd00c9c to 7ddf0c0 Compare July 14, 2022 15:23

valer1435 approved these changes Jul 15, 2022

View reviewed changes

- MultiModalAssumptionsBuilder refactoring [2]

0adccc4

andreygetmanov force-pushed the multimodal_API branch from f7ee2b8 to 0adccc4 Compare July 15, 2022 09:40

gkirgizov approved these changes Jul 15, 2022

View reviewed changes

andreygetmanov merged commit 128c6e6 into master Jul 15, 2022

andreygetmanov deleted the multimodal_API branch July 15, 2022 13:16

andreygetmanov mentioned this pull request Jul 21, 2022

MultiModalData class improvement #789

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adaptation of API for multimodal data #663

Adaptation of API for multimodal data #663

andreygetmanov commented Apr 29, 2022 •

edited

Loading

pep8speaks commented Apr 29, 2022 •

edited

Loading

Dreamlone commented Apr 29, 2022

Dreamlone left a comment

Dreamlone left a comment

codecov bot commented May 20, 2022 •

edited

Loading

Adaptation of API for multimodal data #663

Adaptation of API for multimodal data #663

Conversation

andreygetmanov commented Apr 29, 2022 • edited Loading

pep8speaks commented Apr 29, 2022 • edited Loading

Comment last updated at 2022-07-15 09:40:51 UTC

Dreamlone commented Apr 29, 2022

Dreamlone left a comment

Choose a reason for hiding this comment

Dreamlone left a comment

Choose a reason for hiding this comment

codecov bot commented May 20, 2022 • edited Loading

Codecov Report

andreygetmanov commented Apr 29, 2022 •

edited

Loading

pep8speaks commented Apr 29, 2022 •

edited

Loading

codecov bot commented May 20, 2022 •

edited

Loading