MultiModalData class improvement #789

andreygetmanov · 2022-07-21T14:47:49Z

Now csv files with text and table columns can be read and separated to various data sources just in one motion

MultiModalData.from_csv method added
text fields are defined automatically, if are not predefined by user
tests are added

pep8speaks · 2022-07-21T14:47:53Z

Hello @andreygetmanov! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file examples/advanced/multimodal_text_num_example.py:

Line 34:5: F841 local variable 'prediction' is assigned to but never used

In the file fedot/core/data/data.py:

Line 261:17: F402 import 'field' from line 4 shadowed by loop variable

In the file fedot/core/data/merge/data_merger.py:

Line 7:1: F403 'from fedot.core.data.array_utilities import *' used; unable to detect undefined names
Line 29:21: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 30:31: F405 'find_common_elements' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 32:30: F541 f-string is missing placeholders
Line 59:72: F405 'Optional' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 80:32: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 88:44: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 107:50: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 107:69: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 109:25: F405 'atleast_2d' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 111:45: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 111:59: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 113:16: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 115:53: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 115:66: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 119:46: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 119:75: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 122:22: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 124:18: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 139:36: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 146:50: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 146:69: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 148:38: F405 'atleast_4d' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 161:53: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 161:66: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 163:16: F405 'flatten_extra_dim' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 168:45: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 168:59: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 173:53: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities
Line 173:66: F405 'np' may be undefined, or defined from star imports: fedot.core.data.array_utilities

In the file fedot/preprocessing/data_type_check.py:

Line 29:1: W391 blank line at end of file

Comment last updated at 2022-09-05 14:00:25 UTC

codecov · 2022-07-21T14:57:00Z

Codecov Report

Merging #789 (15a363b) into master (9ae9151) will increase coverage by 0.04%.
The diff coverage is 97.57%.

❗ Current head 15a363b differs from pull request most recent head 70e5af6. Consider uploading reports for the commit 70e5af6 to get more accurate results

@@            Coverage Diff             @@
##           master     #789      +/-   ##
==========================================
+ Coverage   87.73%   87.78%   +0.04%     
==========================================
  Files         193      194       +1     
  Lines       13230    13343     +113     
==========================================
+ Hits        11608    11713     +105     
- Misses       1622     1630       +8

Impacted Files	Coverage Δ
fedot/core/data/data_detection.py	`97.01% <97.01%> (ø)`
fedot/core/data/multi_modal.py	`88.88% <97.29%> (+2.37%)`	⬆️
fedot/preprocessing/preprocessing.py	`98.76% <98.07%> (-0.55%)`	⬇️
fedot/core/data/data.py	`86.82% <100.00%> (+0.54%)`	⬆️
fedot/core/data/data_preprocessing.py	`93.24% <100.00%> (+0.09%)`	⬆️
fedot/core/data/merge/data_merger.py	`98.93% <100.00%> (ø)`
...lementations/data_operations/ts_transformations.py	`86.40% <0.00%> (-1.70%)`	⬇️
...entations/models/ts_implementations/statsmodels.py	`94.38% <0.00%> (-0.52%)`	⬇️
...edot/core/optimisers/gp_comp/operators/mutation.py	`93.27% <0.00%> (-0.45%)`	⬇️
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

examples/advanced/multimodal_text_num_example.py

fedot/core/data/multi_modal.py

examples/advanced/multimodal_text_num_example.py

fedot/core/data/multi_modal.py

test/unit/data/test_multimodal_data.py

examples/advanced/multimodal_text_num_example.py

aPovidlo · 2022-08-25T10:50:38Z

fedot/core/data/data_detection.py

+ALLOWED_NAN_PERCENT = 0.9
+
+
+class DataDetector:


Не совсем понял предназначение данного класса

Не совсем понял предназначение данного класса

Если ты про DataDetector, то это абстракция двух последующих классов. В них есть схожие по механике методы, поэтому решил на это указать, создав абстрактный класс

Схожие по механике методы вижу. Так может один раз реализовать в DataDetector и использовать от туда, чем для каждого по-отдельности? Или это как-то по другому работает?

Схожие по механике методы вижу. Так может один раз реализовать в DataDetector и использовать от туда, чем для каждого по-отдельности? Или это как-то по другому работает?

Это шаблон абстракции для этих классов. В этом классе методы просто унифицируются. Думаю в будущем расширять по мере необходимости

fedot/core/data/data_detection.py

test/unit/data/test_multimodal_data.py

fedot/core/data/data_detection.py

fedot/preprocessing/preprocessing.py

test/unit/data/test_multimodal_data.py

MorrisNein

Поддерживаю идею разделить код для разных типов данных
Возможно, можно было обойтись разными модулями вместо разных классов, но на данном этапе не принципиально, не стоит всё пытаться сделать сразу идеально. По мере усложнения станет понятнее, как удобно.

UPD: Я тут писал про ошибку в property MultiModalData, но сам ошибся. Там всё ок

fedot/core/data/multi_modal.py

fedot/core/data/data_detection.py

fedot/preprocessing/preprocessing.py

Now csv files with text and table data can be read just in one motion - from_csv method added - text fields are defined automatically - tests are added

- added docstring for _column_contains_text - multimodal_wine dataset is moved to more appropriate place - protected funcs of multi_modal.py are now protected

- refactoring of text preprocessing - refactoring of multimodal data test

- now if text column contains a lot of nans, it's dropped

…ing DataDetector parent in data_detection.py

- added description for multimodal_text_num_example.py

- added docstrings

…bility

- DataDetection classes now look better - refactoring of prepare_multimodal_ts logic - preprocessing is now excludes some data types using decorators

fedot/core/data/multi_modal.py

fedot/core/data/data_detection.py

fedot/preprocessing/preprocessing.py

- minor changes

andreygetmanov requested a review from nicl-nno July 21, 2022 14:47

andreygetmanov force-pushed the multimodal_csv branch from c6a7966 to c11ba62 Compare July 21, 2022 14:49

andreygetmanov linked an issue Jul 21, 2022 that may be closed by this pull request

Impement automatic definition of text fields in dataset #754

Closed

aPovidlo self-requested a review July 21, 2022 14:59

andreygetmanov requested review from MorrisNein and Dreamlone July 21, 2022 14:59

nicl-nno requested changes Jul 21, 2022

View reviewed changes

MorrisNein requested changes Jul 21, 2022

View reviewed changes

fedot/core/data/multi_modal.py Outdated Show resolved Hide resolved

fedot/core/data/multi_modal.py Outdated Show resolved Hide resolved

aPovidlo requested changes Jul 21, 2022

View reviewed changes

fedot/core/data/multi_modal.py Outdated Show resolved Hide resolved

andreygetmanov force-pushed the multimodal_csv branch from 6844f9e to 93c3c58 Compare July 21, 2022 16:11

Dreamlone requested changes Jul 23, 2022

View reviewed changes

This comment was marked as resolved.

Sign in to view

nicl-nno approved these changes Aug 2, 2022

View reviewed changes

MorrisNein approved these changes Aug 3, 2022

View reviewed changes

andreygetmanov force-pushed the multimodal_csv branch 3 times, most recently from 59526fb to ff59b62 Compare August 12, 2022 09:58

andreygetmanov force-pushed the multimodal_csv branch from a108477 to a5c3751 Compare August 19, 2022 15:07

MorrisNein self-requested a review August 20, 2022 05:10

andreygetmanov force-pushed the multimodal_csv branch from a5c3751 to e2554e7 Compare August 22, 2022 14:22

andreygetmanov requested review from aPovidlo, nicl-nno and Dreamlone August 23, 2022 15:58

aPovidlo requested changes Aug 25, 2022

View reviewed changes

Dreamlone requested changes Aug 25, 2022

View reviewed changes

fedot/core/data/data_detection.py Outdated Show resolved Hide resolved

fedot/preprocessing/preprocessing.py Outdated Show resolved Hide resolved

test/unit/data/test_multimodal_data.py Outdated Show resolved Hide resolved

test/unit/data/test_multimodal_data.py Outdated Show resolved Hide resolved

andreygetmanov force-pushed the multimodal_csv branch 2 times, most recently from 87e78dd to 38c8796 Compare August 30, 2022 15:53

andreygetmanov requested a review from aPovidlo August 30, 2022 16:51

nicl-nno approved these changes Aug 31, 2022

View reviewed changes

andreygetmanov force-pushed the multimodal_csv branch from a2da375 to 64d320d Compare August 31, 2022 14:21

MorrisNein requested changes Aug 31, 2022

View reviewed changes

Dreamlone reviewed Sep 1, 2022

View reviewed changes

fedot/core/data/data_detection.py Outdated Show resolved Hide resolved

fedot/core/data/data_detection.py Outdated Show resolved Hide resolved

Dreamlone reviewed Sep 1, 2022

View reviewed changes

fedot/preprocessing/preprocessing.py Outdated Show resolved Hide resolved

fedot/preprocessing/preprocessing.py Show resolved Hide resolved

andreygetmanov added 16 commits September 2, 2022 16:25

MultiModalData class improvement

99121be

Now csv files with text and table data can be read just in one motion - from_csv method added - text fields are defined automatically - tests are added

minor changes:

47a53e9

- added docstring for _column_contains_text - multimodal_wine dataset is moved to more appropriate place - protected funcs of multi_modal.py are now protected

- added 2 tests on MultiModalData.from_csv use

b64819b

- fixed bug with incorrect multimodal data preprocessing

c1e7ecb

- fixed bug with incorrect multimodal data preprocessing

c240d69

- added data for tests

74731c2

- rewrote path in test_multimodal_data.py by Path

2fbc879

- task now is defined by str, not by Task class

fa8ab7b

- added substitution of nans to '' in text features

3bf4cbe

- refactoring of text preprocessing - refactoring of multimodal data test

- tests of multimodal data class are finished

9b9a0d0

- now if text column contains a lot of nans, it's dropped

- text and ts preparation methods are now in distinct classes inherit…

9d58be8

…ing DataDetector parent in data_detection.py

- refactoring of prepare_multimodal_ts_data method

20bb336

- added description for multimodal_text_num_example.py

- refactoring structure of test_text_data_only

ebd5f34

- refactoring structure of test_multimodal_data_from_csv

11e953a

- added docstrings

- table and text preprocessing are now distinguished for easier reada…

cad1d58

…bility

- removed duplicate of array_to_input_data method

ef31bae

- DataDetection classes now look better - refactoring of prepare_multimodal_ts logic - preprocessing is now excludes some data types using decorators

andreygetmanov force-pushed the multimodal_csv branch from 64d320d to ef31bae Compare September 2, 2022 13:29

- flake corrections

21f057a

MorrisNein approved these changes Sep 2, 2022

View reviewed changes

- minor changes

15a363b

Dreamlone approved these changes Sep 3, 2022

View reviewed changes

fedot/core/data/multi_modal.py Show resolved Hide resolved

fedot/core/data/data_detection.py Outdated Show resolved Hide resolved

fedot/preprocessing/preprocessing.py Outdated Show resolved Hide resolved

fedot/preprocessing/preprocessing.py Outdated Show resolved Hide resolved

andreygetmanov added 2 commits September 5, 2022 16:49

- decorators moved to the distinguished file

1265ed4

- minor changes

- minor changes

70e5af6

andreygetmanov merged commit e708a83 into master Sep 5, 2022

andreygetmanov deleted the multimodal_csv branch September 5, 2022 14:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MultiModalData class improvement #789

MultiModalData class improvement #789

andreygetmanov commented Jul 21, 2022

pep8speaks commented Jul 21, 2022 •

edited

Loading

codecov bot commented Jul 21, 2022 •

edited

Loading

This comment was marked as resolved.

aPovidlo Aug 25, 2022

andreygetmanov Aug 30, 2022

aPovidlo Aug 31, 2022

andreygetmanov Sep 1, 2022

MorrisNein left a comment •

edited

Loading

MultiModalData class improvement #789

MultiModalData class improvement #789

Conversation

andreygetmanov commented Jul 21, 2022

pep8speaks commented Jul 21, 2022 • edited Loading

Comment last updated at 2022-09-05 14:00:25 UTC

codecov bot commented Jul 21, 2022 • edited Loading

Codecov Report

This comment was marked as resolved.

aPovidlo Aug 25, 2022

Choose a reason for hiding this comment

andreygetmanov Aug 30, 2022

Choose a reason for hiding this comment

aPovidlo Aug 31, 2022

Choose a reason for hiding this comment

andreygetmanov Sep 1, 2022

Choose a reason for hiding this comment

MorrisNein left a comment • edited Loading

Choose a reason for hiding this comment

pep8speaks commented Jul 21, 2022 •

edited

Loading

codecov bot commented Jul 21, 2022 •

edited

Loading

MorrisNein left a comment •

edited

Loading