Add IOptTuner #1102

YamLyubov · 2023-05-24T18:10:41Z

Main changes:

IOptTuner was added. Example: examples/simple/pipeline_tuning_with_iopt.py

The way search space is set have changed
Previously:

{'adareg': {
              'learning_rate': (hp.loguniform, [np.log(1e-3), np.log(1)]),
              'loss': (hp.choice, [["linear", "square", "exponential"]])
}, ... }

Now (IOpt needs info about the type of the parameter: continuous, discete, categorical):

{'adareg': {
              'learning_rate': {
                  'hyperopt-dist': hp.loguniform,
                  'sampling-scope': [1e-3, 1],
                  'type': 'continuous'},
              'loss': {
                  'hyperopt-dist': hp.choice,
                  'sampling-scope': [["linear", "square", "exponential"]],
                  'type': 'categorical'}
}, ... }

TunerBuilder changed slightly: now you can pass additional parameters specific for a tuner with .with_additional_params() method.

Implementation of IOptTuner - aimclub/GOLEM#70
New documentation for hyperparameters tuning (FEDOT) - https://fedot--1102.org.readthedocs.build/en/1102/advanced/hyperparameters_tuning.html
New documentation for hyperparameters tuning (GOLEM) - https://thegolem--70.org.readthedocs.build/en/70/api/tuning.html

aim-pep8-bot · 2023-05-24T18:10:49Z

Hello @YamLyubov! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2023-06-15 15:10:28 UTC

codecov · 2023-05-24T18:34:41Z

Codecov Report

Merging #1102 (39e4878) into master (40312cb) will increase coverage by 0.21%.
The diff coverage is 88.00%.

❗ Current head 39e4878 differs from pull request most recent head e05dc7a. Consider uploading reports for the commit e05dc7a to get more accurate results

@@            Coverage Diff             @@
##           master    #1102      +/-   ##
==========================================
+ Coverage   78.67%   78.88%   +0.21%     
==========================================
  Files         130      130              
  Lines        9308     9306       -2     
==========================================
+ Hits         7323     7341      +18     
+ Misses       1985     1965      -20

Impacted Files	Coverage Δ
fedot/api/api_utils/api_composer.py	`94.84% <ø> (ø)`
...implementations/data_operations/text_pretrained.py	`41.30% <0.00%> (-6.53%)`	⬇️
fedot/core/pipelines/tuning/tuner_builder.py	`83.95% <75.00%> (-0.20%)`	⬇️
fedot/core/data/data.py	`71.28% <100.00%> (ø)`
...t/core/operations/evaluation/gpu/classification.py	`75.00% <100.00%> (ø)`
fedot/core/operations/evaluation/gpu/clustering.py	`51.35% <100.00%> (ø)`
fedot/core/operations/evaluation/gpu/common.py	`46.93% <100.00%> (ø)`
fedot/core/operations/evaluation/gpu/regression.py	`71.42% <100.00%> (ø)`
...lementations/data_operations/text_preprocessing.py	`39.43% <100.00%> (ø)`
...aluation/operation_implementations/models/keras.py	`30.09% <100.00%> (ø)`
... and 2 more

... and 3 files with indirect coverage changes

maypink

а еще хотела спросить: где-то есть сравнение тюнеров? как вообще пользователю понять, когда какой использовать?

maypink · 2023-06-01T09:53:09Z

fedot/core/pipelines/tuning/search_space.py

@@ -25,270 +25,706 @@ def __init__(self,
    def get_parameters_dict(self):
        parameters_per_operation = {
            'kmeans': {
-                'n_clusters': (hp.uniformint, [2, 7])
+                'n_clusters': {


может сделать так, чтобы тип передаваемых параметров автоматически определялся исходя из самих параметров?

Изначально тоже думала о таком решении, но потом все-таки решила оставить так. В таком случае есть возможность не задавать распределение для hyperopt, если его использование не планируется. Так же показалось, что такое задание параметров может вводить в заблуждение, что iOpt будет семплировать именно из такого распределения. Ну и в целом показалось костыльным определение типа по распределению

maypink · 2023-06-01T09:54:04Z

fedot/core/pipelines/tuning/hyperparams.py

-        if isinstance(self.current_params, str):
-            # TODO 'default_params' - need to process
+        try:
+            current_value = self.current_params.get(parameter_name)


а что теперь будет, если строка передастся, а не словарь?

Сейчас такое по идее не должно происходить, так как в ParametersChanger мы передаем параметры, которые хранятся в Node.parameters. Node.parameters - это всегда словарь.

Строка 'default_params' = DEFAULT_PARAMS_STUB преобразуется в словарь при создании ноды (https://github.com/aimclub/FEDOT/blob/master/fedot/core/pipelines/node.py#LL61C1-L75C103), и если мы сеттим parameters (https://github.com/aimclub/FEDOT/blob/master/fedot/core/pipelines/node.py#LL327C1-L341C64)

maypink · 2023-06-02T14:59:55Z

позапускай плиз в этом пр примеры, видимо не все импорты поправили -- сейчас падает как минимум пример с регрессией в мастере

YamLyubov · 2023-06-05T14:23:01Z

а еще хотела спросить: где-то есть сравнение тюнеров? как вообще пользователю понять, когда какой использовать?

Добавила в документацию небольшое сравнение https://fedot--1102.org.readthedocs.build/en/1102/advanced/hyperparameters_tuning.html

…ot work)

deleted redundant files corrected typos simplifeid code remove redundant correct pep8 issues add example add mutations correct visualization fiting process improve visualization add partial in solver.py add example in .py format examples/confidence_intervals move prediction intervals in core/pipelines delete old examples add unit tests Refactoring of ApiParams and ApiMetrics (aimclub#1041) * WIP refactor ApiParams * Remove explicit ApiParams initialization * Move all params initialization to ApiParams * Minor changes * Remove _divide_params * WIP create ApiParamsBuilder * Add ApiParamsBuilder * Minor * Rename history_folder to history_dir * Remove train_data from ApiParams * Fixes after rebase * Move obtain_metric to ApiMetrics * Fix plot_pareto * Fix Fedot.tune * Refactor Fedot.get_metrics * Fix Fedot.tune * Fix metric names * Fixes after rebase * Remove ApiParamsBuilder * Structure parameters in Fedot docstring * Refactor init_composer_requirements * Refactor init_optimizer_params * Refactor init_optimizer_params * Fix docstrings * Add tests for ApiParamsRepository * Minors * Fix test_api_params * Review fixes * Review fixes * Review fixes Hotfix of pipeline import export example (aimclub#1064) meta rules (aimclub#1057) * initial assumption * final architecture * add meta rules * minor * simplify * add meta rule * fix with cv folds * minors * fix types * minor * golem fixes * add log messages * pep8 * remove log file Fix initial assumptions as list of pipelines (aimclub#1070) `Fedot(..., initial_assumption=...)` is expected to get a sequence of pipelines and pass them as initial graphs to an optimizer via composer. Moreover, FEDOT itself generates more than one initial assumption by default. As the result of this bug, composer passed only one of the initial assumptions to an optimizer. This PR fixes the bug and adds the corresponding test. tests fix (aimclub#1073) * remove some assumptions * update requirements * fix pep8 * update to golem Docs updated, badges added (aimclub#1072) * Docs updated, badges added * Mirror workflow fix Improve API documentation (aimclub#1067) - Moved type hints from method headers to corresponding parameters. - Allowed referencing GOLEM objects in FEDOT documentation. - Replaced all url links to documentation pages with sphinx references - it fixed some broken links. - FEDOT now uses its own directory for cache, instead of using GOLEM folder. - Documented `**composer_tuner_params` of `Fedot` with type hints and default values. All parameters with no usage examples are placed to the separate issue aimclub#1076 minor meta fix (aimclub#1078) * minor fix * indent fix * minor Add catboost to default initial assumptions (aimclub#1081) * add catboost to default initial assumptions for classification * restrict mutating loss function in CatBoost * evaluate f1 as expected in the example * pep8 F1 averaging fix (aimclub#1083) Minor logging fix (aimclub#1082) * fix print instead of logging for memory consumption 1059 timestamp bug (aimclub#1065) 879 FEDOT features (aimclub#1075) describe framework's features add example for surrogate optimizer (aimclub#1085) Example with surrogate optimizer was added. External parameters field was removed from api (now we should use partial) related pull request in GOLEM aimclub/GOLEM#82 has_one_root fix (aimclub#1091) * has_one_root fix * test fix Remove outdated test handled in thegolem (aimclub#1101) 358 Reduce execution time for unit tests (aimclub#1098) Update RTD benchmarks tabular data page (aimclub#1099) * +csv support Golem update requirements (aimclub#1088) * Upd RemoveType in Advisor (golem sync) * Upd requirements.txt for stable GOLEM * Fix few imports Add ts bench (aimclub#1104) Add results from ts benchmark Release 0.7.1 and test workflow updates (aimclub#1105) * Upd release version * Add pre-release tests actions on 'release' branch; Disable auto-publish * Upd GOLEM version * Remove manual-build.yml (dup of integration-build.yml) * Add integration tests badge to README * Revert "Upd GOLEM version" (for PR in master) This reverts commit 257ff16. Hotfix some integration tests for release 071 (aimclub#1107) * Fix integration test of ApiParams * Fix integration test of composition_time * Fix integration test of metocean_forecasting * Fix integration test of nemo_multiple.py * Workaround for sqlite exception raised in tests * pep8 fixes * fix different seed in quality imporvement tests * simplify condition * remove test that barely tested anything * fix condition * remove seed from example * fix typo parallel cache files test fix (aimclub#1109) Add IOptTuner (aimclub#1102) * Refactor search space * Fix ParametersChanges * Redact tuner builder * PEP 8 * Fix examples and correct tuning docs * Add IOpt example to docs * Add simple IOpt example * Fix integration tests (6 steal not work) * Fix integration test test_tuner_builder_with_custom_params (6 steal not work) * Fix pep8 * Add tuners comparison in docs * Update golem version in requirements * Fix warn_requirements * Fix warn_requirements * Fix table in tuning docs * Fix credit_scoring_problem_multiobj.py * Fix extra requirements * Edit docs * Fix table in docs * Change requirements * Test requirements * Set stable branch + h2o benchmark's tabular data values (aimclub#1106) * +h2o vals * change csv to html * specified table uuid * change max values style * add df to html converter --------- Co-authored-by: nicl-nno <[email protected]> Added results for multimodal benchmark into FEDOT.docs (aimclub#1115) * - added results for multimodal benchmark ... add docstrings and type hints add prediction intervals unit test simplified solvers code correct pep issues add class defining PredictionIntervals params fix MutationStrength import issue correct test_data path update imports updated unit tests correct type-hint small corrections take short ts for unit test update pred_int test change ql_models to 'max' in pred_ints test update params for ql method ... update example update tests add base_quantiles visualization fix import get_base_quantiles update examples and deleted ql from tests update pep pep issues pep pep update requirements.txt

YamLyubov mentioned this pull request May 26, 2023

Зацикливание при бесконечном значении функции aimclub/iOpt#132

Closed

maypink self-requested a review May 26, 2023 11:23

maypink requested changes Jun 1, 2023

View reviewed changes

YamLyubov force-pushed the add-iopt branch 2 times, most recently from 9f839b7 to ecd5e0a Compare June 5, 2023 13:39

YamLyubov requested a review from maypink June 5, 2023 14:23

YamLyubov force-pushed the add-iopt branch from 80dbdbd to ecab9bc Compare June 8, 2023 09:06

YamLyubov added 18 commits June 8, 2023 12:07

Refactor search space

377c31b

Fix ParametersChanges

ed39831

Redact tuner builder

fed8cc1

PEP 8

c757b28

Fix examples and correct tuning docs

e3f10cb

Add IOpt example to docs

b366320

Add simple IOpt example

bae6d38

Fix integration tests (6 steal not work)

1d61d0a

Fix integration test test_tuner_builder_with_custom_params (6 steal n…

6719dcf

…ot work)

Fix pep8

47c7814

Add tuners comparison in docs

b79fd61

Update golem version in requirements

afa485a

Fix warn_requirements

3e6f449

Fix warn_requirements

c5e45d1

Fix table in tuning docs

510fd9d

Fix credit_scoring_problem_multiobj.py

2570206

Fix extra requirements

5dce3b3

Edit docs

49db14c

YamLyubov force-pushed the add-iopt branch from ecab9bc to 49db14c Compare June 8, 2023 09:08

Fix table in docs

6366946

maypink approved these changes Jun 8, 2023

View reviewed changes

YamLyubov added 3 commits June 13, 2023 14:55

Change requirements

6033465

Test requirements

39e4878

Set stable branch

e05dc7a

YamLyubov merged commit 16d507f into master Jun 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add IOptTuner #1102

Add IOptTuner #1102

YamLyubov commented May 24, 2023 •

edited

Loading

aim-pep8-bot commented May 24, 2023 •

edited

Loading

codecov bot commented May 24, 2023 •

edited

Loading

maypink left a comment

maypink Jun 1, 2023

YamLyubov Jun 5, 2023

maypink Jun 1, 2023

YamLyubov Jun 1, 2023

maypink commented Jun 2, 2023

YamLyubov commented Jun 5, 2023

Add IOptTuner #1102

Add IOptTuner #1102

Conversation

YamLyubov commented May 24, 2023 • edited Loading

aim-pep8-bot commented May 24, 2023 • edited Loading

Comment last updated at 2023-06-15 15:10:28 UTC

codecov bot commented May 24, 2023 • edited Loading

Codecov Report

maypink left a comment

Choose a reason for hiding this comment

maypink Jun 1, 2023

Choose a reason for hiding this comment

YamLyubov Jun 5, 2023

Choose a reason for hiding this comment

maypink Jun 1, 2023

Choose a reason for hiding this comment

YamLyubov Jun 1, 2023

Choose a reason for hiding this comment

maypink commented Jun 2, 2023

YamLyubov commented Jun 5, 2023

YamLyubov commented May 24, 2023 •

edited

Loading

aim-pep8-bot commented May 24, 2023 •

edited

Loading

codecov bot commented May 24, 2023 •

edited

Loading