Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pin optuna to latest version 4.2.0 #83

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Pin optuna to latest version 4.2.0 #83

wants to merge 1 commit into from

Conversation

pyup-bot
Copy link
Collaborator

This PR pins optuna to the latest release 4.2.0.

Changelog

4.2.0

This is the release note of [v4.2.0](https://github.com/optuna/optuna/releases/tag/v4.2.0). In conjunction with the Optuna release, OptunaHub 0.2.0 is released. Please refer to [the release note of OptunaHub 0.2.0](https://github.com/optuna/optunahub/releases/tag/v0.2.0) for more details.

Highlights of this release include:

- 🚀gRPC Storage Proxy for Scalable Hyperparameter Optimization
- 🤖 SMAC3: Support for New State-of-the-art Optimization Algorithm by AutoML.org (automl)
- 📁 OptunaHub Now Supports Benchmark Functions
- 🧑‍💻 Gaussian Process-Based Bayesian Optimization with Inequality Constraints
- 🧑‍💻 c-TPE: Support Constrained TPESampler

Highlights

gRPC Storage Proxy for Scalable Hyperparameter Optimization


The gRPC storage proxy is a feature designed to support large-scale distributed optimization. As shown in the diagram below, gRPC storage proxy sits between the optimization workers and the database server, proxying the calls of Optuna’s storage APIs.

<img width="1241" alt="grpc-proxy" src="https://github.com/user-attachments/assets/00220bc6-75da-4ee3-b9ff-a2b3440ac207" />


In large-scale distributed optimization settings where hundreds to thousands of workers are operating, placing a gRPC storage proxy for every few tens can significantly reduce the load on the RDB server which would otherwise be a single point of failure. The gRPC storage proxy enables sharing the cache about Optuna studies and trials, which can further mitigate load. Please refer to [the official documentation](https://optuna.readthedocs.io/en/latest/reference/generated/optuna.storages.GrpcStorageProxy.html#optuna.storages.GrpcStorageProxy) for further details on how to utilize gRPC storage proxy.

SMAC3: Random Forest-Based Bayesian Optimization Developed by AutoML.org
[SMAC3](https://github.com/automl/SMAC3) is a hyperparameter optimization framework developed by [AutoML.org](http://automl.org/), one of the most influential AutoML research groups. The Optuna-compatible SMAC3 sampler is now available thanks to the contribution to OptunaHub by Difan Deng (dengdifan), one of the core members of AutoML.org. We can now use the method widely used in AutoML research and real-world applications from Optuna.

python
pip install optunahub smac
import optuna
import optunahub
from optuna.distributions import FloatDistribution

def objective(trial: optuna.Trial) -> float:
 x = trial.suggest_float("x", -10, 10)
 y = trial.suggest_float("y", -10, 10)
 return x**2 + y**2

smac_mod = optunahub.load_module("samplers/smac_sampler")
n_trials = 100
sampler = smac_mod.SMACSampler(
 {"x": FloatDistribution(-10, 10), "y": FloatDistribution(-10, 10)},
 n_trials=n_trials,
)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=n_trials)


Please refer to https://hub.optuna.org/samplers/smac_sampler/ for more details.

OptunaHub Now Supports Benchmark Functions
Benchmarking the performance of optimization algorithms is an essential process indispensable to the research and development of algorithms. The newly added OptunaHub Benchmarks in the latest version v0.2.0 of [optunahub](https://hub.optuna.org/) is a new feature for Optuna users to conduct benchmarks conveniently.

python
pip install optunahub>=4.2.0 scipy torch
import optuna
import optunahub

bbob_mod = optunahub.load_module("benchmarks/bbob")
smac_mod = optunahub.load_module("samplers/smac_sampler")
sphere2d = bbob_mod.Problem(function_id=1, dimension=2)

n_trials = 100
studies = []
for study_name, sampler in [
 ("random", optuna.samplers.RandomSampler(seed=1)),
 ("tpe", optuna.samplers.TPESampler(seed=1)),
 ("cmaes", optuna.samplers.CmaEsSampler(seed=1)),
 ("smac", smac_mod.SMACSampler(sphere2d.search_space, n_trials, seed=1)),
]:
 study = optuna.create_study(directions=sphere2d.directions,
     sampler=sampler, study_name=study_name)
 study.optimize(sphere2d, n_trials=n_trials)
 studies.append(study)

optuna.visualization.plot_optimization_history(studies).show()


In the above sample code, we compare and display the performance of the four kinds of samplers using a two-dimensional Sphere function, which is part of a group of benchmark functions widely used in the black-box optimization research community known as [Blackbox Optimization Benchmarking (BBOB)](https://hub.optuna.org/benchmarks/bbob/).

<img width="1250" alt="bbob" src="https://github.com/user-attachments/assets/89b09090-dd37-4679-9fa8-af987f138dcc" />


Gaussian Process-Based Bayesian Optimization with Inequality Constraints
We worked on its extension and adapted `GPSampler` to constrained optimization in Optuna v4.2.0 since Gaussian process-based Bayesian optimization is a very popular method in various research fields such as aircraft engineering and materials science. We show the basic usage below.

python
pip install optuna>=4.2.0 scipy torch
import numpy as np
import optuna

def objective(trial: optuna.Trial) -> float:
 x = trial.suggest_float("x", 0.0, 2 * np.pi)
 y = trial.suggest_float("y", 0.0, 2 * np.pi)
 c = float(np.sin(x) * np.sin(y) + 0.95)
 trial.set_user_attr("c", c)
 return float(np.sin(x) + y)

def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
 return (trial.user_attrs["c"],)

sampler = optuna.samplers.GPSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=50)


Please try out `GPSampler` for constrained optimization especially when only a small number of trials are available!

c-TPE: Support Constrained TPESampler

![c-TPE](https://github.com/user-attachments/assets/03bfbbed-063e-4781-9589-88408189a3be)

Although Optuna has supported constrained optimization for `TPESampler`, which is the default Optuna sampler, since v3.0.0, its algorithm design and performance comparison have not been verified academically. OptunaHub now supports [c-TPE](https://arxiv.org/abs/2211.14411), which is another constrained optimization method for `TPESampler`. Importantly, the algorithm design and its performance comparison are publicly reviewed to be accepted to IJCAI, a top-tier AI international conference. Please refer to https://hub.optuna.org/samplers/ctpe/ for details.

New Features

- Enable `GPSampler` to support constraint functions (5715)
- Update output format options in CLI to include the `value` choice (5822, thanks iamarunbrahma!)
- Add gRPC storage proxy server and client (5852)

Enhancements

- Introduce client-side cache in `GrpcStorageProxy` (5872)

Bug Fixes

- Fix CI (https://github.com/optuna/optuna-integration/pull/185)
- Fix ticks in Matplotlib contour plot (5778, thanks sulan!)
- Adding check in `cli.py` to handle an empty database (5828, thanks willdavidson05!)
- Avoid the input validation fail in Wilcoxon signed ranked test for Scipy 1.15 (5912)
- Fix the default sampler of `load_study` function (5924)

Documentation

- Update OptunaHub example in README (5763)
- Update `distributions.rst` to list deprecated distribution classes (5764)
- Remove deprecation comment for `step` in `IntLogUniformDistribution` (5767)
- Update requirements for OptunaHub in README (5768)
- Use inline code rather than italic for `step` (5769)
- Add notes to `ask_and_tell` tutorial - batch optimization recommendations (5817, thanks SimonPop!)
- Fix the explanation of returned values of `get_trial_params` (5820)
- Introduce `sphinx-notfound-page` for better 404 page (5898)
- Follow-up 5872: Update the docstring of `run_grpc_proxy_server` (5914)
- Modify doc-string of gRPC-related modules (5916)

Examples

- Adapt docker recipes to Python 3.11 (https://github.com/optuna/optuna-examples/pull/292)
- Add version constraint for `wandb` (https://github.com/optuna/optuna-examples/pull/293)

Tests

- Add unit tests for `retry_history` method in `RetryFailedTrialCallback` (5865, thanks iamarunbrahma!)
- Add tests for value format in CLI (5866)
- Import grpc lazy to fix the CI (5878)

Code Fixes

- Fix annotations for `distributions.py` (5755, thanks KannanShilen!)
- Simplify type annotations for `tests/visualization_tests/test_pareto_front.py` (5756, thanks boringbyte!)
- Fix type annotations for `test_hypervolume_history.py` (5760, thanks boringbyte!)
- Simplify type annotations for `tests/test_cli.py` (5765, thanks boringbyte!)
- Simplify type annotations for `tests/test_distributions.py` (5773, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/test_qmc.py` (5775, thanks boringbyte!)
- Simplify type annotations for `tests/sampler_tests/tpe_tests/test_sampler.py` (5779, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py` (5781, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/tpe_tests/test_parzen_estimator.py` (5782, thanks boringbyte!)
- Simplify type annotations for `tests/storage_tests/journal_tests/test_journal.py` (5783, thanks boringbyte!)
- Simplify type annotations for `tests/storage_tests/rdb_tests/create_db.py` (5784, thanks boringbyte!)
- Simplify type annotations for `tests/storage_tests/rdb_tests/` (5785, thanks boringbyte!)
- Simplify type annotations for `tests/test_deprecated.py` (5786, thanks boringbyte!)
- Simplify type annotations for `tests/test_convert_positional_args.py` (5787, thanks boringbyte!)
- Simplify type annotations for `tests/importance_tests/test_init.py` (5790, thanks boringbyte!)
- Simplify type annotations for `tests/storages_tests/test_storages.py` (5791, thanks boringbyte!)
- Simplify type annotations for `tests/storages_tests/test_heartbeat.py` (5792, thanks boringbyte!)
- Simplify type annotations `optuna/cli.py` (5793, thanks willdavidson05!)
- Simplify type annotations for `tests/trial_tests/test_frozen.py` (5794, thanks boringbyte!)
- Simplify type annotations for `tests/trial_tests/test_trial.py` (5795, thanks boringbyte!)
- Simplify type annotations for `tests/trial_tests/test_trials.py` (5796, thanks boringbyte!)
- Refactor some funcs in NSGA-III (5798)
- Simplify type annotations `optuna/_transform.py` (5799, thanks JLX0!)
- Make the assertion messages in `test_trial.py` readable (5800)
- Simplify type annotations for `tests/pruners_tests/test_hyperband.py` (5801, thanks boringbyte!)
- Simplify type annotations for `tests/pruners_tests/test_median.py` (5802, thanks boringbyte!)
- Simplify type annotations for `tests/pruners_tests/test_patient.py` (5803, thanks boringbyte!)
- Use `study.ask()` in tests instead of `create_new_trial` (5807, thanks unKnownNG!)
- Simplify type annotations for `tests/pruners_tests/test_percentile.py` (5808, thanks boringbyte!)
- Simplify type annotations for `tests/pruners_tests/test_successive_halving.py` (5809, thanks boringbyte!)
- Simplify type annotations for `tests/study_tests/test_optimize.py` (5810, thanks boringbyte!)
- Simplify type annotations for `tests/hypervolume_tests/test_hssp.py` (5812, thanks boringbyte!)
- Refactor the MOTPE split (5813)
- Simplify type annotations for `optuna/_callbacks.py` (5818, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/_random.py` (5819, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/_gp/sampler.py` (5823, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_sbx.py` (5824, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_spx.py` (5825, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_undx.py` (5826, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossovers/_vsbx.py` (5827, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/nsgaii/_crossover.py` (5831, thanks boringbyte!)
- Simplify type annotations for `optuna/samplers/testing/threading.py` (5832, thanks boringbyte!)
- Simplify type annotations for `optuna/pruners/_patient.py` (5833, thanks boringbyte!)
- Use `study.ask()` in `tests/pruners_tests/test_percentile.py` (5834, thanks fusawa-yugo!)
- Simplify type annotations for `optuna/search_space/group_decomposed.py` (5836, thanks boringbyte!)
- Simplify type annotations for `optuna/search_space/intersection.py` (5837, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_base.py` (5838, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_redis.py` (5840, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_storage.py` (5841, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_dataframe.py` (5842, thanks boringbyte!)
- Fix logger.warn() deprecation warning in GP module (5843, thanks iamarunbrahma!)
- Simplify type annotations for `optuna/study/_optimize.py` (5844, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_tell.py` (5845, thanks boringbyte!)
- Fix `mypy` errors due to `numpy` 2.2.0 (5848)
- Simplify type annotations for `optuna/visualization/matplotlib/_contour.py` (5851, thanks boringbyte!)
- Refactor the fix for MyPy errors due to NumPy v2.2.0 (5853)
- Simplify type annotations for `optuna/visualization/matplotlib/_parallel_coordinate.py` (5854, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/matplotlib/_param_importances.py` (5855, thanks boringbyte!)
- Use `study.ask()` in `tests/pruners_tests/test_successive_halving.py` (5856, thanks willdavidson05!)
- Simplify type annotations for `optuna/visualization/matplotlib/_pareto_front.py` (5857, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/matplotlib/_rank.py` (5858, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/matplotlib/_slice.py` (5859, thanks boringbyte!)
- Refactor plot contour (5867)
- Refactor MOTPE weighting (5871)
- Simplify type annotations for `optuna/visualization/_utils.py` (5876, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_contour.py` (5877, thanks boringbyte!)
- Simplify type annotations for `optuna/_gp/gp.py` (5879, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_tell.py` (5880, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_heartbeat.py` (5882, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/storage.py` (5883, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/alembic/versions/v3.0.0.a.py` (5884, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/alembic/versions/v3.0.0.c.py` (5885, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/_rdb/alembic/versions/v3.0.0.d.py` (5886, thanks boringbyte!)
- Simplify type annotations for `optuna/_deprecated.py` (5887, thanks boringbyte!)
- Simplify type annotations for `optuna/_experimental.py` (5888, thanks boringbyte!)
- Simplify type annotations for `optuna/_imports.py` (5889, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_slice.py` (5894, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_parallel_coordinate.py` (5895, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_rank.py` (5896, thanks boringbyte!)
- Simplify type annotations for `optuna/visualization/_param_importances.py` (5897, thanks boringbyte!)
- Simplify type annotations for `optuna/_callbacks.py` (5899, thanks boringbyte!)
- Simplify type annotations for `optuna/storages/journal/_file.py` (5900, thanks boringbyte!)
- Simplify type annotations for `tests/storages_tests/test_with_server.py` (5901, thanks boringbyte!)
- Simplify type annotations for `tests/test_multi_objective.py` (5902, thanks boringbyte!)
- Simplify type annotations for `tests/artifacts_tests/test_gcs.py` (5903, thanks boringbyte!)
- Simplify type annotations for `tests/samplers_tests/test_grid.py` (5904, thanks boringbyte!)
- Simplify type annotations for `optuna/study/_dataframe.py` (5905, thanks boringbyte!)
- Simplify type annotations for `tests/visualization_tests/test_optimization_history.py` (5906, thanks boringbyte!)
- Simplify type annotations for `tests/visualization_tests/test_intermediate_plot.py` (5907, thanks boringbyte!)
- Simplify type annotations for multiple test files in `tests/visualization_tests/` (5908, thanks boringbyte!)
- Avoid the port number conflict in the copy study tests (5913)
- Simplify annotations in `tests/study_tests/test_study.py` (5923, thanks sawa3030!)

Continuous Integration

- Fix a GitHub action workflow for publishing to PyPI (https://github.com/optuna/optuna-integration/pull/181)
- Fix CI by adding a version constraint (https://github.com/optuna/optuna-integration/pull/186)
- Rename `test_cache_is_invalidated` and remove `assert study._thread_local.cached_all_trials is None` (5733)
- Use Python 3.12 for docs build CI (5742)
- Fix a GitHub action workflow for publishing to PyPI (5759)
- Install older `kaleido` to fix CI errors (5771)
- Add Python 3.12 to the Docker image CI (5789)
- Move `mypy` related entries in setup.cfg to `pyproject.toml` (5861)

Other

- Bump up version number to v4.2.0.dev (https://github.com/optuna/optuna-integration/pull/178)
- Bump up version number to 4.2.0 (https://github.com/optuna/optuna-integration/pull/191)
- Add `CITATION.cff` (5746)
- Update news to add the autosampler article (5748)
- Bump the version up to v4.2.0.dev (5752)
- Update the news section for Optuna 4.1 release (5758)
- Update pre-commit configuration file (5847)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, JLX0, KannanShilen, SimonPop, boringbyte, c-bata, fusawa-yugo, gen740, himkt, iamarunbrahma, kAIto47802, ktns, mist714, nabenabe0928, not522, nzw0301, porink0424, sawa3030, sulan, unKnownNG, willdavidson05, y0z

4.1.0

This is the release note of [v4.1.0](https://github.com/optuna/optuna/milestone/64?closed=1). Highlights of this release include:
- 🤖 AutoSampler: Automatic Selection of Optimization Algorithms 
- 🚀 More scalable RDB Storage Backend
- 🧑‍💻 Five New Algorithms in OptunaHub (MO-CMA-ES, MOEA/D, etc.)
- 🐍 Support Python 3.13

The updated list of tested and supported Python releases is as follows:
- [Optuna 4.1](https://github.com/optuna/optuna/releases/tag/v4.1.0): supported by Python 3.8 - 3.13
- [Optuna Integration 4.1](https://github.com/optuna/optuna-integration/releases/tag/v4.1.0): supported by Python 3.8 - 3.12
- [Optuna Dashboard 0.17.0](https://github.com/optuna/optuna-dashboard/releases/tag/v0.17.0): supported by Python 3.8 - 3.13

Highlights

AutoSampler: Automatic Selection of Optimization Algorithms

<img width="750" alt="Blog-1" src="https://github.com/user-attachments/assets/f3ecd366-7e7b-49d6-ae38-f35301fa7d11">

[AutoSampler](https://hub.optuna.org/samplers/auto_sampler/) automatically selects a sampler from those implemented in Optuna, depending on the situation. Using AutoSampler, as in the code example below, users can achieve optimization performance equal to or better than Optuna's default without being aware of which optimization algorithm to use.


$ pip install optunahub cmaes torch scipy


python
import optuna
import optunahub

auto_sampler_module = optunahub.load_module("samplers/auto_sampler")
study = optuna.create_study(sampler=auto_sampler_module.AutoSampler())


See the [Medium blog post](https://medium.com/optuna/autosampler-automatic-selection-of-optimization-algorithms-in-optuna-1443875fd8f9) for details.


Enhanced RDB Storage Backend

This release incorporates comprehensive performance tuning on Optuna’s [RDBStorage](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.storages.RDBStorage.html), leading to significant performance improvements. The table below shows the comparison results of execution times between versions 4.0 and 4.1.

|  trials | v4.0.0 | v4.1.0 | Diff |
| --- |  --- |  --- |  --- |

4.0.0

Here is the release note of [v4.0.0](https://github.com/optuna/optuna/milestone/63?closed=1). Please also check out the [release blog post](https://medium.com/optuna/announcing-optuna-4-0-3325a8420d10).

If you want to update the Optuna version of your existing projects to v4.0, please see the [migration guide](https://github.com/optuna/optuna/discussions/5573).

We have also published blog posts about the development items. Please check them out!
- [OptunaHub, a Feature-Sharing Platform for Optuna, Now Available in Official Release!](https://medium.com/optuna/optunahub-a-feature-sharing-platform-for-optuna-now-available-in-official-release-4b99efe9934d)
- [File Management during LLM (Large Language Model) Trainings by Optuna v4.0.0 Artifact Store](https://medium.com/optuna/file-management-during-llm-large-language-model-trainings-by-optuna-v4-0-0-artifact-store-5bdd5112f3c7)
- [Significant Speed Up of Multi-Objective TPESampler in Optuna v4.0.0](https://medium.com/optuna/significant-speed-up-of-multi-objective-tpesampler-in-optuna-v4-0-0-2bacdcd1d99b)

Highlights

Official Release of Feature-Sharing Platform OptunaHub
We officially released [OptunaHub](https://hub.optuna.org/), a feature-sharing platform for Optuna. A large number of optimization and visualization algorithms are available in OptunaHub. Contributors can easily register their methods and deliver them to Optuna users around the world.

Please also read the [OptunaHub release blog post](https://medium.com/optuna/optunahub-a-feature-sharing-platform-for-optuna-now-available-in-official-release-4b99efe9934d).

![optunahub](https://github.com/user-attachments/assets/7995c15b-89e6-474a-ba75-56888ae67230)


Enhanced Experiment Management Feature: Official Support of Artifact Store
Artifact Store is a file management feature for files generated during optimization, dubbed artifacts. In Optuna v4.0, we stabilized the existing file upload API and further enhanced the usability of Artifact Store by adding some APIs such as the artifact download API. We also added features to show JSONL and CSV files on [Optuna Dashboard](https://github.com/optuna/optuna-dashboard) in addition to the existing support for images, audio, and video. With this official support, the API backward compatibility will be guaranteed.

For more details, please check the [blog post](https://medium.com/optuna/file-management-during-llm-large-language-model-trainings-by-optuna-v4-0-0-artifact-store-5bdd5112f3c7).

![artifact](https://github.com/user-attachments/assets/1556ef5a-852a-4256-b2e7-32c7d2eef253)




`JournalStorage`: Official Support of Distributed Optimization via Network File System
`JournalStorage` is a new Optuna storage experimentally introduced in Optuna v3.1 (see the [blog post](https://medium.com/optuna/distributed-optimization-via-nfs-using-optunas-new-operation-based-logging-storage-9815f9c3f932) for details). Optuna has `JournalFileBackend`, a storage backend for various file systems. It can be used on NFS, allowing Optuna to scale to multiple nodes.


In Optuna v4.0, the API for `JournalStorage` has been reorganized, and `JournalStorage` is officially supported. This official support guarantees its backward compatibility from v4.0. For details on the API changes, please refer to the [Optuna v4.0 Migration Guide](https://github.com/optuna/optuna/discussions/5573).

python
import optuna
from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend

def objective(trial: optuna.Trial) -> float:
 ...

storage = JournalStorage(JournalFileBackend("./optuna_journal_storage.log"))
study = optuna.create_study(storage=storage)
study.optimize(objective)



Significant Speedup of Multi-Objective `TPESampler`
Before v4.0, the multi-objective `TPESampler` sometimes limits the number of trials during optimization due to the sampler bottleneck after a few hundred trials. Optuna v4.0 drastically improves the sampling speed, e.g., 300 times faster for three objectives with 200 trials, and enables users to handle much more trials. Please check the [blog post](https://medium.com/optuna/significant-speed-up-of-multi-objective-tpesampler-in-optuna-v4-0-0-2bacdcd1d99b) for details.

Introduction of a New `Terminator` Algorithm
Optuna `Terminator` was originally introduced for hyperparameter optimization of machine learning algorithms using cross-validation. To accept broader use cases, Optuna v4.0 introduced the Expected Minimum Model Regret (EMMR) algorithm. Please refer to the [`EMMREvaluator` document](https://optuna.readthedocs.io/en/v4.0.0/reference/generated/optuna.terminator.EMMREvaluator.html) for details.

Enhancements of Constrained Optimization
We have gradually expanded the support for constrained optimization. In v4.0, [`study.best_trial`](https://optuna.readthedocs.io/en/v4.0.0/reference/generated/optuna.study.Study.html#optuna.study.Study.best_trial) and [`study.best_trials`](https://optuna.readthedocs.io/en/v4.0.0/reference/generated/optuna.study.Study.html#optuna.study.Study.best_trials) start to support constraint optimization. They are guaranteed to satisfy the constraints, which was not the case previously.



Breaking Changes

Optuna removes deprecated features in major releases. To prevent users' code from suddenly breaking, we take a long interval between when a feature is deprecated and when it is removed. By default, features are removed when the major version has increased by two since the feature was deprecated. For this reason, the main target features for removal in v4.0 were deprecated at v2.x. Please refer to the [migration guide](https://github.com/optuna/optuna/discussions/5573) for the removed features list.

- Delete deprecated three integrations, `skopt`, `catalyst`, and `fastaiv1` (https://github.com/optuna/optuna-integration/pull/114)
- Remove deprecated `CmaEsSampler` from integration (https://github.com/optuna/optuna-integration/pull/116)
- Remove verbosity of `LightGBMTuner` (https://github.com/optuna/optuna-integration/pull/136)
- Move positional args of `LightGBMTuner` (https://github.com/optuna/optuna-integration/pull/138)
- Remove `multi_objective` (5390)
- Delete deprecated `_ask` and `_tell` (5398)
- Delete deprecated `--direction(s)` arguments in the `ask` command (5405)
- Delete deprecated three integrations, `skopt`, `catalyst`, and `fastaiv1` (5407)
- Remove the default normalization of importance in f-ANOVA (5411)
- Remove `samplers.intersection` (5414)
- Drop implicit create-study in `ask` command (5415)
- Remove deprecated `study optimize` CLI command (5416)
- Remove deprecated `CmaEsSampler` from integration (5417)
- Support constrained optimization in `best_trial` (5426)
- Drop `--study` in `cli.py` (5430)
- Deprecate `constraints_func` in `plot_pareto_front` function (5455)
- Rename some class names related to `JournalStorage` (5539)
- Remove `optuna.samplers.MOTPESampler` (5640)

New Features

- Add Comet ML integration (https://github.com/optuna/optuna-integration/pull/63, thanks caleb-kaiser!)
- Add Knowledge Gradient candidates functions (https://github.com/optuna/optuna-integration/pull/125, thanks alxhslm!)
- Add `is_exhausted()` function in the `GridSampler` class (5306, thanks aaravm!)
- Remove experimental from plot (5413)
- Implement `download_artifact` (5448)
- Add a function to list linked artifact information (5467)
- Stabilize artifact APIs (5567)
- Stabilize `JournalStorage` (5568)
- Add `EMMREvaluator` and `MedianErrorEvaluator` (5602)

Enhancements

- Pass two arguments to the forward of `ConstrainedMCObjective` to support `botorch=0.10.0` (https://github.com/optuna/optuna-integration/pull/106)
- Speed up non-dominated sort (5302)
- Make 2d hypervolume computation twice faster (5303)
- Reduce the time complexity of HSSP 2d from `O(NK^2 log K)` to `O((N - K)K)` (5346)
- Introduce lazy hypervolume calculations in HSSP for speedup (5355)
- Make `plot_contour` faster (5369)
- Speed up `to_internal_repr` in `CategoricalDistribution` (5400)
- Allow users to modify categorical distance more easily (5404)
- Speed up `WFG` by NumPy vectorization (5424)
- Check whether the study is multi-objective in `sample_independent` of `GPSampler` (5428)
- Suppress warnings from `numpy` in hypervolume computation (5432)
- Make an option to assume Pareto optimality in WFG (5433)
- Adapt multi objective to NumPy v2.0.0 (5493)
- Enhance the error message for integration installation (5498)
- Reduce journal size of `JournalStorage` (5526)
- Simplify a SQL query for getting the `trial_id` of `best_trial` (5537)
- Add import check for artifact store objects (5565)
- Refactor `_is_categorical()` in `optuna/optuna/visualization` (5587, thanks kAIto47802!)
- Speed up WFG by using a fact that hypervolume calculation does not need (second or later) duplicated Pareto solutions (5591)
- Enhance the error message of `multi_objective` deletion (5641)

Bug Fixes

- Log `None` objective trials correctly (https://github.com/optuna/optuna-integration/pull/119, thanks neel04!)
- Pass two arguments to the forward of `ConstrainedMCObjective` in `qnei_candidates_func` (https://github.com/optuna/optuna-integration/pull/124, thanks alxhslm!)
- Allow single split cv in `OptunaSearchCV` (https://github.com/optuna/optuna-integration/pull/128, thanks sgerloff!)
- Update BoTorch samplers to support new constraints interface (https://github.com/optuna/optuna-integration/pull/132, thanks alxhslm!)
- Fix `WilcoxonPruner` bug when `best_trial` has no intermediate value (5354)
- Debug an error caused by convergence in `GPSampler` (5359)
- Fix `average_is_best` implementation in `WilcoxonPruner` (5366)
- Create an unique renaming filename for each release operation in lock systems of `JournalStorage` (5389)
- Fix `_normalize_value` for incomplete trials (5422)
- Fix heartbeat for race condition (5431)
- Guarantee `weights_below` to be finite in MOTPE (5435)
- Fix `_log_complete_trial` for constrained optimization (5462)
- Convert `step` to `int` in `report` (5488)
- Use a fixed seed value when `seed=None` in `GridSampler` (5490)
- Acquire session lock only for write in heartbeat (5496)
- Fix a bug in `_create_new_trial` and refactor it (5497)
- Add rdb create new trial test (5525)
- Fix a bug in `sample_normalized_param` for `GPSampler` (5543)
- Fix the error caused in `plot_contour()` with an impossible pair of variables (5630, thanks kAIto47802!)
- Fix inappropriate behavior in `plot_rank()` when `None` values exist in trial (5634, thanks kAIto47802!)

Installation

- Add an option to install integration dependencies via pip (https://github.com/optuna/optuna-integration/pull/130)
- Add version constraint to numpy (https://github.com/optuna/optuna-integration/pull/131)

Documentation

- Enhance `README.md` (https://github.com/optuna/optuna-integration/pull/126)
- Add document page and fix docstring and comments (https://github.com/optuna/optuna-integration/pull/134)
- Update the docstring of `PyCmaSampler` (https://github.com/optuna/optuna-integration/pull/145)
- Add Dashboard and OptunaHub links to the document header (https://github.com/optuna/optuna-integration/pull/155)
- Remove duplicated license definition (https://github.com/optuna/optuna-integration/pull/156)
- Use sphinx rst syntax (5345)
- Fix typo (5351)
- Update docs about `show_progress_bar` (5393)
- Fix artifact tutorial (5451)
- Revise docs to specify `ArtifactStore` methods as non-public (5474)
- Improve the docstring of `JournalFileStorage` (5475)
- Improve `make clean` in `docs` (5487)
- Improve the example of chemical structures in `optuna artifact tutorial` (5491)
- Add FAQ for artifact store remove API (5501)
- Add a documentation for `ArtifactMeta` (5511)
- Change docs version from latest to stable (5518)
- Add a list of supported storage for artifact store (5532)
- Rename filenames for `JournalFileStorage` in  documents and a tutorial (5535)
- Add explanation for `lock_obj` to `JournalFileStorage` docstring (5540)
- Fix storages document sections (5553)
- Add `Returns` to the docstring of `load_study` (5554, thanks kAIto47802!)
- Fix paths related to `storages.journal` (5560)
- Rename filename for `JournalFileBackend` in examples (5562)
- Add thumbnail (5575)
- Add link to `optunahub-registry` (5586)
- Fix minor artifacts document issues (5592)
- Add the OptunaHub link to the document header (5595)
- Improve a docstring of `CmaEsSampler` (5603)
- Fix the expired links caused by the directory name change in `optuna-examples` (5623, thanks kAIto47802!)
- Remove the use of `evaluator.evaluate` function in the example of `PedAnovaImportanceEvaluator` (5632)
- Add the news section to `README.md` (5636)
- Add SNS URLs to `README.md` (5637)
- Add TPE blog post link to doc (5638)
- Add artifact store post link (5639)
- Add installation guide for visualization tutorial (5644, thanks kAIto47802!)
- Remove a broken link (5648)
- Update the `News` section of `README.md` (5649)

Examples

- Support tensorflow 2.16.1 and separate CI run for tensorflow estimator (https://github.com/optuna/optuna-examples/pull/248)
- Replace deprecated jax function (https://github.com/optuna/optuna-examples/pull/250)
- Drop Python 3.7 support for wandb (https://github.com/optuna/optuna-examples/pull/252)
- Support python 3.12 in the CIs (https://github.com/optuna/optuna-examples/pull/253)
- Remove `dask` version constraint (https://github.com/optuna/optuna-examples/pull/254)
- Update GitHub actions versions to `actions/checkoutv4` and `actions/setup-pythonv5` (https://github.com/optuna/optuna-examples/pull/255)
- Delete an example using deprecated `fastaiv1` (https://github.com/optuna/optuna-examples/pull/256)
- Rename `fastaiv2` to `fastai` (https://github.com/optuna/optuna-examples/pull/257)
- Fix CI (https://github.com/optuna/optuna-examples/pull/258)
- Fix path in pruners workflow (https://github.com/optuna/optuna-examples/pull/259)
- Install `tensorflow-cpu` in CIs (https://github.com/optuna/optuna-examples/pull/260)
- Add python 3.12 and run terminator search cv (https://github.com/optuna/optuna-examples/pull/264)
- Enhance `README.md` (https://github.com/optuna/optuna-examples/pull/265)
- Organize the directory structure (https://github.com/optuna/optuna-examples/pull/266)
- Fix CI with version constraints of packages (https://github.com/optuna/optuna-examples/pull/267)
- Install `numpy<2.0.0` for `catboost` example (https://github.com/optuna/optuna-examples/pull/268)
- Add Python 3.12 to `aim` test python versions (https://github.com/optuna/optuna-examples/pull/270)
- Add Python 3.12 to `fastai` test python versions (https://github.com/optuna/optuna-examples/pull/271)
- Use auc score to unify the intermediate and objective values (https://github.com/optuna/optuna-examples/pull/272)
- Add python 3.12 for `mlflow` CI python versions (https://github.com/optuna/optuna-examples/pull/274)
- Add python 3.12 to CI python versions for `keras` and `tensorboard` examples (https://github.com/optuna/optuna-examples/pull/275)
- Add an example of artifact store (https://github.com/optuna/optuna-examples/pull/276)
- Separate basic and faq directories (https://github.com/optuna/optuna-examples/pull/277, thanks kAIto47802!)
- Remove chainer CI (https://github.com/optuna/optuna-examples/pull/278)

Tests

- Suppress `ExperimentalWarning`s (https://github.com/optuna/optuna-integration/pull/108)
- Add a unit test for convergence of acquisition function in `GPSampler` (5365)
- Remove unused block (5368)
- Implement backward compatibility tests for `JournalStorage` (5486)
- Add some tests for visualization functions (5599)

Code Fixes

- Align run name strategy between `MLflowCallback` and `track_in_mlflow` method (https://github.com/optuna/optuna-integration/pull/111, thanks TTRh!)
- Use `TYPE_CHECKING` for `ObjectiveFuncType` (https://github.com/optuna/optuna-integration/pull/113)
- Ignore `ExperimentalWarning` for `track_in_wandb` (https://github.com/optuna/optuna-integration/pull/122)
- Remove unused module (https://github.com/optuna/optuna-integration/pull/127)
- Apply formatter and follow the Optuna conventions (https://github.com/optuna/optuna-integration/pull/133)
- Refactor an internal process of `BoTorchSampler` (https://github.com/optuna/optuna-integration/pull/147)
- Add missing experimental decorators to the comet integration (https://github.com/optuna/optuna-integration/pull/148)
- Use `__future__.annotations` (https://github.com/optuna/optuna-integration/pull/150)
- Fix E721 errors (https://github.com/optuna/optuna-integration/pull/151)
- Fix future annotations in `percentile.py` (5322, thanks aaravm!)
- Delete examples directory (5329)
- Simplify annotations in `optuna/pruners/_hyperband.py` (5338, thanks keita-sa!)
- Simplify annotations in `optuna/pruners/_successive_halving.py` and `optuna/pruners/_threshold.py` (5343, thanks keita-sa!)
- Reduce test warnings (5344)
- Simplify type annotations for `storages/_base.py` (5352, thanks Obliquedbishop!)
- Use `TYPE_CHECKING` for `Study` in `samplers` (5391)
- Refactor `_normalize_objective_values` in NSGA-III to supress `RuntimeWarning` (5399)
- Make `ObjectiveFuncType` available externally (5401)
- Replace `numpy` with `np` (5412)
- Bundle experimental feature warning (5434)
- Simplify annotations in `_brute_force.py` (5441)
- Add `__future__.annotations` to base sampler (5442)
- Introduce `__future__.annotations` to TPE-related modules (5443)
- Adapt to `__future__.annotations` in `optuna/storages/_rdb/models.py` (5452, thanks aisha-partha!)
- Debug an unintended assertion error in `GPSampler` (5484)
- Make `WFG` a function (5504)
- Enhance the comments in `create_new_trial` (5510)
- Replace single trailing underscore with double trailing underscore in `.rst` files (5514, thanks 47aamir!)
- Fix hyperlinks to use double trailing underscores (5515, thanks virendrapatil24!)
- Replace `_` with `__` in the link in Sphinx (5517)
- Refactor `plot_parallel_coordinate()` (5527, thanks karthikkurella!)
- Remove an obsolete TODO comment (5528)
- Unite the argument order of artifact APIs (5533)
- Make `assume_unique_lexsorted` in `is_pareto_front` required (5534)
- Simplify annotations in `terminator` module (5536)
- Simplify the type annotations in `samplers/_qmc.py` (5538, thanks kAIto47802!)
- Rename `solution_set` to `loss_vals` in hypervolume computation (5541)
- Expand the type of `callbacks` in optimize to `Iterable` (5542, thanks kz4killua!)
- Organize the directory structure of `JournalStorage` (5544)
- Add followup of journal storage module structure organization PR (5546)
- Move `RetryFailedTrialCallback` to `optuna.storages._callbacks` (5551)
- Change `removed_version` of deprecated `JournalStorage` classes (5552)
- Reorganize a journal storage module structure (5555)
- Remove unused private deprecated  class `BaseJournalLogSnapshot` (5561)
- Replace relative import to absolute path (5569, thanks RektPunk!)
- Simplify annotation in `importance` (5578, thanks RektPunk!)
- Simplify annotation in `trial` (5579, thanks RektPunk!)
- Fix mypy warnings in `matplotlib/_contour.py` with Python3.11 (5580)
- Remove redundant loop in `_get_rank_subplot_info` (5581, thanks RektPunk!)
- Simplify annotations in `_partial_fixed.py` (5583)
- Simplify annotation in `_cmaes.py` (5584, thanks RektPunk!)
- Simplify annotations related to `GridSampler` (5588)
- Reorganize names related to `search_space` in `terminator/improvement/evaluator.py` (5594)
- Refactor `_bisect` in `_truncnorm.py` (5598)
- Simplify annotation storages (5608, thanks RektPunk!)
- Add a note about `np.unique` (5615)
- Fix ruff errors except for E731 (5627)

Continuous Integration

- Split ci (https://github.com/optuna/optuna-integration/pull/98)
- Install `pipdeptree` v2.16.2 to hotfix parse error (https://github.com/optuna/optuna-integration/pull/109)
- Remove dask version constraint (https://github.com/optuna/optuna-integration/pull/112)
- Rename CI name (https://github.com/optuna/optuna-integration/pull/120)
- Add `deprecated` arg for the comet CI job (https://github.com/optuna/optuna-integration/pull/135)
- Hotfix the CI in LightGBM test (https://github.com/optuna/optuna-integration/pull/142)
- Hotfix the mypy error of `test_pytorch_lightning.py` (https://github.com/optuna/optuna-integration/pull/144)
- Rename CI jobs (5353, thanks Obliquedbishop!)
- Update GitHub actions versions of `actions/setup-python` and `actions/checkout` (5367)
- Add `type: ignore` for CI hotfix (5419)
- Use CPU-only PyTorch wheels on GitHub Actions (5465)
- Add a version constraint to NumPy (5492)
- Add the note of the Hotfix in the workflow file (5494)
- Remove unnecessary version constraint from workflow (5495)
- Remove skipped tests for BoTorch with Python 3.12 (5519)
- Fix the version conflicts in the workflow for the minimum version constraints (5523)
- Replace `sleep` function with spin waiting to stabilize the frequent Mac test failure (5549)

Other

- Bump up version number to 4.0.0dev (https://github.com/optuna/optuna-integration/pull/102)
- Update PyPI classifiers for Python 3.11 support (https://github.com/optuna/optuna-integration/pull/129)
- Bump up version number to 4.0.0b0 (https://github.com/optuna/optuna-integration/pull/139)
- Bump up version number to 4.0.0 (https://github.com/optuna/optuna-integration/pull/158)
- Bump up to version number v4.0.0.dev (5319)
- Bump up to version number 4.0.0b0 (5572)
- Split `LICENSE` file (5597)
- Add OptunaHub section in `README.md` (5601)
- Remove duplicated definition of license in `pyproject.toml` (5645)
- Bump up to version number 4.0.0 (5653)
- Bump up to version number 4.1.0dev (5654)

Thanks to All the Contributors and Sponsors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

47aamir, Alnusjaponica, HideakiImamura, Obliquedbishop, RektPunk, TTRh, aaravm, aisha-partha, alxhslm, c-bata, caleb-kaiser, contramundum53, eukaryo, gen740, kAIto47802, karthikkurella, keisuke-umezawa, keita-sa, kz4killua, nabenabe0928, neel04, not522, nzw0301, porink0424, sgerloff, toshihikoyanase, virendrapatil24, y0z

Optuna is sponsored by the following sponsors on [GitHub](https://github.com/sponsors/optuna).

AlphaImpact, dec1costello, dubovikmaster, shu65, raquelhortab

4.0.0b0

This is the release note of [v4.0.0-b0](https://github.com/optuna/optuna/milestone/61?closed=1).

If you want to update your existing projects from Optuna v3.x to Optuna v4, please see the [migration guide](https://github.com/optuna/optuna/discussions/5573) and try out Optuna v4.

Highlights

OptunaHub Beta Release

The Optuna team released the beta version of OptunaHub, the feature-sharing platform for Optuna. Registered features can be easily implemented on users’ code and contributors can register the features they implement. The beta version of OptunaHub is now ready to accept contributions from all over the world. Visit [hub.optuna.org](https://hub.optuna.org)!

The following code shows an example to use a sampler registered on OptunaHub.

bash
% pip install optunahub


python
import optunahub
import optuna

def objective(trial):
 x = trial.suggest_float("x", 0, 1)
 return x

mod = optunahub.load_module("samplers/simulated_annealing")

sampler = mod.SimulatedAnnealingSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=20)



Stabilization of Artifact

The stable version of the artifact module is available, now equipped with several new APIs. This module introduces capabilities for managing the relatively large-sized data such as model snapshots in hyperparameter tuning, training/validation datasets, and etc. Compared to third-party libraries for experiment tracking, the advantage of using Optuna’s artifact module is a tight integration of [Optuna Dashboard](https://github.com/optuna/optuna-dashboard). This allows users to see artifacts (files) associated with the Optuna trial or study.

Here is a list of new APIs:
- [`download_artifact`](https://optuna.readthedocs.io/en/v4.0.0-b0/reference/artifacts.html#optuna.artifacts.download_artifact): Download an artifact from the artifact store.
- [`get_all_artifact_meta`](https://optuna.readthedocs.io/en/v4.0.0-b0/reference/artifacts.html#optuna.artifacts.get_all_artifact_meta): List the associated artifact information of the provided trial or study.


Stabilization of `JournalStorage`

The stable version of `JournalStorage` is available. This implies we have decided to maintain backward compatibility of the log format in the future releases.

Please note that this release introduces the following API changes to improve the clarity of class names and the module structure.

| Deprecated APIs | Corresponding active APIs |
-|-
| `optuna.storages.JournalFileStorage` | `optuna.storages.journal.JournalFileBackend` |
| `optuna.storages.JournalFileSymlinkLock` | `optuna.storages.journal.JournalFileSymlinkLock` |
| `optuna.storages.JournalFileOpenLock` | `optuna.storages.journal.JournalFileOpenLock` |
| `optuna.storages.JournalRedisStorage` | `optuna.storages.journal.JournalRedisBackend` |


Breaking Changes

- Delete deprecated three integrations, `skopt`, `catalyst`, and `fastaiv1` (https://github.com/optuna/optuna-integration/pull/114)
- Remove deprecated `CmaEsSampler` from integration (https://github.com/optuna/optuna-integration/pull/116)
- Remove verbosity of `LightGBMTuner` (https://github.com/optuna/optuna-integration/pull/136)
- Move positional args of LightGBM tuner (https://github.com/optuna/optuna-integration/pull/138)
- Remove `multi_objective` (5390)
- Delete deprecated `_ask` and `_tell` (5398)
- Delete deprecated `--direction(s)` arguments in the `ask` command (5405)
- Delete deprecated three integrations, `skopt`, `catalyst`, and `fastaiv1` (5407)
- Remove the default normalization of importance in f-ANOVA (5411)
- Remove `samplers.intersection` (5414)
- Drop implicit create-study in `ask` command (5415)
- Remove deprecated `study optimize` CLI command (5416)
- Remove deprecated `CmaEsSampler` from integration (5417)
- Support constrained optimization in `best_trial` (5426)
- Drop `--study` in `cli.py` (5430)
- Deprecate `constraints_func` in `plot_pareto_front` function (5455)
- Rename some classnames related to `JournalStorage` (5539)

New Features

- Add Comet ML integration (https://github.com/optuna/optuna-integration/pull/63, thanks caleb-kaiser!)
- Add Knowledge Gradient candidates functions (https://github.com/optuna/optuna-integration/pull/125, thanks alxhslm!)
- Add `is_exhausted()` function in the `GridSampler` class (5306, thanks aaravm!)
- Remove experimental from plot (5413)
- Implement `download_artifact` (5448)
- Add a function to list linked artifact information (5467)
- Stabilize artifact APIs (5567)
- Stabilize `JournalStorage` (5568)

Enhancements

- Pass two arguments to the forward of `ConstrainedMCObjective` to support `botorch=0.10.0` (https://github.com/optuna/optuna-integration/pull/106)
- Speed up non-dominated sort (5302)
- Make 2d hypervolume computation twice faster (5303)
- Reduce the time complexity of HSSP 2d from `O(NK^2 log K)` to `O((N - K)K)` (5346)
- Introduce lazy hypervolume calculations in HSSP for speedup (5355)
- Make `plot_contour` faster (5369)
- Speed up `to_internal_repr` in `CategoricalDistribution` (5400)
- Allow users to modify categorical distance more easily (5404)
- Speed up `WFG` by NumPy vectorization (5424)
- Check whether the study is multi-objective in `sample_independent` of `GPSampler` (5428)
- Suppress warnings from `numpy` in hypervolume computation (5432)
- Make an option to assume Pareto optimality in WFG (5433)
- Adapt multi objective to NumPy v2.0.0 (5493)
- Enhance the error message for integration installation (5498)
- Reduce journal size of `JournalStorage` (5526)
- Simplify a SQL query for getting the `trial_id` of `best_trial` (5537)

Bug Fixes

- Log `None` objective trials correctly (https://github.com/optuna/optuna-integration/pull/119, thanks neel04!)
- Pass two arguments to the forward of `ConstrainedMCObjective` in `qnei_candidates_func` (https://github.com/optuna/optuna-integration/pull/124, thanks alxhslm!)
- Allow single split cv in `OptunaSearchCV` (https://github.com/optuna/optuna-integration/pull/128, thanks sgerloff!)
- Fix `WilcoxonPruner` bug when `best_trial` has no intermediate value (5354)
- Debug an error caused by convergence in `GPSampler` (5359)
- Fix `average_is_best` implementation in `WilcoxonPruner` (5366)
- Create an unique renaming filename for each release operation in lock systems of `JournalStorage` (5389)
- Fix `_normalize_value` for incomplete trials (5422)
- Fix heartbeat for race condition (5431)
- Guarantee `weights_below` to be finite in MOTPE (5435)
- Fix `_log_complete_trial` for constrained optimization (5462)
- Convert `step` to `int` in `report` (5488)
- Use a fixed seed value when `seed=None` in `GridSampler` (5490)
- Acquire session lock only for write in heartbeat (5496)
- Fix a bug in `_create_new_trial` and refactor it (5497)
- Add rdb create new trial test (5525)
- Fix a bug in `sample_normalized_param` for `GPSampler` (5543)

Installation

- Add an option to install integration dependencies via pip (https://github.com/optuna/optuna-integration/pull/130)
- Add version constraint to numpy (https://github.com/optuna/optuna-integration/pull/131)

Documentation

- Enhance `README.md` (https://github.com/optuna/optuna-integration/pull/126)
- Add document page and fix docstring and comments (https://github.com/optuna/optuna-integration/pull/134)
- Use sphinx rst syntax (5345)
- Fix typo (5351)
- Update docs about `show_progress_bar` (5393)
- Fix artifact tutorial (5451)
- Revise docs to specify `ArtifactStore` methods as non-public (5474)
- Improve the docstring of `JournalFileStorage` (5475)
- Improve `make clean` in `docs` (5487)
- Improve the example of chemical structures in `optuna artifact tutorial` (5491)
- Add FAQ for artifact store remove API (5501)
- Add a documentation for `ArtifactMeta` (5511)
- Change docs version from latest to stable (5518)
- Add a list of supported storage for artifact store (5532)
- Rename filenames for `JournalFileStorage` in  documents and a tutorial (5535)
- Add explanation for `lock_obj` to `JournalFileStorage` docstring (5540)
- Fix storages document sections (5553)
- Add `Returns` to the docstring of `load_study` (5554, thanks kAIto47802!)
- Fixing paths related to `storages.journal` (5560)
- Rename filename for `JournalFileBackend` in examples (5562)

Examples

- Support tensorflow 2.16.1 and separate CI run for tensorflow estimator (https://github.com/optuna/optuna-examples/pull/248)
- Replace deprecated jax function (https://github.com/optuna/optuna-examples/pull/250)
- Drop Python 3.7 support for wandb (https://github.com/optuna/optuna-examples/pull/252)
- Support python 3.12 in the CIs (https://github.com/optuna/optuna-examples/pull/253)
- Remove dask version constraint (https://github.com/optuna/optuna-examples/pull/254)
- Update GitHub actions versions to `actions/checkoutv4` and `actions/setup-pythonv5` (https://github.com/optuna/optuna-examples/pull/255)
- Delete an example using deprecated `fastaiv1` (https://github.com/optuna/optuna-examples/pull/256)
- Rename fastai v2 to fastai (https://github.com/optuna/optuna-examples/pull/257)
- Fix CI (https://github.com/optuna/optuna-examples/pull/258)
- Fix path in pruners workflow (https://github.com/optuna/optuna-examples/pull/259)
- Install `tensorflow-cpu` in CIs (https://github.com/optuna/optuna-examples/pull/260)
- Add python 3.12 and run terminator search cv (https://github.com/optuna/optuna-examples/pull/264)
- Organize the directory structure (https://github.com/optuna/optuna-examples/pull/266)
- Fix CI with version constraints of packages (https://github.com/optuna/optuna-examples/pull/267)
- Install `numpy<2.0.0` for catboot example (https://github.com/optuna/optuna-examples/pull/268)

Tests

- Suppress `ExperimentalWarning`s (https://github.com/optuna/optuna-integration/pull/108)
- Add a unit test for convergence of acquisition function in `GPSampler` (5365)
- Remove unused block (5368)
- Implement backward compatibility tests for `JournalStorage` (5486)

Code Fixes

- Align run name strategy between MLflowCallback and track_in_mlflow method (https://github.com/optuna/optuna-integration/pull/111, thanks TTRh!)
- Use `TYPE_CHECKING` for `ObjectiveFuncType` (https://github.com/optuna/optuna-integration/pull/113)
- Ignore `ExperimentalWarning` for `track_in_wandb` (https://github.com/optuna/optuna-integration/pull/122)
- Remove unused module (https://github.com/optuna/optuna-integration/pull/127)
- Apply formatter and follow the Optuna conventions (https://github.com/optuna/optuna-integration/pull/133)
- Fix future annotations in `percentile.py` (5322, thanks aaravm!)
- Delete examples directory (5329)
- Simplify annotations in `optuna/pruners/_hyperband.py` (5338, thanks keita-sa!)
- Simplify annotations in `optuna/pruners/_successive_halving.py` and `optuna/pruners/_threshold.py` (5343, thanks keita-sa!)
- Reduce test warnings (5344)
- Simplify type annotations for `storages/_base.py` (5352, thanks Obliquedbishop!)
- Use `TYPE_CHECKING` for `Study` in `samplers` (5391)
- Refactor `_normalize_objective_values` in NSGA-III to supress `RuntimeWarning` (5399)
- Make `ObjectiveFuncType` available externally (5401)
- Replace `numpy` with `np` (5412)
- Bundle experimental feature warning (5434)
- Simplify annotations in `_brute_force.py` (5441)
- Add `__future__.annotations` to base sampler (5442)
- Introduce `__future__.annotations` to TPE-related modules (5443)
- Adapt to `__future__.annotations` in `optuna/storages/_rdb/models.py` (5452, thanks aisha-partha!)
- Debug an unintended assertion error in `GPSampler` (5484)
- Make `WFG` a function (5504)
- Enhance the comments in `create_new_trial` (5510)
- Replace single trailing underscore with double trailing underscore in `.rst` files (5514, thanks 47aamir!)
- Fix hyperlinks to use double trailing underscores (5515, thanks virendrapatil24!)
- Replace `_` with `__` in the link in Sphinx (5517)
- Refactor `plot_parallel_coordinate()` (5527, thanks karthikkurella!)
- Remove an obsolete TODO comment (5528)
- Unite the argument order of artifact APIs (5533)
- Make `assume_unique_lexsorted` in `is_pareto_front` required (5534)
- Simplify annotations in `terminator` module (5536)
- Simplify the type annotations in `samplers/_qmc.py` (5538, thanks kAIto47802!)
- Rename `solution_set` to `loss_vals` in hypervolume computation (5541)
- Expand the type of `callbacks` in optimize to `Iterable` (5542, thanks kz4killua!)
- Organize the directory structure of `JournalStorage` (5544)
- Add followup of journal storage module structure organization PR (5546)
- Move `RetryFailedTrialCallback` to `optuna.storages._callbacks` (5551)
- Change `removed_version` of deprecated `JournalStorage` classes (5552)
- Reorganize a journal storage module structure (5555)
- Remove unused private deprecated  class `BaseJournalLogSnapshot` (5561)

Continuous Integration

- Split ci (https://github.com/optuna/optuna-integration/pull/98)
- Install `pipdeptree` v2.16.2 to hotfix parse error (https://github.com/optuna/optuna-integration/pull/109)
- Remove dask version constraint (https://github.com/optuna/optuna-integration/pull/112)
- Rename CI name (https://github.com/optuna/optuna-integration/pull/120)
- Add `deprecated` arg for the comet CI job (https://github.com/optuna/optuna-integration/pull/135)
- Rename CI jobs (5353, thanks Obliquedbishop!)
- Update GitHub actions versions of `actions/setup-python` and `actions/checkout` (5367)
- Add `type: ignore` for CI hotfix (5419)
- Use CPU-only PyTorch wheels on GitHub Actions (5465)
- Add a version constraint to NumPy (5492)
- Add the note of the Hotfix in the workflow file (5494)
- Remove unnecessary version constraint from workflow (5495)
- Remove skipped tests for BoTorch with Python 3.12 (5519)
- Fix the version conflicts in the workflow for the minimum version constraints (5523)
- Replace `sleep` function with spin waiting to stabilize the frequent Mac test failure (5549)

Other

- Bump up version number to 4.0.0dev (https://github.com/optuna/optuna-integration/pull/102)
- Update PyPI classifiers for Python 3.11 support (https://github.com/optuna/optuna-integration/pull/129)
- Bump up version number to 4.0.0b0 (https://github.com/optuna/optuna-integration/pull/139)
- Bump up to version number v4.0.0.dev (5319)
- Bump up to version number 4.0.0b0 (5572)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

47aamir, Alnusjaponica, HideakiImamura, Obliquedbishop, TTRh, aaravm, aisha-partha, alxhslm, c-bata, caleb-kaiser, contramundum53, eukaryo, gen740, kAIto47802, karthikkurella, keisuke-umezawa, keita-sa, kz4killua, nabenabe0928, neel04, not522, nzw0301, porink0424, sgerloff, toshihikoyanase, virendrapatil24, y0z

3.6.1

This is the release note of [v3.6.1](https://github.com/optuna/optuna/milestone/62?closed=1).

Bug Fixes

- [Backport] Fix Wilcoxon pruner bug when best_trial has no intermediate value 5370
- [Backport] Address issue5358 (5371)
- [Backport] Fix `average_is_best` implementation in `WilcoxonPruner` (5373)

Other

- Bump up version number to v3.6.1 (5372)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

HideakiImamura, eukaryo, nabenabe0928

3.6.0

This is the release note of [v3.6.0](https://github.com/optuna/optuna/milestone/60?closed=1).

Highlights

Optuna 3.6 newly supports the following new features. See [our release blog](https://medium.com/optuna/announcing-optuna-3-6-f5d7efeb5620) for more detailed information.
- Wilcoxon Pruner: New Pruner Based on Wilcoxon Signed-Rank Test
- Lightweight Gaussian Process (GP)-Based Sampler
- Speeding up Importance Evaluation with PED-ANOVA
- Stricter Verification Logic for FrozenTrial
- Refactoring the Optuna Dashboard
- Migration to Optuna Integration

Breaking Changes

- Implement `optuna.terminator` using `optuna._gp` (5241)

These migration-related PRs do not break the backward compatibility as long as optuna-integration v3.6.0 or later is installed in your environment.

- Move TensorBoard Integration (https://github.com/optuna/optuna-integration/pull/56, thanks dheemantha-bhat!)
- Delete TensorBoard integration for migration to `optuna-integration` (5161, thanks dheemantha-bhat!)
- Remove CatBoost integration for isolation (5198)
- Remove PyTorch integration (5213)
- Remove Dask integration (5222)
- Migrate the `sklearn` integration (5225)
- Remove BoTorch integration (5230)
- Remove `SkoptSampler` (5234)
- Remove the `cma` integration (5236)
- Remove the `wandb` integration (5237)
- Remove XGBoost Integration (5239)
- Remove MLflow integration (5246)
- Migrate LightGBM integration (5249)
- Add CatBoost integration (https://github.com/optuna/optuna-integration/pull/61)
- Add PyTorch integration (https://github.com/optuna/optuna-integration/pull/62)
- Add XGBoost integration (https://github.com/optuna/optuna-integration/pull/65, thanks buruzaemon!)
- Add `sklearn` integration (https://github.com/optuna/optuna-integration/pull/66)
- Move Dask integration (https://github.com/optuna/optuna-integration/pull/67)
- Migrate BoTorch integration (https://github.com/optuna/optuna-integration/pull/72)
- Move `SkoptSampler` (https://github.com/optuna/optuna-integration/pull/74)
- Migrate `pycma` integration (https://github.com/optuna/optuna-integration/pull/77)
- Migrate the Weights & Biases integration (https://github.com/optuna/optuna-integration/pull/79)
- Add LightGBM integration (https://github.com/optuna/optuna-integration/pull/81, thanks DanielAvdar!)
- Migrate `MLflow` integration (https://github.com/optuna/optuna-integration/pull/84)

New Features

- Backport the change of the timeline plot in Optuna Dashboard (5168)
- Wilcoxon pruner (5181)
- Add `GPSampler` (5185)
- Add a super quick f-ANOVA algorithm named PED-ANOVA (5212)

Enhancements

- Add `formats.sh` based on `optuna/master` (https://github.com/optuna/optuna-integration/pull/75)
- Use vectorization for categorical distance (5147)
- Unify implementation of fast non-dominated sort (5160)
- Raise `TypeError` if `params` is not a `dict` in `enqueue_trial` (5164, thanks adjeiv!)
- Upgrade `FrozenTrial._validate()` (5211)
- Import SQLAlchemy lazily (5215)
- Add UCB for `optuna._gp` (5224)
- Enhance performance of `GPSampler` (5274)
- Fix inconsistencies between terminator and its visualization (5276, thanks SimonPop!)
- Enhance `GPSampler` performance other than introducing local search (5279)

Bug Fixes

- Fix import path (https://github.com/optuna/optuna-integration/pull/83)
- Fix `README.md` (https://github.com/optuna/optuna-integration/pull/88)
- Fix `LightGBMTuner` test (https://github.com/optuna/optuna-integration/pull/89)
- Fix `JSONDecodeError` in `JournalStorage` (5195)
- Fix trial validation (5229)
- Make `gp.fit_kernel_params` more robust (5247)
- Fix checking value in `study.tell`  (5269, thanks ryota717!)
- Fix `_split_trials` of `TPESampler` for constrained optimization with constant liar (5298)
- Make each importance evaluator compatible with doc (5311)

Documentation

- Remove `study optimize` from CLI tutorial page (5152)
- Clarify the `GridSampler` with ask-and-tell interface (5153)
- Clean-up `faq.rst` (5170)
- Make Methods section hidden from Artifact Docs (5188)
- Enhance README (5189)
- Add a new section explaing how to customize figures (5194)
- Replace legacy `plotly.graph_objs` with `plotly.graph_objects` (5223)
- Add a note section to explain that reseed affects reproducibility (5233)
- Update links to papers (5235)
- adding link for module's example to documetation for the `optuna.terminator` module (5243, thanks HarshitNagpal29!)
- Replace the old example directory (5244)
- Add Optuna Dashboard section to docs (5250, thanks porink0424!)
- Add a safety guard to Wilcoxon pruner, and modify the docstring (5256)
- Replace LightGBM with PyTorch-based example to remove `lightgbm` dependency in visualization tutorial (5257)
- Remove unnecessary comment in `Specify Hyperparameters Manually` tutorial page (5258)
- Add a tutorial of Wilcoxon pruner (5266)
- Clarify that pruners module does not support multi-objective optimization (5270)
- Minor fixes (5275)
- Add a guide to PED-ANOVA for `n_trials>10000` (5310)
- Minor fixes of docs and code comments for `PedAnovaImportanceEvaluator` (5312)
- Fix doc for `WilcoxonPruner` (5313)
- Fix doc example in `WilcoxonPruner` (5315)

Examples

- Remove Python 3.7 and 3.8 from tensorboard CI (https://github.com/optuna/optuna-examples/pull/231)
- Specify black version in the CI (https://github.com/optuna/optuna-examples/pull/232)
- Apply Black 2024 to codebase (https://github.com/optuna/optuna-examples/pull/236)
- Remove MXNet examples (https://github.com/optuna/optuna-examples/pull/237)
- Add an example of Wilcoxon pruner (https://github.com/optuna/optuna-examples/pull/238)
- Make Keras examples Keras 3 friendly (https://github.com/optuna/optuna-examples/pull/239)
- Remove a comment for keras that is not used anymore in this file (https://github.com/optuna/optuna-examples/pull/240)
- Use Keras 3 friendly syntax in MLflow example (https://github.com/optuna/optuna-examples/pull/242)
- Remove `-pre` option in the `rl` integration (https://github.com/optuna/optuna-examples/pull/243)
- Hotfix CI by adding version constraints to `dask` and `tensorflow` (https://github.com/optuna/optuna-examples/pull/245)

Tests

- Unify the implementation of `_create_frozen_trial()` under `testing` module (5157)
- Remove the Python version constraint for PyTorch (5278)

Code Fixes

- Fix unused (and unintended) import (https://github.com/optuna/optuna-integration/pull/68)
- Add Dask to `__init__.py` and fix its documentation generation (https://github.com/optuna/optuna-integration/pull/71)
- Replace `optuna.integration` with `optuna_integration` in the doc and the issue template (https://github.com/optuna/optuna-integration/pull/73)
- Fix the doc for TensorFlow (https://github.com/optuna/optuna-integration/pull/76)
- Add skopt dependency (https://github.com/optuna/optuna-integration/pull/78)
- Fastai readme fix (https://github.com/optuna/optuna-integration/pull/82, thanks DanielAvdar!)
- Fix `__init__.py` (https://github.com/optuna/optuna-integration/pull/86)
- Apply Black 2024 to codebase (https://github.com/optuna/optuna-integration/pull/87)
- Change the order of dependencies by name (https://github.com/optuna/optuna-integration/pull/92)
- Remove the deprecated decorator of `KerasPruningCallback` (https://github.c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pyup-bot