IOHprofiler: IOHexperimenter

Experimenter for Iterative Optimization Heuristics (IOHs), built natively in* C++.

• Documentation: https://arxiv.org/abs/1810.05281
• Wiki page: https://iohprofiler.github.io
• General Contact: [email protected]

IOHexperimenter provides:

• A framework to ease the benchmarking of any iterative optimization heuristic
• Continuous and discrete benchmarking problems
• Pseudo-Boolean Optimization (PBO) problem set (25 pseudo-Boolean problems)
• Integration of the well-known Black-black Optimization Benchmarking (BBOB) problem set (24 continuous problems)
• Interface for adding new problems and suite/problem set
• Advanced logging module that takes care of registering the data in a seamless manner
• Data format is compatible with IOHanalyzer

IOHexperimenter is available for:

• C++ manual can be found here
• Python: please see here for details user manual
• or as a pip package; Wiki Page.

C++ Interface

Installation

The following toolkits are needed for compiling IOHexperimenter:

• A C++ compiler. The minimum compiler version is g++ 7 or equivalent, but we recommend g++ 9 or equivalent.
• CMake, version 3.12 or higher

Please use the following commands to download, compile, and install this package:

> git clone --recursive https://github.com/IOHprofiler/IOHexperimenter.git
> cd IOHexperimenter
> mkdir build
> cd build
> cmake .. && make install

which installs all header files to /usr/local/include/ioh by default. If you want to change this directory, please use the following flag cmake -DCMAKE_INSTALL_PREFIX=your/path ..

If you want to change build options, check the output of cmake -L or use cmake-gui or ccmake.

Examples

To obtain a built-in problem, you could create a problem instance by passing the istance id and the search dimension to the constructor of the intended problem class, e.g.,

#include "ioh.hpp"
const auto om = std::make_shared<ioh::problem::pbo::OneMax>(1, 10); // PBO problem: instance 1, dim 10
const auto sp = std::make_shared<ioh::problem::bbob::Sphere>(1, 5); // BBOB problem: instance 1, dim 5

The instance id is intended to generalize a certain problem by some transformations, where it serves as the random seed for randomizing the transformations, e.g., affine transforms for BBOB problems and scaling of objective values for PBO problems. Please see

• PBO transformations
• BBOB/COCO transformations

We also provide problem factories for this purpose:

const auto &problem_factory = ioh::problem::ProblemRegistry<ioh::problem::Integer>::instance();
const auto om = problem_factory.create("OneMax", 1, 10);

Also, we include some simple examples to demonstrate the basic usage:

• Using a single problem
• Using a pre-defined problem suite/set
• Using the logging ability for storing benchmark data

For the detailed documentation of all available functionality in the IOHexperimenter, please check our this page [under construction].

Add new problems

We offer a very simple and convenient interface for integrating new benchmark problems/functions. First, you could define a new test_problem as you like. Note that the <vector> header is already imported in "ioh.hpp".

#include "ioh.hpp"

std::vector<double> test_problem(const std::vector<double> &)
{
    // the actual function body start here
    // ...
}

Then, you only need to "wrap" this new function as follows:

auto new_problem = ioh::problem::wrap_function<double>(
  &test_problem,
  "test_problem" // name for the new function
);
std::cout << new_problem.meta_data() << std::endl;

After wrapping, we could also create this test_problem from the problem factory. Note that, the instance id is ineffective in this approach since we haven't implemented it for the wrapped problem.

auto &factory = ioh::problem::ProblemRegistry<ioh::problem::Real>::instance();
auto new_problem_f = factory.create(
  "test_problem",  // create by name
  1,               // instance id
  10               // number of search variables
);

Alternatively, one might wish to create the new problem by subclassing the abstract problem class in IOHexperimenter, taking benefits of implementing more details, e.g., aforementioned transformations. This can be done by inheriting the corresponding problem registration class, which is

• ioh::problem::IntegerProblem for pseudo-Boolean problems, and
• ioh::problem::RealProblem for continuous problems.

In the below example, we show how to do this for pseudo-Boolean problems.

class NewBooleanProblem final : public ioh::problem::IntegerProblem<NewBooleanProblem>
{
protected:
    // [mandatory] The evaluate method is mandatory to implement
    std::vector<int> evaluate(const std::vector<int> &x) override
    {
        // the function body goes here
    }

    // [optional] If one wish to implement transformations on objective values
    std::vector<double> transform_objectives(std::vector<double> y) override
    {

    }

    // [optional] If one wish to implement transformations on search variables
    std::vector<double> transform_objectives(std::vector<double> y) override
    {

    }

public:
    /// [mandatory] This constructor always take `instance` as input even
    /// if it is ineffective by default. `instance` would be effective if and only if
    /// at least one of `transform_objectives` and `transform_objectives` is implemented
    NewBooleanProblem(const int instance, const int n_variables) :
        IntegerProblem(
          ioh::problem::MetaData(
            1,                     // problem id, which will be overwritten when registering this class in all pseudo-Boolean problems
            instance,              // the instance id
            "NewBooleanProblem",   // problem name
            n_variables,           // search dimensionality
            1,                     // number of objectives, only support 1 for now
            ioh::common::OptimizationType::Minimization
            )
          )
    {
    }
};

Please check this example for adding continuous problems in this manner.

Python Interface

The python interface is avaible via pip: pip install ioh. Please see here for the functionality it provides for now and some example use cases.

Contact

If you have any questions, comments or suggestions, please don't hesitate contacting us [email protected].

Our team

• Jacob de Nobel, Leiden Institute of Advanced Computer Science,
• Furong Ye, Leiden Institute of Advanced Computer Science,
• Diederick Vermetten, Leiden Institute of Advanced Computer Science,
• Hao Wang, Leiden Institute of Advanced Computer Science,
• Carola Doerr, CNRS and Sorbonne University,
• Thomas Bäck, Leiden Institute of Advanced Computer Science,

When using IOHprofiler and parts thereof, please kindly cite this work as

Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, Thomas Bäck: IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics, arXiv e-prints:1810.05281, 2018.

@ARTICLE{IOHprofiler,
  author = {Carola Doerr and Hao Wang and Furong Ye and Sander van Rijn and Thomas B{\"a}ck},
  title = {{IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics}},
  journal = {arXiv e-prints:1810.05281},
  archivePrefix = "arXiv",
  eprint = {1810.05281},
  year = 2018,
  month = oct,
  keywords = {Computer Science - Neural and Evolutionary Computing},
  url = {https://arxiv.org/abs/1810.05281}
}