Benchmarks#
The following pages compare the performance of moocore, pymoo, BoTorch, DEAP, DESDEO, paretoset, Nevergrad, jMetalPy, fast-pareto, Optuna, moarchiving, and patatune.
Other Python packages are not included in the comparison because they are based on these packages for the functionality benchmarked, so they are at least as slow as them. For example Xopt and BoFire use BoTorch, pysamoo is an extension of pymoo. We do not include functions from the Bayesian optimization toolbox trieste, because it is much slower than BoTorch and too slow to run the benchmarks in a reasonable time (secondmind-labs/trieste#917). Similarly, we do not include paretobench because it is always slower than BoTorch for identifying nondominated points and more than 2000 times slower than moocore for computing the hypervolume in 3D (electronsandstuff/ParetoBench#57), which is too slow to include in the benchmarks.
Not all packages provide the same functionality. For example, pymoo does not provide the epsilon indicator. BoTorch and paretobench only provide the hypervolume. paretoset and fast-pareto only identify nondominated points. DESDEO, DEAP, and jMetalPy use moocore for computing the hypervolume, but other functionality, such as filtering dominated points, is still slower than moocore. Recent versions of seqme and pymoo (≥0.6.2) already use moocore for most functionality benchmarked here.
We would like to benchmark pygmo, however, it is currently impossible to install using pip (See esa/pygmo2#152).
We would also like to benchmark platypus, but its API is too different from
other packages. For example, it only supports filtering dominated points by
creating a new list of points, but it cannot identify their position in the
input. It cannot calculate hypervolume without normalization. It doesn’t
support ArrayLike inputs.
The source code for the benchmarks below can be found at multi-objective/moocore .