MixedInteger Parallel Efficient Global Optimization
A Python
implementation of the Efficient Global Optimization (EGO) / Bayesian Optimization (BO) algorithm for decision spaces composed of either real, integer, catergorical variables, or a mixture thereof.
Underpinned by surrogate models, this algorithm iteratively proposes candidate solutions using the socalled acquisition function which balances exploration with exploitation, and updates the surrogate model with newly observed objective values.
The project is structured as follows:

MIPEGO/base.py
: the base class of Bayesian Optimization. 
MIPEGO/BayesOpt.py
contains several BO variants:
BO
: noiseless + seqential 
ParallelBO
: noiseless + parallel (a.k.a. batchsequential) 
AnnealingBO
: noiseless + parallel + annealling [WEB18] 
SelfAdaptiveBO
: noiseless + parallel + selfadaptive [WEB19] 
NoisyBO
: noisy + parallel 
PCABO
: noiseless + parallel + PCAassisted dimensionality reduction [RaponiWBBD20] [Under Construction]


MIPEGO/InfillCriteria.py
: the implemetation of acquisition functions (see below for the list of implemented ones). 
MIPEGO/Surrogate.py
: the implementation/wrapper of sklearn's random forests model. 
MIPEGO/SearchSpace.py
: implementation of the search/decision space.
Features
This implementation differs from alternative packages/libraries in the following features:
 Parallelization, also known as batchsequential optimization, for which several different approaches are implemented here.
 MomentGenerating Function of the improvment (MGFI) [WvSEB17a] is a recently proposed acquistion function, which implictly controls the explorationexploitation tradeoff.
 MixedInteger Evolution Strategy for optimizing the acqusition function, which is enabled when the search space is a mixture of real, integer, and categorical variables.
Acqusition Functions
The following infillcriteria are implemented in the library:
 Expected Improvement (EI)
 Probability of Improvement (PI) / Probability of Improvement
 Upper Confidence Bound (UCB)
 MomentGenerating Function of Improvement (MGFI)
 Generalized Expected Improvement (GEI) [Under Construction]
For sequential working mode, Expected Improvement is used by default. For parallelization mode, MGFI is enabled by default.
Surrogate Model
The meta (surrogate)model used in Bayesian optimization. The basic requirement for such a model is to provide the uncertainty quantification (either empirical or theorerical) for the prediction. To easily handle the categorical data, random forest model is used by default. The implementation here is based the one in scikitlearn, with modifications on uncertainty quantification.
Installation
pip install mipego
Exemplary Use Case
To use the optimizer you need to define an objective function, the search space and configure the optimizer. Below are two examples that describe most of the functionality.
Optimizing A BlackBox Function
In this example we optimize a mixed integer black box problem.
import numpy as np
from MIPEGO import ParallelBO, ContinuousSpace, OrdinalSpace, NominalSpace, RandomForest
seed = 666
np.random.seed(seed)
dim_r = 2 # dimension of the real values
def obj_fun(x):
x_r = np.array([x['continuous_%d'%i] for i in range(dim_r)])
x_i = x['ordinal']
x_d = x['nominal']
_ = 0 if x_d == 'OK' else 1
return np.sum(x_r ** 2) + abs(x_i  10) / 123. + _ * 2
# Continuous variables can be specified as follows:
# a 2D variable in [5, 5]^2
# for 2 variables, the naming scheme is continuous0, continuous1
C = ContinuousSpace([5, 5], var_name='continuous') * dim_r
# Integer (ordinal) variables can be specified as follows:
# The domain of integer variables can be given as with continuous ones
# var_name is optional
I = OrdinalSpace([5, 15], var_name='ordinal')
# Discrete (nominal) variables can be specified as follows:
# No lb, ub... a list of categories instead
N = NominalSpace(['OK', 'A', 'B', 'C', 'D', 'E', 'F', 'G'], var_name='nominal')
# The whole search space can be constructed:
search_space = C + I + N
# Bayesian optimization also uses a Surrogate model
# For mixed variable type, the random forest is typically used
model = RandomForest(levels=search_space.levels)
opt = ParallelBO(
search_space=search_space,
obj_fun=obj_fun,
model=model,
max_FEs=50,
DoE_size=3, # the initial DoE size
eval_type='dict',
acquisition_fun='MGFI',
acquisition_par={'t' : 2},
n_job=3, # number of processes
n_point=3, # number of the candidate solution proposed in each iteration
verbose=True # turn this off, if you prefer no output
)
xopt, fopt, stop_dict = opt.run()
print('xopt: {}'.format(xopt))
print('fopt: {}'.format(fopt))
print('stop criteria: {}'.format(stop_dict))
Neural Architecture Search
In this example we optimize a neural network architecture on the wellknown MNIST
dataset. The objective function is computed in this file.
In the objective file the neural network architecture is defined and evaluated on the MNIST dataset. In this example, the optimizer proposes 4 candidates architectures which are trained on 4 GPUs simultaneously.
A brief Introduction to Bayesian Optimization
Bayesian Optimization [Moc74, JSW98] (BO) is a sequential optimization strategy originally proposed to solve the singleobjective blackbox optimization problem that is costly to evaluate. Here, we shall restrict our discussion to the singleobjective case. BO typically starts with sampling an initial design of experiment (DoE) of size, X={x_{1},x_{2},...,x_{n}}, which is usually generated by simple random sampling, Latin Hypercube Sampling [SWN03], or the more sophisticated lowdiscrepancy sequence [Nie88] (e.g., Sobol sequences). Taking the initial DoE X and its corresponding objective value, Y={f(x_{1}), f(x_{2}),..., f(x_{n})} ⊆ ℝ, we proceed to construct a statistical model M describing the probability distribution of the objective function conditioned onthe initial evidence, namely Pr(fX,Y). In most application scenarios of BO, there is a lack of a priori knowledge about f and therefore nonparametric models (e.g., Gaussian process regression or random forest) are commonly chosen for M, which gives rise to a predictor f'(x) for all x ∈ X and an uncertainty quantification s'(x) that estimates, for instance, the mean squared error of the prediction E(f'(x)−f(x))^{2}. Based on f' and s', promising points can be identified via the socalled acquisition function which balances exploitation with exploration of the optimization process.
Contributing
Please take a look at our contributing guidelines if you're interested in helping!
Cite Us
You can find our paper on IEEE Explore and on Arxiv.
When using MiPEGO for your research, please cite us as follows:
@inproceedings{van2019automatic,
title={Automatic Configuration of Deep Neural Networks with Parallel Efficient Global Optimization},
author={van Stein, Bas and Wang, Hao and B{\"a}ck, Thomas},
booktitle={2019 International Joint Conference on Neural Networks (IJCNN)},
pages={17},
year={2019},
organization={IEEE}
}
Reference
 [Moc74] Jonas Mockus. "On bayesian methods for seeking the extremum". In Guri I. Marchuk, editor, Optimization Techniques, IFIP Technical Conference, Novosibirsk, USSR, July 17, 1974, volume 27 of Lecture Notes in Computer Science, pages 400–404. Springer, 1974.
 [JSW98] Donald R. Jones, Matthias Schonlau, and William J. Welch. "Efficient global optimization of expensive blackbox functions". J. Glob. Optim., 13(4):455–492, 1998.
 [SWN03] Thomas J. Santner, Brian J. Williams, and William I. Notz. "The Design and Analysis of Computer Experiments". Springer series in statistics. Springer, 2003.
 [Nie88] Harald Niederreiter. "Lowdiscrepancy and lowdispersion sequences". Journal of number theory, 30(1):51–70, 1988.
 [WvSEB17a] Hao Wang, Bas van Stein, Michael Emmerich, and Thomas Bäck. "A New Acquisition Function for Bayesian Optimization Based on the MomentGenerating Function". In Systems, Man, and Cybernetics (SMC), 2017 IEEE International Conference on, pages 507–512. IEEE, 2017.
 [WEB18] Hao Wang, Michael Emmerich, and Thomas Bäck. "Cooling Strategies for the MomentGenerating Function in Bayesian Global Optimization". In 2018 IEEE Congress on Evolutionary Computation, CEC 2018, Rio de Janeiro, Brazil, July 813, 2018, pages 1–8. IEEE, 2018.
 [WEB19] Hao, Wang, Michael Emmerich, and Thomas Bäck. "Towards selfadaptive efficient global optimization". In AIP Conference Proceedings, vol. 2070, no. 1, p. 020056. AIP Publishing LLC, 2019.
 [RaponiWBBD20] Elena Raponi, Hao Wang, Mariusz Bujny, Simonetta Boria, and Carola Doerr: "High Dimensional Bayesian Optimization Assisted by Principal Component Analysis". In International Conference on Parallel Problem Solving from Nature, pp. 169183. Springer, Cham, 2020.