<jupyter_start><jupyter_text>Emukit Bayesian Optimization BenchmarkThis notebook uses the `emukit.benchmarking` package to compare two Bayesian optimization methods against each other, using the Branin test function.<jupyter_code>import emukit
import numpy as np<jupyter_output><empty_output><jupyter_text>Set up test functionWe use the Branin function which is already included in Emukit, both the function and the appropriate input domain are ready made for us.<jupyter_code>from emukit.test_functions.branin import branin_function
branin_fcn, parameter_space = branin_function()<jupyter_output><empty_output><jupyter_text>Set up methods to testWe compare Bayesian optimization using different models. All the methods collect points one at a time in a sequential fashion and use the expected improvement acquisition function. The models we test are:- A Gaussian process with Matern52 covariance function- Random forest using the pyrfr packageWe choose to create lambda functions with a consistent interface that return an instance of a loop with a given initial data set.<jupyter_code>from emukit.examples.enums import ModelType, AcquisitionType
from emukit.examples.optimization_loops import create_bayesian_optimization_loop
from emukit.examples.single_objective_bayesian_optimization import GPBayesianOptimization

loops = [
    ('Random Forest', lambda x, y: create_bayesian_optimization_loop(x, y, parameter_space, AcquisitionType.EI, 
                                                                     ModelType.RandomForest)),
    ('Gaussian Process', lambda x, y: GPBayesianOptimization(parameter_space.parameters, x, y, 
                                                             acquisition_type=AcquisitionType.EI, noiseless=True))
]<jupyter_output><empty_output><jupyter_text>Run benchmarkA total of 10 initial data sets are generated of 5 observations that are randomly sampled from the input domain. For every intial data set, each method is run for 30 optimization iterations. The Gaussian process model has its hyper-parameters optimized after each function observation whereas the other models have fixed hyper-parameters.<jupyter_code>from emukit.benchmarking.benchmarker import Benchmarker
from emukit.benchmarking.metrics import MinimumObservedValueMetric, TimeMetric
n_repeats = 30
n_initial_data = 5
n_iterations = 50

metrics = [MinimumObservedValueMetric(), TimeMetric()]

benchmarkers = Benchmarker(loops, branin_fcn, parameter_space, metrics=metrics)
benchmark_results = benchmarkers.run_benchmark(n_iterations=n_iterations, n_initial_data=n_initial_data, 
                                               n_repeats=n_repeats)<jupyter_output><empty_output><jupyter_text>Plot resultsPlot the results of each method against each other. The plot shows the average value and standard deviation of the lowest observed value up to the given iteration.<jupyter_code>from emukit.benchmarking.benchmark_plot import BenchmarkPlot
colours = ['m', 'c']
line_styles = ['-', '--']

metrics_to_plot = ['minimum_observed_value']
plots = BenchmarkPlot(benchmark_results, loop_colours=colours, loop_line_styles=line_styles, 
                      metrics_to_plot=metrics_to_plot)
plots.make_plot()<jupyter_output><empty_output><jupyter_text>Plot results against timeUsing the `TimeMetric` object above, the time taken to complete each iteration of the loops was recorded. Here we plot the minimum observed value against the time taken.<jupyter_code># Plot against time
plots = BenchmarkPlot(benchmark_results, loop_colours=colours, loop_line_styles=line_styles, x_axis='time',
                      metrics_to_plot=metrics_to_plot)
plots.make_plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05743700894066649, 'fraction_numerical': 0.003251151449471688, 'mean_word_length': 4.964458804523425, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '48185', 'license': 'Apache-2.0', 'n_tokens_mistral': 1031, 'n_tokens_neox': 966, 'n_words': 325, 'path': 'notebooks/Emukit-tutorial-bayesian-optimization-benchmark.ipynb', 'repo_name': 'aaronkl/emukit', 'size': '3691'}