<jupyter_start><jupyter_text>Benchmarking Introduction**BTB** provides a benchmarking framework that allows users and developers to evaluate theperformance of the BTB Tuners or other tuning functions for Machine Learning HyperparameterTuning on hundreds of real world classification problem and classical mathematical optimization problems. PrerequisitesIn order to use this `benchmarking` module, you will have to fork the**BTB** repositoryand install it from source. You can visit our[Get Started](https://hdi-project.github.io/BTB/contributing.htmlget-started)tutorial and follow until step 4, which explains how to clone and install the repositoryfrom it's source. The Benchmarking processThe Benchmarking BTB process has two main concepts. ChallengesA Challenge of the BTB Benchmarking framework is a Python class which has a method that produces ascore that can be optimized by tuning a set of hyperparameters. Tuning FunctionsIn the context of the BTB Benchmarking, `Tuning Functions` are python functions that, given a scoringfunction and its tunable hyperparameters, try to search for the ideal hyperparameter values withina given number of iterations.If you want to add a tuner, you could follow the specific signature a tuning function has:```python3def tuning_function(    scoring_function: callable,    tunable_hyperparameters: dict,    iterations: int) -> score: float``` Creating a tuning functionNow let's create a tuning function that takes as an input takes the following arguments:- `scoring_function`: A function that given the keyword args of the `tunable_hp` generates a score.- `tunable_hp`: A dictionary representation of `HyperParams`.- `iterations`: Amount of tuning iterations to perform.Our function will use a [BTB Tuner](https://github.com/HDI-Project/BTB/blob/master/tutorials/01_Tuning.ipynb) `GPEiTuner` that will iteratively will:1. Propose a new set of hyperparameters to be scored.2. Score them against the `scoring_function`.3. Update the `best_score` so far.And finally will return the `best_score` obtained for the given iterations.<jupyter_code>import numpy as np

from btb.tuning import GPEiTuner, Tunable

def tuning_function(scoring_function, tunable_hp, iterations):
    tunable = Tunable.from_dict(tunable_hp)
    tuner = GPEiTuner(tunable)
    
    best_score = -np.inf
    
    for _ in range(iterations):
        proposal = tuner.propose()
        score = scoring_function(**proposal)
        tuner.record(proposal, score)
        
        best_score = max(score, best_score)
        
    return best_score<jupyter_output><empty_output><jupyter_text>Running the BenchmarkingThe user API for the BTB Benchmarking is the `btb_benchmark.main.run_benchmark` function.The `run_benchmark` function accepts the following arguments:- `tuners`: list of tuners or tuning functions that will be benchmarked.- `challenge_types`: list of types of challenges that will be used for benchmark (optional).- `challenges`: list of names of challenges that will be benchmarked (optional).- `sample`: if specified, run the benchmark on a subset of the available challenges of the given size (optional).- `iterations`: the number of tuning iterations to perform per challenge and tuner.- `output_path`: If given, store the benchmark results in the given path as a CSV file.- `detailed_output`: If ``True`` a dataframe with the elapsed time, score and iterations will be returned.*Note*: as we want to provide a simple usage example, we will be demostrating the benchmarkingfunctionality with a fixed amount of `samples` and low `iterations`.The easiest way to run the benchmarking process is to import `run_benchmark` and run it with thedesired arguments:<jupyter_code>from btb_benchmark import run_benchmark

scores = run_benchmark(sample=1, iterations=1)

scores<jupyter_output><empty_output><jupyter_text>TunersIf you want to run the benchmark on your own tuner implementation, or in a subset of the BTBtuners, you can pass them as a list to the tuners argument. This can be done by either directlypassing the function or the name of the implemented tuners:- `BTB.UniformTuner`- `BTB.GPTuner`- `BTB.GPEiTuner`- `HyperOpt.rand`- `HyperOpt.tpe`For example, if we want to compare the performance of our tuning function and BTB.GPTuner, we cancall the `run_benchmark` function like this:<jupyter_code>tuners = [
    tuning_function,
    'BTB.GPTuner',
]
results = run_benchmark(tuners=tuners, sample=1, iterations=5)
results<jupyter_output><empty_output><jupyter_text>ChallengesIf we want to run the benchmark on a subset of the challenges, we can pass their names to thechallenges argument. If a given challenge is the name of a mathematical optimization problemfunction, the corresponding Mathematical Optimization Challenge will be executed.If the given challenge is the name of a Machine Learning Classification problem, all theimplemented classifiers will be benchmarked on that dataset.For example, if we want to run only on the `stock_1` dataset, we can call the `run_benchmark` function like this:<jupyter_code>challenges = ['stock_1']
results = run_benchmark(challenges=challenges, iterations=5)
results<jupyter_output><empty_output><jupyter_text>Additionally, if we only want to run on a family of challenges or a specific Machine Learningmodel, we can specify it passing the `types` argument.For example, if we want to run all the dataset on the XGBoost model, we can call the run benchmarkfunction like this:<jupyter_code>results = run_benchmark(challenge_types=['xgboost'], sample=1, iterations=5)
results<jupyter_output><empty_output><jupyter_text>Detailed outputIf we want to have a detailed output about the benchmarking, wich contains the elapsed time of each benchmarkingprocess, we can set the argument `detailed_output` to `True`:<jupyter_code>results = run_benchmark(sample=3, iterations=1, detailed_output=True)
results<jupyter_output><empty_output><jupyter_text>Storing the resultsIf we want to store the obtained results directly in to a file, we can pass the path to where wewould like to save our results, by adding the argument `output_path`.For example, if we want to store it as `path/to/my_results.csv` we can use:<jupyter_code>run_benchmark(sample=1, iterations=1, output_path='my_results.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06449044585987261, 'fraction_numerical': 0.0031847133757961785, 'mean_word_length': 5.646560846560846, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '129371', 'license': 'MIT', 'n_tokens_mistral': 1798, 'n_tokens_neox': 1704, 'n_words': 741, 'path': 'benchmark/notebooks/Benchmarking.ipynb', 'repo_name': 'pvk-developer/BTB', 'size': '6280'}