<jupyter_start><jupyter_text>Running the ICML 2018 Experiments In this series of notebooks, we will replicate and analyze the ICML 2018 experiments that were used for benchmarking in BOHB (Falkner et al. 2018).In addition to HpBandSter, we will use CAVE to analyze and visualize the optimization process.  About this experiment Cartpole"A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center." (openAI gym description) Installation requirementsTo run the experiments, please install the requirements, e.g. `pip install -r examples/icml_2018_experiments/icml2018requirements.txt`.  Run the experiment This examples ships with all the code necessary to reproduce the experiment. Because it takes a few days to generate the data, the results of the optimization are provided in `examples/icml_2018_experiments/opt_results/`. If you want to generate the data (from examples/icml_2018_experiments), just run:<jupyter_code># If you want to install the necessary requirements via pip on the current kernel, uncomment the next line
#! pip install -r icml2018requirements.txt

if False:
    ! python scripts/run_experiment.py --exp_name cartpole --num_iterations 1 --min_budget 1 --max_budget 3 --n_workers 1<jupyter_output><empty_output><jupyter_text>You can use this script to generate the optimization data for a variety of optimizers, the results will (by default) be stored in `examples/icml_2018_experiments/EXPERIMENT/OPTIMIZER`. Use `python run_experiment --help` to see how to use the script.If you have access to a cluster, take a look at the scripts provided for cluster computation on a SLURM cluster at `examples/icml_2018_experiments/scripts/cluster/`.  Using the results in CAVE Instantiate CAVEAnalyzing the optimization results with CAVE is very straight-forward. If you want to use CAVE interactively in a notebook, set `show_jupyter=True`. Specify which optimization you want to analyze via the `folders` argument and specify `file_format==SMAC3` or `file_format==BOHB`, depending on which optimizer was used for the results. To analyze how BOHB optimized the cartpole-problem, run:<jupyter_code>from cave.cavefacade import CAVE

cave = CAVE(folders=["opt_results/cartpole/bohb"],
            output_dir="CAVE_reports/cartpole_notebook",  # output for debug/images/etc
            ta_exec_dir=["."],                            # Only important for SMAC-results
            file_format='BOHB',                           # BOHB or SMAC3
            verbose_level='OFF',
            show_jupyter=True,
           )<jupyter_output><empty_output><jupyter_text>To generate the HTML-report you can use the `analyze`-method. The report is located in `output_dir/report.html`, so in this case in `CAVE_reports/cartpole_notebook/report.html`.<jupyter_code>%%capture
cave.analyze()
! firefox CAVE_reports/cartpole_notebook/report.html<jupyter_output><empty_output><jupyter_text>CAVE is fully compatible with Jupyter notebooks. You can invoke the individual analysis methods as follows.The most interesting plot for BOHB might be a visualization of the learning curves:<jupyter_code>cave.bohb_learning_curves();
cave.overview_table();<jupyter_output><empty_output><jupyter_text>For each budget, we can list the cost over incumbents:<jupyter_code>cave.bohb_incumbents_per_budget();<jupyter_output><empty_output><jupyter_text>For parameter-importance analysis, CAVE uses PIMP , a package that provides multiple approaches to parameter-importance analysis. We can easily invoke them via CAVE, of course. To estimate the importance, random forests are used to predict performances of configurations that were not executed. This is difficult for big budgets with few configurations. We can access the individual budgets via the 'run'-keyword-argument of each analysis-method.<jupyter_code>cave.cave_fanova(run='budget_3');
cave.local_parameter_importance(run='budget_3');<jupyter_output><empty_output><jupyter_text>For each budget, we can compare the different parameter-importance-methods that have already been run:<jupyter_code>cave.pimp_comparison_table(run='budget_3');<jupyter_output><empty_output><jupyter_text>To analyze BOHB's behaviour, we can check out the configurator footprint, cost-over-time and parallel coordinated parameters:<jupyter_code>cave.configurator_footprint(use_timeslider=True, num_quantiles=5);
cave.cost_over_time();
cave.parallel_coordinates(run='budget_1');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06770833333333333, 'fraction_numerical': 0.012083333333333333, 'mean_word_length': 5.2922673656618615, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '604793', 'license': 'Apache-2.0', 'n_tokens_mistral': 1421, 'n_tokens_neox': 1344, 'n_words': 520, 'path': 'examples/icml_2018_experiments/cartpole.ipynb', 'repo_name': 'kaddynator/BOAH', 'size': '4800'}