<jupyter_start><jupyter_text>HyperparameterHunter Extended ExampleIn this example, we'll try to simulate a miniature project and go over some of the things you should expect when starting out with HyperparameterHunter and some of the things you might want to adjust.<jupyter_code>import warnings
warnings.filterwarnings('ignore')
# import os
# os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
from sklearn.model_selection import StratifiedKFold
from xgboost import XGBClassifier

from hyperparameter_hunter import Environment, CVExperiment
from hyperparameter_hunter.utils.learning_utils import get_breast_cancer_data
from hyperparameter_hunter.utils.file_utils import print_tree<jupyter_output>Using TensorFlow backend.<jupyter_text>Above, we're importing the cross-validation scheme with which we want to start, along with the first algorithm we'll test (`StratifiedKFold` and `XGBClassifier`, respectively). Then, from `hyperparameter_hunter`, we import `Environment`, and `CVExperiment`, which are at the core of any project. We also import the following utility functions: `get_breast_cancer_data`, which puts the result of `sklearn.datasets.load_breast_cancer` into a DataFrame; and `print_tree`, which prints out a directory's contents so we can easily see what files are being created by completed Experiments.Below, we declare the directory to store hyperparameter_hunter's results, and print out what it looks like. The astute readers will notice that nothing is printed because that directory doesn't exist yet (unless you're running this with a populated directory, in which case the rest of this example won't make much sense).<jupyter_code>example_assets = 'HyperparameterHunterAssets'
print_tree(example_assets)<jupyter_output><empty_output><jupyter_text>Environment<jupyter_code>env = Environment(
    train_dataset=get_breast_cancer_data(),
    root_results_path=example_assets,
    metrics_map=['roc_auc_score'],
    target_column='diagnosis',
    cross_validation_type=StratifiedKFold,
    cross_validation_params=dict(n_splits=3, shuffle=True, random_state=32),
    file_blacklist=['script_backup']
)<jupyter_output>Cross-Experiment Key: icnP8lYZqtpQSHW_tVVuo3Y6LPEQtYdTNMZcpfPf_cs=<jupyter_text>We begin by instantiating an `Environment`, and giving it the following: * Our `train_dataset`,* Our `root_results_path` declared above,* A `metrics_map`, with 'roc_auc_score', because the Wisconsin Breast Cancer dataset is a classification problem,* `target_column`, a string naming the column in our dataset containing the target output (defaults to 'target'),* Our `cross_validation_type` imported earlier,* `cross_validation_params`, a dict containing all the arguments to pass to `cross_validation_type`,* And `file_blacklist`, a list of result file names we don't want to save. Here, we're giving it 'script_backup' because we're executing our Experiments in a Jupyter notebook, instead of a standard Python scriptNotice that upon instantiation, our `Environment` logs the cross-experiment key produced by the provided parameters. This is important because it determines when two `Experiment`s can be properly compared; we'll go over this more later.  First ExperimentLet's dive right in and execute our first `Experiment` to see what happens! All we need to do is give it a `model_initializer` and `model_init_params`, and it'll work with our `Environment` to take care of the rest!`model_init_params` can contain any arguments expected by the `__init__` method of `model_initializer`. Additionally, our Experiment will figure out the default values of any arguments we don't provide and it'll record those too! So there's no need to give it a huge dictionary of arguments just to make sure all the hyperparameters get recorded!<jupyter_code>experiment_0 = CVExperiment(
    model_initializer=XGBClassifier,
    model_init_params=dict(objective='reg:linear', max_depth=3, n_estimators=100, subsample=0.5)
)<jupyter_output><2018-06-08 16:41:42,817> Validated Environment with key: "<KEY>="
<2018-06-08 16:41:42,818> 
<2018-06-08 16:41:42,819> Initialized new Experiment with ID: 1d1627ac-1587-4dbd-8eb3-4611ae74505a
<2018-06-08 16:41:42,820> Skipped creating backup of file: "/Users/Hunter/hyperparameter_hunter/examples/<ipython-input-4-030e3e5c6c1c>"
<2018-06-08 16:41:42,836> Generated hyperparameter key: <KEY>
<2018-06-08 16:41:42,841> Initial preprocessing stage complete
<2018-06-08 16:41:42,845> 

<2018-06-08 16:41:42,850> Starting Repetition 0
<2018-06-08 16:41:42,851> 
<2018-06-08 16:41:42,852> 
<2018-06-08 16:41:42,855> F0/R0  |  Seed: 10967   Time: 16:41:42
<2018-06-08 16:41:43,039> F0/R0  |  OOF(roc_auc_score=0.95354)  |  Time Elapsed: 0.18405 s
<2018-06-08 16:41:43,043> F0.0 AVG:   OOF(roc_auc_score=0.95354)  |  Time Elapsed: 0.19117 s
<2018-06-08 16:41:43,046> 
<2018-06-08 16:41:43,048> F1/R0  |  Seed: 75062   Time: 16:41:[...]<jupyter_text>This may look like a bit much at first, so let's unpack everything. First notice, the log noting the `Experiment`'s connection with the `Environment` we created earlier. Then, it prints out the `Experiment`'s ID, which is a randomly generated UUID. Down another line, we have the `Experiment`'s `hyperparameter_key`, which hashes the inputs to `CVExperiment` to make a "fingerprint" for the `Experiment` to help `hyperparameter_hunter` figure out if we're running an `Experiment` it already has results for. No more wasting time with duplicate experiments! Skipping down a few lines, we see the results of fitting our model during Stratified K-Folds cross-validation for 1 repetition, 3 folds, and 1 run, followed by the final out-of-fold ROC-AUC score for the `Experiment`. The subsequent lines describe which result files are being saved.Now, let's print out the contents of our 'HyperparameterHunterAssets' directory, and notice how dramatically the landscape has changed!<jupyter_code>print_tree(example_assets)<jupyter_output>[;1mHyperparameterHunterAssets/[0m
|-- Heartbeat.log
[;1m|-- Experiments/[0m
[;1m|   |-- Descriptions/[0m
|   |   |-- 1d1627ac-1587-4dbd-8eb3-4611ae74505a.json
[;1m|   |-- Heartbeats/[0m
|   |   |-- 1d1627ac-1587-4dbd-8eb3-4611ae74505a.log
[;1m|   |-- PredictionsOOF/[0m
|   |   |-- 1d1627ac-1587-4dbd-8eb3-4611ae74505a.csv
[;1m|-- KeyAttributeLookup/[0m
|   |-- cross_validation_type.db
|   |-- model_initializer.db
|   |-- prediction_formatter.json
[;1m|   |-- train_dataset/[0m
|   |   |-- XacopYu-zgxwHX7CUJpgmPg6IPRmyjSOb-nWebZim0E=.csv
[;1m|-- Leaderboards/[0m
|   |-- GlobalLeaderboard.csv
[;1m|-- TestedKeys/[0m
|   |-- icnP8lYZqtpQSHW_tVVuo3Y6LPEQtYdTNMZcpfPf_cs=.json
{'fraction_non_alphanumeric': 0.09583648750946253, 'fraction_numerical': 0.0667676003028009, 'mean_word_length': 5.8032955715756955, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '239386', 'license': 'MIT', 'n_tokens_mistral': 2453, 'n_tokens_neox': 2118, 'n_words': 673, 'path': 'examples/extended_example.ipynb', 'repo_name': 'PaulGureghian1/Hyperparameter_Hunter', 'size': '6682'}