<jupyter_start><jupyter_text>Hyperparameter Search for Machine Learning (Advanced)In this tutorial, we will show how to treat a learning method as a hyperparameter in the hyperparameter search. We will consider [Random Forest (RF)](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) classifier and [Gradient Boosting (GB)](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) classifier methods in [Scikit-Learn](https://scikit-learn.org/stable/) for the Airlines data set. Each of these methods have its own set of hyperparameters and some common parameters. We model them using ConfigSpace a python package to express conditional hyperparameters and more.Warning    By design asyncio does not allow nested event loops. Jupyter is using Tornado which already starts an event loop. Therefore the following patch is required to run this tutorial.<jupyter_code>!pip install nest_asyncio

import nest_asyncio
nest_asyncio.apply()<jupyter_output>Requirement already satisfied: nest_asyncio in /Users/romainegele/opt/anaconda3/envs/dh-dev/lib/python3.8/site-packages (1.5.1)<jupyter_text>Create a mapping to record the classification algorithms of interest:<jupyter_code>from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier


CLASSIFIERS = {
    "RandomForest": RandomForestClassifier,
    "GradientBoosting": GradientBoostingClassifier,
}<jupyter_output><empty_output><jupyter_text>Create a baseline code to test the accuracy of the default configuration for both models:<jupyter_code>from deephyper.benchmark.datasets import airlines as dataset
from sklearn.utils import check_random_state

rs_clf = check_random_state(42)
rs_data = check_random_state(42)

ratio_test = 0.33
ratio_valid = (1 - ratio_test) * 0.33

train, valid, test, _ = dataset.load_data(
    random_state=rs_data,
    test_size=ratio_test,
    valid_size=ratio_valid,
    categoricals_to_integers=True,
)

for clf_name, clf_class in CLASSIFIERS.items():
    print(clf_name)

    clf = clf_class(random_state=rs_clf)

    clf.fit(*train)

    acc_train = clf.score(*train)
    acc_valid = clf.score(*valid)
    acc_test = clf.score(*test)

    print(f"Accuracy on Training: {acc_train:.3f}")
    print(f"Accuracy on Validation: {acc_valid:.3f}")
    print(f"Accuracy on Testing: {acc_test:.3f}\n")<jupyter_output>DEBUG:openml.datasets.dataset:Data pickle file already exists and is up to date.<jupyter_text>The accuracy values show that the RandomForest classifier with default hyperparameters results in overfitting and thus poor generalization (high accuracy on training data but not on the validation and test data). On the contrary GradientBoosting does not show any sign of overfitting and has a better accuracy on the validation and testing set, which shows a better generalization than RandomForest.Next, we optimize the hyperparameters, where we seek to find the right classifier and its corresponding hyperparameters to improve the accuracy on the vaidation and test data. Create a `load_data` function to load and return training and validation data:<jupyter_code>import numpy as np
from sklearn.utils import resample

def load_data(verbose=0, subsample=True):

    # In this case passing a random state is critical to make sure
    # that the same data are loaded all the time and that the test set
    # is not mixed with either the training or validation set.
    # It is important to not avoid setting a global seed for safety reasons.
    random_state = np.random.RandomState(seed=42)

    # Proportion of the test set on the full dataset
    ratio_test = 0.33

    # Proportion of the valid set on "dataset \ test set"
    # here we want the test and validation set to have same number of elements
    ratio_valid = (1 - ratio_test) * 0.33

    # The 3rd result is ignored with "_" because it corresponds to the test set
    # which is not interesting for us now.
    (X_train, y_train), (X_valid, y_valid), _, _ = dataset.load_data(
        random_state=random_state,
        test_size=ratio_test,
        valid_size=ratio_valid,
        categoricals_to_integers=True,
    )

    # Uncomment the next line if you want to sub-sample the training data to speed-up
    # the search, "n_samples" controls the size of the new training data
    if subsample:
        X_train, y_train = resample(X_train, y_train, n_samples=int(1e4))
        
    if verbose:
        print(f"X_train shape: {np.shape(X_train)}")
        print(f"y_train shape: {np.shape(y_train)}")
        print(f"X_valid shape: {np.shape(X_valid)}")
        print(f"y_valid shape: {np.shape(y_valid)}")
    return (X_train, y_train), (X_valid, y_valid)

print("With subsampling")
_ = load_data(verbose=1)
print("\nWithout subsampling")
_ = load_data(verbose=1, subsample=False)<jupyter_output>DEBUG:openml.datasets.dataset:Data pickle file already exists and is up to date.<jupyter_text>Tip     Subsampling with `X_train, y_train = resample(X_train, y_train, n_samples=int(1e4))` can be useful if you want to speed-up your search. By subsampling the training time will reduce.    Create a `run` function to train and evaluate a given hyperparameter configuration. This function has to return a scalar value (typically, validation accuracy), which will be maximized by the search algorithm.<jupyter_code>from inspect import signature

def filter_parameters(obj, config: dict) -> dict:
    """Filter the incoming configuration dict based on the signature of obj.
    Args:
        obj (Callable): the object for which the signature is used.
        config (dict): the configuration to filter.
    Returns:
        dict: the filtered configuration dict.
    """
    sig = signature(obj)
    clf_allowed_params = list(sig.parameters.keys())
    clf_params = {
        k: v
        for k, v in config.items()
        if k in clf_allowed_params and not (v in ["nan", "NA"])
    }
    return clf_params
from sklearn.metrics import accuracy_score
from sklearn.utils import check_random_state


def run(config: dict) -> float:

    config["random_state"] = check_random_state(42)

    (X_train, y_train), (X_valid, y_valid) = load_data()

    clf_class = CLASSIFIERS[config["classifier"]]

    # keep parameters possible for the current classifier
    config["n_jobs"] = 4
    clf_params = filter_parameters(clf_class, config)

    try:  # good practice to manage the fail value yourself...
        clf = clf_class(**clf_params)

        clf.fit(X_train, y_train)

        fit_is_complete = True
    except:
        fit_is_complete = False

    if fit_is_complete:
        y_pred = clf.predict(X_valid)
        acc = accuracy_score(y_valid, y_pred)
    else:
        acc = -1.0

    return acc<jupyter_output><empty_output><jupyter_text>Create the `HpProblem` to define the search space of hyperparameters for each model:<jupyter_code>import ConfigSpace as cs
from deephyper.problem import HpProblem


problem = HpProblem(seed=42)

#! Default value are very important when adding conditional and forbidden clauses
#! Otherwise the creation of the problem can fail if the default configuration is not
#! Acceptable
classifier = problem.add_hyperparameter(
    name="classifier",
    value=["RandomForest", "GradientBoosting"],
    default_value="RandomForest",
)

# For both
problem.add_hyperparameter(name="n_estimators", value=(1, 1000, "log-uniform"))
problem.add_hyperparameter(name="max_depth", value=(1, 50))
problem.add_hyperparameter(
    name="min_samples_split", value=(2, 10),
)
problem.add_hyperparameter(name="min_samples_leaf", value=(1, 10))
criterion = problem.add_hyperparameter(
    name="criterion",
    value=["friedman_mse", "squared_error", "gini", "entropy"],
    default_value="gini",
)

# GradientBoosting
loss = problem.add_hyperparameter(name="loss", value=["deviance", "exponential"])
learning_rate = problem.add_hyperparameter(name="learning_rate", value=(0.01, 1.0))
subsample = problem.add_hyperparameter(name="subsample", value=(0.01, 1.0))

gradient_boosting_hp = [loss, learning_rate, subsample]
for hp_i in gradient_boosting_hp:
    problem.add_condition(cs.EqualsCondition(hp_i, classifier, "GradientBoosting"))

forbidden_criterion_rf = cs.ForbiddenAndConjunction(
    cs.ForbiddenEqualsClause(classifier, "RandomForest"),
    cs.ForbiddenInClause(criterion, ["friedman_mse", "squared_error"]),
)
problem.add_forbidden_clause(forbidden_criterion_rf)

forbidden_criterion_gb = cs.ForbiddenAndConjunction(
    cs.ForbiddenEqualsClause(classifier, "GradientBoosting"),
    cs.ForbiddenInClause(criterion, ["gini", "entropy"]),
)
problem.add_forbidden_clause(forbidden_criterion_gb)

if __name__ == "__main__":
    print(problem)<jupyter_output>Configuration space object:
  Hyperparameters:
    classifier, Type: Categorical, Choices: {RandomForest, GradientBoosting}, Default: RandomForest
    criterion, Type: Categorical, Choices: {friedman_mse, squared_error, gini, entropy}, Default: gini
    learning_rate, Type: UniformFloat, Range: [0.01, 1.0], Default: 0.505
    loss, Type: Categorical, Choices: {deviance, exponential}, Default: deviance
    max_depth, Type: UniformInteger, Range: [1, 50], Default: 26
    min_samples_leaf, Type: UniformInteger, Range: [1, 10], Default: 6
    min_samples_split, Type: UniformInteger, Range: [2, 10], Default: 6
    n_estimators, Type: UniformInteger, Range: [1, 1000], Default: 32, on log-scale
    subsample, Type: UniformFloat, Range: [0.01, 1.0], Default: 0.505
  Conditions:
    learning_rate | classifier == 'GradientBoosting'
    loss | classifier == 'GradientBoosting'
    subsample | classifier == 'GradientBoosting'
  Forbidden Clauses:
    (Forbidden: classifier == 'RandomForest' && Forb[...]<jupyter_text>Create an `Evaluator` object using the `ray` backend to distribute the evaluation of the run-function defined previously.<jupyter_code>from deephyper.evaluator import Evaluator
from deephyper.evaluator.callback import LoggerCallback

evaluator = Evaluator.create(run, 
                 method="ray", 
                 method_kwargs={
                     "address": None, 
                     "num_cpus": 2,
                     "num_cpus_per_task": 1,
                     "callbacks": [LoggerCallback()]
                     
                 })

print("Number of workers: ", evaluator.num_workers)<jupyter_output>Number of workers:  2<jupyter_text>Tip     You can open the ray-dashboard at an address like http://127.0.0.1:port in a browser to monitor the CPU usage of the execution.    Finally, you can define a Bayesian optimization search called `AMBS` (for Asynchronous Model-Based Search) and link to it the defined `problem` and `evaluator`.<jupyter_code>from deephyper.search.hps import AMBS

search = AMBS(problem, evaluator)
results = search.search(30)<jupyter_output>[00001] -- best objective: 0.61031 -- received objective: 0.61031
[00002] -- best objective: 0.63312 -- received objective: 0.63312
[00003] -- best objective: 0.63312 -- received objective: 0.61690
[00004] -- best objective: 0.63602 -- received objective: 0.63602
[00005] -- best objective: 0.63602 -- received objective: 0.62558
[00006] -- best objective: 0.63602 -- received objective: 0.61756
[00007] -- best objective: 0.63602 -- received objective: 0.56976
[00008] -- best objective: 0.63790 -- received objective: 0.63790
[00009] -- best objective: 0.63828 -- received objective: 0.63828
[00010] -- best objective: 0.63889 -- received objective: 0.63889
[00011] -- best objective: 0.63889 -- received objective: 0.55800
[00012] -- best objective: 0.63893 -- received objective: 0.63893
[00013] -- best objective: 0.63893 -- received objective: 0.59362
[00014] -- best objective: 0.63934 -- received objective: 0.63934
[00015] -- best objective: 0.63934 -- received objective: 0.61325
[00016] --[...]<jupyter_text>Once the search is over, a file named `results.csv` is saved in the current directory. The same dataframe is returned by the `search.search(...)` call. It contains the hyperparameters configurations evaluated during the search and their corresponding `objective` value (i.e, validation accuracy), `duration` of computation and time of computation with `elapsed_sec`.<jupyter_code>results<jupyter_output><empty_output><jupyter_text>The `deephyper-analytics` command line is a way of analyzing this type of file. For example, we want to output the best configuration we can use the `topk` functionnality.<jupyter_code>!deephyper-analytics topk results.csv<jupyter_output>'0':
  classifier: RandomForest
  criterion: entropy
  duration: 7.3050730228
  elapsed_sec: 61.4623451233
  id: 27
  learning_rate: null
  loss: null
  max_depth: 12
  min_samples_leaf: 9
  min_samples_split: 3
  n_estimators: 883
  objective: 0.6444682956
  subsample: null<jupyter_text>Let us define a test to evaluate the best configuration on the training, validation and test data sets.<jupyter_code>from pprint import pprint
import pandas as pd


config = results.iloc[results.objective.argmax()][:-2].to_dict()
print("Best config is:")
pprint(config)

config["random_state"] = check_random_state(42)

rs_data = check_random_state(42)

ratio_test = 0.33
ratio_valid = (1 - ratio_test) * 0.33

train, valid, test, _ = dataset.load_data(
    random_state=rs_data,
    test_size=ratio_test,
    valid_size=ratio_valid,
    categoricals_to_integers=True,
)

clf_class = CLASSIFIERS[config["classifier"]]
config["n_jobs"] = 4
clf_params = filter_parameters(clf_class, config)

clf = clf_class(**clf_params)

clf.fit(*train)

acc_train = clf.score(*train)
acc_valid = clf.score(*valid)
acc_test = clf.score(*test)

print(f"Accuracy on Training: {acc_train:.3f}")
print(f"Accuracy on Validation: {acc_valid:.3f}")
print(f"Accuracy on Testing: {acc_test:.3f}")<jupyter_output>DEBUG:openml.datasets.dataset:Data pickle file already exists and is up to date.
{'fraction_non_alphanumeric': 0.09693216188967305, 'fraction_numerical': 0.0314705458735417, 'mean_word_length': 4.478106508875739, 'pattern_counts': {'":': 7, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '204068', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 4648, 'n_tokens_neox': 4282, 'n_words': 1289, 'path': 'tutorials/notebooks/02_HPS_Reduce_Overfitting_ML/tutorial_02.ipynb', 'repo_name': 'minesweeter/tutorials', 'size': '13886'}