<jupyter_start><jupyter_text>Hyperparameter OptimizationThat means to find [the best hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_optimization) for the model.*The best means the best score for given hyperparameters*There're several options for tuning:* GridSearch ([GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.htmlsklearn.model_selection.GridSearchCV) in sklearn)* [Random Search](https://en.wikipedia.org/wiki/Random_search) ([RandomizedSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.RandomizedSearchCV.html) in sklearn)![grid and random tuning](../img/tuning.png)* [Bayesian optimization](https://en.wikipedia.org/wiki/Bayesian_optimization) (e.g. [hyperopt](https://hyperopt.github.io/hyperopt/))Let's try to understand the better context and to find where is a pain.Let's consider hyper parameters for [xgboost](https://xgboost.readthedocs.io/en/latest/python/python_api.html):- **max_depth** : int, *Maximum tree depth for base learners.*- **learning_rate** : float, *Boosting learning rate (xgb’s “eta”)*- **n_estimators** : int, *Number of boosted trees to fit.*- silent : boolean, *Whether to print messages while running boosting.*- **objective** : string or callable, *Specify the learning task and the corresponding learning objective or a custom objective function to be used (see note below).*- nthread : int, *Number of parallel threads used to run xgboost.*- **gamma** : float, *Minimum loss reduction required to make a further partition on a leaf node of the tree.*- **min_child_weight** : int, *Minimum sum of instance weight(hessian) needed in a child.*max_delta_step : int, *Maximum delta step we allow each tree’s weight estimation to be.*- **subsample** : float, *Subsample ratio of the training instance.*- **colsample_bytree** : float, *Subsample ratio of columns when constructing each tree.*- **colsample_bylevel** : float, *Subsample ratio of columns for each split, in each level.*- **reg_alpha** : float (xgb’s alpha), *L1 regularization term on weights*- **reg_lambda** : float (xgb’s lambda), *L2 regularization term on weights*- scale_pos_weight : float, *Balancing of positive and negative weights.*- base_score: *The initial prediction score of all instances, global bias.*- **seed** : int, *Random number seed.*missing : float, optional, *Value in the data which needs to be present as a missing value. If None, defaults to np.nan.*There're at least 12 (in **bold**) which could be tuning.For Grid Case search we should define grid (n - values for each hyperparameter). Time needed to compute this:  12 ( hyperparameters) * N (possible values for each) * validation (e.g. k-fold).Example:* hyper_parameters = 12* N (values per parameter) = 10* cv (k-fold validation) = 3`12^10 * 3 * time_per_iteration` = a huge number. Even for case `time_per_iteration` is 1s needed ~6000 years :).And this is why brute-force (grid-search) algorithm failed.Random search tries do it more optimal, but do it to just simple randomization. That means you don't learn anything from previous steps.<jupyter_code>#!pip3 install git+https://github.com/hyperopt/hyperopt
import pandas as pd
import xgboost as xgb
import numpy as np
#import helper as h
import gc

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize']=(12,4)
import seaborn as sns

from hyperopt import hp
from hyperopt import hp, fmin, tpe, STATUS_OK, Trials

%matplotlib inline<jupyter_output>/Users/mateusztybura/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>Let's first relatively simple task to understand better how to use `hyperopt`.The task find 3 weights: w1, w2 and w3 for given function. We know only sensible range for these weights.<jupyter_code>def weigths():
    return 7.65, 12.98, 3.23  #truth weights, looking for by hyperopt

def model(x, w1, w2, w3):
    return np.array(w1*x + w2*x**2 + w3*x**3)
x = np.linspace(0, 10, 500)

w1, w2, w3 = weigths()
y = model(x, *weigths())
plt.plot(x, y);<jupyter_output><empty_output><jupyter_text>Let's try find these hyperparameters (w1, w2 and w3) by using hyperopt<jupyter_code>def objective(space):
    pred_value = model(x, space['w1'], space['w2'], space['w3'])
    true_value = model(x, *weigths())
    
    score = np.mean( (pred_value - true_value)**2 )
    
    return{'loss':score, 'status': STATUS_OK }
    
space ={
    'w1': hp.uniform ('x_w1', 0, 15),
    'w2': hp.uniform ('x_w2', 0, 15),
    'w3': hp.uniform ('x_w3', 0, 15)
}
  
for max_evals in [1, 5, 10, 50, 100]:
    trials = Trials()
    best_params = fmin(fn=objective,
                space=space,
                algo=tpe.suggest,
                max_evals=max_evals,
                trials=trials)
    

    print("iteraions #{0}: {1}".format(max_evals, best_params))
    y_pred = model(x, best_params['x_w1'], best_params['x_w2'], best_params['x_w3'])
    plt.plot(x, y)
    plt.plot(x, y_pred)
    plt.show()<jupyter_output>iteraions #1: {'x_w1': 10.474535832321905, 'x_w2': 12.055647123983604, 'x_w3': 4.982813204452167}<jupyter_text>Let's try optimise more real thingsoptimize real model with hyperopt<jupyter_code>train = pd.read_csv('train.csv')
from sklearn import preprocessing
cat_feats = train.select_dtypes(include=["object"]).columns

for feat in cat_feats:
    train[feat + '_id'] = preprocessing.LabelEncoder().fit_transform(train[feat].values)
num_feats = [feat for feat in train.columns if 'cont' in feat]
id_feats  = [feat for feat in train.columns if '_id' in feat]

x = train[num_feats + id_feats].values
y = train['loss'].values
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=42)
from sklearn.metrics import mean_squared_error
def objective(space):
    model = xgb.XGBRegressor(
        max_depth = space['max_depth'],
        learning_rate = space['learning_rate'],
        n_estimators = 20,
        silent = 0,
        objective = 'reg:linear',
        nthread = -1,
        # gamma = 5290.,
        # min_child_weight = 4.2922,
        subsample = space['subsample'],
        colsample_bytree = 0.6,
        seed = 2017
    )
    
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    score = mean_squared_error(y_test, y_pred)
    return{'loss':score, 'status': STATUS_OK }
    
space ={
    'max_depth': hp.randint ('max_depth', 14) + 1,
    'learning_rate': hp.uniform ('learning_rate', 0.01, 0.2),
    'subsample': hp.uniform ('subsample', 0.1, 1.0)
}
  
for max_evals in [1, 5, 10, 50, 100]:
    trials = Trials()
    best_params = fmin(fn=objective,
                space=space,
                algo=tpe.suggest,
                max_evals=max_evals,
                trials=trials)
    

    print("iterations #{0}: {1}".format(max_evals, best_params))
    
    model = xgb.XGBRegressor(
        max_depth = best_params['max_depth'],
        learning_rate = best_params['learning_rate'],
        n_estimators = 20,
        silent = 0,
        objective = 'reg:linear',
        nthread = -1,
        # gamma = 5290.,
        # min_child_weight = 4.2922,
        subsample = best_params['subsample'],
        colsample_bytree = 0.6,
        seed = 2017
    )
    
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    score = mean_squared_error(y_test, y_pred)
    print('loss', score)<jupyter_output>iterations #1: {'learning_rate': 0.056259268457605004, 'max_depth': 1, 'subsample': 0.3300207662483688}
loss 7283343.64115
iterations #5: {'learning_rate': 0.16658734146533608, 'max_depth': 9, 'subsample': 0.9518188587394154}
loss 3792815.00286
iterations #10: {'learning_rate': 0.1527151547814858, 'max_depth': 9, 'subsample': 0.37715366254314486}
loss 3865574.98974
{'fraction_non_alphanumeric': 0.12209802235597593, 'fraction_numerical': 0.04606313720673136, 'mean_word_length': 4.0983093299937385, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '377546', 'license': 'MIT', 'n_tokens_mistral': 2932, 'n_tokens_neox': 2656, 'n_words': 786, 'path': 'step3.ipynb', 'repo_name': 'tyburam/wit-dataworkshop', 'size': '8141'}