<jupyter_start><jupyter_text>AboutThis notebook demonstrates several additional tools to optimize classification model provided by __Reproducible experiment platform (REP)__ package:* __grid search for the best classifier hyperparameters__* __different optimization algorithms__ * __different scoring models__ (optimization of arbirtary figure of merit)<jupyter_code>%pylab inline<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>Loading data<jupyter_code>!cd toy_datasets; wget -O magic04.data -nc https://archive.ics.uci.edu/ml/machine-learning-databases/magic/magic04.data
import numpy, pandas
from rep.utils import train_test_split
from sklearn.metrics import roc_auc_score

columns = ['fLength', 'fWidth', 'fSize', 'fConc', 'fConc1', 'fAsym', 'fM3Long', 'fM3Trans', 'fAlpha', 'fDist', 'g']
data = pandas.read_csv('toy_datasets/magic04.data', names=columns)
labels = numpy.array(data['g'] == 'g', dtype=int)
data = data.drop('g', axis=1)

train_data, test_data, train_labels, test_labels = train_test_split(data, labels)

list(data.columns)<jupyter_output><empty_output><jupyter_text>Variables used in training<jupyter_code>features = list(set(columns) - {'g'})<jupyter_output><empty_output><jupyter_text>Metric definitionIn Higgs challenge the aim is to maximize `AMS` metrics. To measure the quality one should choose not only classifier, but also an optimal threshold, where the maximal value of `AMS` is achieved.Such metrics (which require a threshold) are called threshold-based.__rep.utils__ contain class __OptimalMetric__, which computes the maximal value for threshold-based metric (and may be used as metric).Use this class to generate metric and use it in grid search.  Prepare quality metricfirst we define AMS metric, and `utils.OptimalMetric` generates<jupyter_code>from rep.report import metrics
def AMS(s, b, s_norm=sum(test_labels == 1), b_norm=6*sum(test_labels == 0)): 
    return s * s_norm / numpy.sqrt(b * b_norm + 10.)
optimal_AMS = metrics.OptimalMetric(AMS)
sum(test_labels == 1),sum(test_labels == 0)<jupyter_output><empty_output><jupyter_text>Compute threshold vs metric qualityrandom predictions for signal and background were used here<jupyter_code>probs_rand = numpy.ndarray((1000, 2))
probs_rand[:, 1] = numpy.random.random(1000)
probs_rand[:, 0] = 1 - probs_rand[:, 1]
labels_rand = numpy.random.randint(0, high=2, size=1000)

optimal_AMS.plot_vs_cut(labels_rand, probs_rand)<jupyter_output>Optimal cut=0.0096, quality=30.5758<jupyter_text>The best quality<jupyter_code>optimal_AMS(labels_rand, probs_rand)<jupyter_output><empty_output><jupyter_text>Hyperparameters optimization algorithms__AbstractParameterGenerator__ is an abstract class to generate new points, where the scorer function will be computed. It is used in grid search to get new set of parameters to train classifier.__Properties__:* `best_params_` - return the best grid point* `best_score_` - return the best quality* `print_results(self, reorder=True)` - print all points with corresponding qualityThe following algorithms inherit from __AbstractParameterGenerator__:* `RandomParameterOptimizer` -  generates random point in parameters space    * `RegressionParameterOptimizer` - generate next point using regression algorithm, which was trained on previous results    * `SubgridParameterOptimizer` - uses subgrids if grid is huge + annealing-like technique (details see in __REP__)  Grid search __GridOptimalSearchCV__ implemets optimal search over specified parameter values for an estimator. Parameters to use it are:* estimator - object of type that implements the "fit" and "predict" methods* params_generator - generator of grid search algorithm (AbstractParameterGenerator)* scorer -  which implement method __call__ with kwargs: "base_estimator", "params", "X", "y", "sample_weight"* Important members are "fit", "fit_best_estimator"<jupyter_code>from rep.metaml import GridOptimalSearchCV
from rep.metaml.gridsearch import RandomParameterOptimizer, FoldingScorer
from rep.estimators import SklearnClassifier
from sklearn.ensemble import AdaBoostClassifier
from collections import OrderedDict<jupyter_output><empty_output><jupyter_text>Grid search with folding scorer__FoldingScorer__ provides folding cross-validation for train dataset:* folds - `k`, number of folds (train on `k-1` fold, test on `1` fold)* folds_check - number of times model will be tested* score_function - function to calculate quality with interface "function(y_true, proba, sample_weight=None)"__NOTE__: if fold_checks > 1, the quality is averaged over tests.<jupyter_code># define grid parameters
grid_param = OrderedDict()
grid_param['n_estimators'] = [30, 50]
grid_param['learning_rate'] = [0.2, 0.1, 0.05]

# use random hyperparameter optimization algorithm 
generator = RandomParameterOptimizer(grid_param)
# define folding scorer
scorer = FoldingScorer(optimal_AMS, folds=6, fold_checks=4)

grid_sk = GridOptimalSearchCV(SklearnClassifier(AdaBoostClassifier(), features=features), generator, scorer)
grid_sk.fit(data, labels)<jupyter_output><empty_output><jupyter_text>Print best parameters<jupyter_code>grid_sk.generator.best_params_<jupyter_output><empty_output><jupyter_text>Print all qualities for used parameters<jupyter_code>grid_sk.generator.print_results()<jupyter_output>58.674:  n_estimators=50, learning_rate=0.2
56.484:  n_estimators=30, learning_rate=0.2
56.100:  n_estimators=50, learning_rate=0.1
53.704:  n_estimators=30, learning_rate=0.1
53.278:  n_estimators=50, learning_rate=0.05
53.091:  n_estimators=30, learning_rate=0.05<jupyter_text>Grid search with user-defined scorerYou can define your own scorer with specific logic by simple way. Scorer must have just the following:* scorer(base_estimator, params, X, y, sample_weight)  Define scorer, which will be train model on all dataset and test it on the pre-defined dataset<jupyter_code>from sklearn import clone
def generate_scorer(test, test_labels, test_weight=None):
    """ Generate scorer which calculate metric on fixed test dataset """
    def custom(base_estimator, params, X, y, sample_weight=None):
        cl = clone(base_estimator)
        cl.set_params(**params)
        cl.fit(X, y)
        res = optimal_AMS(test_labels, cl.predict_proba(test), sample_weight)
        return res
    return custom
# define grid parameters
grid_param = OrderedDict()
grid_param['n_estimators'] = [30, 50]
grid_param['learning_rate'] = [0.2, 0.1, 0.05]
grid_param['features'] = [features[:5], features[:8]]

# define random hyperparameter optimization algorithm 
generator = RandomParameterOptimizer(grid_param)
# define specific scorer
scorer = generate_scorer(test_data, test_labels)

grid = GridOptimalSearchCV(SklearnClassifier(clf=AdaBoostClassifier(), features=features), generator, scorer)
grid.fit(train_data, train_labels)
len(train_data), len(test_data)<jupyter_output><empty_output><jupyter_text>Print all tried combinations of parameters and quality<jupyter_code>grid.generator.print_results()<jupyter_output>60.269:  n_estimators=50, learning_rate=0.2, features=['fLength', 'fDist', 'fConc', 'fWidth', 'fM3Long', 'fSize', 'fConc1', 'fAlpha']
57.120:  n_estimators=30, learning_rate=0.2, features=['fLength', 'fDist', 'fConc', 'fWidth', 'fM3Long', 'fSize', 'fConc1', 'fAlpha']
56.238:  n_estimators=50, learning_rate=0.1, features=['fLength', 'fDist', 'fConc', 'fWidth', 'fM3Long', 'fSize', 'fConc1', 'fAlpha']
55.923:  n_estimators=50, learning_rate=0.05, features=['fLength', 'fDist', 'fConc', 'fWidth', 'fM3Long', 'fSize', 'fConc1', 'fAlpha']
55.677:  n_estimators=30, learning_rate=0.1, features=['fLength', 'fDist', 'fConc', 'fWidth', 'fM3Long', 'fSize', 'fConc1', 'fAlpha']
55.221:  n_estimators=30, learning_rate=0.05, features=['fLength', 'fDist', 'fConc', 'fWidth', 'fM3Long', 'fSize', 'fConc1', 'fAlpha']
37.438:  n_estimators=30, learning_rate=0.2, features=['fLength', 'fDist', 'fConc', 'fWidth', 'fM3Long']
37.281:  n_estimators=50, learning_rate=0.05, features=['fLength', 'fDist', 'fConc', 'fWi[...]<jupyter_text>Results comparison<jupyter_code>from rep.report import ClassificationReport
from rep.data.storage import LabeledDataStorage

lds = LabeledDataStorage(test_data, test_labels)
classifiers = {'grid_fold': grid_sk.fit_best_estimator(train_data[features], train_labels),
               'grid_test_dataset': grid.fit_best_estimator(train_data[features], train_labels) }

report = ClassificationReport(classifiers, lds)<jupyter_output><empty_output><jupyter_text>ROCs<jupyter_code>report.roc().plot()<jupyter_output><empty_output><jupyter_text>Metric<jupyter_code>report.metrics_vs_cut(AMS, metric_label='AMS').plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11062507194658686, 'fraction_numerical': 0.025325198572579716, 'mean_word_length': 7.459591041869523, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 58, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '334888', 'license': 'Apache-2.0', 'n_tokens_mistral': 2957, 'n_tokens_neox': 2798, 'n_words': 722, 'path': 'howto/03-howto-gridsearch.ipynb', 'repo_name': 'carlosf/rep', 'size': '8687'}