<jupyter_start><jupyter_text>Making your own performance metricIn case you want to optimize a metric that is not available in sklearn, you can create it very easily.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import confusion_matrix, make_scorer
# if you want more information about the dataset for this demo:

# scikit-learn dataset
# https://scikit-learn.org/stable/datasets/toy_dataset.html#breast-cancer-dataset

# dataset information: UCI Machine Learning Repository
# https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)
    
# in short, classification problem, trying to predict whether the tumor
# is malignant or benign

# load dataset
breast_cancer_X, breast_cancer_y = load_breast_cancer(return_X_y=True)
X = pd.DataFrame(breast_cancer_X)
y = pd.Series(breast_cancer_y).map({0:1, 1:0})

X.head()
# percentage of benign (0) and malign tumors (1)

y.value_counts() / len(y)<jupyter_output><empty_output><jupyter_text>Confusion matrixWe will use the [confusion matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) function from sklearn to return the false positive, false negatives, true positives and true negatives, to construct the false negative rate.TN | FPFN | TP  False Negative RateBecause it is a dangerous disease, I want to minimize the number of false negatives cases (the false negative rate). That is, I want to minimize the number of cases where the tumor in a patient is malign (1) and it is wrongly classified as benign (0). This way, we can offer treatment to as many patients as possible.Because the **False Negative Rate** is not a metric available in sklearn, we need to create a function ourselves.We will use the [make_scorer](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html) function from sklearn:<jupyter_code># function returns the FNR given probabilities and the 
# grand truth

def fnr(y_true, y_pred):
    
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0,1]).ravel()
    
    FNR = fn / (tp + fn)
    
    return FNR    
fnr_score = make_scorer(
    fnr,
    greater_is_better=False, # smaller is better
    needs_proba=False,
)
# random forests
rf_model = RandomForestClassifier(
    n_estimators=100, max_depth=1, random_state=0, n_jobs=4)

# hyperparameter space
rf_param_grid = dict(
    n_estimators=[10, 20, 50, 100, 200, 500, 1000, 2000],
    max_depth=[1, 2, 3, 4],
)

# search
clf = GridSearchCV(rf_model,
                   rf_param_grid,
                   scoring=fnr_score,
                   cv=5)

search = clf.fit(X, y)

# best hyperparameters
search.best_params_
results = pd.DataFrame(search.cv_results_)[['params', 'mean_test_score', 'std_test_score']]

results.head()
results.sort_values(by='mean_test_score', ascending=False, inplace=True)

results.reset_index(drop=True, inplace=True)

results['mean_test_score'].plot(yerr=[results['std_test_score'], results['std_test_score']], subplots=True)

plt.ylim(-0.3, 0)
plt.ylabel('Mean False Negative Rate')
plt.xlabel('Hyperparameter space')<jupyter_output><empty_output><jupyter_text>Scoring Function that needs a probability<jupyter_code>def fnr(y_true, y_pred):
    
    y_pred_class = np.where(y_pred > 0.37, 1, 0)
    
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred_class, labels=[0,1]).ravel()
    
    FNR = fn / (tp + fn)
    
    return FNR
fnr_score = make_scorer(
    fnr,
    greater_is_better=False, # smaller is better
    needs_proba=True,
)
# random forests
rf_model = RandomForestClassifier(
    n_estimators=100, max_depth=1, random_state=0, n_jobs=4)

# hyperparameter space
rf_param_grid = dict(
    n_estimators=[10, 20, 50, 100, 200, 500, 1000, 2000],
    max_depth=[1, 2, 3, 4],
)

# search
clf = GridSearchCV(rf_model,
                   rf_param_grid,
                   scoring=fnr_score,
                   cv=5)

search = clf.fit(X, y)

# best hyperparameters
search.best_params_
results = pd.DataFrame(search.cv_results_)[['params', 'mean_test_score', 'std_test_score']]

results.sort_values(by='mean_test_score', ascending=False, inplace=True)

results.reset_index(drop=True, inplace=True)

results['mean_test_score'].plot(yerr=[results['std_test_score'], results['std_test_score']], subplots=True)

plt.ylim(-0.3, 0)
plt.ylabel('Mean False Negative Rate')
plt.xlabel('Hyperparameter space')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10496515679442509, 'fraction_numerical': 0.019817073170731708, 'mean_word_length': 4.719800747198008, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '343217', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1642, 'n_tokens_neox': 1523, 'n_words': 403, 'path': 'Section-03-Metrics/03-01-Making-our-own-metrics.ipynb', 'repo_name': 'ankitario/hyperparameter-optimization', 'size': '4592'}