<jupyter_start><jupyter_text>Model Evaluation and Selection **Agenda**- **Regression Evaluation**- What is the usage of **classification accuracy**?- How does a **confusion matrix** describe the performance of a classifier?- **Precision, Recall, F1-score** and **Fb-Score**- **ROC, AUC curve**- Evaluation measures for **multi-class** classification- **Overfitting** and **Underfitting**- **Cross-Validation**- **Learning Curves**- Searching for optimal tuning parameters using **Grid Search** and **Random Search** **Load Libs**<jupyter_code>from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import warnings; warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Regression Evaluation<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.datasets import load_boston

Dataset = load_boston()
X,y = Dataset.data, Dataset.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=.3)

clf = LinearRegression().fit(X_train, y_train)
y_pred = clf.predict(X_test)<jupyter_output><empty_output><jupyter_text>Mean Absolute Error<jupyter_code>from sklearn.metrics import mean_absolute_error
mean_absolute_error(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>Mean Squared Error<jupyter_code>from sklearn.metrics import mean_squared_error
mean_squared_error(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>R2 Score<jupyter_code>from sklearn.metrics import r2_score
r2_score(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>Classification Evaluation  Imbalanced-Data Classification Accuracy**Dataset Link:** https://www.kaggle.com/mlg-ulb/creditcardfraud **Alternatively** unzip credit_card_fraud.rar<jupyter_code>Dataset = pd.read_csv('credit_card_fraud.csv')
print(Dataset.shape)
Dataset.head(10)
X = Dataset.iloc[:,:-1]
target = Dataset.iloc[:,Dataset.shape[1]-1]

target_count = target.value_counts()
print('Not Fraud:', target_count[0])
print('Fraud:', target_count[1])

target_count.plot(kind='bar', title='Count (target)');<jupyter_output>Not Fraud: 284315
Fraud: 492<jupyter_text>**Logistic Regression**<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, target, random_state=42, test_size=.3)

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(random_state = 42).fit(X_train, y_train)
y_pred = lr.fit(X_train, y_train).predict(X_test)
from sklearn import metrics
print("Accurcy Score: {0}".format(metrics.accuracy_score(y_test, y_pred)))<jupyter_output>Accurcy Score: 0.9989466661985184<jupyter_text>**Dummy Classifiers**<jupyter_code>from sklearn.dummy import DummyClassifier

dummy_majority = DummyClassifier(strategy = 'most_frequent').fit(X_train, y_train)
y_dummy_predictions = dummy_majority.predict(X_test)
print("Dummy Score: {0}".format(dummy_majority.score(X_test, y_test)))<jupyter_output>Dummy Score: 0.9984082955888721<jupyter_text>**Accuracy**   Confusion matrix **Example**   **Confusion Matrix of Dummy Classifer**<jupyter_code>from sklearn.metrics import confusion_matrix

confusion_dummy = confusion_matrix(y_true=y_test, y_pred=y_dummy_predictions)
print(pd.DataFrame(data=confusion_dummy, 
             index=["Actuall Not Fraud","Actuall Fraud"],
             columns =["Predicted Not Fraud","Predicted Fraud"]))


labels = ['Not Fraud', 'Fraud']
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(confusion_dummy, cmap=plt.cm.Blues)
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('Actuall')
plt.show()<jupyter_output>Predicted Not Fraud  Predicted Fraud
Actuall Not Fraud                85307                0
Actuall Fraud                      136                0<jupyter_text>**Confusion Matrix of Logistic regression**<jupyter_code>confusion = confusion_matrix(y_true=y_test, y_pred=y_pred)
print(pd.DataFrame(data=confusion, 
             index=["Actuall Not Fraud","Actuall Fraud"],
             columns =["Predicted Not Fraud","Predicted Fraud"]))


labels = ['Not Fraud', 'Fraud']
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(confusion, cmap=plt.cm.Blues)
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('Actuall')
plt.show()<jupyter_output>Predicted Not Fraud  Predicted Fraud
Actuall Not Fraud                85271               36
Actuall Fraud                       54               82<jupyter_text>PrecisionHow many of the predicted positive examples are really positive?   Recall (Sensitivity)When the actual value is positive, how often is the prediction correct? **Trading Off Precision and Recall**Can be achieved by e.g. varying decision **threshold** of a classifier.- Suppose we want to predict y=1 only if very **confident**    - Higher threshold, higher precision, lower recall.- Suppose we want to **avoid missing** positive examples    - Lower threshold, higher recall, lower precision.    **Which metrics should you focus on?**- Choice of metric depends on your **business objective**- **Spam filter** (positive class is "spam"): Optimize for **precision or specificity** because false negatives (spam goes to the inbox) are more acceptable than false positives (non-spam is caught by the spam filter)- **Fraudulent transaction detector** (positive class is "fraud"): Optimize for **Recall (sensitivity)** because false positives (normal transactions that are flagged as possible fraud) are more acceptable than false negatives (fraudulent transactions that are not detected)  F1-Score **F1-Score** is the **Harmonic Mean** between precision and recall   Fb-ScoreGives a percentage more **importance/weight** to either **precision** or **recall**<jupyter_code>from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
print("Score of Logistic Regression Classifer:")
print(pd.DataFrame(data=[accuracy_score(y_test, y_pred),
                        precision_score(y_test, y_pred),
                        recall_score(y_test, y_pred),
                        f1_score(y_test, y_pred)],
                   index=["Accuracy","Precision","Recall","F1-Score"],
                   columns =["Value"]))

print("Score of Dummy Regression Classifer:")
print(pd.DataFrame(data=[accuracy_score(y_test, y_dummy_predictions),
                        precision_score(y_test, y_dummy_predictions),
                        recall_score(y_test, y_dummy_predictions),
                        f1_score(y_test, y_dummy_predictions)],
                   index=["Accuracy","Precision","Recall","F1-Score"],
                   columns =["Value"]))
from sklearn.metrics import classification_report
print("Score of Logistic Regression Classifier:")
print(classification_report(y_test, y_pred, target_names=['Not Fraud', 'Fraud']))

print("Score of Dummy Classifier:")
print(classification_report(y_test, y_dummy_predictions, target_names=['Not Fraud', 'Fraud']))
from sklearn.metrics import fbeta_score
print("Fb-Score: {:.2f}".format(fbeta_score(y_test, y_pred, average='macro', beta=2)))
print("Fb-Score: {:.2f}".format(fbeta_score(y_test, y_pred, average='macro', beta=1)))
for THRESHOLD in np.arange(0.1, 1.0, 0.1):
    preds = np.where(lr.predict_proba(X_test)[:,1] > THRESHOLD, 1, 0)
    print ("THRESHOLD= {:.1f}".format(THRESHOLD))
    print(pd.DataFrame(data=[precision_score(y_test, preds),
                   recall_score(y_test, preds),], 
                    index=["Precision", "Recall"],
                    columns =["Value"]))
    print()
    print()<jupyter_output>THRESHOLD= 0.1
              Value
Precision  0.123393
Recall     0.705882


THRESHOLD= 0.2
              Value
Precision  0.286624
Recall     0.661765


THRESHOLD= 0.3
              Value
Precision  0.530120
Recall     0.647059


THRESHOLD= 0.4
              Value
Precision  0.685484
Recall     0.625000


THRESHOLD= 0.5
              Value
Precision  0.694915
Recall     0.602941


THRESHOLD= 0.6
              Value
Precision  0.720721
Recall     0.588235


THRESHOLD= 0.7
              Value
Precision  0.750000
Recall     0.551471


THRESHOLD= 0.8
              Value
Precision  0.776596
Recall     0.536765


THRESHOLD= 0.9
              Value
Precision  0.806818
Recall     0.522059<jupyter_text>ROC curves, Area-Under-Curve (AUC) **Recall (Sensitivity)****When the actual value is positive, how often is the prediction correct?****Specificity****When the actual value is negative, how often is the prediction correct?**<jupyter_code>from sklearn.metrics import roc_curve, auc

y_score_lr = lr.fit(X_train, y_train).decision_function(X_test)
fpr_lr, tpr_lr, _ = roc_curve(y_test, y_score_lr)

plt.figure(num=None, figsize=(7, 7), dpi=80, facecolor='w', edgecolor='k')
plt.xlim([-0.01, 1.00])
plt.ylim([-0.01, 1.01])
plt.plot(fpr_lr, tpr_lr, lw=3, label='AUC: {:0.2f})'.format(metrics.roc_auc_score(y_test, y_score_lr)))
plt.xlabel('False Positive Rate (Specifity)', fontsize=16)
plt.ylabel('True Positive Rate (Sensitivity)', fontsize=16)
plt.title('ROC curve', fontsize=16)
plt.legend(loc='lower right', fontsize=13)
plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')
plt.axes().set_aspect('equal')
plt.show()<jupyter_output><empty_output><jupyter_text>**AUC = 0 (worst) AUC = 1 (best)****ROC/AUC advantages:**- Does not require you to **set a classification threshold**- Still useful when there is **high class imbalance**- Gives a **single number** for easy comparison.  Evaluation measures for multi-class classification **Data Preparation**<jupyter_code>from sklearn.ensemble import RandomForestClassifier
import seaborn as sns
from sklearn.datasets import fetch_olivetti_faces

dataset = fetch_olivetti_faces()
X, y = dataset.data, dataset.target
X_train_mc, X_test_mc, y_train_mc, y_test_mc = train_test_split(X, y, random_state=0)<jupyter_output><empty_output><jupyter_text>**Classification**<jupyter_code>clf = RandomForestClassifier(n_estimators=200, max_depth=15,
                             random_state=42).fit(X_train_mc, y_train_mc)

clf_predicted_mc = clf.predict(X_test_mc)<jupyter_output><empty_output><jupyter_text>**Confusion Matrix**<jupyter_code>confusion_mc = confusion_matrix(y_test_mc, clf_predicted_mc)

df_cm = pd.DataFrame(confusion_mc, 
                     index = [i for i in range(0,confusion_mc.shape[0])], columns = [i for i in range(0,confusion_mc.shape[1])])

plt.figure(num=None, figsize=(7, 7), dpi=80, facecolor='w', edgecolor='k')

sns.heatmap(df_cm, annot=True)
plt.title('Random Forest Classifier \nAccuracy:{0:.3f}'.format(accuracy_score(y_test_mc, 
                                                                       clf_predicted_mc)))
plt.ylabel('Actuall label')
plt.xlabel('Predicted label')
print(classification_report(y_test_mc, clf_predicted_mc))<jupyter_output>precision    recall  f1-score   support

          0       1.00      0.67      0.80         6
          1       1.00      1.00      1.00         4
          2       0.67      1.00      0.80         2
          3       1.00      1.00      1.00         1
          4       1.00      1.00      1.00         1
          5       1.00      1.00      1.00         5
          6       1.00      1.00      1.00         4
          7       0.75      1.00      0.86         3
          9       1.00      1.00      1.00         1
         10       1.00      1.00      1.00         4
         11       1.00      1.00      1.00         1
         12       0.67      1.00      0.80         2
         13       1.00      1.00      1.00         3
         14       1.00      1.00      1.00         5
         15       0.67      0.67      0.67         3
         17       1.00      0.67      0.80         6
         19       1.00      1.00      1.00         4
         20       1.00      1.00      1.00   [...]<jupyter_text>Overfitting and Underfitting- Machine Learning models have one sole purpose; to generalize well.- A model that **generalizes** well is a model that is neither **underfit** nor **overfit.**- **Generalization** is the model’s ability to give sensible outputs to sets of input that it has never seen before.<jupyter_code>from sklearn.tree import DecisionTreeRegressor

# Create a random dataset
rng = np.random.RandomState(1)
X = np.sort(5 * rng.rand(80, 1), axis=0)
y = np.sin(X).ravel()
y[::5] += 3 * (0.5 - rng.rand(16))


# Plot the results
plt.figure(num=None, figsize=(7, 7), dpi=80, facecolor='w', edgecolor='k')
plt.scatter(X, y, s=20, edgecolor="black",
            c="darkorange", label="data")
plt.xlabel("data")
plt.ylabel("target")
plt.title("Data")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Overfitting** is the case where the overall cost is really small, but the generalization of the model is unreliable. This is due to the model learning “too much” from the training data set.<jupyter_code># Fit regression model
clf = DecisionTreeRegressor(max_depth=15)
clf.fit(X, y)

# Predict
y_1 = clf.predict(X)

# Plot the results
plt.figure(num=None, figsize=(7, 7), dpi=80, facecolor='w', edgecolor='k')
plt.scatter(X, y, s=20, edgecolor="black",
            c="darkorange", label="data")
plt.plot(X, y_1, color="cornflowerblue",
         label="max_depth=15", linewidth=2)
plt.xlabel("data")
plt.ylabel("target")
plt.title("Overfitting")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Overfitting** happens when the model is too complex relative to the amount and noisiness of the training data.**Possible Solutions:**- Simplify the model by selecting one with fewer parameters.- Gather more training data.- Reduce the noise in the training data.- Apply Cross- Validation. **Underfitting** is the case where the model has “ not learned enough” from the training data, resulting in low generalization and unreliable predictions.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, test_size=0.9)

# Fit regression model
clf = DecisionTreeRegressor(max_depth=1)
clf.fit(X_train, y_train)

# Predict
X_test = np.arange(0.0, 5.0, 0.01)[:, np.newaxis]
y_test = np.arange(0.0, 5.0, 0.01)[:, np.newaxis]
y_1 = clf.predict(X_test)

# Plot the results
plt.figure(num=None, figsize=(7, 7), dpi=80, facecolor='w', edgecolor='k')
plt.scatter(X, y, s=20, edgecolor="black",
            c="darkorange", label="data")
plt.plot(X_test, y_1, color="cornflowerblue",
         label="max_depth=1", linewidth=2)
plt.xlabel("data")
plt.ylabel("target")
plt.title("Underfitting")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Underfitting** occurs when your model is too simple to learn the underlying structure of the data.**Possible Solutions:**- Selecting a more powerful model, with more parameters.- Feeding better features to the learning algorithm.- Reducing the constraints on the model,  Model selection using evaluation metrics**Train/test on same data**   - Typically overfits and likely won't generalize well to new data.   - **Not recommended**   Single train/test split (Holdout)   - Fast and simple.   - In most patterns we don't have the **luxury** of large data set.   - Generally, this split will be close to 80% of the data for training and 20% of the data for testing.   - Splitting the data set into training and testing sets will leave us with either **insufficient training or testing patterns**.    - Clearly the testing data set contains **useful information** for learning. Yet, they are **ignored** and not used for training purposes in the data splitting error rate estimation method.   - If dataset is not completely even. In splitting our dataset we may end up splitting it in such a way that our training set is very different from the test<jupyter_code>from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn import metrics
digits = load_breast_cancer()
X = digits.data
y = digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2 , random_state=2233)

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

          0       0.90      0.82      0.86        45
          1       0.89      0.94      0.92        69

avg / total       0.90      0.89      0.89       114<jupyter_text>K-fold cross-validation   - K train-test splits.   - Average metric over all splits.   - Randomly split our dataset into K equally sized parts. **K-fold cross-validation algorithm**1. Split the dataset into K **equal** partitions (or "folds").2. Use fold 1 as the **testing set** and the union of the other folds as the **training set**.3. Calculate **testing accuracy**.4. Repeat steps 2 and 3 K times, using a **different fold** as the testing set each time.5. Use the **average testing accuracy** as the estimate of out-of-sample accuracy.<jupyter_code>data = load_breast_cancer()
X, y = data.data, data.target

from sklearn.model_selection import cross_val_score

knn = KNeighborsClassifier(n_neighbors=5)

# use recall as scoring metric
scores = cross_val_score(knn, X, y, cv=5, scoring = 'f1')
print('Cross-validation (F1-Score)', scores)
print("Average cross-validation score (F1-Score) : {:.3f}".format(scores.mean()))
print()<jupyter_output>Cross-validation (F1-Score) [ 0.91503268  0.95238095  0.95104895  0.95833333  0.94285714]
Average cross-validation score (F1-Score) : 0.944<jupyter_text>RepeatedKFold-  the most robust of cross validation methods.- Similar to K-Fold, we set a value for K which signifies the number of times we will train our model. However, in this case K will not represent the number of equally sized partitions.- on each training iteration, we randomly select points to be for the testing set.- The advantage of this method over K-Fold is that the proportion of the train-test split is not dependent on the number of iterations.- The disadvantage of this method is that some points may never be selected to be in the test subset at all — at the same time, some points might be selected multiple times.<jupyter_code>from sklearn.metrics import f1_score
from sklearn.model_selection import RepeatedKFold
data = load_breast_cancer()
X, y = data.data, data.target
rkf = RepeatedKFold(n_splits=5, n_repeats=2, random_state=42)

l = []
knn = KNeighborsClassifier(n_neighbors=5)

for train_index, test_index in rkf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    knn.fit(X_train, y_train)
    y_pred = knn.predict(X_test)
    l.append(f1_score(y_test, y_pred))  

print('RepeatedKFold (F1-Score)', l)
print("Average RepeatedKFold score (F1-Score) : {:.3f}".format(sum(l) / len(l)))<jupyter_output>RepeatedKFold (F1-Score) [0.9659863945578232, 0.97333333333333327, 0.91666666666666663, 0.965034965034965, 0.92753623188405798, 0.93877551020408156, 0.93877551020408168, 0.9419354838709677, 0.96453900709219853, 0.94117647058823539]
Average RepeatedKFold score (F1-Score) : 0.947<jupyter_text>Learning Curves - **Train Learning Curve:** Learning curve calculated from the training dataset that gives an idea of how well the model is learning. - **Validation Learning Curve:** Learning curve calculated from a hold-out validation dataset that gives an idea of how well the model is generalizing.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.datasets import load_digits
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit
from sklearn.tree import DecisionTreeClassifier


def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
                        n_jobs=None, train_sizes=np.linspace(.1, 1.0, 5)):

    plt.figure()
    plt.title(title)
    if ylim is not None:
        plt.ylim(*ylim)
    plt.xlabel("Training examples")
    plt.ylabel("Score")
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="validation score")

    plt.legend(loc="best")
    return plt

digits = load_digits()
X, y = digits.data, digits.target<jupyter_output><empty_output><jupyter_text>**Underfitting:** A model of a given complexity will underfit a large dataset: this means that the training score will decrease, but the validation score will increase1. Example of Training Learning Curve Showing an Underfit Model That Requires Further Training<jupyter_code>title = "Learning Curves (Naive Bayes)"
estimator = GaussianNB()
plot_learning_curve(estimator, title, X, y, ylim=(0.7, 1.01))<jupyter_output><empty_output><jupyter_text>2. Example of Training Learning Curve Showing An Underfit Model That Does Not Have Sufficient Capacity<jupyter_code>title = "Learning Curves (SVC)"
estimator = SVC(gamma=0.1)
plot_learning_curve(estimator, title, X, y)
plt.show()<jupyter_output><empty_output><jupyter_text>**Overfitting:** A model of a given complexity will overfit a small dataset: this means the training score will be relatively high, while the validation score increases to a point and begins decreasing again.<jupyter_code>title = "Learning Curves (Decision Tree)"
cv = ShuffleSplit(n_splits=10, test_size=0.001, random_state=0)
estimator = DecisionTreeClassifier(random_state=42, max_depth=100)
plot_learning_curve(estimator, title, X, y, (0.7, 1.01), cv=cv)
plt.show()<jupyter_output><empty_output><jupyter_text>**Good Fit:** A model will never, except by chance, give a better score to the validation set than the training set: this means the curves should keep getting closer together but never cross.<jupyter_code>title = "Learning Curves (SVC)"
cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0)
estimator = SVC(gamma=0.001)
plot_learning_curve(estimator, title, X, y, (0.7, 1.01), cv=cv)
plt.show()<jupyter_output><empty_output><jupyter_text>Searching for optimal tuning parameters using Grid Search and Random Search Grid Search<jupyter_code>from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
dataset = load_digits()
X, y = dataset.data, dataset.target == 1
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

clf = SVC(kernel='rbf')
grid_values = {'C': [ 0.01, 0.1, 1, 10, 100],
               'gamma': [ 0.005, 0.01, 0.05, 0.1, 1, 10, 100]}

grid_clf = GridSearchCV(clf, param_grid = grid_values,cv=5, scoring = 'f1_micro')
grid_clf.fit(X_train, y_train)
y_decision_fn_scores = grid_clf.decision_function(X_test) 

print('Grid best score (F1): {:.2f}'.format(grid_clf.best_score_))
print('Grid best parameter (max. F1): ', grid_clf.best_params_)
print('Grid best Estimator: ',grid_clf.best_estimator_)<jupyter_output><empty_output><jupyter_text>Random Search<jupyter_code>from sklearn.model_selection import RandomizedSearchCV
from sklearn.neighbors import KNeighborsClassifier

param_dist = dict(n_neighbors=list(range(1, 200)), weights=['uniform', 'distance']
                  ,algorithm =['auto', 'ball_tree', 'kd_tree', 'brute']
                  ,leaf_size=list(range(1, 70)), p = [1,2]
                 )

knn = KNeighborsClassifier()
rand = RandomizedSearchCV(knn, param_dist, cv=5, scoring='f1_micro', n_iter=100)
rand.fit(X_train, y_train)
y_decision_fn_scores = rand.predict(X_test) 

print('Grid best score (F1): {:.2f}'.format(rand.best_score_))
print('Grid best parameter (max. F1): ', rand.best_params_)
print('Grid best Estimator: ',rand.best_estimator_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1130718954248366, 'fraction_numerical': 0.040236928104575166, 'mean_word_length': 4.1119231572353305, 'pattern_counts': {'":': 0, '<': 103, '<?xml version=': 0, '>': 104, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '24607', 'license': 'MIT', 'n_tokens_mistral': 8852, 'n_tokens_neox': 8042, 'n_words': 2160, 'path': 'Model Evaluation and Selection.ipynb', 'repo_name': 'aoso3/Efficient-Machine-Learning', 'size': '24480'}