<jupyter_start><jupyter_text>_Lambda School Data Science — Model Validation_  Validate classification problemsObjectives- Imbalanced Classes- Confusion Matrix- ROC AUC Reading- [Simple guide to confusion matrix terminology](https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/)- [Precision and Recall](https://en.wikipedia.org/wiki/Precision_and_recall)  Preliminary We'll use [mlxtend](http://rasbt.github.io/mlxtend/) and [yellowbrick](http://www.scikit-yb.org/en/latest/) for visualizations. These libraries are already installed on Google Colab. But if you are running locally with Anaconda Python, you'll probably need to install them:```conda install -c conda-forge mlxtend conda install -c districtdatalabs yellowbrick``` We'll reuse the `train_validation_test_split` function from yesterday's lesson.<jupyter_code>from sklearn.model_selection import train_test_split

def train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, 
    random_state=None, shuffle=True):
        
    assert train_size + val_size + test_size == 1
    
    X_train_val, X_test, y_train_val, y_test = train_test_split(
        X, y, test_size=test_size, random_state=random_state, shuffle=shuffle)
    
    X_train, X_val, y_train, y_val = train_test_split(
        X_train_val, y_train_val, test_size=val_size/(train_size+val_size), 
        random_state=random_state, shuffle=shuffle)
    
    return X_train, X_val, X_test, y_train, y_val, y_test<jupyter_output><empty_output><jupyter_text>Fun demo!The next code cell does five things: 1. Generate dataWe use scikit-learn's [make_classification](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html) function to generate fake data for a binary classification problem, based on several parameters, including:- Number of samples- Weights, meaning "the proportions of samples assigned to each class."- Class separation: "Larger values spread out the clusters/classes and make the classification task easier."(We are generating fake data so it is easy to visualize.) 2. Split dataWe split the data three ways, into train, validation, and test sets. (For this toy example, it's not really necessary to do a three-way split. A two-way split, or even no split, would be ok. But I'm trying to demonstrate good habits, even in toy examples, to avoid confusion.) 3. Fit modelWe use scikit-learn to fit a [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) on the training data.We use this model parameter:> **class_weight : _dict or ‘balanced’, default: None_**> Weights associated with classes in the form `{class_label: weight}`. If not given, all classes are supposed to have weight one.> The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as `n_samples / (n_classes * np.bincount(y))`. 4. Evaluate modelWe use our Logistic Regression model, which was fit on the training data, to generate predictions for the validation data.Then we print [scikit-learn's Classification Report](https://scikit-learn.org/stable/modules/model_evaluation.htmlclassification-report), with many metrics, and also the accuracy score. We are comparing the correct labels to the Logistic Regression's predicted labels, for the validation set.  5. Visualize decision functionBased on these examples- https://imbalanced-learn.readthedocs.io/en/stable/auto_examples/combine/plot_comparison_combine.html- http://rasbt.github.io/mlxtend/user_guide/plotting/plot_decision_regions/example-1-decision-regions-in-2d<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score, classification_report
from sklearn.linear_model import LogisticRegression
from mlxtend.plotting import plot_decision_regions


#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.50, 0.50)
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = None

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0);<jupyter_output>precision    recall  f1-score   support

           0       0.96      0.92      0.94        48
           1       0.93      0.96      0.94        52

   micro avg       0.94      0.94      0.94       100
   macro avg       0.94      0.94      0.94       100
weighted avg       0.94      0.94      0.94       100

accuracy 0.94<jupyter_text>Try re-running the cell above with different values for these four parameters:- `n_samples`- `weights`- `class_sep`- `class_balance`For example, with a 50% / 50% class distribution:```n_samples = 1000weights = (0.50, 0.50)class_sep = 0.8class_balance = None```With a 95% / 5% class distribution:```n_samples = 1000weights = (0.95, 0.05)class_sep = 0.8class_balance = None```With the same 95% / 5% class distribution, but changing the Logistic Regression's `class_balance` parameter to `'balanced'` (instead of its default `None`)```n_samples = 1000weights = (0.95, 0.05)class_sep = 0.8class_balance = 'balanced'```With the same 95% / 5% class distribution, but with different values for `class_balance`:- `{0: 1, 1: 1}` _(equivalent to `None`)_- `{1: 1, 1: 2}`- `{1: 1, 1: 10}` _(roughly equivalent to `'balanced'` for this dataset)_- `{1: 1, 1: 100}`- `{1: 1, 1: 10000}`How do the evaluation metrics and decision region plots change?  What you can do about imbalanced classes [Learning from Imbalanced Classes](https://www.svds.com/tbt-learning-imbalanced-classes/) gives "a rough outline of useful approaches" : - Do nothing. Sometimes you get lucky and nothing needs to be done. You can train on the so-called natural (or stratified) distribution and sometimes it works without need for modification.- Balance the training set in some way:  - Oversample the minority class.  - Undersample the majority class.  - Synthesize new minority classes.- Throw away minority examples and switch to an anomaly detection framework.- At the algorithm level, or after it:  - Adjust the class weight (misclassification costs).  - Adjust the decision threshold.  - Modify an existing algorithm to be more sensitive to rare classes.- Construct an entirely new algorithm to perform well on imbalanced data. We demonstrated just one of these options: many scikit-learn classifiers have a `class_balance` parameter, which we can use to "adjust the class weight (misclassification costs)."The [imbalance-learn](https://github.com/scikit-learn-contrib/imbalanced-learn) library can be used to "oversample the minority class, undersample the majority class, or synthesize new minority classes."You can see how to "adjust the decision threshold" in a great blog post, [Visualizing Machine Learning Thresholds to Make Better Business Decisions](https://blog.insightdatascience.com/visualizing-machine-learning-thresholds-to-make-better-business-decisions-4ab07f823415).  Bank Marketing — getting startedhttps://archive.ics.uci.edu/ml/datasets/Bank+MarketingThe data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed. bank-additional-full.csv with all examples (41188) and 20 inputs, **ordered by date (from May 2008 to November 2010)**  Download data<jupyter_code>!wget https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip
!unzip bank-additional.zip
%cd bank-additional<jupyter_output>/content/bank-additional<jupyter_text>Load data, assign to X and y<jupyter_code>import pandas as pd

bank = pd.read_csv('bank-additional-full.csv', sep=';')

X = bank.drop(columns='y')
y = bank['y'] == 'yes'<jupyter_output><empty_output><jupyter_text>Split data We want to do "model selection (hyperparameter optimization) and performance estimation" so we'll choose a validation method from the diagram's green box.There is no one "right" choice here, but I'll choose "3-way holdout method (train/validation/test split)."  Source: https://sebastianraschka.com/blog/2018/model-evaluation-selection-part4.html There's no one "right" choice here, but I'll choose to split by time, not with a random shuffle, based on this advice by [<NAME>](https://www.fast.ai/2017/11/13/validation-sets/):> If your data is a time series, choosing a random subset of the data will be both too easy (you can look at the data both before and after the dates your are trying to predict) and not representative of most business use cases (where you are using historical data to build a model for use in the future).[According to UCI](https://archive.ics.uci.edu/ml/datasets/Bank+Marketing), this data is "ordered by date (from May 2008 to November 2010)" so if I don't shuffle it when splitting, then it will be split by time.<jupyter_code>X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, shuffle=False)<jupyter_output><empty_output><jupyter_text>Bank Marketing — live coding!  Naive Baseline Model<jupyter_code>import numpy as np

majority_class = y_train.mode()[0]
y_pred = np.full(shape=y_val.shape, fill_value=majority_class)
y_val.shape, y_pred.shape
y_pred
from sklearn.metrics import accuracy_score

accuracy_score(y_val, y_pred)
y_val.value_counts(normalize=True)    # Doing just this will tell you the majority class
from sklearn.metrics import accuracy_score, classification_report

print(classification_report(y_val, y_pred))
# roc_auc_score for majority classifier is always .5

from sklearn.metrics import roc_auc_score

roc_auc_score(y_val, y_pred)
!pip install category_encoders
from sklearn.preprocessing import StandardScaler
import category_encoders as ce
from sklearn.pipeline import make_pipeline

pipeline = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True),
    StandardScaler(),
    LogisticRegression()
)

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_val)

accuracy_score(y_val, y_pred)
roc_auc_score(y_val, y_pred)
from sklearn.metrics import confusion_matrix

confusion_matrix(y_val, y_pred)

%matplotlib inline
#from matplotlib.colors import ListedColorsMap
import matplotlib.pyplot as plt
import seaborn as sns

fig, ax = plt.subplots(figsize=(12, 8))

def confusion_viz(y_true, y_pred):
  matrix = confusion_matrix(y_val, y_pred)
  # cmap = ListedColorMap(['white'])
  return sns.heatmap(matrix, cbar=True, annot=True,
                    fmt=',', linewidth=1, linecolor='grey',
                    square=True,
                    xticklabels=['Predicted\nNO', 'Predicted\nYES'],
                    yticklabels=['Actual\nNO', 'Actual\nYES'])

confusion_viz(y_val, y_pred)<jupyter_output><empty_output><jupyter_text>ASSIGNMENT optionsReplicate code from the lesson or other examples. [Do it "the hard way" or with the "Benjamin Franklin method."](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit)Work with one of these datasets- [Bank Marketing](https://archive.ics.uci.edu/ml/datasets/Bank+Marketing)- [Synthetic Financial Dataset For Fraud Detection](https://www.kaggle.com/ntnu-testimon/paysim1)- Any imbalanced binary classification datasetContinue improving your model. Measure validation performance with a variety of classification metrics, which could include:- Accuracy- Precision- Recall- F1- ROC AUCTry one of the other options mentioned for imbalanced classes- The [imbalance-learn](https://github.com/scikit-learn-contrib/imbalanced-learn) library can be used to "oversample the minority class, undersample the majority class, or synthesize new minority classes."- You can see how to "adjust the decision threshold" in a great blog post, [Visualizing Machine Learning Thresholds to Make Better Business Decisions](https://blog.insightdatascience.com/visualizing-machine-learning-thresholds-to-make-better-business-decisions-4ab07f823415).  Working on Bank Data<jupyter_code>import pandas as pd

df = pd.read_csv('bank-additional-full.csv', sep=';')

df.head()

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

df.head()
df.shape
df.dtypes<jupyter_output><empty_output><jupyter_text>The Baseline models above had Accuracy Score of ~85% and ROC_AUC_SCORE of around 56%, and so those are what I'm trying to beat<jupyter_code>df['y'].value_counts(normalize=True)
import numpy as np

df['y_encoded'] = np.where(df['y'] == 'yes', 1, 0)
df.head()
df.dtypes
df_numeric = df.select_dtypes(include=['number']).copy()
df_numeric.head()<jupyter_output><empty_output><jupyter_text>Baseline Model 2 Using Logistic Regression on just the numerical data<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X = df_numeric.loc[:, 'age':'nr.employed'].values
y = df_numeric['y_encoded'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

lr = LogisticRegression(solver='lbfgs', max_iter=1000).fit(X_train, y_train)
y_pred_lr = lr.predict(X_test)

print("Baseline Model #2 Test score: {:.2f}".format(lr.score(X_test, y_test)))
# The test score alone does not tell us anything really meaningful, so let's evaluate Baseline Model 2:

from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, y_pred_lr)
print("Confusion Matrix:\n{}".format(confusion))
from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred_lr))
from sklearn.metrics import roc_auc_score

lr_auc = roc_auc_score(y_test, lr.predict_proba(X_test)[:, 1])

print("AUC (Area Under the Curve): {:.3f}".format(lr_auc))<jupyter_output>AUC (Area Under the Curve): 0.926<jupyter_text>The **ROC-AUC** is considered the gold-standard when it comes to evaluating classification models, and it's the area of the curve which shows FPR (False Positive Rate) and on the x-axis, and TPR (True Positive Rate) on the y-axis.  The ROC Curve shows the tradeoff between TPR and FPR for all different thresholds of the classfier which the classifier uses to make its predictions.  You want to be as far on the left corner as possible exemplifying **High TPR, and Low FPR**  Baseline Model 3: Simply running a LogisticRegression on just the numerical variables seems to have increased accuracy score.  But this one accuracy score doesn't give us vital clues how it might really perform on new test data.  So I'm going to re-run the Logistic but separating the dataset into a training set, a validation set, and test set with different parameters using a combination of **GridSearchCV with Cross-Validation** where scikit-learn's GridSearchCV will automatically perform Cross Validation for us on just the training set, so we can test the model on new data it hasn't seen before.We will summarize the performance of this Baseline model with:1. The best parameters2. Best Cross-Validation Score3. And a Test set score that will give us a clue on generalization performance<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import MinMaxScaler

X = df_numeric.loc[:, 'age':'nr.employed'].values
y = df_numeric['y_encoded'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


pipe = make_pipeline(
    MinMaxScaler(),
    LogisticRegression(solver='lbfgs', max_iter=1000)
)

# Define the parameter grid that we will search over to find the best parameters
param_grid = {
    'logisticregression__C': [0.001, 0.01, 0.1, 1, 10, 100]
}

# Fitting using GridSearchCV automatically performs cross-validation with different parameter settings
grid = GridSearchCV(pipe, param_grid=param_grid, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)

y_pred_grid = grid.predict(X_test)

print("Baseline Model #3:")
print()
print("Best parameters: {}".format(grid.best_params_))
print()
print("Best cross-validation scores: {:.2f}".format(grid.best_score_))
print()
print("Test set score: {:.2f}".format(grid.score(X_test, y_test)))
from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, y_pred_grid)
print("Confusion Matrix:\n{}".format(confusion))
from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred_grid))<jupyter_output>precision    recall  f1-score   support

           0       0.93      0.98      0.95      9144
           1       0.67      0.39      0.49      1153

   micro avg       0.91      0.91      0.91     10297
   macro avg       0.80      0.68      0.72     10297
weighted avg       0.90      0.91      0.90     10297<jupyter_text>This makes clear that the best model for Baseline Model 3 performs slightly worse than Baseline Model 2  in terms of Recall on the 'yes's for the bank dataset   Let's check out the ROC-AUC:<jupyter_code>from sklearn.metrics import roc_auc_score

grid_auc = roc_auc_score(y_test, grid.predict_proba(X_test)[:, 1])

print("AUC (Area Under the Curve): {:.3f}".format(grid_auc))<jupyter_output>AUC (Area Under the Curve): 0.926<jupyter_text>However, when we consider the AUC, Baseline Model 3 that has been scaled, with C paramter equal to 10 (higher C equals higher compelexity therefore more variance in terms of bias-variance tradeoff) we find a slightly better model.  Baseline Model 4: SVM with unscaled data<jupyter_code>from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

X = df_numeric.loc[:, 'age':'nr.employed'].values
y = df_numeric['y_encoded'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

svm = SVC().fit(X_train, y_train)
y_pred_svm = svm.predict(X_test)

print("\nBaseline Model #4 Test score: {:.2f}".format(svm.score(X_test, y_test)))
from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, y_pred_svm)
print("Confusion Matrix:\n{}".format(confusion))
from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred_svm))
# Very interesting result from SVM classification report that the SVM is barely correctly classifying the True Positives for the 1
# in dataset

from sklearn.metrics import roc_auc_score

svm_auc = roc_auc_score(y_test, svm.decision_function(X_test))

print("AUC (Area Under the Curve): {:.3f}".format(svm_auc))
# Again, because it predicts the 1('yes') so poorly, this SVM does no better than the naive baseline model which just predicts
# the most frequent class.<jupyter_output><empty_output><jupyter_text>Let's try using GridSearchCV to find a Support Vector Classifier with the best parameters and run Cross-Validation on it to get more reliable estimate of generalization performance:<jupyter_code>from sklearn.svm import SVC
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import MinMaxScaler

X = df_numeric.loc[:, 'age':'nr.employed'].values
y = df_numeric['y_encoded'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)


pipe = make_pipeline(
    MinMaxScaler(),
    SVC()
)

# Define the parameter grid that we will search over to find the best parameters
param_grid = {
    'svc__C': [0.001, 0.01, 0.1, 1, 10, 100],
    'svc__gamma': [0.001, 0.01, 0.1, 1, 10, 100]
}

# Fitting using GridSearchCV automatically performs cross-validation with different parameter settings
grid = GridSearchCV(pipe, param_grid=param_grid, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)

y_pred_grid = grid.predict(X_test)

print("Baseline Model #3:")
print()
print("Best parameters: {}".format(grid.best_params_))
print()
print("Best cross-validation scores: {:.2f}".format(grid.best_score_))
print()
print("Test set score: {:.2f}".format(grid.score(X_test, y_test)))<jupyter_output>Baseline Model #3:

Best parameters: {'svc__C': 100, 'svc__gamma': 1}

Best cross-validation scores: 0.91

Test set score: 0.91<jupyter_text>The scaled data did give us better scores for SVC and also with different parameter settings.  Let's evaluate this Model:<jupyter_code>from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, y_pred_grid)
print("Confusion Matrix:\n{}".format(confusion))
from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred_grid))<jupyter_output>precision    recall  f1-score   support

           0       0.92      0.98      0.95      9144
           1       0.68      0.33      0.45      1153

   micro avg       0.91      0.91      0.91     10297
   macro avg       0.80      0.66      0.70     10297
weighted avg       0.89      0.91      0.89     10297<jupyter_text>The Recall Score for this best model went up by a factor of 10!  While still low, this is definitely an improvement.<jupyter_code>from sklearn.metrics import roc_auc_score

svm_auc = roc_auc_score(y_test, grid.decision_function(X_test))

print("AUC (Area Under the Curve): {:.3f}".format(svm_auc))<jupyter_output>AUC (Area Under the Curve): 0.907<jupyter_text>Again, the ROC-AUC score tells us that this grid-searched fitted model is better, but still a little worse than the LogisticRegression  Let's try a Baseline Model 5 Using a RandomForestClassifier without scaled data<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X = df_numeric.loc[:, 'age':'nr.employed'].values
y = df_numeric['y_encoded'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

rf = RandomForestClassifier().fit(X_train, y_train)
y_pred_rf = rf.predict(X_test)

print("\nBaseline Model #5 Test score: {:.2f}".format(rf.score(X_test, y_test)))<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/ensemble/forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
  "10 in version 0.20 to 100 in 0.22.", FutureWarning)<jupyter_text>Right off the at, the RandomForestClassifier is performing at the high end of range.  Let's try to change some parameters and evaluate.<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X = df_numeric.loc[:, 'age':'nr.employed'].values
y = df_numeric['y_encoded'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

rf = RandomForestClassifier(n_estimators=100, min_samples_split=5).fit(X_train, y_train)
y_pred_rf = rf.predict(X_test)

print("\nBaseline Model #5 Test score: {:.2f}".format(rf.score(X_test, y_test)))
from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, y_pred_rf)
print("Confusion Matrix:\n{}".format(confusion))
from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred_rf))
from sklearn.metrics import roc_auc_score

rf_auc = roc_auc_score(y_test, rf.predict_proba(X_test)[:, 1])

print("AUC (Area Under the Curve): {:.3f}".format(rf_auc))<jupyter_output>AUC (Area Under the Curve): 0.942<jupyter_text>The RandomForestClassifier which only uses the numeric data outperforms both SVC and Logistic in terms of the AUC  Using SMOTE to balance the data<jupyter_code>!pip install imbalanced_learn
from imblearn.over_sampling import SMOTE

os = SMOTE(random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3, random_state=42)

os_data_X, os_data_y = os.fit_sample(X_train, y_train)


rf = RandomForestClassifier(n_estimators=100, min_samples_split=5).fit(os_data_X, os_data_y)
y_pred_rf = rf.predict(X_test)

print("\nBaseline Model #6 Test score: {:.2f}".format(rf.score(X_test, y_test)))
from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, y_pred_rf)
print("Confusion Matrix:\n{}".format(confusion))
from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred_rf))
from sklearn.metrics import roc_auc_score

rf_auc = roc_auc_score(y_test, rf.predict_proba(X_test)[:, 1])

print("AUC (Area Under the Curve): {:.3f}".format(rf_auc))
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import MinMaxScaler



pipe = make_pipeline(
    MinMaxScaler(),
    LogisticRegression(solver='lbfgs', max_iter=1000)
)

# Define the parameter grid that we will search over to find the best parameters
param_grid = {
    'logisticregression__C': [0.001, 0.01, 0.1, 1, 10, 100]
}

# Fitting using GridSearchCV automatically performs cross-validation with different parameter settings
grid = GridSearchCV(pipe, param_grid=param_grid, cv=5, n_jobs=-1)
grid.fit(os_data_X, os_data_y)

y_pred_grid = grid.predict(X_test)

print("Baseline Model #3:")
print()
print("Best parameters: {}".format(grid.best_params_))
print()
print("Best cross-validation scores: {:.2f}".format(grid.best_score_))
print()
print("Test set score: {:.2f}".format(grid.score(X_test, y_test)))
from sklearn.metrics import roc_auc_score

grid_auc = roc_auc_score(y_test, grid.predict_proba(X_test)[:, 1])

print("AUC (Area Under the Curve): {:.3f}".format(grid_auc))<jupyter_output>AUC (Area Under the Curve): 0.928<jupyter_text>Using SMOTE did slightly worse for RandomForestClassifier and slightly better for the LogisticRegression GridSearchCV model with respect to ROC-AUC score.<jupyter_code># y_train has been balanced half and half

os_data_y_s = pd.Series(os_data_y)
os_data_y_s.value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09522020141289644, 'fraction_numerical': 0.02378626183676537, 'mean_word_length': 5.117931034482758, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 71, 'https://': 19, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '475990', 'license': 'MIT', 'n_tokens_mistral': 8931, 'n_tokens_neox': 8303, 'n_words': 2627, 'path': 'Sammy Lee - LS_DS_242_Validate_classification_problems.ipynb', 'repo_name': 'Captmoonshot/DS-Unit-2-Sprint-4-Model-Validation', 'size': '26619'}