<jupyter_start><jupyter_text>Model evaluation for logistic regressionModel metrics with sklearn for logistic regression on the Titanic dataset *** Environment`conda activate sklearn-env`*** Goals- Build a pipeline- Use the pipeline to transform data- Use the pipeline to predict- Compute various metrics specific to classification models *** Referenceshttps://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.htmlhttps://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.htmlhttps://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.htmlhttps://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.htmlhttps://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.htmlhttps://scikit-learn.org/stable/modules/model_evaluation.html<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import random


# Make numpy printouts easier to read.
np.set_printoptions(precision=3, suppress=True)<jupyter_output><empty_output><jupyter_text>Dataset load from CSV located on UCI website.http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data  If the URL does not work the dataset can be loaded from the data folder `./data/auto-mpg.data`.<jupyter_code>from sklearn.datasets import fetch_openml

# Load data from https://www.openml.org/d/40945
raw_dataset = fetch_openml("titanic", version=1, as_frame=True).frame
dataset = raw_dataset.copy()
dataset.head(10)
dataset.drop(['boat', 'body', 'home.dest', 'fare', 'cabin'],  axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Dataset split- row base in test and train datasets- column base in features and labels<jupyter_code>train_dataset = dataset.sample(frac=0.8, random_state=random.randint(0, 1000))
test_dataset = dataset.drop(train_dataset.index)

train_features = train_dataset.copy()
test_features = test_dataset.copy()

train_labels = train_features.pop('survived')
test_labels = test_features.pop('survived')

test_labels.reset_index(drop=True, inplace=True)<jupyter_output><empty_output><jupyter_text>Build the pipeline<jupyter_code>from sklearn.preprocessing import MinMaxScaler
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder,OrdinalEncoder
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import FunctionTransformer

from sklearn.pipeline import Pipeline

numeric_features = ['age', 'sibsp', 'parch']
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', MinMaxScaler())])

custom_features = ['pclass']
custom_transformer = FunctionTransformer(np.square, validate=True)

categorical_features = ['embarked', 'sex']
ohe_transformer = OneHotEncoder(handle_unknown='ignore')
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('ohe', ohe_transformer)])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('ohe', categorical_transformer, categorical_features),
        ('cust', custom_transformer, custom_features)])

data_prep_pipeline = Pipeline(steps=[('preprocessor', preprocessor)])<jupyter_output><empty_output><jupyter_text>Use the pipeline to transform data<jupyter_code>data_prep_data = data_prep_pipeline.fit_transform(train_features, train_labels)

transformed_df = pd.DataFrame(data = data_prep_data)
transformed_df.head(10)<jupyter_output><empty_output><jupyter_text>Build the full pipeline<jupyter_code>from sklearn.linear_model import LogisticRegression
pipeline = Pipeline(steps=[('preprocessor', preprocessor), ('regressor', LogisticRegression())])
pipeline_model = pipeline.fit(train_features, train_labels)
print(pipeline_model)
from sklearn import set_config

set_config(display='diagram')
pipeline<jupyter_output><empty_output><jupyter_text>Use pipeline to predict values<jupyter_code>scored_data = pipeline.predict(test_features)<jupyter_output><empty_output><jupyter_text>Show predicted values<jupyter_code>scored_df = pd.DataFrame(data = scored_data)

scored_df = pd.concat([scored_df, test_labels], axis=1)
scored_df.head(10)<jupyter_output><empty_output><jupyter_text>Model evaluation  Confusion matrixhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html<jupyter_code>from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(test_labels, scored_data).ravel()<jupyter_output><empty_output><jupyter_text>Plot Confusion Matrixhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html<jupyter_code>from sklearn.metrics import plot_confusion_matrix

def plotConfusionMatrix(test_features, test_labels, pipeline):
    title = "Confusion matrix"
    
    disp = plot_confusion_matrix(pipeline, test_features, test_labels,
                                 display_labels=['Survived', 'Not survived'],
                                 cmap=plt.cm.Blues)
    disp.ax_.set_title(title)
    
    print(title)
    print(disp.confusion_matrix)
    
    plt.show()
plotConfusionMatrix(test_features, test_labels, pipeline)<jupyter_output><empty_output><jupyter_text>Metrics https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html  https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html  https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html  https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html  https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.htmlhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html<jupyter_code>def showMetrics(test_features, test_labels, scored_data, pipeline):
    from sklearn.metrics import accuracy_score
    print("Accuracy: ", accuracy_score(test_labels, scored_data))
    
    from sklearn.metrics import precision_score
    print( "Precission", precision_score(test_labels, scored_data, average='binary', pos_label='1'))
    
    from sklearn.metrics import recall_score
    print( "Sensitivity/Recall: ", recall_score(test_labels, scored_data, average='binary', pos_label='1'))

    print("Specify", tn / (tn+fp))
    
    from sklearn.metrics import f1_score
    print( "F1 score", f1_score(test_labels, scored_data, average='binary', pos_label='1'))
    
    from sklearn.metrics import roc_auc_score
    print( "AuROC", roc_auc_score(test_labels, pipeline.predict_proba(test_features)[:, 1]))

    from sklearn.metrics import log_loss
    print( "Cross-entropy loss", log_loss(test_labels, pipeline.predict_proba(test_features)[:, 1]))

    log_loss
    
showMetrics(test_features, test_labels, scored_data, pipeline)<jupyter_output><empty_output><jupyter_text>Roc Curve - plothttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.RocCurveDisplay.htmlhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_roc_curve.html<jupyter_code>import matplotlib.pyplot as plt  
import numpy as np
from sklearn import metrics

def plotAuROC(test_features, test_labels, pipeline, ax = None):
    fpr, tpr, thresholds = metrics.roc_curve(test_labels, pipeline.predict_proba(test_features)[:, 1], pos_label='1')
    roc_auc = metrics.auc(fpr, tpr)
    display = metrics.RocCurveDisplay(fpr=fpr, tpr=tpr, roc_auc=roc_auc, estimator_name='Logistic regression estimator')
    display.plot(ax)  
plotAuROC(test_features, test_labels, pipeline)
plt.show()<jupyter_output><empty_output><jupyter_text>Build basic model without feature engineering and compare metrics<jupyter_code>train_dataset_new = train_dataset[['pclass','sibsp','parch', 'survived' ]].copy()
test_dataset_new = test_dataset[['pclass','sibsp','parch', 'survived' ]].copy()

train_features_new = train_dataset_new.copy()
test_features_new = test_dataset_new.copy()

train_labels_new = train_features_new.pop('survived')
test_labels_new = test_features_new.pop('survived')
from sklearn.linear_model import LogisticRegression

logistic_regressor_new = LogisticRegression().fit(train_features_new, train_labels_new)
scored_data_new = logistic_regressor_new.predict(test_features_new)
tn, fp, fn, tp = confusion_matrix(test_labels_new, scored_data_new).ravel()
plotConfusionMatrix(test_features_new, test_labels_new, logistic_regressor_new)
showMetrics(test_features_new, test_labels_new, scored_data_new, logistic_regressor_new)
fig, ax = plt.subplots(1,1)
plotAuROC(test_features, test_labels, pipeline, ax)
plotAuROC(test_features_new, test_labels_new, logistic_regressor_new, ax)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11052389705882353, 'fraction_numerical': 0.0040211397058823525, 'mean_word_length': 7.112767940354147, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 17, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '93514', 'license': 'Apache-2.0', 'n_tokens_mistral': 2856, 'n_tokens_neox': 2791, 'n_words': 462, 'path': 'notebooks/05_01_Logistic_SkLearn_Metrics.ipynb', 'repo_name': 'datagridsoftware/practical-ml', 'size': '8704'}