<jupyter_start><jupyter_text>Exercise 5.5    In Chapter 4, we used logistic regression to predict the probability of $\mathrm{default}$ using $\mathrm{income}$ and $\mathrm{balance}$ on the Default data set. We will now estimate the test error of this logistic regression model using the validation set approach. Do not forget to set a random seed before beginning your analysis.            Fit a logistic regression model that uses $\mathrm{income}$ and $\mathrm{balance}$ to predict $\mathrm{default}$.        Using the validation set approach, estimate the test error of this model. In order to do this, you must perform the following steps:                            Split the sample set into a training set and a validation set.                Fit a multiple logistic regression model using only the training observations.                Obtain a prediction of default status for each individual in the validation set by computing the posterior probability of default for that individual, and classifying the individual to the $\mathrm{default}$ category if the posterior probability is greater than $0.5$.                Compute the validation set error, which is the fraction of the observations in the validation set that are misclassified.                            Repeat the process in 2 three times, using three different splits of the observations into a training set and a validation set. Comment on the results obtained.        Now consider a logistic regression model that predicts the probability of $\mathrm{default}$ using $\mathrm{income}$, $\mathrm{balance}$, and a dummy variable for $\mathrm{student}$. Estimate the test error for this model using the validation set approach. Comment on whether or not including a dummy variable for $\mathrm{student}$ leads to a reduction in the test error rate.<jupyter_code>import pandas as pd
import numpy as np

%run ../../customModules/usefulFunctions.ipynb
# https://stackoverflow.com/questions/34398054/ipython-notebook-cell-multiple-outputs
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

from sklearn.model_selection import train_test_split
import statsmodels.api as sm<jupyter_output><empty_output><jupyter_text>Exercise 5.5.1    Fit a logistic regression model that uses $\mathrm{income}$ and $\mathrm{balance}$ to predict $\mathrm{default}$.<jupyter_code>df = pd.read_csv("../../DataSets/Default/Default.csv")
df['default01'] = np.where(df['default'] == 'Yes', 1, 0)
df.insert(0, 'Intercept', 1)
targetColumn = ['default01']
descriptiveColumns = ['Intercept', 'balance', 'income']
df_X = df[descriptiveColumns]
df_Y = df[targetColumn]
model = sm.Logit(df_Y, df_X)
fitted = model.fit()
fitted.summary()<jupyter_output>Optimization terminated successfully.
         Current function value: 0.078948
         Iterations 10<jupyter_text>Exercise 5.5.2    Using the validation set approach, estimate the test error of this model. In order to do this, you must perform the following steps:                            Split the sample set into a training set and a validation set.                Fit a multiple logistic regression model using only the training observations.                Obtain a prediction of default status for each individual in the validation set by computing the posterior probability of default for that individual, and classifying the individual to the $\mathrm{default}$ category if the posterior probability is greater than $0.5$.                Compute the validation set error, which is the fraction of the observations in the validation set that are misclassified.<jupyter_code>df_train, df_test = train_test_split(df, test_size=0.5, random_state=42)
df_X_train = df_train[descriptiveColumns]
df_Y_train = df_train[targetColumn]
df_X_test = df_test[descriptiveColumns]
df_Y_test = df_test[targetColumn]

model = sm.Logit(df_Y_train, df_X_train)
fitted = model.fit()
fitted.summary()

sr_Y_pred = fitted.predict(df_X_test)
df_Y_test_and_pred = pd.DataFrame({
    'Observed': df_Y_test['default01'],
    'Predicted': np.where(sr_Y_pred > 0.5, 1, 0),
})
df_confusion, df_confusion_pct = createConfusionMatrixFromOutOfSampleData(df=df_Y_test_and_pred, binaryMap={0: 'less than median', 1: 'greater than median'})
df_confusion  
df_confusion_pct.round(2)

confusion_matrix = df_confusion.to_numpy()
TN, FP, FN, TP = confusion_matrix[0, 0], confusion_matrix[0, 1], confusion_matrix[1, 0], confusion_matrix[1, 1]
missclass_acc = 100 * ((FP + FN) / (TN + FP + FN + TP))
f'The validation set error is {missclass_acc:.2f}%.'<jupyter_output>Optimization terminated successfully.
         Current function value: 0.078493
         Iterations 10<jupyter_text>Exercise 5.5.3    Repeat the process in 2 three times, using three different splits of the observations into a training set and a validation set. Comment on the results obtained.<jupyter_code>df_train, df_test = train_test_split(df, test_size=0.5, random_state=42)
df_X_train = df_train[descriptiveColumns]
df_Y_train = df_train[targetColumn]
df_X_test = df_test[descriptiveColumns]
df_Y_test = df_test[targetColumn]

model = sm.Logit(df_Y_train, df_X_train)
fitted = model.fit()

sr_Y_pred = fitted.predict(df_X_test)
df_Y_test_and_pred = pd.DataFrame({
    'Observed': df_Y_test['default01'],
    'Predicted': np.where(sr_Y_pred > 0.5, 1, 0),
})
df_confusion, df_confusion_pct = createConfusionMatrixFromOutOfSampleData(df=df_Y_test_and_pred, binaryMap={0: 'less than median', 1: 'greater than median'})
confusion_matrix = df_confusion.to_numpy()
TN, FP, FN, TP = confusion_matrix[0, 0], confusion_matrix[0, 1], confusion_matrix[1, 0], confusion_matrix[1, 1]
missclass_acc = 100 * ((FP + FN) / (TN + FP + FN + TP))
f'The validation set error is {missclass_acc:.2f}%.'

'-----------------------'

f_train, df_test = train_test_split(df, test_size=0.5, random_state=43)
df_X_train = df_train[descriptiveColumns]
df_Y_train = df_train[targetColumn]
df_X_test = df_test[descriptiveColumns]
df_Y_test = df_test[targetColumn]

model = sm.Logit(df_Y_train, df_X_train)
fitted = model.fit()

sr_Y_pred = fitted.predict(df_X_test)
df_Y_test_and_pred = pd.DataFrame({
    'Observed': df_Y_test['default01'],
    'Predicted': np.where(sr_Y_pred > 0.5, 1, 0),
})
df_confusion, df_confusion_pct = createConfusionMatrixFromOutOfSampleData(df=df_Y_test_and_pred, binaryMap={0: 'less than median', 1: 'greater than median'})
confusion_matrix = df_confusion.to_numpy()
TN, FP, FN, TP = confusion_matrix[0, 0], confusion_matrix[0, 1], confusion_matrix[1, 0], confusion_matrix[1, 1]
missclass_acc = 100 * ((FP + FN) / (TN + FP + FN + TP))
f'The validation set error is {missclass_acc:.2f}%.'

'-----------------------'

f_train, df_test = train_test_split(df, test_size=0.5, random_state=44)
df_X_train = df_train[descriptiveColumns]
df_Y_train = df_train[targetColumn]
df_X_test = df_test[descriptiveColumns]
df_Y_test = df_test[targetColumn]

model = sm.Logit(df_Y_train, df_X_train)
fitted = model.fit()

sr_Y_pred = fitted.predict(df_X_test)
df_Y_test_and_pred = pd.DataFrame({
    'Observed': df_Y_test['default01'],
    'Predicted': np.where(sr_Y_pred > 0.5, 1, 0),
})
df_confusion, df_confusion_pct = createConfusionMatrixFromOutOfSampleData(df=df_Y_test_and_pred, binaryMap={0: 'less than median', 1: 'greater than median'})
confusion_matrix = df_confusion.to_numpy()
TN, FP, FN, TP = confusion_matrix[0, 0], confusion_matrix[0, 1], confusion_matrix[1, 0], confusion_matrix[1, 1]
missclass_acc = 100 * ((FP + FN) / (TN + FP + FN + TP))
f'The validation set error is {missclass_acc:.2f}%.'<jupyter_output>Optimization terminated successfully.
         Current function value: 0.078493
         Iterations 10<jupyter_text>The average validation set error of these 3 fitted models is $2.66 \%$.Exercise 5.5.4    Now consider a logistic regression model that predicts the probability of $\mathrm{default}$ using $\mathrm{income}$, $\mathrm{balance}$, and a dummy variable for $\mathrm{student}$. Estimate the test error for this model using the validation set approach. Comment on whether or not including a dummy variable for $\mathrm{student}$ leads to a reduction in the test error rate.<jupyter_code>df['student01'] = np.where(df['student'] == 'Yes', 1, 0)
descriptiveColumns = ['Intercept', 'balance', 'income', 'student01']

df_train, df_test = train_test_split(df, test_size=0.5, random_state=42)
df_X_train = df_train[descriptiveColumns]
df_Y_train = df_train[targetColumn]
df_X_test = df_test[descriptiveColumns]
df_Y_test = df_test[targetColumn]

model = sm.Logit(df_Y_train, df_X_train)
fitted = model.fit()

sr_Y_pred = fitted.predict(df_X_test)
df_Y_test_and_pred = pd.DataFrame({
    'Observed': df_Y_test['default01'],
    'Predicted': np.where(sr_Y_pred > 0.5, 1, 0),
})
df_confusion, df_confusion_pct = createConfusionMatrixFromOutOfSampleData(df=df_Y_test_and_pred, binaryMap={0: 'less than median', 1: 'greater than median'})
confusion_matrix = df_confusion.to_numpy()
TN, FP, FN, TP = confusion_matrix[0, 0], confusion_matrix[0, 1], confusion_matrix[1, 0], confusion_matrix[1, 1]
missclass_acc = 100 * ((FP + FN) / (TN + FP + FN + TP))
f'The validation set error is {missclass_acc:.2f}%.'

'-----------------------'

f_train, df_test = train_test_split(df, test_size=0.5, random_state=43)
df_X_train = df_train[descriptiveColumns]
df_Y_train = df_train[targetColumn]
df_X_test = df_test[descriptiveColumns]
df_Y_test = df_test[targetColumn]

model = sm.Logit(df_Y_train, df_X_train)
fitted = model.fit()

sr_Y_pred = fitted.predict(df_X_test)
df_Y_test_and_pred = pd.DataFrame({
    'Observed': df_Y_test['default01'],
    'Predicted': np.where(sr_Y_pred > 0.5, 1, 0),
})
df_confusion, df_confusion_pct = createConfusionMatrixFromOutOfSampleData(df=df_Y_test_and_pred, binaryMap={0: 'less than median', 1: 'greater than median'})
confusion_matrix = df_confusion.to_numpy()
TN, FP, FN, TP = confusion_matrix[0, 0], confusion_matrix[0, 1], confusion_matrix[1, 0], confusion_matrix[1, 1]
missclass_acc = 100 * ((FP + FN) / (TN + FP + FN + TP))
f'The validation set error is {missclass_acc:.2f}%.'

'-----------------------'

f_train, df_test = train_test_split(df, test_size=0.5, random_state=44)
df_X_train = df_train[descriptiveColumns]
df_Y_train = df_train[targetColumn]
df_X_test = df_test[descriptiveColumns]
df_Y_test = df_test[targetColumn]

model = sm.Logit(df_Y_train, df_X_train)
fitted = model.fit()

sr_Y_pred = fitted.predict(df_X_test)
df_Y_test_and_pred = pd.DataFrame({
    'Observed': df_Y_test['default01'],
    'Predicted': np.where(sr_Y_pred > 0.5, 1, 0),
})
df_confusion, df_confusion_pct = createConfusionMatrixFromOutOfSampleData(df=df_Y_test_and_pred, binaryMap={0: 'less than median', 1: 'greater than median'})
confusion_matrix = df_confusion.to_numpy()
TN, FP, FN, TP = confusion_matrix[0, 0], confusion_matrix[0, 1], confusion_matrix[1, 0], confusion_matrix[1, 1]
missclass_acc = 100 * ((FP + FN) / (TN + FP + FN + TP))
f'The validation set error is {missclass_acc:.2f}%.'<jupyter_output>Optimization terminated successfully.
         Current function value: 0.077900
         Iterations 10
{'fraction_non_alphanumeric': 0.1157752808988764, 'fraction_numerical': 0.022651685393258427, 'mean_word_length': 5.046739130434783, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '447911', 'license': 'MIT', 'n_tokens_mistral': 3893, 'n_tokens_neox': 3593, 'n_words': 1030, 'path': 'Chapter05/Exercise05/5_5.ipynb', 'repo_name': 'HerrBroekman/Applied-solution-of-An-Introduction-to-Statistical-Learning-written-in-Python', 'size': '11125'}