<jupyter_start><jupyter_text>**Credit Default** - Simple Models Source:  [https://github.com/d-insight/code-bank.git](https://github.com/d-insight/code-bank.git)  License: [MIT License](https://opensource.org/licenses/MIT). See open source [license](LICENSE) in the Code Bank repository.  -------------  Overview In this demo, we try to predict the probability of default on credit card bills using a dataset of customers payments from a Taiwanese bank. A credit default happens when you fail to pay the minimum payment by the due date mentioned on your credit card bill for more than 6 months. From a risk management perspective, the accuracy of the predicted probability of default is more valuable than just a binary prediction (classification) of default itself. Image: https://greendayonline.com/wp-content/uploads/2017/03/Recovering-From-Student-Loan-Default.jpg  The Credit Card Default Dataset  We will try to predict the probability of defaulting on a credit card account at a Taiwanese bank. A credit card default happens when a customer fails to pay the minimum due on a credit card bill for more than 6 months. We will use a dataset from a Taiwanese bank with 30,000 observations (Source: *<NAME>., & <NAME>. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2), 2473-2480.*). Each observation represents an account at the bank at the end of October 2005.  We renamed the variable default_payment_next_month to customer_default. The target variable to predict is `customer_default` -- i.e., whether the customer will default in the following month (1 = Yes or 0 = No). The dataset also includes 23 other explanatory features. Variables are defined as follows: | Feature name     | Variable Type | Description |------------------|---------------|--------------------------------------------------------| customer_default | Binary        | 1 = default in following month; 0 = no default | LIMIT_BAL        | Continuous    | Credit limit   | SEX              | Categorical   | 1 = male; 2 = female| EDUCATION        | Categorical   | 1 = graduate school; 2 = university; 3 = high school; 4 = others| MARRIAGE         | Categorical   | 0 = unknown; 1 = married; 2 = single; 3 = others| AGE              | Continuous    | Age in years  | PAY1             | Categorical   | Repayment status in September, 2005 | PAY2             | Categorical   | Repayment status in August, 2005 | PAY3             | Categorical   | Repayment status in July, 2005 | PAY4             | Categorical   | Repayment status in June, 2005 | PAY5             | Categorical   | Repayment status in May, 2005 | PAY6             | Categorical   | Repayment status in April, 2005 | BILL_AMT1        | Continuous    | Balance in September, 2005  | BILL_AMT2        | Continuous    | Balance in August, 2005  | BILL_AMT3        | Continuous    | Balance in July, 2005  | BILL_AMT4        | Continuous    | Balance in June, 2005 | BILL_AMT5        | Continuous    | Balance in May, 2005  | BILL_AMT6        | Continuous    | Balance in April, 2005  | PAY_AMT1         | Continuous    | Amount paid in September, 2005| PAY_AMT2         | Continuous    | Amount paid in August, 2005| PAY_AMT3         | Continuous    | Amount paid in July, 2005| PAY_AMT4         | Continuous    | Amount paid in June, 2005| PAY_AMT5         | Continuous    | Amount paid in May, 2005| PAY_AMT6         | Continuous    | Amount paid in April, 2005 The measurement scale for repayment status is:       -2 = payment two months in advance       -1 = payment one month in advance       0 = pay duly       1 = payment delay for one month       2 = payment delay for two months       3 = payment delay for three months       4 = payment delay for four months       5 = payment delay for five months       6 = payment delay for six months       7 = payment delay for seven months       8 = payment delay for eight months       9 = payment delay for nine months or more   -------  **Part 0**: SetupPut all import statements, constants and helper functions at the top of your notebook.<jupyter_code># Standard imports
import numpy  as np
import pandas as pd
import pandas_profiling
import itertools

# Visualization packages 
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="white")

from sklearn.metrics         import confusion_matrix, roc_auc_score
from sklearn.linear_model    import LogisticRegression
from sklearn.dummy           import DummyClassifier
from sklearn.model_selection import train_test_split

# Special code to ignore un-important warnings 
import warnings
warnings.filterwarnings('ignore')

%matplotlib inline  
# Constants 
FIGSIZE = (10, 6)
# Define a helper function to visualize the confusion matrix
# Note optional parameter for normalization; apply normalization by setting `normalize=True`

def plot_confusion_matrix(cm, classes=[0,1], normalize=False, title='Confusion Matrix', cmap=plt.cm.Reds):
    """ 
    Function to plot a sklearn confusion matrix, showing number of cases per prediction condition 
    
    Args:
        cm         an sklearn confusion matrix
        classes    levels of the class being predicted; default to binary outcome
        normalize  apply normalization by setting `normalize=True`
        title      title for the plot
        cmap       color map
    """
    
    plt.figure(figsize=FIGSIZE)
    plt.imshow(cm, aspect='auto', interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    plt.locator_params(nbins=2)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
    
    thresh = cm.max() / 2.
    
    # add FP, TP, FN, TN counts
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, round (cm[i, j],2), horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black", fontsize = 12)
        
    plt.tight_layout()
    plt.ylabel('Actual label')
    plt.xlabel('Predicted label')<jupyter_output><empty_output><jupyter_text>**Part 1**: Data Preprocessing and EDAFirst, we would like to understand the main characteristics of the dataset. We might need to transform and clean some features before we can specify a statistical model.<jupyter_code># Load data into a dataframe
data = pd.read_csv('credit_data.csv')
data.head()
# Print data columns
data.columns
# Dimension of data in the form of (number of observations, number of features)
data.shape
# Check distribution of an arbitrary column
data['PAY_1'].hist(bins=40)
# Another way to check distribution of value
data['MARRIAGE'].value_counts()
# Get ovrall statistics about data frame
data.describe()
# Investigate 'EDUCATION' field 
# Notice the mismatch between data description and actual data !
data['EDUCATION'].value_counts()
# Cross-tab 'MARRIAGE' and the target 
pd.crosstab(data['MARRIAGE'], data['customer_default'], margins = True)
# Check data types
data.dtypes
# Check distribution of target feature
data['customer_default'].hist()
# Divide dataframe into two data frames, with and without credit default
data_def_1 = data[data['customer_default'] == 1]
data_def_0 = data[data['customer_default'] == 0]

# Check distribution of different features in with and without default dataframes as well as full dataframe
plt.figure(figsize=FIGSIZE)
plt.style.use('seaborn-deep')
col = 'LIMIT_BAL'
plt.xlabel(col)
plt.ylabel('count')
plt.hist([data_def_0[col], data_def_1[col], data[col]], label=['non_default-'+col, 'default-'+col, 'all-'+col])
plt.legend(loc='upper right')
plt.show()
# Check distribution of different features using box plot. How are the box and whisker positions defined? Click Shift + Tab
plt.figure(figsize=FIGSIZE)
data.boxplot(column='LIMIT_BAL')
plt.ylabel('Feature value')
plt.show()
# Visualize corelation of different features with respect to each other

# Compute the correlation matrix
corr = data.corr()

# Generate a mask for the upper triangle
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# Setup the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(220, 10, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0, square=True, linewidths=.2)
# Visualize distribution of one feature with respect to another one
sns.jointplot(x = "LIMIT_BAL", y = "customer_default", data = data)
# Visualize pair-wise distribution of a set of features
sample_data = data.iloc[:,1:5]
sns.pairplot(sample_data)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis (the automated approach)Instead of performing all of the steps above manually, you can also run a "profile" on the dataset first, and then drill down into specific cases of interest.<jupyter_code># Use the automated pandas profiling utility to examine the dataset
# data.profile_report()<jupyter_output><empty_output><jupyter_text>**Part 2**: Data Preprocessing and Designing Cross-validation Schema Among different features, SEX and MARRIAGE are categoical while others are either numerical or ordinal with acceptable labeling.<jupyter_code># One hot encoding of sex and marital status
# Applying one-hot encoding for other field doesn't make sense since they are either numeric or ordinal
cols_to_transform = ['SEX', 'MARRIAGE']
data_with_dummies = pd.get_dummies(data=data, columns = cols_to_transform)
data_with_dummies.head()
# Separate features and target
cols_X = [ 'LIMIT_BAL', 'EDUCATION', 'AGE', 'PAY_1', 'PAY_2', 'PAY_3',
           'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3',
           'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2',
           'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'SEX_1', 'SEX_2', 'MARRIAGE_0',
           'MARRIAGE_1', 'MARRIAGE_2', 'MARRIAGE_3']
col_y  =   'customer_default'

X = data_with_dummies.loc[:, cols_X]
y = data_with_dummies[col_y]
# Divide data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1, stratify=y)
print(X_train.shape , X_test.shape , y_train.shape , y_test.shape)<jupyter_output>(24000, 27) (6000, 27) (24000,) (6000,)<jupyter_text>**Part 3**: Classification using Dummy Classifier and Logistic Regression  Model 1: BaselineA good practice is to start with a "dumb" model that simply predicts the average (for a regression) or the most frequent outcome (for classification). That gives you sense of how well a complex model performs compared to the most simple model. As this is a classification problem, we will start by predicting No Default.<jupyter_code># Define a baseline using dummy classifier that predicts the most frequent label
dummy_clf = DummyClassifier(strategy = 'most_frequent')
dummy_clf.fit(X_train, y_train)

# Predict default - as a binary outcome and a probability between 0 and 1
y_pred_dummy       = dummy_clf.predict(X_test)
y_pred_dummy_proba = dummy_clf.predict_proba(X_test)

# Plot confusion matrix
cm_dummy = confusion_matrix(y_test, y_pred_dummy)
plot_confusion_matrix(cm_dummy)
# How many errors did the dumb model make?
errors_dummy  = cm_dummy[0][1] + cm_dummy[1][0]
correct_dummy = cm_dummy[0][0] + cm_dummy[1][1]

print('Num Errors =', errors_dummy, '\n')
print('Accuracy   =', "{0:.4f}".format(float(correct_dummy)/len(y_test)))
print('AUC Score  =', "{0:.4f}".format(roc_auc_score(y_test, y_pred_dummy_proba[:, 1])))<jupyter_output>Num Errors = 1327 

Accuracy   = 0.7788
AUC Score  = 0.5000<jupyter_text>Model 2: Logit model (no regularization)<jupyter_code># Binary classification using a logit model
lr_clf = LogisticRegression(C = 10e9)
lr_clf.fit(X_train, y_train)

# Predict default - as a binary outcome and a probability between 0 and 1
y_pred_lr       = lr_clf.predict(X_test)
y_pred_lr_proba = lr_clf.predict_proba(X_test)

# Plot confusion matrix
cm_lr = confusion_matrix(y_test, y_pred_lr)
plot_confusion_matrix(cm_lr,)
# How many errors did the dumb model make?
errors_lr = cm_lr[0][1] + cm_lr[1][0]
correct_lr = cm_dummy[0][0] + cm_dummy[1][1]

print('Num Errors =', errors_lr, '\n')
print('Accuracy   =', "{0:.4f}".format(float(correct_lr)/len(y_test)))
print('AUC Score  =', "{0:.4f}".format(roc_auc_score(y_test, y_pred_lr_proba[:, 1])))<jupyter_output>Num Errors = 1328 

Accuracy   = 0.7788
AUC Score  = 0.6491<jupyter_text>**Summary of performance metrics**  Accuracy<jupyter_code>width = 30
models = ['Dummy', 'Logit']
results = [float((sum(cm_dummy)[0] - errors_dummy)/sum(cm_dummy)[0]), float((sum(cm_lr)[0] - errors_lr)/sum(cm_lr)[0])]
print('', '=' * width, '\n', 'Summary of Accuracy Scores'.center(width), '\n', '=' * width)  
for i in range(len(models)):
    print(models[i].center(width-8), '{0:.4f}'.format(results[i]))<jupyter_output>============================== 
   Summary of Accuracy Scores   
 ==============================
        Dummy          0.7788
        Logit          0.7786<jupyter_text>AUC<jupyter_code>width = 25
models = ['Dummy', 'Logit']
results = [roc_auc_score(y_test, y_pred_dummy_proba[:, 1]), roc_auc_score(y_test, y_pred_lr_proba[:, 1])]
print('', '=' * width, '\n', 'Summary of AUC Scores'.center(width), '\n', '=' * width)  
for i in range(len(models)):
    print(models[i].center(width-8), '{0:.4f}'.format(results[i]))<jupyter_output>========================= 
   Summary of AUC Scores   
 =========================
      Dummy       0.5000
      Logit       0.6491
{'fraction_non_alphanumeric': 0.11490522899918873, 'fraction_numerical': 0.0248543402905819, 'mean_word_length': 4.012939001848429, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 31, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '292126', 'license': 'MIT', 'n_tokens_mistral': 4451, 'n_tokens_neox': 4108, 'n_words': 1401, 'path': 'demos/credit-default-simple/credit-default-simple.ipynb', 'repo_name': 'dsfm-org/code-bank', 'size': '13566'}