<jupyter_start><jupyter_text>Default of Credit Card Clients Project - Group 4 *Authors: <NAME>, <NAME>, <NAME>, <NAME>*  0. AbstractFinancial institutions incur monetary loss when a client or borrower is unable to pay their interest or their initial principal on time. Thus, it is necessary for such institutions to assess the risk that potential borrowers cannot repay their loan in determining their eligibility for the loan in the first place. The present study endeavors to answer the question "Is there a way to effectively predict whether or not a client will default on their credit card payment?" and uncover the most significant features that contribute to the higher likelihood of defaulting. The result of predictive accuracy of the projected likelihood of default will be more beneficial than the binary result of categorization - credible or not credible customers - from the standpoint of risk management.  I. IntroductionAmidst growing financial insecurity during the pandemic, unsecured debt has continued to rise (Frech et al., 2021). Consequently, the consumer credit market and risk prediction has been a matter of great speculation and fear, lest there be a repeat of the financial crises that rocked the economic world in the late 2000s: In 2006, Taiwan was rocked by a credit card debt crisis with debt from credit cards and cash cards reaching $268 billion USD and over half a million people unable to repay their loans (Yeh & Lien, 2009). As many could barely afford to pay the minimum credit card debt balance every month or continued to default on their payments, significant societal problems consequently plagued the country, many banks incurred heavy losses and the government eventually needed to step in to stabilize the financial system (Yeh & Lien, 2009). This situation arose because many banks in Taiwan had lowered the requirements for credit card approval in order to gain more customers within the increasingly competitive industry (Tsai, 2010). Such examples indicate that a strict assessment of an applicant’s capability to make their card payment is critical to a well-developed financial system and a business’s survivability in the banking industry. This project focuses on the case of customers default payments in Taiwan and finds the predictive accuracy of the probability of the customers to default. The purpose of this study is to assess the true probability of default because the real probability of default is unknown.This project used the data from UCI Machine learning repository. As the response variable, this project used data from a binary variable, default payment (Yes = 1, No = 0). The following 23 factors were considered as explanatory variables in this study, which was based on a review of the literature:* Amount of the given credit (NT dollar): it includes both the individual consumer credit and his/her family (supplementary) credit.* Gender (1 = male; 2 = female).* Education (1 = graduate school; 2 = university; 3 = high school; 4 = others).* Marital status (1 = married; 2 = single; 3 = others).* Age (year).* History of past payment. The measurement scale for the repayment status is: -1 = pay duly; 1 = payment delay for one month; 2 = payment delay for two months . . . 8 = payment delay for eight months; 9 = payment delay for nine months and above.* Amount of bill statement (NT dollar). * Amount of previous payment (NT dollar).The objective of this project is to maintain ease of interpretation for the average reader. In line with this goal, we will simplify the models and methods of analysis we choose to use as well as exclude some features in the data set in favor of greater readability.  II. Environment setup  II-1. Import libraries<jupyter_code>import os

%matplotlib inline
import sys
sys.path.append('..')
from group4package.function_count_plot import count_plot
from group4package import preprocess as pp
from group4package import summary_stats_function as ss
from group4package import metrics_function as cm

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import xgboost as xgb
from sklearn.compose import ColumnTransformer, make_column_transformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    make_scorer,
    plot_confusion_matrix,
)
from sklearn.model_selection import (
    GridSearchCV,
    RandomizedSearchCV,
    cross_val_score,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler
from sklearn.svm import SVC
import random
import plotly.express as px
import plotly.figure_factory as ff<jupyter_output><empty_output><jupyter_text>II-2. Data import and preparation<jupyter_code># load the dataset
df = pd.read_excel("https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls", skiprows=1)
# rename column to get rid of spaces
df.rename(columns = {"default payment next month":"default_payment"}, inplace = True)
train_df, test_df = pp.preprocess(df, 0.8, 200)

# split into training and testing set for x and y
X_train, y_train = (train_df.drop(columns=["default_payment"]),
                   train_df["default_payment"])

X_test, y_test = (test_df.drop(columns=["default_payment"]),
                  test_df["default_payment"])<jupyter_output><empty_output><jupyter_text>The dataset was split into a 80% training and 20% testing set. The model will be built using only the training data. This gives the abilithy to compute a final performance metric for our model by evaluating it on the testing data. train_test_split() function shuffles the data to ensure the data ending up in the training and test sets is randomized.   III. Exploratory Data Analysis  III-1. Dataset overview First we have a look at our dataset:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Then we check the number of rows and columns:<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>Hence, there are 25 variables and 30,000 entries in this dataset.  III-2. Check missing values Now let's check if the dataset has null/missing values that may affect further analysis:<jupyter_code>df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Great! The dataset is super clean, and no missing value is found.  III-3. Statistics information This part we have a look at some basic statistics of the training set:<jupyter_code>summary = ss.get_summary_stats(train_df)
summary<jupyter_output><empty_output><jupyter_text>We can see the following information:1. The average amount of given credit in NT dollars is 166849.320000;2. The average age of all clients is 35.419625, etc. To better understand the correlation between variables, we would like to compute and visualize the correlations:<jupyter_code>plt.figure(figsize=(25,25))
sns.heatmap(train_df.corr(),cbar=True,annot=True,cmap='Reds')<jupyter_output><empty_output><jupyter_text>The heatmap shows some positive/negative correlations:Positive correlations:1. Default payment - PAY_0 to PAY_6 (Repayment status from April to September, 2005);2. Limit balance - BILL_AMT1 to BILL_AMT6 (Amount of bill statement from April to September, 2005), etc.Negative correlations:1. Limit balance - PAY_0 to PAY_6 (Repayment status from April to September, 2005), etc.Specifically, PAY_0 has the highest correlation with default_payment. This will give us a signal that PAY_0 plays an important role for predicting default_payment.  III-4. Exploring variables  1. LIMIT_BAL First, we look at the amount of given credit (in NT dollars). Credit card limits are likely an indictor of how wealthy someone is since banks tend to give higher limits to clients that have more money with them. Thus, this may be an important feature when predicting if someone is able to pay the bill on time.<jupyter_code>plt.figure(figsize = (16,9))
sns.distplot(train_df['LIMIT_BAL'],kde=True,bins=50, rug = True, color="green")<jupyter_output>/opt/miniconda3/lib/python3.9/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)
/opt/miniconda3/lib/python3.9/site-packages/seaborn/distributions.py:2103: FutureWarning: The `axis` variable is no longer used and will be removed. Instead, assign variables directly to `x` or `y`.
  warnings.warn(msg, FutureWarning)<jupyter_text>2. Repayment Status<jupyter_code>for column in train_df[["PAY_0", "PAY_2", "PAY_3", "PAY_4", "PAY_5", "PAY_6"]]:
    edu_plot = count_plot(df=train_df, x=column, name ="Repayment status:" + column)
    plt.show()<jupyter_output><empty_output><jupyter_text>Looking at the above plots on repayment status shows that if a client defaults on their payment for 2 months (e.g PAY_X = 2), it is a indicator to predict that default_payment = 1. It looks like repayment status will be an important feature in the model.  IV. Analysis  IV-1. Class Imbalance<jupyter_code>plt.figure(figsize=(6,4))

default_plot = sns.countplot(x = "default_payment",data = train_df)
plt.ylim(0,25000)
total = float(len(train_df))
for p in default_plot.patches:
    default_plot.annotate((str(100* p.get_height()/total) + "%"), (p.get_x()+0.32, p.get_height()+1000), ha="center")


plt.show()<jupyter_output><empty_output><jupyter_text>The above plot shows the percentage of rows with default payment = 0 versus default payment = 1. Cleary there is class imbalance in this dataset. Because of this, we will use the area under the Receiver Operating Characterisitc curve (ROC AUC) as our primary metric to evaluate our model instead of accuracy, which tends to be missleading in cases of class imbalance. ROC AUC evalauates how good the model is as distinguishing between classes, and gives a more accurate sense of how well our model generalizes when dealing with class imbalance.  IV-2. Preprossesing  We apply scaling to numeric features to ensure the model built will be robust and not sensitive to the scale of each individual feature. To do this, we use the StandardScaler() function to set the sample mean to 0 and standard deviation to 1.To handle categorical values, we will apply one-hot-encoding. This creates binary dummy variables for each category. There are no null values in the dataset so there is no need to handle missing values.<jupyter_code>train_df.info()
numeric_feats = [     # apply scaling
    "LIMIT_BAL",
    "BILL_AMT1",
    "BILL_AMT2",
    "BILL_AMT3",
    "BILL_AMT4", 
    "BILL_AMT5",
    "BILL_AMT6",
    "PAY_AMT1",
    "PAY_AMT2",
    "PAY_AMT3",
    "PAY_AMT4",
    "PAY_AMT5",
    "PAY_AMT6",
    "AGE",
]

categorical_feats = [  # apply one-hot encoding
    "PAY_0",
    "PAY_2",
    "PAY_3",
    "PAY_4",
    "PAY_5",
    "PAY_6",
    "EDUCATION",
    "SEX",
    "MARRIAGE",
]<jupyter_output><empty_output><jupyter_text>Next, we apply scaling and one-hot-encoding through using a column transformer, which applies the transformations to each column specified.<jupyter_code>ct = make_column_transformer(
    (StandardScaler(), numeric_feats,),
    (OneHotEncoder(handle_unknown="ignore"), categorical_feats,),

)<jupyter_output><empty_output><jupyter_text>IV-3. Model 0: Dummy Classifier Firstly, we will try a baseline model to act as a comparison measure for the final model built. Using a pipeline to do this ensures that the model is built using just the training data, and that the testing data has no influence on the model. 5-fold Cross validation is used to give a more robust measure of performance error. K-fold Cross-validation spilts the training data into k folds, and each time one fold is the validation set. Each fold fits the model on the training portion and uses one fold as a validation set to calculate a performance metric, which can be averaged to get a overall score of how well the model does. This ensures that outliers don't negatively influence the performance metric.<jupyter_code>from sklearn.model_selection import cross_val_predict

dummy = DummyClassifier()

pipe = make_pipeline(ct, dummy)

scores = cross_validate(pipe, X_train, y_train, return_train_score=True, cv=5, scoring="roc_auc")
scores = pd.DataFrame(scores)
scores<jupyter_output><empty_output><jupyter_text>IV-4. Model 1: Logistic Regression Next, a logistic regression model is fitted to the training data. Logistic regression uses the training data to learn coefficients, which then can be used to calculate prediction probabilities of the each class using the sigmoid function. This allows us to calculate the probability of a client defaulting on their credit card based on their data. The hyperparameter C is used to control the fundatamental tradeoff of bias and variance, to reduce the likeliness of the model overfitting or underfitting. We chose logistic regression over more complex classifiers such as ensemble trees because it is easier to interpret and efficient to train. The predicted coeficients give information about feature importance and direction of association, making the model easily interpreted. Moreover, since the dataset is significantly larger than the number of features, logistic regression is less likely to overfit because it is a low variance model. Hyperparameter optimization is carried out to find the best value of C for the data in hopes to reduce bias and variance. Cross-validation is used to test how well the model performs on unseen data during hyperparemter opimization, enabling performance metrics to be calculated to compare different values of C.<jupyter_code>random.seed()

train_scores = []
cv_scores = []

C = 10.0 ** np.arange(-1.5, 2, 0.5)

for c in C:
    
    pipe_lr = make_pipeline(
    ct, 
    LogisticRegression(max_iter=1000, C=c),
    )
    
    results = cross_validate(pipe_lr, X_train, y_train, return_train_score=True, scoring = "roc_auc")
    
    train_scores.append(results["train_score"].mean())
    cv_scores.append(results["test_score"].mean())
    

scores = pd.DataFrame({"C": C, "Train Scores": train_scores, "CV Scores": cv_scores })

scores
best_C = scores.loc[scores["CV Scores"].idxmax(), "C"]
print("best C value: " + str(best_C))<jupyter_output>best C value: 0.03162277660168379<jupyter_text>Next, the logistic regression model with the optimized C value is fitted to the training data. The model is built using pipelines to ensure all data preprocessing is constant and that no information from the testing set leaks into the training of the model.<jupyter_code>random.seed()

model = make_pipeline(
    ct, 
    LogisticRegression(max_iter=1000, C=best_C),
    )

model.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>IV-5. Feature Importance Next, we will look at feature importance. In logistic regression, the magnitude and direction of the learned coefficients explain the relationship between a explanatory feature and the response variable.<jupyter_code>ohe_feature_names = (
    model.named_steps["columntransformer"]
    .named_transformers_["onehotencoder"]
    .get_feature_names()
    .tolist()
)


feat_names = (numeric_feats + ohe_feature_names)


coefs = {
    "coefficient": model.named_steps["logisticregression"].coef_[0].tolist(),
    "absolute_value": np.absolute(
        model.named_steps["logisticregression"].coef_[0].tolist()
    ),
}
df_coefs = pd.DataFrame(coefs, index=feat_names).sort_values(
    "absolute_value", ascending=False
)

df_coefs[:10]<jupyter_output><empty_output><jupyter_text>The table above shows the most important features according to the model. Positive coefficients indicate that an increase in the feature increases the probability that the response variable is class 1, and negative coefficients indicate that a increases in the feature decreases the probability that the response variable is class 1.The most important feature is repayment status in April with a 2 month delay. x0_2 $= \begin{cases}    1     &\text{if repayment status in April was delayed 2 months} \\    0     &\text{else}    \end{cases}$ To further interpret this, the odds ratio can be calculated. $OR = e^{\beta}$,  where $\beta$ is a model coefficient.<jupyter_code>x0_2_OR = np.exp(df_coefs.iat[0,0])
x0_2_OR<jupyter_output><empty_output><jupyter_text>The odds ratio for x0_2 can be interpreted as clients who delayed payment for 2 months in April have about 2.9 times the odds of defaulting their credit card payment next month than those who did not delay for 2 months of, controlling for the other features since there is some correlation. It makes logical sense that this feature is important since it was highly correlated with the response variable during the exploratory data analysis, and because it makes sense that if someone has delayed payment in the past they may not be able to pay in the future as well. Another important feature is x0_0, corresponding to repayment status in April with a revolving credit. The magnitude is negative here meaning that a value of 1 for this binary variable is negatively associated with defaulting next month. This makes sense because revolving credit lets clients pay a minimum balance instead of the full bill, making them less likely to default. The odds ratio is calculated below.<jupyter_code>x0_0_OR = np.exp(df_coefs.iat[1,0])
x0_0_OR<jupyter_output><empty_output><jupyter_text>The odds ratio above shows that clients who had a repayment status of using a revolving credit in April decreases the odds of class 1 versus class 0 by about 63%, controlling for the other features. Due to the large number of features, we only looked at the first 2 most important features, however this method can be done for the remaining features as well.  IV-6. Testing the Model Finally, the model will be evaluated by predicting on the test dataset. Performance metrics including area under the ROC curve, f1-score, precision and recall will be calculated. Precision is the ratio of true postive and total positives. Recall is the measure of the model correctly identifying true positives. F1-score is the harmonic mean of precision and recall.<jupyter_code>from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score

# prediction probabilities on test set 
lr_prob = model.predict_proba(X_test)[:,1]

# roc_auc score on test set 
roc_lr = roc_auc_score(y_test, lr_prob)

fpr, tpr, thresholds = roc_curve(y_test, lr_prob)

roc_lr = round(roc_lr, 4)

auc_label = "AUC is " + str(roc_lr)

plt.plot(fpr, tpr, label="ROC Curve")
plt.xlabel("FPR")
plt.ylabel("TPR (recall)")
plt.fill_between(fpr, tpr, color='blue', alpha=0.2, label=auc_label)

plt.legend(loc="best");<jupyter_output><empty_output><jupyter_text>The test ROC AUC is almost the same as the cross-validation ROC AUC so we can conclude that there is little optimization bias and we are not overfitting. Area under the ROC curve is the classifier's ability to distinguish between classes. So 78% of the time, the model is able to correctly distinguish between class 0 and class 1.<jupyter_code>predict = model.predict(X_test)

TN, FP, FN, TP = confusion_matrix(y_test, predict).ravel()

FP = int(FP)
FN = int(FN)
TP = int(TP)


res = cm.calculate_metrics(FP, FN, TP)

# res = classification_report(
#         y_test, predict, target_names=["non-default", "default"]
#     )

res
plot_confusion_matrix(
    model,
    X_test,
    y_test,
    display_labels=["Non default", "default"],
    values_format="d",
    cmap=plt.cm.Blues,
);<jupyter_output>/opt/miniconda3/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function plot_confusion_matrix is deprecated; Function `plot_confusion_matrix` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: ConfusionMatrixDisplay.from_predictions or ConfusionMatrixDisplay.from_estimator.
  warnings.warn(msg, category=FutureWarning)
{'fraction_non_alphanumeric': 0.06827964969571025, 'fraction_numerical': 0.015239226163970115, 'mean_word_length': 5.0733173076923075, 'pattern_counts': {'":': 6, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '210977', 'license': 'MIT', 'n_tokens_mistral': 5875, 'n_tokens_neox': 5523, 'n_words': 2477, 'path': 'reports/data_analysis.ipynb', 'repo_name': 'DSCI-310/DSCI-310-Group-4', 'size': '20236'}