<jupyter_start><jupyter_text>The goal of this Jupyter notebook is to introduce Machine Learning techniques through a concrete case study of credit scoring.To do this, we will use the [UCI default of credit card clients Dataset](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients) to build a machine learning model that predict credit card default. The dataset contains 30.000 records (6.636 defaults) of credit card users in Taiwain. Each record contains 23 attributes describing the credit card user and his/her payment history.  1. Import the libraries that we will use in this tutorial<jupyter_code>%matplotlib inline
import numpy as np
import pandas as pd

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score, roc_curve
from sklearn import tree

import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>2. Read Data<jupyter_code>data = pd.read_excel("../data/default_of_credit_card_clients.xls", sheet_name=0, skiprows=1, index_col="ID")
data.head()<jupyter_output><empty_output><jupyter_text>* **ID**: ID of each client* **LIMIT_BAL**: Amount of given credit in NT dollars (includes individual and family/supplementary credit* **SEX**: Gender (1=male, 2=female)* **EDUCATION**: (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown)* **MARRIAGE**: Marital status (1=married, 2=single, 3=others)* **AGE**: Age in years* **PAY_0**: Repayment status in September, 2005 (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, ... 8=payment delay for eight months, 9=payment delay for nine months and above)* **PAY_2**: Repayment status in August, 2005 (scale same as above)* **PAY_3**: Repayment status in July, 2005 (scale same as above)* **PAY_4**: Repayment status in June, 2005 (scale same as above)* **PAY_5**: Repayment status in May, 2005 (scale same as above)* **PAY_6**: Repayment status in April, 2005 (scale same as above)* **BILL_AMT1**: Amount of bill statement in September, 2005 (NT dollar)* **BILL_AMT2**: Amount of bill statement in August, 2005 (NT dollar)* **BILL_AMT3**: Amount of bill statement in July, 2005 (NT dollar)* **BILL_AMT4**: Amount of bill statement in June, 2005 (NT dollar)* **BILL_AMT5**: Amount of bill statement in May, 2005 (NT dollar)* **BILL_AMT6**: Amount of bill statement in April, 2005 (NT dollar)* **PAY_AMT1**: Amount of previous payment in September, 2005 (NT dollar)* **PAY_AMT2**: Amount of previous payment in August, 2005 (NT dollar)* **PAY_AMT3**: Amount of previous payment in July, 2005 (NT dollar)* **PAY_AMT4**: Amount of previous payment in June, 2005 (NT dollar)* **PAY_AMT5**: Amount of previous payment in May, 2005 (NT dollar)* **PAY_AMT6**: Amount of previous payment in April, 2005 (NT dollar)* **default payment next month**: Default payment (1=yes, 0=no)  3. Data Discovery  How many defaults in the data set?<jupyter_code>data["default payment next month"].value_counts()<jupyter_output><empty_output><jupyter_text>Default Ratio<jupyter_code>data["default payment next month"].sum() / len(data["default payment next month"])<jupyter_output><empty_output><jupyter_text>6,636 out of 30,000 (or 22%) of clients will default next month.   Let's check if there are any anomalies in the data<jupyter_code>data.isnull().sum()<jupyter_output><empty_output><jupyter_text>Based on the numbers above, we can see that all the columns have no missing data<jupyter_code># Bill Statement description
data[['BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6']].describe()<jupyter_output><empty_output><jupyter_text>Negative values can be interpreted as credit? Has to be investigated  Visualization<jupyter_code>class_0 = data.loc[data["default payment next month"] == 0]["LIMIT_BAL"]
class_1 = data.loc[data["default payment next month"] == 1]["LIMIT_BAL"]
plt.figure(figsize = (14,5))
plt.title('Default amount of credit limit  - grouped by Payment Next Month')
plt.hist(class_0, bins=200, color="green")
plt.hist(class_1, bins=200, color="red")
plt.show()<jupyter_output><empty_output><jupyter_text>Training / Validation Split 80% for training and 20% for validation<jupyter_code>train_df, val_df = train_test_split(data, test_size=0.20, random_state=2018, shuffle=True )
#Ratio of default in training data
print( train_df["default payment next month"].sum() / len(train_df["default payment next month"]) )

#Ratio of default in validation data
print( val_df["default payment next month"].sum() / len(val_df["default payment next month"]) )<jupyter_output>0.222666666667
0.215333333333<jupyter_text>Let's build our first model - Decision Trees<jupyter_code>target = "default payment next month"

predictors = ['LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 
              'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 
              'BILL_AMT1','BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6',
              'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
clf_dt = DecisionTreeClassifier()<jupyter_output><empty_output><jupyter_text>PUSH THE BUTTON FOR TRAINING<jupyter_code>clf_dt.fit(train_df[predictors], train_df[target].values)<jupyter_output><empty_output><jupyter_text>Predicition on training data to check if the model could model the data<jupyter_code>preds_training = clf_dt.predict(train_df[predictors])
preds_proba_training = clf_dt.predict_proba(train_df[predictors])
roc_auc_score(train_df[target], preds_proba_training[:,1])<jupyter_output><empty_output><jupyter_text>Predicition on validation data<jupyter_code>preds = clf_dt.predict(val_df[predictors])
preds_proba = clf_dt.predict_proba(val_df[predictors])
roc_auc_score(val_df[target], preds_proba[:,1])
fpr, tpr, threshold = roc_curve(val_df[target], preds_proba[:,1])
roc_auc = roc_auc_score(val_df[target], preds_proba[:,1])

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's build a stronger model - Random Forest<jupyter_code>target = "default payment next month"

predictors = ['LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 
              'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 
              'BILL_AMT1','BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6',
              'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
clf_rf1 = RandomForestClassifier(criterion="gini",
                             n_estimators=500,
                             random_state=2018,
                             verbose=False)<jupyter_output><empty_output><jupyter_text>PUSH THE BUTTON FOR TRAINING<jupyter_code>clf_rf1.fit(train_df[predictors], train_df[target].values)<jupyter_output><empty_output><jupyter_text>Predicition on training data to check if the model could model the data<jupyter_code>preds_training = clf_rf1.predict(train_df[predictors])
preds_proba_training = clf_rf1.predict_proba(train_df[predictors])
roc_auc_score(train_df[target], preds_proba_training[:,1])<jupyter_output><empty_output><jupyter_text>Predicition on validation data<jupyter_code>preds = clf_rf1.predict(val_df[predictors])
preds_proba = clf_rf1.predict_proba(val_df[predictors])
roc_auc_score(val_df[target], preds_proba[:,1])
fpr, tpr, threshold = roc_curve(val_df[target], preds_proba[:,1])
roc_auc = roc_auc_score(val_df[target], preds_proba[:,1])

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's improve on the model - Features Engineering<jupyter_code>data['Closeness_6'] = (data["LIMIT_BAL"] - data["BILL_AMT6"]) / data["LIMIT_BAL"]
data['Closeness_5'] = (data["LIMIT_BAL"] - data["BILL_AMT5"]) / data["LIMIT_BAL"]
data['Closeness_4'] = (data["LIMIT_BAL"] - data["BILL_AMT4"]) / data["LIMIT_BAL"]
data['Closeness_3'] = (data["LIMIT_BAL"] - data["BILL_AMT3"]) / data["LIMIT_BAL"]
data['Closeness_2'] = (data["LIMIT_BAL"] - data["BILL_AMT2"]) / data["LIMIT_BAL"]
data['Closeness_1'] = (data["LIMIT_BAL"] - data["BILL_AMT1"]) / data["LIMIT_BAL"]
train_df, val_df = train_test_split(data, test_size=0.20, random_state=2018, shuffle=True )
target = "default payment next month"

predictors = ['LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 
              'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 
              'BILL_AMT1','BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6',
              'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6',
             "Closeness_1", "Closeness_2", "Closeness_3", "Closeness_4", "Closeness_5", "Closeness_6"]
 clf_rf2 = RandomForestClassifier(criterion="gini",
                             n_estimators=500,
                             random_state=2018,
                             verbose=False)
clf_rf2.fit(train_df[predictors], train_df[target].values)<jupyter_output><empty_output><jupyter_text>Predicition on training data to check if the model could model the data<jupyter_code>preds_training = clf_rf2.predict(train_df[predictors])
preds_proba_training = clf_rf2.predict_proba(train_df[predictors])
roc_auc_score(train_df[target], preds_proba_training[:,1])<jupyter_output><empty_output><jupyter_text>Predicition on validation data<jupyter_code>preds = clf_rf2.predict(val_df[predictors])
preds_proba = clf_rf2.predict_proba(val_df[predictors])
roc_auc_score(val_df[target], preds_proba[:,1])
fpr, tpr, threshold = roc_curve(val_df[target], preds_proba[:,1])
roc_auc = roc_auc_score(val_df[target], preds_proba[:,1])

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>Features Importance<jupyter_code>clf_rf2.feature_importances_
tmp = pd.DataFrame({'Feature': predictors, 'Feature importance': clf_rf2.feature_importances_})
tmp = tmp.sort_values(by='Feature importance',ascending=False)
plt.figure(figsize = (7,4))
plt.title('Features importance',fontsize=14)
s = sns.barplot(x='Feature',y='Feature importance',data=tmp)
s.set_xticklabels(s.get_xticklabels(),rotation=90)
plt.show()<jupyter_output>C:\Program Files\Anaconda2\envs\analytics3\lib\site-packages\seaborn\categorical.py:1428: FutureWarning: remove_na is deprecated and is a private function. Do not use.
  stat_data = remove_na(group_data)<jupyter_text>Let's try another model - Gradient Boosting<jupyter_code>target = "default payment next month"

predictors = ['LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 
              'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 
              'BILL_AMT1','BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6',
              'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6',
             "Closeness_1", "Closeness_2", "Closeness_3", "Closeness_4", "Closeness_5", "Closeness_6"]
clf_gb = GradientBoostingClassifier(learning_rate=0.05, max_depth=4, random_state=2018)
clf_gb.fit(train_df[predictors], train_df[target].values)<jupyter_output><empty_output><jupyter_text>Predicition on training data to check if the model could model the data<jupyter_code>preds_training = clf_gb.predict(train_df[predictors])
preds_proba_training = clf_gb.predict_proba(train_df[predictors])
roc_auc_score(train_df[target], preds_proba_training[:,1])<jupyter_output><empty_output><jupyter_text>Predicition on validation data<jupyter_code>preds = clf_gb.predict(val_df[predictors])
preds_proba = clf_gb.predict_proba(val_df[predictors])
roc_auc_score(val_df[target], preds_proba[:,1])
fpr, tpr, threshold = roc_curve(val_df[target], preds_proba[:,1])
roc_auc = roc_auc_score(val_df[target], preds_proba[:,1])

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()
tmp = pd.DataFrame({'Feature': predictors, 'Feature importance': clf_gb.feature_importances_})
tmp = tmp.sort_values(by='Feature importance',ascending=False)
plt.figure(figsize = (7,4))
plt.title('Features importance',fontsize=14)
s = sns.barplot(x='Feature',y='Feature importance',data=tmp)
s.set_xticklabels(s.get_xticklabels(),rotation=90)
plt.show()<jupyter_output>C:\Program Files\Anaconda2\envs\analytics3\lib\site-packages\seaborn\categorical.py:1428: FutureWarning: remove_na is deprecated and is a private function. Do not use.
  stat_data = remove_na(group_data)
{'fraction_non_alphanumeric': 0.1431585292344786, 'fraction_numerical': 0.031042796865581677, 'mean_word_length': 6.63254744105808, 'pattern_counts': {'":': 0, '<': 90, '<?xml version=': 0, '>': 90, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '662863', 'license': 'Apache-2.0', 'n_tokens_mistral': 5183, 'n_tokens_neox': 4894, 'n_words': 1046, 'path': 'notebooks/credit-default-models.ipynb', 'repo_name': 'adilmoujahid/ml-credit-card-default', 'size': '13272'}