<jupyter_start><jupyter_text>Bank Loan Default Prediction *Created by: <NAME>, data scientist  Antwerpen, Belgium  Date: October 28, 2019  Contacts: <EMAIL>*  *** Table of ContentsStage 1 : Business Understanding  Stage 2 : Analytic Approach  Stage 3 : Data Requirements  Stage 4 : Data Collection  Stage 5 : Data Understanding  Stage 6 : Data Preparation  Stage 7 : Modeling  Stage 8 : Evaluation  Sample of use  *** Stage 1 : Business Understanding**Problem:**  The loan is one of the most important products of the banking. All the banks are trying to figure out effective business strategies to persuade customers to apply their loans. However, there are some customers behave negatively after their application are approved. Loans default will cause huge loss for the banks. To prevent this situation, banks have to find some methods to predict customers’ behaviours.   **Question:**  Can we predict a loan default? Stage 2 : Analytic ApproachAs the question requires a yes/no answer, next classification models will be built:- K Nearest Neighbor(KNN)- Decision Tree- Support Vector Machine- Logistic Regression  To evaluate models performance, we use:+ Jaccard score, + F1 score,+ Logistic regression scores (where applicable)To evaluate models performance, we use the out of sample set.  Stage 3 : Data Requirements**Data content:** To answer the question we need data about past loans, having the next information:  data about loan (loan status, loan amount, terms, effective date, due date) and data about customer (age, gender, education level).  **Data formats:** CSV format  **Data sources:** corporative information from the bank. Stage 4 :  Data CollectionImporting libraries:<jupyter_code>! pip install seaborn
# computation
import numpy as np
import pandas as pd
import itertools

# visualization
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
from matplotlib.ticker import NullFormatter
%matplotlib inline
import seaborn as sns


from sklearn import preprocessing
from sklearn.model_selection import train_test_split

# machine learning
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn import svm
from sklearn.linear_model import LogisticRegression

from sklearn import metrics
from sklearn.metrics import f1_score, jaccard_similarity_score
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.metrics import log_loss<jupyter_output><empty_output><jupyter_text>Collecting the data, [source](https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_train.csv):<jupyter_code>!wget -O loan_train.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_train.csv
df = pd.read_csv('loan_train.csv')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Stage 5 : Data Understanding   The dataset consists of 2 parts:* Train dataset,* Out of sample dataset.This dataset is about past loans.  The train dataset includes details of 346 customers whose loan are already paid off or defaulted. It includes following fields:+ **Loan_status**: Whether a loan is paid off on in collection  + **Principal**: Basic principal loan amount  + **Terms**: Origination terms which can be weekly (7 days), biweekly, and monthly payoff schedule  + **Effective_date**: When the loan got originated and took effects  + **Due_date**: Since it’s one-time payoff schedule, each loan has one single due date  + **Age**: Age of applicant  + **Education**: Education of applicant  + **Gender**: The gender of applicant<jupyter_code># Converting to date time object 

df['due_date'] = pd.to_datetime(df['due_date'])
df['effective_date'] = pd.to_datetime(df['effective_date'])
df.head()<jupyter_output><empty_output><jupyter_text>Data visualization and pre-processing **How many of each class is in our data set?**<jupyter_code>df['loan_status'].value_counts()
# Bar chart: Paid Off vs Collection

height = [260, 86]
bars = ['Paid off', 'Collection']
y_pos = np.arange(len(bars))
bar_width = 0.35

fig, ax = plt.subplots()
plt.bar(y_pos, height, color = ('grey', 'mediumvioletred'), width=0.4)
plt.xticks(y_pos, bars)

ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

plt.title('Paid Off vs Collection')

plt.show()<jupyter_output><empty_output><jupyter_text>**Conclusion**: 260 people have paid off the loan on time while 86 have gone into collection. **Do gender & loan amount have impact on loan status?**<jupyter_code># Bar Chart: Do gender & loan amount have impact on loan status?

bins = np.linspace(df.Principal.min(), df.Principal.max(), 10)
palette ={"PAIDOFF":"grey","COLLECTION":"mediumvioletred"}
gender = sns.FacetGrid(df, col="Gender", hue="loan_status", palette=palette, col_wrap=2)
gender.map(plt.hist, 'Principal', bins=bins, ec="k")
gender.set_axis_labels("Loan amount", "Principal")
gender.axes[-1].legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Conclusion**: Yes, the default loans are tent to be over 700 for male customers, over 900 for female customers.**Do age & loan amount have impact on loan status?**<jupyter_code># Bar Chart: Do age & loan amount have impact on loan status?

palette ={"PAIDOFF":"grey","COLLECTION":"mediumvioletred"}
bins = np.linspace(df.age.min(), df.age.max(), 10)
g = sns.FacetGrid(df, col="Gender", hue="loan_status", palette=palette, col_wrap=2)
g.map(plt.hist, 'age', bins=bins, ec="k")
g.axes[-1].legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Conclusion**: Yes, most of the default loans are tent to be: for female customers between 25 and 40 y.o., for male customers: 25-40. In other way, as histogram of default loans for male customers follow the shape of paid off loans, we think age have less impact. Stage 6 : Data Preparation & Feature engineering  Checking the day of the weekAt what day of the week default loans tend to be taken?<jupyter_code>#At what day of the week default loans tend to be taken?

df['dayofweek'] = df['effective_date'].dt.dayofweek
bins = np.linspace(df.dayofweek.min(), df.dayofweek.max(), 10)
palette ={"PAIDOFF":"grey","COLLECTION":"mediumvioletred"}
gender = sns.FacetGrid(df, col="Gender", hue="loan_status", palette=palette, col_wrap=2)
gender.map(plt.hist, 'dayofweek', bins=bins, ec="k")
gender.axes[-1].legend()
gender.set_axis_labels("Day of week", "dayofweek")
plt.show()<jupyter_output><empty_output><jupyter_text>**Conclusion**: We see that people who get the loan at the end of the week don't pay it off, so we use Feature binarization to set a threshold values less then day 4 :<jupyter_code>df['weekend'] = df['dayofweek'].apply(lambda x: 1 if (x>3)  else 0)
df.head()<jupyter_output><empty_output><jupyter_text>Checking gender<jupyter_code>df.groupby(['Gender'])['loan_status'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>**Conclusion**: 86 % of female pay there loans while only 73 % of males pay there loan.  Converting male to 0 and female to 1:<jupyter_code>df['Gender'].replace(to_replace=['male','female'], value=[0,1],inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Checking education<jupyter_code>df.groupby(['education'])['loan_status'].value_counts(normalize=True)
# Feature before One Hot Encoding
df[['Principal','terms','age','Gender','education']].head()<jupyter_output><empty_output><jupyter_text>Converting categorical varables to binary variables:<jupyter_code>Feature = df[['Principal','terms','age','Gender','weekend']]
Feature = pd.concat([Feature, pd.get_dummies(df['education'])], axis=1)
Feature.drop(['Master or Above'], axis = 1,inplace=True)
Feature.head()<jupyter_output><empty_output><jupyter_text>Stage 7 : Modeling Selecting feature sets, X:<jupyter_code>X = Feature
X[0:2]<jupyter_output><empty_output><jupyter_text>What are the lables?<jupyter_code>y = df['loan_status'].values
y[0:5]<jupyter_output><empty_output><jupyter_text>Normalizing data Data Standardization give data zero mean and unit variance<jupyter_code>X = preprocessing.StandardScaler().fit(X).transform(X)
X[0:2]
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)<jupyter_output>Train set: (276, 8) (276,)
Test set: (70, 8) (70,)<jupyter_text>Modeling**K Nearest Neighbor(KNN)**<jupyter_code>k = 7
knn = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)
knn
y_predict_knn = knn.predict(X_test)
y_predict_knn[0:5]<jupyter_output><empty_output><jupyter_text>Looking for the best k:<jupyter_code>Ks = 15
mean_acc = np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))
ConfustionMx = [];
for n in range(1,Ks):
    
    #Train Model and Predict  
    knn = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)
    y_predict_knn = knn.predict(X_test)
    mean_acc[n-1] = metrics.accuracy_score(y_test, y_predict_knn)

    
    std_acc[n-1]=np.std(y_predict_knn==y_test)/np.sqrt(y_predict_knn.shape[0])

fig, ax = plt.subplots()
plt.plot(range(1,Ks),mean_acc,'purple')
plt.fill_between(range(1,Ks), mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10, color = 'grey')
plt.legend(('Accuracy ', '+/- 3xstd'))
plt.ylabel('Accuracy ')
plt.xlabel('Number of Nabors (K)')
plt.tight_layout()
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
plt.show()

print( "The best accuracy was with", mean_acc.max().round(2), "with k=", mean_acc.argmax()+1)<jupyter_output><empty_output><jupyter_text>**Decision Tree**<jupyter_code>max_depth = 4
criterion = "entropy"
decitionTree = DecisionTreeClassifier(criterion= criterion, max_depth = max_depth)
decitionTree
decitionTree.fit(X_train, y_train)
y_predict_decitionTree = decitionTree.predict(X_test)
y_predict_decitionTree[0:5]<jupyter_output><empty_output><jupyter_text>**Support Vector Machine**<jupyter_code>svm = svm.SVC(kernel='rbf')
svm.fit(X_train, y_train) 
svm
y_predict_svm = svm.predict(X_test)
y_predict_svm[0:5]<jupyter_output><empty_output><jupyter_text>**Logistic Regression**<jupyter_code>C=0.01
solver='liblinear'
logisticRegression = LogisticRegression(C=C, solver=solver).fit(X_train,y_train)
logisticRegression
y_predict_logisticRegression = logisticRegression.predict(X_test)
y_predict_logisticRegression[0:5]
#evaluation
knn_score = metrics.accuracy_score(y_test, y_predict_knn).round(2)
print("KNN Accuracy: ", knn_score)

decisionTree_score = metrics.accuracy_score(y_test, y_predict_decitionTree).round(2)
print("Decision Trees's Accuracy: ", decisionTree_score)

SVM_score = metrics.accuracy_score(y_test, y_predict_svm).round(2)
print("SVM Accuracy: ", SVM_score)

LogisticRegression_score = metrics.accuracy_score(y_test, y_predict_logisticRegression).round(2)
print("LogisticRegression Accuracy: ", LogisticRegression_score)


d_ = {'Accuracy score': [knn_score, decisionTree_score, SVM_score, LogisticRegression_score]}
train_report = pd.DataFrame(data=d_, index=['KNN', 'Decision Tree', 'SVM', 'LogisticRegression'])

#visualization
color = [('grey', 'grey', 'mediumvioletred', 'grey')]
train_report.plot(kind='bar', color = color, legend=None)
plt.title('Model Evaluation: train set')
plt.show()<jupyter_output>KNN Accuracy:  0.7
Decision Trees's Accuracy:  0.61
SVM Accuracy:  0.74
LogisticRegression Accuracy:  0.69<jupyter_text>Stage 8 : Model Evaluation using Out of sample set Downloading the set:<jupyter_code>!wget -O loan_test.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_test.csv<jupyter_output>--2019-10-28 14:27:45--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_test.csv
Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 192.168.3.11
Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3642 (3.6K) [text/csv]
Saving to: ‘loan_test.csv’

loan_test.csv       100%[===================>]   3.56K  --.-KB/s    in 0s      

2019-10-28 14:27:45 (90.7 MB/s) - ‘loan_test.csv’ saved [3642/3642]<jupyter_text>Making same data preparation as with the train set:<jupyter_code>outOfSample = pd.read_csv('loan_test.csv')

#coreecting data 

outOfSample['due_date'] = pd.to_datetime(outOfSample['due_date'])
outOfSample['effective_date'] = pd.to_datetime(outOfSample['effective_date'])
outOfSample['dayofweek'] = outOfSample['effective_date'].dt.dayofweek
outOfSample['weekend'] = outOfSample['dayofweek'].apply(lambda x: 1 if (x>3)  else 0)
outOfSample['Gender'].replace(to_replace=['male','female'], value=[0,1],inplace=True)

Feature_outOfSample = outOfSample[['Principal','terms','age','Gender','weekend']]
Feature_outOfSample = pd.concat([Feature_outOfSample,pd.get_dummies(outOfSample['education'])], axis=1)
Feature_outOfSample.drop(['Master or Above'], axis = 1,inplace=True)
Feature_outOfSample.head()
X_outOfSample = Feature_outOfSample
y_outOfSample = outOfSample['loan_status'].values
X_outOfSample = preprocessing.StandardScaler().fit(X_outOfSample).transform(X_outOfSample)<jupyter_output>/home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/sklearn/preprocessing/data.py:625: DataConversionWarning: Data with input dtype uint8, int64 were all converted to float64 by StandardScaler.
  return self.partial_fit(X, y)
/home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/ipykernel_launcher.py:3: DataConversionWarning: Data with input dtype uint8, int64 were all converted to float64 by StandardScaler.
  This is separate from the ipykernel package so we can avoid doing imports until<jupyter_text>**Running the models:**<jupyter_code>#K Neighbor
y_predict_knn_outOfSample = knn.predict(X_outOfSample)

knn_jaccard_score = jaccard_similarity_score(y_outOfSample, y_predict_knn_outOfSample).round(2)
knn_f1_score = f1_score(y_outOfSample, y_predict_knn_outOfSample, average='weighted').round(2)
print("K Neighbor jaccard_score: ", knn_jaccard_score)
print("K Neighbor f1_score: ", knn_f1_score)
#Decision Tree
y_predict_decitionTree_outOfSample = decitionTree.predict(X_outOfSample)

DecisionTrees_jaccard_score = jaccard_similarity_score(y_outOfSample, y_predict_decitionTree_outOfSample).round(2)
DecisionTrees_f1_score = f1_score(y_outOfSample, y_predict_decitionTree_outOfSample, average='weighted').round(2)
print("DecisionTrees's jaccard_score: ", DecisionTrees_jaccard_score)
print("DecisionTrees's f1_score: ", DecisionTrees_f1_score)
#SVM
y_predict_outOfSample = svm.predict(X_outOfSample)

SVM_jaccard_score = jaccard_similarity_score(y_outOfSample, y_predict_outOfSample).round(2)
SVM_f1_score = f1_score(y_outOfSample, y_predict_outOfSample, average='weighted').round(2)
print("SVM f1_score: ", SVM_jaccard_score)
print("SVM jaccard_score: ", SVM_f1_score)
#LogRes
y_predict_logisticRegression_outOfSample = logisticRegression.predict(X_outOfSample)
y_outOfSample_prob = logisticRegression.predict_proba(X_outOfSample)

LogisticRegression_jaccard_score = jaccard_similarity_score(y_outOfSample, y_predict_logisticRegression_outOfSample).round(3)
LogisticRegression_f1_score = f1_score(y_outOfSample, y_predict_logisticRegression_outOfSample, average='weighted').round(3)
LogisticRegression_LogLoss = log_loss(y_outOfSample, y_outOfSample_prob).round(3)
print("LogisticRegression jaccard_score: ", LogisticRegression_jaccard_score)
print("LogisticRegression f1_score: ", LogisticRegression_f1_score)
print("LogisticRegression LogLoss: ", LogisticRegression_LogLoss)<jupyter_output>LogisticRegression jaccard_score:  0.741
LogisticRegression f1_score:  0.66
LogisticRegression LogLoss:  0.567<jupyter_text>Report:<jupyter_code>d = {
    'Jaccard score': [knn_jaccard_score, DecisionTrees_jaccard_score, SVM_jaccard_score, LogisticRegression_jaccard_score],
    'F1 score': [knn_f1_score, DecisionTrees_f1_score, SVM_f1_score, LogisticRegression_f1_score],
    'Logistic Regression score': ['NA', 'NA', 'NA', LogisticRegression_LogLoss]
}
report = pd.DataFrame(data=d, index=['KNN', 'Decision Tree', 'SVM', 'LogisticRegression'])
report
color = [('grey', 'grey', 'mediumvioletred', 'grey'), ('lightgrey','lightgrey', 'thistle', 'lightgrey')]
report.plot(kind='bar', color = color)

plt.title('Model Evaluation: out of sample set')

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1247417669218617, 'fraction_numerical': 0.019564953214242314, 'mean_word_length': 7.431864754098361, 'pattern_counts': {'":': 6, '<': 94, '<?xml version=': 0, '>': 97, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '609875', 'license': 'MIT', 'n_tokens_mistral': 6001, 'n_tokens_neox': 5599, 'n_words': 1283, 'path': 'Labs/Banking: Loan default forecast.ipynb', 'repo_name': 'AnnaSukhareva/DataScienceLabs', 'size': '16479'}