<jupyter_start><jupyter_text>The Predictor to Permit Loan ***  A note on the Dataset This data-set was obtained from Kaggle (https://www.kaggle.com/zhijinzhai/loandata/data) Some information about the data columns:**Loan_id** A unique loan number assigned to each loan customers**Loan_status** Whether a loan is paid off, in collection, new customer yet to payoff, or paid off after the collection efforts**Principal Basic** principal loan amount at the origination terms Can be weekly (7 days), biweekly, and monthly payoff schedule**Effective_date** When the loan got originated and took effects**Due_date Since** it’s one-time payoff schedule, each loan has one single due date**Paidoff_time** The actual time a customer pays off the loan**Pastdue_days** How many days a loan has been past due**Age**, **education**, **gender** A customer’s basic demographic information  k-Nearest Neighbour<jupyter_code>import itertools
import numpy as np # linear algebra
import matplotlib.pyplot as plt
from matplotlib.ticker import NullFormatter
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import numpy as np
import matplotlib.ticker as ticker
import seaborn as sns
from sklearn import preprocessing
%matplotlib inline

data_frame = pd.read_csv("Loan_payments_data.csv")
# print ("Shape of dataset before cleaning: ", data_frame.size)

# data_frame = data_frame.dropna()
# data_frame = data_frame.reset_index(drop=True)
# print ("Shape of dataset after cleaning: ", data_frame.size)
# data_frame.head(5)

# Shape of dataset before cleaning:  5500
# Shape of dataset after cleaning:  1100
data_frame.head()
data_frame.describe()
data1 = data_frame.copy()
data_frame['Principal'].unique()
data_frame['loan_status'].unique()<jupyter_output><empty_output><jupyter_text>There are 3 types of loan status, Now let me explain the following:**'PAIDOFF'** :  means those people gave their money before or on due date without the help of any external agencies**'COLLECTION'** :  Risky people, they havenot given the money and the lenders will have to acquire the money with the help of some 3rd party organisations**'COLLECTION_PAIDOFF'** : These people have given the money after the 3rd party organisations have forced them to do so.  We will see that there are categorical string values which are needed to be converted to numeric "one-hot-encoded" values for better working of the algorithm and reducing the dimension<jupyter_code>#  For converting it to One hot encoded values

from sklearn import preprocessing
le=preprocessing.LabelEncoder()
#data_frame['loan_status']=le.fit_transform(data_frame['loan_status'])
data_frame['Gender']=le.fit_transform(data_frame['Gender'])
data_frame['education']=le.fit_transform(data_frame['education'])
data_frame['past_due_days']=le.fit_transform(data_frame['past_due_days'])
df =data1[data_frame['past_due_days'] ==60 ]
df.head(5)
# convert 'PAIDOFF' to 1 and other two categories to 0
data_frame['loan_status'] = data_frame['loan_status'].map({'PAIDOFF': 1, 'COLLECTION': 0, 'COLLECTION_PAIDOFF': 0})
data_frame['Principal'].value_counts()
data_frame.head()<jupyter_output><empty_output><jupyter_text>A plot of the histogram showing the basic statistics of the Principal amount taken<jupyter_code>data_frame.hist(column='Principal', bins=50)<jupyter_output><empty_output><jupyter_text>Note 0 = 'PAIDOFF', 1 = 'COLLECTION', and 1 = 'COLLECTION_PAIDOFF' are the mapped values<jupyter_code>data_frame.hist(column='loan_status', bins=50)
# Note 0 = 'PAIDOFF', 1 = 'COLLECTION', and 2 = 'COLLECTION_PAIDOFF'
# Creating a new column for storing the difference between due date and paid off time
# data_frame['diff_effective_due'] = data_frame['due_date'].sub(data_frame['effective_date'], axis=0)

data_frame.columns<jupyter_output><empty_output><jupyter_text>Now we create the features X for training<jupyter_code>X = data_frame[[ 'Principal','terms', 'past_due_days', 'Gender', 'age', 'education']] .values  #.astype(float)
X[0:5]<jupyter_output><empty_output><jupyter_text>and y for output labels<jupyter_code>y = data_frame['loan_status'].values
y[0:5]
X = preprocessing.StandardScaler().fit(X).transform(X.astype(float))
X[0:5]
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)
from sklearn import metrics
# print("Train set Accuracy: ", metrics.accuracy_score(y_train, neigh.predict(X_train)))
# print("Test set Accuracy: ", metrics.accuracy_score(y_test, yhat))<jupyter_output><empty_output><jupyter_text>Let us see which K values suits the dataset more<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

Ks = 30
mean_acc = np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))
ConfustionMx = [];
for n in range(1,Ks):
    
    #Train Model and Predict  
    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)
    yhat=neigh.predict(X_test)
    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)

    
    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])

mean_acc
plt.plot(range(1,Ks),mean_acc,'g')
plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)
plt.legend(('Accuracy ', '+/- 3xstd'))
plt.ylabel('Accuracy ')
plt.xlabel('Number of Nabors (K)')
plt.tight_layout()
plt.show()
print( "The best accuracy was with", mean_acc.max(), "with k=", mean_acc.argmax()+1)<jupyter_output>The best accuracy was with 0.93 with k= 1<jupyter_text>So, by using the features of the given dataset, we find K = 1, 8 and 9, wierd isn't it? But this dataset have a pattern like this.<jupyter_code>from sklearn.metrics import jaccard_similarity_score
jaccard_similarity_score(y_test, yhat)
from sklearn.metrics import classification_report, confusion_matrix
import itertools
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
print(confusion_matrix(y_test, yhat, labels=[1,0]))
# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])
np.set_printoptions(precision=2)


# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['churn=1','churn=0'],normalize= False,  title='Confusion matrix')
print (classification_report(y_test, yhat))<jupyter_output>precision    recall  f1-score   support

           0       0.78      0.91      0.84        34
           1       0.95      0.86      0.90        66

    accuracy                           0.88       100
   macro avg       0.86      0.89      0.87       100
weighted avg       0.89      0.88      0.88       100<jupyter_text>***   Decision Tree Method<jupyter_code>import numpy as np 
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
data_frame = pd.read_csv("Loan_payments_data.csv")
#  For converting it to One hot encoded values

from sklearn import preprocessing
le=preprocessing.LabelEncoder()
# let us keep the default 3 classes of the dataframe...
data_frame['loan_status']=le.fit_transform(data_frame['loan_status'])
data_frame['Gender']=le.fit_transform(data_frame['Gender'])
data_frame['education']=le.fit_transform(data_frame['education'])
data_frame['past_due_days']=le.fit_transform(data_frame['past_due_days'])
data_frame = data_frame.drop(['Loan_ID','effective_date', 'due_date','paid_off_time'], axis=1)
data_frame.head()
X = data_frame[['Principal', 'terms', 'past_due_days', 'age', 'education', 'Gender']].values
X[0:5]
y = data_frame["loan_status"]
y[0:5]
from sklearn.model_selection import train_test_split
X_trainset, X_testset, y_trainset, y_testset = train_test_split(X, y, test_size=0.3, random_state=5)
print(X_trainset.shape)
print(y_trainset.shape)

loanTree = DecisionTreeClassifier(criterion="entropy", max_depth = 9)
loanTree # it shows the default parameters
loanTree.fit(X_trainset,y_trainset)
predTree = loanTree.predict(X_testset)
print (predTree [0:5])
print (y_testset [0:5])<jupyter_output>[2 1 2 2 1]
241    2
448    1
75     2
212    2
481    1
Name: loan_status, dtype: int64<jupyter_text>Printing the accuracy of the decision tree<jupyter_code>from sklearn import metrics
import matplotlib.pyplot as plt
print("DecisionTrees's Accuracy: ", metrics.accuracy_score(y_testset, predTree))
from sklearn.externals.six import StringIO
import pydotplus
import matplotlib.image as mpimg
from sklearn import tree
%matplotlib inline 
pd.DataFrame.from_records(X)
#couldn't resolve the issue to draw this graph


dot_data = StringIO()
filename = "loanTree.png"
featureNames = [ "Principal","terms","past_due_days","age","education","Gender"]
targetNames = ["loan_status"]
out=tree.export_graphviz(loanTree,feature_names=featureNames, out_file=dot_data, class_names= "loan_status", filled=True,  special_characters=True,rotate=False)  
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
graph.write_png(filename)
img = mpimg.imread(filename)
plt.figure(figsize=(100, 200))
plt.imshow(img,interpolation='nearest')
tree.plot_tree(loanTree.fit(X_trainset,y_trainset))<jupyter_output><empty_output><jupyter_text>***   Support Vector Machine<jupyter_code>import pandas as pd
import pylab as pl
import numpy as np
import scipy.optimize as opt
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
%matplotlib inline 
import matplotlib.pyplot as plt
data_frame = pd.read_csv("Loan_payments_data.csv")
from sklearn import preprocessing
le=preprocessing.LabelEncoder()
# let us keep the default 3 classes of the dataframe...
#data_frame['loan_status']=le.fit_transform(data_frame['loan_status'])
data_frame['Gender']=le.fit_transform(data_frame['Gender'])
data_frame['education']=le.fit_transform(data_frame['education'])
data_frame['past_due_days']=le.fit_transform(data_frame['past_due_days'])
# convert 'PAIDOFF' to 1 and other two categories to 0
data_frame['loan_status'] = data_frame['loan_status'].map({'PAIDOFF': 1, 'COLLECTION': 0, 'COLLECTION_PAIDOFF': 0})

data_frame.head()<jupyter_output><empty_output><jupyter_text>Lets plot the dataset in terms of the **age** and **past_due_date** with two categories i.e., defaulters and paidoffs<jupyter_code>ax = data_frame[data_frame['loan_status'] == 1][0:50].plot(kind='scatter', x='past_due_days', y='age', color='DarkBlue', label='Paidoff');
data_frame[data_frame['loan_status'] == 0][0:50].plot(kind='scatter', x='past_due_days', y='age', color='Yellow', label='Defaulter', ax=ax);
plt.show()<jupyter_output><empty_output><jupyter_text>Creating the X and Y labels for training<jupyter_code>X = data_frame[['Principal', 'terms', 'past_due_days', 'age', 'education', 'Gender']].values
X[0:5]
data_frame['loan_status'].describe
y = np.asarray(data_frame['loan_status'])
y [0:5]
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)
from sklearn import svm
clf = svm.SVC(kernel='rbf')
clf.fit(X_train, y_train) 
yhat = clf.predict(X_test)
yhat[0:5]
from sklearn.metrics import classification_report, confusion_matrix
import itertools
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, yhat, labels=[0,1])
np.set_printoptions(precision=2)

print (classification_report(y_test, yhat))

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['Paidoff','Defaulter'],normalize= False,  title='Confusion matrix')<jupyter_output>precision    recall  f1-score   support

           0       0.79      1.00      0.88        34
           1       1.00      0.86      0.93        66

    accuracy                           0.91       100
   macro avg       0.90      0.93      0.90       100
weighted avg       0.93      0.91      0.91       100

Confusion matrix, without normalization
[[34  0]
 [ 9 57]]<jupyter_text>Plotting the f1-score<jupyter_code>from sklearn.metrics import f1_score
f1_score(y_test, yhat, average='weighted')<jupyter_output><empty_output><jupyter_text>Plotting the similarity score<jupyter_code>from sklearn.metrics import jaccard_similarity_score
jaccard_similarity_score(y_test, yhat)<jupyter_output>/usr/lib/python3.8/site-packages/sklearn/metrics/_classification.py:656: FutureWarning: jaccard_similarity_score has been deprecated and replaced with jaccard_score. It will be removed in version 0.23. This implementation has surprising behavior for binary and multiclass classification tasks.
  warnings.warn('jaccard_similarity_score has been deprecated and replaced '<jupyter_text>***   Logistic Regression<jupyter_code>import pandas as pd
import pylab as pl
import numpy as np
import scipy.optimize as opt
from sklearn import preprocessing
%matplotlib inline 
import matplotlib.pyplot as plt
data_frame = pd.read_csv("Loan_payments_data.csv")
from sklearn import preprocessing
le=preprocessing.LabelEncoder()
# let us keep the default 3 classes of the dataframe...
#data_frame['loan_status']=le.fit_transform(data_frame['loan_status'])
data_frame['Gender']=le.fit_transform(data_frame['Gender'])
data_frame['education']=le.fit_transform(data_frame['education'])
data_frame['past_due_days']=le.fit_transform(data_frame['past_due_days'])
# convert 'PAIDOFF' to 1 and other two categories to 0
data_frame['loan_status'] = data_frame['loan_status'].map({'PAIDOFF': 1, 'COLLECTION': 0, 'COLLECTION_PAIDOFF': 0})
data_frame.head()<jupyter_output><empty_output><jupyter_text>Lets prepare the dataset<jupyter_code>X = data_frame[['Principal', 'terms', 'past_due_days', 'age', 'education', 'Gender']].values
X[0:5]
y = np.asarray(data_frame['loan_status'])
y [0:5]
from sklearn import preprocessing
X = preprocessing.StandardScaler().fit(X).transform(X)
X[0:5]
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
LR = LogisticRegression(C=0.01, solver='liblinear').fit(X_train,y_train)
LR
yhat = LR.predict(X_test)
yhat
yhat_prob = LR.predict_proba(X_test)
yhat_prob[:5]<jupyter_output><empty_output><jupyter_text>Evaluation  jaccard index<jupyter_code>from sklearn.metrics import jaccard_similarity_score
jaccard_similarity_score(y_test, yhat)
from sklearn.metrics import classification_report, confusion_matrix
import itertools
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
print(confusion_matrix(y_test, yhat, labels=[1,0]))<jupyter_output>[[56 10]
 [ 0 34]]<jupyter_text>Plotting the confusion matrix<jupyter_code># Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])
np.set_printoptions(precision=2)


# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['churn=1','churn=0'],normalize= False,  title='Confusion matrix')
print (classification_report(y_test, yhat))<jupyter_output>precision    recall  f1-score   support

           0       0.77      1.00      0.87        34
           1       1.00      0.85      0.92        66

    accuracy                           0.90       100
   macro avg       0.89      0.92      0.89       100
weighted avg       0.92      0.90      0.90       100<jupyter_text>Plotting the logloss<jupyter_code>from sklearn.metrics import log_loss
log_loss(y_test, yhat_prob)<jupyter_output><empty_output><jupyter_text>***   Some Extra Analysis on the dataset   (may skip if you want to save time, just for playing around)<jupyter_code>import numpy as np 
import pandas as pd
from scipy import ndimage 
from scipy.cluster import hierarchy 
from scipy.spatial import distance_matrix 
from matplotlib import pyplot as plt 
from sklearn import manifold, datasets 
from sklearn.cluster import AgglomerativeClustering 
from sklearn.datasets.samples_generator import make_blobs 
%matplotlib inline
data_frame.head()
featureset = data_frame[[ 'Principal','terms', 'past_due_days', 'Gender', 'age', 'education', 'education']]

from sklearn.preprocessing import MinMaxScaler
x = featureset.values #returns a numpy array
min_max_scaler = MinMaxScaler()
feature_mtx = min_max_scaler.fit_transform(x)
feature_mtx [0:5]
import scipy
leng = feature_mtx.shape[0]
D = np.zeros([leng,leng])
for i in range(leng):
    for j in range(leng):
        D[i,j] = scipy.spatial.distance.euclidean(feature_mtx[i], feature_mtx[j])
import pylab
import scipy.cluster.hierarchy
Z = scipy.cluster.hierarchy.linkage(D, 'complete')
from scipy.cluster.hierarchy import fcluster
max_d = 3
clusters = fcluster(Z, max_d, criterion='distance')
clusters<jupyter_output><empty_output><jupyter_text>Hierarchical Clustering according to Principal, Age and education<jupyter_code># loan_status 	Principal 	terms 	effective_date 	due_date 	paid_off_time 	past_due_days 	age 	education 	Gender

fig = pylab.figure(figsize=(18,50))
def llf(id):
    return '[%s %s %s]' % (data_frame['Principal'][id], data_frame['age'][id], int(float(data_frame['education'][id])) )
    
dendro = scipy.cluster.hierarchy.dendrogram(Z,  leaf_label_func=llf, leaf_rotation=0, leaf_font_size =12, orientation = 'right')


import os
import numpy as np
import pandas as pd
import datetime 
import seaborn as sns
sns.set_style("dark")

import matplotlib.pyplot as plt
%matplotlib inline

import warnings
warnings.filterwarnings("ignore")



def my_read_file(filename):
    df = pd.read_csv(filename)
    print("{}: Reading {}.".format(now(), filename))
    print("{}: The data contains {} observations with {} columns".format(now(), df.shape[0], df.shape[1]))
    return df

def now():
    tmp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    return tmp

# Self-defined function to read dataframe and find the missing data on the columns and # of missing
def checking_na(df):
    try:
        if (isinstance(df, pd.DataFrame)):
            df_na_bool = pd.concat([df.isnull().any(), df.isnull().sum(), (df.isnull().sum()/df.shape[0])*100],
                                   axis=1, keys=['df_bool', 'df_amt', 'missing_ratio_percent'])
            df_na_bool = df_na_bool.loc[df_na_bool['df_bool'] == True]
            return df_na_bool
        else:
            print("{}: The input is not panda DataFrame".format(now()))

    except (UnboundLocalError, RuntimeError):
        print("{}: Something is wrong".format(now()))
loan_data = my_read_file("Loan_payments_data.csv")
print("\n\n")
print(checking_na(loan_data))

import seaborn as sns
sns.set_style("dark")
fig, axs = plt.subplots(1, 2, figsize=(16, 5))
sns.boxplot(x='loan_status', y='Principal', data=data_frame, hue='loan_status', ax=axs[0])
sns.distplot(loan_data.Principal, bins=range(300, 1000, 100), ax=axs[1], kde=True)
plt.show();

print(loan_data.loan_status.unique())

fig = plt.figure(figsize=(5,5))
ax = sns.countplot(loan_data.loan_status)
ax.set_title("Count of Loan Status")
for p in ax.patches:
    ax.annotate(str(format(int(p.get_height()), ',d')), (p.get_x(), p.get_height()*1.01))
plt.show()

fig, axs = plt.subplots(1, 2, figsize=(16,5))
sns.countplot(loan_data.terms, ax=axs[0])
axs[0].set_title("Count of Terms of loan")
for p in axs[0].patches:
    axs[0].annotate(str(format(int(p.get_height()), ',d')), (p.get_x(), p.get_height()*1.01))

sns.countplot(x='terms', hue='loan_status', data=loan_data, ax=axs[1])
axs[1].set_title("Term count breakdown by loan_status")
for t in axs[1].patches:
    if (np.isnan(float(t.get_height()))):
        axs[1].annotate(0, (t.get_x(), 0))
    else:
        axs[1].annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))

axs[1].legend(loc='upper left')
plt.show();

fig = plt.figure(figsize=(10,5))
ax = sns.countplot(x='effective_date', hue='loan_status', data=loan_data)
ax.set_title('Loan date')
for t in ax.patches:
    if (np.isnan(float(t.get_height()))):
        ax.annotate(0, (t.get_x(), 0))
    else:
        ax.annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))
plt.show();

# Note: if we think that the day of week, or month has the significant factor to the loan status
# Below is the function which we can use to extract the year, month, or day:
# pd.DatetimeIndex(loan_data.effective_date).year
# pd.DatetimeIndex(loan_data.effective_date).month
# pd.DatetimeIndex(loan_data.effective_date).day<jupyter_output><empty_output><jupyter_text>People are more prone to pay off in the last day<jupyter_code>loan_data['paid_off_date'] = pd.DatetimeIndex(loan_data.paid_off_time).normalize()
fig = plt.figure(figsize=(16, 6))
ax = sns.countplot(x='paid_off_date', data=loan_data.loc[loan_data.loan_status.isin(['COLLECTION_PAIDOFF', 'PAIDOFF'])] , hue='loan_status')
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
for t in ax.patches:
    if (np.isnan(float(t.get_height()))):
        ax.annotate(0, (t.get_x(), 0))
    else:
        ax.annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))

ax.legend(loc='upper right')
plt.show();

# Compute the day to pay-off the loan
loan_data['day_to_pay'] = (pd.DatetimeIndex(loan_data.paid_off_time).normalize() - pd.DatetimeIndex(loan_data.effective_date).normalize()) / np.timedelta64(1, 'D')

fig = plt.figure(figsize=(15, 5))
ax = sns.countplot(x='day_to_pay', hue='terms', data=loan_data)
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
for t in ax.patches:
    if (np.isnan(float(t.get_height()))):
        ax.annotate('', (t.get_x(), 0))
    else:
        ax.annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))
        
plt.show();


fig = plt.figure(figsize=(15, 5))
ax = sns.countplot(x='day_to_pay', hue='terms', data=loan_data.loc[loan_data.loan_status == 'PAIDOFF'])
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
for t in ax.patches:
    if (np.isnan(float(t.get_height()))):
        ax.annotate('', (t.get_x(), 0))
    else:
        ax.annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))
        
plt.show();
tmp = loan_data.loc[(loan_data.day_to_pay > 30) & (loan_data.loan_status == 'PAIDOFF')]
print("{}: Incorrect status: {} observations".format(now(), tmp.shape[0]))
print(tmp[['loan_status', 'terms', 'effective_date', 'due_date', 'paid_off_time']])
fig, axs = plt.subplots(3, 2, figsize=(16, 15))
sns.distplot(loan_data.age, ax=axs[0][0])
axs[0][0].set_title("Total age distribution across dataset")
sns.boxplot(x='loan_status', y='age', data=loan_data, ax=axs[0][1])
axs[0][1].set_title("Age distribution by loan status")
sns.countplot(x='education', data=loan_data, ax=axs[1][0])
axs[1][0].set_title("Education count")
for t in axs[1][0].patches:
    if (np.isnan(float(t.get_height()))):
        axs[1][0].annotate('', (t.get_x(), 0))
    else:
        axs[1][0].annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))

sns.countplot(x='education', data=loan_data, hue='loan_status', ax=axs[1][1])
axs[1][1].set_title("Education by loan status")
for t in axs[1][1].patches:
    if (np.isnan(float(t.get_height()))):
        axs[1][1].annotate('', (t.get_x(), 0))
    else:
        axs[1][1].annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))

axs[1][1].legend(loc='upper right')
sns.countplot(x='Gender', data=loan_data, ax=axs[2][0])
axs[2][0].set_title("# of Gender")
for t in axs[2][0].patches:
    if (np.isnan(float(t.get_height()))):
        axs[2][0].annotate('', (t.get_x(), 0))
    else:
        axs[2][0].annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))

sns.countplot(x='Gender', data=loan_data, hue='education', ax=axs[2][1])
axs[2][1].set_title("Education of the gender")
for t in axs[2][1].patches:
    if (np.isnan(float(t.get_height()))):
        axs[2][1].annotate('', (t.get_x(), 0))
    else:
        axs[2][1].annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))

plt.show();
status_map = {"PAIDOFF": 1, "COLLECTION": 2, "COLLECTION_PAIDOFF": 2 }
loan_data['loan_status_trgt'] = loan_data['loan_status'].map(status_map)

fig, axs = plt.subplots(1, 2, figsize=(15, 5))
sns.countplot(x='loan_status', data=loan_data, ax=axs[0])
axs[0].set_title("Count using original target labels")
for t in axs[0].patches:
    if (np.isnan(float(t.get_height()))):
        axs[0].annotate('', (t.get_x(), 0))
    else:
        axs[0].annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))

sns.countplot(x='loan_status_trgt', data=loan_data, ax=axs[1])
axs[1].set_title("Count using new target labels")
for t in axs[1].patches:
    if (np.isnan(float(t.get_height()))):
        axs[1].annotate('', (t.get_x(), 0))
    else:
        axs[1].annotate(str(format(int(t.get_height()), ',d')), (t.get_x(), t.get_height()*1.01))
    
plt.show();
dummies = pd.get_dummies(loan_data['education']).rename(columns=lambda x: 'is_' + str(x))
loan_data = pd.concat([loan_data, dummies], axis=1)
loan_data = loan_data.drop(['education'],  axis=1)

dummies = pd.get_dummies(loan_data['Gender']).rename(columns=lambda x: 'is_' + str(x))
loan_data = pd.concat([loan_data, dummies], axis=1)
loan_data = loan_data.drop(['Gender'], axis=1)

loan_data = loan_data.drop(['Loan_ID', 'loan_status', 'effective_date', 'due_date', 'paid_off_time', 'past_due_days', 'paid_off_date', 'day_to_pay'], axis=1)

dummy_var = ['is_female', 'is_Master or Above']
loan_data = loan_data.drop(dummy_var, axis = 1)

print(loan_data.head(2))

# ML library

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn import svm

### Create function to evaluate the score of each classification model
def eval_model_classifier(model, data, target, split_ratio):
    trainX, testX, trainY, testY = train_test_split(data, target, train_size=split_ratio, random_state=0)
    model.fit(trainX, trainY)    
    return model.score(testX,testY)


### 1st round: RandomForestClassification

# Initialise values
num_estimators_array = np.array([1,5,10,50,100,200,500]) 
num_smpl = 5 # Test run the model according to samples_number
num_grid = len(num_estimators_array)
score_array_mu = np.zeros(num_grid) # Keep mean
score_array_sigma = np.zeros(num_grid) # Keep Standard deviation 
j=0

print("{}: RandomForestClassification Starts!".format(now()))
for n_estimators in num_estimators_array:
    score_array = np.zeros(num_smpl) # Initialize
    for i in range(0,num_smpl):
        rf_class = RandomForestClassifier(n_estimators = n_estimators, n_jobs=1, criterion="gini")
        score_array[i] = eval_model_classifier(rf_class, X, y, 0.8)
        print("{}: Try {} with n_estimators = {} and score = {}".format(now(), i, n_estimators, score_array[i]))
    score_array_mu[j], score_array_sigma[j] = np.mean(score_array), np.std(score_array)
    j=j+1

print("{}: RandomForestClassification Done!".format(now()))
fig = plt.figure(figsize=(7,3))
plt.errorbar(num_estimators_array, score_array_mu, yerr=score_array_sigma, fmt='k.-')
plt.xscale("log")
plt.xlabel("number of estimators",size = 16)
plt.ylabel("accuracy",size = 16)
plt.xlim(0.9,600)
plt.ylim(0.3,0.8)
plt.title("Random Forest Classifier", size = 18)
plt.grid(which="both")
plt.show();
C_array = np.array([0.5, 0.1, 1, 5, 10])
score_array = np.zeros(len(C_array))
i=0
for C_val in C_array:
    svc_class = svm.SVC(kernel='linear', random_state=1, C = C_val)
    score_array[i] = eval_model_classifier(svc_class, X, y, 0.8)
    i=i+1

score_mu, score_sigma = np.me`an(score_array), np.std(score_array)

fig = plt.figure(figsize=(7,3))
plt.errorbar(C_array, score_array, yerr=score_sigma, fmt='k.-')
plt.xlabel("C assignment",size = 16)
plt.ylabel("accuracy",size = 16)
plt.title("SVM Classifier (Linear)", size = 18)
plt.grid(which="both")
plt.show();
# Note: 
# Gamma: Kernel coefficient - the higher, it will try to exact fit to the training data, hence, can cause overfitting

gamma_array = np.array([0.001, 0.01, 0.1, 1, 10])
score_array = np.zeros(len(gamma_array))
score_mu = np.zeros(len(gamma_array))
score_sigma = np.zeros(len(gamma_array))
i=0
for gamma_val in gamma_array:
    svc_class = svm.SVC(kernel='rbf', random_state=1, gamma = gamma_val)
    score_array[i] = eval_model_classifier(svc_class, X, y, 0.8)
    score_mu[i], score_sigma[i] = np.mean(score_array[i]), np.std(score_array[i])
    i=i+1


fig = plt.figure(figsize=(10,5))
plt.errorbar(gamma_array, score_mu, yerr=score_sigma, fmt='k.-')
plt.xscale('log')
plt.xlabel("Gamma",size = 16)
plt.ylabel("accuracy",size = 16)
plt.title("SVM Classifier (RBF)", size = 18)
plt.grid(which="both")
plt.show();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14376856919468334, 'fraction_numerical': 0.021266614542611414, 'mean_word_length': 5.019578313253012, 'pattern_counts': {'":': 3, '<': 86, '<?xml version=': 0, '>': 90, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '517836', 'license': 'MIT', 'n_tokens_mistral': 12002, 'n_tokens_neox': 11318, 'n_words': 2490, 'path': 'python/coursera_python/ML_WITH_PYTHON_IBM/Proj/Loan_Prediction_Project.ipynb', 'repo_name': 'SayanGhoshBDA/code-backup', 'size': '31975'}