<jupyter_start><jupyter_text>UnderwittingLoan data for all loans issued by XYZ Corp. through 2007-2015.Objective :-You have to build a data model to predict the probability of default, and choose a cut-off based on what you feel is suitable. Alternatively you can also use a modelling technique which gives binary output.<jupyter_code>import pandas as pd
import math
import numpy as np 
from scipy import stats 
import matplotlib.pyplot as plt
import seaborn as sns 
import datetime
from  collections import Counter 
from sklearn.metrics import confusion_matrix , classification_report , accuracy_score
from sklearn.linear_model import LogisticRegression 
#from sklearn.cross_validation import KFold
#from sklearn.grid_search import GridSearchCV
from xgboost import XGBClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
import warnings
warnings.filterwarnings("ignore")
df = pd.read_table(r"C:\Users\kusha\Desktop\data sci\python\pythonProject\XYZCorp_LendingData.text" , index_col = None)
df.shape
df.head()<jupyter_output><empty_output><jupyter_text>Creating Copy of dataframe.<jupyter_code>df_copy = pd.DataFrame.copy(df)
pd.set_option('display.max_columns', 80)
df_copy.head(n = 2)<jupyter_output><empty_output><jupyter_text>Before proceeding ahead , lets remove those variables which are not important for modeling or prediction. Variables "id" , "member_id" , "zip_code" , "addr_state" , "last_pymnt_d" , "next_pymnt_d" , "last_credit_pull_d","desc" and "policy_code" should be removed as they are not important for modeling. We wont remove "issue_d" for now cause we will have to use it for spliting our data in train and test .<jupyter_code>remove_list = ["id" , "member_id" , "zip_code" , "addr_state" , "last_pymnt_d" , "next_pymnt_d" , "last_credit_pull_d" 
               , "desc" ,"policy_code"]
for i in remove_list:
    del df_copy[i]<jupyter_output><empty_output><jupyter_text>The on going payment processes should not be considered as these values can mislead the machine to predict wrong valuesAs the problem statement states : "In this project you will have to put yourself in the shoes of a loan issuer and manage credit risk by using the past data and deciding whom to give the loan to in the future. The text files contain complete loan data for all loans issued by XYZ Corp. through 2007-2015. The data contains the indicator of default, payment information, credit history, etc.The data should be divided into train ( June 2007 - May 2015 ) and out-of-time test ( June 2015 - Dec 2015 ) data.""DECIDING WHOM TO GIVE LOAN IN FUTURE"So , variables like total_pymnt	: Payments received to date for total amount fundedtotal_pymnt_inv :	Payments received to date for portion of total amount funded by investorstotal_rec_int :	Interest received to date total_rec_late_fee :	Late fees received to date total_rec_prncp	: Principal received to dateSo whenever a customer comes for loan i wont ask him "How much amount of loan have you cleared yet ?" , how will the customer answer this question , when we have not yet decided whether to give loan or not to the customer.And even this values may mislead the model to predict wrong values cause we have to predict on data of customers who's final loan payment date hasnt even arrived or have taken the loan recently .<jupyter_code>remove_list = ["total_pymnt" , "total_pymnt_inv" , "total_rec_int" , "total_rec_late_fee" , "total_rec_prncp"]
for i in remove_list:
    del df_copy[i]
df_copy.shape<jupyter_output><empty_output><jupyter_text>Missing Values<jupyter_code>def nulls():
    mvs = pd.DataFrame()
    labels = []
    percent = []
    
    for cols in df.columns.values:
        if df[cols].isnull().sum() > 0:
            p = (df[cols].isnull().sum()/df.shape[0])
            labels.append(cols)
            percent.append(p)
    mvs["labels"] = labels
    mvs["percent"] = percent
    mvs = mvs.sort_values("percent")
    y_pos = np.arange(len(mvs))
    plt.barh(y_pos , mvs["percent"]  , align = "center" , alpha = 0.5)
    plt.yticks(y_pos , mvs["labels"])
    plt.xlabel("Percent of missing values")
    plt.title("Missing value plot") 
    plt.rcParams["figure.figsize"] = (15,5)
    plt.show()
nulls()<jupyter_output><empty_output><jupyter_text>1)As we can infer from the plot that many variables have missing values more than 40% of data.2)In this case we can remove these variables which have missing values more than 40% .Before removing these column we can use dti_joint to combine dti and dti_joint of applicants type joint and replace these calculated dti values of applicants type joint with dti value.Cause Applicant type joint means 2 people together have issued loan .So the dept to income(dti) ratio of one individual from applicant type joint is stored in dti and the dti ratio of other individual is saved in dti_joint.If we remove dti_joint because of missing values then we might loose important information.Dept to income ratio is very important feature or variable in underwritting , to predict whether the loan issuer will default or not.Formula to calculate Dept To Income Ratio :Debt to income ratio = Monthly Expenses / Monthly Income  To get joint dti of Applicant joint we will have find Monthly expenses of both the individuals of applicant type joint.We can calculate Monthly Expense by multiplying  dept to income ratio into monthly incomeAnother reason to compute this value is because ,the applicant type individual is more than 90% and applicant type joint are only 10% .This is a case of imbalanced independent variable, which can lead the model to perform bad.Check the plot below.<jupyter_code>labels = 'individual', 'joint'
sizes = [855527, 442 ]
colors = ['lightskyblue', 'yellow']
explode = (0.10, 0.5)  # explode 1st slice
 
# Plot
patches , text = plt.pie(sizes, explode=explode, labels=labels, colors=colors, shadow=True, startangle=140)
plt.legend(patches, labels, loc="best")
plt.axis('equal')
plt.show()
df["application_type"].value_counts()<jupyter_output><empty_output><jupyter_text>There are only 442 joint applicants out of 855969 applicants ,which is very less.Therefore creating new dti values for applicant type joint is a better option.Same we can do with annual income and annual income joint.So now lets create a functions to replace the dti ratio and annual income of joint applicant's.<jupyter_code>def newDti(df):
    monthly_inc = (df["annual_inc"][df["dti_joint"].isnull() == False]/12)
    monthly_inc_joint = (df["annual_inc_joint"][df["dti_joint"].isnull() == False]/12)
 
    dti_ = (df["dti"][df["dti_joint"].isnull() == False])
    dti_j = (df["dti_joint"][df["dti_joint"].isnull() == False])
    
    monthly_expenses = dti_*monthly_inc
    monthly_expenses_joint = dti_j*monthly_inc_joint
    
    numerator = monthly_expenses + monthly_expenses_joint
    denominator = monthly_inc + monthly_inc_joint    
    
    new_dti = numerator/denominator
    
    ind = df["dti"][df["dti_joint"].isnull() == False].index
    
    df["dti"][ind] = new_dti

def new_annual_inc(df):
    
    combined_inc = df["annual_inc"][df["annual_inc_joint"].isnull() == False] + df["annual_inc_joint"][df["annual_inc_joint"].isnull() ==  False]
    ind = df["annual_inc"][df["annual_inc_joint"].isnull() == False].index
    df["annual_inc"][ind] = combined_inc

newDti(df_copy)
new_annual_inc(df_copy)<jupyter_output><empty_output><jupyter_text>Removing variables which have missing values more than 40%.<jupyter_code>def remove_variables_missing_values_more_than_40_percent(df):
    for var in df.columns.values:
        if (df[var].isnull().sum()/df.shape[0]) >= 0.4:
            del df[var]
            
remove_variables_missing_values_more_than_40_percent(df_copy)
df_copy.shape<jupyter_output><empty_output><jupyter_text>Below are the variables which are important according to the problem statement.<jupyter_code>final_var= [
"annual_inc","delinq_2yrs","dti","emp_length","funded_amnt","grade","home_ownership","initial_list_status",
"installment","int_rate","loan_amnt","open_acc","inq_last_6mths","pub_rec","purpose","revol_bal","term",
"verification_status","tot_cur_bal","issue_d","recoveries","collections_12_mths_ex_med","default_ind"
]
def kep_col(data,var):
    for col in data.columns.values:
        if not col in var:
            del data[col]

kep_col(df_copy , final_var)
df_copy.shape
df_copy.isnull().sum()<jupyter_output><empty_output><jupyter_text>Treating missing values<jupyter_code>print("mode : ",df_copy["tot_cur_bal"].mode()[0])
print("mean : ",np.mean(df_copy["tot_cur_bal"]))
print("quantile 0.05 : ",df_copy["tot_cur_bal"].quantile(0.05),"quantile 0.25 : ",df_copy["tot_cur_bal"].quantile(0.25) ,
      "quantile 0.50 : ", df_copy["tot_cur_bal"].quantile(0.50) , 
      "quantile 0.75 : ",df_copy["tot_cur_bal"].quantile(0.75) ,"quantile 0.95 : ", df_copy["tot_cur_bal"].quantile(0.95))
df_copy['tot_cur_bal'].fillna(df_copy['tot_cur_bal'].mean(),inplace=True) 
df_copy['collections_12_mths_ex_med'].fillna(df_copy['collections_12_mths_ex_med'].mean(),inplace=True)<jupyter_output><empty_output><jupyter_text>Correlation Matrix<jupyter_code>correlation_matrix = df_copy.corr()
a4_dimens = (15.7 , 10.27)
fig , ax = plt.subplots(figsize = a4_dimens)
sns.heatmap(correlation_matrix, cmap="Blues", vmax=0.8, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5} , 
            annot=True,
            xticklabels=correlation_matrix.columns.values ,
            yticklabels=correlation_matrix.columns.values,
            )

plt.xticks(rotation = 90)
plt.yticks(rotation = 360)
plt.show()<jupyter_output><empty_output><jupyter_text>From the below plot we infer that "loan_amnt","funded_amnt"and"installment" are highly correlated to each other and can lead the model to perform poor. So in this case we can remove funded_amnt and installment , both are derieved from loan_amnt .<jupyter_code>del df_copy["installment" ]
del df_copy["funded_amnt"]
df_copy.shape
df_copy.describe()
def biased_IF(data):
    for x in data:
        val = data[x].unique()[0]
        s = data[x][data[x] == val].size
        p = (s/data.shape[0])*100
        
        if p >= 50 and (data[x].dtype == "int" or data[x].dtype == "float64") :
            
            print(x)
            print("value : ",val,"appearance : ",s , " : percent : ",p , "\n")
            
biased_IF(df_copy)<jupyter_output>delinq_2yrs
value :  0.0 appearance :  692685  : percent :  80.9240755214266 

pub_rec
value :  0.0 appearance :  725455  : percent :  84.75248519514142 

recoveries
value :  0.0 appearance :  831782  : percent :  97.17431355574793 

collections_12_mths_ex_med
value :  0.0 appearance :  844768  : percent :  98.69142457261887<jupyter_text>From the above output we see that varibles recoveries  and collection_12_mths_ex_med have more than 90% biased values . This lead the model to perform poor . in this case we can remove these 2 variables .But we can still use pub_rec and delinq_2yrs , we can convert these variables into categorical data .<jupyter_code>df_copy = df_copy.drop(['recoveries','collections_12_mths_ex_med'],axis=1) 
df_copy.shape
df_copy["delinq_2yrs"] = df_copy["delinq_2yrs"].astype(int)
df_copy["delinq_2yrs"][df_copy["delinq_2yrs"] > 0 ] = 1
df_copy["delinq_2yrs"].unique()
df_copy["pub_rec"] = df_copy["pub_rec"].astype(int)
df_copy["pub_rec"][df_copy["pub_rec"] > 0 ] = 1
df_copy["pub_rec"].unique()<jupyter_output><empty_output><jupyter_text>Binning<jupyter_code>print("min : " ,min(df_copy["annual_inc"]) ,"max : ", max(df_copy["annual_inc"]))
print("mode : " ,df_copy["annual_inc"].mode()[0])
print("mean : ",np.mean(df_copy["annual_inc"]) ,"median : ",np.median(df["annual_inc"]))
print("quantile 0.25 : ", df_copy["annual_inc"].quantile(0.25) ," quantile 0.50 : ", df_copy["annual_inc"].quantile(0.50) , 
     " quantile 0.75 : ", df_copy["annual_inc"].quantile(0.75) ," quantile 0.95 : ", df_copy["annual_inc"].quantile(0.95))
cat_an = pd.cut(df_copy["annual_inc"] , bins = [0,30000,50000,60000,90000,150000,9600000] , 
                labels = [6 , 5 , 4 , 3 , 2 , 1])
print(cat_an.value_counts())
c = Counter(cat_an)
labels = []
for i in np.arange(0 , 6):
        labels.append(c.most_common()[i][0])

sns.countplot(y = cat_an , order=labels )
plt.show()
df_copy["annual_inc"] = cat_an
df_copy["annual_inc"] = df_copy["annual_inc"].astype(int)
print(df_copy["annual_inc"].head())
print("min : ",min(df_copy["revol_bal"]) ,"max : ", max(df_copy["revol_bal"]))
print("mode : ",df_copy["revol_bal"].mode()[0])
print("mean : ",np.mean(df_copy["revol_bal"]) ,"median : ",np.median(df["revol_bal"]))
print("quantile 0.25 : ",df_copy["revol_bal"].quantile(0.25) ," quantile 0.50 : ", df_copy["revol_bal"].quantile(0.50) , 
      " quantile 0.75 : ",df_copy["revol_bal"].quantile(0.75) ," quantile 0.95 : ", df_copy["revol_bal"].quantile(0.95))
cat_revol = pd.cut(df_copy["revol_bal"] , bins = [-1,5000,10000,20000,40000,3000000 ] , 
                   labels = [1,2,3,4,5])
print(cat_revol.value_counts())
c_revol = Counter(cat_revol)
labels_revol = []
for i in np.arange(0 , 5):
        labels_revol.append(c_revol.most_common()[i][0])
sns.countplot(y = cat_revol , order=labels_revol)
plt.show()
df_copy["revol_bal"] = cat_revol
df_copy["revol_bal"] = df_copy["revol_bal"].astype(int)
print(df_copy["revol_bal"].head())
print("min : ",min(df_copy["tot_cur_bal"]) ,"max : ", max(df_copy["tot_cur_bal"]))
print("mode : ",df_copy["tot_cur_bal"].mode()[0])
print("mean : ",np.mean(df_copy["tot_cur_bal"]) ,"median : ", np.median(df_copy["tot_cur_bal"]))
print("quantile 0.25 : ",df_copy["tot_cur_bal"].quantile(0.25) ," quantile 0.50 : ", df_copy["tot_cur_bal"].quantile(0.50) , 
      " quantile 0.75 : ",df_copy["tot_cur_bal"].quantile(0.75) ," quantile 0.95 : ", df_copy["tot_cur_bal"].quantile(0.95))
cat_tot_cur = pd.cut(df_copy["tot_cur_bal"] , 
                     bins = [-1,10000,30000,60000,120000,200000,400000,9000000],
                     labels = [7,6,5,4,3,2,1])
print(cat_tot_cur.value_counts())
c_tot = Counter(cat_tot_cur)
labels_tot = []
for i in np.arange(0 , 7):
        labels_tot.append(c_tot.most_common()[i][0])
sns.countplot(y = cat_tot_cur , order=labels_tot)
plt.show()
df_copy["tot_cur_bal"] = cat_tot_cur
df_copy["tot_cur_bal"] = df_copy["tot_cur_bal"].astype(int)
print(df_copy["tot_cur_bal"].head())
df_copy.dtypes<jupyter_output><empty_output><jupyter_text>Outliers Treatment<jupyter_code>plt.figure(1)
plt.subplot(141)
sns.boxplot(df_copy["int_rate"] , orient="v" , color="r")

plt.figure(1)
plt.subplot(142)
sns.boxplot(df_copy["dti"] , orient="v" , color="y")

plt.figure(1)
plt.subplot(143)
sns.boxplot(df_copy["inq_last_6mths"] , orient="v" , color="g")

plt.figure(1)
plt.subplot(144)
sns.boxplot(df_copy["open_acc"] , orient="v" , color="b")

plt.show()<jupyter_output><empty_output><jupyter_text>We will replace the outliers with upper whishker.<jupyter_code>outlier_var_list = ["int_rate" , "dti" , "inq_last_6mths" , "open_acc"]
for var in outlier_var_list:
    B= plt.boxplot(df_copy[var])
    value = [item.get_ydata()[1] for item in B['whiskers']]
    upper_whisker = value[1]
    df_copy[var][df_copy[var] > upper_whisker] = upper_whisker
plt.figure(1)
plt.subplot(141)
sns.boxplot(df_copy["int_rate"] , orient="v" , color="r")

plt.figure(1)
plt.subplot(142)
sns.boxplot(df_copy["dti"] , orient="v" , color="y")

plt.figure(1)
plt.subplot(143)
sns.boxplot(df_copy["inq_last_6mths"] , orient="v" , color="g")

plt.figure(1)
plt.subplot(144)
sns.boxplot(df_copy["open_acc"] , orient="v" , color="b")

plt.show()<jupyter_output><empty_output><jupyter_text>Label encoding.<jupyter_code>df_copy["emp_length"][df_copy["emp_length"] == "n/a"] = 0
df_copy["emp_length"][df_copy["emp_length"] == "< 1 year"] = 1
df_copy["emp_length"][df_copy["emp_length"] == "1 year"] = 2
df_copy["emp_length"][df_copy["emp_length"] == "2 years"] = 3
df_copy["emp_length"][df_copy["emp_length"] == "3 years"] = 4
df_copy["emp_length"][df_copy["emp_length"] == "4 years"] = 5
df_copy["emp_length"][df_copy["emp_length"] == "5 years"] = 6
df_copy["emp_length"][df_copy["emp_length"] == "6 years"] = 7
df_copy["emp_length"][df_copy["emp_length"] == "7 years"] = 8
df_copy["emp_length"][df_copy["emp_length"] == "8 years"] = 9
df_copy["emp_length"][df_copy["emp_length"] == "9 years"] = 10
df_copy["emp_length"][df_copy["emp_length"] == "10+ years"] = 11
df_copy["emp_length"] = df_copy["emp_length"].astype(int)
print(df_copy["emp_length"].head())
print(df_copy["emp_length"].unique())
for_label_encode = ["term","grade","home_ownership",
                    "verification_status","purpose","initial_list_status"]
from sklearn import preprocessing
le = {}
for x in for_label_encode:
    le[x] = preprocessing.LabelEncoder()
for x in for_label_encode:
    df_copy[x] = le[x].fit_transform(df_copy.__getattr__(x))
df_copy.head()
for x in for_label_encode:
    print(x,": ")
    print(df[x].unique())
    print(df_copy[x].unique())<jupyter_output>term : 
[' 36 months' ' 60 months']
[0 1]
grade : 
['B' 'C' 'A' 'E' 'F' 'D' 'G']
[1 2 0 4 5 3 6]
home_ownership : 
['RENT' 'OWN' 'MORTGAGE' 'OTHER' 'NONE' 'ANY']
[5 4 1 3 2 0]
verification_status : 
['Verified' 'Source Verified' 'Not Verified']
[2 1 0]
purpose : 
['credit_card' 'car' 'small_business' 'other' 'wedding'
 'debt_consolidation' 'home_improvement' 'major_purchase' 'medical'
 'moving' 'vacation' 'house' 'renewable_energy' 'educational']
[ 1  0 11  9 13  2  4  6  7  8 12  5 10  3]
initial_list_status : 
['f' 'w']
[0 1]<jupyter_text>Now lets split the data into train and test according to date's."The data should be divided into train ( June 2007 - May 2015 ) and out-of-time test ( June 2015 - Dec 2015 ) data."<jupyter_code>issue_dates = []
for year in np.arange(2007,2016) :
    for month in np.arange(1 , 13):
        d = datetime.date(year , month , 1)
        m = str(d.strftime("%b"))
        sep = "-"
        y = str(d.year)
        full = m + sep + y
        issue_dates.append(full)
print(issue_dates[:101])
data_split_type = []
for date in df_copy["issue_d"].values:
    if date in issue_dates[:101]:
        data_split_type.append("train")
    else:
        data_split_type.append("test")
df_copy["type"] =  data_split_type
df_till_may2015 = pd.DataFrame(df_copy[df_copy["type"] == "train"])
df_test = pd.DataFrame(df_copy[df_copy["type"] == "test"])
del df_till_may2015["type"]
del df_till_may2015["issue_d"]

del df_test["type"]
del df_test["issue_d"]
print(df_till_may2015.shape)
print(df_test.shape)<jupyter_output>(598978, 18)
(256991, 18)<jupyter_text>Modeling.As we know that the dependent variable has imbalanced classes or skewed class , so we will take different approaches for modeling like :- 1) Adding penalty to the loss function or cost function.2) Up sampling3) Down sampling 4) Up sampling and down sampling Logistic Regression<jupyter_code>x = df_till_may2015.iloc[ : , :-1].values
y = df_till_may2015.iloc[ : , -1 ].values
print(x)
print(y)
from sklearn.preprocessing import StandardScaler
scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)
from sklearn.model_selection import train_test_split as split
x_train , x_test , y_train , y_test = split(x , y ,
                                            test_size = 0.3 , 
                                            random_state = 123)<jupyter_output><empty_output><jupyter_text>First we will check the classification using Logistic regression without tunning the parameters<jupyter_code>algo = (LogisticRegression())
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)

print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))<jupyter_output>On train : 
confusion matrix
[[387007      2]
 [ 32275      0]]

 
report

             precision    recall  f1-score   support

          0       0.92      1.00      0.96    387009
          1       0.00      0.00      0.00     32275

avg / total       0.85      0.92      0.89    419284

0.923018765324

 On test : 
confusion matrix
[[165813      0]
 [ 13881      0]]

 
report

             precision    recall  f1-score   support

          0       0.92      1.00      0.96    165813
          1       0.00      0.00      0.00     13881

avg / total       0.85      0.92      0.89    179694

0.922752011753<jupyter_text>From above output we see that the model is not predicting 1("defaulters")Now lets tune the parameters .<jupyter_code>algo = (LogisticRegression(class_weight= "balanced" ,  solver= "sag" , max_iter = 500 , tol= 0.0001))
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)

print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))<jupyter_output>On train : 
confusion matrix
[[250540 136469]
 [ 11047  21228]]

 
report

             precision    recall  f1-score   support

          0       0.96      0.65      0.77    387009
          1       0.13      0.66      0.22     32275

avg / total       0.89      0.65      0.73    419284

0.648171644995

 On test : 
confusion matrix
[[107121  58692]
 [  4739   9142]]

 
report

             precision    recall  f1-score   support

          0       0.96      0.65      0.77    165813
          1       0.13      0.66      0.22     13881

avg / total       0.89      0.65      0.73    179694

0.647005464846<jupyter_text>Now he we got better results then before , even though the accuracy is decreasing but model predicted 1("defaulters")And even the precision and recall rate has increased for 1("defaulters")Now as we can see this model is working fine lets change the threshold and see the results<jupyter_code>pred_y_thres = algo.predict_proba(x_train)

pred_y_thres_50_99 = []
for i in pd.DataFrame(pred_y_thres)[1]:
    if i >= 0.5099:
        pred_y_thres_50_99.append(1)
    else :
        pred_y_thres_50_99.append(0)

print("confusion matrix")
print(confusion_matrix(y_train , pred_y_thres_50_99))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_thres_50_99))
print(accuracy_score(y_train , pred_y_thres_50_99))<jupyter_output>confusion matrix
[[257353 129656]
 [ 11626  20649]]

 
report

             precision    recall  f1-score   support

          0       0.96      0.66      0.78    387009
          1       0.14      0.64      0.23     32275

avg / total       0.89      0.66      0.74    419284

0.663039848885<jupyter_text>So we got 2 percent increase in our accuracy .As we can see the accuracy on train is less than 70 which is underfiting lets try by adding penalty to the cost function or loss function and then check acccuracy respectively and the classification reportSo now we will have to bare implementation of logistic and then add penalty to it.<jupyter_code>def costFunction(x,y,theta):
    z = x @ (-(theta.T))
    euler = np.power(math.e , z)
    pred_y = 1/(1 + euler)
    
    costF = ( (-y*((np.log(pred_y)) ) ) - (1-y)* (np.log(1-pred_y)) )
    
    return sum(costF)/len(x)
    
def gradientDescent(x , y , theta , alpha , iterations , convergence_threshold = None):
    cost = np.zeros(iterations)
    for i in range(iterations):
        
        z = x @ (-(theta.T))
        eulers = np.power(math.e , z)
        pred_y = 1/(1+eulers)
        
        penaltied_cost = []
        for p_y , a_y in zip(pred_y,y):
            if a_y == 1:
                penaltied_cost.append( float(5.5*(p_y - a_y)) )
            else:
                penaltied_cost.append( float((p_y - a_y)) )
        penaltied_cost = np.array([penaltied_cost])
        penaltied_cost = penaltied_cost.reshape(penaltied_cost.shape[1] , 1)
        partialDerivativeoftheta = (1/len(x))* (sum((penaltied_cost)*x))
        
        theta = theta - (alpha*partialDerivativeoftheta)
        cost[i] = costFunction(x,y,theta)
        
        
        if not convergence_threshold == None and cost[i] < convergence_threshold:
            break 
    return theta , cost

def logistic_regression( x , theta):
	z  = x @ (-theta.T)
	eulers = np.power(math.e , z)
	pred_y = 1 / (1 + eulers)

	classified = []
	for i in pred_y:
		if i >= 0.5099:
			classified.append(1)
		else:
			classified.append(0)

	return pred_y , classified
alpha = 0.01
iterations = 80
x = df_till_may2015.iloc[ : , :-1].values
y = df_till_may2015.iloc[ : , -1 ].values

x0 = np.ones([x.shape[0],1])
x = np.concatenate((x0,x) , axis = 1)
y = y.reshape(y.shape[0],1)

from sklearn.preprocessing import StandardScaler
scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)

theta = np.zeros([1,x.shape[1]])

from sklearn.model_selection import train_test_split as split
x_train , x_test , y_train , y_test = split(x , y ,
                                            test_size = 0.3 , 
                                            random_state = 123)
coef , new_cost = gradientDescent(x_train , y_train , theta , alpha , iterations)
print(coef)
pred_y_cv , classified_cv = logistic_regression(x_train , coef)
print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , classified_cv))
print("\n \nreport\n")
print(classification_report(y_train , classified_cv))
print(accuracy_score(y_train , classified_cv))

pred_y , classified = logistic_regression(x_test , coef)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , classified))
print("\n \nreport\n")
print(classification_report(y_test , classified))
print(accuracy_score(y_test , classified))<jupyter_output>On train : 
confusion matrix
[[237930 149079]
 [ 10959  21316]]

 
report

             precision    recall  f1-score   support

          0       0.96      0.61      0.75    387009
          1       0.13      0.66      0.21     32275

avg / total       0.89      0.62      0.71    419284

0.618306446227

 On test : 
confusion matrix
[[101723  64090]
 [  4715   9166]]

 
report

             precision    recall  f1-score   support

          0       0.96      0.61      0.75    165813
          1       0.13      0.66      0.21     13881

avg / total       0.89      0.62      0.71    179694

0.617099068416<jupyter_text>Now lets try with up sampling and down sampling.<jupyter_code>from sklearn.utils import resample

df_majority = df_till_may2015[df_till_may2015.default_ind==0]
df_minority = df_till_may2015[df_till_may2015.default_ind==1]

df_minority_upsampled = resample(df_minority, 
                                replace=True,     
                                 n_samples=200000,    
                                random_state=123) 

df_majority_downsampled = resample(df_majority, 
                                 replace=False,    
                                 n_samples=350000,
                                 random_state=123)
 

df_sampled = pd.concat([df_majority_downsampled, df_minority_upsampled])
 
df_sampled.default_ind.value_counts()

x = df_sampled.iloc[ : , :-1].values
y = df_sampled.iloc[ : , -1 ].values


from sklearn.preprocessing import StandardScaler
scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)

from sklearn.model_selection import train_test_split as split
x_train , x_test , y_train , y_test = split(x , y ,
                                            test_size = 0.3 , 
                                            random_state = 123)
algo = (LogisticRegression(solver= "sag" , max_iter = 1500 , tol= 0.00001))
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)

print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))
pred_y_thres = algo.predict_proba(x_train)

pred_y_thres_50_99 = []
for i in pd.DataFrame(pred_y_thres)[1]:
    if i >= 0.5099:
        pred_y_thres_50_99.append(1)
    else :
        pred_y_thres_50_99.append(0)

print("confusion matrix")
print(confusion_matrix(y_train , pred_y_thres_50_99))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_thres_50_99))
print(accuracy_score(y_train , pred_y_thres_50_99))<jupyter_output>confusion matrix
[[211747  33305]
 [ 90947  49001]]

 
report

             precision    recall  f1-score   support

          0       0.70      0.86      0.77    245052
          1       0.60      0.35      0.44    139948

avg / total       0.66      0.68      0.65    385000

0.677267532468<jupyter_text>As we can see the type two error is increasing , lets try Up sampling<jupyter_code>df_majority = df_till_may2015[df_till_may2015.default_ind==0]
df_minority = df_till_may2015[df_till_may2015.default_ind==1]

df_minority_upsampled = resample(df_minority, 
                                replace=True,     
                                 n_samples=500000,    
                                random_state=123) 

df_majority_downsampled = resample(df_majority, 
                                 replace=True,    
                                 n_samples=950000,
                                 random_state=123)
 

df_sampled = pd.concat([df_majority_downsampled, df_minority_upsampled])
 
df_sampled.default_ind.value_counts()
x = df_sampled.iloc[ : , :-1].values
y = df_sampled.iloc[ : , -1 ].values


from sklearn.preprocessing import StandardScaler
scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)

from sklearn.model_selection import train_test_split as split
x_train , x_test , y_train , y_test = split(x , y ,
                                            test_size = 0.3 , 
                                            random_state = 123)
algo = (LogisticRegression(class_weight="balanced" , solver= "sag" , max_iter = 1500 , tol= 0.00001))
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)

print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))
pred_y_thres = algo.predict_proba(x_train)

pred_y_thres_50_99 = []
for i in pd.DataFrame(pred_y_thres)[1]:
    if i >= 0.5099:
        pred_y_thres_50_99.append(1)
    else :
        pred_y_thres_50_99.append(0)

print("confusion matrix")
print(confusion_matrix(y_train , pred_y_thres_50_99))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_thres_50_99))
print(accuracy_score(y_train , pred_y_thres_50_99))<jupyter_output>confusion matrix
[[442079 223463]
 [125981 223477]]

 
report

             precision    recall  f1-score   support

          0       0.78      0.66      0.72    665542
          1       0.50      0.64      0.56    349458

avg / total       0.68      0.66      0.66   1015000

0.655720197044<jupyter_text>As we infer from all the outputs of logistic Regression that , logistic regression is not performing well .Now lets try ensemble models Random Forest and Xgboost<jupyter_code>x = df_till_may2015.iloc[ : , :-1].values
y = df_till_may2015.iloc[ : , -1 ].values

scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)

from sklearn.model_selection import train_test_split as split
x_train , x_test , y_train , y_test = split(x , y ,
                                            test_size = 0.3 , 
                                            random_state = 123)
algo = (RandomForestClassifier())
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)
print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))<jupyter_output>On train : 
confusion matrix
[[387005      4]
 [  5910  26365]]

 
report

             precision    recall  f1-score   support

          0       0.98      1.00      0.99    387009
          1       1.00      0.82      0.90     32275

avg / total       0.99      0.99      0.99    419284

0.985895001956

 On test : 
confusion matrix
[[165434    379]
 [ 13739    142]]

 
report

             precision    recall  f1-score   support

          0       0.92      1.00      0.96    165813
          1       0.27      0.01      0.02     13881

avg / total       0.87      0.92      0.89    179694

0.921433102942<jupyter_text>As we can infer that it performed good on the data it was trained and performed poor on unknown data , this is a case of overfiting lets tune the parameters .<jupyter_code>algo = (RandomForestClassifier(n_estimators = 500 ))
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)
print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))<jupyter_output>On train : 
confusion matrix
[[387009      0]
 [     0  32275]]

 
report

             precision    recall  f1-score   support

          0       1.00      1.00      1.00    387009
          1       1.00      1.00      1.00     32275

avg / total       1.00      1.00      1.00    419284

1.0

 On test : 
confusion matrix
[[165797     16]
 [ 13875      6]]

 
report

             precision    recall  f1-score   support

          0       0.92      1.00      0.96    165813
          1       0.27      0.00      0.00     13881

avg / total       0.87      0.92      0.89    179694

0.922696361592<jupyter_text>Again overfitting<jupyter_code>algo = (RandomForestClassifier(n_estimators = 500 , max_depth= 18))
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)
print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))<jupyter_output>On train : 
confusion matrix
[[387009      0]
 [ 28236   4039]]

 
report

             precision    recall  f1-score   support

          0       0.93      1.00      0.96    387009
          1       1.00      0.13      0.22     32275

avg / total       0.94      0.93      0.91    419284

0.93265662415

 On test : 
confusion matrix
[[165813      0]
 [ 13879      2]]

 
report

             precision    recall  f1-score   support

          0       0.92      1.00      0.96    165813
          1       1.00      0.00      0.00     13881

avg / total       0.93      0.92      0.89    179694

0.922763141785<jupyter_text>Lets try up sampling and down sampling<jupyter_code>from sklearn.utils import resample
df_majority = df_till_may2015[df_till_may2015.default_ind==0]
df_minority = df_till_may2015[df_till_may2015.default_ind==1]

df_minority_upsampled = resample(df_minority, 
                                replace=True,     
                                 n_samples=200000,    
                                random_state=123) 

df_majority_downsampled = resample(df_majority, 
                                 replace=False,    
                                 n_samples=350000,
                                 random_state=123)
 

df_sampled = pd.concat([df_majority_downsampled, df_minority_upsampled])
 
df_sampled.default_ind.value_counts()
x = df_sampled.iloc[ : , :-1].values
y = df_sampled.iloc[ : , -1 ].values


from sklearn.preprocessing import StandardScaler
scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)

from sklearn.model_selection import train_test_split as split
x_train , x_test , y_train , y_test = split(x , y ,
                                            test_size = 0.3 , 
                                            random_state = 123)
algo = (RandomForestClassifier(n_estimators = 501 ))
algo.fit(x_train , y_train)

pred_y_cv = algo.predict(x_train)
print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = algo.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))<jupyter_output>On train : 
confusion matrix
[[245052      0]
 [     0 139948]]

 
report

             precision    recall  f1-score   support

          0       1.00      1.00      1.00    245052
          1       1.00      1.00      1.00    139948

avg / total       1.00      1.00      1.00    385000

1.0

 On test : 
confusion matrix
[[102663   2285]
 [  2658  57394]]

 
report

             precision    recall  f1-score   support

          0       0.97      0.98      0.98    104948
          1       0.96      0.96      0.96     60052

avg / total       0.97      0.97      0.97    165000

0.970042424242<jupyter_text>As we can see we got an amazing result , precision and recall of both the class is proper . Now lets try with XgBoost and check whether we get better results than this.<jupyter_code>x = df_till_may2015.iloc[ : , :-1].values
y = df_till_may2015.iloc[ : , -1 ].values

scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)

from sklearn.model_selection import train_test_split as split
x_train , x_test , y_train , y_test = split(x , y ,
                                            test_size = 0.3 , 
                                            random_state = 123)
classifier=XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=0.8, gamma=0.5, learning_rate=0.01, max_delta_step=2,
       max_depth=9, min_child_weight=5, missing=None, n_estimators=1000,
       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,
       reg_alpha=0, reg_lambda=1, scale_pos_weight=12, seed=27,
       silent=True, subsample=0.8)
classifier.fit(x_train,y_train)
pred_y_cv = classifier.predict(x_train)

print("On train : ")
print("confusion matrix")
print(confusion_matrix(y_train , pred_y_cv))
print("\n \nreport\n")
print(classification_report(y_train , pred_y_cv))
print(accuracy_score(y_train , pred_y_cv))

pred_y = classifier.predict(x_test)
print("\n On test : ")
print("confusion matrix")
print(confusion_matrix(y_test , pred_y))
print("\n \nreport\n")
print(classification_report(y_test , pred_y))
print(accuracy_score(y_test , pred_y))<jupyter_output>On train : 
confusion matrix
[[275564 111445]
 [  6203  26072]]

 
report

             precision    recall  f1-score   support

          0       0.98      0.71      0.82    387009
          1       0.19      0.81      0.31     32275

avg / total       0.92      0.72      0.78    419284

0.71940737066

 On test : 
confusion matrix
[[116025  49788]
 [  4748   9133]]

 
report

             precision    recall  f1-score   support

          0       0.96      0.70      0.81    165813
          1       0.16      0.66      0.25     13881

avg / total       0.90      0.70      0.77    179694

0.696506282903<jupyter_text>So finally we infer that we are getting better results with Random Forest after down sampling majority class and up sampling minority class , with better score of precision and recall and 97% accuracy .<jupyter_code>features = df_till_may2015.columns.values[:-1]
importances = algo.feature_importances_
indices = np.argsort(importances)

plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [features[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()<jupyter_output><empty_output><jupyter_text>Now predicting using random forest on data from June 2015.<jupyter_code>x = df_test.iloc[ : , :-1].values

scalar = StandardScaler()
scalar.fit(x)
x = scalar.transform(x)

y_pred = algo.predict(x)
data_split_type = []
for date in df["issue_d"].values:
    if date in issue_dates[:101]:
        data_split_type.append("train")
    else:
        data_split_type.append("test")
df["type"] =  data_split_type

df_submition = pd.DataFrame(df[df["type"] == "test"])
df_submition["default_ind"] = y_pred

c = Counter(df_submition["default_ind"])
labels = []
size = df_submition["default_ind"].unique().size
for i in np.arange(0 , size):
    labels.append(c.most_common()[i][0])
a4_dimens = (5.7 , 5.27)
fig , ax = plt.subplots(figsize = a4_dimens)
sns.set(style = "dark")
sns.countplot(x = "default_ind" , data = df , order = labels , ax = ax )
plt.xlabel("Loan Status")
plt.show()
df_submition["default_ind"].value_counts()
df_submition.to_csv("final_results.csv" , header = True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11836086554641766, 'fraction_numerical': 0.05991515180243168, 'mean_word_length': 3.35573448952204, 'pattern_counts': {'":': 2, '<': 117, '<?xml version=': 0, '>': 125, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 80, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '469592', 'license': 'MIT', 'n_tokens_mistral': 16468, 'n_tokens_neox': 14769, 'n_words': 3344, 'path': 'Underwritting.ipynb', 'repo_name': 'kushal9090/Data-Science-Final-Project-on-Loan-Default', 'size': '42193'}