<jupyter_start><jupyter_text>Data DescriptionPersonal Loan product is an unsecured loan therefore it is vital to assess the risk of the customers by checking their credit worthiness. This must be done to prevent loan defaults.The objective is to build a Risk model using the dataset which will assess the risk of a customer defaulting after cross-selling the Personal Loan.Column Descriptions:V1: Customer ID\v2: If a customer has bounced in first EMI (1 : Bounced, 0 : Not bounced)\V3: Number of times bounced in recent 12 months\V4: Maximum MOB (Month of business with TVS Credit)\V5: Number of times bounced while repaying the loan\V6: EMI {monthly payment}\V7: Loan Amount\V8: Tenure {length of loan}\V9: Dealer codes from where customer has purchased the Two wheeler\V10: Product code of Two wheeler (MC : Motorcycle , MO : Moped, SC : Scooter)\V11: No of advance EMI paid\V12: Rate of interest\V13: Gender (Male/Female)\V14: Employment type (HOUSEWIFE : housewife, SELF : Self-employed, SAL : Salaried, PENS : Pensioner, STUDENT : Student)\V15: Resident type of customer\V16: Date of birth\V17: Age at which customer has taken the loan\V18: Number of loans\V19: Number of secured loans\V20: Number of unsecured loans\V21: Maximum amount sanctioned in the Live loans\V22: Number of new loans in last 3 months\V23: Total sanctioned amount in the secured Loans which are Live\V24: Total sanctioned amount in the unsecured Loans which are Live\V25: Maximum amount sanctioned for any Two wheeler loan\V26: Time since last Personal loan taken (in months)\V27: Time since first consumer durables loan taken (in months)\V28: Number of times 30 days past due in last 6 months\V29: Number of times 60 days past due in last 6 months\V30: Number of times 90 days past due in last 3 months\V31: Tier ; (Customer's geographical location)\V32: Target variable ( 1: Defaulters / 0: Non-Defaulters)\ Initial Imports<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
!pip install pandas-profiling==2.7.1
from pandas_profiling import ProfileReport
!pip install category_encoders==2.*
!pip install eli5
# !pip install pdpbox
!pip install shap<jupyter_output>Requirement already satisfied: pandas-profiling==2.7.1 in /usr/local/lib/python3.6/dist-packages (2.7.1)
Requirement already satisfied: matplotlib>=3.2.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.7.1) (3.2.2)
Requirement already satisfied: confuse>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.7.1) (1.3.0)
Requirement already satisfied: jinja2>=2.11.1 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.7.1) (2.11.2)
Requirement already satisfied: astropy>=4.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.7.1) (4.0.1.post1)
Requirement already satisfied: htmlmin>=0.1.12 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.7.1) (0.1.12)
Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.7.1) (1.18.5)
Requirement already satisfied: requests>=2.23.0 in /usr/local/lib/python3.6/dist-packages (from pandas-profiling==2.7.1) (2[...]<jupyter_text>Download data, initial examination of data<jupyter_code>url = 'https://raw.githubusercontent.com/SaraWestWA/DS-Unit-2-Applied-Modeling/master/TVS.csv'

df1 = pd.read_csv(url)
df1.head(2)
# Rename columns

column_names = {'V1': 'ID',
'V2':'miss_1st',
'V3':'annual_bounce',
'V4':'max_MOB',
'V5':'total_bounce',
'V6':'monthly_payment',
'V7':'loan_amount',
'V8': 'installments',
'V9': 'dealer',
'V10': 'product',
'V11': 'advance_payments',
'V12': 'int_rate',
'V13': 'sex',
'V14': 'employment',
'V15': 'residence',
'V16': 'DOB',
'V17': 'age',
'V18': 'num_loans',
'V19': 'num_secured_loans',
'V20': 'num_unsecured_loans',
'V21': 'totaL_credit_max',
'V22': 'new_loans',
'V23': 'secured_max',
'V24': 'unsecured_max',
'V25': 'max_loan',
'V26': 'months_last_loan',
'V27': 'months_first_loan',
'V28': 'late_30_6',
'V29': 'late_60_6',
'V30': 'late_90_3',
'V31': 'location',
'V32': 'defaulter'}

df1.rename(columns = column_names, inplace=True)
list(df1)<jupyter_output><empty_output><jupyter_text>Initial Profile Report<jupyter_code>df1.tail(2)
# #Examine the data
# profile = ProfileReport(df1, minimal=True).to_notebook_iframe()

# profile<jupyter_output><empty_output><jupyter_text>Initial CleaningNote that 12 of 32 columns are all missing exactly 34,480 values, 28.8% of the data: 'max_MOB',  'total_bounce',  'monthly_payment',  'loan_amount',  'installments',  'dealer',  'product',  'advance_payments',  'int_rate',  'sex',  'employment',   'DOB',  'age' Interestingly 'residence' is missing 35,397 values, 29.6%, due to the location of the column it is likely missing values in the same rows as above. To address this these rows will be dropped and a comparison made to the initial ratios for the target (defaulters/nondefaulters). The result is that these rows are permanently dropped.<jupyter_code># Create a copy of the dataframe, so downloading only has to be preformed once
df2=df1.copy()
df2.shape
# Drop the one instance of the product RETOP
df2 = df2[df2['product'] != 'RETOP']
df2.shape
"""Note that 12 of 32 columns are all missing exactly 34,480 values, 28.8% of the data:
 'max_MOB', 
 'total_bounce', 
 'monthly_payment', 
 'loan_amount', 
 'installments', 
 'dealer', 
 'product', 
 'advance_payments', 
 'int_rate', 
 'sex', 
 'employment', 
  'DOB', 
 'age'
Interestingly 'residence' is missing 35,397 values, 29.6%, due to the location
of the column it is likely missing values in the same rows as above.

To address this these rows will be dropped and a comparison made to the
initial ratios for the target (defaulters/nondefaulters).

"""
df2.dropna(subset=['monthly_payment'], inplace=True)
df2.shape

# The result is that these rows are permanently dropped
"""Compare Maximum 2 Wheekler Loan, 12.6% missing values to
Loan Amount"""
dfx=df2[['max_loan','loan_amount','defaulter']]
dfx['2_percent']=dfx['max_loan']/dfx['loan_amount']
dfx.describe()
# Fill missing max_loan values with loan_amount, ratio is mostly 1
df2['max_loan'].fillna(df2['loan_amount'], inplace=True)
print(df2['max_loan'].mean())
# Compare data to verifty ratios are similar
print(df1['defaulter'].value_counts(normalize=True)) #original 
print(df2['defaulter'].value_counts(normalize=True))
#Examine the data again
# profile = ProfileReport(df2, minimal=True).to_notebook_iframe()

# profile
df2.info()
# Wrangle residence column, so as not to unneccisarily lose good data when
# dropping rows to achieve balance, 917 missing, 1.1%
df2['residence'] = df2['residence'].replace({np.nan:'UNKNOWN'})

df2['residence'].value_counts()
df = df2.copy() #copy dataframe, a good place to "run after"<jupyter_output><empty_output><jupyter_text>Train, Validate, Test Split<jupyter_code>from sklearn.model_selection import train_test_split
# Train / Test Split, stratified, highly imbalanced data

# split off test set
train1, test1 = train_test_split(df, test_size=0.2, stratify=df['defaulter'], random_state=42)

#split validation set out of the training data
train2, val2 = train_test_split(train1, test_size=0.2, stratify=train1['defaulter'], random_state=42)

train1.shape, test1.shape, train2.shape, val2.shape
train2.info() # note that 5 columns contain null values
# profile = ProfileReport(train2, minimal=True).to_notebook_iframe()

# profile
# Get ratio of defaulters for train, it matches full data set
train2['defaulter'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Imports for Modeling<jupyter_code># Imports for pipelines
import shap
from shap import TreeExplainer
import category_encoders as ce
import eli5
from eli5.sklearn import PermutationImportance
from numpy import sqrt
from numpy import argmax
from sklearn.impute import SimpleImputer
# from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
# from sklearn.model_selection  import RandomizedSearchCV
from sklearn.metrics import confusion_matrix
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import classification_report
from sklearn.metrics import  f1_score, roc_auc_score, roc_curve, precision_score, recall_score, accuracy_score
from sklearn.datasets import make_classification
from sklearn.metrics import precision_recall_curve
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Run After<jupyter_code>#Create fresh copies of all of the data
test = test1.copy()
train = train2.copy()
val = val2.copy()
train_down = train2.copy() #this will be a downsampled version of train

train_big = train1.copy() #to be used with cross validation
worker = train.copy() #dataframe to test wrangle on individual features<jupyter_output><empty_output><jupyter_text>Pre-DownsamplingThis data is very imbalanced. One way to achieve balance while keeping the best data is to drop incomplete rows. However, many of the columns with incomplete data may be dropped in data wrangling. This provides the most complete data for the model.A second copy of the train data will be made, so that testing can be done on both. There are five numerical columns with an high percentage of missing values    total_credit_max, 64.7%   secured_max, 81.5%   unsecured_max, 81.6%    months_last_loan, 87.0%   months_first_loan, 80.3%<jupyter_code>"""Drop rows with missing data from the majority class to downsample
# does not work well to wrangle with columns at the same time"""

  # Print out shape info before wrangling
print('Before Wrangling:')
print('Train:', train.shape)
print('Train Downsampled:', train_down.shape)
print(train_down['defaulter'].value_counts())
print('---'*20)

def wrangle_rows(X):
    # copy to avoid errors
    X=X.copy()

    not_default = X[X['defaulter']==0]
    default = X[X['defaulter']==1]

    """data is still too imbalanced, drop additional columns based on missing 
    data, dropping all columns wth missing data makes a nearly perfect balance"""
    not_default = not_default.dropna(thresh = (32))
    X = pd.concat([not_default,default])

    return X

train_down = wrangle_rows(train_down)
# train_big = wrangle_rows(train_big)

print('After Wrangling')
print('Train Downsampled:', train_down.shape)
print('Downsampled Defaulters:')
train_down['defaulter'].value_counts()

# Dowsampled baselines
train_down['defaulter'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Wrangle Data<jupyter_code># Columns to drop - 3
goodbye = [
           'ID', #all unique
           'new_loans', #all zero
           'num_loans',  #sum of loans, will magnify outliers from num secured loans
          ]

# Data about the loan - 6 
personal_loan = [
          'loan_amount',
          'monthly_payment',
          'installments',
          'advance_payments',
          'int_rate',
          'miss_1st' # this loan was already pre-sold, presume it's been at least one month
]

# Future data - 6
future=[
        'max_MOB',
        'annual_bounce',
        'total_bounce',
        'late_30_6',
        'late_60_6',
        'late_90_3'
]

# Data on current 2 wheeler loan - 3
loan_info = [
          'dealer',
          'product',
          'max_loan'
]

# Columns about the borrower - 7
person_col = [
          'sex',
          'employment',
          'residence'
          'age',
          'num_unsecured_loans',
          'num_secured_loans', #skewed
          'location',

          ]

# Date to modify - 1
to_date =['DOB']

# Columns with missing data - 5
missing_to_drop =[
          'totaL_credit_max', #includes loan amount
          'secured_max',
          'months_last_loan',
          'unsecured_max',
          'months_first_loan'
          ]
# Create function to wrangle data sets

def wrangle(X):
    # copy to avoid errors
    X=X.copy()

    # Change DOB column to year only
    for col in to_date:
      X[col] = pd.to_datetime(X[col],infer_datetime_format=True)
      X['YOB'] =X[col].dt.year
    
    X = X.drop(columns = to_date) #replaced by YOB
    # X = X.drop(columns = ['age']) #replaced by YOB

    # Drop extraneous columns & suspected future columns
    X = X.drop(columns = goodbye)
    X = X.drop(columns = future)
    # X = X.drop(columns = personal_loan)


    # Drop columns with high % missing
    X = X.drop(columns = missing_to_drop)
    
    # Return wrangled dataframe
    return X
    
train = wrangle(train)
train_down = wrangle(train_down)
val = wrangle(val)
test = wrangle(test)
train_big = wrangle(train_big)

print('After Wrangling')
print('Train:', train.shape)
print('Validation:',val.shape)
print('Test:', test.shape)
print('Train & Val:', train_big.shape)
print('Train Downsampled:', train_down.shape)
# # Inspect new columns
# profile = ProfileReport(train, minimal=True).to_notebook_iframe()

# profile<jupyter_output><empty_output><jupyter_text>Run After 2 Target Vector / Features Matrix Split<jupyter_code># Target, is the client a defaulter
target = 'defaulter'

# Features
features = list(train.drop(columns = [target]))
features
# Arrange data into X features matrix and y target vector 
X_train = train[features]
y_train = train[target]
X_val = val[features]
y_val = val[target]
X_test = test[features]
y_test = test[target]
X_train_big = train_big[features]
y_train_big = train_big[target]

X_down = train_down[features]
y_down = train_down[target]
# Print ratios for ease of reference
y_train.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Transformations<jupyter_code># Encode catagorical features on X_train
encoder = ce.OrdinalEncoder()
X_train_e = encoder.fit_transform(X_train)
X_down_e = encoder.transform(X_down)
X_val_e = encoder.transform(X_val)
X_test_e = encoder.transform(X_test)
# scale data - logistic regression
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train_e)
X_down_s = encoder.transform(X_down_e)
X_val_s = scaler.transform(X_val_e)
X_test_s = scaler.transform(X_test_e)<jupyter_output><empty_output><jupyter_text>SMOTE<jupyter_code># Balance data using SMOTE
from imblearn.over_sampling import SMOTE

smote = SMOTE(ratio='minority')
X_smote, y_smote = smote.fit_sample(X_train_s, y_train)
X_smote.shape<jupyter_output>Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.<jupyter_text>Tracking Dataframe<jupyter_code># Create dataframe to track outcomes
columns= ['Model','Paid', 'True -', 'False +', 'False -','True +','Default']
tracker= pd.DataFrame(columns=columns)
tracker
# Function to add rows to tracking dataframe
# m_name = Model Name

def row_maker(m_name, y, pred):
  tn, fp, fn, tp = confusion_matrix(y, pred).ravel()
  recall = round((tp/(tp+fn)),2)
  n_recall = round((tn/(tn+fp)),2)
  new_row = {'Model':m_name,'True -':tn, 'False +':fp, 'False -':fn,'True +':tp ,'Paid':n_recall,'Default':recall }
  return new_row<jupyter_output><empty_output><jupyter_text>XGBoost  XG Boost Simple Model<jupyter_code># First XGBoost Model

from xgboost import XGBClassifier
booster= XGBClassifier(n_estimators=100, random_state=42, n_jobs=-1)

# Fit on train, score on val
booster.fit(X_train_e, y_train)

y_pred_v = booster.predict(X_val_e)

print('Classification Report:\n\n', classification_report(y_val, y_pred_v))
# Complete tracking dataframe

m_name = 'XG Boost'
y = y_val
pred = y_pred_v

new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output><empty_output><jupyter_text>XG Boost Simple Model with PRC Curve Not Useful<jupyter_code># Create Precision Recall Curve to find best threshold
# for smiple model
y_hat_v = booster.predict_proba(X_val_e)

# Keep only positive outcome probabilies (defaulters)
y_hat_v=y_hat_v[:,1]

# Precision Recall Curve
precision, recall, thresholds = precision_recall_curve(y_val,y_hat_v)

# Calculate f score
fscore = (2*precision*recall)/(precision + recall)
fscore=np.nan_to_num(fscore) #remove Nan(divide by 0)

# index of larges f score (best result)
ix = argmax(fscore)

best=thresholds[ix]

print('Best Threshold = ',best)
print('F-Score =', fscore[ix])

# Plot precision recall curve
plt.figure(figsize=(2,4))
no_skill = len(y_val[y_val==1])/len(y_val)

# plot no skill line
plt.plot([0,1], [no_skill, no_skill], linestyle ='--', label="No Skill")

# plot PRC
plt.plot(recall, precision, marker='.', label='RF')

# plot best threshold point
plt.scatter(recall[ix], precision[ix], marker='o', color='black', label='Best')
# label axes
plt.xlabel('Recall')
plt.ylabel('Precison')
plt.legend();

# creat prediction with new threshold
y_pred_v_new = (booster.predict_proba(X_val_e)[:,1]>=best).astype(int)

print('XG Boost Downsampled with PRC')
print('---'*20)
# print classification report
print('Classification Report:\n\n', classification_report(y_val, y_pred_v_new))<jupyter_output>XG Boost Downsampled with PRC
------------------------------------------------------------
Classification Report:

               precision    recall  f1-score   support

           0       0.98      0.93      0.95     13311
           1       0.05      0.18      0.08       297

    accuracy                           0.91     13608
   macro avg       0.52      0.55      0.52     13608
weighted avg       0.96      0.91      0.93     13608<jupyter_text>XG Boost Downsampled Data<jupyter_code># XGBoost Model with downsampled data
# (does not work with smote)
m_name = 'XG Boost Down'
from xgboost import XGBClassifier
xgb= XGBClassifier(n_estimators=100, random_state=42, n_jobs=-1)

# Fit on train, score on val
xgb.fit(X_down_e, y_down)

y_pred_v = xgb.predict(X_val_e)

print('Classification Report:\n\n', classification_report(y_val, y_pred_v))

# Complete tracking dataframe

m_name = m_name = 'XG Boost Down'
y = y_val
pred = y_pred_v

new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output><empty_output><jupyter_text>XG Boost SMOTEDOES NOT WORK<jupyter_code># # XGBoost Model with SMOTE
# # Does not work without much data modification
# m_name = '<NAME>'
# from xgboost import XGBClassifier
# xgb= XGBClassifier(n_estimators=100, random_state=42, n_jobs=-1)

# # Fit on train, score on val
# xgb.fit(X_smote, y_smote)

# predictions = xgb.predict(test_data_process.values)  

# y_pred_v = xgb.predict(X_val_e)


# print('Classification Report:\n\n', classification_report(y_val, y_pred_v))<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code># Logistic Regression
# define basic model
logreg = LogisticRegression(solver='lbfgs', n_jobs=-1, random_state=42)<jupyter_output><empty_output><jupyter_text>Simple Logistic Regression<jupyter_code># Fit simple model on train
lr_simple= logreg.fit(X_train_s, y_train)

# Predict
y_pred_v = lr_simple.predict(X_val_s)
y_pred_test = lr_simple.predict(X_test_s)

m_name = 'Simple Logistic Regression'
y = y_val
pred = y_pred_v

print('Classification Report:\n\n', classification_report(y, pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Classification Report:

               precision    recall  f1-score   support

           0       0.98      1.00      0.99     13311
           1       0.00      0.00      0.00       297

    accuracy                           0.98     13608
   macro avg       0.49      0.50      0.49     13608
weighted avg       0.96      0.98      0.97     13608<jupyter_text>Simple Logistic Regression Precision Recall Curve<jupyter_code>m_name= '<NAME>'
model = lr_simple
x = X_val_s
y = y_val

# This was supposed to be a function, but it stopped working
# Create Precision Recall Curve to find best threshold 
# standard data 
y_hat_v = model.predict_proba(x)

# Keep only positive outcome probabilies (defaulters)
y_hat_v=y_hat_v[:,1]

# Precision Recall Curve
precision, recall, thresholds = precision_recall_curve(y,y_hat_v)

# Calculate f score
fscore = (2*precision*recall)/(precision + recall)
fscore=np.nan_to_num(fscore) #remove Nan(divide by 0)

# index of largest f score (best result)
ix = argmax(fscore)
best =thresholds[ix]
my_fscore= fscore[ix]

new_pred = (model.predict_proba(x)[:,1]>=best).astype(int)

pred = new_pred

print('Best Threshold = ',best)
print('F-Score =', my_fscore)
print('---'*20)
print('Classification Report:\n\n', classification_report(y,new_pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Best Threshold =  0.03096337916808546
F-Score = 0.06830052229811169
------------------------------------------------------------
Classification Report:

               precision    recall  f1-score   support

           0       0.98      0.84      0.91     13311
           1       0.04      0.29      0.07       297

    accuracy                           0.83     13608
   macro avg       0.51      0.56      0.49     13608
weighted avg       0.96      0.83      0.89     13608<jupyter_text>Logistic Regression - Downsampled Data<jupyter_code># Logistic Regression with downsampled data
# Fit on downsampled train
lr_model = logreg.fit(X_down_s, y_down)

# Predict on downsampled data
y_pred_v = lr_model.predict(X_val_s)
y_pred_test = lr_model.predict(X_test_s)

m_name = 'Logistic Regression Down'
y = y_val
pred = y_pred_v

# print classification report
print('Classification Report:\n\n', classification_report(y, pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Classification Report:

               precision    recall  f1-score   support

           0       0.98      0.26      0.41     13311
           1       0.02      0.79      0.05       297

    accuracy                           0.27     13608
   macro avg       0.50      0.53      0.23     13608
weighted avg       0.96      0.27      0.40     13608<jupyter_text>ROC Curve Downsampled Data<jupyter_code>from numpy import sqrt
m_name='Logistic Regression Down ROC'
model = lr_model
x = X_val_s
y = y_val

# Create ROC Curve 
y_hat_v = model.predict_proba(x)

# Keep only positive outcome probabilies (defaulters)
y_hat_v=y_hat_v[:,1]

# Calculate ROC curve
fpr, tpr, thresholds = roc_curve(y_val,y_hat_v)

# calculate the g-mean for each threshold
gmeans = sqrt(tpr * (1-fpr))

# locate the index of the largest g-mean
ix = argmax(gmeans)

best=thresholds[ix]

# Plot ROC
plt.figure(figsize=(4,8))

# plot no skill line
plt.plot([0,1], [0,1], linestyle ='--', label="No Skill")

# plot roc curve
plt.plot(fpr, tpr, marker='.', label='Logistic')

# plot best threshold point
plt.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# label axes
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend();

print('Best Threshold = ',best)
#Logistic Regression Downsampled ROC
m_name= 'Logistic Regression  ROC'
model = lr_model
x = X_val_s
y = y_val

# create prediction with new threshold
y_pred_v = (model.predict_proba(x)[:,1]>=best).astype(int)

pred = y_pred_v

print('Classification Report:\n\n', classification_report(y,pred))

# Complete tracking datafram
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Classification Report:

               precision    recall  f1-score   support

           0       0.98      0.51      0.67     13311
           1       0.02      0.55      0.05       297

    accuracy                           0.51     13608
   macro avg       0.50      0.53      0.36     13608
weighted avg       0.96      0.51      0.65     13608<jupyter_text>Logistic Regression SMOTE<jupyter_code># Logistic Regression SMOTE
my_lr_model = logreg.fit(X_smote, y_smote)

# Predict on downsampled data
y_pred_v = my_lr_model.predict(X_val_s)
y_pred_test = my_lr_model.predict(X_test_s)
#Logistic Regression SMOTE
m_name= 'Logistic Regression SMOTE '
model = my_lr_model
x = X_val_s
y = y_val
pred = y_pred_v

print('---'*20)
print('Classification Report:\n\n', classification_report(y,pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker
#Logistic Regression SMOTE - Test Data
m_name= 'Logistic Regression SMOTE - Test Data '
model = my_lr_model
x = X_test_s
y = y_test
pred = y_pred_test

print('Classification Report:\n\n', classification_report(y,pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker
# Plot confusion matrix
print('Logisitic Regression with SMOTE')
print('---'*20)

plot_confusion_matrix(my_lr_model, X_test_s, y_test, values_format='.0f');<jupyter_output>Logisitic Regression with SMOTE
------------------------------------------------------------<jupyter_text>Logistic Regression ROC SMOTEMakes minor difference<jupyter_code># ROC SMOTE - Validation

from numpy import sqrt
# Create ROC Curve 
y_hat_v = my_lr_model.predict_proba(X_val_s)

# Keep only positive outcome probabilies (defaulters)
y_hat_v=y_hat_v[:,1]

# Calculate ROC curve
fpr, tpr, thresholds = roc_curve(y_val,y_hat_v)

# calculate the g-mean for each threshold
gmeans = sqrt(tpr * (1-fpr))

# locate the index of the largest g-mean
ix = argmax(gmeans)
best=thresholds[ix]

# Plot ROC
plt.figure(figsize=(4,8))

# plot no skill line
plt.plot([0,1], [0,1], linestyle ='--', label="No Skill")

# plot roc curve
plt.plot(fpr, tpr, marker='.', label='Logistic')

# plot best threshold point
plt.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# label axes
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend();

print('Best Threshold = ',best)

#Logistic Regression SMOTE
m_name= 'Logistic Regression SMOTE ROC'
model = my_lr_model
x = X_val_s
y = y_val

# create prediction with new threshold
y_pred_v = (my_lr_model.predict_proba(x)[:,1]>=best).astype(int)

pred = y_pred_v

print('Classification Report:\n\n', classification_report(y,pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker
# Logistic Regression SMOTE ROC - Test Data

# Create ROC Curve 
y_hat_v = my_lr_model.predict_proba(X_test_s)

# Keep only positive outcome probabilies (defaulters)
y_hat_v=y_hat_v[:,1]

# Calculate ROC curve
fpr, tpr, thresholds = roc_curve(y_test,y_hat_v)

# calculate the g-mean for each threshold
gmeans = sqrt(tpr * (1-fpr))

# locate the index of the largest g-mean
ix = argmax(gmeans)

best=thresholds[ix]

# Plot ROC
plt.figure(figsize=(4,8))

# plot no skill line
plt.plot([0,1], [0,1], linestyle ='--', label="No Skill")

# plot roc curve
plt.plot(fpr, tpr, marker='.', label='Logistic')

# plot best threshold point
plt.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# label axes
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend();

print('Best Threshold = ',best)

m_name= 'Logistic Regression SMOTE ROC - Test Data'
model = my_lr_model
x = X_test_s
y = y_test

# create prediction with new threshold
y_pred_test = (my_lr_model.predict_proba(x)[:,1]>=best).astype(int)

pred = y_pred_test

print('Classification Report:\n\n', classification_report(y,pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker
# Plot confusion matrix
print('       Logisitic Regression SMOTE ROC - Test Data')


plot_confusion_matrix(my_lr_model, X_test_s, y_test, values_format='.0f', display_labels = ['Non-Defaulter', 'Defaulter'], cmap=plt.cm.Blues);<jupyter_output>Logisitic Regression SMOTE ROC - Test Data<jupyter_text>Random Forest<jupyter_code># not necessary with ordinal encoder
# encoder2=ce.OneHotEncoder()
# X_train_e2=encoder2.fit_transform(X_train)
# X_down_e2 = encoder2.transform(X_down)
# X_val_e2 = encoder2.transform(X_val)
# X_test_e2 = encoder2.transform(X_test)
# X_train_big_e2 = encoder2.transform(X_train_big)
# X_train_big_e2.shape, X_test_e2.shape<jupyter_output><empty_output><jupyter_text>Random Forest Simple Model<jupyter_code># Random Forest Simple Model
# Define model
rf_simple = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)

# Fit on train
rf_simple.fit(X_train_e, y_train)

# Predict on data
y_pred_v = rf_simple.predict(X_val_e)
y_pred_d = rf_simple.predict(X_down_e)
y_pred_test = rf_simple.predict(X_test_e)

m_name = '<NAME>'
y = y_val
pred = y_pred_v

print('Classification Report:\n\n', classification_report(y, pred))

# Complete tracking dataframe

new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Classification Report:

               precision    recall  f1-score   support

           0       0.98      1.00      0.99     13311
           1       0.00      0.00      0.00       297

    accuracy                           0.98     13608
   macro avg       0.49      0.50      0.49     13608
weighted avg       0.96      0.98      0.97     13608<jupyter_text>Random Forest Using Downsampled Data<jupyter_code>rf_model =RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)

# Fit on train
rf_model.fit(X_down_e, y_down)

# Predict on data
y_pred_t = rf_model.predict(X_down_e)
y_pred_v = rf_model.predict(X_val_e)

m_name = '<NAME>'
y = y_val
pred = y_pred_v

print('Classification Report:\n\n', classification_report(y, pred))

# Complete tracking dataframe
new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Classification Report:

               precision    recall  f1-score   support

           0       0.98      0.11      0.19     13311
           1       0.02      0.89      0.04       297

    accuracy                           0.12     13608
   macro avg       0.50      0.50      0.12     13608
weighted avg       0.96      0.12      0.19     13608<jupyter_text>Permutation ImportancesConcluded that all features were likely important in all models<jupyter_code># Use eli5 to create permuter
permuter = PermutationImportance(
    rf_model, #as defined above
    scoring = 'f1',
    n_iter=5, #number of random shuffles
    random_state=42
)
permuter.fit(X_down_e, y_down)
feature_names = X_train_e.columns.tolist()
feature_names

# Run permuter
pd.Series(permuter.feature_importances_, feature_names)

# Display in a useful way with eli5
eli5.show_weights(
    permuter,
    top=None, #no limit, show all features
    feature_names= feature_names
)
print('Shape before removing features:', X_train_e.shape)

#Remove features less than 0 because they slow down the model and don't improve performance
minimum_importance = 0
mask = permuter.feature_importances_ > minimum_importance #apply conditional statement to all elements of array
mask
new_features = X_train_e.columns[mask] #Select columns where mask is true (permutation score > 0)
X_trainx = X_train_e[new_features] #Save new X_train

print('Shape after removing features:', X_trainx.shape)<jupyter_output>Shape before removing features: (54429, 17)
Shape after removing features: (54429, 17)<jupyter_text>Random Forest Using SMOTE Data<jupyter_code># Random Forest Model SMOTE Data
# Tuning does not help
smote_rf_model =RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
# Fit on train
smote_rf_model.fit(X_smote,y_smote)

# Predict on data
y_pred_v = smote_rf_model.predict(X_val_e)
y_pred_test = smote_rf_model.predict(X_test_e)

print('Classification Report:\n\n', classification_report(y_val, y_pred_v))

# Complete tracking dataframe
m_name='Random Forest SMOTE'
y = y_val
pred = y_pred_v

new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker
# Random Forest Model SMOTE Data - Test
print('Classification Report:\n\n', classification_report(y_test, y_pred_test))

# Complete tracking dataframe
m_name='Random Forest SMOTE - Test Data'
y = y_test
pred = y_pred_test

new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Classification Report:

               precision    recall  f1-score   support

           0       0.98      0.16      0.28     16639
           1       0.02      0.87      0.04       371

    accuracy                           0.18     17010
   macro avg       0.50      0.52      0.16     17010
weighted avg       0.96      0.18      0.27     17010<jupyter_text>Random Forest SMOTE with ROC<jupyter_code># Validation Data

m_name='RF SMOTE ROC'
# Create ROC Curve 
y_hat_v = smote_rf_model.predict_proba(X_val_e)

y_hat_v=y_hat_v[:,1] # Keep only positive outcome probabilies (defaulters)

fpr, tpr, thresholds = roc_curve(y_val,y_hat_v) # Calculate ROC curve

gmeans = sqrt(tpr * (1-fpr))# calculate the g-mean for each threshold

ix = argmax(gmeans) # locate the index of the largest g-mean
best=thresholds[ix]

print('Best Threshold = ',best)

smote_rf_best = best

# create prediction with new threshold
y_pred_v_new = (smote_rf_model.predict_proba(X_val_e)[:,1]>=best).astype(int)

print('Classification Report:\n\n', classification_report(y_val, y_pred_v_new))

# Complete tracking dataframe
m_name= 'Random Forest SMOTE ROC'
y = y_val
pred = y_pred_v_new

new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker
# RF SMOTE ROC - Test Data
from numpy import sqrt
m_name='RF SMOTE ROC'
# Create ROC Curve 
y_hat_v = smote_rf_model.predict_proba(X_test_e2)

y_hat_v=y_hat_v[:,1] # Keep only positive outcome probabilies (defaulters)

fpr, tpr, thresholds = roc_curve(y_test,y_hat_v) # Calculate ROC curve

gmeans = sqrt(tpr * (1-fpr))# calculate the g-mean for each threshold

ix = argmax(gmeans) # locate the index of the largest g-mean
best=thresholds[ix]

print('Best Threshold = ',best)
best_rf = best

smote_rf_best = best# create prediction with new threshold - Test Data
y_pred_v_new_test = (smote_rf_model.predict_proba(X_test_e)[:,1]>=best).astype(int)

print('Classification Report:\n\n', classification_report(y_test, y_pred_v_new_test))

# Complete tracking dataframe
m_name= 'Random Forest SMOTE ROC - Test'
y = y_test
pred = y_pred_v_new_test

new_row=row_maker(m_name, y, pred)
tracker = tracker.append(new_row, ignore_index=True)
tracker<jupyter_output>Best Threshold =  0.67
Classification Report:

               precision    recall  f1-score   support

           0       0.98      0.58      0.73     16639
           1       0.03      0.49      0.05       371

    accuracy                           0.58     17010
   macro avg       0.50      0.53      0.39     17010
weighted avg       0.96      0.58      0.71     17010<jupyter_text>Shapley ValuesUse Random Forest with SMOTE, it is a poor model, but it is what I can successfully use with Shapley<jupyter_code># Use shapley values to examine why the model made this prediction
explainer = shap.TreeExplainer(smote_rf_model) # for tree based models
# Create dataframe with encoded data
X_test_final = pd.DataFrame(X_test_e,columns=X_test.columns)
# verify that columns match for training & test data
X_smote.shape,X_test_final.shape
#list of predicted values
p_list= smote_rf_model.predict(X_test_e[0:]) 
p_list.shape
# Generate a list of incorrect predictions
wrong_list =[]
for i in range(0,17009):
  row = X_test.iloc[[i]]
  my_a_row = y_test.iloc[[i]].item()
  if p_list[i] != my_a_row:
    wrong_list.append(i)
print(len(wrong_list))
# Generate list of defaulters
# error appeared in final run-through, a.item suggestion does not work
naughty_list=[]
for i in range(0,17009):
  row2 = y_test.iloc[[i]].item()
  if row2==1:
    naughty_list.append(i)
len(naughty_list)
print(len(naughty_list))
naughty_list[0:19] #print 20 values
# Find defaulters predicted incorrectly
my_set = set(naughty_list) & set(wrong_list)
sneaky_list= list(sorted(my_set))
print(len(sneaky_list))
sneaky_list
# Select an observation to test, FN:99, TP:453, TN:3
i=0
row = X_test_e.iloc[[i]] #select a row to create a Shapley Plot
a_row = y_test.iloc[[i]].values.tolist() # find the actual value
p_row = smote_rf_model.predict(row)#find the predicted value
pp_row = smote_rf_model.predict_proba(row) #see the probability that this would be the choice
print('Actual:', a_row)
print('Predicted:', p_row)
print('Probility of Prediction:', pp_row)
# Shapley Plot
print('True Value:', a_row,'  ','Predicted Value:', p_row)
shap_values = explainer.shap_values(row) #two arrays, one for each outcome
feature_names = row.columns

feature_values = row.values[0] #an array of the values in the selected row  
shap.initjs() #initialize Javascript
shap.force_plot(
    base_value = explainer.expected_value[p_row], # baseline value for this model predicted value
    # base_value = base_value, #ROC adjusted baseline
    shap_values = shap_values[(p_row).item()], # predicted value for this instance
    features=feature_names
)<jupyter_output>True Value: [0]    Predicted Value: [1]
{'fraction_non_alphanumeric': 0.11075463050335661, 'fraction_numerical': 0.03178668679733628, 'mean_word_length': 4.2657580919931855, 'pattern_counts': {'":': 0, '<': 114, '<?xml version=': 0, '>': 130, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '411440', 'license': 'MIT', 'n_tokens_mistral': 13790, 'n_tokens_neox': 12872, 'n_words': 3402, 'path': 'module4-model-interpretation/SW_DSTP7_Unit_2_Build.ipynb', 'repo_name': 'SaraWestWA/DS-Unit-2-Linear-Models', 'size': '37149'}