<jupyter_start><jupyter_text>ML Project - Predicting Bad Cases - Sprint 2    19-04-23 15:19    <NAME>    Credit Team IntroductionThe aim of this project is to attempt to predict bad cases using the Lender data. The deliverables of this project are not only to predict the bad cases themselves, but also to see which features are most important in making these predictions. In the previous sprint 1, we achieved an accuracy of 96% and a recall of 11%. The main objective of sprint 2 is to boost the recall value as much as possible, to be able to provide more meaningful results.Bad cases in this project are considered to be any loan reference that is in arrears of over 30 days.The Lender Portal download will be used as our dataset for the time being. High level aims of the project are to then take the results of the predictors and use them to help fine tune the Decision Engine. Exploratory Data AnalysisThe goal of this section is to gain an understanding of our data in order to inform what we do in the feature engineering section.We begin our exploratory data analysis by loading our standard modules. We then proceed to do some initial tidying of our data before looking at the features one by one.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

import time
import ipaddress
from datetime import date
import datetime as dt
from pandas.tseries.offsets import BMonthEnd
import gc
import os

from imblearn.over_sampling import SMOTE

from sklearn.feature_selection import VarianceThreshold
from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold, cross_val_predict
from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import confusion_matrix, recall_score, precision_score, accuracy_score, roc_curve
from sklearn.utils.multiclass import unique_labels<jupyter_output>C:\Users\<NAME>\Anaconda3\lib\site-packages\sklearn\ensemble\weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d<jupyter_text>We then load the data, which we have downloaded from duoBI.<jupyter_code>lender_data_raw = pd.read_csv('input/Lender_Raw_Data.csv')
print("Raw data shape: " + str(lender_data_raw.shape))

# Only select Activated, Accepted and Signed cases
print("Removed " + str(len(lender_data_raw[~((lender_data_raw['Status.1'] == 'ACTIVATED') | (lender_data_raw['Status.1'] == 'ACCEPTED') | (lender_data_raw['Status.1'] == 'SIGNED'))])) + " records that were not activated, accepted or signed.")
lender_data = lender_data_raw[(lender_data_raw['Status.1'] == 'ACTIVATED') | (lender_data_raw['Status.1'] == 'ACCEPTED') | (lender_data_raw['Status.1'] == 'SIGNED')]
lender_data.shape<jupyter_output><empty_output><jupyter_text>Next we need to upload a Multi-tier dataset to match records by.<jupyter_code>mtier_data = pd.read_csv('input/mtier 19-04-12.csv')
lender_data['mtier'] = pd.Series(lender_data['LoanRef'].isin(mtier_data['AgreementRef']).astype(int))
print(str(sum(lender_data['mtier'])) + " m-tier records")<jupyter_output>1136 m-tier records<jupyter_text>We then remove all records that are too young to have arrears. As payments are made monthly, the minimum arrears possible is one month, meaning any application younger than a month is pointless.<jupyter_code>print("Removed " + str(sum((date.today() - pd.to_datetime(lender_data['CreatedAt'], format='%Y-%m-%d %H:%M:%S').dt.date) < "91 days")) + " records under 91 days old")
lender_data = lender_data[~((date.today() - pd.to_datetime(lender_data['CreatedAt'], format='%Y-%m-%d %H:%M:%S').dt.date) < "91 days")]<jupyter_output>Removed 9158 records under 91 days old<jupyter_text>Now we've done our initial data work, we upload our arrears data to get labels from. This is the full historic dataset of all loans gone bad at each month end, so includes duplicated Loan References (for loans that have remained bad over the months). We'll remove duplicate loan references, retaining records with the bad case label. We'll also make sure to save the arrears dataset WITH duplicates for future use in the oversampling section.<jupyter_code># Import Arrears data
arrears_data = pd.read_csv('input/arrears_history.csv',
                           #engine = 'python', # Only need to use this for the skipfooter arg
                           skiprows = 1,
                           #skipfooter = 3,
                           thousands = ',', # Recognise thousand separator
                           decimal = '.', # Recognise decimal point
                           usecols = [3,7,8,9,10,11,12],
                           names = ['LoanRef','1-30','31-60','61-90','91-180','181-270','271-365']
                          )
def map_bad_case(row):
    """
    Map the arrears columns to our bad case labels
    """
    if row['31-60'] > 12:
        val = 1
    elif row['61-90'] > 12:
        val = 1
    elif row['91-180'] > 12:
        val = 1
    elif row['181-270'] > 12:
        val = 1
    elif row['271-365'] > 12:
        val = 1
    else:
        val = 0
    return val
# Map arrears data to bad case label
arrears_data['bad_case'] = arrears_data.apply(map_bad_case, axis = 1)
# Subset arrears_data to only bad cases
arrears_data_with_dupes = arrears_data[arrears_data['bad_case'] == 1]
# Subset arrears_data to just LoanRef and labels, and save 
arrears_data_no_dupes = arrears_data_with_dupes.drop_duplicates('LoanRef')
arrears_data_no_dupes = arrears_data_no_dupes.iloc[:, [0,7]]

print("Number of arrears bad cases: " + str(len(arrears_data_no_dupes)) + " (Unique records)")
df = lender_data.merge(arrears_data_no_dupes, on='LoanRef', how='left')
df['bad_case'].fillna(0, inplace = True)
print("arrears data bad cases: " + str(len(arrears_data_no_dupes)))

print("df records: " + str(len(df)))
print("df bad cases: " + str(sum(df['bad_case'])) + "  //  " + str(sum(df['bad_case'])/len(df)))
# Save the datasets so far
df.to_csv('df_before_feature_engineering.csv')
arrears_data_no_dupes.to_csv('final_arrears_data.csv')<jupyter_output><empty_output><jupyter_text>First, let's take a look at the shape and summary of all the data. Immediately, we note that several features have nulls that we'll have to deal with.<jupyter_code>df.shape
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 22003 entries, 0 to 22002
Data columns (total 92 columns):
ApplicationID                         22003 non-null object
ActiveCAIS3Applicant                  21826 non-null object
ActiveCAIS3AssociateID                10197 non-null object
ApplicantinIVA                        21844 non-null object
BankClosedWarning                     21898 non-null object
BankEnhancedScore                     0 non-null float64
BankruptcyInformationApplicant        21844 non-null float64
BankruptcyInformationAssociate        10844 non-null float64
CAISSpecialInstructionSet             21774 non-null object
ConfirmatoryDobs                      21889 non-null float64
ConfirmedOnVotersRoll                 21845 non-null object
CreditScore                           21845 non-null float64
DeceasedWarning                       21897 non-null object
DefaultCAISApplicant                  21844 non-null object
DefaultCAISAssociate                  2530 non-nul[...]<jupyter_text>It appears that we can drop all the ID features as these will be useless*. We've removed all empty features too. We've also removed the Status columns, as we've already filtered them to only show Activated records, as well as any other features that are rendered void since this filter. **We've kept the LoanRef field just for later use in oversampling.*<jupyter_code>cols_remove = ['BankEnhancedScore','DeviceRisk','HasactivePaydayLoan','Householdincome','JudgementApplicant', 
               'JudgementAssociate','NumberofBankruptcies','MerchantSector','Nationality','ApplicationID',
               'Status', 'Status.1','StatusBriefing','MerchantID','DeclinedAt','InfoNeededAt',
                'DepositPaidAt', 'FinanceID', 'Reference', 'ActivatedAt', 'SignedAt', 'LoanRef',
               'AwaitingActivationAt','AddedToWatchlist','PGVariable','AddedToWatchlist','Comments'
              ]
df = df.drop(cols_remove, 1)<jupyter_output><empty_output><jupyter_text>Bad CasesWe need to add labels to our dataset for the Bad Cases. A 1 will signify a Bad Case, and a 0 will signify all others.<jupyter_code>df.bad_case.value_counts(normalize=True)
sns.countplot(df.bad_case)
plt.show()<jupyter_output><empty_output><jupyter_text>As expected, the data is really skewed towards the non-bad cases. We'll need to figure out the best way to fix this.   ActiveCAIS3ApplicantThis variable is in part numerical. We'll need to map the Default (D) and Unknown (U) cases into numerical values. Looks like there is some useful information to be extracted here in terms of bad cases - higher numbers of D values have bad cases than others. We'll map D = 10, and U = 0.<jupyter_code>print('bad_case summarised for each ActiveCAIS3Applicant')
df.bad_case.groupby(df.ActiveCAIS3Applicant).agg(['mean', 'count'])<jupyter_output>bad_case summarised for each ActiveCAIS3Applicant<jupyter_text>NaNs will be considered U, as as far as I'm concerned they are the same thing.<jupyter_code>df['ActiveCAIS3Applicant'].fillna('U', inplace = True)
# Replace 'D' values with 10
df.loc[df.ActiveCAIS3Applicant == 'D', 'ActiveCAIS3Applicant'] = 10
# Replace 'U' values with 0
df.loc[df.ActiveCAIS3Applicant == 'U', 'ActiveCAIS3Applicant'] = 0
# Make col numeric
df['ActiveCAIS3Applicant'] = pd.to_numeric(df['ActiveCAIS3Applicant'])
sns.countplot(df['ActiveCAIS3Applicant'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>ActiveCAIS3AssociateIDSimilar to ActiveCAIS3Applicant - we'll need to map the Default (D) and Unknown (U) cases into 10 and 0.<jupyter_code>print('bad_case summarised for each ActiveCAIS3AssociateID')
df.bad_case.groupby(df.ActiveCAIS3AssociateID).agg(['mean', 'count'])<jupyter_output>bad_case summarised for each ActiveCAIS3AssociateID<jupyter_text>NaNs will again be considered U.<jupyter_code>df['ActiveCAIS3AssociateID'].fillna('U', inplace = True)
# Replace 'D' values with 10
df.loc[df.ActiveCAIS3AssociateID == 'D', 'ActiveCAIS3AssociateID'] = 10
# Replace 'U' values with 0
df.loc[df.ActiveCAIS3AssociateID == 'U', 'ActiveCAIS3AssociateID'] = 0
# Make col numeric
df['ActiveCAIS3AssociateID'] = pd.to_numeric(df['ActiveCAIS3AssociateID'])
sns.countplot(df['ActiveCAIS3AssociateID'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>ApplicantinIVAThis feature doesn't seem to be too useful given that all records but 116 are False, and there are currently no True values. We'll get rid of this one.<jupyter_code>df.bad_case.groupby(df.ApplicantinIVA).agg(['mean', 'count'])
print('Count of NaNs: ',df['ApplicantinIVA'].isna().sum())
cols_remove_2 = ['ApplicantinIVA']<jupyter_output><empty_output><jupyter_text>BankClosedWarningSparse feature - we'll get rid of this.<jupyter_code>df.bad_case.groupby(df.BankClosedWarning).agg(['mean', 'count'])
print('Count of NaNs: ',df['BankClosedWarning'].isna().sum())
cols_remove_2.append('BankClosedWarning')<jupyter_output><empty_output><jupyter_text>BankruptcyInformationApplicantSparse feature - makes sense as we are very unlikely to be lending to people who have had bankrupcies. We'll get rid of this feature.<jupyter_code>df.bad_case.groupby(df.BankruptcyInformationApplicant).agg(['mean', 'count'])
print('Count of NaNs: ',df['BankruptcyInformationApplicant'].isna().sum())
cols_remove_2.append('BankruptcyInformationApplicant')<jupyter_output><empty_output><jupyter_text>BankruptcyInformationAssociateThis is an interesting variable - There is potential use in keeping this feature as there are so many more bad cases for NaN records that for others. We will engineer the feature to be `is_BankruptcyInformationAssociate` and map to 0 = False, 1 = Unknown, 2 = True.<jupyter_code>df['bad_case'].groupby(df['BankruptcyInformationAssociate'].isnull()).mean()
df.bad_case.groupby(df.BankruptcyInformationAssociate).agg(['mean', 'count'])
sns.countplot(df['BankruptcyInformationAssociate'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>CAISSpecialInstructionSetSparse feature - we'll get rid of this.<jupyter_code>df.bad_case.groupby(df.CAISSpecialInstructionSet).agg(['mean', 'count'])
print('Count of NaNs: ',df['CAISSpecialInstructionSet'].isna().sum())
cols_remove_2.append('CAISSpecialInstructionSet')<jupyter_output><empty_output><jupyter_text>ConfirmatoryDobsQuite normally distributed, and a noticeable amount of bad cases for NaN values. We will make a null_flag feature for this.<jupyter_code>sns.countplot(df['ConfirmatoryDobs'], hue = df['bad_case'])
df.bad_case.groupby(df.ConfirmatoryDobs).agg(['mean', 'count'])
df['bad_case'].groupby(df['ConfirmatoryDobs'].isnull()).agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>ConfirmedOnVotersRollThis feature seems overall pretty useless - we'll get rid of this one.<jupyter_code>df.bad_case.groupby(df.ConfirmedOnVotersRoll).agg(['mean', 'count'])
print('Count of NaNs: ',df['ConfirmedOnVotersRoll'].isna().sum())
cols_remove_2.append('ConfirmedOnVotersRoll')<jupyter_output><empty_output><jupyter_text>CreditScoreA really useful indicator. There are 115 nulls for CreditScore, and they interestingly have a lower bad case rate than the non-nulls. Before imputing values for the nulls, we will include a CreditScoreNull flag just to make sure we can account for this characteristic of the data.<jupyter_code>print('Count of NaNs: ',df['CreditScore'].isnull().sum())
df['bad_case'].groupby(df['CreditScore'].isnull()).agg(['mean', 'count'])
df['bad_case'].groupby(pd.qcut(df['CreditScore'],5)).mean()
pd.qcut(df['CreditScore'],5).value_counts()<jupyter_output><empty_output><jupyter_text>DeceasedWarningSparse feature - we'll remove this.<jupyter_code>df.bad_case.groupby(df.DeceasedWarning).agg(['mean', 'count'])
print('Count of NaNs: ',df['DeceasedWarning'].isna().sum())
cols_remove_2.append('DeceasedWarning')<jupyter_output><empty_output><jupyter_text>DefaultCAISApplicantThis is a useful feature - NaNs and U values mapped to 0 and D mapped to 10.<jupyter_code>df.bad_case.groupby(df.DefaultCAISApplicant).agg(['mean', 'count'])
print('Count of NaNs: ',df['DefaultCAISApplicant'].isna().sum())
df['DefaultCAISApplicant'].fillna('U', inplace = True)
# Replace 'D' values with 10
df.loc[df.DefaultCAISApplicant == 'D', 'DefaultCAISApplicant'] = 10
# Replace 'U' values with 0
df.loc[df.DefaultCAISApplicant == 'U', 'DefaultCAISApplicant'] = 0
# Make col numeric
df['DefaultCAISApplicant'] = pd.to_numeric(df['DefaultCAISApplicant'])
sns.countplot(df['DefaultCAISApplicant'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>DefaultCAISAssociateThere's too little data in this feature. We'll remove this one.<jupyter_code>df.bad_case.groupby(df.DefaultCAISAssociate).agg(['mean', 'count'])
print('Count of NaNs: ',df['DefaultCAISAssociate'].isna().sum())
cols_remove_2.append('DefaultCAISAssociate')<jupyter_output><empty_output><jupyter_text>EmailCreatedIf this feature was less sparse, we could create a useful feature out of it - such as how long the email has been created for. But we're going to get rid of it as it is too sparse.<jupyter_code>print('Count of NaNs: ',df['EmailCreated'].isna().sum())
cols_remove_2.append('EmailCreated')<jupyter_output><empty_output><jupyter_text>EmailDomainStatusIsActiveSome useful stuff here - we'll keep this variable. NaNs have been mapped to Unknown. We'll create an ordinal scale here for Active = 2, Unknown = 1 and Inactive = 0.<jupyter_code>df['EmailDomainStatusIsActive'].fillna('Unknown', inplace = True)
df.bad_case.groupby(df.EmailDomainStatusIsActive).agg(['mean', 'count'])
sns.countplot(df['EmailDomainStatusIsActive'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>EmailStatusisActiveWe've mapped Expired to Inactive and NaNs to Unknown. We'll create an ordinal scale for this feature where Active = 2, Unknown = 1 and Inactive = 0.<jupyter_code>df['EmailStatusisActive'].fillna('Unknown', inplace = True)
df.loc[df.EmailStatusisActive == 'Expired', 'EmailStatusisActive'] = 'Inactive'
df.bad_case.groupby(df.EmailStatusisActive).agg(['mean', 'count'])
sns.countplot(df['EmailStatusisActive'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>EmailageScoreLooks like it could be a really useful indicator I think - we'll divide the values by ten and round just so it scales better. There are some nulls in there that we'll need to impute, but we should definitely create a null_flag for the feature.<jupyter_code>print('Count of NaNs: ',df['EmailageScore'].isna().sum())
df['bad_case'].groupby(df['EmailageScore'].isnull()).agg(['mean', 'count'])
df['bad_case'].groupby(pd.qcut(df['EmailageScore'],5)).mean()
pd.qcut(df['EmailageScore'],5).value_counts()<jupyter_output><empty_output><jupyter_text>EmploymentStatusA really useful feature. We'll one hot encode this.<jupyter_code>df.bad_case.groupby(df.EmploymentStatus).agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>GenderCould be useful, we'll one-hot encode this feature.<jupyter_code>df.bad_case.groupby(df.Gender).agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>GrossIncomeObviously will be a useful feature. We'll log transform this for better performance.`fcc_survey_df['Income_log'] = np.log((1+ fcc_survey_df['Income'])``income_log_mean = np.round(np.mean(fcc_survey_df['Income_log']), 2)fig, ax = plt.subplots()fcc_survey_df['Income_log'].hist(bins=30, color='A9C5D3',                                  edgecolor='black', grid=False)plt.axvline(income_log_mean, color='r')ax.set_title('Developer Income Histogram after Log Transform',              fontsize=12)ax.set_xlabel('Developer Income (log scale)', fontsize=12)ax.set_ylabel('Frequency', fontsize=12)ax.text(11.5, 450, r'$\mu$='+str(income_log_mean), fontsize=10)`<jupyter_code>print('Count of NaNs: ',df['GrossIncome'].isna().sum())
df['bad_case'].groupby(pd.qcut(df['GrossIncome'],5)).mean()
pd.qcut(df['GrossIncome'],5).value_counts()<jupyter_output><empty_output><jupyter_text>HasappliedforPaydayLoanSparse feature - we'll get rid of this.<jupyter_code>df.bad_case.groupby(df.HasappliedforPaydayLoan).agg(['mean', 'count'])
cols_remove_2.append('HasappliedforPaydayLoan')<jupyter_output><empty_output><jupyter_text>HasCCJWe'll map the NaNs to False for now, I'm pretty sure we'd know if there was a CCJ. This may be really bad practise, but we'll see what happens.<jupyter_code>df['HasCCJ'].fillna(False, inplace = True)
df.bad_case.groupby(df.HasCCJ).agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>IncomeConfidenceFactorThis feature is too sparse - we'll have to get rid of it.<jupyter_code>print('Count of NaNs: ',df['IncomeConfidenceFactor'].isna().sum())
df.bad_case.groupby(df.IncomeConfidenceFactor).agg(['mean', 'count'])
cols_remove_2.append('IncomeConfidenceFactor')<jupyter_output><empty_output><jupyter_text>IncomeDebtRatiosWould be a really useful feature if we had more data. We'll get rid of this one.<jupyter_code>print('Count of NaNs: ',df['IncomeDebtRatios'].isna().sum())
cols_remove_2.append('IncomeDebtRatios')<jupyter_output><empty_output><jupyter_text>IncomeMaxIDAgain too sparse. We'll remove this feature.<jupyter_code>print('Count of NaNs: ',df['IncomeMaxID'].isna().sum())
cols_remove_2.append('IncomeMaxID')<jupyter_output><empty_output><jupyter_text>IndebtednessScoreAgain too sparse. We'll remove this feature.<jupyter_code>print('Count of NaNs: ',df['IndebtednessScore'].isna().sum())
cols_remove_2.append('IndebtednessScore')<jupyter_output><empty_output><jupyter_text>IPAddressIf we knew a bit more about networking we might be able to extract more from this feature. We can see there is something interesting happening where two IP addresses are being recorded for some records. So we'll map the IP Address to validated or non_validated. Non validated IPs will be mapped to 0 and validated to 1.A potential thing to do in future is extract the class of IP address too.<jupyter_code>non_validated_idx = [] # For the indexes of non validated IP Addresses
non_validated_val = [] # For the values of non validated IP Addresses

for idx, val in enumerate(df['IPAddress']):
    try:
        ipaddress.ip_address(val)
    except:
        non_validated_idx.append(idx)
        non_validated_val.append(val)

print(' Number of non-validated IP Addresses: ', len(non_validated_val))
df.iloc[non_validated_idx].bad_case.agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>MatchLevelThis feature is quite sparse, keeping it in will probably add to overfitting. We'll exclude it.<jupyter_code>df.bad_case.groupby(df.MatchLevel).agg(['mean', 'count'])
cols_remove_2.append('MatchLevel')<jupyter_output><empty_output><jupyter_text>MobileRiskNumStatusCould be a potentially useful feature. We'll map this to ACTIVE = 2, Unknown = 1, DISCONNECTED = 0.<jupyter_code>df['MobileRiskNumStatus'].fillna('Unknown', inplace = True)
df.bad_case.groupby(df.MobileRiskNumStatus).agg(['mean', 'count'])
sns.countplot(df['MobileRiskNumStatus'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>MobileRiskPhoneTypeThis feature might be useful, although we can't really see any specific relationship with the values and bad cases. The decision tree might be able to find more useful relationships.<jupyter_code>df['MobileRiskPhoneType'].fillna(0, inplace = True)
df.bad_case.groupby(df.MobileRiskPhoneType).agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>MobileRiskRegLocationSparse feature - we'll remove this.<jupyter_code>df.bad_case.groupby(df.MobileRiskRegLocation).agg(['mean', 'count'])
cols_remove_2.append('MobileRiskRegLocation')<jupyter_output><empty_output><jupyter_text>MobileRiskScoreThis could be a useful feature, but the values are not distributed well. We'll round the score to the nearest 100.<jupyter_code>df['MobileRiskScore'].fillna(0, inplace = True)
df.bad_case.groupby(df.MobileRiskScore).agg(['mean', 'count']).head()<jupyter_output><empty_output><jupyter_text>NumberofActiveCIFASRecordsThis could be a useful feature - but we'll transform it to a new feature `ActiveCifasRecords` with a value of 1 or zero denoting if there are CIFAS records or not.<jupyter_code>print('Count of NaNs: ',df['NumberofActiveCIFASRecords'].isna().sum())
df.bad_case.groupby(df.NumberofActiveCIFASRecords).agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>NumberOfActivePaydayLoansSparse feature - we'll remove this.<jupyter_code>df.bad_case.groupby(df.NumberOfActivePaydayLoans).agg(['mean', 'count'])
cols_remove_2.append('NumberOfActivePaydayLoans')<jupyter_output><empty_output><jupyter_text>NumberofNoticesofCorrectionSparse feature - we'll remove this too.<jupyter_code>df.bad_case.groupby(df.NumberofNoticesofCorrection).agg(['mean', 'count'])
cols_remove_2.append('NumberofNoticesofCorrection')<jupyter_output><empty_output><jupyter_text>NumberofNoticesofDisputeSparse feature - we'll remove this.<jupyter_code>df.bad_case.groupby(df.NumberofNoticesofDispute).agg(['mean', 'count'])
cols_remove_2.append('NumberofNoticesofDispute')<jupyter_output><empty_output><jupyter_text>OccupancyStatusLooks like this feature could be really useful. We'll one-hot encode this.<jupyter_code>df.bad_case.groupby(df.OccupancyStatus).agg(['mean', 'count'])
sns.countplot(df['OccupancyStatus'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>PepWarningSparse feature - we'll remove this.<jupyter_code>print('Count of NaNs: ',df['PepWarning'].isna().sum())
df.bad_case.groupby(df.PepWarning).agg(['mean', 'count'])
cols_remove_2.append('PepWarning')<jupyter_output><empty_output><jupyter_text>PercentageOfTotalPaymentsSparse feature - we'll remove this.<jupyter_code>print('Count of NaNs: ',df['PercentageOfTotalPayments'].isna().sum())
df.bad_case.groupby(df.PercentageOfTotalPayments).agg(['mean', 'count'])
cols_remove_2.append('PercentageOfTotalPayments')<jupyter_output><empty_output><jupyter_text>PrimaryChecksReturnedThis could be a useful feature, maybe consider any rounding or general transforming though if prone to overfitting.<jupyter_code>print('Count of NaNs: ',df['PrimaryChecksReturned'].isna().sum())
df['PrimaryChecksReturned'].fillna(0, inplace = True)
df.bad_case.groupby(df.PrimaryChecksReturned).agg(['mean', 'count']).head()<jupyter_output><empty_output><jupyter_text>RatioOfIncomeChangeOverThreeMonthsSparse feature - we'll remove this.<jupyter_code>print('Count of NaNs: ',df['RatioOfIncomeChangeOverThreeMonths'].isna().sum())
cols_remove_2.append('RatioOfIncomeChangeOverThreeMonths')<jupyter_output><empty_output><jupyter_text>SanctionsWarningSparse feature - we'll remove this.<jupyter_code>print('Count of NaNs: ',df['SanctionsWarning'].isna().sum())
df.bad_case.groupby(df.SanctionsWarning).agg(['mean', 'count'])
cols_remove_2.append('SanctionsWarning')<jupyter_output><empty_output><jupyter_text>SDNWarningSparse feature - we'll remove this.<jupyter_code>print('Count of NaNs: ',df['SDNWarning'].isna().sum())
df.bad_case.groupby(df.SDNWarning).agg(['mean', 'count'])
cols_remove_2.append('SDNWarning')<jupyter_output><empty_output><jupyter_text>MerchantNameThere are a lot of Merchants, and varying counts for each. We want to be able to capture some information here, but if we one-hot encode every Merchant this would definitely lead to over fitting (and be super inefficient computationally). Merchants like Adaptation Station LTD have high bad case rate, but only because there are 3 records in total. Simba Sleep on the other hand has a high bad case rate and a lot of records - we want to capture this distinction. I think for now let's just take a handful of the most active merchants to one-hot encode and leave the rest. We might want to come back to this feature, but between this one and the sector feature (which we need to add ourselves) most of the information should be captured.<jupyter_code>df.bad_case.groupby(df.MerchantName).agg(['mean', 'count']).sort_values(by=['count'], ascending=False).head(10)
print('Number of Merchants: ',len(df.MerchantName.unique()))<jupyter_output>Number of Merchants:  237<jupyter_text>CountrySparse feature - get rid of this.<jupyter_code>df.bad_case.groupby(df.Country).agg(['mean', 'count'])
cols_remove_2.append('Country')<jupyter_output><empty_output><jupyter_text>CurrencySparse feature - get rid of this.<jupyter_code>df.bad_case.groupby(df.Currency).agg(['mean', 'count'])
cols_remove_2.append('Currency')<jupyter_output><empty_output><jupyter_text>CreatedAtThis could be a useful feature - we'll transform the datetime into an 'app_age' feature (minusing the CreatedDate from the Current Date).We'll map to new features: - `day_of_week` - `is_weekend` - `day_of_month` - `day_of_year` - `month_of_year` - `year` - `absolute_time` - `hour_of_day` - `minute_of_day`Other features we'll extract are: - `is_holiday` - `is_payday`<jupyter_code># Convert string to datetime obj
df['CreatedAt'] = pd.to_datetime(df['CreatedAt'], format='%Y-%m-%d %H:%M:%S')
df['CreatedDate'] = df['CreatedAt'].dt.date<jupyter_output><empty_output><jupyter_text>AcceptedAtWe'll extract a new feature called `days_between_create_accept` that shows the number of days between app creation and app acceptance.<jupyter_code># Convert string to datetime obj
df['AcceptedAt'] = pd.to_datetime(df['AcceptedAt'], format = '%Y-%m-%d %H:%M:%S')<jupyter_output><empty_output><jupyter_text>ChannelWe'll remove this feature in favour of the `ChannelType` feature.<jupyter_code>df.bad_case.groupby(df.Channel).agg(['mean', 'count'])
cols_remove_2.append('Channel')<jupyter_output><empty_output><jupyter_text>ChannelTypeWe'll remove this.<jupyter_code>cols_remove_2.append('ChannelType')<jupyter_output><empty_output><jupyter_text>FinanceDescriptionThis is a complex feature to work with. For now we will extract into new features: - Term - BNPL - Interest Rate - Subsidised<jupyter_code># Just take the unique finance descriptions
finances = pd.Series(df['FinanceDescription'].unique()).sort_values().reset_index(drop=True)
print(str(len(finances)) + " unique finance descriptions")
finances.values<jupyter_output><empty_output><jupyter_text>LenderSparse feature - we'll get rid of this.<jupyter_code>cols_remove_2.append('Lender')<jupyter_output><empty_output><jupyter_text>CreditAmountObviously will be a useful feature. We'll log transform this for better performance.`fcc_survey_df['Income_log'] = np.log((1+ fcc_survey_df['Income'])``income_log_mean = np.round(np.mean(fcc_survey_df['Income_log']), 2)``fig, ax = plt.subplots()fcc_survey_df['Income_log'].hist(bins=30, color='A9C5D3',                                  edgecolor='black', grid=False)plt.axvline(income_log_mean, color='r')ax.set_title('Developer Income Histogram after Log Transform',              fontsize=12)ax.set_xlabel('Developer Income (log scale)', fontsize=12)ax.set_ylabel('Frequency', fontsize=12)ax.text(11.5, 450, r'$\mu$='+str(income_log_mean), fontsize=10)`<jupyter_code>df['bad_case'].groupby(pd.qcut(df['CreditAmount'],5)).mean()
pd.qcut(df['CreditAmount'],5).value_counts()<jupyter_output><empty_output><jupyter_text>DepositAmountAnother useful feature - we'll log transform this and also create a new feature of `depost_credit_ratio`  DepositStatusNot sure if this one is useful beyond the DepositAmount. We'll just extract a new feature called `no_deposit` to mark the ones where there was none.<jupyter_code>df.bad_case.groupby(df.DepositStatus).agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>CustomerNameFor this feature we will transform it to remove whitespace and punctuation. Then we'll extract a feature that shows the number of characters in the customer name. This may or may not prove useful...<jupyter_code>print('Count of NaNs: ',df['CustomerName'].isna().sum())<jupyter_output>Count of NaNs:  0<jupyter_text>DOBWe'll minus this date from the current date to create a new 'age' feature, then remove this one.<jupyter_code># Convert string to datetime obj
df['DOB'] = pd.to_datetime(df['DOB'], format='%Y-%m-%d')
df['DOB'] = df['DOB'].dt.date<jupyter_output><empty_output><jupyter_text>EmailSimilar to CustomerName, we'll extract the length of the email address to a new feature. But first we will split the email address into name and domain. The length will be computed on the name part of the string, and we'll map the domain to another feature.<jupyter_code>print('Count of NaNs: ',df['Email'].isna().sum())<jupyter_output>Count of NaNs:  0<jupyter_text>PhoneNumberThis feature doesn't tell us an awful lot - we'll get rid of it.<jupyter_code>df['PhoneNumber'].head(10)
cols_remove_2.append('PhoneNumber')<jupyter_output><empty_output><jupyter_text>BuildingNumberWe'll extract whether the record is numeric or not in an attempt to remove flat names, street names and errors then map all these records to 0, and adding a new feature `is_numeric_building_num` (this will also include NaNs). That way we have a numeric feature that might be useful.<jupyter_code>print('Count of NaNs: ',df['BuildingNumber'].isna().sum())
not_building_num_idx = [] # For the indexes of non numeric building numbers
not_building_num_val = [] # For the values of non numeric building numbers

for idx, val in enumerate(df['BuildingNumber']):
    try:
        if val.isnumeric():
            continue
        else:
            not_building_num_idx.append(idx)
            not_building_num_val.append(val)
    except:
        not_building_num_idx.append(idx)
        not_building_num_val.append(val)

print('Number of non-numeric Building Numbers: ',len(not_building_num_val))
print(not_building_num_val[45:55])
print('Non-numerical building numbers')
df.iloc[not_building_num_idx].bad_case.agg(['mean', 'count'])
print('Numerical building numbers')
not_building_num_df = df.index.isin(not_building_num_idx)
df[~not_building_num_df].bad_case.agg(['mean', 'count'])<jupyter_output>Numerical building numbers<jupyter_text>StreetWe'll get rid of this feature in favour of a looked up region feature.<jupyter_code>cols_remove_2.append('Street')<jupyter_output><empty_output><jupyter_text>CityWe'll get rid of this feature in favour of a looked up region feature.<jupyter_code>cols_remove_2.append('City')<jupyter_output><empty_output><jupyter_text>PostcodeWe map the geographies to region instead so we can prevent overfitting. WE'll then one-hot encode the region.<jupyter_code>pcd_lookup = pd.read_csv('input/pcd_lookup.csv')
pcd_lookup.head()
df['Postcode'] = df.Postcode.str.replace(' ', '') # Remove whitespace from Postcode for lookup
df.rename(columns = {'Postcode':'pcd_no_space'}, inplace=True)

df = df.merge(pcd_lookup, 'left')
df['reg_name'].fillna('Unknown', inplace = True)
df.bad_case.groupby(df.reg_name).agg(['mean', 'count'])
sns.countplot(df['reg_name'], hue = df['bad_case'])
cols_remove_2.append('pcd_no_space')<jupyter_output><empty_output><jupyter_text>IsDuplicatedSparse feature - we'll get rid of this.<jupyter_code>df.bad_case.groupby(df.IsDuplicated).agg(['mean', 'count'])
cols_remove_2.append('IsDuplicated')<jupyter_output><empty_output><jupyter_text>Multi-TierWe don't have to change this variable.<jupyter_code>sns.countplot(df['mtier'], hue = df['bad_case'])<jupyter_output><empty_output><jupyter_text>Feature EngineeringHaving done our cursory exploration of the variables, we now have a pretty good idea of how we want to transform our variables in preparation for our final dataset. We will perform our feature engineering through a series of helper functions that each serve a specific purpose.We'll begin by removing the columns we found were not helpful to our algorithm.<jupyter_code># Remove columns
df = df.drop(cols_remove_2, 1)
# BankruptcyInformationAssociate
    
def transform_bankruptcy_information_associate(df):
    """"
    Map BankruptcyInformationAssociate feature so shows 2 if 
    there are Bancrupcies, 1 if Unknown and 0 if there are none.
    Then remove original BankruptcyInformationAssociate feature.
    """
    df['is_bankruptcy_information_associate'] = df['BankruptcyInformationAssociate'].apply(lambda x: 2 if x > 0 else(0 if x == 0 else 1))
    df = df.drop('BankruptcyInformationAssociate', 1)
    return df
# ConfirmatoryDobs

def transform_confirmatory_dobs(df):
    """
    Map ConfirmatoryDobs feature so shows 1 if is a 
    null record, and 0 if not. Then map NaNs to 0.
    """
    df['is_null_confirmatory_dobs'] = df['ConfirmatoryDobs'].apply(lambda x: 1 if pd.isnull(x) else 0)
    df['ConfirmatoryDobs'].fillna(0, inplace = True)
    return df
# CreditScore

def transform_credit_score(df):
    """"
    Log transform credit score, then remove original 
    CreditScore feature.
    """
    df['log_credit_score'] = np.log((1+ df['CreditScore']))
    df['log_credit_score'].fillna(2.8, inplace = True)
    df = df.drop('CreditScore', 1)
    return df
# EmailDomainStatusIsActive

def transform_email_domain_status_is_active(df):
    """
    Map EmailDomainStatusIsActive to 2 if Active, 
    1 if Unknown and 0 if Inactive.
    """
    df['EmailDomainStatusIsActive'] = df['EmailDomainStatusIsActive'].apply(lambda x: 2 if x == 'Active' else(1 if x == 'Unknown' else 0))
    return df
# EmailStatusisActive

def transform_email_status_is_active(df):
    """
    Map EmailStatusisActive to 2 if Active, 1
    if Unknown and 0 if Inactive.
    """
    df['EmailStatusisActive'] = df['EmailStatusisActive'].apply(lambda x: 2 if x == 'Active' else(1 if x == 'Unknown' else 0))
    return df
# EmailageScore

def impute_emailage_score(df):
    """
    Create null flag feature for EmailageScore. 
    Divide EmailageScore by 10 and round.
    Impute null values using the median.
    """
    df['is_null_emailage_score'] = df['EmailageScore'].apply(lambda x: 1 if pd.isnull(x) else 0)
    df['EmailageScore'] = round(df['EmailageScore'] / 10, 1)
    df['EmailageScore'].fillna(df['EmailageScore'].median(), inplace = True)
    return df
# GrossIncome

def transform_gross_income(df):
    """
    Log transform GrossIncome. Then impute null
    values using the median, remove GrossIncome.
    """
    df['log_gross_income'] = np.log((1+ df['GrossIncome']))
    df['log_gross_income'].fillna( np.log((df['GrossIncome'].median())), inplace = True)
    df = df.drop('GrossIncome', 1)
    return df
# IPAddress

def is_ip_validated(df):
    """
    Map records to show 1 if IP address is
    validated and 0 if not. Remove IPAddress feature.
    """
    df['is_ip_validated'] = pd.Series(df.index.values).apply(lambda x: 1 if x in non_validated_idx else 0)
    df = df.drop('IPAddress', 1)
    return df
# MobileRiskNumStatus

def transform_mobile_risk_num_status(df):
    """
    Map MobileRiskNumStatus to show 2 if Active, 
    1 if Unknown and 0 if Inactive.
    """
    df['MobileRiskNumStatus'] = df['MobileRiskNumStatus'].apply(lambda x: 2 if x == 'Active' else(1 if x == 'Unknown' else 0))
    return df
# MobileRiskScore

def transform_mobile_risk_score(df):
    """
    Divide MobileRiskScore by 100 and round.
    """
    df['MobileRiskScore'] = round(df['MobileRiskScore'] / 100)
    return df
# NumberofActiveCIFASRecords

def transform_is_active_cifas(df):
    """
    Map NumberofActiveCIFASRecords to show 1 if
    there are active Cifas records and 0 if there
    aren't. Also map another feature that shows 1
    if the NumberofActiveCIFASRecords was originally null.
    """
    df['is_null_number_of_active_cifas'] = df['NumberofActiveCIFASRecords'].apply(lambda x: 1 if pd.isnull(x) else 0)
    df['is_active_cifas'] = df['NumberofActiveCIFASRecords'].apply(lambda x: 1 if x > 0 else 0)
    df['NumberofActiveCIFASRecords'].fillna(0, inplace = True)
    return df
# MerchantName

top_merchants = pd.Series(df['MerchantName'].value_counts().head(10).index.values)

def is_top_merchant(df): 
    """
    Creates individaul encoded features for each
    of the 10 most active merchants. Then maps all
    records to a is_top_ten_merchant feature.
    """
    df['is_top_10_merchant'] = df['MerchantName'].apply(lambda x: 1 if x in top_merchants else 0)
    df = df.drop('MerchantName', 1)
    return df
# CreatedAt

def calculate_app_age(df):
    """
    
    """
    today = date.today()
    #df['app_age'] = df['CreatedDate'].apply(lambda x: today - x).astype(str).str[:-24].astype(float)
    
    df['day_of_week'] = df['CreatedAt'].dt.dayofweek
    df['is_weekend'] = df['CreatedAt'].dt.dayofweek.apply(lambda x: 1 if x > 4 else 0)
    df['day_of_month'] = df['CreatedAt'].dt.day
    #df['day_of_year'] = df['CreatedAt'].dt.dayofyear
    #df['month_of_year'] = df['CreatedAt'].dt.month
    #df['year'] = df['CreatedAt'].dt.year
    df['hour_of_day'] = df['CreatedAt'].dt.hour
    df['minute_of_day'] = df['CreatedAt'].dt.minute
    
    bmonth_ends = pd.Series(df['CreatedAt'].apply(lambda x: BMonthEnd().rollforward(x)))
    df['is_payday'] = df['CreatedDate'].apply(lambda x: 1 if x in bmonth_ends else 0)
    
    df = df.drop(['CreatedDate'], 1)
    return df
# AcceptedAt

def calculate_time_accept_create(df):
    """
    
    """
    df['time_between_accept_create'] = df['AcceptedAt'].apply(lambda x: time.mktime(x.timetuple())) - df['CreatedAt'].apply(lambda x: time.mktime(x.timetuple()))
    df['log_time_between_accept_create'] = np.log((1 + df['time_between_accept_create']))
    df['log_time_between_accept_create'].fillna( np.log((df['time_between_accept_create'].median())), inplace = True)
    df = df.drop(['AcceptedAt', 'CreatedAt', 'time_between_accept_create'], 1)
    return df
# FinanceDescription

def transform_finance_decription(df):
    """
    Extract Interest Rate, Term, Subsidised, and BNPL
    from string into new features.
    """
    # Remove BNPL
    df = df[~df['FinanceDescription'].str.contains("BNPL")]
    
    def extract_term(df):
        """
        Extract the loan term from string
        """
        term = []
        for i in df['FinanceDescription']:
            if i.split(' Mon', 1)[0][-2:] == 'PR':
                term.append(float(i.split(' 18', 1)[0][-2:]))
            elif (len(i.replace(" ", "").split('Mon')[0]) == 7 and '+' in i):
                term.append(float(i.replace(" ", "")[4:5]) + float(i.replace(" ", "")[6:7]))
                
            elif (len(i.replace(" ", "").split('Mon')[0]) == 8 and '+' in i):    
                term.append(float(i.replace(" ", "")[4:5]) + float(i.replace(" ", "")[6:8]))
                
            elif (len(i.replace(" ", "").split('Mon')[0]) == 9 and '+' in i):    
                term.append(float(i.replace(" ", "")[4:6]) + float(i.replace(" ", "")[7:9]))
                
            else:
                term.append(float(i.split(' Mon', 1)[0][-2:]))
        return term
    df['term'] = extract_term(df)
    
    def extract_apr(df):
        """
        Extract the interest from string
        """
        apr = []
        for i in df['FinanceDescription']:
            if " 0%" in i:
                apr.append(float(0))
            elif "s " in i.split('%', 1)[0][-4:]:
                apr.append(float(i.split('%', 1)[0][-2:]))
            elif "nths" in i.split('%', 1)[0][-4:]:
                # Missing BNPL APRs are replaced with the BNPL APR mean
                apr.append(float(sum([20.9,18.9,16.9])/len([20.9,18.9,16.9])))
            else:
                apr.append(float((i.split('%', 1)[0][-4:])))
        return apr
    df['apr'] = extract_apr(df)
    
    def extract_subsidised(df):
        """
        Extract whether the finance is 
        subsidised from string
        """
        is_subsidised_list = []
        for i in df['FinanceDescription']:
            if "Sub" in i:
                is_subsidised_list.append(1)
            elif "sub" in i:
                is_subsidised_list.append(1)
            else:
                is_subsidised_list.append(0)
        return is_subsidised_list
    df['is_subsidised'] = extract_subsidised(df)
    
    df = df.drop(['FinanceDescription'], 1)
    
    return df
# CreditAmount

def credit_amount_transform(df):
    """
    
    """
    df['log_credit_amount'] = np.log((1+ df['CreditAmount']))
    df['log_credit_amount'].fillna(2.8, inplace = True)
    return df
# DepositAmount

def transform_deposit_amount(df):
    """
    
    """
    df['log_deposit_amount'] = np.log((1+ df['DepositAmount']))
    df['log_deposit_amount'].fillna( np.log((df['DepositAmount'].median())), inplace = True)
    df['deposit_credit_ratio'] = df['DepositAmount'] / df['CreditAmount']
    df = df.drop('DepositAmount', 1)
    return df
# DepositStatus

def is_no_depoist(df):
    """
    
    """
    df['is_no_deposit'] = df['DepositStatus'].apply(lambda x: 1 if x == 'NO-DEPOSIT' else 0)
    df = df.drop('DepositStatus', 1)
    return df
# CustomerName

def calculate_name_len(df):
    """
    
    """
    df['customer_name_len'] = df['CustomerName'].apply(lambda x: len(x))
    df = df.drop('CustomerName', 1)
    return df
# DOB

def calculate_customer_age(df):
    """
    
    """
    today = date.today()
    df['customer_age'] = df['DOB'].apply(lambda x: today.year - x.year - ((today.month, today.day) < (x.month, x.day)))
    df = df.drop('DOB', 1)
    return df
# Email

def calculate_email_len(df):
    """
    
    """
    df['email_name_len'] = df['Email'].apply(lambda x: len(x.split('@', 1)[0]))
    df = df.drop('Email', 1)
    return df
# BuildingNumber

def is_building_name_numerical(df):
    """
    
    """
    df['BuildingNumber'] = df['BuildingNumber'].apply(lambda x: 0 if x in not_building_num_val else x).astype('float')
    return df
def dummies(df, columns = ['EmploymentStatus', 'Gender', 'HasCCJ', 'OccupancyStatus', 'ChannelType', 'reg_name']):
    for column in columns:
        df[column] = df[column].apply(lambda x: str(x))
        df = pd.concat((df, pd.get_dummies(df[column], prefix = column)), axis = 1)
        del df[column]
    return df<jupyter_output><empty_output><jupyter_text>Having built our helper functions, we can now execute them in order to build our dataset that will be used in the model<jupyter_code>df = transform_bankruptcy_information_associate(df)
df = transform_confirmatory_dobs(df)
df = transform_credit_score(df)
df = transform_email_domain_status_is_active(df)
df = transform_email_status_is_active(df)
df = impute_emailage_score(df)
df = transform_gross_income(df)
df = is_ip_validated(df)
df = transform_mobile_risk_num_status(df)
df = transform_mobile_risk_score(df)
df = transform_is_active_cifas(df)
df = is_top_merchant(df)
df = calculate_app_age(df)
df = calculate_time_accept_create(df)
df = transform_finance_decription(df)
df = credit_amount_transform(df)
df = transform_deposit_amount(df)
df = is_no_depoist(df)
df = calculate_name_len(df)
df = calculate_customer_age(df)
df = calculate_email_len(df)
df = is_building_name_numerical(df)
df = dummies(df)<jupyter_output>C:\Users\<NAME>\Anaconda3\lib\site-packages\ipykernel_launcher.py:8: RuntimeWarning: invalid value encountered in log
  
C:\Users\<NAME>\Anaconda3\lib\site-packages\ipykernel_launcher.py:31: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
C:\Users\<NAME>s\Anaconda3\lib\site-packages\ipykernel_launcher.py:49: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
C:\Users\<NAME>s\Anaconda3\lib\site-packages\ipykernel_launcher.py:65: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[[...]<jupyter_text>We can see that our final dataset has 91 columns, composed of our target column and 90 predictor variables. Although highly dimensional datasets can result in high variance, I think we should be fine here.<jupyter_code>print('Number of columns: ', len(df.columns))
print('Number of records: ', len(df))<jupyter_output>Number of columns:  78
Number of records:  21430<jupyter_text>Feature Selection & SamplingWe will employ two sampling techniques. The first will be undersampling good cases.The second methodology will be Synthetic Minority Oversampling Technique (SMOTE). SMOTE uses a K-Nearest Neighbour algorithm to synthetically add new data to an imbalanced dataset. We'll try this method to correct our dataset and hope that it helps the recall value.<jupyter_code>df_undersample = df.drop(df.query('bad_case == 0').sample(frac=.70).index)
X = df.loc[:, df.columns != 'bad_case']
y = df[['bad_case']] 
def rank_gauss(x):
    from scipy.special import erfinv
    N = x.shape[0]
    temp = x.argsort()
    rank_x = temp.argsort() / N
    rank_x -= rank_x.mean()
    rank_x *= 2
    efi_x = erfinv(rank_x)
    efi_x -= efi_x.mean()
    return efi_x
for i in X.columns:
    #print('Categorical: ',i)
    X[i] = rank_gauss(X[i].values)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state = 10)
gc.collect()
sm = SMOTE(random_state = 12)
X_train_res, y_train_res = sm.fit_sample(X_train, y_train)
print("Before SMOTE:")
print("---------------------")
print("X_train: " + str(len(X_train)) + " records")
print("y_train: " + str(len(y_train)) + " records")
print("X_test: " + str(len(X_test)) + " records")
print("y_test: " + str(len(y_test)) + " records")
print("After SMOTE:")
print("---------------------")
print("X_train: " + str(len(X_train_res)) + " records")
print("y_train: " + str(len(y_train_res)) + " records")
print("X_test: " + str(len(X_test)) + " records")
print("y_test: " + str(len(y_test)) + " records")<jupyter_output>After SMOTE:
---------------------
X_train: 28054 records
y_train: 28054 records
X_test: 6429 records
y_test: 6429 records<jupyter_text>Hyperparameter TuningWe will use grid search to identify the optimal parameters of our random forest model. On my Windows 8GB machine, the process took quite a while so should consider upgrading to Azure computing.<jupyter_code>from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import roc_auc_score
from keras.layers import Dropout, BatchNormalization
from keras.layers.advanced_activations import PReLU, LeakyReLU
from keras.optimizers import Adam
from sklearn.model_selection import KFold
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.models import load_model
import tensorflow as tf
from keras.callbacks import Callback
import logging
class IntervalEvaluation(Callback):
    def __init__(self, validation_data=(), interval=10):
        super(Callback, self).__init__()

        self.interval = interval
        self.X_val, self.y_val = validation_data

    def on_epoch_end(self, epoch, logs={}):
        y_pred = self.model.predict_proba(self.X_val, verbose=0)
        score = roc_auc_score(self.y_val, y_pred)
        
        logging.info("interval evaluation - epoch: {:d} - score: {:.6f}".format(epoch, score))
        print("interval evaluation - epoch: {:d} - score: {:.6f}".format(epoch, score))
        logs['val_auc'] = score
n_folds = 10
folds = KFold(n_splits=n_folds, shuffle=True, random_state=42)
sub_preds = np.zeros(X_test.shape[0])
oof_preds = np.zeros(X_train_res.shape[0])
for n_fold, (trn_idx, val_idx) in enumerate(folds.split(X_train_res)):
    trn_x, trn_y = X_train_res[trn_idx], y_train_res[trn_idx]
    val_x, val_y = X_train_res[val_idx], y_train_res[val_idx]
    earlystop = EarlyStopping(monitor='val_auc', min_delta=0, patience=3, verbose=0, mode='max')
    file_path = "fold " + str(n_fold+1) + " best_model.hdf5"
    check_point = ModelCheckpoint(file_path, monitor = "val_auc", verbose = 1, save_best_only = True, mode = "max")    
    print( 'Setting up neural network...' )
    nn = Sequential()
    nn.add(Dense(units = 400 , kernel_initializer = 'normal', input_dim = X_train_res.shape[1]))
    nn.add(PReLU())
    nn.add(Dropout(.5))
    nn.add(Dense(units = 160 , kernel_initializer = 'normal'))
    nn.add(PReLU())
    nn.add(BatchNormalization())
    nn.add(Dropout(.5))
    nn.add(Dense(units = 64 , kernel_initializer = 'normal'))
    nn.add(PReLU())
    nn.add(BatchNormalization())
    nn.add(Dropout(.5))
    nn.add(Dense(units = 26, kernel_initializer = 'normal'))
    nn.add(PReLU())
    nn.add(BatchNormalization())
    nn.add(Dropout(.5))
    nn.add(Dense(units = 12, kernel_initializer = 'normal'))
    nn.add(PReLU())
    nn.add(BatchNormalization())
    nn.add(Dropout(.5))
    nn.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
    nn.compile(loss='binary_crossentropy', optimizer='adam')
    
    print( 'Fitting neural network...' )
    ival = IntervalEvaluation(validation_data=(val_x, val_y), interval=10)
    nn.fit(trn_x, trn_y, validation_data = (val_x, val_y), epochs=20, verbose=0, callbacks=[ival, earlystop, check_point], batch_size=128)
    
    best_model = load_model(file_path)
    
    oof_preds[val_idx] = best_model.predict(val_x).flatten()
    
    print(roc_auc_score(val_y, oof_preds[val_idx]))
    
    
    print( 'Predicting...' )
    sub_preds += best_model.predict(X_test).flatten().clip(0,1) / folds.n_splits
    
    gc.collect()
print('FULL AUC: {}'.format(roc_auc_score(y_train_res, oof_preds)))
y_pred_train = best_model.predict(X_train_res).flatten().round()
y_true_train = y_train_res

print("###########################")
print("#### Training Results ####")
print("###########################")
print("")

print('Accuracy score', accuracy_score(y_true_train, y_pred_train))
print('Precision score: ', precision_score(y_true_train, y_pred_train))
print('Recall score: ', recall_score(y_true_train, y_pred_train))<jupyter_output>###########################
#### Training Results ####
###########################

Accuracy score 0.9820702930063449
Precision score:  0.9665378777425141
Recall score:  0.9987167605332573<jupyter_text>Let's take a brief look at our variable importance according to our random forest model. We can see that some of the original columns we predicted would be important in fact were, including gender, fare, and age. But we also see title, name length, and ticket length feature prominently, so we can pat ourselves on the back for creating such useful variables.<jupyter_code># Plot Confusion Matrix - VALIDATION RESULTS
def plot_confusion_matrix(y_true_test, y_pred_test, classes,
                          normalize = False,
                          title = None,
                          cmap = plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if not title:
        if normalize:
            title = 'Normalized confusion matrix'
        else:
            title = 'Confusion matrix, without normalization'

    # Compute confusion matrix
    cm = confusion_matrix(y_true_train, y_pred_crossval)
    # Only use the labels that appear in the data
    classes = unique_labels(y_true_train, y_pred_crossval)
    if normalize:
        cm = cm.astype('float') / cm.sum(axis = 1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    fig, ax = plt.subplots()
    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)
    ax.figure.colorbar(im, ax=ax)
    # We want to show all ticks...
    ax.set(xticks = np.arange(cm.shape[1]),
           yticks = np.arange(cm.shape[0]),
           # ... and label them with the respective list entries
           xticklabels = classes, yticklabels = classes,
           title = title,
           ylabel = 'True label',
           xlabel = 'Predicted label')

    # Rotate the tick labels and set their alignment.
    plt.setp(ax.get_xticklabels(), rotation=45, ha = "right",
             rotation_mode = "anchor")

    # Loop over data dimensions and create text annotations.
    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i in range(cm.shape[0]):
        for j in range(cm.shape[1]):
            ax.text(j, i, format(cm[i, j], fmt),
                    ha="center", va="center",
                    color="white" if cm[i, j] > thresh else "black")
    fig.tight_layout()
    print(classes)
    return ax


np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plot_confusion_matrix(y_train_res, y_pred_crossval, classes=y_true_train,
                      title='Confusion matrix, without normalization')

# Plot normalized confusion matrix
plot_confusion_matrix(y_train_res, y_pred_crossval, classes=y_true_train, normalize=True,
                      title='Normalized confusion matrix')
y_scores_crossval = gs.predict_proba(X_train_res)[:, 1]
# Plot ROC-AUC curve
def plot_roc_curve(fpr, tpr, label=None):
    """
    The ROC curve, modified from 
    Hands-On Machine learning with Scikit-Learn and TensorFlow; p.91
    """
    plt.figure(figsize=(8,8))
    plt.title('ROC Curve')
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'k--')
    plt.axis([-0.005, 1, 0, 1.005])
    plt.xticks(np.arange(0,1, 0.05), rotation=90)
    plt.xlabel("False Positive Rate")
    plt.ylabel("True Positive Rate (Recall)")
    plt.legend(loc='best')
    
fpr, tpr, auc_thresholds = roc_curve(y_true_train, y_scores_crossval)
#print(auc(fpr, tpr)) # AUC of ROC
plot_roc_curve(fpr, tpr, 'recall_optimized')<jupyter_output><empty_output><jupyter_text>Final Results (Test Set)<jupyter_code>y_pred_test = best_model.predict(X_test).flatten().round()
y_true_test = y_test
# Save results
df.to_csv('rf_dataset_final.csv', index=False)

rf_results_final = pd.concat([X_test.reset_index(drop = True),
                              pd.DataFrame(y_pred_test, columns = ['bad_case_pred']),
                              pd.DataFrame(y_true_test.values, columns = ['bad_case_true'])],
                             axis = 1)
pd.DataFrame.to_csv(rf_results_final, 'rf_results_final.csv', index = False)

feature_importances_final = pd.concat((pd.DataFrame(X_test.columns, columns = ['variable']), 
                                           pd.DataFrame(rf.feature_importances_, columns = ['importance'])), 
                                          axis = 1).sort_values(by = 'importance', ascending = False)
pd.DataFrame.to_csv(feature_importances_final, 'feature_importances_final.csv', index = False)
# Plot Confusion Matrix - TEST RESULTS
def plot_confusion_matrix(y_true_test, y_pred_test, classes,
                          normalize = False,
                          title = None,
                          cmap = plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if not title:
        if normalize:
            title = 'Normalized confusion matrix'
        else:
            title = 'Confusion matrix, without normalization'

    # Compute confusion matrix
    cm = confusion_matrix(y_true_test, y_pred_test)
    # Only use the labels that appear in the data
    classes = unique_labels(y_true_test, y_pred_test)
    if normalize:
        cm = cm.astype('float') / cm.sum(axis = 1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    fig, ax = plt.subplots()
    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)
    ax.figure.colorbar(im, ax=ax)
    # We want to show all ticks...
    ax.set(xticks = np.arange(cm.shape[1]),
           yticks = np.arange(cm.shape[0]),
           # ... and label them with the respective list entries
           xticklabels = classes, yticklabels = classes,
           title = title,
           ylabel = 'True label',
           xlabel = 'Predicted label')

    # Rotate the tick labels and set their alignment.
    plt.setp(ax.get_xticklabels(), rotation=45, ha = "right",
             rotation_mode = "anchor")

    # Loop over data dimensions and create text annotations.
    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i in range(cm.shape[0]):
        for j in range(cm.shape[1]):
            ax.text(j, i, format(cm[i, j], fmt),
                    ha="center", va="center",
                    color="white" if cm[i, j] > thresh else "black")
    fig.tight_layout()
    print(classes)
    return ax


np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plot_confusion_matrix(y_test, y_pred_test, classes=y_true_test,
                      title='Confusion matrix, without normalization')

# Plot normalized confusion matrix
plot_confusion_matrix(y_test, y_pred_test, classes=y_true_test, normalize=True,
                      title='Normalized confusion matrix')
print("###########################")
print("#### Testing Results ####")
print("###########################")
print("")

print('Accuracy score', accuracy_score(y_true_test, y_pred_test))
print('Precision score: ', precision_score(y_true_test, y_pred_test))
print('Recall score: ', recall_score(y_true_test, y_pred_test))
# Find the correlation coefficients and list alongside the feature importance. Export .csv
corr = pd.DataFrame(df.corr().loc[:,'bad_case'].to_frame()).reset_index()
corr.columns = ['variable', 'corr']

feature_importances_final.merge(corr).to_csv('feature_importances_final_with_corr.csv')
# Display feature importances with correlations 
pd.concat((pd.DataFrame(X_train.columns, columns = ['variable']), 
           pd.DataFrame(rf.feature_importances_, columns = ['importance'])),
          axis = 1).sort_values(by='importance', ascending = False)[:20]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1280541556475891, 'fraction_numerical': 0.013039934800325998, 'mean_word_length': 4.756798161623899, 'pattern_counts': {'":': 0, '<': 321, '<?xml version=': 0, '>': 328, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '81', 'dir': 'jupyter-structured-clean-dedup', 'id': '343206', 'license': 'Apache-2.0', 'n_tokens_mistral': 20346, 'n_tokens_neox': 19343, 'n_words': 4778, 'path': 'credit-card-default-prediction-nn.ipynb', 'repo_name': 'TheNerdyCat/credit-card-default-prediction', 'size': '60151'}