<jupyter_start><jupyter_text>Loan Data Set Exploration  Importing required libraries<jupyter_code>import pandas as pd
import numpy as np
import scipy.stats

import matplotlib
import matplotlib.pyplot as pp
import seaborn as sns

from string import ascii_letters
import pandas.plotting

from IPython import display
from ipywidgets import interact, widgets

from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
import joblib

%matplotlib inline
sns.set_theme(context='notebook', style='darkgrid')
loan = pd.read_csv('loan.csv', dtype={'Current Credit Balance': 'int64'})
loan.head()
loan.info()
loan.describe()<jupyter_output><empty_output><jupyter_text>Step 1: Cleaning the Dataframe  Credit Score: Dealing with missing values<jupyter_code>loan.isna().sum()<jupyter_output><empty_output><jupyter_text>* Credit score: 61676 missing entries, that's a significant proportion of the data. Dropping the rows is not a good option in this case. We will replace those with the mean credit score.<jupyter_code>loan['Credit Score'] = loan['Credit Score'].fillna(loan['Credit Score'].mean())

loan['Credit Score'].isna().sum()<jupyter_output><empty_output><jupyter_text>Years in current job* There are 11476 missing entries* We can change it to integer, by dropping the '+ years' part of the entry and any other invalud string<jupyter_code>loan['Years in current job'] = loan['Years in current job'].str.strip(r'+ years').replace('< 1', '1').fillna(0)
loan['Years in current job'] = loan['Years in current job'].astype(int)
loan['Years in current job'].describe()<jupyter_output><empty_output><jupyter_text>Annual Income* There are 61676 missing entries, the same number of missing entries as Credit Score. Similaliry, we can fill these with the mean.<jupyter_code>loan['Annual Income'] = loan['Annual Income'].fillna(loan['Annual Income'].mean())
loan['Annual Income'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Months since last delinquent * 140383 nan entries for Months since last delinquent.* Looking at the data, it can be result of not having any late payments, therefore we can replace it with 0.<jupyter_code>loan['Months since last delinquent']= loan['Months since last delinquent'].fillna(0)
loan['Months since last delinquent'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Bankruptcies* 529 missing entries, we can fill it with 0 as the mean is 0.11<jupyter_code>loan['Bankruptcies'] = loan['Bankruptcies'].fillna(0)<jupyter_output><empty_output><jupyter_text>Tax Liens* Only 29 missing entries, with mean of 0.02, we will fill it with 0<jupyter_code>loan['Tax Liens'] = loan['Tax Liens'].fillna(0)
loan.isna().sum()<jupyter_output><empty_output><jupyter_text>* We have completed filling all the missing entries  Duplicate records* There are 215700 unique records for 'Loan ID' and 'Customer ID', resulting in 41284 duplicate records in the dataset to be removed.<jupyter_code>loan['Loan ID'].nunique()
loan['Customer ID'].nunique()
loan.drop_duplicates(subset=['Loan ID', 'Customer ID'], inplace=True)
loan.value_counts().sum()<jupyter_output><empty_output><jupyter_text>Correcting dtypes* Monthly Debt column contains '$' signs in the begining of the recods* Maximum Open Credit has VALUE! as entry<jupyter_code>loan['Monthly Debt'] = loan['Monthly Debt'].str.strip(r'$')
loan['Monthly Debt'][0]
loan['Monthly Debt'] = loan['Monthly Debt'].apply( lambda x:   x.replace(',', ''))
loan['Monthly Debt'] = loan['Monthly Debt'].astype(float)
loan.isna().sum()<jupyter_output><empty_output><jupyter_text>Correcting Maximum Open Credit<jupyter_code>loan['Maximum Open Credit'] .value_counts().sort_values(ascending=False)
loan['Maximum Open Credit']  = loan['Maximum Open Credit'] .replace(to_replace=r'#VALUE!', value=0, regex=True)
loan['Maximum Open Credit'] = loan['Maximum Open Credit'].astype(float)
loan['Maximum Open Credit'].dtypes
loan['Maximum Open Credit'] .value_counts().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Correcting purpose entries<jupyter_code>loan['Purpose'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>* There are two 'Other' entries for purpose, one with a capital and one without. We will map them into one.<jupyter_code>loan['Purpose'] = loan['Purpose'].replace(['Other', 'other'], 'Other Reasons')
loan['Purpose'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Correcting Home Ownership Entries<jupyter_code>loan['Home Ownership'].value_counts()
loan['Home Ownership'] = loan['Home Ownership'].replace('HaveMortgage', 'Home Mortgage')
loan['Home Ownership'].value_counts()<jupyter_output><empty_output><jupyter_text>Outliers  Credit Score<jupyter_code>sns.boxplot(x=loan['Credit Score'])<jupyter_output><empty_output><jupyter_text>* Above plot shows block of entries for credit score around 6000 to 8000, these are outliers as there not included in the box of other observations (near the quartiles).<jupyter_code>credit_outlier = loan[loan['Credit Score'] > 5000].count()[0]
credit_in = loan[loan['Credit Score'] < 2000].count()[0]
credit_outlier, credit_in<jupyter_output><empty_output><jupyter_text>* There are 14,433 records that have credit score above 5000 and 201, 267 entries below credit score of 2000* There's no credit score entries between 2000 and 5000, indicating that there is an issue here.* One explaination could be that the records are from different credit agencies with different credit scoring range, we will remove the records above 2000 Credit Score.<jupyter_code>loan = loan[loan['Credit Score'] < 2000]
loan.shape
fig, ax = pp.subplots(figsize=(8,4))
ax.scatter(loan['Credit Score'], loan['Annual Income'])
ax.set_xlabel('Credit Score')
ax.set_ylabel('Annual Income')
pp.show()<jupyter_output><empty_output><jupyter_text>* There's still an empty gap between the records in the range of 550-750 and 1200+* Research shows that most credit scores are in the range of 350 and 850<jupyter_code>loan[loan['Credit Score'] > 850].count()[0]
loan = loan[loan['Credit Score'] <=  850]
loan.shape
sns.boxplot(x=loan['Credit Score'])
fig, ax = pp.subplots(figsize=(8,4))
ax.scatter(loan['Credit Score'], loan['Annual Income'])
ax.set_xlabel('Credit Score')
ax.set_ylabel('Annual Income')
pp.show()<jupyter_output><empty_output><jupyter_text>* The box plot and scatter diagram show that the spread of data is more even, with very few outliers now.  Current Loan Amount<jupyter_code>sns.boxplot(x=loan['Current Loan Amount'])
loan['Current Loan Amount'].value_counts().sort_index(ascending=False)<jupyter_output><empty_output><jupyter_text>* Clearly there is a problem here, we have over 35210 entries with 99,999,999 (99 million)* Sometimes 9's are used for unknowns or if  there is a floating-point computation problem.* This is a standard artifact of how computers do arithmetic -- convert decimal number to binary (which means most fractions cannot be exactly represented in a finite amount of memory), perform the operation, then convert back to decimal. This is a common and well known feature/bug in computer arithmetic and has been well described for many years<jupyter_code>odd_amount  = loan[loan['Current Loan Amount'] == 99999999]

odd_amount['Loan Status'].value_counts()<jupyter_output><empty_output><jupyter_text>* All the current loan amounts with 99 million have been fully paid.* As a result, I am going to assume that they are meant to be 0's instead of 99million and replace those values in the dataset.<jupyter_code>loan['Current Loan Amount'] = loan['Current Loan Amount'].replace(99999999, 0)
sns.boxplot(x=loan['Current Loan Amount'])
outlier  = loan[loan['Current Loan Amount'] >= 37000]
outlier.shape<jupyter_output><empty_output><jupyter_text>* The boxplot looks much better now, we have a range of values from 0 to 35,000.* Only 19 outliers above the 37,000 range, we can leave them in this case.  Part 2: Exploratory Data Analysis * Plotting a diagonal correlation matrix<jupyter_code>#Matrix correlation
corr = loan.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
f, ax = pp.subplots(figsize=(11, 9))
cmap = sns.diverging_palette(230, 20, as_cmap=True)

sns.heatmap(corr, mask=mask, cmap=cmap, vmax = 1, center=0, square=True, linewidths=.5, cbar_kws={'shrink': .5}, annot=True)<jupyter_output><empty_output><jupyter_text>* The correlation map shows the strength of linear relationship between two variables.* Right from the start, you can see there is a positive strong correlation between Bankruptcies and Number of Credit Problems. * There is also strong correlation between Monthly Debt and Current Credit Balance and Number of Open Accounts.* Majority of other variables have no relationship  Histogram of Credit Score and Purpose<jupyter_code>ax = sns.histplot(data=loan, x='Credit Score', hue='Purpose', multiple='stack')

sns.move_legend(
    ax, "lower center",
    bbox_to_anchor=(.5, 1), ncol=3, title=None, frameon=True,
)<jupyter_output><empty_output><jupyter_text>* Majority of the credit scores are in the 725- 750 range, which is in the good to excellent band range of most credit providers* The diagram also shows that the main purpose of the credit loans are debt consolidation* Debt consolidation is a form of debt refinancing that entails taking out one loan to pay off many others. This commonly refers to a personal finance process of individuals addressing high consumer debt, but occasionally it can also refer to a country's fiscal approach to consolidate corporate debt or government debt  Years in current job and Purpose<jupyter_code>ax = sns.histplot(data=loan, x='Years in current job', hue='Purpose', multiple='stack')
sns.move_legend(
    ax, "lower center",
    bbox_to_anchor=(.5, 1), ncol=3, title=None, frameon=True,
)<jupyter_output><empty_output><jupyter_text>* Over 40,000 of the records are from those with over 10 years in their current job.* With over 20,000 with a year in their current job* This can indicate that people are more likely to get a loan in their first year of after 10 years in their current role.  Credit Score and Loan Status<jupyter_code>ax = sns.ecdfplot(data=loan, x='Credit Score', hue='Loan Status', stat='count', complementary=False, )
loan['Loan Status'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>* The diagram shows that majority of the loans have been fully paid. * 91% of loans have been fully paid off, while 8% have been charged off.  Home Ownership<jupyter_code>g = sns.catplot(data=loan, x= 'Loan Status', y='Monthly Debt', col='Home Ownership', saturation=.5,
               kind='bar', ci=None, aspect=.6)
g.set_axis_labels("", 'Monthly Debt')
g.set_xticklabels([' Fully Paid', 'Chaged off'])
g.set_titles("{col_name}")
g.set(ylim=(0,1200))
g.despine(left=True)<jupyter_output><empty_output><jupyter_text>* Interesting to see that for each Home Ownership category, customers that were charged off had a slightly higher monthly debt than those that fully paid it.<jupyter_code>loan.head()
loan.Term.value_counts()<jupyter_output><empty_output><jupyter_text>Part 3: Cleaning and Creating Features  Converting categorical data to numeric<jupyter_code>for feature in ['Loan Status', 'Term', 'Home Ownership', 'Purpose']:
    le = LabelEncoder()
    loan[feature] = le.fit_transform(loan[feature].astype(str))

loan.head()<jupyter_output><empty_output><jupyter_text>Part 4: Split the data into Train, Validate and Test<jupyter_code>#We will not be using the Loan ID and Customer ID in our models as it can cause problems with the results, given that they are not meaningful to Loan Status.
loan_ml = loan.drop(['Loan ID', 'Customer ID'], axis=1)
loan_ml.head()
features = loan_ml.drop(['Loan Status'], axis=1)
labels = loan_ml['Loan Status']

x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size=0.4, random_state=42)
x_val, x_test, y_val, y_test = train_test_split(x_test, y_test, test_size=0.5, random_state=42)

x_train.head()
for dataset in [y_train, y_val, y_test]:
    print(round(len(dataset)/ len(labels), 2))<jupyter_output>0.6
0.2
0.2<jupyter_text>Part 5: Machine Learning: Applying Random Forest x_train.to_csv('data/train_features.csv', index=False)x_val.to_csv('data/val_features.csv', index=False)x_test.to_csv('data/test_features.csv', index=False)y_train.to_csv('data/train_labels.csv', index=False)y_val.to_csv('data/val_labels.csv', index=False)y_test.to_csv('data/test_labels.csv', index=False)  GridSearchCV<jupyter_code>def print_results(results):
    print(f'BEST PARAMS: {results.best_params_}')
    
    means = results.cv_results_['mean_test_score']
    stds = results.cv_results_['std_test_score']
    for mean, std, params in zip(means, stds, results.cv_results_['params']):
        print(f'{round(mean,3)} (+/- { round(std*2, 3)}) for { params}')
# Conduct search for best params while running crosss-validation (GridSearchCV)

rf = RandomForestClassifier()
parameters = {
    'n_estimators': [2**i for i in range(3, 6)],
    'max_depth': [2, 4, 6, 8, None]
}

cv = GridSearchCV(rf, parameters, cv=5)
cv.fit(x_train, y_train.values.ravel())

print_results(cv)
cv.best_estimator_
cv.best_params_
cv.best_score_<jupyter_output><empty_output><jupyter_text>The results for the Random Forest Classifier gives us a best score of 91%, with best params of max depth 2 and estimators of 8.<jupyter_code>#Generate feature importance plot
feat_imp = cv.best_estimator_.feature_importances_
indices = np.argsort(feat_imp)
pp.yticks(range(len(indices)), [x_train.columns[i] for i in indices])
pp.barh(range(len(indices)), feat_imp[indices], color='r', align='center')
pp.show()<jupyter_output><empty_output><jupyter_text>* Term plays the most significant role in determing Loan status, with a weight of 0.35.* Credit score surprisnly comes second in determing Loan status, with a weight of 2.25* The other two factors that play a significant role are Current Loan Amount and Annual Income.<jupyter_code># Write out the model fit on the whole training set
joblib.dump(cv.best_estimator_, 'rfc_1.pk1')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11300429486728156, 'fraction_numerical': 0.02006618320073224, 'mean_word_length': 6.8086860912589335, 'pattern_counts': {'":': 0, '<': 132, '<?xml version=': 0, '>': 131, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '484231', 'license': 'MIT', 'n_tokens_mistral': 4866, 'n_tokens_neox': 4409, 'n_words': 1429, 'path': 'Loan Dataset Exploration.ipynb', 'repo_name': 'supermanimmy/loan-dataset-exploration', 'size': '14203'}