<jupyter_start><jupyter_text>Effects of Borrowers' Information on Their Interest Rate and Loan Completion Status by <NAME>  Investigation OverviewIn this investigation, I will try to determine two things:  first, which features are best at predicting the `LoanStatus`; second, which features determine the `BorrowerAPR ` and `BorrowerRate` at most? `Credit Score (CreditScoreRangeLower/CreditScoreRangeUpper)`, `ProsperScore`, and  `Term` will be used primarily to determine thd three pivot variables (`LoanStatus`, `BorrowerAPR`, and `BorrowerRate`).  Dataset OverviewThis data set contains 113,937 loans with 81 variables on each loan, including loan amount, borrower rate, current loan status, borrower income, and many others.Some points were removed from teh dataset due to thier potential influence (outliers) in visualization.<jupyter_code># import all packages and set plots to be embedded inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb

%matplotlib inline

# suppress warnings from final output
import warnings
warnings.simplefilter("ignore")
# load in the dataset into a pandas dataframe
df = pd.read_csv('prosperLoanData.csv', encoding='latin-1', low_memory=False)
cols = ['Term', 'LoanStatus', 'BorrowerAPR', 'BorrowerRate', 'ListingCategory', 'EmploymentStatus'
          , 'IsBorrowerHomeowner', 'CreditScoreRangeLower', 'CreditScoreRangeUpper', 'DebtToIncomeRatioff', 
           'StatedMonthlyIncome', 'PercentFunded', 'Recommendations', 'InvestmentFromFriendsCount', 'ProsperScore']
df_sub = df.copy()
df_sub = df_sub.loc[:, df.columns.isin(cols)]
df_sub.drop(index=df_sub.query('StatedMonthlyIncome == 0').index, inplace = True)
df_sub.drop(index=df_sub.query('StatedMonthlyIncome >= 50000').index, inplace = True)
df_sub.drop(index=df_sub.query('StatedMonthlyIncome <= 200').index, inplace=True)

def renameLoanStatus(x):
    if 'Past Due' in x:
        return 'Past Due'
    elif x == 'FinalPaymentInProgress':
        return 'InProgress'
    return x

df_sub.EmploymentStatus = df_sub.EmploymentStatus.astype('category')


df_sub.LoanStatus = df_sub.LoanStatus.apply(renameLoanStatus)
df_sub.LoanStatus = df_sub.LoanStatus.astype('category')

df_sub['Term'] = df_sub['Term'].astype('category')
df_sub['ProsperScore'] = df_sub['ProsperScore'].astype('category')

df_sub.drop(axis=0, index=df_sub.query('CreditScoreRangeLower < 200').index, inplace=True);<jupyter_output><empty_output><jupyter_text>Distribution of Loan StatusMore than half of the loans are in the `Current` status,and with 30 per cent already `Completed` loans. Nevertheless, attention should be focused on the **bad loans**, which are: `Defaulted`, `Past Due`, `Chargedoff`, and `Cancelled`. They are about 15 per cent of the total loans.<jupyter_code># get the relative frequency of LoanStatus, i.e., by proportion
base_color = sb.color_palette()[4]
n_points = df_sub.shape[0]
max_count = df_sub['LoanStatus'].value_counts().max()
max_prop = max_count / n_points

# generate tick mark locations and names
tick_props = np.arange(0, max_prop, 0.05)
tick_names = ['{:0.2f}'.format(v) for v in tick_props]

# create the plot
plt.figure(figsize=[14.70, 8.27])
sb.countplot(data = df_sub, x = 'LoanStatus', color = base_color, )
plt.yticks(tick_props * n_points, tick_names)
plt.ylabel('proportion')

locs, labels = plt.xticks() # get the current tick locations and labels

# loop through each pair of locations and labels
for loc, label in zip(locs, labels):

    # get the text property for the label to get the correct count
    count = df_sub['LoanStatus'].value_counts()[label.get_text()]
    pct_string = '{:0.3f}%'.format(100*count/n_points)

    # print the annotation just below the top of the bar
    plt.text(loc, count-8, pct_string, ha = 'center', color = 'black')

plt.title('Distribution of LoanStatus'.title(),
               fontsize = 14, weight = "bold")
plt.xticks(rotation = 15);<jupyter_output><empty_output><jupyter_text>Distribution of BorrowerAPR and BorrowerRateThe two distributions look very similar, noting that `BarrowerAPR` is higher than `BorrowerRate` on average. They both have extremely high count between 0.3 and 0.4<jupyter_code># combine the two plots
plt.figure(figsize=[14.70, 8.27])
bins_edge = np.arange(0, df_sub.BorrowerAPR.max()+0.002, 0.002)
plt.hist(data=df_sub, x='BorrowerAPR', bins=bins_edge)

bins_edge = np.arange(0, df_sub.BorrowerRate.max()+0.002, 0.002)
plt.hist(data=df_sub, x='BorrowerRate', bins=bins_edge)
plt.xlabel('BorrowerRate/APR combined')
plt.ylabel('count')
plt.title('Distribution of BorrowerAPR and BorrowerRate',
               fontsize = 14, weight = "bold")
plt.legend(['BorrowerAPR', 'BorrowerRate']);
plt.tight_layout();<jupyter_output><empty_output><jupyter_text>Correlation between Credit Score and Borrower Ratethere is a moderate negative raltionship between the `Credit Score` and the two `borrorwers rate`. Meanwhile, the two pairs of vriables are almost podtively related to each other perfectly. `StatedMonthlyIncome` though, it has a very weak relation with `borrowers rate`, probably because income alone would be less convincing than `Credit Score` in demonstrating a borrowers' credibility.<jupyter_code>#initialize some values
numeric_vars = ['BorrowerAPR', 'BorrowerRate', 'CreditScoreRangeLower', 'CreditScoreRangeUpper', 
                'StatedMonthlyIncome']
categoric_vars = ['Term', 'LoanStatus', 'EmploymentStatus', 'IsBorrowerHomeowner', 'ProsperScore']
# correlation plot
plt.figure(figsize=[14.70, 8.27])
sb.heatmap(df_sub[numeric_vars].corr(), annot = True, fmt = '.3f',
           cmap = 'Blues_r', center = 0.25)

plt.title('Correlation of Borrower Rate and Other Numeric Variables'.title(),
               fontsize = 14, weight = "bold")
plt.tight_layout();
plt.show()<jupyter_output><empty_output><jupyter_text>ProsperScore vs. Borrower RateAs observed from above, `ProsperScore` seemed to be negatively related to the two `borrowers rate`. When `ProsperScore` was as low as 1.0, typical `borrowers rate` went up high to 3.0. In the middle, however, `borrowers rate` could take a wide range of vlaue, from 0.1 to 0.3. And when `ProsperScore` reached the right end extreme, few `borrowers rate` went up to 0.3.<jupyter_code>i = 1
def scattergrid(x, y, **kwargs):
    """ Quick hack for creating scatter plots with pyplot's subplot. """
    
    
    global i
    plt.subplot(4,2,i)
    plt.subplots_adjust(wspace=0.2, hspace= 1, )
    sb.stripplot(x, y, color = base_color, alpha = 0.1, jitter=0.35)
    i += 1

plt.figure(figsize=[14.70, 8.27])
scattergrid(df_sub.ProsperScore, df_sub.BorrowerAPR)
plt.title('BorrowerAPR and ProsperScore'.title(),
               fontsize = 14)
scattergrid(df_sub.ProsperScore, df_sub.BorrowerRate)
plt.title('BorrowerRate and ProsperScore'.title(),
               fontsize = 14)
plt.suptitle('Borrower Rate and ProsperScore'.title(),
               fontsize = 14, weight = "bold", y = 1.05)
plt.tight_layout();<jupyter_output><empty_output><jupyter_text>Credit Score vs. LoanStatusJust as expected, the `CreditScore Bounds` has wave-like distribution, this is most visible in the **Current** `LoanStatus`. And not suprisingly, the **Defaulted**, **Cancelled**, **Chargedoff** and **Past Due**  `LoanStatus`, which all are bad loans, have a lower median `CreditScore Bounds` than the **Current** and **InProgress** LoanStatus, which are good loans (or about to be good loans). This might reveal that good debts came from a high credit score borrower on average.<jupyter_code># plot matrix of LoanStatus against numeric features.
i = 1
def Boxgrid(x, y, **kwargs):
    """ Quick hack for creating scatter plots with pyplot's subpplot """
    
    
    global i
    plt.subplot(2,2,i)
    plt.subplots_adjust(wspace=0.2, hspace= 0.3, )
    sb.violinplot(x, y, color = base_color)
    i += 1

plt.figure(figsize=[14.70, 8.27])
Boxgrid(df_sub.LoanStatus, df_sub.CreditScoreRangeLower)
plt.title('Loan Status and CreditScoreRangeLower'.title(),
               fontsize = 14)
Boxgrid(df_sub.LoanStatus, df_sub.CreditScoreRangeUpper)
plt.title('Loan Status and CreditScoreRangeUpper'.title(),
               fontsize = 14)
plt.suptitle('Loan Status and Credit Score'.title(),
               fontsize = 14, weight = "bold", y = 0.97);<jupyter_output><empty_output><jupyter_text>Term vs. LoanStatus**60** months' `Term` seem to have a better performance in `LoanStatus`, compared to the mode **36** months, fewer loans were ***Chargedoff*** or went ***Defaulted*** in **60** months loans.<jupyter_code># plot matrix of LoanStatus against categorical features.

i = 1
def Countgrid(x, hue, subtitile, **kwargs):
    """ Quick hack for creating counts plots with pyplot's subplot. """
    
    
    global i
    plt.subplot(1,1,i)
    plt.subplots_adjust(wspace=0.2, hspace= 0.2, )
    sb.countplot(x, hue=hue, palette='Reds')
    plt.legend(loc=1)
    plt.gca().set_title(subtitile)
    i += 1
    
plt.figure(figsize=[14.70, 8.27])
plt.suptitle('LoanStatus and Term'.title(),
               fontsize = 14, weight = "bold", y = 0.93);
Countgrid(df_sub.LoanStatus, df_sub.Term, "Term")<jupyter_output><empty_output><jupyter_text>Given the extreme similarity in distributions between `BorrowerAPR` and `BorrowerRate` , and as well as between `CreditScoreRangeLower` and `CreditScoreRangeUpper`, I would only use one of each pair to analyze in the next section: namely, `BorrowerAPR` and `CreditScoreRangeLower`.    Borrower Rate in Differnt ProsperScore and Credit ScoreThe trend is most obvious in high `ProsperScore`, like **9.0**, **10.0**, and **11.0**. In high `ProsperScore`, the interst rate decreases as borrowers' `CreditScoreRangeLower` increases. However, the trend is not so evident in low `ProsperScore`.<jupyter_code># create faceted heat maps on levels of the ProsperScore variable
bins_x = np.arange(df_sub.CreditScoreRangeLower.min()+150, df_sub.CreditScoreRangeLower.max(), 25.0)
bins_y = np.arange(0.0, 0.3, 0.05)

j = 1
def hist2dgrid(ProsperScore, **kwargs):
    """ Quick hack for creating heatmap plots with pyplot's hist2d """
    
    df_sub_temp = df_sub[df_sub.ProsperScore == ProsperScore]
    
    global j
    plt.subplot(6,2,i)
    plt.subplots_adjust(wspace=0.1, hspace= 0.5, )
    plt.hist2d(data=df_sub_temp, x='CreditScoreRangeLower', y='BorrowerRate' , cmin=0.5, 
               cmap = 'viridis_r', bins = [bins_x, bins_y])
    plt.gca().set_title('ProsperScore = '+ str(j))
    j = int(j)
    j += 1

plt.figure(figsize=(14,17))
for i in range(1, 7):
    hist2dgrid(i)
    
plt.suptitle('Borrower Rate in Different Credit Score and ProsperScore'.title(),
               fontsize = 14, weight = "bold", y = 1.025);
plt.colorbar()
plt.tight_layout();
plt.show()
plt.figure(figsize=(14,17))
for i in range(7, 12):
    hist2dgrid(i)

plt.colorbar()
plt.tight_layout();
plt.show()<jupyter_output><empty_output><jupyter_text>LoanStatus in Differnt Term and Credit ScoreLet's look at each plot individually, the **Cancelled** plot can not provide much information since it only has the `Term` of ***36***; in the **Chargedoff** plot, lower median of `CreditScore` is observed in `Term` ***36***; in btoh **Completed** and **Current** plots, differnt `Term` values did not show much difference in median; in `Defaulted` and `Inprogress` plots, the `Term`***36*** revealed a low median `Creditscore`; finally, the **Past Due** plot did show significant enough difference in medians.<jupyter_code># plot the matrices between LoanStatus, Term, and CreditScoreRangeLower. 
g = sb.FacetGrid(data = df_sub, col = 'LoanStatus', height = 8.27/2, col_wrap=4)
g.map(sb.boxenplot, 'Term', 'CreditScoreRangeLower', color=base_color,)
plt.suptitle('Loan Status in Different Credit Score and Term'.title(),
               fontsize = 14, weight = "bold", y = 1.05);
plt.tight_layout();<jupyter_output><empty_output><jupyter_text>LoanStatus in Different ProsperScore and Credit ScoreOne interesting fact from the plots: if a borrower has a high `ProsperScore` of **11.0** and a high `CreditScoreRangeLower`, then his/her loan is almost guaranteed to not go bad. This is witnessed in the **Defaulted**, **Chargedoff**, and **PastDue** boxplots, they have close to zero amount of data with high `ProsperScore` and high `CreditScoreRangeLower` at the same time.<jupyter_code># plot the matrices between LoanStatus, Term, and CreditScoreRangeLower. 
g = sb.FacetGrid(data = df_sub, col = 'LoanStatus', height = 8.27/2, col_wrap=4)
g.map(sb.boxenplot, 'ProsperScore', 'CreditScoreRangeLower', color=sb.color_palette()[6])
plt.suptitle('Loan Status in Different Credit Score and ProsperScore'.title(),
               fontsize = 14, weight = "bold", y = 1.05);
plt.tight_layout();<jupyter_output><empty_output><jupyter_text>LoanStatus in Different ProsperScore and TermJust as expected, median `ProsperScore` like **5.0**, **6.0** and **7.0**, and the `Term` of ***36*** contributed most to bad loans. The low `ProsperScore` though, might have fewer data in total, and therefore resulted in less count of bad loans.<jupyter_code># plot the matrices between LoanStatus, Term, and ProsperScore. 

#since our attention can be focused on the bad loans distribution, I will limit the LoanSatus here.
LoanStatus_sub = ['Cancelled', 'Chargedoff', 'Defaulted', 'Past Due']
ProsperScore_sub_1 = [1, 2, 3]
ProsperScore_sub_2 = [4, 5, 6]
ProsperScore_sub_3 = [7, 8, 9]
ProsperScore_sub_4 = [10, 11]

# ProsperScore Group 1
df_sub_temp_1 = df_sub.copy()
df_sub_temp_1 = df_sub_temp_1[df_sub_temp_1['LoanStatus'].isin(LoanStatus_sub)]
df_sub_temp_1 = df_sub_temp_1[df_sub_temp_1['ProsperScore'].isin(ProsperScore_sub_1)]

LoanStatus_sub_order = pd.api.types.CategoricalDtype(categories=LoanStatus_sub, ordered=True)
df_sub_temp_1.LoanStatus =  df_sub_temp_1.LoanStatus.astype(LoanStatus_sub_order)

ProsperScore_sub_order_1 = pd.api.types.CategoricalDtype(categories=ProsperScore_sub_1, ordered=True)
df_sub_temp_1.ProsperScore =  df_sub_temp_1.ProsperScore.astype(ProsperScore_sub_order_1)

# ProsperScore Group 2
df_sub_temp_2 = df_sub.copy()
df_sub_temp_2 = df_sub_temp_2[df_sub_temp_2['LoanStatus'].isin(LoanStatus_sub)]
df_sub_temp_2 = df_sub_temp_2[df_sub_temp_2['ProsperScore'].isin(ProsperScore_sub_2)]

LoanStatus_sub_order = pd.api.types.CategoricalDtype(categories=LoanStatus_sub, ordered=True)
df_sub_temp_2.LoanStatus =  df_sub_temp_2.LoanStatus.astype(LoanStatus_sub_order)

ProsperScore_sub_order_2 = pd.api.types.CategoricalDtype(categories=ProsperScore_sub_2, ordered=True)
df_sub_temp_2.ProsperScore =  df_sub_temp_2.ProsperScore.astype(ProsperScore_sub_order_2)

# ProsperScore Group 3
df_sub_temp_3 = df_sub.copy()
df_sub_temp_3 = df_sub_temp_3[df_sub_temp_3['LoanStatus'].isin(LoanStatus_sub)]
df_sub_temp_3 = df_sub_temp_3[df_sub_temp_3['ProsperScore'].isin(ProsperScore_sub_3)]

LoanStatus_sub_order = pd.api.types.CategoricalDtype(categories=LoanStatus_sub, ordered=True)
df_sub_temp_3.LoanStatus =  df_sub_temp_3.LoanStatus.astype(LoanStatus_sub_order)

ProsperScore_sub_order_3 = pd.api.types.CategoricalDtype(categories=ProsperScore_sub_3, ordered=True)
df_sub_temp_3.ProsperScore =  df_sub_temp_3.ProsperScore.astype(ProsperScore_sub_order_3)

# ProsperScore Group 4
df_sub_temp_4 = df_sub.copy()
df_sub_temp_4 = df_sub_temp_3[df_sub_temp_3['LoanStatus'].isin(LoanStatus_sub)]
df_sub_temp_4 = df_sub_temp_3[df_sub_temp_3['ProsperScore'].isin(ProsperScore_sub_4)]

LoanStatus_sub_order = pd.api.types.CategoricalDtype(categories=LoanStatus_sub, ordered=True)
df_sub_temp_4.LoanStatus =  df_sub_temp_4.LoanStatus.astype(LoanStatus_sub_order)

ProsperScore_sub_order_4 = pd.api.types.CategoricalDtype(categories=ProsperScore_sub_4, ordered=True)
df_sub_temp_4.ProsperScore =  df_sub_temp_4.ProsperScore.astype(ProsperScore_sub_order_4)

# Group 1
g = sb.FacetGrid(data = df_sub_temp_1, col = 'Term',row='ProsperScore', height = 8.27/3, aspect=(14.70/3)/(8.27/3))
g.map(sb.countplot, 'LoanStatus', color=sb.color_palette()[7])
plt.suptitle('Loan Status in Different Term and ProsperScore'.title(),
               fontsize = 14, weight = "bold", y = 1.05);
plt.tight_layout();
# Group 2
g = sb.FacetGrid(data = df_sub_temp_2, col = 'Term',row='ProsperScore', height = 8.27/3, aspect=(14.70/3)/(8.27/3))
g.map(sb.countplot, 'LoanStatus', color=sb.color_palette()[7]);
plt.tight_layout();
# Group 3
g = sb.FacetGrid(data = df_sub_temp_3, col = 'Term',row='ProsperScore', height = 8.27/3, aspect=(14.70/3)/(8.27/3))
g.map(sb.countplot, 'LoanStatus', color=sb.color_palette()[7])
plt.tight_layout();
# Group 4
g = sb.FacetGrid(data = df_sub_temp_4, col = 'Term',row='ProsperScore', height = 8.27/2, aspect=(14.70/3)/(8.27/2))
g.map(sb.countplot, 'LoanStatus', color=sb.color_palette()[7])
plt.tight_layout();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1223154262597418, 'fraction_numerical': 0.025403057885656492, 'mean_word_length': 5.495363214837712, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '558130', 'license': 'Apache-2.0', 'n_tokens_mistral': 6187, 'n_tokens_neox': 5722, 'n_words': 1488, 'path': 'Explanatory_Analysis.ipynb', 'repo_name': 'Peta0228/DAND_ProsperLoan-Analysis-Bad-Loan-and-Borrower-APR', 'size': '16811'}