<jupyter_start><jupyter_text>Prosper Loan Data Exploration by <NAME> Preliminary WranglingThis dataset is composed of 113,937 loans and for each entry there are 81 variables. Those variables include loan amount, borrower rate, current loan status, borrower income, and many others.<jupyter_code># Import all packages and set plots to be embedded inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb

%matplotlib inline
# Load in the dataset into a pandas dataframe
df = pd.read_csv('prosperLoanData.csv')
df.head(10)
# High-level overview of data shape and composition
# Display all columns and data types
pd.set_option('display.max_rows', 81)
print(df.info())
print(df.shape)
print(df.describe())<jupyter_output>(113937, 81)
       ListingNumber           Term    BorrowerAPR   BorrowerRate  \
count   1.139370e+05  113937.000000  113912.000000  113937.000000   
mean    6.278857e+05      40.830248       0.218828       0.192764   
std     3.280762e+05      10.436212       0.080364       0.074818   
min     4.000000e+00      12.000000       0.006530       0.000000   
25%     4.009190e+05      36.000000       0.156290       0.134000   
50%     6.005540e+05      36.000000       0.209760       0.184000   
75%     8.926340e+05      36.000000       0.283810       0.250000   
max     1.255725e+06      60.000000       0.512290       0.497500   

         LenderYield  EstimatedEffectiveYield  EstimatedLoss  EstimatedReturn  \
count  113937.000000             84853.000000   84853.000000     84853.000000   
mean        0.182701                 0.168661       0.080306         0.096068   
std         0.074516                 0.068467       0.046764         0.030403   
min        -0.010000                -0.18[...]<jupyter_text>There are variables with a lot of null values. Also, there are many variables which seem to be of institutional use and do not bring any useful information.  There are many variables set as object type, which could be changed to facilitate its' usage.  What is the structure of your dataset?The data is structured in 113,937 loan entries in which each entry has 81 different variables. These variables are mostly numeric, however there are some categorical variables. Numeric variables are mainly values indicating total loan amount, yield, payment rates and overall rates. On the other hand, categorical variables indicate dates and characteristics of the borrowers. What is/are the main feature(s) of interest in your dataset?The main features of interest in the dataset is to understand what factors influence the loan's status and what factors influence the payment of the loan.This implies the analysis of the LoanStatus and its relation with some of the other variables. What features in the dataset do you think will help support your investigation into your feature(s) of interest?It is expected that the LoanStatus is affected by: Term, BorrowerRate, Occupation, EmploymentStatus, IsBorrowerHomeowner, CurrentDelinquencies, DelinquenciesLast7Years, DebtToIncomeRatio, IncomeRange, StatedMonthlyIncome, LoanOriginalAmount and MonthlyLoanPayment. Moreover, I expect that completed and current status are linked with a smaller number of terms, a lower borrower rate, a positive employment status, homeownership, no delinquencies, a small debt to income ratio, high income range and monthly income.    Some useful modifications to the original data<jupyter_code>df_loan = df.copy()
df_loan = df_loan.rename(columns={'ProsperRating (Alpha)': 'ProsperRating', 'ListingCategory (numeric)': 'ListingCategory'})
df_loan.columns
variables = ['Term', 'LoanStatus', 'BorrowerRate','LenderYield', 'ListingCategory',
'EstimatedEffectiveYield', 'EstimatedLoss','EstimatedReturn',  'ProsperRating', 
'ProsperScore', 'Occupation', 'EmploymentStatus','IsBorrowerHomeowner',  'CurrentDelinquencies',
             'AmountDelinquent', 'DelinquenciesLast7Years','DebtToIncomeRatio', 'IncomeRange', 
             'StatedMonthlyIncome', 'LoanOriginalAmount', 'MonthlyLoanPayment']
not_variables = df_loan.columns[df_loan.columns.isin(variables)==False]
df_loan.drop(not_variables, axis=1, inplace=True)
df_loan.head()
df_loan.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 113937 entries, 0 to 113936
Data columns (total 21 columns):
 #   Column                   Non-Null Count   Dtype  
---  ------                   --------------   -----  
 0   Term                     113937 non-null  int64  
 1   LoanStatus               113937 non-null  object 
 2   BorrowerRate             113937 non-null  float64
 3   LenderYield              113937 non-null  float64
 4   EstimatedEffectiveYield  84853 non-null   float64
 5   EstimatedLoss            84853 non-null   float64
 6   EstimatedReturn          84853 non-null   float64
 7   ProsperRating            84853 non-null   object 
 8   ProsperScore             84853 non-null   float64
 9   ListingCategory          113937 non-null  int64  
 10  Occupation               110349 non-null  object 
 11  EmploymentStatus         111682 non-null  object 
 12  IsBorrowerHomeowner      113937 non-null  bool   
 13  CurrentDelinquencies     113240 non-null  float64
 14  Amou[...]<jupyter_text>There are some issues with data types:* LoanStatus, ProsperScore, Occupation, EmploymentStatus and IncomeRange should be categories.<jupyter_code># Fixing categories types
# Convert IncomeRange into ordered categorical type
income_range = ['Not displayed', 'Not employed', '$0', '$1-24,999', '$25,000-49,999', '$50,000-74,999', 
                '$75,000-99,999', '$100,000+']
income_ranges = pd.api.types.CategoricalDtype(ordered = True, categories = income_range)
df_loan['IncomeRange'] = df_loan['IncomeRange'].astype(income_ranges)

# Convert IncomeRange into ordered categorical type
prosper_rate = ['N/A','HR','E','D','C', 'B', 'A', 'AA']
prosper_rates = pd.api.types.CategoricalDtype(ordered = True, categories = prosper_rate)
df_loan['ProsperRating'] = df_loan['ProsperRating'].astype(prosper_rates)

#Adjust datatype for all other categorical columns
df_loan['LoanStatus'] = df_loan['LoanStatus'].astype('category')
df_loan['ProsperScore'] = df_loan['ProsperScore'].astype('category')
df_loan['Occupation'] = df_loan['Occupation'].astype('category')
df_loan['EmploymentStatus'] = df_loan['EmploymentStatus'].astype('category')

# Including definitions to ListingCategory
df_loan['ListingCategory'].replace({0: 'Not Available',
                                    1: 'Debt Consolidation',
                                    2: 'Home Improvement',
                                    3: 'Business', 
                                    4: 'Personal Loan', 
                                    5: 'Student Use', 
                                    6: 'Auto', 
                                    7: 'Other', 
                                    8: 'Baby&Adoption', 
                                    9:'Boat', 
                                    10: 'Cosmetic Procedure', 
                                    11: 'Engagement Ring', 
                                    12: 'Green Loans', 
                                    13: 'Household Expenses', 
                                    14: 'Large Purchases', 
                                    15: 'Medical/Dental', 
                                    16: 'Motorcycle', 
                                    17: 'RV', 
                                    18: 'Taxes', 
                                    19: 'Vacation', 
                                    20: 'Wedding Loans'}, inplace  = True)
df_loan.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 113937 entries, 0 to 113936
Data columns (total 21 columns):
 #   Column                   Non-Null Count   Dtype   
---  ------                   --------------   -----   
 0   Term                     113937 non-null  int64   
 1   LoanStatus               113937 non-null  category
 2   BorrowerRate             113937 non-null  float64 
 3   LenderYield              113937 non-null  float64 
 4   EstimatedEffectiveYield  84853 non-null   float64 
 5   EstimatedLoss            84853 non-null   float64 
 6   EstimatedReturn          84853 non-null   float64 
 7   ProsperRating            84853 non-null   category
 8   ProsperScore             84853 non-null   category
 9   ListingCategory          113937 non-null  object  
 10  Occupation               110349 non-null  category
 11  EmploymentStatus         111682 non-null  category
 12  IsBorrowerHomeowner      113937 non-null  bool    
 13  CurrentDelinquencies     113240 non-null  fl[...]<jupyter_text>Univariate Exploration<jupyter_code># The main reasons people take loans
fig, ax = plt.subplots(figsize = [8,8])

default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, y = 'ListingCategory', color = default_color, order = df_loan['ListingCategory'].value_counts().index)

plt.show()<jupyter_output><empty_output><jupyter_text>The most frequent reason for someone taking a loan is debt consolidation.<jupyter_code># First, I want to observe to distribution of loan status in the dataset.
fig, ax = plt.subplots(figsize = [8,8])

default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, y = 'LoanStatus', color = default_color)

plt.show()<jupyter_output><empty_output><jupyter_text>It is possible to observe that the majority of loans are current and this value is followed by loans which were completed.  It is possible to observe a considerable number of charged-off and defaulted loans.    It is not possible to have a clean observation of past due loans, which need a closer look.Loans that are defaulted and charged off will be directly linked to "bad payers". This is so because loans in "default" are those in which borrower is not paying for an extended period of time and loans "charged off" are no longer expected to be payed. [1][1] https://help.lendingclub.com/hc/en-us/articles/216127747-What-is-the-difference-between-a-loan-that-is-in-default-and-a-loan-that-has-been-charged-off-<jupyter_code>past_due = ['Past Due (1-15 days)', 'Past Due (16-30 days)', 'Past Due (61-90 days)', 'Past Due (31-60 days)', 'Past Due (91-120 days)', 'Past Due (>120 days)']

fig, ax = plt.subplots(figsize = [8,8])

default_color = sb.color_palette()[0]
sb.countplot(data = df_loan[df_loan.LoanStatus.isin(past_due)], y = 'LoanStatus', color = default_color)
plt.ylim(5.5,11.5)
plt.show();<jupyter_output><empty_output><jupyter_text>Here it is possible to see that majority of past due loans are within 15 days.  As delinquents are considered to be those that exceed 30 past due days [2], therefore will be considered delinquents those with Loan Status:* Defaulted * Chargedoff* Past Due (31-60 days)* Past Due (61-90 days)* Past Due (91-120 days)* Past Due (>120 days)'[2] https://www.investopedia.com/terms/p/past-due.asp<jupyter_code># Now, let's give a look in the EmploymentStatus in order to understand if borrowers have any source of money
# to pay the loan.
fig, ax = plt.subplots(figsize = [8,8])

default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, y = 'EmploymentStatus', color = default_color,  order = df_loan['EmploymentStatus'].value_counts().index)

plt.show()<jupyter_output><empty_output><jupyter_text>Here, it is possible to noticed that majority of borrowers are employed.<jupyter_code># Understand what are the main occupation among borrowers
fig, ax = plt.subplots(figsize = [15,15])

default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, y = 'Occupation', color = default_color,  order = df_loan['Occupation'].value_counts().index)
plt.show();
# Now, let's try to investigate if borrowers are homeowners, as this information could be an indication of stability 
# and a higher likelihood to pay the loan.
fig, ax = plt.subplots(figsize = [8,8])

default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, x = 'IsBorrowerHomeowner', color = default_color)

plt.show()<jupyter_output><empty_output><jupyter_text>The number of borrowers that do and do not own a house is similar.<jupyter_code># Distribution of number of delinquencies each borrower has:

fig, ax = plt.subplots(figsize = [20,10])
ticks = np.arange(0, df_loan.CurrentDelinquencies.max()+1,1).astype(int)
default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, x = 'CurrentDelinquencies', color = default_color)
plt.xlabel('Number of accounts delinquent')
plt.xticks(ticks, ticks)
plt.show()
df_loan[df_loan['CurrentDelinquencies'] > 10]<jupyter_output><empty_output><jupyter_text>Most borrowers have no delinquent accounts.  We can also observe that those with high number of current delinquencies are quite few compared to the total number of borrowers.<jupyter_code># Distribution of number of delinquencies in the last 7 years:

fig, ax = plt.subplots(figsize = [20,10])
ticks = np.arange(0, df_loan.DelinquenciesLast7Years.max()+1,1).astype(int)
default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, x = 'DelinquenciesLast7Years', color = default_color)
plt.xlabel('Number of delinquencies in the last 7 years')
plt.xticks(ticks, ticks)
plt.show()
df_loan[df_loan['DelinquenciesLast7Years']!=0].DelinquenciesLast7Years.mode()<jupyter_output><empty_output><jupyter_text>Most borrowers have no delinquencies in the last 7 years. Accounting for only those with delinquencies, the mode is 1 delinquency in the last 7 years.<jupyter_code># Distribution of debt to income ratio
bins = np.arange(0, df_loan['DebtToIncomeRatio'].max()+0.01, 0.01)
plt.figure(figsize=[8, 5])
plt.hist(data = df_loan, x = 'DebtToIncomeRatio', bins =bins)
plt.xlabel('The debt to income ratio')
plt.xlim(0,2)
plt.show()<jupyter_output><empty_output><jupyter_text>The debt to income ratio is right-skewed and it indicates that the majority of borrowers have a debt that is smaller than their income.<jupyter_code># Distribution of income range
fig, ax = plt.subplots(figsize = [20,10])
default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, x = 'IncomeRange', color = default_color)
plt.xlabel('Income Range')
plt.show()<jupyter_output><empty_output><jupyter_text>Most of borrowers earn between more than \$25,000, and most of borrowers earn in range between \\$25,000 and \\$49,999.<jupyter_code># Distribution of Stated Monthly Income 
x_ticks = [500, 1000, 5000, 10000, 50000, 100000]
log_binsize = 0.04
bins = 10 ** np.arange(1, np.log10(df_loan['StatedMonthlyIncome'].max())+log_binsize, log_binsize)
plt.figure(figsize=[8, 5])
plt.hist(data = df_loan, x = 'StatedMonthlyIncome', bins =bins)
plt.xlabel('Stated Monthly Income')
plt.xscale('log')
plt.xticks(x_ticks, x_ticks)
plt.xlim(400,80000)
plt.show()
print(df_loan['StatedMonthlyIncome'].mean())<jupyter_output>5608.025568224836<jupyter_text>The distribution looks normal distribution, with a mean monthly income of $5608.02.<jupyter_code># Distribution of Loans initial amount:
x_ticks = [1000, 2000,4000, 10000,15000,20000]
bins = np.arange(0, df['LoanOriginalAmount'].max()+500, 500)
plt.figure(figsize=[8, 5])
plt.hist(data = df_loan, x = 'LoanOriginalAmount', bins =bins)
plt.xlabel('Loan Original Amount')
plt.xlim(800,30000)
plt.show();<jupyter_output><empty_output><jupyter_text>It is possible to notice some peaks of values in the distribution around \$4,000, \\$10,000 and \\$15,000.<jupyter_code># Distribution of Monthly Loan Payment 
x_ticks = [10, 20, 50, 100, 200, 500, 1000, 2000]
log_binsize = 0.025
bins = 10 ** np.arange(1, np.log10(df_loan['MonthlyLoanPayment'].max())+log_binsize, log_binsize)
plt.figure(figsize=[8, 5])
plt.hist(data = df_loan, x = 'MonthlyLoanPayment', bins =bins)
plt.xlabel('Monthly Loan Payment')
plt.xscale('log')
plt.xticks(x_ticks, x_ticks)
plt.show()
print(df_loan['MonthlyLoanPayment'].mean())
print(df_loan['MonthlyLoanPayment'].mode())<jupyter_output>272.47578310823104
0    173.71
dtype: float64<jupyter_text>The distribution a little bimodal, with a peak between \$100 and \\$200.<jupyter_code># Distribution of number of terms:
fig, ax = plt.subplots(figsize = [20,10])
default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, x = 'Term', color = default_color)
plt.xlabel('Number of terms')
plt.show()<jupyter_output><empty_output><jupyter_text>Most loans are 36 months long.<jupyter_code># Distribution of ratings
fig, ax = plt.subplots(figsize = [20,10])
default_color = sb.color_palette()[0]
sb.countplot(data = df_loan, x = 'ProsperRating', color = default_color)
plt.xlabel('Prosper Ratings')
plt.show()<jupyter_output><empty_output><jupyter_text>Most of loans had a rating of C. The distribution assimilates with a normal distribution.<jupyter_code># Distribution of lander yield 
bins = np.arange(0, df_loan['EstimatedReturn'].max()+0.003, 0.003)
plt.figure(figsize=[8, 5])
plt.hist(data = df_loan, x = 'EstimatedReturn', bins =bins)
plt.xlabel('Estimated Return')
plt.xlim(0,0.2)
plt.show()
print(df_loan['EstimatedReturn'].mean())
print(df_loan['EstimatedReturn'].mode())<jupyter_output>0.09606829611209916
0    0.1246
dtype: float64<jupyter_text>The distribution looks right-skewed, with a mean return of 9.60% and a mode of 12.46%. As it was observed a small number of borrowers with really high delinquencies, those will be not considered during the further analysis. From here on, it will be only considered borrowers up to 15 deliquencies.<jupyter_code>df_loan = df_loan[df_loan['CurrentDelinquencies'] <=15]<jupyter_output><empty_output><jupyter_text>Discuss the distribution(s) of your variable(s) of interest. Were there any unusual points? Did you need to perform any transformations?* The variables of interest display that most loans are current and those who do not pay have less than 5 delinquencies.* Most borrowers have no past delinquencis in 7 years, are employed, have a low debt to income ratio, earning more than 25,000.* The loan initial amount have peaks around \$4,000, \\$10,000 and \\$15,000, and the most frequent loan payments are between 100 and 200.It was not observed any unusual point and log transformation was needed for income ranges.  Of the features you investigated, were there any unusual distributions? Did you perform any operations on the data to tidy, adjust, or change the form of the data? If so, why did you do this?The features investigated displayed usual distributions, however in many of them it is possible to observe some peaks. No operations were needed in order change data, apart from the motifications of data types in some variables.  Bivariate Exploration As an initial step, it will be observed the correlation between the different variables in the data.<jupyter_code>numeric_vars = ['BorrowerRate', 'LenderYield',
       'EstimatedEffectiveYield', 'EstimatedLoss', 'EstimatedReturn',
       'CurrentDelinquencies', 'AmountDelinquent',
       'DelinquenciesLast7Years', 'DebtToIncomeRatio', 
       'StatedMonthlyIncome', 'LoanOriginalAmount', 'MonthlyLoanPayment']
categoric_vars = ['Term', 'LoanStatus', 'ProsperRating', 'ProsperScore', 'Occupation', 'EmploymentStatus',
       'IsBorrowerHomeowner','IncomeRange',]

# correlation plot
plt.figure(figsize = [8, 5])
sb.heatmap(df_loan[numeric_vars].corr(), annot = True, fmt = '.3f',
           cmap = 'vlag_r', center = 0)
plt.show()
# plot matrix considers a sample of 500 loans in order to render faster
samples = np.random.choice(df_loan.shape[0], 500, replace = False)
loans_samp = df_loan.reindex(samples)
g = sb.PairGrid(data = loans_samp, vars = numeric_vars)
g = g.map_diag(plt.hist, bins = 20);
g.map_offdiag(plt.scatter);<jupyter_output><empty_output><jupyter_text>From the two plots above, we can clearly verify that the borrower rate is related with the lender yield (as expected).  Also, higher rates are applied to those with more delinquencies.  The borrower rate has a wide range for small loans (original amount), however the values tend to be lower for higher amounts.<jupyter_code># Understand relation between LoanStatus and number of Terms
plt.figure(figsize=(15,10))
sb.countplot(data = df_loan, y = 'LoanStatus', hue = 'Term')
plt.ylabel('Loan Status')
plt.xlabel('Count')
plt.legend(framealpha = 1, title = 'Number of Terms');
df_loan.Term.value_counts()<jupyter_output><empty_output><jupyter_text>The majority of completed and current loans are of 36 terms. Current loans also have a significant amount of 60 terms loans.<jupyter_code>def cat_scatterplots(dataframe, feature_a, feature_b, feature_a_label, feature_b_label, limx_min = None, 
                     limx_max = None, limy_min = None, limy_max = None):
    plt.figure(figsize=[8, 5])
    plt.scatter(data = dataframe, x = feature_a, y = feature_b, alpha = 1/8)
    plt.xlabel(feature_a_label)
    plt.ylabel(feature_b_label)
    plt.xlim(limx_min, limx_max)
    plt.ylim(limy_min, limy_max);
# Understand relation between LoanStatus and BorrowerRate
cat_scatterplots(df_loan, 'BorrowerRate', 'LoanStatus', 'Loan Status', 'Borrower Rate')<jupyter_output><empty_output><jupyter_text>It is possible to notice that current loans have a range of rates between 0.05 and 0.35. Past due loans seem to have a higher rate, with values closer to 0.2 and 0.3.<jupyter_code># Understand relation between LoanStatus and EmploymentStatus
plt.figure(figsize=(10,10))
sb.countplot(data = df_loan.query('LoanStatus == ["Chargedoff","Completed", "Current"]'), x = 'LoanStatus', hue = 'EmploymentStatus')
plt.xlabel('Loan Status')
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Employment Status')
plt.xlim(0.5,3.5);<jupyter_output><empty_output><jupyter_text>Most current loans are made by employed borrowers.<jupyter_code># Understand relation between LoanStatus and EmploymentStatus
plt.figure(figsize=(10,10))
sb.countplot(data = df_loan.query('LoanStatus != ["Chargedoff","Completed", "Current"]'), x = 'LoanStatus', hue = 'EmploymentStatus')
plt.xlabel('Loan Status')
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Employment Status')
plt.xlim(5.5,12)
plt.ylim(0,700)
plt.xticks(rotation=15);<jupyter_output><empty_output><jupyter_text>Most past due are made by employed borrowers.<jupyter_code># Understand relation between LoanStatus and IsBorrowerHomeowner
plt.figure(figsize=(15,10))
sb.countplot(data = df_loan, x = 'LoanStatus', hue = 'IsBorrowerHomeowner')
plt.xlabel('Loan Status')
plt.xticks(rotation=15)
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Homeownership');<jupyter_output><empty_output><jupyter_text>A higher percentage of current loans are made by home owners, but the past due loans have a similar percentage of homeownership.<jupyter_code># Understand relation between LoanStatus and CurrentDelinquencies
cat_scatterplots(df_loan, 'CurrentDelinquencies', 'LoanStatus', 'Current Delinquencies', 'Loan Status', 0, 15)<jupyter_output><empty_output><jupyter_text>The number of current delinquencies does not seem to influence the loan payment, as there is a wide range of number of delinquencies and past due loans do not necessary have a high number.<jupyter_code># Understand relation between LoanStatus and DelinquenciesLast7Years
cat_scatterplots(df_loan, 'DelinquenciesLast7Years', 'LoanStatus', 'Delinquencies Last 7 Years', 'Loan Status')<jupyter_output><empty_output><jupyter_text>Similarly to the observed for the number of delinquencies, the number of delinquencies in the last 7 years have a wide range of values and do not present any special indication for the loan status.<jupyter_code># Understand relation between LoanStatus and DebtToIncomeRatio
cat_scatterplots(df_loan, 'DebtToIncomeRatio', 'LoanStatus', 'Debt To Income Ratio', 'Loan Status',0, 1.5)
default_color = sb.color_palette()[0]
sb.boxplot(data = df_loan[df_loan['DebtToIncomeRatio']<1], y = 'DebtToIncomeRatio', x = 'LoanStatus', color = default_color)
plt.xticks(rotation = 25);<jupyter_output><empty_output><jupyter_text>The debt to income ratio does not present any special indication to the loan status.<jupyter_code># Understand relation between LoanStatus and IncomeRange
plt.figure(figsize=(15,10))
sb.countplot(data = df_loan.query('LoanStatus == ["Chargedoff","Completed", "Current"]'), x = 'LoanStatus', hue = 'IncomeRange')
plt.xlabel('Loan Status')
plt.xticks(rotation=15)
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Income Range')
plt.xlim(0.5,3.5);<jupyter_output><empty_output><jupyter_text>The majority of current loans are made by individual with an income range of \$50,000 - \\$75,999.<jupyter_code># Understand relation between LoanStatus and IncomeRange
plt.figure(figsize=(15,10))
sb.countplot(data = df_loan.query('LoanStatus != ["Chargedoff","Completed", "Current"]'), x = 'LoanStatus', hue = 'IncomeRange')
plt.xlabel('Loan Status')
plt.xticks(rotation=15)
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Income Range')
plt.xlim(5.5,12)
plt.ylim(0,280);<jupyter_output><empty_output><jupyter_text>But if it is regarded the past due loans, most are made by individual with an income range between \$25,000 - \\$49,999.<jupyter_code># Understand relation between LoanStatus and StatedMonthlyIncome
cat_scatterplots(df_loan, 'StatedMonthlyIncome', 'LoanStatus', 'Stated Monthly Income', 'Loan Status',0, 60000)<jupyter_output><empty_output><jupyter_text>It is possible to notice that the past due loan are mostly made by individuals with a lower stated monthly income.<jupyter_code># Understand relation between LoanStatus and LoanOriginalAmount
cat_scatterplots(df_loan, 'LoanOriginalAmount', 'LoanStatus', 'Loan Original Amount', 'Loan Status')<jupyter_output><empty_output><jupyter_text>Past due loans have a high frequency in amount around \$4,000, \\$10,000 and \\$15,000.<jupyter_code># Understand relation between LoanStatus and MonthlyLoanPayment
cat_scatterplots(df_loan, 'MonthlyLoanPayment', 'LoanStatus', 'Monthly Loan Payment', 'Loan Status', 0, 1500)<jupyter_output><empty_output><jupyter_text>Past due loans have a high frequency of occurrences in values around monthly loan payments of 200.<jupyter_code>samples = np.random.choice(df_loan.shape[0], 1000, replace = False)
loans_samp = df_loan.reindex(samples)
categoric_vars = ['LoanStatus', 'IncomeRange', 'CurrentDelinquencies']

fig, ax = plt.subplots(ncols = 2, nrows = 3 , figsize = [20,20])

for i in range(len(categoric_vars)):
    var = categoric_vars[i]
    sb.violinplot(data = loans_samp, x = var, y = 'MonthlyLoanPayment', ax = ax[i,0],
               color = default_color)
    ax[i,0].set_xticklabels(ax[i,0].get_xticklabels(), rotation=45)
    sb.violinplot(data = loans_samp, x = var, y = 'BorrowerRate', ax = ax[i,1],
               color = default_color)
    ax[i,1].set_xticklabels(ax[i,1].get_xticklabels(), rotation=45)
fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>* Past due loans have monthly payments around 200 and they present higher borrowers rates.* Higher income ranges have some higher values of monthly payment, however the most frequent values are similar for all income ranges. The most frequent values of borrower rate present a decreasing tendency as the income increases.* The most frequent monthly payments values are similar for most number of delinquencies, however those with zero delinquencies present higher monthly payments. The borrower rate has an increasing trend according to the number of delinquencies up to 4 delinquencies.<jupyter_code>plt.figure(figsize=(15,10))
sb.countplot(data = df_loan, x = 'CurrentDelinquencies', hue = 'EmploymentStatus')
plt.xlabel('Current deliquencies')
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Employment status')
plt.xlim(-0.5,5.5)
plt.ylim(0,60000);<jupyter_output><empty_output><jupyter_text>The plot is a little messy, but we are interested in those with 0 delinquencies, which means the ones that have everything payed. Analyzing the plot, it is possible to observe that the majority people with 0 delinquencies are those employed, full-time and self-employed.This confirms the initial hypothesis that having a job is a high indication that the person is likely to pay the loan.<jupyter_code>plt.figure(figsize=(15,10))
sb.countplot(data = df_loan, x = 'CurrentDelinquencies', hue = 'IsBorrowerHomeowner')
plt.xlabel('Current deliquencies')
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Home ownership')
plt.xlim(-0.5,5.5);<jupyter_output><empty_output><jupyter_text>This plot depicts that home ownership is has almost the same percentage among those with no delinquencies. Again, this probably means that home ownership has no influence on wether someone will pay or not the loan.<jupyter_code>def cat_regplots(dataframe, feature_a, feature_b, feature_a_label, feature_b_label, limx_min = None, 
                     limx_max = None, limy_min = None, limy_max = None):
    plt.figure(figsize=[8, 5])
    sb.regplot(data = dataframe, x = feature_a, y = feature_b, fit_reg = False,
               x_jitter = 0.2, y_jitter = 0.2, scatter_kws = {'alpha': 1/8})
    plt.xlabel(feature_a_label)
    plt.ylabel(feature_b_label)
    plt.xlim(limx_min, limx_max)
    plt.ylim(limy_min, limy_max);
# Understand the relation between number of delinquencies and borrower rate
cat_regplots(df_loan, 'BorrowerRate', 'CurrentDelinquencies', 'Borrower Rate', 'Current Delinquencies',  
                 limy_min = 0, limy_max = 15)<jupyter_output><empty_output><jupyter_text>Individuals with no delinquencies have a wide range of rates. However, as the number of delinquencies increases, the rate tends to be higher.<jupyter_code># Understand the relation between number of delinquencies and loan amount
cat_regplots(df_loan, 'LoanOriginalAmount', 'CurrentDelinquencies', 'Loan Amount', 'Current Delinquencies',  
                 limy_min = 0, limy_max = 15)<jupyter_output><empty_output><jupyter_text>It is possible to notice a higher frequency of higher loan amounts for the individuals with a smaller number of current delinquencies.<jupyter_code># Understand a possible relation between the number of delinquencies and income
cat_regplots(df_loan, 'StatedMonthlyIncome', 'CurrentDelinquencies', 'Stated Monthly Income', 'Current Delinquencies',  
                 limx_min = 0, limx_max = 30000, limy_min = 0, limy_max = 15)<jupyter_output><empty_output><jupyter_text>It is possible to observe that higher the income, smaller the number of delinquencies tend to be.<jupyter_code># Understand the variation if there is any link between a specific prosper rating with smaller number of deliquencies
plt.figure(figsize=(15,10))
sb.countplot(data = df_loan, x = 'CurrentDelinquencies', hue = 'ProsperRating')
plt.xlabel('Current Delinquencies')
plt.ylabel('Count')
plt.legend(framealpha = 1, title = 'Prosper Rating')
plt.xlim(-0.5,5.5);<jupyter_output><empty_output><jupyter_text>It is possible to notice that individual with no delinquencies have all different  ratings. However, there is a higher number of individuals with rating C, B and A.<jupyter_code># Understand the variation of amount deliquent according to the income
cat_regplots(df_loan, 'StatedMonthlyIncome', 'AmountDelinquent', 'Stated Monthly Income', 'Amount Delinquent',  
                 limx_min = 0, limx_max = 30000, limy_min = 0, limy_max = 15000)<jupyter_output><empty_output><jupyter_text>In the previous plot we can observe again that those individuals with smaller income are the ones with a higher amount delinquent.<jupyter_code># Understand the variation of Monthly Loan Payment according to the Stated Monthly Income
plt.hist2d(data = df_loan[(df_loan['StatedMonthlyIncome']<10000)&(df_loan['MonthlyLoanPayment']<2000)], x = 'StatedMonthlyIncome', y = 'MonthlyLoanPayment', cmap = 'viridis_r')
plt.colorbar()
plt.xlabel('Stated Monthly Income')
plt.ylabel('Monthly Loan Payment')
plt.xlim(0,10000);<jupyter_output><empty_output><jupyter_text>It is possible to notice that there is a high frequency of individual with a monthly income between \\$2000 and \\$3000 who pay a value up to \\$200.<jupyter_code># Understand the variation of Lender Yield according to the Loan Original Amount
cat_scatterplots(df_loan, 'LenderYield', 'LoanOriginalAmount', 'Lender Yield', 'Loan Original Amount',  
                 limx_min = 0, limx_max = 0.35)<jupyter_output><empty_output><jupyter_text>The lender yield seems to be higher for loan amounts smaller than $10,000.<jupyter_code># Understand the income monthly payment distrbution according to the income range
plt.figure(figsize=(15,10))
base_color = sb.color_palette()[3]
sb.violinplot(data = df_loan, x = 'IncomeRange', y = 'MonthlyLoanPayment', color = base_color, inner='quartile')
plt.xlabel('Income Range')
plt.ylabel('Monthly Loan Payment');<jupyter_output><empty_output><jupyter_text>Those with a higher income range have a higher mean monthly loan payment. Also, it is possible to notice an increase in the monthly loan payment as the income range increases.<jupyter_code># Understand the income monthly payment distrbution according to the Stated Monthly Income
x_ticks = [500, 1000, 5000, 10000,20000, 50000,100000]
plt.figure(figsize=[8, 5])
plt.scatter(data = df_loan, x = 'StatedMonthlyIncome', y = 'MonthlyLoanPayment',  alpha = 1/8)
plt.xlabel('Stated Monthly Income')
plt.ylabel('Monthly Loan Payment')
plt.xscale('log')
plt.xticks(x_ticks, x_ticks)
plt.xlim(500,100000);<jupyter_output><empty_output><jupyter_text>The same pattern as before can be observed if the stated monthly income is regarded instead of income range. There is an increase in the monthly loan payment as the stated monthly income rises. Also, it is possible to notice a higher number of loans with monthly payment bellow $500 for the majority of stated monthly incomes.<jupyter_code># Understand the Debt To Income Ratio distribution according to the Stated Monthly Income
x_ticks = [500, 1000, 5000, 10000,20000, 50000,100000]
plt.figure(figsize=[8, 5])
plt.scatter(data = df_loan, x = 'StatedMonthlyIncome', y = 'DebtToIncomeRatio',  alpha = 1/8)
plt.xlabel('Stated Monthly Income')
plt.ylabel('Debt To Income Ratio')
plt.xscale('log')
plt.xticks(x_ticks, x_ticks)
plt.xlim(500,100000)
plt.ylim(0,4);<jupyter_output><empty_output><jupyter_text>It is possible to notice a negative trend between debt to income ratio and stated monthly income. This means that people with higher income usually have a loan value that is less significant when compared to their total income.  Talk about some of the relationships you observed in this part of the investigation. How did the feature(s) of interest vary with other features in the dataset?Analyzing the previous plots, it is possible to notice that loans of 36 terms are the ones which are mostly completed and current. This could possibly be an indication of the status, however as it is the option with higher number of borrowers, it is not possible to state without any further investigation. Other factors that seem not to imply the loan status are the employment status and homeownership. Both "current" and "past due" loans are mainly made by employed individuals and homeownership is not highly predominant in any loan status.  The number of current delinquencies and delinquencies in the past seven years take a wide range of values and does not present any specific relation to the loan status. The debt to income ratio does not present any special relationship with the loan status, but the income range which is more common in the the "past due" loans is the range between \$25,000 - \\$49,999, indicating that lower income might be an indication of the loan status. The latter is confirmed with the analysis of the distribution of the stated monthly income, in which is noticed a higher presence of individuals with lower income with a "past due" loan.  Also, it was possible to notice that loans have higher borrowers rates for those that are "past due". Another interesting fact is that "past due" loans have a high frequency of loans amount of \\$4,000, \\$10,000 and \\$15,000, and the most frequent monthly loan payments are around \\$200. Did you observe any interesting relationships between the other features (not the main feature(s) of interest)?Some interesting and expected relationships were confirmed: a higher number of delinquencies implies in a higher borrower rate, higher income ranges present higher values of monthly loan payment, higher the incomes tend to have a smaller  number of delinquencies. Also, the majority of individuals with 0 delinquencies are those employed, full-time and self-employed, indicating that employment could influence the payment of the loan. This fact is reinforced by those individuals with smaller income presenting higher delinquent amounts.   Also, as expected, it is possible to notice a higher frequency of higher loan amounts by individuals who have a smaller number of current delinquencies, and it is possible to notice a negative trend between debt to income ratio and stated monthly income, meaning that people with higher income usually ask for a loan amount that is less significant to their total income value.   Differently from what was supposed in the beginning, homeownership has no influence on wether someone will pay or not the loan, and the lender yield seems to be higher for loan amounts smaller than \\$10,000, indicating that higher loan amounts do not lead to higher yields.  Multivariate ExplorationIn the multivariate explaration, my main objective in to explore the relation between Loan Status, Borrower Rate, Income Range, Loan Orignal Amount and Monthly Loan Payment.<jupyter_code># Creating a new dataframe composed of LoanStatus of interest
df_loan_stat = df_loan.copy()
df_loan_stat['LoanStatus'] = df_loan_stat['LoanStatus'].astype('str')
stat = ['Completed','Current', 'Past Due (1-15 days)','Past Due (16-30 days)', 'Past Due (31-60 days)','Past Due (61-90 days)','Past Due (91-120 days)','Past Due (>120 days)']
df_loan_stat = df_loan_stat.loc[df_loan_stat['LoanStatus'].isin(stat)]
df_loan_stat['LoanStatus'] = df_loan_stat['LoanStatus'].astype('category')
df_loan_stat.to_csv('Prosper_loan_clean.csv',index=False)
# Understand the relation between LoanStatus, LoanOriginalAmount and BorrowerRate
g = sb.FacetGrid(data = df_loan_stat, col = 'LoanStatus', size = 4, col_wrap = 4)
g.map(plt.scatter, 'BorrowerRate','LoanOriginalAmount', alpha = 1/8)
g.set_ylabels('Loan Original Amount')
g.set_xlabels('Borrower Rate')
g.set_titles('{col_name}');<jupyter_output><empty_output><jupyter_text>It is possible to observe the completed and current loans have original amounts and borrower rates in a wide range of values. However, if regarded the past due loans, it is possible to observe a higher frequency of loans around \$4,000, \\$10,000 and \\$15,000 and higher rates, around 0.3.<jupyter_code># Understand the relation between LoanStatus, MonthlyLoanPayment and BorrowerRate
g = sb.FacetGrid(data = df_loan_stat, col = 'LoanStatus', size = 4, col_wrap = 4)
g.map(plt.scatter, 'BorrowerRate','MonthlyLoanPayment', alpha = 1/8)
g.set_ylabels('Monthly Loan Payment')
g.set_xlabels('Borrower Rate')
g.set_titles('{col_name}');<jupyter_output><empty_output><jupyter_text>It is possible to notice that for current and completed loans there s an increasing trend between borrower rate and monthly loan payment. This trend can be observed in past due loans, however it is less intense. Also, it is possible to notice that past due loans usually have lower monthly loan payments and rates around 0.3.<jupyter_code># Understand the relation between LoanStatus, MonthlyLoanPayment and LoanOriginalAmount
g = sb.FacetGrid(data = df_loan_stat, col = 'LoanStatus', size = 4, col_wrap = 4)
g.map(plt.scatter, 'LoanOriginalAmount','MonthlyLoanPayment', alpha = 1/8)
g.set_xlabels('Loan Original Amount')
g.set_ylabels('Monthly Loan Payment')
g.set_titles('{col_name}');<jupyter_output><empty_output><jupyter_text>As expected, higher loan original amounts have higher monthly payments. Another noticeable fact is that past due loans have lower loan original amount values.<jupyter_code># Understand the relation between LoanStatus, IncomeRange and LoanOriginalAmount
plt.figure(figsize=[15,5])
sb.boxplot(data = df_loan_stat, x = 'LoanStatus', y = 'LoanOriginalAmount', hue = 'IncomeRange')
plt.xlabel('Loan Status')
plt.ylabel('Loan Original Amount')
plt.xticks(rotation=20)
plt.legend(title = 'Income Range', loc =0);<jupyter_output><empty_output><jupyter_text>It is verified an increasing trend in the median of loan original amounts as the income range increases. This pattern can be observed in most loans status.<jupyter_code># Understand the relation between LoanStatus, BorrowerRate and IncomeRange
plt.figure(figsize=[15,5])
sb.boxplot(data = df_loan_stat, x = 'LoanStatus', y = 'BorrowerRate', hue = 'IncomeRange')
plt.xlabel('Loan Status')
plt.ylabel('Borrower Rate')
plt.xticks(rotation=20)
plt.legend(title = 'Income Range', loc =0);<jupyter_output><empty_output><jupyter_text>It is verified a decreasing trend in the borrower rate as the income range increases. We can also observe higher median borrower rates for past due loans and that this rate is higher for the income range \$1 - \\$24,999 with loans past due over 60 days.<jupyter_code># Understand the relation between LoanStatus, MonthlyLoanPayment and IncomeRange
plt.figure(figsize=[15,5])
sb.boxplot(data = df_loan_stat, x = 'LoanStatus', y = 'MonthlyLoanPayment', hue = 'IncomeRange')
plt.xlabel('Loan Status')
plt.ylabel('Monthly Loan Payment')
plt.xticks(rotation=20)
plt.ylim(0,1500)
plt.legend(title = 'Income Range', loc =0);<jupyter_output><empty_output><jupyter_text>It is verified an increasing trend in the median monthly loan payment for increasing income ranges. It is also possible to verify that past due loans have lower monthly loan payments.Another interesting fact to be pointed out on the last three plots is that the Income range classified as "Not displayed" is only present in "Completed and "Past Due (>120 days)".<jupyter_code>fig = plt.figure(figsize = [8,6])
ax = sb.pointplot(data = df_loan_stat, x = 'LoanStatus', y = 'MonthlyLoanPayment', hue = 'IncomeRange',
           palette = 'Blues', linestyles = '', dodge = 0.4)
plt.title('Monthly Loan Payments across Income Ranges and Loan Status')
plt.ylabel('Monthly Loan Payments ($)')
plt.xticks(rotation=35)
plt.show();<jupyter_output><empty_output><jupyter_text>Again, it is verified an increasing central tendency between the monthly loan payments and higher income ranges. It is also noticeable a high variability in the monthly payment for past due loans.<jupyter_code># Understand the realtion between MonthlyLoanPayment, LoanOriginalAmount and BorrowerRate
plt.figure(figsize = (8,7))
plt.scatter(data = df_loan_stat, y = 'LoanOriginalAmount', x = 'MonthlyLoanPayment', c = 'BorrowerRate', alpha = 1/2)
plt.colorbar(label = 'Borrower Rate')
plt.ylabel('Loan Original Amount')
plt.xlabel('Monthly Loan Payment');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09743195376584639, 'fraction_numerical': 0.039149888143176735, 'mean_word_length': 4.7065159574468085, 'pattern_counts': {'":': 0, '<': 218, '<?xml version=': 0, '>': 219, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '54', 'dir': 'jupyter-structured-clean-dedup', 'id': '219413', 'license': 'MIT', 'n_tokens_mistral': 14665, 'n_tokens_neox': 12841, 'n_words': 4371, 'path': 'Prosper_Loan_Data.ipynb', 'repo_name': 'dsllt/Udacity_DA_P5', 'size': '42919'}