<jupyter_start><jupyter_text>Prosper Loan Data Exploration  by <NAME> 1- Preliminary Wrangling> This data set contains 113,937 loans with 81 variables on eachloan, including loan amount, borrower rate (or interest rate),current loan status, borrower income, and many others.  a) Import important libraries<jupyter_code># import all packages and set plots to be embedded inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb

%matplotlib inline<jupyter_output><empty_output><jupyter_text>b) Dataset loading and observation<jupyter_code>#load data set
df=pd.read_csv("prosperLoanData.csv")
df.head(2)
#observe certain columns
df['Occupation']
#observe dataset dimensionlity
df.shape<jupyter_output><empty_output><jupyter_text>c) Data cleaning<jupyter_code>#first  make a dataset of interest
prosper=df[['Investors','LoanOriginalAmount','MonthlyLoanPayment','Recommendations','PercentFunded','TotalTrades','IncomeRange','StatedMonthlyIncome','CurrentlyInGroup','IsBorrowerHomeowner','Occupation','EmploymentStatus','ListingCategory (numeric)','BorrowerRate','BorrowerAPR','LoanStatus','Term']]
prosper.head(2)

# observe dataset of interest information
prosper.info()

# observe dataset of interest dimensionality
prosper.shape
#drop missing values
prosper.dropna(inplace=True)
# assure there is no missing data
prosper.isna().any().any()<jupyter_output><empty_output><jupyter_text>What is the structure of your dataset?113937 rows and 17 columns What is/are the main feature(s) of interest in your dataset? features wrapped relationships between borrower and loan. What features in the dataset do you think will help support your investigation into your feature(s) of interest?The features that describe the borrower income ,employment status and what kind of loan he needs.  Univariate Explorationwe look to data distributions and frequency seperately  a) what does borrower monthly income look like ?<jupyter_code># make function to be called whenever needed to change the labels
def custom_labels(x_label,y_label,title):
    plt.title(title,fontsize=20)
    plt.xlabel(x_label,fontsize=14)
    plt.ylabel(y_label,fontsize=14)
# the log  base 10 of income values is under 6.25
np.log10(prosper['StatedMonthlyIncome'].describe())
# apply logarithmics transformation
bins= 10**np.arange(3,6.25,0.1)
ticks=[100,500,1000,3000,8000,50000]
labels=['{}'.format(v) for v in ticks]
prosper['StatedMonthlyIncome'].hist(figsize=(10,5), bins=bins);
plt.xscale('log')
# use our function to represent current graph labels
custom_labels('Income','count','Monthly Income')
plt.xticks(ticks,labels);<jupyter_output><empty_output><jupyter_text>> the most frequent monthly income is 3K-8K  b) what is frequent income range of the borrower?<jupyter_code>#distribution using Seaborn bar plot 
plt.figure(figsize=(10,5));
sb.countplot(data = prosper, y='IncomeRange',color=sb.color_palette()[0]);
plt.title('Prosper Borrower Incom Range');<jupyter_output><empty_output><jupyter_text>> the most frequent income range is 25K-50K and 50K-75K  c) what is the borrower employment status?<jupyter_code># barplot for employment status
plt.figure(figsize=(10,5));

sb.countplot(data = prosper, y='EmploymentStatus',color=sb.color_palette()[0],order=prosper['EmploymentStatus'].value_counts().index);
plt.title('Prosper Borrower Employment Status');<jupyter_output><empty_output><jupyter_text>> most of borrowers have to be employed or have full time job  c) what the distribution of borrower reputation?<jupyter_code># describe rate values
prosper['BorrowerRate'].describe()
# histogram plot of borrower rate
(prosper['BorrowerRate']*10).hist(figsize=(10,5),bins=10*np.arange(0,0.5,.01));
#manage labels
custom_labels('Rate','Count',' Borrowe Rate')<jupyter_output><empty_output><jupyter_text>> the distribution is skewed to the right,the majority of borrower rate is from 1 to 2 on a scale from 0 to 5  d) what is the loan amount distribution?<jupyter_code>#observe loan amount statistics after scaling to logarithmic scale
np.log10(prosper['LoanOriginalAmount'].describe())
# apply logarithmics transformation
plt.figure(figsize=(10,5))
bins= 10**np.arange(2,6,0.1)
ticks=[1000,2000,4000,10000,20000,40000,100000]
labels=['{}'.format(v) for v in ticks]
prosper['LoanOriginalAmount'].hist( bins=bins);
#manage labels
custom_labels('Amount','Count','Loan Original Amount')
plt.xscale('log')
plt.xticks(ticks,labels);<jupyter_output><empty_output><jupyter_text>> the most frequent loan amount is between 4K to 10K  e) what is loan duration ?<jupyter_code># barplot for loan duration
plt.figure(figsize=(10,5));
sb.countplot(data = prosper, x='Term',color=sb.color_palette()[0],order=prosper['Term'].value_counts().index);
#manage labels
custom_labels('Months','Count','Loan duration (months)')<jupyter_output><empty_output><jupyter_text>> duration is calculated by months and there are 3 options 36 which is frequent, 60 and 12 months  d) what does monthly payment look like?<jupyter_code>#monthly payment statistics 
prosper['MonthlyLoanPayment'].describe()
# statistics after logarithmics transformation
np.log10(prosper['MonthlyLoanPayment'].describe())
# apply logarithmics transformation
plt.figure(figsize=(10,5));
bins= 10**np.arange(-1,5+0.1,0.1)
ticks=[10,50,100,500,1500]
labels=['{}'.format(v) for v in ticks]
prosper['MonthlyLoanPayment'].hist(bins=bins);

#manage labels
custom_labels('Payment',
'count','Monthly Payment')


plt.xscale('log')
plt.xticks(ticks,labels);<jupyter_output><empty_output><jupyter_text>> the most frequent monthyl amount is 100-500 $  Of the features you investigated, were there any unusual distributions? Did you perform any operations on the data to tidy, adjust, or change the form of the data? If so, why did you do this?> the distribution of monthly payment , original loan amount and borrower monthly incom has unusual distributions, so we need to make tha logarithmic scale, I did so because these are financial data which has to be handeled in scaling  Bivariate Exploration> we investigate relations between two feature.<jupyter_code># remove outlier
Q1 = prosper.quantile(0.25)
Q3 = prosper.quantile(0.75)
IQR = Q3 - Q1
prosper_copy= prosper[~((prosper < (Q1 - 1.5 * IQR)) |(prosper > (Q3 + 1.5 * IQR))).any(axis=1)]<jupyter_output><empty_output><jupyter_text>a) what is the relationship between original amount and loan monthly payment?<jupyter_code>#scatter plot between original amount and loan monthly payment
plt.figure(figsize=(10,5))
plt.scatter(data=prosper_copy,x='LoanOriginalAmount',y='MonthlyLoanPayment');
#manage labels
custom_labels('original amount',
'monthly payment','Relationship between borrower monthly payment and loan amount')<jupyter_output><empty_output><jupyter_text>> there is a strong positive linear relationship between original amount and loan monthly payment?  b) what is the relationship between borrower  monthly payment and monthly income ?<jupyter_code>#scatter plot between borrower monthly payment and monthly income
plt.figure(figsize=(10,5))
plt.scatter(data=prosper_copy,x='MonthlyLoanPayment',y='StatedMonthlyIncome');
#manage labels
custom_labels('monthly payment',
'monthly income','Relationship between borrower monthly payment and monthly income')<jupyter_output><empty_output><jupyter_text>> there no linear Relationship between borrower monthly payment and monthly income  c) what is the relationship between borrower monthly income and loan amount ?<jupyter_code>#scatter plot between borrower monthly income and loan amount
plt.figure(figsize=(10,5))
plt.scatter(data=prosper_copy,x='StatedMonthlyIncome',y='LoanOriginalAmount');
#manage labels
custom_labels('monthly income',
'original amount','Relationship between borrower monthly income and loan amount')<jupyter_output><empty_output><jupyter_text>> there no linear Relationship between borrower monthly income and loan amount  d) what is the relationship between Borrower Rate and Percent Funded?<jupyter_code>#scatter plot between Borrower Rate and Percent Funded
plt.figure(figsize=(10,5))
plt.scatter(data=prosper_copy,x='BorrowerRate',y='PercentFunded');
#manage labels
custom_labels('Borrower Rate',
'Percent Funded','Relationship between Borrower Rate and Percent Funded')<jupyter_output><empty_output><jupyter_text>>there is no Relationship between Borrower Rate and Percent Funded  e) what is the relationship between Percent Funded and Original Amount?<jupyter_code>#scatter plot between Percent Funded and Original Amount 
plt.figure(figsize=(10,5))
plt.scatter(data=prosper_copy,x='PercentFunded',y='LoanOriginalAmount');
#manage labels
custom_labels('Percent Funded',
'Original Amount','Relationship Percent Funded and Original Amount')<jupyter_output><empty_output><jupyter_text>> there is no relationship between Percent Funded and Original Amount  Talk about some of the relationships you observed in this part of the investigation. How did the feature(s) of interest vary with other features in the dataset?> we found a strong positive linear relationship between borrower monthly payment and loan amount> ,no linear Relationship between borrower monthly payment and monthly income> and no linear Relationship between borrower monthly income and loan amount Did you observe any interesting relationships between the other features (not the main feature(s) of interest)?> no, there is a no  Relationship between Original Amount and Percent Funded and between Borrower Rate and Percent Funded  Multivariate Exploration> plots of three or more variables to investigate the data  a) How being a home owner borrower affects loan status?<jupyter_code>plt.figure(figsize=(20,10))
plt.subplot(1,2,1)
fig1=sb.countplot(x='LoanStatus',data=prosper[prosper['IsBorrowerHomeowner']==True],color=sb.color_palette()[0],
                 order=prosper['LoanStatus'].value_counts().index)
custom_labels('Loan Status','count','Borrower is a home owner')
plt.xticks(rotation=55)
plt.ylim([0,35000])

plt.subplot(1,2,2)
fig2=sb.countplot (x='LoanStatus',data=prosper[prosper['IsBorrowerHomeowner']==False],color=sb.color_palette()[0],
                  order=prosper['LoanStatus'].value_counts().index)
custom_labels('Loan Status','count','Borrower is not a home owner')
plt.xticks(rotation=55)

plt.ylim([0,35000])

plt.show()<jupyter_output><empty_output><jupyter_text>> if the borrower was a home owner, it's more likely to have current loans more than not home owner borrowers.  b) how much monthly payment varies according to borrower employment status and loan amount ?<jupyter_code>plt.figure(figsize=(10,10))
# regplotting
# prepare plottig to be approperaite to screen
sb.regplot(data=prosper,x='LoanOriginalAmount',y='EmploymentStatus',fit_reg=False,
              scatter_kws={'s':prosper['MonthlyLoanPayment']/2});
# manage labels
custom_labels('Orginal amount','Employment Status','Monthly Payment')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10617034931695242, 'fraction_numerical': 0.023654533785642248, 'mean_word_length': 7.7614457831325305, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 96, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '485449', 'license': 'MIT', 'n_tokens_mistral': 3537, 'n_tokens_neox': 3281, 'n_words': 1003, 'path': 'exploration_template.ipynb', 'repo_name': 'khadija267/Data-Visualization', 'size': '10916'}