<jupyter_start><jupyter_text>Prosper - Loan Data by <NAME> Preliminary Wrangling> [Data set](https://www.google.com/url?q=https://s3.amazonaws.com/udacity-hosted-downloads/ud651/prosperLoanData.csv&sa=D&source=editors&ust=1622283415087000&usg=AOvVaw0vpDb-LcekmeJjNUwDsRyP) has information on peer-to-peer loans facilitated by, a credit company, [Prosper](https://www.prosper.com/). The data set contains 113,937 loans with 81 variables on each loan, including loan amount, borrower rate (or interest rate), current loan status, borrower income, and many others.<jupyter_code># import all packages and set plots to be embedded inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb

%matplotlib inline
# setting color scheme for plots
base_color = sb.color_palette()[0]
# reading data from csv file to Dataframe
loan_data = pd.read_csv('prosperLoanData.csv')

# Displaying the structure of data
print(loan_data.shape)
# Displaying the top 5 rows of data
loan_data.head()
# Displaying the critical info like index dtype and columns, non-null values and memory usage.
loan_data.info()
# Displaying the random 5 rows of data
loan_data.sample(5)
# Displaying the statistical summary of numerical columns
loan_data.describe()<jupyter_output><empty_output><jupyter_text>> The data set has 81 features. For the purpose of analysis of this data set, I've chosen some of the features out of 81 features.<jupyter_code># Name of cols assigning to variable
target_col =  ['Term', 'LoanStatus', 'BorrowerRate', 
               'ProsperRating (Alpha)', 'ListingCategory (numeric)', 
               'EmploymentStatus', 'DelinquenciesLast7Years', 
               'StatedMonthlyIncome', 'TotalProsperLoans', 
               'LoanOriginalAmount', 'LoanOriginationDate', 
               'Recommendations', 'Investors']

# Creating a subset of Data with selected features for analysis
loan_target = loan_data[target_col]

# Displaying the structure of data
loan_target.shape
# Displaying the top 5 rows of data
loan_target.head()
# Displaying the random 5 rows of data
loan_target.sample(5)
# Displaying the critical info like index dtype and columns, non-null values and memory usage.
loan_target.info()
# Displaying the statistical summary of numerical columns
loan_target.describe()<jupyter_output><empty_output><jupyter_text>> Column names are inconsistent and will cause erros<jupyter_code># Renaming the columns as Title case
loan_target.columns = loan_target.columns.str.replace('[(,)]', '').str.replace(' ','')
loan_target = loan_target.rename(columns={'ListingCategorynumeric' : 'ListingCategoryNumeric'})
loan_target.columns<jupyter_output><empty_output><jupyter_text>> Convert `LoanOriginationDate` to `datetime` data type form `object`<jupyter_code># Changing the data type to datetime
loan_target.LoanOriginationDate = pd.to_datetime(loan_target.LoanOriginationDate)<jupyter_output><empty_output><jupyter_text>> Filling `NaN` by `0` in `TotalProsperLoans` columns.<jupyter_code># filling null/NaN values with 0 
loan_target.TotalProsperLoans = loan_target.TotalProsperLoans.fillna(0)<jupyter_output><empty_output><jupyter_text>> *Dropping the missing values*: As the Prosper use their own proprietary Ratings since 2009, we have a lot of missing values in `ProsperRatingAlpha` column.<jupyter_code># Drop the missing values
loan_target = loan_target.dropna(subset=['ProsperRatingAlpha']).reset_index()
# Displaying the critical info like index dtype and columns, non-null values and memory usage.
loan_target.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 84853 entries, 0 to 84852
Data columns (total 14 columns):
index                      84853 non-null int64
Term                       84853 non-null int64
LoanStatus                 84853 non-null object
BorrowerRate               84853 non-null float64
ProsperRatingAlpha         84853 non-null object
ListingCategoryNumeric     84853 non-null int64
EmploymentStatus           84853 non-null object
DelinquenciesLast7Years    84853 non-null float64
StatedMonthlyIncome        84853 non-null float64
TotalProsperLoans          84853 non-null float64
LoanOriginalAmount         84853 non-null int64
LoanOriginationDate        84853 non-null datetime64[ns]
Recommendations            84853 non-null int64
Investors                  84853 non-null int64
dtypes: datetime64[ns](1), float64(4), int64(6), object(3)
memory usage: 9.1+ MB<jupyter_text>What is the structure of your dataset?> The [data set](https://www.google.com/url?q=https://s3.amazonaws.com/udacity-hosted-downloads/ud651/prosperLoanData.csv&sa=D&source=editors&ust=1622283415087000&usg=AOvVaw0vpDb-LcekmeJjNUwDsRyP) contains 113,937 loans with 81 features on each loan, including loan amount, borrower rate (or interest rate), current loan status, borrower income, and many others. [Complete list](https://docs.google.com/spreadsheets/d/1gDyi_L4UvIrLTEC6Wri5nbaMmkGmLQBk-Yx3z0XDEtI/editgid=0) of features and thier details can be found [here](https://docs.google.com/spreadsheets/d/1gDyi_L4UvIrLTEC6Wri5nbaMmkGmLQBk-Yx3z0XDEtI/editgid=0).>> But some of the features I selected for alanysis are listed below with their descriptions taken from [here](https://docs.google.com/spreadsheets/d/1gDyi_L4UvIrLTEC6Wri5nbaMmkGmLQBk-Yx3z0XDEtI/editgid=0).- **Term**: The length of the loan expressed in months.- **LoanStatus**: The current status of the loan: Cancelled,  Chargedoff, Completed, Current, Defaulted, FinalPaymentInProgress, PastDue. The PastDue status will be accompanied by a delinquency bucket.- **BorrowerRate**: The Borrower's interest rate for this loan.- **ProsperRatingAlpha**: The Prosper Rating assigned at the time the listing was created between AA - HR.  Applicable for loans originated after July 2009.- **ListingCategoryNumeric**: The category of the listing that the borrower selected when posting their listing:     -  *0* - Not Available    -  *1* - Debt Consolidation    -  *2* - Home Improvement    -  *3* - Business    -  *4* - Personal Loan    -  *5* - Student Use    -  *6* - Auto    -  *7* - Other    -  *8* - Baby&Adoption    -  *9* - Boat    - *10* - Cosmetic Procedure    - *11* - Engagement Ring    - *12* - Green Loans    - *13* - Household Expenses    - *14* - Large Purchases    - *15* - Medical/Dental    - *16* - Motorcycle    - *17* - RV    - *18* - Taxes    - *19* - Vacation    - *20* - Wedding Loans- **EmploymentStatus**: The employment status of the borrower at the time they posted the listing.- **DelinquenciesLast7Years**: Number of delinquencies in the past 7 years at the time the credit profile was pulled.- **StatedMonthlyIncome**: The monthly income the borrower stated at the time the listing was created.- **TotalProsperLoans**: Number of Prosper loans the borrower at the time they created this listing. This value will be null if the borrower had no prior loans.- **LoanOriginalAmount**: The origination amount of the loan.- **LoanOriginationDate**: The date the loan was originated.- **Recommendations**: Number of recommendations the borrower had at the time the listing was created.- **Investors**: The number of investors that funded the loan. What is/are the main feature(s) of interest in your dataset?> I'm curious to know what are major factors connected with prosper credit rating.> And what features can be used to predict default on credit. What features in the dataset do you think will help support your investigation into your feature(s) of interest?> Borrowers Prosper rating will've the highest impact on chances of default, the amount of loan will play vital role and may be credit category too. Prosper rating will rely on employment status and monthly income.  Univariate Exploration> In this section, investigate distributions of individual variables. Ifyou see unusual points or outliers, take a deeper look to clean things upand prepare yourself to look at relationships between variables.  Employment Status<jupyter_code># Setting the order for plot
order = loan_target.EmploymentStatus.value_counts().index

# Plotting the Plot
plt.figure(figsize = [8, 5])
sb.countplot(data = loan_target, y = 'EmploymentStatus', color = base_color, order = order);
plt.xlabel('Count');
plt.ylabel('Employment Status');
plt.title('Employment Status of Loan Borrowers');<jupyter_output><empty_output><jupyter_text>By excluding the status 'empolyed' and zooming the graph<jupyter_code># Setting the order for plot
order = loan_target.EmploymentStatus.value_counts().index[1:]

# Plotting the Plot
plt.figure(figsize = [8, 5])
sb.countplot(data = loan_target, y = 'EmploymentStatus', color = base_color, order = order);
plt.xlabel('Count');
plt.ylabel('Employment Status');
plt.title('Employment Status of Loan Borrowers');<jupyter_output><empty_output><jupyter_text>> **Observation Employment Status**- The highest number of loan borrowers are **employed** with *600K+* in count.- The people who belongs to **Full-time or Self-employed or Other** category are *few thousands* in count.- The least number of people who borrow loans are **Not employed or Retired or Part-time** and they are just *few hundreds* in count.   Stated Monthly Income<jupyter_code># Plotting the Plot
plt.figure(figsize = [8, 5])
plt.hist(data=loan_target, x='StatedMonthlyIncome', bins=1000, rwidth=1.0);
plt.xticks(rotation = 90);
plt.xlabel('Count');
plt.ylabel('Stated Monthly Income');
plt.title('Stated Monthly Income of Loan Borrowers');<jupyter_output><empty_output><jupyter_text>> The distribution of stated monthly income is highly skewed to the right. We need to check about the outliers<jupyter_code># Calculating the boundry for better visulization 
smi_std = loan_target.StatedMonthlyIncome.std()
smi_mean = loan_target.StatedMonthlyIncome.mean()
smi_boundary = smi_mean + smi_std * 2
smi_std, smi_mean, smi_boundary, len(loan_target[loan_target.StatedMonthlyIncome >= smi_boundary])
# Plotting the Plot
plt.figure(figsize = [8, 5])
plt.hist(data=loan_target, x='StatedMonthlyIncome', bins=1000, rwidth=0.75);
plt.xlim(0, smi_boundary);
plt.xlabel('Count');
plt.ylabel('Stated Monthly Income');
plt.title('Stated Monthly Income of Loan Borrowers');<jupyter_output><empty_output><jupyter_text>> **Observation Stated Monthly Income**- With a boundary of mean and 2 times standard deviation, distribution of stated monthly income still has high right skewness.- Any how, now one can infer that the mode of data is nearly to 5000.  Loan Status<jupyter_code># Setting the order for plot
order = loan_target.LoanStatus.value_counts().index

# Plotting the Plot
plt.figure(figsize = [8, 5])
sb.countplot(data = loan_target, y = 'LoanStatus', color = base_color, order = order);
plt.xlabel('Count');
plt.ylabel('Loan Status');
plt.title('Loan Status of Borrowers');<jupyter_output><empty_output><jupyter_text>> **Observation Loan Status**- Second highest majority in Loan Status is of Completed ones in the data set.- In the data set, Loan Status with Current are the highest ones. - Chargedoff loan also have substantial amount.- Past due loans are categorized in several groups on the basis of length of payment delay.  Discuss the distribution(s) of your variable(s) of interest. Were there any unusual points? Did you need to perform any transformations?> - The majority of borrowers are employed in the data set. And all other categories hold a small part of borrowers.- Stated Monthly Income's distribution is highly right skewed with very large range along with plenty of outliers.- Most of the loans lie in the category of current loans in our data set. Of the features you investigated, were there any unusual distributions? Did you perform any operations on the data to tidy, adjust, or change the form of the data? If so, why did you do this?> - As our main goal is to identify the driving factors of outcome of loan, we aren't much interested in current loans, though the category of current loans hold highest portion among all categories of loan status.  Bivariate Exploration> In this section, investigate relationships between pairs of variables in yourdata. Make sure the variables that you cover here have been introduced in somefashion in the previous section (univariate exploration).<jupyter_code># All Loan Statuses except 'Current' type
loan_target = loan_target[(loan_target.LoanStatus != 'Current')]

def to_defaulted(val):
    '''
    Take any Dataframe and checks if the specific column has 
    these values in each record, if not it will change it.
    '''
    if val.LoanStatus == 'Completed':
        return val.LoanStatus
    else:
        return 'Defaulted'

# Applying the function 
loan_target.LoanStatus = loan_target.apply(to_defaulted, axis=1)
loan_target.LoanStatus.value_counts()<jupyter_output><empty_output><jupyter_text>> After transforming dataset we have 19664 completed loans and 8613 defaulted. > ***NOTE*** Extracting Top 5 Categories (ListingCategoryNumeric)<jupyter_code># Returning a Series containing counts of unique values (top 5) for 'ListingCategoryNumeric' col.
loan_target.ListingCategoryNumeric.value_counts()[:5]
# Top 5 most frequent key pair-values of 'ListingCategoryNumeric' col
categories = {1 : 'Debt Consolidation', 7 : 'Other', 2 : 'Home Improvement',
              3 : 'Business', 6 : 'Auto'}

def categorize(val):
    '''
    Take any Dataframe and checks if the specific column has 
    these values in each record, if not it will change it.
    '''
    loan_cat = val.ListingCategoryNumeric
    if  loan_cat in categories:
        return categories[loan_cat]
    else:
        return categories[7]

# Applying the function 
loan_target.ListingCategoryNumeric = loan_target.apply(categorize, axis=1)
loan_target.ListingCategoryNumeric.value_counts()<jupyter_output><empty_output><jupyter_text>> Variable Listing Category is set up as numeric and most of the values have very low frequency, for the easier visualization so we have changed it to categorical categories and extracted only top 5 categories.  Loan Status & Prosper Rating<jupyter_code># Setting the order for plot
order = loan_target.ProsperRatingAlpha.value_counts(ascending=True).index

# Plotting the Plot
plt.figure(figsize = [8, 5])
sb.countplot(data = loan_target, y = 'LoanStatus', 
             hue = 'ProsperRatingAlpha', palette = 'Blues', 
             hue_order=order);
plt.xlabel('Count');
plt.ylabel('Loan Status');
plt.title('Loan Status & Rating');<jupyter_output><empty_output><jupyter_text>> **Observation Loan Status & Prosper Rating**- Rating 'D' is the most frequent one in both 'Completed' and 'Defaulted' statuses.- Rating 'AA' is the least frequent one in both 'Completed' and 'Defaulted' statuses.  Employment Status & Prosper Rating<jupyter_code># Setting the order for plot
order = loan_target.EmploymentStatus.value_counts(ascending=False).index

# Plotting the Plot
plt.figure(figsize = [12, 9])
sb.countplot(data = loan_target, x = 'ProsperRatingAlpha', 
             hue = 'EmploymentStatus', palette = 'Blues', 
             hue_order=order);
plt.xlabel('Prosper Rating Alpha');
plt.ylabel('Count');
plt.title('Employment Status & Rating');<jupyter_output><empty_output><jupyter_text>> **Observation Employment Status & Prosper Rating**- Lower Ratings seem to have greater proportions of individuals with employment statuses 'Self-employed', 'Not employed', 'Retired' and 'Part-time'.  Loan Status & Loan Amount<jupyter_code># Plotting the Plot
plt.figure(figsize = [8, 5])
sb.boxplot(data = loan_target, x = 'LoanStatus', y = 'LoanOriginalAmount', color = base_color);
plt.xlabel('Loan Status');
plt.ylabel('Loan Amount');
plt.title('Loan Status & Loan Amount');<jupyter_output><empty_output><jupyter_text>> **Loan Status & Loan Amount**- Completed credits tend to be smaller incomparison to Defaulted credits ones.- Loans of amount more than USD25000/- ones belong to Completed credits mostly. On the other hand, very few Defaulted credits are of amount more than USD25000/-.  Loan Status & Listing Category<jupyter_code># Setting the order for plot
order = loan_target.ListingCategoryNumeric.value_counts().index

# Plotting the Plot
plt.figure(figsize = [8, 5])
sb.countplot(data = loan_target, x = 'LoanStatus', 
             hue = 'ListingCategoryNumeric', palette = 'Blues', 
             hue_order=order);
plt.xlabel('Loan Status');
plt.ylabel('Count');
plt.title('Loan Status & Listing Category');<jupyter_output><empty_output><jupyter_text>> **Loan Status & Listing Category**>> - 'Debt Consolidation' is the most frequent category of all the time in Completed credits as well as Defaulted credits.  Talk about some of the relationships you observed in this part of the investigation. How did the feature(s) of interest vary with other features in the dataset?> - In Loan status & Loan amount, Completed credits tend to be smaller incomparison to Defaulted credits ones.- In Employment Status & Rating, Lower Ratings seem to have greater proportions of individuals with employment statuses 'Self-employed', 'Not employed', 'Retired' and 'Part-time'. Did you observe any interesting relationships between the other features (not the main feature(s) of interest)?> - Rating D is the most common in both Completed and Defaulted credits.  Multivariate Exploration> Create plots of three or more variables to investigate your data evenfurther. Make sure that your investigations are justified, and follow fromyour work in the previous sections.  Rating, Loan Amount & Loan Status<jupyter_code># Plotting the Plot
plt.figure(figsize = [15, 8])
sb.boxplot(data = loan_target, x = 'ProsperRatingAlpha', y = 'LoanOriginalAmount',
           hue = 'LoanStatus', color=base_color);
plt.xlabel('Rating');
plt.ylabel('Loan Amount');
plt.title('Rating, Loan Amount & Loan Status');<jupyter_output><empty_output><jupyter_text>> **Rating, Loan Amount & Loan Status**- Most of the defaulted credits comes from individuals with low Prosper rating.- Defaulted credits tend to be larger than completed ones, except the lowest ratings.  Listing Category, Loan Amount and Loan Status<jupyter_code># Plotting the Plot
plt.figure(figsize = [15, 8])
sb.violinplot(data=loan_target, x='ListingCategoryNumeric', y='LoanOriginalAmount',
              hue='LoanStatus', color=base_color);
plt.xlabel('Category');
plt.ylabel('Loan Amount');
plt.title('Category, Loan Amount & Loan Status');<jupyter_output><empty_output><jupyter_text>> **Category, Loan Amount & Loan Status**- Debt Consolidation and Other category tend to have larger amount.  Relationships between Credit category, Rating and Status of Credit.<jupyter_code># Plotting the Plot
sb.catplot(x = 'ProsperRatingAlpha', hue = 'LoanStatus', col = 'ListingCategoryNumeric',
           data = loan_target, kind = 'count', palette = 'Blues', col_wrap = 3);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09186114507469034, 'fraction_numerical': 0.01796821008984105, 'mean_word_length': 4.992991398534565, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 117, 'https://': 8, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '583513', 'license': 'MIT', 'n_tokens_mistral': 5903, 'n_tokens_neox': 5408, 'n_words': 2001, 'path': 'exploration.ipynb', 'repo_name': 'Usama-Tariq/Udacity_Communicat-Data-Findings_Project-5_DAND', 'size': '18816'}