<jupyter_start><jupyter_text>EDA & Data VisualizationBecause there are so many features in our loaning dataset, it's important to explore associations between them and identify the most significant factors contributing to default. In this notebook, univariate, bivariate, and multivariate analysis will be performed to understand the data better to build the model.   Imports<jupyter_code>import pandas as pd
import numpy as np

from matplotlib import pyplot as plt
import seaborn as sns
import plotly.plotly as py
import plotly.graph_objs as go
%matplotlib inline
sns.set_style("whitegrid")

import warnings
warnings.filterwarnings("ignore")

%load_ext autoreload
%autoreload 2

import sys
sys.path.insert(0, "../src/visualization") # access custom files

# import custom files
import plots

import os 
print(os.listdir("../data/LendingClub/processed"))<jupyter_output>['Processed.csv']<jupyter_text>Loading and Inspecting DataWe have already cleaned parts of the raw data from LendingClub in the previous notebook, *Preprocessing.ipynb*, and we will use this version to create plots and look for interesting details about certain features. Some attributes like the loan *id*, and other unique features aren't very useful to inspect so they will be dropped from analysis.<jupyter_code>processed_path = "../data/LendingClub/processed/Processed.csv"
data = pd.read_csv(processed_path)
print("Rows: " + str(data.shape[0]) + "  Columns: " + str(data.shape[1]))
data.head()
data.columns<jupyter_output><empty_output><jupyter_text>Loan StatusWe should first explore the label we are trying to predict, which is the loan status detailing whether the loan is current, has been paid, is late, or has been charged off (default). Previously, we had encoded these categories into numerical values. The dictionary of encoding is referenced below, and we can see our data contains loans that fall into all of these categories.<jupyter_code>status_dict = {0: "Charged Off", 1: "Current", 2: "Fully Paid", 3: "In Grace Period", 4: "Late (16-30 days)",
               5: "Late (31-120 days)"}

print(set(data["loan_status"]))
print(data["loan_status"].value_counts())<jupyter_output>{0, 1, 2, 3, 4, 5}
1    134839
2      5929
5      1042
3       689
4       364
0       156
Name: loan_status, dtype: int64<jupyter_text>We can make a barplot of counts for all the loan status outcomes. It appears we have loans falling into each of these categories. Now, we will dive deeper to understand what attributes of a loan contribute to this outcome.<jupyter_code>plt.figure(figsize=(14, 6))

g = sns.countplot(x="loan_status", data=data)
g.set_xticklabels(status_dict.values(), rotation=45)
g.set_xlabel("Loan Status Categories", fontsize=13)
g.set_ylabel("Count", fontsize=13)
g.set_title("Loan Status Types Distribution", fontsize=15)

proportions = []
for status in g.patches:
    height = status.get_height()
    proportions.append(height)
    g.text(status.get_x()+status.get_width()/2.,
            height + 3,
            "{:1.2f}%".format(height/len(data)*100),
            ha="center", fontsize=12) 

g.set_ylim(0, max(proportions)*1.1)<jupyter_output><empty_output><jupyter_text>Loan AmountThe loan amount requested is a good place to start, as understanding the distribution of loan amounts can give us a good idea of the range where most of the loans fall in. Large outliers may be much harder to pay back, which is probably a useful indicator of perdicting the loan status.  *Distribution*To visualize the ranges of the loans, a histogram is helpful. It appears that most of the requested loans fall somewhere within the range of 0-40000 dollars, with various modes to consider; $10,000 still appears to be the most common loan amount. It can also be observed that most of these loans were funded by the lender by observing the distirbution of *funded_amnt*, which is the total amounts funded to the loan.<jupyter_code>plt.figure()
fig = plt.gcf() #creates the plot figure object
fig.set_size_inches(16, 12) #sets the plot size
plt.subplot(2, 2, 1) #first subplot, 2 rows and 2 columns
plots.histogram("loan_amnt", data, kde=True)
plt.subplot(2, 2, 2) #second subplot, 2 rows and 2 columns
plots.histogram("funded_amnt", data, kde=True)
plt.subplots_adjust(left=0) #shift to make plots legible<jupyter_output><empty_output><jupyter_text>*Grade*It's also beneficial to understand the loan amounts in context with other features like the loan *grade*, which can provide insight to the total amounts requested by each grade, so we can detect outliers grade-wise.<jupyter_code>plt.figure()
fig = plt.gcf() #creates the plot figure object
fig.set_size_inches(10, 5) #sets the plot size
plots.barplot(data, "grade", "loan_amnt")<jupyter_output><empty_output><jupyter_text>We can also observe the loan status for each of the grades as well by making *loan_status* a hue for our bar plot. This will display the amounts of loans that are ongoing, were fully paid, were fully charged off, etc.<jupyter_code>plt.figure()
fig = plt.gcf() #creates the plot figure object
fig.set_size_inches(20, 10) #sets the plot size
plots.barplot(data, "grade", "loan_amnt", hue="loan_status")
# Each grade also consists of a subgrade, so we can dive deeper into different loan categories
plt.figure()
fig = plt.gcf() 
fig.set_size_inches(30, 10)
g = plots.barplot(data, "grade", "loan_amnt", hue="sub_grade")
g.set_xlabel("Grade", fontsize=13)
g.set_ylabel("loan_amnt", fontsize=13)
g.set_title("Loan Amount by Subgrade", fontsize=15)<jupyter_output><empty_output><jupyter_text>Loan Purpose & EmploymentIt may be beneficial to understand the purposes for the loans in the data, since the use of the borrowed money may contribute to whether it is paid back or not. Some may borrow money for a new car, to pay off their house, or for other casual purposes. Others may need financing for debt consolidation, which is to pay off all their other current liabilities, or to finance a business. These activities seem to carry more risk and may have an impact on default.  *Loan Purpose*We can observe the number of loans made for each type of purpose and understand what the majority of borrowers are looking to do. Some loan purposes may have higher risks and lead to more defaults, while other purposes may be easier to pay off.<jupyter_code>plt.figure(figsize=(14, 6))

g = sns.countplot(x='purpose', data=data)
g.set_xticklabels(g.get_xticklabels(), rotation=45)
g.set_xlabel("Purpose Titles", fontsize=13)
g.set_ylabel('Count', fontsize=13)
g.set_title("Client Purposes for Loan Credit", fontsize=15)

sizes=[]

for p in g.patches:
    height = p.get_height()
    sizes.append(height)
    g.text(p.get_x()+p.get_width()/2.,
            height + 3,
            '{:1.2f}%'.format(height/len(data)*100),
            ha="center", fontsize=12) 
    
g.set_ylim(0, max(sizes) * 1.1)<jupyter_output><empty_output><jupyter_text>*Employment Title**IP*: Text processing/regular expressions<jupyter_code>set(data["emp_title"])<jupyter_output><empty_output><jupyter_text>*Employment Length*The amount of time that someone has been employed can reveal information about not only job stability, but also experience with managing finances and responsibility to repay loans. Longer employment length could mean that the borrower is more likely to pay off their debt since they have less risk of becoming unemploymed and a more organized financial plan than borrowers with less experience.<jupyter_code>set(data["emp_length"])<jupyter_output><empty_output><jupyter_text>Loan DateLoans are requested at different times throughout the year. However, some months may be more popular than others for a vareity of reasons which could have an impact to default. For example, there may be a time during the year where the economy performs poorly and people are in need of financing. The *issue_d* column contains data about the month in which the loan was funded, which will be very useful when joining our economy data with the lending data. For now, we sohuld understand which months and years are the most common for requesting a loan.<jupyter_code>data["issue_d"].head()
data["year"] = pd.to_datetime(data["issue_d"]).dt.year
data["month"] = pd.to_datetime(data["issue_d"]).dt.month
plt.figure(figsize=(10, 5))
sns.barplot('month', 'loan_amnt', data=data)
plt.title('Months of Loan Requests', fontsize=15)
plt.xlabel('Month', fontsize=13)
plt.ylabel('Number of Loans', fontsize=13)<jupyter_output><empty_output><jupyter_text>Similar to previous features explored, the *loan_status* can be be a hue for the bar plot that shows the total amount of loans ongoing, fully paid, charged off, etc... by month of the loan.<jupyter_code>plt.figure(figsize=(12, 8))
sns.barplot('month', 'loan_amnt', data=data, hue="loan_status")
plt.title('Months of Loan Requests', fontsize=15)
plt.xlabel('Month', fontsize=13)
plt.ylabel('Number of Loans', fontsize=13)
'''
plt.figure(figsize=(12, 8))
sns.barplot('year', 'loan_amnt', data=data)
plt.title('Years of Loan Requests', fontsize=15)
plt.xlabel('Year', fontsize=13)
plt.ylabel('Number of Loans', fontsize=13)
'''<jupyter_output><empty_output><jupyter_text>It can be observed that some months and years have more requested loans than others, which may be helpful information when put into context of the macroeconomic condition. As shown in the plots where *loan_status* is an additional hue, there are varying amounts of loans that were charged off throughout these time periods. Some months and years have much more charged off loans than others, which is very interesting: the time of a loan seems to have an impact on its status. When there is a turndown and factors like unemployment rise, people may be inclined to attain financing for their activities, but may have difficulty paying back. This topic will explored in another notebook with the joined data, where we will evaluate the extent to which macroeconomic factors contribute towards default.  LocationThe address of the borrower is also a potential indicator for *loan_status* since different areas of the countries have varying economic conditions. For example, areas near the rust belt could have lower incomes and higher unemployment, while areas in California could be significantly wealthier. This feature is important when joining the macroeconomic data because states have varying macroecjonomic factors and the data will be joined by the corresponding location. For now, it's good to visualze and analyze which states had more charge offs than others, which could provide insight about locations with poorer economies, a potentially important factor to predicting the outcome of a loan.<jupyter_code>data["addr_state"]
plt.figure()
fig = plt.gcf() #creates the plot figure object
fig.set_size_inches(20, 10) #sets the plot size
plots.barplot(data, "addr_state", "loan_amnt")
# *In progress* making geospatial plot w/ plotly<jupyter_output><empty_output><jupyter_text>FICO Score (Credit Score)One of the most fundamental indicators of a borrower's reliability to pay back their loan is their credit score. The FICO score is a trusted source of credit score determination and is evaluated when issuing loans. It will be useful to understand the distribution of FICO scores in the data which will be important in predicting the *loan_status*, as people with lower FICO scores are expected to have more charge offs. The data contains two FICO score columns, one containing the highest score that the borrower had during the loan period, and the other containing the lowest score. To combine these scores, we can simply take the average of them since it is highly unlikely that someone's credit score will go from poor to excellent in the short-term duration of the loan.<jupyter_code>data["fico_average"] = (data["fico_range_high"] + data["fico_range_low"])/2
data["fico_average"].head()<jupyter_output><empty_output><jupyter_text>Here is the distribution of FICO scores across the dataset for borrowers. It appears unimodal and skewed right, with more people in the data having scores around 700. This makes sense, as many people may need financing to pay off some debt or for other purposes, while those with very high credit scores (>800) are less likely in need of a loan.<jupyter_code>plots.histogram("fico_average", data, kde=True)<jupyter_output><empty_output><jupyter_text>There has been a generally agreed upon distribution of FICO score ranges that fall into different categories of someone's credit. This range is used in the function below, where we can convert each borrower's FICO score to a category for better understanding and analysis of default.<jupyter_code>def fico_rating(fico_score):
    """
    Converts raw FICO score to a categorical assessment of borrower's credit.
    
    :param fico_score: FICO score of borrower
    :returns: rating of FICO score
    """
    if 579 >= fico_score >= 300:
        return "Poor"
    elif 669 >= fico_score >= 580:
        return "Fair"
    elif 739 >= fico_score >= 670:
        return "Good"
    elif 799 >= fico_score >= 740:
        return "Very Good"
    elif fico_score >= 800:
        return "Exceptional"
    
    
data["fico_rating"] = data["fico_average"].apply(fico_rating)
data["fico_rating"].head()<jupyter_output><empty_output><jupyter_text>Now we can visualize the loan amounts requested by each category of FICO scores, which will show whether borrowers with higher scores request more than lower scores. Furthermore, it's very useful to include a hue of loan status as well to understand if certain score categories had higher rates of charge offs when requesting larger amounts.<jupyter_code>plt.figure()
fig = plt.gcf() 
fig.set_size_inches(20, 10)
plots.barplot(data, "fico_rating", "loan_amnt", hue="loan_status")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.081397037467325, 'fraction_numerical': 0.015538774324716816, 'mean_word_length': 5.25192918747163, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '604519', 'license': 'MIT', 'n_tokens_mistral': 4152, 'n_tokens_neox': 3863, 'n_words': 1696, 'path': 'notebooks/Loaning EDA & Data Visualization.ipynb', 'repo_name': 'samsonq/Macroeconomic-Default-Analysis', 'size': '13772'}