<jupyter_start><jupyter_text>Finance Project: Deep Neural Network  Supervised machine learning algorithm using Lending Club dataset.Csv Download :-https://www.kaggle.com/code/truesight/finance-predictive-model-deep-neural-network/data?select=lending_club_loan_two.csv  Description: While staring into the future through a crytal ball is a myth, technology can help investors to seek true sight in their investment prospects. It is the financial institutes' dream to grasp the untapped knowledge what is the return on investment in a given project. This is an interesting project to uncover the risky versus the high-profitable loan borrowers in the ocean of dataset.   Project Objective: To develop a supervised machine learning model to identify which borrowers will payoff their loans. The project implications can be beneficial to the financial institute in risk assessments, whether the prospective borrower would default or payoff the loan. Strategy for loan approval and profitable target market can be identified. Ultimately, this model serves as the blueprint to decrease bussiness risks and increase profitability of the organization.  Predictive Model: Deep Neural NetworkReal life dataset by one of the financial powerhouses, Lending Club. Supervised machine learning deep neural network will be used to perform binary classification. In this project, the target feature or y-variable will be "Loan Status".    Process:This project will start off with exploratory data analysis, data visualization, feature-engineering, and preparing the dataset for machine learning. The end result the accuracy of the model to predict payoff or default loan.   Importing libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
data = pd.read_csv("D:\\DC Universe\\Ucsc\\Third Year\\ENH 3201 Industrial Placements\\Flask Projects\\Financial Analysis\\lending_club_loan_two.csv")
data.head()<jupyter_output><empty_output><jupyter_text>This is a large dataset, which has ~396000 observations as shown below.  Noted that they are float and object data type.<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 396030 entries, 0 to 396029
Data columns (total 27 columns):
 #   Column                Non-Null Count   Dtype  
---  ------                --------------   -----  
 0   loan_amnt             396030 non-null  float64
 1   term                  396030 non-null  object 
 2   int_rate              396030 non-null  float64
 3   installment           396030 non-null  float64
 4   grade                 396030 non-null  object 
 5   sub_grade             396030 non-null  object 
 6   emp_title             373103 non-null  object 
 7   emp_length            377729 non-null  object 
 8   home_ownership        396030 non-null  object 
 9   annual_inc            396030 non-null  float64
 10  verification_status   396030 non-null  object 
 11  issue_d               396030 non-null  object 
 12  loan_status           396030 non-null  object 
 13  purpose               396030 non-null  object 
 14  title                 394275 non-null  object 
 15  [...]<jupyter_text>Visualizing loan payoff vs chargeoff.<jupyter_code>plt.figure(figsize=(12,7))
sns.set_context("paper")
sns.countplot(x="loan_status", data=data, palette="Spectral")<jupyter_output><empty_output><jupyter_text>Visualizing the distribution of loan amount borrowed.<jupyter_code>plt.figure(figsize=(15,7))
sns.set_context("paper", font_scale=2)
sns.distplot(data["loan_amnt"], bins=15, kde=False, color="seagreen")<jupyter_output>C:\ProgramData\Anaconda3\lib\site-packages\seaborn\distributions.py:2557: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>Descriptive analysis and correlation of features.<jupyter_code>data.corr()<jupyter_output><empty_output><jupyter_text>Noted that loan amount and interest rate has high correlation,, which is expected. Total account & open account also has high correlation. Lastly, public record and bankcruptcies, which also makes sense.<jupyter_code>plt.figure(figsize=(15,7))
sns.set_context("poster", font_scale=0.5)
sns.heatmap(data.corr(), annot=True, cmap="coolwarm", alpha=0.6)<jupyter_output><empty_output><jupyter_text>Visualizing the relationship between installment and loan amount.<jupyter_code>plt.figure(figsize=(15,7))
sns.scatterplot(x="installment", y="loan_amnt", data=data, alpha=0.8, hue="loan_status", palette="RdYlBu" )<jupyter_output><empty_output><jupyter_text>Noted that there is only a slight difference in full paid and charged off on loan amount.<jupyter_code>plt.figure(figsize=(15,7))
sns.boxplot(x="loan_status", y="loan_amnt", data=data, palette="plasma")
data.groupby("loan_status")["loan_amnt"].describe()<jupyter_output><empty_output><jupyter_text>Diving in the feature "grade", presumably the level of worthiness of the borrowers.<jupyter_code>data["grade"].value_counts()<jupyter_output><empty_output><jupyter_text>Visualizing the relationship between the grade and loan status. As expected, borrowers tend to have a higher charged off in lower grade categories. This shows that the grade can potentially be a good indicator if the borrower has the ability to payoff or default.<jupyter_code>plt.figure(figsize=(15,7))
sns.set_context("paper", font_scale=2)
sns.countplot(x="grade", data=data, hue="loan_status", color="seagreen")<jupyter_output><empty_output><jupyter_text>Visualizing in a sorted order of grade gives a better understanding of the impact of grade in loan status. The lower the grade, the higher the ratio in fully paid to charged off.<jupyter_code>plt.figure(figsize=(15,7))
sns.set_context("paper", font_scale=2)
sorted_grade = sorted(data["grade"].unique())
sns.countplot(x="grade", data=data, hue="loan_status", color="seagreen", order=sorted_grade)
data["loan_status"].value_counts()
data["loan_repaid"] = data["loan_status"].map({"Fully Paid":1, "Charged Off":0})
data[["loan_repaid", "loan_status"]]<jupyter_output><empty_output><jupyter_text>Visualizing the correlation between "loan repaid" and other features. Noted that interest rate has relatively high correlation compared to the rest. This is expected as the higher the interest rate, the harder it is to pay off a loan.<jupyter_code>plt.figure(figsize=(15,7))
data.corr()["loan_repaid"].sort_values().drop("loan_repaid").plot(kind="bar", color="seagreen", alpha=0.6)<jupyter_output><empty_output><jupyter_text>Dealing with missing values. The number of features that have missing values is shown below. The number of missing values in each feature will determine the treatment of the feature; to drop or replace it with some other values.<jupyter_code>data.isnull().sum()
data.isnull().sum() / len(data) * 100
data["emp_title"].nunique()
data["emp_title"].value_counts()<jupyter_output><empty_output><jupyter_text>The employment title feature has >170000 unique values. It is not feasible to keep them as a feature to be used in our machine learning model. Will just drop it.<jupyter_code>data = data.drop("emp_title", axis=1)
sorted(data["emp_length"].dropna().unique())
sorted_emp_length = ['< 1 year',
 '1 year',
 '2 years',
 '3 years',
 '4 years',
 '5 years',
 '6 years',
 '7 years',
 '8 years',
 '9 years',
 '10+ years']<jupyter_output><empty_output><jupyter_text>Visualizing the employment length feature. It appears that most of the borrowers have >10yrs employment length, meaning most of the borrowers are middle-aged adults and/or matured instead of young adults.<jupyter_code>plt.figure(figsize=(16,7))
sns.set_context("paper", font_scale=1.5)
sns.countplot(x="emp_length", data=data, hue="loan_status", order=sorted_emp_length, palette="coolwarm")
emp_co = data[data["loan_status"] == "Charged Off"].groupby("emp_length").count()["loan_status"]
emp_fp = data[data["loan_status"] == "Fully Paid"].groupby("emp_length").count()["loan_status"]
emp_length_graph = emp_co/(emp_co+emp_fp)<jupyter_output><empty_output><jupyter_text>Visualizing the number of charged off and total borrowers in percentage in each intervals of employment length. They are relatively the same across the board shown in the graph below.<jupyter_code>emp_length_graph.plot(kind="bar")<jupyter_output><empty_output><jupyter_text>The employment length feature does not really help us to distinguish borrowers who payoff or default. Will just drop.<jupyter_code>data = data.drop("emp_length", axis=1)
data.isnull().sum()
data["purpose"].value_counts()<jupyter_output><empty_output><jupyter_text>Visualizing the loan purpose feature. It is apparent that top reason to borrow loans from Lending Club is debt consolidation as provided by borrowers. This is matches with common perception that people often attempt to pay off high interest rate credit card accounts with unsecured personal loan.<jupyter_code>plt.figure(figsize=(18,7))
sns.set_context("paper", font_scale=1)
sns.countplot(x="purpose", data=data, hue="loan_status", palette="seismic")
data["title"].value_counts()<jupyter_output><empty_output><jupyter_text>It appears that the feature "title" provides the same information as loan purpose. Will drop this feature.<jupyter_code>data = data.drop("title", axis=1)
data["mort_acc"].value_counts()<jupyter_output><empty_output><jupyter_text>Recalling that mortage account feature has ~38000 missing values. This is pretty significant as dropping this feature will significantly reduce the size of the dataset. It is probably a good idea to replace the missing values with some other values. After checking out the correlation of mortageg with other features, it is noted that total account has the highest correlation with mortgage account. it is not surprising that people have more mortgages when they have more accounts.<jupyter_code>data.corr()["mort_acc"].sort_values()<jupyter_output><empty_output><jupyter_text>Decided to replace missing values in mortgage account with the mean value based on the total account.<jupyter_code>total_acc_avg = data.groupby("total_acc").mean()["mort_acc"]
def fill_in_mort_acc(total_acc, mort_acc):
    
    if np.isnan(mort_acc):
        return total_acc_avg[total_acc]
    else:
        return(mort_acc)<jupyter_output><empty_output><jupyter_text>Replacing the missing values with lambda function.<jupyter_code>data["mort_acc"] = data.apply(lambda x: fill_in_mort_acc(x["total_acc"], x["mort_acc"]), axis=1)<jupyter_output><empty_output><jupyter_text>Finding out the current standing of our missing values. Since the remaining 2 features have very low missing values, decided to just drop those missing values as it is more time-saving.<jupyter_code>data.isnull().sum()
data = data.dropna()
data.isnull().sum()
data.dtypes<jupyter_output><empty_output><jupyter_text>Dealing with non-numeric type of data<jupyter_code>data.select_dtypes(["object"]).columns
data["term"].value_counts()<jupyter_output><empty_output><jupyter_text>Grabbing the numeric values "36" and "60".<jupyter_code>data["term"] = data["term"].apply(lambda term: int(term[:3]))
data["term"].value_counts()<jupyter_output><empty_output><jupyter_text>Since sub-grade provides more information than grade, this featue will be dropped.<jupyter_code>data = data.drop("grade", axis=1)<jupyter_output><empty_output><jupyter_text>Preparing the data with binary classification (dummy data).<jupyter_code>dummy = pd.get_dummies(data["sub_grade"], drop_first=True)
data = pd.concat([data.drop("sub_grade", axis=1), dummy], axis=1)
data.head()
data["verification_status"].value_counts()
data["application_type"].value_counts()
data["initial_list_status"].value_counts()
dummy = pd.get_dummies(data[["verification_status", "application_type", "initial_list_status", "purpose"]], drop_first=True)
data = pd.concat([data.drop(["verification_status", "application_type", "initial_list_status", "purpose"], axis=1), dummy], axis=1)
data.head()
data["home_ownership"].value_counts()
data["home_ownership"] = data["home_ownership"].replace(["NONE", "ANY"], "OTHER")
data["home_ownership"].value_counts()
dummy = pd.get_dummies(data["home_ownership"], drop_first=True)
data = pd.concat([data.drop("home_ownership", axis=1), dummy], axis=1)
data.head()<jupyter_output><empty_output><jupyter_text>Four more features with object as data type to deal with.<jupyter_code>data.select_dtypes("object").columns
data["address"].value_counts()<jupyter_output><empty_output><jupyter_text>Address has no values in our machine learning model but the ZIP code may have some sort of influence in the outcome. Grabbing the ZIP code from the address.<jupyter_code>data["zip_code"] = data["address"].apply(lambda address: address[-5:])
data["zip_code"].value_counts()<jupyter_output><empty_output><jupyter_text>Noted that the newly engineered feature of ZIP has only a few unique counts so it is feasible to keep this feature. Getting dummy data on this feature.<jupyter_code>dummy = pd.get_dummies(data["zip_code"], drop_first=True)
data = pd.concat([data.drop("zip_code", axis=1), dummy], axis=1)
data.head()
data = data.drop("address", axis=1)
data = data.drop("issue_d", axis=1)
data["earliest_cr_line"].value_counts()<jupyter_output><empty_output><jupyter_text>The feature "Earliest credit line" may be a key factor as it provides some sort of a time series information. Grabbing the year as our tim series feature.<jupyter_code>data["earliest_cr_line"] = data["earliest_cr_line"].apply(lambda year: int(year[-4:]))
data["earliest_cr_line"].value_counts()
data.select_dtypes("object").columns<jupyter_output><empty_output><jupyter_text>Recalling that we have converted "fully paid" and "charged off" with binary digits, it is safe to just drop the original feature.<jupyter_code>data = data.drop("loan_status", axis=1)<jupyter_output><empty_output><jupyter_text>Data cleansing process and feature-engineering complete. The dataset now has 79 features. Now preparing for training data and test data.<jupyter_code>data.head(3)<jupyter_output><empty_output><jupyter_text>Trainning data will be set at 80% of the dataset and test size is 20%. Random state will be set at 42, which is arbitrary - I heard 42 is THE number of universe, life, and everything :)  The target feature or y-variable is "loan repaid" (yes = 1; no = 0).<jupyter_code>from sklearn.model_selection import train_test_split
X = data.drop("loan_repaid", axis=1).values
y = data["loan_repaid"].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Scalar is used when preparing the dataset for deep learning so the data will have a more meaningful relationship among features. This can enable the machine to learn the data better.<jupyter_code>from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Preparation is complete. Importing deep neural network libraries.<jupyter_code>import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.models import Sequential<jupyter_output><empty_output><jupyter_text>Recalling the dataset has ~310000 observations and 78 features.<jupyter_code>X_train.shape<jupyter_output><empty_output><jupyter_text>Setting "rectified linear unit" as the activation function in neural network as this is a commonly used activation. Dense will be set at 78, which is an arbitrary number, as the first layer, then followed by half of it and so forth. On the final layer of the neural network, the "sigmoid" is used as the activation function, which is sorta similar to logistic regression. For loss function, the binary cross entropy will be used since this is a binary classification model. Optimizer is set as Adam is this is the most commonly used.<jupyter_code>model = Sequential()

model.add(Dense(78, activation="relu"))
model.add(Dropout(0.2))

model.add(Dense(39, activation="relu"))
model.add(Dropout(0.2))

model.add(Dense(19, activation="relu"))
model.add(Dropout(0.2))

model.add(Dense(1, activation="sigmoid"))

model.compile(loss="binary_crossentropy", optimizer="adam")<jupyter_output><empty_output><jupyter_text>Setting epochs as 25, which is also an arbitrary number; the batch size is set at 256 (64-bit ~ personal preference).<jupyter_code>model.fit(X_train, y_train, epochs=25, batch_size=256, validation_data=(X_test, y_test))
losses = pd.DataFrame(model.history.history)<jupyter_output><empty_output><jupyter_text>Loss function graph to see the performance of the deep neural network model. Noted that the loss funtion decreased sharply at the beginning, which is desirable, then trending down slowly below the validation loss.<jupyter_code>losses.plot()<jupyter_output><empty_output><jupyter_text>Evalution of the supervised machine learning deep neural network performance.<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix
prediction = model.predict_classes(X_test)
print(classification_report(y_test, prediction))
print(confusion_matrix(y_test, prediction))<jupyter_output>precision    recall  f1-score   support

           0       0.98      0.44      0.61     15421
           1       0.88      1.00      0.94     63623

    accuracy                           0.89     79044
   macro avg       0.93      0.72      0.77     79044
weighted avg       0.90      0.89      0.87     79044

[[ 6832  8589]
 [  163 63460]]
{'fraction_non_alphanumeric': 0.10163061940890046, 'fraction_numerical': 0.021854829577624277, 'mean_word_length': 5.905003909304144, 'pattern_counts': {'":': 2, '<': 164, '<?xml version=': 0, '>': 165, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '41', 'dir': 'jupyter-structured-clean-dedup', 'id': '68328', 'license': 'Apache-2.0', 'n_tokens_mistral': 5642, 'n_tokens_neox': 5283, 'n_words': 1642, 'path': 'ECL Models/Financial Analysis/Financial Predictive Deep Learning Hierarchy.ipynb', 'repo_name': 'Pandula1234/Data-Analysis-Intern', 'size': '17662'}