<jupyter_start><jupyter_text>Loan Application Prediction - End-to-End Project  In this project, we aim to categorize customers who may or may not receive loans, based on information about the customer's conditions.  1 - Requirements First we must import some libraries that we will use throughout the code.<jupyter_code>import pandas as pd

from sklearn.model_selection import train_test_split, StratifiedShuffleSplit

import matplotlib.pyplot as plt #for the graphics

import numpy as np

import seaborn as sns

import matplotlib as mpl

from sklearn.impute import SimpleImputer

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import StandardScaler

%matplotlib inline

sns.set_theme()<jupyter_output><empty_output><jupyter_text>2 - Get the data We then collect data from the website: https://www.kaggle.com/vipin20/loan-application-data<jupyter_code>dataset = pd.read_csv('https://raw.githubusercontent.com/M-MSilva/Predict-The-Status-of-Loan-End-to-End-Project/master/Dataset/LOAN.csv',  on_bad_lines='skip',decimal='.', thousands=',',encoding='utf-8')<jupyter_output><empty_output><jupyter_text>3 - Initial Data Insights Now to better understand the dataframe we must show its first rows.<jupyter_code>dataset.head()<jupyter_output><empty_output><jupyter_text>In the dataset, each row characterizes an applicant who has 13 attributes. Of these attributes, many are easy to understand, so I have outlined a few:* ApplicantIncome: Applicant's Total Income;* CoapplicantIncome: Income from someone close, which will possibly help theindividual pay off the debt;* Loan_Amount_Term: Installments ex: 360 = 12 months;* Credit_History = 1 if the individual has a good credit history, 0 otherwise;* Loan_Status = means whether the loan will be approved or not.<jupyter_code>dataset.info()#to have a summary of the data
#statistical information
dataset.describe()<jupyter_output><empty_output><jupyter_text>5 - Understanding the data Now let's try to understand the data better and make some assumptions, from graphs extracted from the dataframe data.<jupyter_code>cmap = mpl.cm.Dark2

listA = ['ApplicantIncome','CoapplicantIncome','LoanAmount','Loan_Amount_Term']
fig=plt.figure(figsize=(12, 8))


for i, x in enumerate(listA):
  ax=fig.add_subplot(2,2,i+1)
  dataset[x].hist(bins=50,ax=ax,color=cmap(i/ float((i + 1))))
  if(x == 'Loan_Amount_Term'):
    ax.set_xlabel('Days')
  else:
    ax.set_xlabel('₹')
  ax.set_ylabel('Frequency Density')
  ax.set_title(x)
fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>In the graphs above we notice that the highest occurrences for Applicant's Income, Coapplicant's Income, Loan Amount, Term of Loan Amount are respectively ₹ 200,  ₹ 100, ₹ 120,  370 days approximately. The important thing to ask is, are people who have a higher income, or who apply for a higher loan amount, more likely to receive the loan ? Also, in our data data an interesting thing is that we have a lot of categorical data, so we'll see what we can get from that.<jupyter_code>cmap2 = mpl.cm.tab20

listFirst = ['Property_Area','Married','Dependents','Education','Self_Employed','Gender']

fig=plt.figure(figsize=(10,8))

for i, col in enumerate(listFirst):
  ax=fig.add_subplot(3,3,i+1)
  plt.xlabel('Number of Occurrences')
  plt.ylabel('Categories')
  valuec = dataset[col].value_counts().plot(kind='barh',title=col,color=cmap2(i/ float((i + 5))),grid=True,ax=ax)
  plt.subplots_adjust(left=0.125,
                    bottom=0.1, 
                    right=1.4, 
                    top=1.35, 
                    wspace=0.4, 
                    hspace=0.35)
plt.show()<jupyter_output><empty_output><jupyter_text>In these data we have, for example 233 people living in Semi-Urban areas, 202 people living in Urban areas and 179 cities living in Rural areas. In this way, another important issue comes into play. People who live in what type of area are most likely to receive a loan? This question and the previous one we will investigate in this project.  6 - Stratified Sampling: To perform the stratified population, we take subpopulations of a population and use the slicing method to subdivide the data. Stratification must be done, because if our dataset has a lot of data of a single type, our model may notice some wrong trend. But first we need to know which is the highest income and the lowest income:<jupyter_code>print('Maximum : ₹ ', max(dataset['ApplicantIncome']))
print('Minimum : ₹ ', min(dataset['ApplicantIncome']))<jupyter_output>Maximum : ₹  81000
Minimum : ₹  150<jupyter_text>we change some null data<jupyter_code>dataset['Credit_History'].fillna(dataset['Credit_History'].median(),inplace = True)
dataset.isnull().sum()<jupyter_output><empty_output><jupyter_text>We use the cut method and plot its content.<jupyter_code>dataset["hist_cat"] = pd.cut(dataset["Credit_History"],
                               bins=[-1,0.5,1.5],
                               labels=[0,1])

plt.xlabel('Categories of Income')
plt.ylabel('Frequency Density')
plt.title('Category Histograms For each Income')
dataset['hist_cat'].hist(color='tomato')
plt.show()<jupyter_output><empty_output><jupyter_text>After breaking down the populations, we actually performed the stratification.<jupyter_code>dataset.replace({'Loan_Status':{'Y': 1, 'N':0}},inplace=True)# I need to do this

from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(dataset, dataset["hist_cat"]):
    strat_train_set = dataset.iloc[train_index]
    strat_test_set = dataset.iloc[test_index]<jupyter_output><empty_output><jupyter_text>To check the consistency of our stratification we make the comparison:<jupyter_code>print(strat_train_set['hist_cat'].value_counts()/len(strat_train_set))
print(dataset['hist_cat'].value_counts()/len(dataset))<jupyter_output>1    0.855049
0    0.144951
Name: hist_cat, dtype: float64<jupyter_text>and we noticed a consistency in the results.  7 - Returning the data to the original state<jupyter_code>import warnings
warnings.filterwarnings("ignore")

strat_train_set.drop('hist_cat',axis=1, inplace=True)
strat_test_set.drop('hist_cat',axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>8 - Data Visualization To better interpret the concepts and notice patterns, let's visualize the data, which will also help us understand the possible decisions of our algorithm.<jupyter_code>dataset = strat_train_set.copy()

listaB = ['Gender','Married','Dependents','Education','Self_Employed','Property_Area','Credit_History','Loan_Amount_Term']

fig=plt.figure(figsize=(20, 14))

for i, col in enumerate(listaB):
  ax=fig.add_subplot(4,4,i+1)
  sns.countplot(x=col,hue='Loan_Status', data=dataset, ax=ax)
fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>The graph above is self-explanatory, it seems to indicate that married, childless, college-educated, non-self-employed men who live in semi-urban areas, who have a good credit history and who make 360-day payments are supposedly more likely to receive loans. Above we find a supposed answer to one of our questions raised in section 5, the other question asked outside whether there is any relationship between the applicant's income and the possibility of receiving loans. To speculate this question, we are going to plot a pairplot plot, which relates two variables simultaneously, in addition, we are going to distinguish in this data those who receive loans and those who do not.<jupyter_code>selected_columns = dataset[['ApplicantIncome','CoapplicantIncome','LoanAmount','Loan_Status']]

DATA2 = selected_columns.copy()

sns.pairplot(data=DATA2, hue="Loan_Status",height=3)
plt.show()<jupyter_output><empty_output><jupyter_text>From the graph above, it appears that for smaller amounts of applicant income, the co-applicant and loan amount are those that are usually approved for loan. Despite this, we cannot claim anything, as it may be that people with higher incomes do not take out loans, or our data lack information about this type of population. Maybe later we'll say something. Since the graph above didn't make any good statements, let's now plot a heatmap graph by re-correlating two data points and using a third ApplicantIncome value to tell us the magnitude of the Applicant Income for each bi-value. For example, in the chart below we can see the Applicant Income of women and men and whether they are married or not.<jupyter_code>List1 = ['Gender','Dependents','Loan_Amount_Term','Self_Employed']
List2 = ['Married','Education','Credit_History','Property_Area']

num = [0,1,2,3]

fig=plt.figure(figsize=(12, 10))
for (z,j,k) in zip(num,List1 ,List2):                   
  infos = dataset.pivot_table(index=j,columns=k,values="ApplicantIncome")
  ax=fig.add_subplot(2,2,z+1)
  sns.heatmap(infos, annot=True, fmt='.2f', linewidths=.5, ax=ax,cmap=plt.get_cmap("magma"))<jupyter_output><empty_output><jupyter_text>To make the assumptions, let's look at some information from Applicant Income:<jupyter_code>dataset['ApplicantIncome'].describe()<jupyter_output><empty_output><jupyter_text>Based on this information and the graph above, we note that some married men, people without children and graduates, individuals who request a lot of tuition with a good track record and people who are not self-employed, who live in semi-urban areas have incomes below the average ~ ₹ 5255.00 are more likely to receive a loan. From our first graph in section 7, we understand that these groups surveyed here would likely receive credit, as these groups fall into positive credit status. While these groups of people with below-average incomes indicate a likely certainty of receiving credit, the only certainty we can have will be with the implementation of the machine learning model, but these insights are precious.  9 - Exploratory Data Analysis and Feature Engineering  9.1 Search for Correlations Let's now concern ourselves in how variables depend on each other in our dataset. To measure the correlation coefficient we will use the corr() method.<jupyter_code>dataset2 = dataset.copy()
dataset2.replace({'Married':{'No':0,'Yes':1},'Gender':{'Male':1,'Female':0},'Self_Employed':{'No':0,'Yes':1},
                 'Property_Area':{'Rural':1,'Urban':2,'Semiurban':3},
                 'Education':{'Graduate':1,'Not Graduate':0}, 'Loan_Status':{'Y': 1, 'N':0}},inplace=True)


dataset2 = dataset2.replace(to_replace='3+', value=4)
Matrix_corr = dataset2.corr()
Matrix_corr<jupyter_output><empty_output><jupyter_text>The correlation coefficient close to 1 reveals a positive correlation between the data, if the correlation coefficient is close to -1 it expresses a negative correlation. For example, loan status has a strong correlation with credit history, and the lower the Coaplicant's salary, the lower the applicant's salary, as its correlation coefficient is close to -1. Furthermore, we can look at the data in the form of a heat map.<jupyter_code>mask = np.triu(np.ones_like(Matrix_corr))

f,ax = plt.subplots(figsize=(11,9))

sns.heatmap(Matrix_corr,mask=mask,cmap=plt.get_cmap("cividis"),center=0,square=True,linewidths=.5,cbar_kws={"shrink": .5})
plt.show()<jupyter_output><empty_output><jupyter_text>The conclusions seem to be the same, and we note the strong relationship between credit history and loan status.  9.2 - Creating New Data We will now make resource combinations to create new data.<jupyter_code>def feature_engineering(data):
  data['Total_Income'] = data['ApplicantIncome'] + data['CoapplicantIncome']
  data['Loan_Term_per_T_Income'] = data['Loan_Amount_Term']/data['CoapplicantIncome'] #installments per couple or helpers
  data['LoanA_per_Total_Income'] = data['LoanAmount']/data['CoapplicantIncome'] #Loan Amount by Total Income

  return data
corr_m = feature_engineering(dataset2).corr() #TOTAL
corr_m["Loan_Status"].sort_values(ascending=False)
corr_m = feature_engineering(dataset2).corr()#Aplicant
corr_m["Loan_Status"].sort_values(ascending=False)
corr_m = feature_engineering(dataset2).corr()#Coaplicant
corr_m["Loan_Status"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Although small, Loan Amount per Total Income is more closely tied to Loan Status than Loan Amount. Also, Loan amount Term per Total Income gives us more valuable information than Loan Amount Term. Similarly, Total Income is more closely associated with Loan Status than Applicant Income and Co-Applicant Income. The latter is undeniable, as the joint income of a couple is more relevant for the realization of a loan, than the income that only one person has, in this way, loans for people with lower wages tend to be accepted more frequently in our data. Dice. We arrive at the same statement as in section 8. Despite all this, we will not implement the model with this new data, as we have many values ​​that would give nan and zeros that divided would give nan.  9.3 Prepare the Data<jupyter_code>dataset = strat_train_set.drop("Loan_Status", axis=1) # drop labels for training set
dataset_labels = strat_train_set["Loan_Status"].copy()

dataset_num = dataset.select_dtypes(include=[np.number])#only the numerical data<jupyter_output><empty_output><jupyter_text>9.4 Transformations by Pipelines As seen so far, our data has many imperfections, to remedy some of these limitations, we employ a pepilines transformation that sequentially employs several transformations and a final estimator.In our pipeline:1.  First, we fill in data that is missing from the database, with the median of the column;2.   We apply RobustScaler(), to remove out-of-curve points and scale data between 25% of the sample and 75% of the sample;3. Finally, we use the StandardScaler, which makes the features have similar weights, normalizing them through the elimination of the mean and scaling to the range of units.<jupyter_code>from imblearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import RobustScaler

num_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy='median')),
        ('rbscaler',RobustScaler()),
        ('std_scaler', StandardScaler()),
        
    ])

dataset_num_tr = num_pipeline.fit_transform(dataset_num)
print(list(dataset_num))<jupyter_output>['ApplicantIncome', 'CoapplicantIncome', 'LoanAmount', 'Loan_Amount_Term', 'Credit_History']<jupyter_text>Later, we joined our numerical and categorical data through the ColumnTransformer and used the OneHotEncoder method, to transform words into numbers, as the models do not understand words, only numbers.We turn names into numbers such as labels that the pc understands.<jupyter_code>from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder


num_attribs = list(dataset_num)
cat_attribs = ['Gender','Married','Dependents','Education','Self_Employed','Property_Area']

full_pipeline = ColumnTransformer([
        ("num", num_pipeline, num_attribs),
        ("cat", OneHotEncoder(), cat_attribs)
    ])


dataset_before = full_pipeline.fit_transform(dataset)<jupyter_output><empty_output><jupyter_text>Finally, we apply the SMOTE() method, which oversamples the minority class and balances the distribution of classes, despite not providing additional content to the model.<jupyter_code>from imblearn.over_sampling import SMOTE


dataset_prepared, dataset_labels = SMOTE().fit_resample(dataset_before, dataset_labels)<jupyter_output><empty_output><jupyter_text>10 - Choose and Train The Model In the k-nearest neighbor model, objects are classified according to the plural vote of their neighbors, where each object is assigned the most similar class among its k-nearest neighbors.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

clf = KNeighborsClassifier(n_neighbors = 15)

clf.fit(dataset_prepared, dataset_labels)<jupyter_output><empty_output><jupyter_text>Our algorithm is already qualified, however, we will search for several hyperparameters to be tested in our model using cross validation. For this purpose we will use the grid search.<jupyter_code>from sklearn.model_selection import GridSearchCV

grid_params = {
    'n_neighbors' : [3,5,7,9,11,13,15],
    'weights' : ['uniform','distance'],
    'metric' : ['minkowski','euclidean','manhattan'],
    'algorithm' : ['auto', 'ball_tree', 'kd_tree', 'brute'],
}

gs = GridSearchCV(
    KNeighborsClassifier(),
    grid_params,
    verbose = 1,
    cv=3,
    n_jobs = -1,
    scoring='roc_auc'
)

clf = gs.fit(dataset_prepared, dataset_labels)

print('score: {:.3f}'
     .format(clf.score(dataset_prepared, dataset_labels)))<jupyter_output>Fitting 3 folds for each of 168 candidates, totalling 504 fits
score: 1.000<jupyter_text>We found the best parameters:<jupyter_code>print(f'Best parameters: {clf.best_params_}')
print(f'Best score: {clf.best_score_}')<jupyter_output>Best parameters: {'algorithm': 'auto', 'metric': 'manhattan', 'n_neighbors': 9, 'weights': 'distance'}
Best score: 0.8966989843028624<jupyter_text>Although our model is good, it is overfitting , which is basically learning very well but only the training data, i.e. it's like our model is biased in the training data, and when it comes across new data, it won't behave well then I will apply n_neighbors = 15.<jupyter_code>clf = KNeighborsClassifier(n_neighbors = 15)

clf.fit(dataset_prepared, dataset_labels)<jupyter_output><empty_output><jupyter_text>For a quick evaluation of the model, we used the Mean Square Error, which measured the mean squared difference between the recorded values ​​and the actual value.<jupyter_code>from sklearn.metrics import mean_squared_error


dataset_predictions = clf.predict(dataset_prepared)
lin_mse = mean_squared_error(dataset_labels, dataset_predictions)
lin_rmse = np.sqrt(lin_mse)
lin_rmse<jupyter_output><empty_output><jupyter_text>Our model is good, because the closer to zero the better.  11 - Evaluation method in TRAINING SET Now we will evaluate our model by applying the roc_curve curve and the auc value. The ROC curve describes the true positive rate by the false positive rate. In our case, we aim to have a high rate of true positives, as we would like to know the individuals who should receive the loan. The AUC value ranges from 0 to 1, and the closer to 1, the better are our estimates. To have a ROC curve we need something to use as a reference in our decision threshold, what we use is the score, which if it is greater than the threshold it conceives the instance in the positive class, otherwise not. To get the scores of the instances, we use the cross_val_predict method.<jupyter_code>from sklearn.model_selection import cross_val_predict 

y_scores_proba = cross_val_predict(clf, dataset_prepared, dataset_labels, cv=3,
                             method="predict_proba")

y_scores = y_scores_proba[:,1]<jupyter_output><empty_output><jupyter_text>Having the scores in hand, we check recall and accuracy.<jupyter_code>from sklearn.metrics import roc_curve,auc

pr, rec, thresholds = roc_curve(dataset_labels, y_scores)<jupyter_output><empty_output><jupyter_text>Finally we plot the graph:<jupyter_code>roc_auc_lr = auc(pr, rec)

fig, ax = plt.subplots(1, figsize= (8,6))
ax.set_aspect('equal') 
plt.xlim([-0.01, 1.00])
plt.ylim([-0.01, 1.01])


#plots the roc curve, with its coordinates 
plt.plot(pr, rec, lw=3, label='ROC curve (area = {:0.2f})'.format(roc_auc_lr))

plt.xlabel('False Positive Rate', fontsize=16)
plt.ylabel('True Positive Rate', fontsize=16)
plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16)
plt.legend(loc='lower right', fontsize=13)

#plots a list of values ​​that goes from zero to one, and how x = y is plotted a 45 degree line
plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')#plot the stippling

plt.show()<jupyter_output><empty_output><jupyter_text>12 - Implementation in Test Set Finally, we do the same procedure as before, only now in our test set.<jupyter_code>from sklearn.metrics import mean_squared_error

X_test = strat_test_set.drop("Loan_Status", axis=1)
y_test = strat_test_set["Loan_Status"].copy()

X_test_prepared = full_pipeline.transform(X_test)

final_predictions = clf.predict(X_test_prepared)

final_mse = mean_squared_error(y_test, final_predictions)
final_rmse = np.sqrt(final_mse)
final_rmse
from sklearn.model_selection import cross_val_predict

y_scores_proba = cross_val_predict(clf, X_test_prepared, y_test, cv=3,
                             method="predict_proba")


y_scores = y_scores_proba[:,1]
from sklearn.metrics import roc_curve,auc

pr, rec, thresholds = roc_curve(y_test, y_scores)
roc_auc_lr = auc(pr, rec)

fig, ax = plt.subplots(1, figsize= (8,6))
ax.set_aspect('equal') 
plt.xlim([-0.01, 1.00])
plt.ylim([-0.01, 1.01])


#plots the roc curve, with its coordinates
plt.plot(pr, rec, lw=3, label='ROC curve (area = {:0.2f})'.format(roc_auc_lr))

plt.xlabel('False Positive Rate', fontsize=16)
plt.ylabel('True Positive Rate', fontsize=16)
plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16)
plt.legend(loc='lower right', fontsize=13)

#plots a list of values ​​that goes from zero to one, and how x = y is plotted a 45 degree line
plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--') #plot the stippling

plt.show()<jupyter_output><empty_output><jupyter_text>Our model is great.  13 - To Deploy  13.1 - Saving the trained model Now let's save the model, to deploy it later.<jupyter_code>import pickle<jupyter_output><empty_output><jupyter_text>We use a dump method, which takes the clf model as an input and writes our classifier in binary to the file "wb".<jupyter_code>filename = 'trained_model.sav'
pickle.dump(clf, open(filename, 'wb'))<jupyter_output><empty_output><jupyter_text>loading the saved model<jupyter_code>loaded_model = pickle.load(open('trained_model.sav', 'rb'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09781021897810219, 'fraction_numerical': 0.01583029197080292, 'mean_word_length': 5.839625585023401, 'pattern_counts': {'":': 1, '<': 136, '<?xml version=': 0, '>': 136, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '477050', 'license': 'MIT', 'n_tokens_mistral': 7144, 'n_tokens_neox': 6669, 'n_words': 2218, 'path': 'ColabProject/Loan_Application_Prediction_End_to_End_Project.ipynb', 'repo_name': 'M-MSilva/Predict-The-Status-of-Loan_End-to-End-Project', 'size': '21920'}