<jupyter_start><jupyter_text>Predict Loan Eligibility for Dream Housing Finance companyDream Housing Finance company deals in all kinds of home loans. They have presence across all urban, semi urban and rural areas. Customer first applies for home loan and after that company validates the customer eligibility for loan.Company wants to automate the loan eligibility process (real time) based on customer detail provided while filling online application form. These details are Gender, Marital Status, Education, Number of Dependents, Income, Loan Amount, Credit History and others. To automate this process, they have provided a dataset to identify the customers segments that are eligible for loan amount so that they can specifically target these customers. Note: Submission.csv should contain dataset of size 367 x 2 with fields Loan_id and Loan_status In this notebook we will be doing:--> EDA--> See how different algorithms perform without preprocessing, parameter tuning--> See which algorithm is performing best--> We will then perform preprocessing, then evaluate and later we will perform parameter tuning Steps to perform:=> EDA:1. visulaize2. write inferences=> Preprocessing:1. Handle null values2. removing not necessary features3. label encoding boolean/ordinal features and scaling numerical features=> Feature generation=> Training=> Parameter tuning<jupyter_code># Importing Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set_style("whitegrid")
# Importing dataset
train_df = pd.read_csv("train.csv")
test_df = pd.read_csv("test.csv")<jupyter_output><empty_output><jupyter_text>EDAEvery data science projects starts by understanding Data. So lets understand what our features are and what our target is<jupyter_code>train_df<jupyter_output><empty_output><jupyter_text>==> So it is a Binary classification problemLet us see what are our categorical features and numerical features<jupyter_code>null_data = train_df[train_df.isnull().any(axis=1)]
null_data
train_df.info()
train_df.describe()<jupyter_output><empty_output><jupyter_text>Here are a few inferences, you can draw by looking at the output of describe() function:1. LoanAmount has (614 – 592) 22 missing values.2. Loan_Amount_Term has (614 – 600) 14 missing values.3. Credit_History has (614 – 564) 50 missing values.4. We can also look that about 84% applicants have a credit_history. How? The mean of Credit_History field is 0.84 (Remember, Credit_History has value 1 for those who have a credit history and 0 otherwise)5. The ApplicantIncome distribution seems to be in line with expectation. Same with CoapplicantIncome6. Please note that we can get an idea of a possible skew in the data by comparing the mean to the median, i.e. the 50% figure.For the non-numerical values (e.g. Property_Area, Credit_History etc.), we can look at frequency distribution to understand whether they make sense or not. The frequency table can be printed by following command:<jupyter_code>train_df['Property_Area'].value_counts()<jupyter_output><empty_output><jupyter_text>Now let us see all null values in our dataset<jupyter_code>train_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>We do have null values in our dataset. Now it is time to figure out why do we have null values. Is it because the data was not provided by users or is it because the data doesn't exist with the user.Also let us first look if our test dataset contain any null values.<jupyter_code>test_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Test data too contains the null values. What to do with null values?We can do either of the following:Remove rows with null values or1. Replace with mean (mean imputation)2. Replace with median (median imputation)3. Replace with mode4. Replace with some constant and Add new feature isNull5. Train a model to predict null values6. Train our dataset with GBDT as it can handle null values.<jupyter_code>train_missing_cols = []
test_missing_cols = []
for col in train_df.columns:
    if train_df[col].isnull().sum() > 0:
        train_missing_cols.append(col)

for col in test_df.columns:
    if test_df[col].isnull().sum() > 0:
        test_missing_cols.append(col)
train_missing_cols
cat_cols = []
num_cols = []
for col in train_df.columns[1:-1]:
    if train_df[col].dtype=='O':
        cat_cols.append(col)
    else:
        num_cols.append(col)
cat_cols
l=len(cat_cols)
fig, axes = plt.subplots(int(l/3),3, figsize=(15,10))

for i in range(l):
    sns.countplot(x=cat_cols[i],data=train_df,hue='Loan_Status',ax=axes[i%2,i//2])<jupyter_output><empty_output><jupyter_text>Loan_Status is 'Y' (Generalization based on majority count):1. dependents is 02. self_employed is fasle3. education is graduate4. property_area is urban or semiurban5. Gender is male6. Married is yes<jupyter_code>#plot hist for numerical cols
l = len(num_cols)
fig, ax = plt.subplots(l,1,figsize=(10,18))
for i in range(l):
    ax[i].hist(train_df[num_cols[i]], bins=100)
    ax[i].set_xlabel(num_cols[i])<jupyter_output>/home/sidharthjain/anaconda3/envs/tensorflow_ml/lib/python3.7/site-packages/numpy/lib/histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/home/sidharthjain/anaconda3/envs/tensorflow_ml/lib/python3.7/site-packages/numpy/lib/histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>Insights:1. Most of the peoples income is between 2k-5k2. Many coapplicant's income is 03. Loan Amount mostly is between 90-1704. Loan Amount Term is taking only values 180(nearly 50) and 360 (500+) and few 3005. Many of them have credit history (1)<jupyter_code># # Below plot is simply plotted

# target = labelEncoder.fit_transform(train_df['ApplicantIncome'])
# l = len(cat_cols)
# fig, ax = plt.subplots(int(l/3),3, figsize=(15,10))

# for n in range(l):
#     i = n%2
#     j = n//2
#     sns.barplot(train_df[cat_cols[i]], target, ax=ax[i,j])
l = len(cat_cols)
fig, axes = plt.subplots(int(l/3),3, figsize=(15,10))

for i in range(l):
    sns.boxplot(x=cat_cols[i], y='ApplicantIncome', hue = 'Loan_Status', data=train_df, ax=axes[i%2, i//2])<jupyter_output><empty_output><jupyter_text>Pandas Profiling Report is just awesome!<jupyter_code># from pandas_profiling import ProfileReport
# prof = ProfileReport(train_df)
# prof<jupyter_output><empty_output><jupyter_text>Preprocessing1. Categorical encoding2. Handling missing values3. Scaling (We will see how the models perform with and without scaling)<jupyter_code>features = cat_cols+num_cols<jupyter_output><empty_output><jupyter_text>We can handle numerical and categorical features in many ways as follows.This is the easiest way of handling null values.Here, we are replacing categorical feature nan with the highest occuring values and numerical feature nan with mean values.<jupyter_code># #Filling null values

# imputed_train_df = train_df.copy()
# imputed_test_df = test_df.copy()

# for col in num_cols:
#     imputed_train_df[col].fillna(train_df[col].mean(), inplace=True)
#     imputed_test_df[col].fillna(train_df[col].mean(), inplace=True)

# for col in cat_cols:
#     imputed_train_df[col].fillna(train_df[col].mode()[0], inplace=True)
#     imputed_test_df[col].fillna(train_df[col].mode()[0], inplace=True)

# imputed_train_df = imputed_train_df[features]
# imputed_test_df = imputed_test_df[features]

# # Categorical encoding

# from sklearn.preprocessing import LabelEncoder

# for col in cat_cols:
#     labelEncoder = LabelEncoder()
#     imputed_train_df[col] = labelEncoder.fit_transform(imputed_train_df[col])
#     imputed_test_df[col] = labelEncoder.transform(imputed_test_df[col])

# imputed_train_df.head()<jupyter_output><empty_output><jupyter_text>Note: pd.get_dummnies() doesn't retain nan values and thus nan values may be filled with incorrect data.However, parameter dummy_na=True will add new features denoting null values<jupyter_code># #### Encoding categrical Features : ##########
# imputed_train_df = pd.get_dummies(train_df[features],dummy_na=True, drop_first=True)
# imputed_test_df = pd.get_dummies(test_df[features], dummy_na=True,drop_first=True)

# for col in num_cols:
#     imputed_train_df[col].fillna(train_df[col].mean(), inplace=True)
#     imputed_test_df[col].fillna(train_df[col].mean(), inplace=True)
# imputed_train_df.head()<jupyter_output><empty_output><jupyter_text>Now, we will use SimpleImputer from sklearn.impute to handle numerical null values by replacing them with mean.Note:  mean can be highly affected by outliers and thus median might be a better option.Try tweaking strategy='median'<jupyter_code>from sklearn.impute import SimpleImputer

### Encoding categrical Features: ##########
imputed_train_df = pd.get_dummies(train_df[features],drop_first=True)
retain_imputed_train_df_cols = imputed_train_df.columns
imputed_test_df = pd.get_dummies(test_df[features],drop_first=True)
retain_imputed_test_df_cols = imputed_test_df.columns

### Handling Numerical null values ########
simpleImputer = SimpleImputer(missing_values=np.nan, strategy='median')
imputed_train_df = pd.DataFrame(simpleImputer.fit_transform(imputed_train_df))
imputed_test_df = pd.DataFrame(simpleImputer.transform(imputed_test_df))
imputed_train_df.columns = retain_imputed_train_df_cols
imputed_test_df.columns = retain_imputed_test_df_cols
imputed_train_df.head()<jupyter_output><empty_output><jupyter_text>reference:https://towardsdatascience.com/using-pandas-transform-and-apply-to-deal-with-missing-data-on-a-group-level-cb6ccf060531<jupyter_code># Last observation carried forward
# train_df = train_df.fillna(method = 'ffill')  #forward-fill
# test_df = test_df.fillna(method='ffill')
# train_df = train_df.dropna(axis=0, subset=["LoanAmount"])

X = imputed_train_df
y = train_df[train_df.columns[-1]]
test_X = imputed_test_df<jupyter_output><empty_output><jupyter_text>Data splitting<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.33, random_state=42)
X<jupyter_output><empty_output><jupyter_text>Training without scaling<jupyter_code>from sklearn.linear_model import LogisticRegression as lrc
from sklearn.svm import SVC as svc
from sklearn.neighbors import KNeighborsClassifier as knnc
from sklearn.tree import DecisionTreeClassifier as dtc
from sklearn.ensemble import RandomForestClassifier as rfc
from sklearn.ensemble import GradientBoostingClassifier as gbc
from sklearn.naive_bayes import GaussianNB as gnb

from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
models = [lrc(max_iter=1000), svc(), knnc(), dtc(), rfc(), gbc(), gnb()]

print("Validation accuracy:\n")

i=0
for model in models:
    i+=1
    model.fit(X_train, y_train)
    cvpred = model.predict(X_val)
    pred = model.predict(test_X)
    my_submission = pd.DataFrame({'Loan_ID': test_df.Loan_ID, 'Loan_Status': pred})
    my_submission.to_csv('pred'+str(i)+'.csv', index=False)
    print(accuracy_score(y_val, cvpred))
    print(classification_report(y_val, cvpred))<jupyter_output>Validation accuracy:

0.7980295566502463
              precision    recall  f1-score   support

           N       0.94      0.46      0.62        72
           Y       0.77      0.98      0.86       131

    accuracy                           0.80       203
   macro avg       0.86      0.72      0.74       203
weighted avg       0.83      0.80      0.78       203

0.645320197044335
              precision    recall  f1-score   support

           N       0.00      0.00      0.00        72
           Y       0.65      1.00      0.78       131

    accuracy                           0.65       203
   macro avg       0.32      0.50      0.39       203
weighted avg       0.42      0.65      0.51       203

0.5665024630541872
              precision    recall  f1-score   support

           N       0.25      0.11      0.15        72
           Y       0.63      0.82      0.71       131

    accuracy                           0.57       203
   macro avg       0.44      0.46      0.43       [...]<jupyter_text>ScalingNote: Scaling shouldn't be done on categorical featuresDistance based algorithms(Knn, kmeans ,svm) and gradient based algorithms(Neural networks, Logistic regression, linear regression) doesn't perform well without scaling and thus let's apply scaling and see how algorithms perform.reference:https://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/<jupyter_code>from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train2 = X_train.copy()
X_val2 = X_val.copy()
X_train2[num_cols] = scaler.fit_transform(X_train[num_cols])
X_val2[num_cols] = scaler.transform(X_val[num_cols])


models = [lrc(max_iter=1000), svc(), knnc(), dtc(), rfc(), gbc(), gnb()]

print("Validation accuracy:\n")

i=0
for model in models:
    i+=1
    model.fit(X_train2, y_train)
    cvpred = model.predict(X_val2)
    pred = model.predict(test_X)
    my_submission = pd.DataFrame({'Loan_ID': test_df.Loan_ID, 'Loan_Status': pred})
    my_submission.to_csv('pred2'+str(i)+'.csv', index=False)
    print(accuracy_score(y_val, cvpred))
#     print(classification_report(y_val, cvpred))
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range = (0,1))
X_train3 = X_train.copy()
X_val3 = X_val.copy()
X_train3[num_cols] = scaler.fit_transform(X_train[num_cols])
X_val3[num_cols] = scaler.transform(X_val[num_cols])

models = [lrc(max_iter=1000), svc(), knnc(), dtc(), rfc(), gbc(), gnb()]

print("Validation accuracy:\n")
i=0
for model in models:
    i+=1
    model.fit(X_train3, y_train)
    cvpred = model.predict(X_val3)
    pred = model.predict(test_X)
    my_submission = pd.DataFrame({'Loan_ID': test_df.Loan_ID, 'Loan_Status': pred})
    my_submission.to_csv('pred3'+str(i)+'.csv', index=False)
    print(accuracy_score(y_val, cvpred))
#     print(classification_report(y_val, cvpred))<jupyter_output>Validation accuracy:

0.7980295566502463
0.7980295566502463
0.7093596059113301
0.6945812807881774
0.7783251231527094
0.7783251231527094
0.7980295566502463<jupyter_text>Note: Scaling does increase accuracy of svm, logistic regression. However, it doesn't affect tree based models and probability bases models.  Parameter Tuning<jupyter_code># from sklearn.model_selection import GridSearchCV
# models = [lrc(max_iter=2000), svc(), knnc(), dtc(), rfc()]
# param_grid = [{'C': np.logspace(0, 4, 10)},
#              {'C': [0.1, 1, 10, 100, 1000],  
#               'gamma': [1, 0.1, 0.01, 0.001, 0.0001], 
#               'kernel': ['rbf']}, 
#              {'leaf_size' : list(range(1,50)),
#               'n_neighbors' : list(range(1,30)),
#               'p': [1,2]},
#              {'criterion' : ['gini', 'entropy'], 
#               'max_depth' : [4,6,8,12]},
#              {'n_estimators': [200, 500],
#               'max_features': ['auto', 'sqrt', 'log2'],
#               'max_depth' : [4,5,6,7,8],
#               'criterion' :['gini', 'entropy']}
#              ]

# for i in range(len(models)):
#     clf = GridSearchCV(models[i], param_grid[i])
#     clf.fit(X_train, y_train)
#     cvpred = clf.predict(X_val)
#     pred = clf.predict(test_X)
#     my_submission = pd.DataFrame({'Loan_ID': test_df.Loan_ID, 'Loan_Status': pred})
#     my_submission.to_csv('pred4'+str(i)+'.csv', index=False)
#     print(accuracy_score(y_val, cvpred))
from sklearn import model_selection
results = []
seed = 7
scoring = 'accuracy'
models = [lrc(max_iter=2000), svc(), knnc(), dtc(), rfc(), gbc(), gnb()]
for model in models:
	kfold = model_selection.KFold(n_splits=10, random_state=seed, shuffle=True)
	cv_results = model_selection.cross_val_score(model, X_val2, y_val, cv=kfold, scoring=scoring)
	results.append(cv_results)
# 	names.append(name)
	msg = "%f (%f)" % (cv_results.mean(), cv_results.std())
	print(msg)<jupyter_output>0.778095 (0.075389)
0.782857 (0.060719)
0.792143 (0.071888)
0.724762 (0.075030)
0.772619 (0.085703)
0.757857 (0.066744)
0.802619 (0.067733)<jupyter_text>Among all the classifiers, only logistic regression and naive bayes performed well.Let us get more deeper if we can manipulate hyper parameters for the best hypothesis.<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import validation_curve
### 1. Use of validation curves for both datasets.
C_param_range = [0.001,0.01,0.1,1,10,100,1000]


# Apply logistic regression model to training data
lr = LogisticRegression(penalty='l2',C = i,random_state = 0, max_iter=1000)

# SEPAL Plot validation curve
train_score, test_score = validation_curve(estimator=lr
                                                            ,X=X
                                                            ,y=y
                                                            ,param_name='C'
                                                            ,param_range=C_param_range
                                                            )

train_mean = np.mean(train_score,axis=1)
train_std = np.std(train_score,axis=1)
test_mean = np.mean(test_score,axis=1)
test_std = np.std(test_score,axis=1)

plt.plot(C_param_range
            ,train_mean
            ,color='blue'
            ,marker='o'
            ,markersize=5
            ,label='training accuracy')
    
plt.plot(C_param_range
            ,test_mean
            ,color='green'
            ,marker='x'
            ,markersize=5
            ,label='test accuracy') 
    
plt.xlabel('C_parameter')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')
plt.ylim([0.5,1])<jupyter_output><empty_output><jupyter_text>Their is no way to increase its score by parameter tuning.Only way to increase score is to feature engineer if that is possible. Let us see how a neural network will perform on it.<jupyter_code>from keras.models import Sequential
from keras.layers import Dense

# train2 is standard scaled data
train_data = X_train2.copy()
train_label = [1 if _=='Y' else 0 for _ in y_train]
test_data = X_val2.copy()
test_label = [1 if _=='Y' else 0 for _ in y_val]

input_dim = train_data.shape[1]
output_dim = 1

model = Sequential()
#first hidden layer
model.add(Dense(10,input_dim=input_dim, activation='relu'))
# output layer
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(train_data, train_label, epochs=150, batch_size=10)
model.evaluate(test_data, test_label)
pred = model.predict_classes(test_data)
pred = pred.reshape(len(test_label),)
accuracy_score(pred, test_label)
print(classification_report(pred, test_label))<jupyter_output>precision    recall  f1-score   support

           0       0.49      0.85      0.62        41
           1       0.95      0.77      0.85       162

    accuracy                           0.79       203
   macro avg       0.72      0.81      0.74       203
weighted avg       0.86      0.79      0.81       203
{'fraction_non_alphanumeric': 0.106055851898337, 'fraction_numerical': 0.04120907854826901, 'mean_word_length': 4.041655681518587, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 93, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '331139', 'license': 'MIT', 'n_tokens_mistral': 6826, 'n_tokens_neox': 6303, 'n_words': 1631, 'path': 'loan-prediction/Loan prediction.ipynb', 'repo_name': 'Sidharth99/Data-Science', 'size': '19122'}