<jupyter_start><jupyter_text>Project:                 ****Bank Credit Card Default Prediction**** Problem Statement                      The banks with the invent of credit card were more focused on the number of customers using their credit service but the drawback of them not being able to pay back the creditin time was an issue that soon followed, a system was in need to effectively decide the credit limit to be allowed to a person based on his previous credit history. We are going to build a classification model using **Logistic Regression**, **Random Forest** and **XGBoost** to predict the credibility of the customer,in order to minimize the risk and maximize the profit of German Credit Bank.                        Loading Required Libraries<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os as os

%matplotlib inline

import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Let's Check out the Data Set<jupyter_code>cr_train = pd.read_csv("Credit_Risk_Train_data.csv")
cr_valid = pd.read_csv("Credit_Risk_Validate_data.csv")
cr_test = pd.read_csv("Credit_Risk_Test_data.csv")
print("Dimension of Train Dataset:{}".format(cr_train.shape))
print("Dimension of Validate Dataset:{}".format(cr_valid.shape))
print("Dimension of Test Dataset:{}".format(cr_test.shape))
cr_train.head()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>cr_train.describe()
cr_train.info()
cr_train = cr_train.drop('Loan_ID', axis = 1)<jupyter_output><empty_output><jupyter_text>Missing Values<jupyter_code>cr_train.isnull().sum()
## Here we will check the percentage of nan values present in each feature

## 1 -step make the list of features which has missing values

features_with_na=[features for features in cr_train.columns if cr_train[features].isnull().sum()>1]

## 2- step print the feature name and the percentage of missing values

for feature in features_with_na:
    print(feature, np.round(cr_train[feature].isnull().mean(), 4),  ' % missing values')
#using heatmap for nullvalues
sns.heatmap(cr_train.isnull(), cmap = None, annot = None)
plt.show()
#columns name
cr_train.columns
# list of numerical variables
num_var = [feature for feature in cr_train.columns if cr_train[feature].dtypes != 'O']
cr_train[num_var]
# list of actegorical variables
cat_var = [feature for feature in cr_train.columns if cr_train[feature].dtypes == 'O']
cr_train[cat_var]<jupyter_output><empty_output><jupyter_text>Univariate Analysis<jupyter_code>## Lets analyse the numeric variables by creating histograms to understand the distribution

for feature in num_var:
    data=cr_train.copy()
    data[feature].hist(bins=25)
    plt.xlabel(feature)
    plt.ylabel("Count")
    plt.title(feature)
    plt.show()<jupyter_output><empty_output><jupyter_text>'ApplicantIncome', 'CoapplicantIncome' and 'LoanAmount' all these features are right skewed. This simply indicate that there are very less number of Applicant who has high income and also less number of applicant has asked for large amount for loan.<jupyter_code>cr_train['Credit_History'].value_counts()
sns.countplot(cr_train['Credit_History'])<jupyter_output><empty_output><jupyter_text>In this dataset we have more number of applicant having credit history as 1<jupyter_code>cr_train['Loan_Amount_Term'].value_counts()
sns.countplot(cr_train['Loan_Amount_Term'])
## Lets analyse the categorical varaibles

for feature in cat_var:
    data=cr_train.copy()
    sns.countplot(data[feature])
    plt.xlabel(feature)
    plt.ylabel("Count")
    plt.title(feature)
    plt.show()<jupyter_output><empty_output><jupyter_text>Bivariate Analysis<jupyter_code>## Lets analyse the categorical varaibles with respect to Target Variable

for feature in cat_var:
    data=cr_train.copy()
    tab = pd.crosstab(data[feature], cr_train["Loan_Status"])
    tab.plot.bar(stacked=True)
    plt.xlabel(feature)
    plt.ylabel("Count")
    plt.title("{} vs Loan Status".format(feature))
    plt.show()
## Lets analyse the numerical varaibles with respect to Target Variable

for feature in num_var:
    data=cr_train.copy()
    sns.barplot(x= "Loan_Status", y = feature, data = data)
    plt.xlabel("Loan_Status")
    plt.ylabel(feature)
    plt.title("{} vs Loan Status".format(feature))
    plt.show()<jupyter_output><empty_output><jupyter_text>Multivariate Analysis<jupyter_code>sns.pairplot(cr_train, diag_kind='kde')
cr_train.corr()<jupyter_output><empty_output><jupyter_text>LoanAmount is 57% correlated with ApplicantIncome, This indicate that applicant who has good income ask for big amount for loan.  Feature Engineering<jupyter_code>cr_train.head()
cr_train.isnull().sum()<jupyter_output><empty_output><jupyter_text>Handling Missing Values  Categorical VariablesImputation of missing values with their modular value in case of categorical feature.  Gender<jupyter_code>cr_train["Gender"].value_counts()
cr_train["Gender"].fillna("Male", inplace = True)<jupyter_output><empty_output><jupyter_text>Married<jupyter_code>cr_train["Married"].value_counts()
cr_train["Married"].fillna("Yes", inplace = True)<jupyter_output><empty_output><jupyter_text>Dependents<jupyter_code>cr_train["Dependents"].value_counts()
cr_train["Dependents"].fillna(0, inplace = True)<jupyter_output><empty_output><jupyter_text>Self_Employed<jupyter_code>cr_train["Self_Employed"].value_counts()
cr_train["Self_Employed"].fillna("No", inplace = True)<jupyter_output><empty_output><jupyter_text>Credit_History<jupyter_code>cr_train["Credit_History"].value_counts()
cr_train["Credit_History"].fillna(1, inplace = True)<jupyter_output><empty_output><jupyter_text>Numeric VariablesImputation of missing values with median value of feature in case of numerical feature.  LoanAmount<jupyter_code>med = cr_train["LoanAmount"].median(skipna=True);med
cr_train["LoanAmount"].fillna(128, inplace = True)<jupyter_output><empty_output><jupyter_text>Loan_Amount_Term<jupyter_code>med1 = cr_train["Loan_Amount_Term"].median(skipna=True);med1
cr_train["Loan_Amount_Term"].fillna(360, inplace = True)
cr_train.isnull().sum()<jupyter_output><empty_output><jupyter_text>All missing values get imputated  Validation Data Set<jupyter_code>cr_valid = cr_valid.drop('Loan_ID', axis = 1)
cr_valid.isnull().sum()
#Imputation of missing values

#categorical variables

cr_valid["Gender"].fillna("Male", inplace = True)
cr_valid["Dependents"].fillna(0, inplace = True)
cr_valid["Self_Employed"].fillna("No", inplace = True)
cr_valid["Credit_History"].fillna(1, inplace = True)

#numerical variables

cr_valid["LoanAmount"].fillna(cr_valid["LoanAmount"].median(skipna=True), inplace = True)
cr_valid["Loan_Amount_Term"].fillna(cr_valid["Loan_Amount_Term"].median(skipna=True), inplace = True)
cr_valid.isnull().sum()<jupyter_output><empty_output><jupyter_text>Testing Data set<jupyter_code>cr_test = cr_test.drop('Loan_ID', axis = 1)
cr_test.isnull().sum()
#Imputation of missing values

#categorical variables

cr_test["Gender"].fillna("Male", inplace = True)
cr_test["Dependents"].fillna(0, inplace = True)
cr_test["Self_Employed"].fillna("No", inplace = True)
cr_test["Credit_History"].fillna(1, inplace = True)

#numerical variables

cr_test["LoanAmount"].fillna(cr_test["LoanAmount"].median(skipna=True), inplace = True)
cr_test["Loan_Amount_Term"].fillna(cr_test["Loan_Amount_Term"].median(skipna=True), inplace = True)
cr_test.isnull().sum()<jupyter_output><empty_output><jupyter_text>Handling Categorical Variables  Training Data Set<jupyter_code>cr_train.head()
cr_train.info()
#creating dummy variables for categorical variables

#Gender
gender = pd.get_dummies(cr_train["Gender"], drop_first = True, dtype=int)

#Married
married = pd.get_dummies(cr_train["Married"], drop_first = True, dtype=int)

#Dependents
dependents = pd.get_dummies(cr_train["Dependents"], drop_first = True, dtype=int)

#Education
education = pd.get_dummies(cr_train["Education"], drop_first = True, dtype=int)

#Self_Employed
self_employed = pd.get_dummies(cr_train["Self_Employed"], drop_first = True, dtype=int)

#Credit_History
credit_history = pd.get_dummies(cr_train["Credit_History"], drop_first = True, dtype=int)

#Property_Area
property_area = pd.get_dummies(cr_train["Property_Area"], drop_first = True, dtype=int)

#Dropping actual columns 

cr_train.drop("Gender", axis = 1, inplace = True)
cr_train.drop("Married", axis = 1, inplace = True)
cr_train.drop("Education", axis = 1, inplace = True)
cr_train.drop("Self_Employed", axis = 1, inplace = True)
cr_train.drop("Dependents", axis = 1, inplace = True)
cr_train.drop("Property_Area", axis = 1, inplace = True)
cr_train.drop("Credit_History", axis = 1, inplace = True)
cr_train.head()
#concatenating dummy variables with cr_train

cr_train = pd.concat([gender,married,dependents,education,self_employed,credit_history,property_area, cr_train],axis = 1)
cr_train.head()<jupyter_output><empty_output><jupyter_text>Convert Loan_Status value of Y and N to a numerical value<jupyter_code>def loan_status(Loan_Status):
    if Loan_Status == "Y":
        return 1
    else :
        return 0
cr_train["Loan_Status"] = cr_train["Loan_Status"].apply(loan_status)
cr_train.head()
#final training data set

training_data = cr_train.copy()
x_train = training_data.drop("Loan_Status", axis = 1)

y_train = training_data["Loan_Status"]
x_train.head()
y_train.head()<jupyter_output><empty_output><jupyter_text>Validation Data Set<jupyter_code>cr_valid.head()
cr_valid.rename(columns = {"outcome":"Loan_Status"}, inplace = True)
cr_valid.head()
#creating dummy variables for categorical variables

#Gender
gender = pd.get_dummies(cr_valid["Gender"], drop_first = True, dtype=int)

#Married
married = pd.get_dummies(cr_valid["Married"], drop_first = True, dtype=int)

#Dependents
dependents = pd.get_dummies(cr_valid["Dependents"], drop_first = True, dtype=int)

#Education
education = pd.get_dummies(cr_valid["Education"], drop_first = True, dtype=int)

#Self_Employed
self_employed = pd.get_dummies(cr_valid["Self_Employed"], drop_first = True, dtype=int)

#Credit_History
credit_history = pd.get_dummies(cr_valid["Credit_History"], drop_first = True, dtype=int)

#Property_Area
property_area = pd.get_dummies(cr_valid["Property_Area"], drop_first = True, dtype=int)

#Dropping actual columns 

cr_valid.drop("Gender", axis = 1, inplace = True)
cr_valid.drop("Married", axis = 1, inplace = True)
cr_valid.drop("Education", axis = 1, inplace = True)
cr_valid.drop("Self_Employed", axis = 1, inplace = True)
cr_valid.drop("Dependents", axis = 1, inplace = True)
cr_valid.drop("Property_Area", axis = 1, inplace = True)
cr_valid.drop("Credit_History", axis = 1, inplace = True)
cr_valid.head()
#concatenating dummy variables with cr_valid

cr_valid = pd.concat([gender,married,dependents,education,self_employed,credit_history,property_area, cr_valid],axis = 1)
cr_valid.head()<jupyter_output><empty_output><jupyter_text>Convert Loan_Status value of Y and N to a numerical value<jupyter_code>def loan_status(Loan_Status):
    if Loan_Status == "Y":
        return 1
    else :
        return 0
cr_valid["Loan_Status"] = cr_valid["Loan_Status"].apply(loan_status)
cr_valid.head()
# final validation data set

validation_data = cr_valid.copy()
x_valid = validation_data.drop("Loan_Status", axis = 1)

y_valid = validation_data["Loan_Status"]
x_valid.head()
y_valid.head()<jupyter_output><empty_output><jupyter_text>Testing Data Set<jupyter_code>cr_test.head()
#creating dummy variables for categorical variables

#Gender
gender = pd.get_dummies(cr_test["Gender"], drop_first = True, dtype=int)

#Married
married = pd.get_dummies(cr_test["Married"], drop_first = True, dtype=int)

#Dependents
dependents = pd.get_dummies(cr_test["Dependents"], drop_first = True, dtype=int)

#Education
education = pd.get_dummies(cr_test["Education"], drop_first = True, dtype=int)

#Self_Employed
self_employed = pd.get_dummies(cr_test["Self_Employed"], drop_first = True, dtype=int)

#Credit_History
credit_history = pd.get_dummies(cr_test["Credit_History"], drop_first = True, dtype=int)

#Property_Area
property_area = pd.get_dummies(cr_test["Property_Area"], drop_first = True, dtype=int)

#Dropping actual columns 

cr_test.drop("Gender", axis = 1, inplace = True)
cr_test.drop("Married", axis = 1, inplace = True)
cr_test.drop("Education", axis = 1, inplace = True)
cr_test.drop("Self_Employed", axis = 1, inplace = True)
cr_test.drop("Dependents", axis = 1, inplace = True)
cr_test.drop("Property_Area", axis = 1, inplace = True)
cr_test.drop("Credit_History", axis = 1, inplace = True)
cr_test.head()
#concatenating dummy variables with cr_test

cr_test = pd.concat([gender,married,dependents,education,self_employed,credit_history,property_area, cr_test],axis = 1)
cr_test.head()
# final testing data set 

testing_data = cr_test.copy()
x_test = testing_data.copy()<jupyter_output><empty_output><jupyter_text>Model Creation I am going to create multiple binary classification models and then the model which will give good accuarcy and perform better in term of evaluation metrics, will going to select that model.I am also going to use hyperparameter tuning for hyperparameters.  Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
#intializing LogisticRegression to object logmodel

logmodel = LogisticRegression()
logmodel.fit(x_train,y_train)
#making prediction using validation data set
predictions = logmodel.predict(x_valid)<jupyter_output><empty_output><jupyter_text>Evaluation Metrics<jupyter_code>from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
print(confusion_matrix(predictions,y_valid))
print(accuracy_score(predictions,y_valid))
print(classification_report(predictions,y_valid))<jupyter_output>[[ 56   3]
 [ 21 287]]
0.9346049046321526
              precision    recall  f1-score   support

           0       0.73      0.95      0.82        59
           1       0.99      0.93      0.96       308

    accuracy                           0.93       367
   macro avg       0.86      0.94      0.89       367
weighted avg       0.95      0.93      0.94       367<jupyter_text>When using logistic regression, we are getting 93.46% as accuracy but here our dataset is imbalanced so we also have to take care of precision and recall values.Let's see if we able to improve these two values in further models.  Random Forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
rf=RandomForestClassifier()<jupyter_output><empty_output><jupyter_text>Hyperparameter optimization using RandomizedSearchCV<jupyter_code># Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]

# Number of features to consider at every split
max_features = ['auto', 'sqrt','log2']

# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 1000,10)]

# Minimum number of samples required to split a node
min_samples_split = [1, 2, 3, 4, 5, 7, 9]

# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4, 6, 8]

# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
              'criterion':['entropy','gini']}

print(random_grid)
rf_randomcv=RandomizedSearchCV(estimator=rf,param_distributions=random_grid,n_iter=100,cv=3,verbose=2,
                               random_state=100,n_jobs=-1)
### fit the randomized model
rf_randomcv.fit(x_train,y_train)
#best parameter
rf_randomcv.best_params_
#best estimator
rf_randomcv.best_estimator_
#best model from random search cv
rf_model = rf_randomcv.best_estimator_
#making prediction using validation data set
rf_predictions = rf_model.predict(x_valid)<jupyter_output><empty_output><jupyter_text>Evaluation Metrics<jupyter_code>print(confusion_matrix(rf_predictions,y_valid))
print(accuracy_score(rf_predictions,y_valid))
print(classification_report(rf_predictions,y_valid))<jupyter_output>[[ 58   1]
 [ 19 289]]
0.9455040871934605
              precision    recall  f1-score   support

           0       0.75      0.98      0.85        59
           1       1.00      0.94      0.97       308

    accuracy                           0.95       367
   macro avg       0.87      0.96      0.91       367
weighted avg       0.96      0.95      0.95       367<jupyter_text>Xgboost Classifier Here i am going to change column name because one duplicate name in column exist such as "Yes". And xgboost does not take dublicate feature name while training model.<jupyter_code>x_train.head()
 col_name = ['Male', 'Married', '0', '1', '2', '3+', 'Not Graduate', 'Self_Employed',
       'Credit_History_1', 'Semiurban', 'Urban', 'ApplicantIncome',
       'CoapplicantIncome', 'LoanAmount', 'Loan_Amount_Term']
x_train.columns = col_name
x_train.head()
x_valid.columns = col_name
x_test.columns = col_name
from sklearn.model_selection import RandomizedSearchCV
import xgboost
classifier=xgboost.XGBClassifier()<jupyter_output><empty_output><jupyter_text>Hyperparameter optimization using RandomizedSearchCV<jupyter_code>params={
 "learning_rate"    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] ,
 "max_depth"        : [ 3, 4, 5, 6, 8, 10, 12, 15],
 "min_child_weight" : [ 1, 3, 5, 7 ],
 "gamma"            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ],
 "colsample_bytree" : [ 0.3, 0.4, 0.5 , 0.7 ]
    
}
xg_randomcv = RandomizedSearchCV(classifier,param_distributions=params,n_iter=5,scoring='roc_auc',n_jobs=-1,cv=5,verbose=3)
xg_randomcv.fit(x_train,y_train)
#best parameters
xg_randomcv.best_params_
#best estimator
xg_randomcv.best_estimator_
#best model from random search cv
xg_model = xg_randomcv.best_estimator_
#mking prediction using validation data set
xg_predictions = xg_model.predict(x_valid)<jupyter_output><empty_output><jupyter_text>Evaluation Metrics<jupyter_code>print(confusion_matrix(xg_predictions,y_valid))
print(accuracy_score(xg_predictions,y_valid))
print(classification_report(xg_predictions,y_valid))
classifier=xgboost.XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=0.3, gamma=0.1, gpu_id=-1,
              importance_type='gain', interaction_constraints='',
              learning_rate=0.25, max_delta_step=0, max_depth=10,
              min_child_weight=5, monotone_constraints='()',
              n_estimators=100, n_jobs=4, num_parallel_tree=1, random_state=0,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,
              tree_method='exact', validate_parameters=1, verbosity=None)
from sklearn.model_selection import cross_val_score
score=cross_val_score(classifier,x_train,y_train,cv=10)
score
score.mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11228568423267066, 'fraction_numerical': 0.018407489218470602, 'mean_word_length': 5.8276481149012564, 'pattern_counts': {'":': 3, '<': 127, '<?xml version=': 0, '>': 128, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '564345', 'license': 'Apache-2.0', 'n_tokens_mistral': 6734, 'n_tokens_neox': 6343, 'n_words': 1416, 'path': 'BankCreditPrediction.ipynb', 'repo_name': 'imkushwaha/BankCreditPrediction', 'size': '19014'}