<jupyter_start><jupyter_text>Table of ContentsAbout the companyBusiness problemThis notebook has two parts:Import librariesGet to know your dataTrain dataTest dataSome helper functions for null value treatmentImputing train and test dataExploratory data analysisUnivariate analysisLoan statusGender, Married, Self_Employed, Credit_HistoryDependents, Education, Property_AreaApplicant IncomeCoapplicant IncomeLoan AmountLoan Amount TermCredit historyBivariate AnalysisCategorical vs Target VariableNumerical vs Target VariableApplicantIncomeCoapplicantIncomeLoan amountData preparationEncoding categorical variableReducing skewnessOutputLink to part 2Show your support by upvoting the kernel 👍 <img src="https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcTQ2OEfjwBg26hXQEle2XqFBDp0coC1dVugLBxLZ_5_EFTIS20O"              width="650px">  About the companyDream Housing Finance company deals in all home loans. They have presence across all urban, semi urban and rural areas. Customer first apply for home loan after that company validates the customer eligibility for loan. Business problemCompany wants to automate the loan eligibility process (real time) based on customer detail provided while filling online application form. These details are Gender, Marital Status, Education, Number of Dependents, Income, Loan Amount, Credit History and others. To automate this process, they have given a problem to identify the customers segments, those are eligible for loan amount so that they can specifically target these customers. Here they have provided a partial data set. **Reference**: [Analytics vidhya - Loan status prediction competition](https://datahack.analyticsvidhya.com/contest/practice-problem-loan-prediction-iii/) ---  *This notebook has two parts:***Part 1: [Data exploration & cleaning](https://www.kaggle.com/psvishnu/loan-prediction-part-1/)**    **Part 2: [Modelling & Tuning](https://www.kaggle.com/psvishnu/loan-prediction-part-2/)**  Import libraries Load all the necessary dependencies which will be required for the purpose of analysis.<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
import seaborn as sns
from statistics import mode
from scipy import stats
from warnings import filterwarnings 
import os
filterwarnings('ignore')
print(os.listdir("../input/"))
plt.rcParams['figure.figsize'] = 10,6
%matplotlib inline<jupyter_output>['train3', 'test-file']<jupyter_text>Get to know your data<jupyter_code>train_df = pd.read_csv("../input/train3/train.csv",index_col="Loan_ID")
test_df = pd.read_csv("../input/test-file/test.csv")

print('Train data shape', train_df.shape)
print('Test data shape', test_df.shape)
# print(train_df.columns)<jupyter_output>Train data shape (614, 12)
Test data shape (367, 12)<jupyter_text>*Train data*<jupyter_code>train_df.head().style.background_gradient(cmap='Blues')<jupyter_output><empty_output><jupyter_text>*Test data*<jupyter_code>test_df.sample(3).style.background_gradient(cmap='Greens')
pd.DataFrame(train_df.dtypes,columns=['Data type']).T<jupyter_output><empty_output><jupyter_text>We have two types of variables in our dataframe - *categorical and numerical.***1. Categorical features**- Nominal features: These features have categories (Gender, Married, Self_Employed, Credit_History, Loan_Status).- Ordinal features: Variables in categorical features having some order involved (Dependents, Education, Property_Area).**2. Numerical features**: - Continuous features: These features have continuously distributed values (ApplicantIncome, CoapplicantIncome, LoanAmount).- Discrete featuress: Values in these columns are unique and non-continous in nature( Loan_Amount_Term, Credit_history ). Let's check for null values in the Train Test data<jupyter_code>plt.figure(figsize=(18,6))
plt.subplot(1,2,1)
train_df.isna().sum().sort_values().plot.barh(color='blue',alpha=0.7)
plt.title('Train data')
plt.subplot(1,2,2)
test_df.isna().sum().sort_values().plot.barh(color='green',alpha=0.7)
plt.title('Test data')
plt.show()<jupyter_output><empty_output><jupyter_text>There are many missing values in both Train and Test data, next we will impute them (replace with appropirate values).  Some helper functions for null value treatment Imputing null values with mean, median or mode value is a straight forward method.But for best quality data we should impute based on the domain knowledge of the column. One important drawback of first method, if the number of null values are more then it can introduce spikes in the distribution.In this case, we will be replacing the null values with forward/backward fill method. This method is fast and maintains the overall distribution of the data.<jupyter_code>def get_null_columns(df):
    null_cols = df.isna().sum().sort_values(ascending=False)
    null_cols = null_cols[null_cols > 0]
    if len(null_cols):
        plt.title('Null value count before imputation')
        sns.barplot(null_cols.values,null_cols.index)
        return null_cols
    else:
        return None
    
def get_num_cols(df):
    return df.select_dtypes(include="number").columns.values

def get_cat_cols(df):
    return df.select_dtypes(exclude="number").columns.values

def treat_null_cols(df):
    nulls= get_null_columns(df)
    
    if nulls is not None:
        null_columns = nulls.index.values.tolist()
        for col in null_columns:
            if df[col].dtype == 'object':
                df.loc[:,col]=df.loc[:,col].fillna(method='ffill')
                df.loc[:,col]=df.loc[:,col].fillna(method='bfill')
            # Numerical columns    
            elif col in ("Credit_History","Loan_Amount_Term"):
                df[col].fillna(df[col].mode()[0],inplace=True)
            else:
                df[col].fillna(df[col].median(),inplace=True)
        print("Columns after imputation".center(38,'='))
        df.info()
    else:
        print('No null values !')<jupyter_output><empty_output><jupyter_text>Imputing train and test data<jupyter_code>treat_null_cols(train_df)
treat_null_cols(test_df)<jupyter_output>=======Columns after imputation=======
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 367 entries, 0 to 366
Data columns (total 12 columns):
Loan_ID              367 non-null object
Gender               367 non-null object
Married              367 non-null object
Dependents           367 non-null object
Education            367 non-null object
Self_Employed        367 non-null object
ApplicantIncome      367 non-null int64
CoapplicantIncome    367 non-null int64
LoanAmount           367 non-null float64
Loan_Amount_Term     367 non-null float64
Credit_History       367 non-null float64
Property_Area        367 non-null object
dtypes: float64(3), int64(2), object(7)
memory usage: 34.5+ KB<jupyter_text>Exploratory data analysis  Univariate analysis> For categorical features we can use frequency table or % distribution which will calculate the value per category.> For numerical features, probability density plots can be used to check the distribution of the variable.  Loan status **Dependant variable**<jupyter_code>f, ax  = plt.subplots(1,2,figsize = (18,6))
sns.countplot(train_df['Loan_Status'],ax=ax[0],palette = 'bright',alpha=0.7)

loan_status_count = train_df['Loan_Status'].value_counts(normalize=True)
loan_status_count.plot.pie(autopct="%.2f%%",explode = [0,0.2],ax=ax[1])
plt.title('% of loan approval')
plt.show()<jupyter_output><empty_output><jupyter_text>We can see an imbalance here, the number of **Loan approved is more than twice the rejections.** ---  Gender, Married, Self_Employed, Credit_History**Nominal variables**<jupyter_code>plt.figure(1)
plt.subplot(221)
train_df['Gender'].value_counts().plot.pie(title='% distribution of Gender',figsize=(20,10),autopct="%1.1f",explode=[0,.1])
# train_df['Gender'].value_counts(normalize=True).plot.bar(figsize=(20,10), title= 'Gender')

plt.subplot(222)
train_df['Married'].value_counts().plot.pie(title='% distribution of Marital Status',autopct="%1.1f",explode=[0,.1])

plt.subplot(223)
train_df['Self_Employed'].value_counts().plot.pie(title='% distribution of Employment type',autopct="%1.1f",explode=[0,.1])

plt.subplot(224)
train_df['Credit_History'].value_counts().plot.pie(title='% distribution of credit history',autopct="%1.1f",explode=[0,.1])
plt.show()<jupyter_output><empty_output><jupyter_text>From the above chart we can see that most of the people who applied for loans were either `Males`, `Married`, `No self employment` or `Had a credit history`. ---  Dependents, Education, Property_Area**Ordinal Variables**<jupyter_code>plt.figure(figsize=(20, 10))
plt.subplot(231)
sns.countplot(train_df['Dependents'],palette="Accent")
plt.subplot(234)
train_df['Dependents'].value_counts().plot.pie(cmap="Accent",autopct="%1.1f%%",explode=[0,0,0,0.2])


plt.subplot(232)
sns.countplot(train_df['Education'],palette="Accent")
plt.subplot(235)
train_df['Education'].value_counts().plot.pie(cmap="Accent",autopct="%1.1f%%",explode=[0,0.1])


plt.subplot(233)
sns.countplot(train_df['Property_Area'],palette="Accent")
plt.subplot(236)
train_df['Property_Area'].value_counts().plot.pie(cmap="Accent",autopct="%1.1f%%",explode=[0,0,0.1])
plt.show()<jupyter_output><empty_output><jupyter_text>From the above charts we can infer that, - Most of the applicants(57.6%) did not have any dependants- 78.2% are Graduates applying for loan.- Eventhough most of the applicants are from Semiurban, an almost equal distribution can be seen among other categories. --- **Numerical Variables**  Applicant Income<jupyter_code>plt.figure(1,figsize=(16,5))
plt.subplot(121)
print("Skewness:",train_df['ApplicantIncome'].skew())
sns.distplot(train_df['ApplicantIncome'],fit = stats.norm);

plt.subplot(122)
sns.boxplot(train_df['ApplicantIncome'])

plt.show()<jupyter_output>Skewness: 6.539513113994625<jupyter_text>- Applicant Income is left skewed (6.53).- Many outliers are present in the distribution.- Most of the applicant income is near a range of 5000. **Let us check whether this non uniformity in Income is due to the difference in education level**<jupyter_code>plt.figure(figsize=(14,5))
ax = sns.boxplot(data=train_df,x="ApplicantIncome",y="Education",palette='Blues')
fig = plt.gcf()
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>It can be observed that variation in Application income is more among individuals who are Graduates. ---  Coapplicant Income<jupyter_code>plt.figure(1,figsize=(16,5))
plt.subplot(121)
print(train_df['CoapplicantIncome'].skew())
sns.distplot(train_df['CoapplicantIncome'],fit=stats.norm)

plt.subplot(122)
sns.boxplot(train_df['CoapplicantIncome'],color='c')
fig = plt.gcf()
plt.tight_layout()

plt.figure(2,figsize=(16,5))
ax = sns.boxplot(data=train_df,x="CoapplicantIncome",y="Education")
plt.show()<jupyter_output>7.491531216657306<jupyter_text>- Similar to Applicant Income, Coapplicant income is also left skewed(7.49) and not normally distributed.- Also, there are a numer of outliers in the distribution. One of which can be attribued to the level of education. ---  Loan Amount<jupyter_code>plt.figure(1)
plt.subplot(121)

# distribution plots cannot handle NaN
train_df = train_df.dropna()
sns.distplot(train_df['LoanAmount'],fit=stats.norm)

plt.subplot(122)
# Figsize = width * length
train_df['LoanAmount'].plot.box(figsize=(15,7))
plt.tight_layout()
print("Skewness:",train_df['LoanAmount'].skew())<jupyter_output>Skewness: 2.743052748820892<jupyter_text>The above three columns(Applicant income, coapplicant income and loan amount) have outliers and a high level of skewness.We can later use `log transformation` to remove the skewness and it will even help to scale down the outliers. --- Discrete Numeric values  Loan Amount Term<jupyter_code>plt.figure(figsize=(16,5))
lat = train_df['Loan_Amount_Term'].value_counts(normalize=True)
sns.barplot(x=lat.values,y=lat.index, palette="rocket",orient='h')
plt.show()<jupyter_output><empty_output><jupyter_text>Either most of the applicant opted for a specific 30 year (360 months) term followed by 15 years (180 month) term or our sample data is having more of such datapoints. ---  Credit history<jupyter_code>sns.countplot(train_df['Credit_History'],palette="spring")
plt.show()<jupyter_output><empty_output><jupyter_text>Having more data points with credit history might make our model better at predicting values with a history. ---  Bivariate Analysis<jupyter_code># Removing Loan_Status, since it is our target variable
cat_columns = get_cat_cols(train_df).tolist()
cat_columns.remove('Loan_Status')
n_cat_cols = len(cat_columns)

num_cols = get_num_cols(train_df)
n_num_cols = len(num_cols)

target = 'Loan_Status'<jupyter_output><empty_output><jupyter_text>Categorical vs Target Variable We will try to find answers to questions like these1. Is gender or being self employed a factor for loan approvals?2. Does being married decreases your chance of getting a loan?3. Does having lower number of dependants gives you an upper hand?4. Will being in Urban location make loan approval easy?<jupyter_code>fig,ax = plt.subplots(n_cat_cols,1,figsize=(8,6*n_cat_cols))

for i,col in enumerate(cat_columns):
    # Create a cross table for stacked graph
    pd.crosstab(train_df[col],train_df[target])
    ct = pd.crosstab(train_df[col],train_df[target],normalize="index")
    ct.plot.barh(stacked=True,ax=ax[i])
    
plt.show()<jupyter_output><empty_output><jupyter_text>- There was no significant pattern observed based on the `gender` or `Self_Employed` status.- Proportion of `married applicants` is higher for the approved loans.- If one is having 2 `Dependents` the person is having more approval rate compared to `0`(no dependent) or other dependents number, which is quite strange.- `Graduates` were having more loan approval rate than their counterpart.- Property area plays a role in approval too, but most of it belongs to `Suburban` region rather than `Urban` or `Rural`. ---  Numerical vs Target Variable Lets make some new features for performing better EDA.Three bins Low, Medium and High each for1. Applicant Income2. Coapplicant Income3. LoanAmount<jupyter_code>groups = ['Low','Medium','High']

def get_categories(x):
    if x < q1:
        return groups[0]
    elif x < q3:
        return groups[1]
    else:
        return groups[2]    

for col_name in ['ApplicantIncome','CoapplicantIncome','LoanAmount']:
    q1 = train_df[col_name].quantile(q=0.25)
    q3 = train_df[col_name].quantile(q=0.75)
    train_df[col_name+'_cat'] = train_df[col_name].apply(get_categories)<jupyter_output><empty_output><jupyter_text>ApplicantIncome<jupyter_code>train_df.groupby('Loan_Status')['ApplicantIncome'].mean().plot.bar()
plt.show()<jupyter_output><empty_output><jupyter_text>There are not much changes in the mean income. Lets check based on applicant income category.<jupyter_code>cross = pd.crosstab(train_df['ApplicantIncome_cat'],train_df['Loan_Status'],normalize="index")
print(cross)
cross.plot.bar(stacked=True)<jupyter_output>Loan_Status                 N         Y
ApplicantIncome_cat                    
High                 0.318182  0.681818
Low                  0.311688  0.688312
Medium               0.310458  0.689542<jupyter_text>It can be inferred that Applicant income does not affect the chances of loan approval which contradicts our hypothesis where we assumed that if the applicant income is high the chances of loan approval will also be high. ---  CoapplicantIncome<jupyter_code>cross = pd.crosstab(train_df['CoapplicantIncome_cat'],train_df['Loan_Status'],normalize="index")
print(cross)
cross.plot.barh(stacked=True)<jupyter_output>Loan_Status                   N         Y
CoapplicantIncome_cat                    
High                   0.311688  0.688312
Medium                 0.313043  0.686957<jupyter_text>--- Loan amount<jupyter_code>cross = pd.crosstab(train_df['LoanAmount_cat'],train_df['Loan_Status'],normalize="index")
print(cross)
cross.plot.barh(stacked=True)<jupyter_output>Loan_Status            N         Y
LoanAmount_cat                    
High            0.357143  0.642857
Low             0.305195  0.694805
Medium          0.294118  0.705882<jupyter_text>It can be seen that the proportion of approved loans is higher for Low and Medium Loan Amount as compared to that of High Loan Amount. ---  Data preparation- Let’s drop the bins which we created for EDA. - Categorical columns: We will encode the categorical columns to numbers.- Numerical columns: There were continuous numerical variables with outliers and skewness, those will be transformed by applying log operation.We will also convert the target variable’s categories into 0 and 1 so that we can find its correlation with numerical variables. One more reason to do so is that few models like logistic regression only understands numeric values as input. We will replace N with 0 and Y with 1.<jupyter_code>train_df = train_df.drop(['ApplicantIncome_cat', 'CoapplicantIncome_cat', 'LoanAmount_cat'], axis=1)<jupyter_output><empty_output><jupyter_text>Encoding categorical variable<jupyter_code># Property_Area
col_name="Property_Area"
d = {'Urban':2,'Semiurban':1,'Rural':0}
train_df[col_name].replace(d,inplace=True)
test_df[col_name].replace(d,inplace=True)

# Self_Employed
col_name="Self_Employed"
d = {'Yes':1,'No':0}
train_df[col_name].replace(d,inplace=True)
test_df[col_name].replace(d,inplace=True)

# Education
col_name="Education"
d ={'Graduate':1, 'Not Graduate':0}
train_df[col_name].replace(d,inplace=True)
test_df[col_name].replace(d,inplace=True)

# Married
d = {'Yes':1,'No':0}
train_df['Married'].replace(d,inplace=True)
test_df['Married'].replace(d,inplace=True)

# Gender
gender = {'Male':1,'Female':0}
train_df['Gender'].replace(gender,inplace=True)
test_df['Gender'].replace(gender,inplace=True)

# Dependents
d = {'3+':3}
train_df['Dependents'].replace(d,inplace=True)
test_df['Dependents'].replace(d,inplace=True)

#Loan status
loan_status = {'N':0,'Y':1}
train_df['Loan_Status'].replace(loan_status,inplace=True)<jupyter_output><empty_output><jupyter_text>Reducing skewness<jupyter_code>#Loan Amount
col_name = 'LoanAmount'
train_df[col_name] = np.log(train_df[col_name])
print(train_df[col_name].skew())
ax = sns.distplot(train_df[col_name],fit=stats.norm)
# Applicant Income
col_name = 'ApplicantIncome'
train_df[col_name] = np.log(train_df[col_name])
print(train_df[col_name].skew())
ax = sns.distplot(train_df[col_name],fit=stats.norm)
# Coapplicant Income
col_name = 'CoapplicantIncome'
# Since there are 0 values we will have to use log1p to remove infinite values
train_df[col_name] = np.log1p(train_df[col_name])
print(train_df[col_name].skew())
ax = sns.distplot(train_df[col_name],fit=stats.norm)<jupyter_output>-0.17307327599230946<jupyter_text>--- Now lets look at the correlation between all the numerical variables. We will use the heat map to visualize the correlation.Heatmaps visualize data through variations in coloring. The variables with darker color means their correlation is more.<jupyter_code>matrix = train_df.corr()
f, ax = plt.subplots(figsize=(12, 6))
with sns.axes_style("white"):
    sns.heatmap(matrix,mask=np.triu(matrix,1),annot=True,fmt=".2f", vmax=.8,cbar=False,cmap="coolwarm");<jupyter_output><empty_output><jupyter_text>Looking at the correlation map we can see that `credit history` is the most important feature for `Loan Status`, with no other feature even close to it. So existing internal customers will be always preferred over the new ones.Also, `Applicant Income` and `LoanAmount` are correlated with each other. ---  Output<jupyter_code># Sending out the cleaned train and test data as output.
train_df.to_csv('train_clean.csv')
test_df.to_csv('test_clean.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11338646222132553, 'fraction_numerical': 0.025622919398769293, 'mean_word_length': 5.680256064690027, 'pattern_counts': {'":': 0, '<': 107, '<?xml version=': 0, '>': 108, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '128168', 'license': 'MIT', 'n_tokens_mistral': 6860, 'n_tokens_neox': 6319, 'n_words': 1671, 'path': 'notebooks/Loan prediction - part 1.ipynb', 'repo_name': 'p-s-vishnu/Loan-status-predictor', 'size': '19826'}