<jupyter_start><jupyter_text>**Credit Card Default Prediction**  **Introduction** In this notebook, I will using the dataset from [Credit Card Payment Default Prediction Hackathon](https://www.kaggle.com/reverie5/av-janata-hack-payment-default-prediction?select=train_20D8GL3.csv). The dataset contains information about the payment default, demographic factors, credits, payment history, and bill of amount of the credit card users in Taiwan from April to September 2005. The goal of this notebook is to `preprocess` the dataset so it can be used for `Exploratory Data Analysis (EDA)` and `Machine Learning Modeling`. Overall, the goal of project is to build a model that could predict which client payment will default in the next month.   **Data Preprocessing**  **A. Import Libraries and Load Dataset**<jupyter_code>## import libraries
import numpy as np
import pandas as pd
from scipy import stats

import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use("ggplot")
pd.options.display.max_columns = 100

%matplotlib inline

import warnings
warnings.filterwarnings('ignore')
## connect to drive
from google.colab import drive
drive.mount("/content/gdrive")
 ## load dataset to dataframe
df = pd.read_csv("/content/gdrive/MyDrive/final-project/datasets/train_20D8GL3.csv")

print("dataset's shape: ", df.shape)<jupyter_output>dataset's shape:  (21000, 25)<jupyter_text>**B. Dataset Exploration** We will start by explore our raw dataset<jupyter_code>## show first five rows of the dataset
df.head()<jupyter_output><empty_output><jupyter_text>We can take a look the summary, characteristics and descriptive statistics of the dataset we have berfore we go more deeper.<jupyter_code>## summary information
df.info()
## desctriptive statistics
df.describe().T<jupyter_output><empty_output><jupyter_text>**Dataset Description** This dataset had 25 columns (24 features and 1 target) and 21000 entry rows. Generally there's no missing or null values in every rows. Average credit limit that customer have is 167214 New Taiwan Dollar (NTD), while the highest credit limit is 800000 NTD. Average credit card holder age is 35.4 years old, the youngest age is 21 and the oldest is 75. On target column, *default_payment_next_month*, we can see there's 22.1% credit card holder that will default on the next month (October). All these features could have strong or weark connection to our target. That's why we need to explore more so we can get better understanding.  **Columns Description** *   **ID**: Unique id for every clients*   **LIMIT_BAL**: Credit limit given (New Taiwan Dollar)* **SEX**: Gender (1=man, 2=women)* **EDUCATION**: Education level (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown)* **MARIAGE**: Mariage Status (1=married, 2=single, 3=others)* **AGE**: Client's age* **PAY_0**: Payment status in September 2005 (-1=on time, 1=one month overdue, 2=two month overdue,....8=eight month overdue)* **PAY_2**: Payment status in August 2005* **PAY_3**: Payment status in July 2005* **PAY_4**: Payment status in June 2005* **PAY_5**: Payment Status in May 2005* **PAY_6**: Payment status in April 2005* **BILL_AMT1**: Bill amount in September 2005* **BILL_AMT2**: Bill amount in August 2005* **BILL_AMT3**: Bill amount in July 2005* **BILL_AMT4**: Bill amount in June 2005* **BILL_AMT5**: Bill amount in May 2005* **BILL_AMT6**: Bill amount in April* **PAY_AMT1**: Payment amount in September 2005* **PAY_AMT2**: Payment amount in August 2005* **PAY_AMT3**: Payment amount in July 2005* **PAY_AMT4**: Payment amount in June 2005* **PAY_AMT5**: Payment amount in May 2005* **PAY_AMT6**: Payment amount in April 2005* **default_payment_next_month**: Target, dafault payment status (1=yes, 0=no) **Additional notes**:Base on description from dataset's metadata in Kaggle, value -1 that we have in `PAY_0` to `PAY_6` column represent for on time payment, while value 1 to 8 represent overdue payment. But as we can see when we called describe() function on our dataset, there's value -2 and value 0. From what I understand, value -2 represent that the credit card holder doesn't have any bill, while value 0 represent half or some payment from the baill amount. We can visualize PAY_0 column to how many -2 and 0 represent on payment status.<jupyter_code>plt.figure(figsize=(10,8))
plt.title("Payment Status in September 2005")
sns.countplot(data=df, x="PAY_0")

plt.show()<jupyter_output><empty_output><jupyter_text>We can see that we have a lot of value 0 and value -2, that's why we cannot ignore it even it's not in metadata description.  **C. Data Cleansing** In this section, we'l try to cleansing our dataset such as handling missing values, duplicated data, and outliers.  **Missing Values Handling** When we called `info()` funtion on our dataset abuve, we can see that there's no missing values for every row in every column that we had. That's why we don't need to handling anything here, because we don't have missing values. But for your next reference in the next project, we can create a funtion that will detect how many missing values in the dataset.<jupyter_code>def missing_val(data):
    df_cols = list(data.columns)
    cols_total_count = len(list(data.columns))
    cols_count = 0
    for loc, col in enumerate(df_cols):
        null_count = data[col].isnull().sum()
        total_count = data[col].isnull().count()
        percent_null = round(null_count/total_count*100, 2)
        if null_count > 0:
            cols_count += 1
            print('{} column has {} missing/null values: {}% null'.format(loc, col, null_count, percent_null))
    cols_percent_null = round(cols_count/cols_total_count*100, 2)
    print('From {} columns, {} has missing/null values; {}% column has missing/nuls_totalll values'.format(cols_total_count, cols_count, cols_percent_null))
## call the function
missing_val(df)<jupyter_output>From 25 columns, 0 has missing/null values; 0.0% column has missing/nuls_totalll values<jupyter_text>**Duplicated Values Handling** We'll try to check if we have duplicated data or not using two approach; first look at the clients `ID`, and second check the unique values for column `ID`.<jupyter_code>## check duplicated id
id_count = df["ID"].value_counts(ascending=False)
dup_id = id_count[id_count>1].count()

## print result
print("There is {} client's id that appear more than one time in the dataset".format(dup_id))<jupyter_output>There is 0 client's id that appear more than one time in the dataset<jupyter_text>After this two approach, we know that we don't have any duplicated values in out dataset.  **Outliers Values Handling** After we checked the missing and duplicated values, we'll try to check the outliers in the dataset by using two approach. First, we'll create a function that use `Tukey's Method` to check the outliers, and second we can visualize every continous column to see spreading values in our dataset.<jupyter_code>## create a function
def outlier_count(col, data=df):
    print(15*'-' + col + 15*'-')
    q75, q25 = np.percentile(data[col], [75, 25])
    iqr = q75 - q25
    min_val = q25 - (iqr*1.5)
    max_val = q75 + (iqr*1.5)
    outlier_count = len(np.where((data[col] > max_val) | (data[col] < min_val))[0])
    outlier_percent = round(outlier_count/len(data[col])*100, 2)
    print('Total outliers value: {}'.format(outlier_count))
    print('Outliers percentage: {}%'.format(outlier_percent))<jupyter_output><empty_output><jupyter_text>With this function, we'll considering for every values that outside 1.5 times IQR is the outliers, as in `<NAME>` method.<jupyter_code>## take every contious column
cont_cols = list(df[['LIMIT_BAL','BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 
                           'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']])

## call the function
for col in cont_cols:
    outlier_count(col)<jupyter_output>---------------LIMIT_BAL---------------
Total outliers value: 111
Outliers percentage: 0.53%
---------------BILL_AMT1---------------
Total outliers value: 1663
Outliers percentage: 7.92%
---------------BILL_AMT2---------------
Total outliers value: 1655
Outliers percentage: 7.88%
---------------BILL_AMT3---------------
Total outliers value: 1696
Outliers percentage: 8.08%
---------------BILL_AMT4---------------
Total outliers value: 1789
Outliers percentage: 8.52%
---------------BILL_AMT5---------------
Total outliers value: 1906
Outliers percentage: 9.08%
---------------BILL_AMT6---------------
Total outliers value: 1879
Outliers percentage: 8.95%
---------------PAY_AMT1---------------
Total outliers value: 1905
Outliers percentage: 9.07%
---------------PAY_AMT2---------------
Total outliers value: 1903
Outliers percentage: 9.06%
---------------PAY_AMT3---------------
Total outliers value: 1823
Outliers percentage: 8.68%
---------------PAY_AMT4---------------
Total outliers value: 205[...]<jupyter_text>From the result above, we can see that `LIMIT_BAL` have the smallest ourliers, only 0.53% or 111 outliers. The biggest outliers is in `PAY_AMT6` column with 9.82% or 2063 ourliers. We can also check the outliers by visualize it with boxplot and histogram.<jupyter_code>## create a function
def outliers_detection(data):
    plt.figure(figsize=(35, 50))
    i = 0
    for col in cont_cols:
        i += 1
        plt.subplot(9, 4, i)
        plt.boxplot(data[col])
        plt.title('{} boxplot'.format(col))
        plt.grid(True)
        i += 1
        plt.subplot(9, 4, i)
        plt.hist(data[col], bins=20)
        plt.title('{} histogram'.format(col))
        plt.grid(True)
    plt.show()
## call the function
outliers_detection(df)<jupyter_output><empty_output><jupyter_text>You can zoom in to the picture above if you want to see clearly the outliers value. As example, in `LIMIT_BAL`, we know that every values that outside 520000 NTD is outliers. In this binary classification project, I don't think we need to do any special treatments to our outliers such as remove or transform them. It's because we'll lose a lot of information that maybe needed for our modeling. Let's consider that outliers are **not always** a problem that must be resolved. Beside that, decision trees can perform well even with outliers.   **Detect Imbalance Data** Let's see the class imbalance on our target variable by visualize it.<jupyter_code>## count default freaquent
yes = df.default_payment_next_month.sum()
no = len(df)-yes

# perecentage
yes_perc = round(yes/len(df)*100, 1)
no_perc = round(no/len(df)*100, 1)
## plot default
plt.figure(figsize = (8,8))
plt.title('Default Credit Card Clients \n(Default = 1, Not Default = 0)')
ax=sns.countplot(x=df['default_payment_next_month'], data=df);
for p in ax.patches:
    ax.annotate('Non-default: {}'.format(no), xy=(1, 15000), xytext=(-0.25, 3000), size=12)
    ax.annotate('Default: {}'.format(yes), xy=(1, 15000), xytext=(0.8, 3000), size=12)
    plt.annotate(str(no_perc)+" %", xy=(-0.3, 15000), xytext=(-0.1, 16400), size=12)
    plt.annotate(str(yes_perc)+" %", xy=(0.7, 15000), xytext=(0.9, 4700), size=12)
locs, labels = plt.xticks()
plt.show()<jupyter_output><empty_output><jupyter_text>From 21000 credit card holder, percentage client that predicted will default in the next month is 22.1%, while 77.9% predicted will not default. With this, we know that our target is **highly imbalance**. But for now I think we don't need to do anything in our target (like under-sampling and over-sampling) because I want to do some experiment with both imbalanced and balanced target in our modeling section.  **D.Feature Engineering** After we feel enough in our data cleaning process, we'll do features engineering to our dataset.<jupyter_code>df.describe().T<jupyter_output><empty_output><jupyter_text>Although our dataset is pretty clean and already encoded in the nominal-type columns, there's some column that have anomaly values. For example, in `MARRIAGE` column there's value 0, where it doesn't represent anything because in our dataset description there's only three categories; 1 represent married, 2 represent single, and 3 represent others.In this features engineering, we'll try to change and add features in our dataset so it'll useable for our next process.  **Rename and Drop Columns** We'll rename our target column, `PAY_0` and `SEX` column so it'll be more easily to we understand.<jupyter_code>## rename column
df.rename(columns={"default_payment_next_month":"DEFAULT",
                  "SEX":"GENDER",
                   "PAY_0":"PAY_1"}, inplace=True)<jupyter_output><empty_output><jupyter_text>We don't need the `ID` column again, so we can remove it using drop function.<jupyter_code>df.drop(columns="ID", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>**Change on Marriage Column**<jupyter_code>## check unique value
df['MARRIAGE'].unique()<jupyter_output><empty_output><jupyter_text>We can assign value 0 to value 3 because it doesn't represent anything in our column.<jupyter_code>## assign value 0 to value 3
df["MARRIAGE"] = np.where(df["MARRIAGE"] == 0, 3, df["MARRIAGE"])

## check the unique values again
df["MARRIAGE"].unique()<jupyter_output><empty_output><jupyter_text>**Change on Education Column**<jupyter_code>## check unique values on education column
df["EDUCATION"].unique()<jupyter_output><empty_output><jupyter_text>We will assign value 0, 5, and 6 to value 4<jupyter_code>## assign values
df['EDUCATION'] = np.where(df['EDUCATION'] == 0, 4, df['EDUCATION'])
df['EDUCATION'] = np.where(df['EDUCATION'] == 5, 4, df['EDUCATION'])
df['EDUCATION'] = np.where(df['EDUCATION'] == 6, 4, df['EDUCATION'])

## check unique values again
df['EDUCATION'].unique()<jupyter_output><empty_output><jupyter_text>**One-Hot Encoding** Before we do `One-Hot Encoding` to our dataset, we'll copy our dataset that we already processed so far to new dataset.<jupyter_code>df_ohe = df.copy()
## check first five rows of the new dataset
df_ohe.head()<jupyter_output><empty_output><jupyter_text>Next, we will separate features and target on our new dataset.<jupyter_code>## set target
target_col = df_ohe["DEFAULT"]

## set features
features_col = df_ohe[['LIMIT_BAL', 'GENDER', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_1', 'PAY_2','PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2','BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5',
                       'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6',]]<jupyter_output><empty_output><jupyter_text>We will separate our features into categorical and numerical. Only categorical columns that will we process in `One-Hot Encoding`.<jupyter_code>## set categorical columns
categorical_cols = ['GENDER','EDUCATION','MARRIAGE','PAY_1','PAY_2','PAY_3','PAY_4','PAY_5','PAY_6']

## set numericak columns
numerical_cols = ['LIMIT_BAL','BILL_AMT1', 'BILL_AMT2','BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5','BILL_AMT6','PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
## one-hot encoding

for col in categorical_cols:
    dum_df = pd.get_dummies(df_ohe[col], prefix=col)
    df_ohe = df_ohe.join(dum_df)<jupyter_output><empty_output><jupyter_text>We can check the result by show the first five rows<jupyter_code>df_ohe.head()<jupyter_output><empty_output><jupyter_text>We'll move our target column to the most left of the dataset so it'll more easier to read<jupyter_code>## set target
target_var = "DEFAULT"

## create a list
col_names = list(df_ohe.columns)

## move target to the first index of the list
col_names.insert(0, col_names.pop(col_names.index(target_var)))

## apply it to the dataframe
df_ohe = df_ohe.loc[:, col_names]<jupyter_output><empty_output><jupyter_text>Now we can show the our preprocessed dataframes<jupyter_code>## show the first five rows of the df_ohe
df_ohe.head()
## show the first five rows of the df
df.head()
print("df's shape: ", df.shape)
print("df_ohe's shape: ", df_ohe.shape)<jupyter_output>df's shape:  (21000, 24)
df_ohe's shape:  (21000, 96)<jupyter_text>Both is our preprocessed dataset; with and without one-hot encoding. The reason why I made two dataset because I want to see the performance of our model later if we use dataset without OHE and dataset with OHE.  E. Export to CSV We'll export our two `preprocessed` dataset (df and df_ohe) to csv format so we can reuse it on our next notebook.<jupyter_code>## reconnect to drive
#connect notebook ke google drive
from google.colab import drive
drive.mount("/content/gdrive")
## export df
df.to_csv("/content/gdrive/MyDrive/Colab Notebooks/Defaulters/preprocessed_data.csv", index=False)

## export df_ohe
df_ohe.to_csv("/content/gdrive/MyDrive/Colab Notebooks/Defaulters/preprocessed_data_ohe.csv", index=False)<jupyter_output><empty_output><jupyter_text>We can check the csv files that we just export by load it into this notebook to make sure that the format is correct<jupyter_code>## load csv files

df1 = pd.read_csv("/content/gdrive/MyDrive/Colab Notebooks/Defaulters/preprocessed_data.csv")
df2 = pd.read_csv("/content/gdrive/MyDrive/Colab Notebooks/Defaulters/preprocessed_data_ohe.csv")

print("df1's shape: ", df1.shape)
print("df2's shape: ", df2.shape)
df2.head()
df1.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1291802136553646, 'fraction_numerical': 0.030306549001393405, 'mean_word_length': 4.7899159663865545, 'pattern_counts': {'":': 3, '<': 98, '<?xml version=': 0, '>': 100, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '72062', 'license': 'MIT', 'n_tokens_mistral': 5977, 'n_tokens_neox': 5412, 'n_words': 1911, 'path': 'notebook files/Preprocessing.ipynb', 'repo_name': 'alvinnoza-ph/Credit-Card-Default-Prediction', 'size': '17228'}