<jupyter_start><jupyter_text><jupyter_code># import necessary packages
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import gc

from sklearn.metrics import auc, roc_curve, log_loss
from sklearn.preprocessing import LabelEncoder
from sklearn.utils import shuffle
from imblearn.over_sampling import RandomOverSampler

import h2o
from h2o.frame import H2OFrame
from h2o.estimators.glm import H2OGeneralizedLinearEstimator
from h2o.estimators.random_forest import H2ORandomForestEstimator
from h2o.grid.grid_search import H2OGridSearch

from keras import models, layers


%matplotlib inline<jupyter_output>//anaconda/lib/python3.5/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
//anaconda/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
  return f(*args, **kwds)<jupyter_text>Lending Club Loan Data Analysis and Modeling Classification is one of two most common data science problems (another one is regression). For the supervised classification problem, imbalanced data is pretty common yet very challanging. For example, credit card fraud detection, disease classification, network intrusion and so on, are classification problem with imbalanced data. In this project, working with the Lending Club loan data, we hope to correctlly predict whether or not on loan will be default using the history data.  Contents1. Problem Statment2. Data Exploration3. Data Cleaning and Initial Feature Engineering    - Feature transformation    - Missing values    - Feature transformation    - Feature normalization    - And so on ......4. Visualization5. Further Feature Engineering6. Machine Learning    - Logistic Regression    - Random Forest    - Deep Learning7. Conclusions  1. Problem Statment For companies like Lending Club, correctlly predicting whether or not one loan will be default is very important. In this project, using the historical data, more specifically, the Lending Club loan data from 2007 to 2015, we hope to build a machine learning model such that we can predict the chance of default for the future loans. As I will show later, this dataset is highly imbalanced and includes a lot of features, which makes this problem more challanging.   2. Data Exploration There are several ways to download the dataset, for example, you can go to Lending Club's [website](https://www.lendingclub.com/info/download-data.action), or you can go to [Kaggle](https://www.kaggle.com/wendykan/lending-club-loan-data).I will use the loan data from 2007 to 2015 as the training set (+ validation set), and use the data from 2016 as the test set.<jupyter_code># Load the training and test data set
train = pd.read_csv('./data/2007-2015-loan.csv', low_memory=False, encoding='ISO-8859-1')
test = pd.read_csv('./data/2016-loan.csv', low_memory=False, encoding='ISO-8859-1')

# There are 74 features in total
print('Train:\t', train.shape)
print('Test:\t', test.shape)
train.head()
test.head()
# Create a concise table for the data
train_dtype = train.dtypes.values
train_missing = train.isnull().sum().values
train_missing_ratio = train_missing / len(train)

test_dtype = test.dtypes.values
test_missing = test.isnull().sum().values
test_missing_ratio = test_missing / len(test)

# Calculate unique values
train_unique = []
test_unique = []
for name in train.columns:
    train_unique.append(train[name].unique().shape[0])
    test_unique.append(test[name].unique().shape[0])

# Useful information
tmp_map = {'Columns': train.columns, 'Train dtype': train_dtype, 'Train missing': train_missing, 
           'Train missing ratio': train_missing_ratio, 'Train unique': train_unique, 'Test dtype': test_dtype,
           'Test missing': test_missing, 'Test missing ratio': test_missing_ratio, 'Test unique': test_unique}
columns = ['Columns', 'Train dtype', 'Train missing', 'Train missing ratio', 'Train unique',
           'Test dtype', 'Test missing', 'Test missing ratio', 'Test unique']
df = pd.DataFrame(tmp_map, columns=columns)

pd.options.display.max_rows = 75
df<jupyter_output><empty_output><jupyter_text>We should notice some differences between the training and test set, and look into details. Some major difference are:1. For test set, id, member_id, and url are totally missing, which is different from training set2. For training set, open_acc_6m, open_il_6m, open_il_12m, open_il_24m, mths_since_rcnt_il, total_bal_il, il_util, open_rv_12m, open_rv_24m, max_bal_bc, all_util, inq_fi, total_cu_tl, and inq_last_12m	are almost missing in training set, which is different from test set3. desc, mths_since_last_delinq, mths_since_last_record, mths_since_last_major_derog, annual_inc_joint, dti_joint, and verification_status_joint have large amount of missing values4. There are multiple loan status, but we only concern whether or not the load is default  3. Data Cleaning and Initial Feature Engineering  I. Transform feature `int_rate` and `revol_util` in test set<jupyter_code># Check the difference values for int_rate, revol_util
print('Item\t', 'int_rate', '\t', 'revol_util')
print('-' * 40)
print('Train:\t', train['int_rate'][0], '\t\t', train['revol_util'][0])
print('Test:\t', test['int_rate'][0], '\t', test['revol_util'][0])
# Using lambda function
test['int_rate'] = test['int_rate'].apply(lambda x: float(x[:-1]))

index = test[~test['revol_util'].isnull()].index
test.loc[index, 'revol_util'] = test.loc[index, 'revol_util'].apply(lambda x: float(x[:-1]))
test['revol_util'] = test['revol_util'].astype(float)<jupyter_output><empty_output><jupyter_text>II. Transform target values `loan_status`<jupyter_code># Check the target values
print('Train:\n', list(train['loan_status'].unique()))
print('\nTest:\n', list(test['loan_status'].unique()))
# Let's only keep the data that is not in process currentlly, for more details, refer to 
# https://help.lendingclub.com/hc/en-us/articles/215488038-What-do-the-different-Note-statuses-mean-
drop_status = ['Issued']
train = train[~train['loan_status'].isin(drop_status)]
test = test[~test['loan_status'].isin(drop_status)]
# Treat this as a binary classification problem
maps = {'Current': 0, 'Fully Paid': 0, 'Charged Off': 1, 'Default': 1, 
        'Does not meet the credit policy. Status:Fully Paid': 0, 
        'Does not meet the credit policy. Status:Charged Off': 1, 
        'In Grace Period': 0, 'Late (16-30 days)': 1, 'Late (31-120 days)': 1}
train['loan_status'] = train['loan_status'].apply(lambda x: maps[x])
test['loan_status'] = test['loan_status'].apply(lambda x: maps[x])
# Check the imbalance
train_default_ratio = np.round(len(train[train['loan_status'] == 1]) / len(train), 4)
train_nondefault_ratio = np.round(len(train[train['loan_status'] == 0]) / len(train), 4)
test_default_ratio = np.round(len(test[test['loan_status'] == 1]) / len(test), 4)
test_nondefault_ratio = np.round(len(test[test['loan_status'] == 0]) / len(test), 4)
print('Item\t', 'Default', '\t', 'Not Default')
print('-' * 40)
print('Train:\t', train_default_ratio, '\t', train_nondefault_ratio)
print('Test:\t', test_default_ratio, '\t', test_nondefault_ratio)<jupyter_output>Item	 Default 	 Not Default
----------------------------------------
Train:	 0.0696 	 0.9304
Test:	 0.0363 	 0.9637<jupyter_text>It's clear that our dataset is highly imbalanced.<jupyter_code># Create a concise table for the data
train_dtype = train.dtypes.values
train_missing = train.isnull().sum().values
train_missing_ratio = train_missing / len(train)

test_dtype = test.dtypes.values
test_missing = test.isnull().sum().values
test_missing_ratio = test_missing / len(test)

# Calculate unique values
train_unique = []
test_unique = []
for name in train.columns:
    train_unique.append(train[name].unique().shape[0])
    test_unique.append(test[name].unique().shape[0])

# Useful information
df = pd.DataFrame({'Columns': train.columns, 'Train dtype': train_dtype, 'Train missing': train_missing, 
                   'Train missing ratio': train_missing_ratio, 'Train unique': train_unique, 
                   'Test dtype': test_dtype, 'Test missing': test_missing, 
                   'Test missing ratio': test_missing_ratio, 'Test unique': test_unique}, 
                  columns=['Columns', 'Train dtype', 'Train missing', 'Train missing ratio', 'Train unique',
                           'Test dtype', 'Test missing', 'Test missing ratio', 'Test unique'])

df<jupyter_output><empty_output><jupyter_text>III. Drop useless features<jupyter_code># Drop the features that has too many missing values
useless = ['id', 'member_id', 'url', 'open_acc_6m', 'open_il_6m', 'open_il_12m', 'open_il_24m', 
           'mths_since_rcnt_il', 'total_bal_il', 'il_util', 'open_rv_12m', 'open_rv_24m',
           'max_bal_bc', 'all_util', 'inq_fi', 'total_cu_tl', 'inq_last_12m', 'desc', 
           'mths_since_last_delinq', 'mths_since_last_record', 'mths_since_last_major_derog', 
           'annual_inc_joint', 'dti_joint', 'verification_status_joint']
train = train.drop(labels=useless, axis=1)
test = test.drop(labels=useless, axis=1)
# Drop meaningless features based on understanding
meaningless = ['emp_title', 'issue_d', 'last_pymnt_d', 'next_pymnt_d', 'zip_code', 'title', 
               'grade', 'earliest_cr_line', 'last_credit_pull_d', 'policy_code']
train = train.drop(labels=meaningless, axis=1)
test = test.drop(labels=meaningless, axis=1)
_ = gc.collect()
print(train.shape, test.shape)<jupyter_output>(878919, 40) (434407, 40)<jupyter_text>Now, we have successfully reduce the features from 74 to 40. Next, let's focus on more detailed feature engineering First, let's look at the data again. From the below table, we can see that:* Most features are numerical, but there are severl categorical features.* There are still some missing values among numerical and categorical features.<jupyter_code># Create a concise table for the data
train_dtype = train.dtypes.values
train_missing = train.isnull().sum().values
train_missing_ratio = train_missing / len(train)

test_dtype = test.dtypes.values
test_missing = test.isnull().sum().values
test_missing_ratio = test_missing / len(test)

# Calculate unique values
train_unique = []
test_unique = []
for name in train.columns:
    train_unique.append(train[name].unique().shape[0])
    test_unique.append(test[name].unique().shape[0])

# Useful information
df = pd.DataFrame({'Columns': train.columns, 'Train dtype': train_dtype, 'Train missing': train_missing, 
                   'Train missing ratio': train_missing_ratio, 'Train unique': train_unique, 
                   'Test dtype': test_dtype, 'Test missing': test_missing, 
                   'Test missing ratio': test_missing_ratio, 'Test unique': test_unique}, 
                  columns=['Columns', 'Train dtype', 'Train missing', 'Train missing ratio', 'Train unique',
                           'Test dtype', 'Test missing', 'Test missing ratio', 'Test unique'])

df<jupyter_output><empty_output><jupyter_text>IV. Feature transformation Transform numerical values into categorical values<jupyter_code>transform_featurs = ['total_rec_late_fee', 'recoveries', 'collection_recovery_fee', 'tot_coll_amt', 
                     'collections_12_mths_ex_med', 'acc_now_delinq', 'out_prncp', 'out_prncp_inv']
for name in transform_featurs:
    train[name] = (train[name] > 0).astype(str)
    test[name] = (test[name] > 0).astype(str)<jupyter_output><empty_output><jupyter_text>Transform categorical values into numerical values (discrete)<jupyter_code># Transform sub_grade
sub_grade_encoder = LabelEncoder()
sub_grade_encoder.fit(train['sub_grade'])
train['sub_grade'] = sub_grade_encoder.transform(train['sub_grade'])
test['sub_grade'] = sub_grade_encoder.transform(test['sub_grade'])
# Transform emp_length (first fill NA value with mode)
mode = train['emp_length'].mode().values[0]
train['emp_length'] = train['emp_length'].fillna(value=mode)
test['emp_length'] = test['emp_length'].fillna(value=mode)

# Manually do encoding due to sort problems
emp_map = {'< 1 year': 0, '1 year': 1, '3 years': 2, '4 years': 3, '5 years': 4, '6 years': 5, 
           '2 years': 6, '7 years': 7, '8 years': 8, '9 years': 9, '10+ years': 10}
train['emp_length'] = train['emp_length'].apply(lambda x: emp_map[x])
test['emp_length'] = test['emp_length'].apply(lambda x: emp_map[x])<jupyter_output><empty_output><jupyter_text>V. Fill missing values* For numerical features, use median* For categorical features, use mode (here, we don't have missing categorical values)<jupyter_code># Seperate numerical and categorical features (16 categorical features, 23 numerical features)
numerical_feature = train.select_dtypes(exclude=['object']).columns.drop('loan_status')
categorical_feature = train.select_dtypes(include=['object']).columns

print('Numerical:\t', len(numerical_feature))
print('Categorical:\t', len(categorical_feature))
# Fill numerical features with median
medians = train[numerical_feature].median(axis=0, skipna=True)
train[numerical_feature] = train[numerical_feature].fillna(value=medians)
test[numerical_feature] = test[numerical_feature].fillna(value=medians)<jupyter_output><empty_output><jupyter_text>4. Visualization   I. Visualize categorical features<jupyter_code># For better visualization purpose, we set the y to be log scale
fig, ax = plt.subplots(nrows=6, ncols=3, figsize=(18, 30))
plt.tight_layout(h_pad=3)
for i in range(len(categorical_feature)):
    name = categorical_feature[i]
    sns.countplot(x=name, hue='loan_status', data=train, ax=ax[i//3][i%3])
    ax[i//3][i%3].set_yscale('log')
plt.show()
_ = gc.collect()<jupyter_output><empty_output><jupyter_text>II. Visualize numerical features<jupyter_code># Histogram of numerical values
fig, ax = plt.subplots(nrows=8, ncols=3, figsize=(18, 42))
plt.tight_layout(h_pad=3)
hist_kws={'histtype': 'bar', 'edgecolor':'black', 'alpha': 0.2}
for i in range(len(numerical_feature)):
    name = numerical_feature[i]
    sns.distplot(train[train['loan_status'] == 0][name], label='Not Default',
                 hist_kws=hist_kws, ax=ax[i//3][i%3])
    sns.distplot(train[train['loan_status'] == 1][name], label='Default',
                 hist_kws=hist_kws, ax=ax[i//3][i%3])
    ax[i//3][i%3].legend()
plt.show()
_ = gc.collect()
# Heatmap of the correlation
corr = train[numerical_feature].corr()
fig, ax = plt.subplots(figsize=(12, 10))
sns.heatmap(corr, ax=ax)
plt.axis('image')
plt.show()<jupyter_output><empty_output><jupyter_text>5. Further Feature Engineering From the above heatmap and the categorical variable countplot, we can see that some feature has strong correlation* loan_amnt, funded_amnt, funded_amnt_inv, installment* int_rate, sub_grade* total_pymnt, total_pymnt_inv, total_rec_prncp* out_prncp, out_prncp_inv* recoveries, collection_recovery_feeWe can drop some of them to reduce redundancy<jupyter_code># Drop redundant features
useless = ['funded_amnt', 'funded_amnt_inv', 'installment', 'sub_grade', 'total_pymnt_inv', 
           'total_rec_prncp', 'out_prncp_inv', 'collection_recovery_fee']
train = train.drop(useless, axis=1)
test = test.drop(useless, axis=1)
# Seperate numerical and categorical features (13 categorical features, 25 numerical features)
numerical_feature = train.select_dtypes(exclude=['object']).columns.drop('loan_status')
categorical_feature = train.select_dtypes(include=['object']).columns

print('Numerical:\t', len(numerical_feature))
print('Categorical:\t', len(categorical_feature))<jupyter_output>Numerical:	 17
Categorical:	 14<jupyter_text>Now, we only 14 categorical features, 17 numerical features. Let's check the correlation again.<jupyter_code>print('Numerical features:', '\n', list(numerical_feature), '\n')
print('Categorical Features:', '\n', list(categorical_feature))
# Check the heatmap again
corr = train[numerical_feature].corr()
fig, ax = plt.subplots(figsize=(12, 10))
sns.heatmap(corr, ax=ax)
plt.axis('image')
plt.show()<jupyter_output><empty_output><jupyter_text>6. Machine Learning After the above procedures, we are ready to build the predictive models. In this part, I explored three different models: Logistic regression, Random Forest, and Deep Learning.I used to use scikit-learn a lot. But there is one problem with scikit-learn: you need to do one-hot encoding manually, which can sometimes dramatically increase the feature space. In this part, for logistic regression and random forest, I use H2O package, which has a better support with categorical features. For deep learning model, I use Keras with TensorFlow backend.<jupyter_code># Define x and y variables
train = shuffle(train, random_state=42)
x = list(train.columns.drop('loan_status'))
y = 'loan_status'

train_y = train['loan_status'].values
test_y = test['loan_status'].values
# Initialize H2O cluster
h2o.init()
h2o.remove_all()

# Transform to H2O Frame, and make sure the target variable is categorical
h2o_train = H2OFrame(train)
h2o_test = H2OFrame(test)

h2o_train['loan_status'] = h2o_train['loan_status'].asfactor()
h2o_test['loan_status'] = h2o_test['loan_status'].asfactor()<jupyter_output>Checking whether there is an H2O instance running at http://localhost:54321..... not found.
Attempting to start a local H2O server...
  Java Version: openjdk version "1.8.0_121"; OpenJDK Runtime Environment (Zulu 8.20.0.5-macosx) (build 1.8.0_121-b15); OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-macosx) (build 25.121-b15, mixed mode)
  Starting server from /anaconda/lib/python3.5/site-packages/h2o/backend/bin/h2o.jar
  Ice root: /<KEY>
  JVM stdout: /var/folders/gx/b15jqbt1567grsfr7l_znn2h0000gn/T/tmpqdjoac_3/h2o_jifu_started_from_python.out
  JVM stderr: /var/folders/gx/b15jqbt1567grsfr7l_znn2h0000gn/T/tmpqdjoac_3/h2o_jifu_started_from_python.err
  Server is running at http://127.0.0.1:54321
Connecting to H2O server at http://127.0.0.1:54321... successful.<jupyter_text>I. Logistic Regression<jupyter_code># Train logistic regression model with Lasso using grid search
hyper_parameters = {'alpha': [0, 0.2, 0.4, 0.6, 0.8, 1.0], 
                    'lambda': [1e-4, 1e-5, 1e-6, 1e-7, 1e-8]}

# Create GLM model
glm = H2OGeneralizedLinearEstimator(family='binomial', balance_classes=True, 
                                    early_stopping=True, custom_metric_func='auc',
                                    keep_cross_validation_predictions=True, nfolds=5)

# Grid search
glm_grid = H2OGridSearch(glm, hyper_parameters, grid_id='GLM')
glm_grid.train(x=x, y=y, training_frame=h2o_train)
# Get the grid search result, sorted by AUC
glm_models = glm_grid.get_grid(sort_by='auc', decreasing=True)

# Choose the best model
best_glm = glm_grid.models[0]

glm_models
# Make predictions
glm_train_pred = best_glm.predict(h2o_train).as_data_frame()['p1'].values
glm_test_pred = best_glm.predict(h2o_test).as_data_frame()['p1'].values
# Build the ROC curve
train_fpr, train_tpr, _ = roc_curve(train_y, glm_train_pred)
test_fpr, test_tpr, _ = roc_curve(test_y, glm_test_pred)
glm_train_auc = np.round(auc(train_fpr, train_tpr), 3)
glm_test_auc = np.round(auc(test_fpr, test_tpr), 3)

fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(train_fpr, train_tpr, label='Train AUC: ' + str(glm_train_auc))
ax.plot(test_fpr, test_tpr, label='Test  AUC: ' + str(glm_test_auc))
ax.set_xlabel('False Positive Rate', fontsize=12)
ax.set_ylabel('True Positive Rate', fontsize=12)
ax.legend(fontsize=12)
plt.show()
# Log-loss
print('Train:\t', log_loss(train_y, glm_train_pred))
print('Test:\t', log_loss(test_y, glm_test_pred))<jupyter_output>Train:	 0.10566966640063173
Test:	 0.10066479508447695<jupyter_text>II. Random Forest<jupyter_code># Train random forest model using grid search
hyper_parameters = {'max_depth': [10, 15, 20], 
                    'min_rows': [3, 5]}

# Build random forest model
rf = H2ORandomForestEstimator(balance_classes=True, ntrees=200, stopping_rounds=5, nfolds=5, 
                              stopping_metric='auc', keep_cross_validation_predictions=True)

# Grid search
rf_grid = H2OGridSearch(rf, hyper_parameters, grid_id='DRF')
rf_grid.train(x=x, y=y, training_frame=h2o_train)
# Get the grid search result, sorted by AUC
rf_models = rf_grid.get_grid(sort_by='auc', decreasing=True)

# Choose the best model
best_rf = rf_grid.models[0]

rf_models
# Make prediction
rf_train_pred = best_rf.predict(h2o_train).as_data_frame()['p1'].values
rf_test_pred = best_rf.predict(h2o_test).as_data_frame()['p1'].values
# Build the ROC curve
train_fpr, train_tpr, _ = roc_curve(train_y, rf_train_pred)
test_fpr, test_tpr, _ = roc_curve(test_y, rf_test_pred)
rf_train_auc = np.round(auc(train_fpr, train_tpr), 3)
rf_test_auc = np.round(auc(test_fpr, test_tpr), 3)

fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(train_fpr, train_tpr, label='Train AUC: ' + str(rf_train_auc))
ax.plot(test_fpr, test_tpr, label='Test  AUC: ' + str(rf_test_auc))
ax.set_xlabel('False Positive Rate', fontsize=12)
ax.set_ylabel('True Positive Rate', fontsize=12)
ax.legend(fontsize=12)
plt.show()
# Log-loss
print('Train:\t', log_loss(train_y, rf_train_pred))
print('Test:\t', log_loss(test_y, rf_test_pred))
# Feature importance
best_rf.varimp_plot(num_of_features=20)
plt.show()
# Shutdown h2o instance
h2o.cluster().shutdown()<jupyter_output>H2O session _sid_b3ad closed.<jupyter_text>III. Deep Learning In this part, let's manually build a fully-connected neural network (NN) model to finish the classification task.<jupyter_code># Feature Normalization
mean = train[numerical_feature].mean()
std = train[numerical_feature].std()
train[numerical_feature] = (train[numerical_feature] - mean) / std
test[numerical_feature] = (test[numerical_feature] - mean) / std

# Dummy encoding
tmp_data = pd.concat([train[categorical_feature], test[categorical_feature]], axis=0)
dummy_data = pd.get_dummies(tmp_data)
train_dummy = dummy_data[:len(train)]
test_dummy = dummy_data[len(train):]

# Get train, test data and target
train_data = pd.concat([train[numerical_feature], train_dummy], axis=1).values
train_target = train['loan_status'].values
test_data = pd.concat([test[numerical_feature], test_dummy], axis=1).values
test_target = test['loan_status'].values
# Define NN model
def build_model(data):
    # build a NN with input-layer1-layer2-output structure
    model = models.Sequential()
    model.add(layers.Dense(128, activation='relu', input_shape=(data.shape[1],)))
    model.add(layers.Dense(64, activation='relu'))
    model.add(layers.Dense(1, activation='sigmoid'))
    model.compile(optimizer='Adam', loss='binary_crossentropy')
    return model


def kfold(data, target, k, num_epochs=100, batch_size=128):
    """ function to perform k-fold cross validation """
    n = len(data) // k
    
    # Keep recording the training and validation loss
    train_loss = np.zeros((k, num_epochs))
    val_loss = np.zeros((k, num_epochs))

    for i in range(k):
        print('Processing cross-validation round #', i + 1)
        
        # Get the train and validation set
        val_x = data[i * n: (i + 1) * n]
        val_y = target[i * n: (i + 1) * n]
        
        train_x = np.concatenate([data[: i * n], data[(i + 1) * n:]], axis=0)
        train_y = np.concatenate([target[: i * n], target[(i + 1) * n:]], axis=0)
        
        # Over sampling training set
        ros = RandomOverSampler(ratio='minority', random_state=42)
        train_x_resampled, train_y_resampled = ros.fit_sample(train_x, train_y)
        
        # Build and train NN model
        model = build_model(data)
        history = model.fit(train_x_resampled, train_y_resampled, epochs=num_epochs, verbose=0, 
                            validation_data=(val_x, val_y), batch_size=batch_size)
        
        # Retrieve the training and validation history
        
        train_loss[i] = history.history['loss']
        val_loss[i] = history.history['val_loss']
        
    return train_loss, val_loss
# Begin 5-fold cross validataion
num_epochs = 100
train_loss, val_loss = kfold(data=train_data, target=train_target, 
                             k=5, num_epochs=100, batch_size=128)
# Visualization
fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(range(1, num_epochs + 1), np.mean(train_loss, axis=0), label='Train Loss')
ax.plot(range(1, num_epochs + 1), np.mean(val_loss, axis=0), label='Validation Loss')
ax.set_xlabel('Epochs', fontsize=12)
ax.set_ylabel('Loss', fontsize=12)
ax.legend(fontsize=12)
plt.show()
# Over sampling training set
ros = RandomOverSampler(ratio='minority', random_state=42)
train_x_resampled, train_y_resampled = ros.fit_sample(train_data, train_target)
sum(train_target == 1) / len(train_target)
sum(train_y_resampled == 1) / len(train_y_resampled)
# Re-fit the model with the best epochs
nn_model = build_model(train_data)
model.fit(train_x_resampled, train_y_resampled, epochs=?, verbose=1 batch_size=128)
# Make prediction
nn_train_pred = nn_model.predict(train_data)
nn_test_pred = nn_model.predict(train_data)
# Build the ROC curve
train_fpr, train_tpr, _ = roc_curve(train_y, nn_train_pred)
test_fpr, test_tpr, _ = roc_curve(test_y, nn_test_pred)
nn_train_auc = np.round(auc(train_fpr, train_tpr), 3)
nn_test_auc = np.round(auc(test_fpr, test_tpr), 3)

fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(train_fpr, train_tpr, label='Train AUC: ' + str(nn_train_auc))
ax.plot(test_fpr, test_tpr, label='Test  AUC: ' + str(nn_test_auc))
ax.set_xlabel('False Positive Rate', fontsize=12)
ax.set_ylabel('True Positive Rate', fontsize=12)
ax.legend(fontsize=12)
plt.show()
# Log-loss
print('Train:\t', log_loss(train_y, nn_train_pred))
print('Test:\t', log_loss(test_y, nn_test_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12705519104592822, 'fraction_numerical': 0.02257815515245079, 'mean_word_length': 5.295189504373178, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 69, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '351439', 'license': 'MIT', 'n_tokens_mistral': 9208, 'n_tokens_neox': 8633, 'n_words': 2187, 'path': 'deprecated/Lending Club Modeling.ipynb', 'repo_name': 'JifuZhao/Lending-Club-Loan-Analysis', 'size': '25964'}