<jupyter_start><jupyter_text>Credit Data ExampleThis example demonstrate how to use classical tree base model to do classification prediction of an imbalance data like predicting a default for loan/credit.   DataThe data is a public data from UCL : http://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clientsHere we will have a brief inspection of the data, then test it with 2 methods: XGBoost and IsolationForest. EDA and feature engineering are be added.<jupyter_code>import pandas as pd
pd.set_option('display.max_columns', 1000)
df = pd.read_excel('default of credit card clients.xls', header=1)
df.head()
df.shape
na_cols = []
for col in df.columns:
    if df[col].count() != df.shape[0]:
        na_cols.append(col)
df[na_cols].count()<jupyter_output><empty_output><jupyter_text>There is no NA's<jupyter_code>df.dtypes
df['default payment next month'].value_counts()<jupyter_output><empty_output><jupyter_text>It's a slightly imbalance data with 'default payment next month' being the label.  Prepare for Training<jupyter_code>label = df['default payment next month'].values
label
data = df.drop(['default payment next month'],axis=1, inplace=False)<jupyter_output><empty_output><jupyter_text>Split the data statisticallySicne the data is inbalance, the train test split need to follow the statistic of the origianl data.<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(data.values, label, test_size=0.2, random_state=42, stratify=label)

X_training, X_var, y_training, y_var = train_test_split(X_train, y_train,\
                                                        test_size=0.2, random_state=42, stratify=y_train)
X_training.shape<jupyter_output><empty_output><jupyter_text>Using XGBoostXGBoost provide a flixiable and fast model training.<jupyter_code>import os

mingw_path = 'C:\\Program Files\\mingw-w64\\x86_64-7.3.0-posix-seh-rt_v5-rev0\\mingw64\\bin'

os.environ['PATH'] = mingw_path + ';' + os.environ['PATH']
from xgboost import XGBClassifier

model = XGBClassifier(objective='binary:logistic',n_estimators=200)
model.fit(X_training, y_training, eval_set=[(X_training, y_training), (X_var, y_var)], eval_metric='auc', verbose=10)
print(model)<jupyter_output>[0]	validation_0-auc:0.733729	validation_1-auc:0.712399
[10]	validation_0-auc:0.774028	validation_1-auc:0.764049
[20]	validation_0-auc:0.780168	validation_1-auc:0.768258
[30]	validation_0-auc:0.785753	validation_1-auc:0.771366
[40]	validation_0-auc:0.792319	validation_1-auc:0.774259
[50]	validation_0-auc:0.797579	validation_1-auc:0.774792
[60]	validation_0-auc:0.80159	validation_1-auc:0.775595
[70]	validation_0-auc:0.805092	validation_1-auc:0.775335
[80]	validation_0-auc:0.807562	validation_1-auc:0.775902
[90]	validation_0-auc:0.809775	validation_1-auc:0.775219
[100]	validation_0-auc:0.812344	validation_1-auc:0.775658
[110]	validation_0-auc:0.814872	validation_1-auc:0.775924
[120]	validation_0-auc:0.81696	validation_1-auc:0.776366
[130]	validation_0-auc:0.819119	validation_1-auc:0.776506
[140]	validation_0-auc:0.821087	validation_1-auc:0.776762
[150]	validation_0-auc:0.82246	validation_1-auc:0.776422
[160]	validation_0-auc:0.824194	validation_1-auc:0.776326
[170]	validation_0-auc:0.825[...]<jupyter_text>We can do some parameter tuning following a detailed guide online: https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/* we only turn the most important parameter<jupyter_code>from sklearn.model_selection import StratifiedKFold
from sklearn.grid_search import GridSearchCV

param_test1 = {
 'max_depth':[3,5,7,9],
 'min_child_weight':[1,3,5]
}
gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=200, max_depth=5,
 min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), 
 param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch1.fit(X_train, y_train)
gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_
param_test2 = {
 'max_depth':[3,4,5],
 'min_child_weight':[2,3,4]
}
gsearch2 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=200, max_depth=5,
 min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), 
 param_grid = param_test2, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch2.fit(X_train, y_train)
gsearch2.grid_scores_, gsearch2.best_params_, gsearch2.best_score_<jupyter_output><empty_output><jupyter_text>seems like the best max_depth is 4 and the min_child_weight is 2. Right now the validation score is around 0.78. Since parameter tuning will not significantly improve the result, ensembeling could be considered to further improve the model if needed.<jupyter_code>model = XGBClassifier(objective='binary:logistic',max_depth=4, \
                      min_child_weight=2,n_estimators=60) # early stopping at 60
model.fit(X_training, y_training, eval_set=[(X_training, y_training), (X_var, y_var)], eval_metric='auc', verbose=10)
print(model)<jupyter_output>[0]	validation_0-auc:0.748791	validation_1-auc:0.730067
[10]	validation_0-auc:0.782635	validation_1-auc:0.76739
[20]	validation_0-auc:0.793164	validation_1-auc:0.773296
[30]	validation_0-auc:0.798885	validation_1-auc:0.775004
[40]	validation_0-auc:0.804768	validation_1-auc:0.776849
[50]	validation_0-auc:0.810866	validation_1-auc:0.77803
[59]	validation_0-auc:0.81517	validation_1-auc:0.777942
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,
       max_depth=4, min_child_weight=2, missing=None, n_estimators=60,
       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,
       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
       silent=True, subsample=1)<jupyter_text>For threshold = 0.85 gives the best F1 Score  Using IsolationForestAnother method to be considered is anomaly detection for picking up the sparse claimed case. We can compare it with the model with XGBoost.<jupyter_code>from sklearn.ensemble import IsolationForest
from sklearn import metrics

clf = IsolationForest()
clf.fit(X_train)
y_pred = clf.predict(X_train)
y_pred = (-y_pred+1)/2

print("Accuracy : %.4g" % metrics.accuracy_score(y_train, y_pred))
print("AUC Score (Train): %f" % metrics.roc_auc_score(y_train, y_pred))
print("F1 Score (Train): %f" % metrics.f1_score(y_train, y_pred))<jupyter_output>Accuracy : 0.7361
AUC Score (Train): 0.518998
F1 Score (Train): 0.178493<jupyter_text>Final evaliuation using the test set **Bsaseline: Constant prediction**<jupyter_code>import numpy as np
baseline = np.zeros(len(y_test))

print("Accuracy : %.4g" % metrics.accuracy_score(y_test, baseline))
print("AUC Score (Test): %f" % metrics.roc_auc_score(y_test, baseline))
print("F1 Score (Test): %f" % metrics.f1_score(y_test, baseline))<jupyter_output>Accuracy : 0.7788
AUC Score (Test): 0.500000
F1 Score (Test): 0.000000<jupyter_text>**For XGBoost**<jupyter_code>#Predict test set:
threshold = .5
predprob = model.predict_proba(X_test)[:,1]
pred = (predprob > threshold).astype(int)
        
print("Accuracy : %.4g" % metrics.accuracy_score(y_test, pred))
print("AUC Score (Test): %f" % metrics.roc_auc_score(y_test, predprob))
print("F1 Score (Test): %f" % metrics.f1_score(y_test, pred))<jupyter_output>Accuracy : 0.8188
AUC Score (Test): 0.780556
F1 Score (Test): 0.467418<jupyter_text>The model here gives a good accuracy of 0.82 and AUC of 0.78.<jupyter_code>pd.Series(pred).value_counts()<jupyter_output><empty_output><jupyter_text>**For IsolationForest**<jupyter_code>y_pred = clf.predict(X_test)
y_pred = (-y_pred+1)/2

print("Accuracy : %.4g" % metrics.accuracy_score(y_test, y_pred))
print("AUC Score (Test): %f" % metrics.roc_auc_score(y_test, y_pred))
print("F1 Score (Test): %f" % metrics.f1_score(y_test, y_pred))
pd.Series(y_pred).value_counts()
pd.Series(y_test).value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12628324056895485, 'fraction_numerical': 0.08175633889919605, 'mean_word_length': 6.613935969868173, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '446273', 'license': 'MIT', 'n_tokens_mistral': 3399, 'n_tokens_neox': 3073, 'n_words': 595, 'path': 'Credit_data_example.ipynb', 'repo_name': 'Cheukting/credit-data-example', 'size': '8085'}