<jupyter_start><jupyter_text>Credit Risk Prediction Model with LightGBM  1 Introduction Our goal for this project is not only to find a relatively accurate and robust model for risk prediction, but more importantly, to improve its interpretability and give understandable explanations for sales representatives in a bank/credit card company can use to decide on accepting or rejecting applications. The variable names are somewhat cryptic and the user might want more intuitive information for both of the mechanism and the outcome. For that purpose, we manually classified 23 variables into groups and labeled them using a name that makes sense in the business context. Combined with the most important contributing factors, users can easily get insights about  the predictive relationship. To improve the user experience, the full model and explanations can be conditionally displayed in an interactive interface. We deployed our best models on GitHub and Heroku. Users can access our web app by:- opening  http://credit-risk.herokuapp.com/ via web browser, or- running these command lines on terminal```pip install --upgrade streamlitstreamlit run https://raw.githubusercontent.com/chriskhanhtran/credit-risk-prediction/master/app.py```A 5-minute explanation and live demonstration can be found at  https://youtu.be/IEk8rmLnJDk/. **Import libraries**<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import joblib

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

%matplotlib inline
sns.set_style('darkgrid')<jupyter_output><empty_output><jupyter_text>**Load data**<jupyter_code>data = pd.read_csv('../data/heloc_dataset_v1.csv')
print(data.shape)
data.head(3)
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10459 entries, 0 to 10458
Data columns (total 24 columns):
RiskPerformance                       10459 non-null object
ExternalRiskEstimate                  10459 non-null int64
MSinceOldestTradeOpen                 10459 non-null int64
MSinceMostRecentTradeOpen             10459 non-null int64
AverageMInFile                        10459 non-null int64
NumSatisfactoryTrades                 10459 non-null int64
NumTrades60Ever2DerogPubRec           10459 non-null int64
NumTrades90Ever2DerogPubRec           10459 non-null int64
PercentTradesNeverDelq                10459 non-null int64
MSinceMostRecentDelq                  10459 non-null int64
MaxDelq2PublicRecLast12M              10459 non-null int64
MaxDelqEver                           10459 non-null int64
NumTotalTrades                        10459 non-null int64
NumTradesOpeninLast12M                10459 non-null int64
PercentInstallTrades                  10459 non-null int64
MSinc[...]<jupyter_text>2 Data Preprocessing<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer

# Specify X and y
X = data.drop('RiskPerformance', axis=1)
y = data.RiskPerformance

# Factorizing `y`
y = y.map({'Bad': 1, 'Good': 0})

# One-hot encode for 'MaxDelq2PublicRecLast12M' and 'MaxDelqEver'
dummies = pd.get_dummies(X[['MaxDelq2PublicRecLast12M', 'MaxDelqEver']].astype(str), drop_first=True)
X.drop(['MaxDelq2PublicRecLast12M', 'MaxDelqEver'], axis=1, inplace=True)

# One-hot encode for -7, -8, -9 values
def get_special_dummies(X, col):
  """
  One-hot encode for -7, -8, -9 values in each column
  """
  X[col + '_-7'] = X[col].apply(lambda row:int(row==-7))
  X[col + '_-8'] = X[col].apply(lambda row:int(row==-8))
  X[col + '_-9'] = X[col].apply(lambda row:int(row==-9))

for col in X.columns:
    get_special_dummies(X, col)

# Concatenate X and dummies
X = pd.concat([X, dummies], axis=1)

# Replace negative values with NaN
X[X < 0] = np.nan

# Train test split
X_train, X_test, y_train, y_test = train_test_split(X,
                                                    y,
                                                    test_size=0.1,
                                                    random_state=1)
# Imputing Missing Values and Feature Scaling
preprocessor = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='mean')),
    ('scaler', StandardScaler())
])

# Feature scaling
X_train = preprocessor.fit_transform(X_train)
X_test = preprocessor.transform(X_test)<jupyter_output><empty_output><jupyter_text>3 Modeling<jupyter_code>from sklearn.metrics import accuracy_score, roc_curve, auc, roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from lightgbm import LGBMClassifier
from sklearn.svm import SVC

def gridSearchCV(model, params):
    """
    @param    model: sklearn estimator
    @param    params (dict): Dictionary of possible parameters
    
    @return   cv_results (DataFrame)
    """
    model_cv = GridSearchCV(model, param_grid=params, scoring='roc_auc', cv=5)
    model_cv.fit(X_train, y_train)
    cv_results = pd.DataFrame(model_cv.cv_results_)[['params', 'mean_test_score']]
    
    return cv_results

def evaluate(model, plotROC=False):
    """
    1. Print AUC and accuracy on the test set
    2. Plot ROC
    """
    model.fit(X_train, y_train)
    probs = model.predict_proba(X_test)
    preds = probs[:,1]
    fpr, tpr, threshold = roc_curve(y_test, preds)
    roc_auc = auc(fpr, tpr)
    print(f'AUC: {roc_auc:.4f}')
    
    # Find optimal threshold
    rocDf = pd.DataFrame({'fpr': fpr, 'tpr':tpr, 'threshold':threshold})
    rocDf['tpr - fpr'] = rocDf.tpr - rocDf.fpr
    optimalThreshold = rocDf.threshold[rocDf['tpr - fpr'].idxmax()]
    
    # Get accuracy over the test set
    y_pred = np.where(preds >= optimalThreshold, 1, 0)
    accuracy = accuracy_score(y_test, y_pred)
    print(f'Accuracy: {accuracy*100:.2f}%')
    
    # Plot ROC AUC
    if plotROC:
        plt.title('Receiver Operating Characteristic')
        plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
        plt.legend(loc = 'lower right')
        plt.plot([0, 1], [0, 1],'r--')
        plt.xlim([0, 1])
        plt.ylim([0, 1])
        plt.ylabel('True Positive Rate')
        plt.xlabel('False Positive Rate')
        plt.show()<jupyter_output><empty_output><jupyter_text>3.1 Random Forest<jupyter_code>params = {'n_estimators': [50, 200]}
rfc = RandomForestClassifier(random_state=1)
print(gridSearchCV(rfc, params))
rfc = RandomForestClassifier(n_estimators=500, random_state=1)
evaluate(rfc, plotROC=True)<jupyter_output>AUC: 0.7981
Accuracy: 74.00%<jupyter_text>3.2 Logistic Regression<jupyter_code>params = {'C': [0.01, 0.1, 1, 10]}
logit = LogisticRegression()
print(gridSearchCV(logit, params))
logit = LogisticRegression(C=0.01)
evaluate(logit, plotROC=True)<jupyter_output>AUC: 0.7992
Accuracy: 72.94%<jupyter_text>3.3 LightGBM<jupyter_code>params = {'learning_rate': [0.051, 0.052, 0.055]}
lightGBM = LGBMClassifier(learning_rate=0.05,
                          n_estimators=200,
                          num_leaves=5,
                          min_data_in_leaf=4,
                          max_depth=5,
                          max_bin=55,
                          bagging_fraction=0.5,
                          bagging_freq=5,
                          feature_fraction=0.24,
                          feature_fraction_seed=9,
                          bagging_seed=9,
                          min_sum_hessian_in_leaf=11)
print(gridSearchCV(lightGBM, params))
lightGBM = LGBMClassifier(learning_rate=0.05,
                          n_estimators=200,
                          num_leaves=5,
                          min_data_in_leaf=4,
                          max_depth=5,
                          max_bin=55,
                          bagging_fraction=0.5,
                          bagging_freq=5,
                          feature_fraction=0.24,
                          feature_fraction_seed=9,
                          bagging_seed=9,
                          min_sum_hessian_in_leaf=11)
evaluate(lightGBM, plotROC=True)<jupyter_output>AUC: 0.8012
Accuracy: 73.80%<jupyter_text>3.4 SVM<jupyter_code>params = {'C': [0.1]}
svc = SVC(gamma='auto', random_state=1)
print(gridSearchCV(svc, params))
svc = SVC(C=0.1, kernel='linear', degree=2,
          gamma=0.1, random_state=1, probability=True)
evaluate(svc, plotROC=True)<jupyter_output>AUC: 0.7918
Accuracy: 72.56%<jupyter_text>3.5 Averaging Model<jupyter_code>from sklearn.base import BaseEstimator, RegressorMixin, TransformerMixin, clone

class AveragingModel(BaseEstimator, RegressorMixin, TransformerMixin):
    def __init__(self, models):
        self.models = models

    def fit(self, X, y):
        # Create clone models
        self.models_ = [clone(x) for x in self.models]

        # Train cloned models
        for model in self.models_:
            model.fit(X, y)

        return self

    def predict(self, X):
        # Get predictions from trained clone models
        predictions = np.column_stack(
            [model.predict(X) for model in self.models_])
        # Average predictions
        predictions = np.mean(predictions, axis=1)
        predictions = np.where(predictions >= 0.5, 1, 0)
        return predictions
    
    def predict_proba(self, X):
        # Get predictions from trained clone models
        probs_1 = np.column_stack(
            [model.predict_proba(X)[:, 1] for model in self.models_])
        # Average predictions
        probs_1 = np.mean(probs_1, axis=1)
        probs_0 = 1 - probs_1
        probs = np.column_stack([probs_0, probs_1])
        return probs
avg_model = AveragingModel([lightGBM, rfc, logit])
evaluate(avg_model, plotROC=True)<jupyter_output>AUC: 0.8035
Accuracy: 73.52%<jupyter_text>4 Save models<jupyter_code>from sklearn.pipeline import Pipeline
import joblib

def save_model(model, model_name):
    print('Saving ' + model_name + ' ...')
    pipeline = Pipeline(steps=[('imputer', SimpleImputer(strategy='mean')),
                               ('scaler', StandardScaler()),
                               ('model', model)])
    pipeline.fit(X, y)
    filename = 'models/' + model_name + '.joblib.pkl'
    joblib.dump(pipeline, filename, compress=9)
    print('Complete!\n')
save_model(logit, 'logit')
save_model(rfc, 'rfc')
save_model(svc, 'svc')
save_model(lightGBM, 'lgbm')
### Load models
#import joblib
#model_name = 'lgbm'
#filename = 'models/' + model_name + '.joblib.pkl'
#model = joblib.load(filename)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10274688717802491, 'fraction_numerical': 0.03260146373918829, 'mean_word_length': 2.9556390977443607, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '480080', 'license': 'MIT', 'n_tokens_mistral': 3626, 'n_tokens_neox': 3368, 'n_words': 808, 'path': 'files/Credit Risk Prediction with LightBGM.ipynb', 'repo_name': 'chenxitao-simon/chenxitao-simon.github.io', 'size': '10521'}