<jupyter_start><jupyter_text>Tutorial: FICO Explainable Machine Learning Challenge In this tutorial, we use the dataset form the FICO Explainable Machine Learning Challenge: https://community.fico.com/s/explainable-machine-learning-challenge. The goal is to create a pipeline by combining a binning process and logistic regression to obtain an explainable model and compare it against a black-box model using Gradient Boosting Tree (GBT) as an estimator.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from optbinning import BinningProcess

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report
from sklearn.metrics import auc, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline<jupyter_output><empty_output><jupyter_text>Download the dataset from the link above and load it.<jupyter_code>df = pd.read_csv("data/FICO_challenge/heloc_dataset_v1.csv", sep=",")
variable_names = list(df.columns[1:])
X = df[variable_names].values<jupyter_output><empty_output><jupyter_text>Transform the categorical dichotomic target variable into numerical.<jupyter_code>y = df.RiskPerformance.values
mask = y == "Bad"
y[mask] = 1
y[~mask] = 0
y = y.astype(int)<jupyter_output><empty_output><jupyter_text>Modeling The data dictionary of this challenge includes three special values/codes:    * -9 No Bureau Record or No Investigation* -8 No Usable/Valid Trades or Inquiries* -7 Condition not Met (e.g. No Inquiries, No Delinquencies)<jupyter_code>special_codes = [-9, -8, -7]<jupyter_output><empty_output><jupyter_text>This challenge imposes monotonicity constraints with respect to the probability of a bad target for many of the variables. We apply these rules by passing the following dictionary of parameters for these variables involved.<jupyter_code>binning_fit_params = {
    "ExternalRiskEstimate": {"monotonic_trend": "descending"},
    "MSinceOldestTradeOpen": {"monotonic_trend": "descending"},
    "MSinceMostRecentTradeOpen": {"monotonic_trend": "descending"},
    "AverageMInFile": {"monotonic_trend": "descending"},
    "NumSatisfactoryTrades": {"monotonic_trend": "descending"},
    "NumTrades60Ever2DerogPubRec": {"monotonic_trend": "ascending"},
    "NumTrades90Ever2DerogPubRec": {"monotonic_trend": "ascending"},
    "PercentTradesNeverDelq": {"monotonic_trend": "descending"},
    "MSinceMostRecentDelq": {"monotonic_trend": "descending"},
    "NumTradesOpeninLast12M": {"monotonic_trend": "ascending"},
    "MSinceMostRecentInqexcl7days": {"monotonic_trend": "descending"},
    "NumInqLast6M": {"monotonic_trend": "ascending"},
    "NumInqLast6Mexcl7days": {"monotonic_trend": "ascending"},
    "NetFractionRevolvingBurden": {"monotonic_trend": "ascending"},
    "NetFractionInstallBurden": {"monotonic_trend": "ascending"},
    "NumBank2NatlTradesWHighUtilization": {"monotonic_trend": "ascending"}
}<jupyter_output><empty_output><jupyter_text>Instantiate a ``BinningProcess`` object class with variable names, special codes and dictionary of binning parameters. Create a explainable model pipeline and a black-blox pipeline.<jupyter_code>binning_process = BinningProcess(variable_names, special_codes=special_codes,
                                 binning_fit_params=binning_fit_params)
clf1 = Pipeline(steps=[('binning_process', binning_process),
                      ('classifier', LogisticRegression(solver="lbfgs"))])

clf2 = LogisticRegression(solver="lbfgs")

clf3 = GradientBoostingClassifier()<jupyter_output><empty_output><jupyter_text>Split dataset into train and test. Fit pipelines with training data, then generate classification reports to show the main classification metrics.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf1.fit(X_train, y_train)
clf2.fit(X_train, y_train)
clf3.fit(X_train, y_train)
y_pred = clf1.predict(X_test)
print(classification_report(y_test, y_pred))
y_pred = clf2.predict(X_test)
print(classification_report(y_test, y_pred))
y_pred = clf3.predict(X_test)
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.71      0.66      0.68      1004
           1       0.70      0.75      0.73      1088

    accuracy                           0.71      2092
   macro avg       0.71      0.70      0.70      2092
weighted avg       0.71      0.71      0.70      2092<jupyter_text>Plot the Receiver Operating Characteristic (ROC) metric to evaluate and compare the classifiers' prediction.<jupyter_code>probs = clf1.predict_proba(X_test)
preds = probs[:,1]
fpr1, tpr1, threshold = roc_curve(y_test, preds)
roc_auc1 = auc(fpr1, tpr1)

probs = clf2.predict_proba(X_test)
preds = probs[:,1]
fpr2, tpr2, threshold = roc_curve(y_test, preds)
roc_auc2 = auc(fpr2, tpr2)

probs = clf3.predict_proba(X_test)
preds = probs[:,1]
fpr3, tpr3, threshold = roc_curve(y_test, preds)
roc_auc3 = auc(fpr3, tpr3)
plt.title('Receiver Operating Characteristic')
plt.plot(fpr1, tpr1, 'b', label='Binning+LR: AUC = {0:.2f}'.format(roc_auc1))
plt.plot(fpr2, tpr2, 'g', label='LR: AUC = {0:.2f}'.format(roc_auc2))
plt.plot(fpr3, tpr3, 'r', label='GBT: AUC = {0:.2f}'.format(roc_auc3))
plt.legend(loc='lower right')
plt.plot([0, 1], [0, 1],'k--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>The plot above shows the increment in terms of model performance after binning when the logistic estimator is chosen. Furthermore, a previous binning process might reduce numerical instability issues, as confirmed when fitting the classifier ``clf2``.  Binning process statistics The binning process of the pipeline can be retrieved to show information about the problem and timing statistics.<jupyter_code>binning_process.information(print_level=2)<jupyter_output>optbinning (Version 0.13.0)
Copyright (c) 2019-2021 <NAME>, Apache License 2.0

  Begin options
    max_n_prebins                         20   * d
    min_prebin_size                     0.05   * d
    min_n_bins                            no   * d
    max_n_bins                            no   * d
    min_bin_size                          no   * d
    max_bin_size                          no   * d
    max_pvalue                            no   * d
    max_pvalue_policy            consecutive   * d
    selection_criteria                    no   * d
    fixed_variables                       no   * d
    categorical_variables                 no   * d
    special_codes                        yes   * U
    split_digits                          no   * d
    binning_fit_params                   yes   * U
    binning_transform_params              no   * d
    verbose                            False   * d
  End options

  Statistics
    Number of records                   83[...]<jupyter_text>The ``summary`` method returns basic statistics for each binned variable.<jupyter_code>binning_process.summary()<jupyter_output><empty_output><jupyter_text>The ``get_binned_variable`` method serves to retrieve an optimal binning object, which can be analyzed in detail afterward.<jupyter_code>optb = binning_process.get_binned_variable("NumBank2NatlTradesWHighUtilization")
optb.binning_table.build()
optb.binning_table.plot(metric="event_rate")
optb.binning_table.analysis()<jupyter_output>---------------------------------------------
OptimalBinning: Binary Binning Table Analysis
---------------------------------------------

  General metrics

    Gini index               0.30840174
    IV (Jeffrey)             0.33485338
    JS (Jensen-Shannon)      0.04101657
    Hellinger                0.04143062
    Triangular               0.16088993
    KS                       0.26468885
    HHI                      0.25839135
    HHI (normalized)         0.13478991
    Cramer's V               0.28612357
    Quality score            0.22212566

  Monotonic trend             ascending

  Significance tests

    Bin A  Bin B  t-statistic      p-value      P[A > B]  P[B > A]
        0      1   234.556206 6.049946e-53 5.576352e-107  1.000000
        1      2    15.402738 8.686234e-05  7.665294e-06  0.999992
        2      3     8.386138 3.780934e-03  1.290940e-03  0.998709
        3      4     0.113909 7.357368e-01  3.678894e-01  0.632111
{'fraction_non_alphanumeric': 0.10256410256410256, 'fraction_numerical': 0.0467919177596597, 'mean_word_length': 3.2277722277722276, 'pattern_counts': {'":': 32, '<': 43, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '17646', 'license': 'Apache-2.0', 'n_tokens_mistral': 2978, 'n_tokens_neox': 2705, 'n_words': 663, 'path': 'doc/source/tutorials/tutorial_binning_process_FICO_xAI.ipynb', 'repo_name': 'philip-khor/optbinning', 'size': '8481'}