<jupyter_start><jupyter_text>ShapRFECV - Recursive Feature Elimination using SHAP importance[![open in colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ing-bank/probatus/blob/master/docs/tutorials/nb_shap_feature_elimination.ipynb)Recursive Feature Elimination allows you to efficiently reduce the number of features in your dataset, without losing the predictive power of the model. `probatus` implements the following feature elimination routine for **tree-based & linear models**:    While any features left, iterate:        1. (Optional) Tune hyperparameters, in case sklearn compatible search CV e.g. `GridSearchCV` or            `RandomizedSearchCV` or `BayesSearchCV`are passed as clf,        2. Calculate SHAP feature importance using Cross-Validation,        3. Remove `step` lowest importance features.The functionality is similar to [RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html), yet it removes the lowest importance features, based on SHAP features importance. It also supports the use of any hyperparameter search schema that is consistent with sklearn API e.g. [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html), [RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) and [BayesSearchCV](https://scikit-optimize.github.io/stable/modules/generated/skopt.BayesSearchCV.htmlskopt.BayesSearchCV) passed as a `clf`, thanks to which you can perform hyperparameter optimization at each step of the search.hyperparameters of the model at each round, to tune the model for each features set. Lastly, it supports categorical features (`object` and `category` dtype) and missing values in the data, as long as the model supports them. The main advantages of using this routine are:- It uses a tree-based or a linear model to detect the complex relations between features and the target.- It uses SHAP importance, which is one of the most reliable ways to estimate features importance. Unlike many other techniques, it works with missing values and categorical variables.- Supports the use of sklearn compatible hyperparameter search schemas e.g. [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html), [RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) and [BayesSearchCV](https://scikit-optimize.github.io/stable/modules/generated/skopt.BayesSearchCV.htmlskopt.BayesSearchCV), in order to optimize hyperparameters at each iteration. This way you can assess if the removal of a given feature reduces the predictive power, or simply requires additional tuning of the model.- You can also provide a list of features that should not be eliminated e.g. incase of prior knowledge.The disadvantages are:- Removing lowest [SHAP](https://shap.readthedocs.io/en/latest/) importance feature does not always translate to choosing the feature with the lowest impact on a model's performance. Shap importance illustrates how strongly a given feature affects the output of the model, while disregarding correctness of this prediction.- Currently, the functionality only supports tree-based & linear binary classifiers, in the future the scope might be extended.- For large datasets, performing hyperparameter optimization can be very computationally expensive. For gradient boosted tree models, one alternative is to use early stopping of the training step. For this, see [EarlyStoppingShapRFECV](EarlyStoppingShapRFECV) Setup the datasetIn order to use the functionality, let's set up an example dataset with:- 18 numerical features- 1 static feature- 1 static feature- 1 feature with missing values<jupyter_code>%%capture
!pip install probatus
!pip install lightgbm
from probatus.feature_elimination import ShapRFECV
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd
import lightgbm
from sklearn.model_selection import RandomizedSearchCV
 
feature_names = ['f1', 'f2_missing', 'f3_static', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', 'f10', 'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f20']
 
# Prepare two samples
X, y = make_classification(n_samples=1000, class_sep=0.05, n_informative=6, n_features=20,
                           random_state=0, n_redundant=10, n_clusters_per_class=1)
X = pd.DataFrame(X, columns=feature_names)

X['f2_missing'] = X['f2_missing'].apply(lambda x: x if np.random.rand()<0.8 else np.nan)
X['f3_static'] = 0
#First 5 rows of first 5 columns
X[feature_names[:5]].head()<jupyter_output><empty_output><jupyter_text>Set up the model and model tuningYou need to set up the model that you would like to use in the feature elimination. `probatus` requires a **tree-based or linear binary classifier** in order to speed up the computation of SHAP feature importance at each step. We recommend using [LGBMClassifier](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html), which by default handles missing values and categorical features.The example below applies randomized search in order to optimize the hyperparameters of the model at each iteration of the search.<jupyter_code>clf = lightgbm.LGBMClassifier(max_depth=5, class_weight='balanced')

param_grid = {
    'n_estimators': [5, 7, 10],
    'num_leaves': [3, 5, 7, 10],
}
search = RandomizedSearchCV(clf, param_grid)<jupyter_output><empty_output><jupyter_text>Apply ShapRFECVNow let's apply the [ShapRFECV](https://ing-bank.github.io/probatus/api/feature_elimination.html).<jupyter_code>shap_elimination = ShapRFECV(
    clf=search, step=0.2, cv=10, scoring='roc_auc', n_jobs=3)
report = shap_elimination.fit_compute(X, y)<jupyter_output><empty_output><jupyter_text>At the end of the process, you can investigate the results for each iteration.<jupyter_code>#First 5 rows of first 5 columns
report[['num_features', 'features_set', 'val_metric_mean']]<jupyter_output><empty_output><jupyter_text>Once the process is completed, you can visualize the results. Let's investigate the performance plot. In this case, the Validation AUC score has the highest Validation AUC at 11 features and a peak at 6 features.<jupyter_code>performance_plot = shap_elimination.plot()<jupyter_output><empty_output><jupyter_text>Let's see the final feature set:<jupyter_code>shap_elimination.get_reduced_features_set(num_features=6)<jupyter_output><empty_output><jupyter_text>You can also provide a list of features that should not be eliminated. Say based on your prior knowledge you know that the features `f10,f19,f15` are important and should not be eliminated. This can be done by providing a list of columns to `columns_to_keep` parameter in the `fit()` function.<jupyter_code>shap_elimination = ShapRFECV(
    clf=search, step=0.2, cv=10, scoring='roc_auc', n_jobs=3  ,min_features_to_select=4)
report = shap_elimination.fit_compute(X, y, columns_to_keep=['f10','f15','f19'])

performance_plot = shap_elimination.plot()<jupyter_output><empty_output><jupyter_text>Let's see the final feature set:<jupyter_code>shap_elimination.get_reduced_features_set(num_features=4)<jupyter_output><empty_output><jupyter_text>EarlyStoppingShapRFECV[Early stopping](https://en.wikipedia.org/wiki/Early_stopping) is a type of regularization, common in [gradient boosted trees](https://en.wikipedia.org/wiki/Gradient_boostingGradient_tree_boosting), such as [LightGBM](https://lightgbm.readthedocs.io/en/latest/index.html) and [XGBoost](https://xgboost.readthedocs.io/en/latest/index.html). It consists of measuring how well the model performs after each base learner is added to the ensemble tree, using a relevant scoring metric. If this metric does not improve after a certain number of training steps, the training can be stopped before the maximum number of base learners is reached. Early stopping is thus a way of mitigating overfitting in a relatively cheaply, without having to find the ideal regularization hyperparameters. It is particularly useful for handling large datasets, since it reduces the number of training steps which can decrease the modelling time.`EarlyStoppingShapRFECV` is a child of `ShapRFECV` with limited support for early stopping and the example below shows how to use it with LightGBM.<jupyter_code>from probatus.feature_elimination import EarlyStoppingShapRFECV

clf = lightgbm.LGBMClassifier(n_estimators=200, max_depth=3)

# Run feature elimination
shap_elimination = EarlyStoppingShapRFECV(
    clf=search, step=0.2, cv=10, scoring='roc_auc', eval_metric='auc', early_stopping_rounds=5, n_jobs=3)
report = shap_elimination.fit_compute(X, y)
# Make plots
performance_plot = shap_elimination.plot()

# Get final feature set
final_features_set = shap_elimination.get_reduced_features_set(num_features=9)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08485320284697509, 'fraction_numerical': 0.012566725978647686, 'mean_word_length': 6.131641554321967, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 37, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '191115', 'license': 'MIT', 'n_tokens_mistral': 2716, 'n_tokens_neox': 2578, 'n_words': 893, 'path': 'docs/tutorials/nb_shap_feature_elimination.ipynb', 'repo_name': 'lproth/probatus', 'size': '8992'}