<jupyter_start><jupyter_text>Multiple Explainer Performance Comparison: Benchmark Tree, Permutation, and Exact Explainers This notebook demonstrates the use of benchmark framework to compare the performance across multiple explainers. For this purpose, tabular data is chosen and the explainers used to benchmark are: tree, permutation, and exact explainers. The metrics used to evaluate are "keep positive" and "keep negative". The masker used is Independent Masker but can also generalize to other tabular maskers.The new benchmark utility uses the new API with MaskedModel as wrapper around user-imported model and evaluates masked values of inputs.Please refer to 'Tabular Prediction Benchmark Demo.ipynb' for more description about this benchmarking example.<jupyter_code>import numpy as np 
import pandas as pd 
import xgboost
import shap 
from sklearn.model_selection import train_test_split
import shap.benchmark as benchmark
untrained_model = xgboost.XGBRegressor(n_estimators=100, subsample=0.3)
X,y = shap.datasets.boston()
X = X.values

test_size = 0.3 
random_state = 0
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=random_state)

model = untrained_model.fit(X_train, y_train)
masker = shap.maskers.Independent(X)
permutation_explainer = shap.Explainer(model.predict, masker, algorithm='permutation')
tree_explainer = shap.Explainer(model, masker, algorithm='tree')
exact_explainer = shap.Explainer(model.predict, masker, algorithm='exact')
permutation_shap_values = permutation_explainer(X_train)
tree_shap_values = tree_explainer(X_train)
exact_shap_values = exact_explainer(X_train)
metrics = {'sort_order': ['positive', 'negative'], 'perturbation': ['keep']}
benchmarks = dict()
benchmarks['permutation explainer'] = benchmark.framework.get_benchmark(permutation_explainer.model, permutation_shap_values, X_train, y_train, permutation_explainer.masker, metrics)
benchmarks['tree explainer'] = benchmark.framework.get_benchmark(tree_explainer.model, tree_shap_values, X_train, y_train, tree_explainer.masker, metrics)
benchmarks['exact explainer'] = benchmark.framework.get_benchmark(exact_explainer.model, exact_shap_values, X_train, y_train, exact_explainer.masker, metrics)
benchmark.framework.trend_plot(benchmarks)
benchmark.framework.compare_plot(benchmarks)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08106960950764007, 'fraction_numerical': 0.003395585738539898, 'mean_word_length': 7.9961832061068705, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '386978', 'license': 'MIT', 'n_tokens_mistral': 745, 'n_tokens_neox': 702, 'n_words': 197, 'path': 'notebooks/benchmark/others/Explainers Benchmark Comparison.ipynb', 'repo_name': 'anusham1990/shap', 'size': '2356'}