<jupyter_start><jupyter_text>Interpretation of individual predictions with Local-DIFFI In this notebook are reproduced the experimental results on the Local-DIFFI method described in our paper [Interpretable Anomaly Detection with DIFFI: Depth-based Feature Importance for the Isolation Forest](https://arxiv.org/pdf/2007.11117.pdf).The present code represents the very first release of the DIFFI algorithm: the code in this version is not yet optimized.<jupyter_code>import os
import numpy as np
import pickle as pkl 
import time
import matplotlib.pyplot as plt 
%matplotlib inline

from DIFFI import DIFFI
from DIFFI.data import load_glass_dataset, load_syntethic_dataset
from DIFFI.comparison import local_shap_batch

from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.utils import shuffle
import shap
import interpretability_module as interp
from utils import *
from graphics import plot_ranking_glass<jupyter_output><empty_output><jupyter_text>"Glass" dataset  Load data<jupyter_code>X_train, y_train, X_test, y_test = load_glass_dataset()<jupyter_output><empty_output><jupyter_text>Train the Isolation Forest<jupyter_code>iforest = IsolationForest(
    n_estimators= 100, 
    max_samples=64, 
    contamination='auto', 
    random_state=0, 
    bootstrap=False)
iforest.fit(X_train)
y_tr_pred = np.array(iforest.decision_function(X_train) < 0).astype('int')
f1 = f1_score(y_train, y_tr_pred)
print('\nF1 score (on training data): {}'.format(round(f1, 3)))<jupyter_output>F1 score (on training data): 0.556<jupyter_text>Get predictions and interpretations for test outliersWe consider data points in the "headlamps glass" class (class 7 in the original dataset) as test outliers. We compare feature importance scores provided by SHAP with those obtained with Local-DIFFI.  Predictions<jupyter_code>y_te_pred = np.array(iforest.decision_function(X_test) < 0).astype('int')
print('Detected anomalies (test, class 7): {} out of {}'.format(int(sum(y_te_pred)), len(y_test)))<jupyter_output>Detected anomalies (test, class 7): 28 out of 29<jupyter_text>Local-DIFFI<jupyter_code>diffi_te, ord_idx_diffi_te = DIFFI(iforest).explain(X_test)
plot_ranking_glass(ord_idx_diffi_te, title = 'Feature ranking glass dataset class 7 outliers - Local DIFFI')
#print('Average computational time Local-DIFFI: {}'.format(round(np.mean(exec_time_diffi_te),3)))<jupyter_output><empty_output><jupyter_text>SHAP<jupyter_code>shap_te, ord_idx_shap_te, exec_time_shap_te = local_shap_batch(iforest, X_test[np.where(y_te_pred == 1)])
plot_ranking_glass(ord_idx_shap_te, title = 'Feature ranking glass dataset class 7 outliers - SHAP')
print('Average computational time SHAP: {}'.format(round(np.mean(exec_time_shap_te),3)))<jupyter_output>Average computational time SHAP: 0.097<jupyter_text>Synthetic datasetWe consider data points represented by a 6-dimensional feature vector. Features $f_1$ and $f_2$ are the coordinates of the points on the 2-D plane, features $f_3$, $f_4$, $f_5$, $f_6$ are white noise samples. Inliers are concentrated near the origin, while outliers are far away from the origin.  Load data<jupyter_code>X_tr, y_tr, X_xaxis, X_yaxis, X_bisec = load_syntethic_dataset()<jupyter_output><empty_output><jupyter_text>Train the Isolation Forest<jupyter_code>iforest = IsolationForest(n_estimators=100, 
                          max_samples=256, 
                          contamination='auto', 
                          random_state=0, 
                          bootstrap=False)
iforest.fit(X_tr)
y_tr_pred = np.array(iforest.decision_function(X_tr) < 0).astype('int')
f1 = f1_score(y_tr, y_tr_pred)
print('\nF1 score (on training data): {}'.format(round(f1, 3)))<jupyter_output>F1 score (on training data): 0.716<jupyter_text>Get predictions and interpretations for test outliersWe consider as test outliers the points displayed in the figure below (projected on the subspace of the relevant features, i.e. $f_1$ and $f_2$). We have:- outliers on the x-axis (blue points)- outliers on the y-axis (orange points)- outliers on the bisector (green points).We compare feature importance scores provided by SHAP with those obtained with Local-DIFFI.<jupyter_code>fig, ax = plt.subplots()
ax.scatter(X_xaxis[:,0], X_xaxis[:,1], cmap='Blues')
ax.scatter(X_yaxis[:,0], X_yaxis[:,1], cmap='Greens')
ax.scatter(X_bisec[:,0], X_bisec[:,1], cmap='Oranges')
ax.set_title('Synthetic outliers');<jupyter_output><empty_output><jupyter_text>Predictions - Outliers on the x-axis<jupyter_code>y_xaxis_pred = np.array(iforest.decision_function(X_xaxis) < 0).astype('int')
print('Detected anomalies: {} out of {}'.format(int(sum(y_xaxis_pred)), len(y_xaxis_pred)))<jupyter_output>Detected anomalies: 48 out of 100<jupyter_text>Local-DIFFI - Outliers on the x-axis<jupyter_code>diffi_xaxis, ord_idx_diffi_xaxis = DIFFI(iforest).explain(X_xaxis[np.where(y_xaxis_pred == 1)])
plot_ranking_syn(ord_idx_diffi_xaxis, 'Feature ranking syn dataset, ' + r'$f_1$'+ ' outliers - Local DIFFI')
#print('Average computational time Local-DIFFI: {}'.format(round(np.mean(exec_time_diffi_xaxis),3)))<jupyter_output><empty_output><jupyter_text>SHAP - Outliers on the x-axis<jupyter_code>shap_xaxis, ord_idx_shap_xaxis, exec_time_shap_xaxis = local_shap_batch(iforest, X_xaxis[np.where(y_xaxis_pred == 1)])
plot_ranking_syn(ord_idx_shap_xaxis, 'Feature ranking syn dataset, ' + r'$f_1$'+ ' outliers - SHAP')
print('Average computational time SHAP: {}'.format(round(np.mean(exec_time_shap_xaxis),3)))<jupyter_output>Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.[...]<jupyter_text>Predictions - Outliers on the y-axis<jupyter_code>y_yaxis_pred = np.array(iforest.decision_function(X_yaxis) < 0).astype('int')
print('Detected anomalies: {} out of {}'.format(int(sum(y_yaxis_pred)), len(y_yaxis_pred)))<jupyter_output>Detected anomalies: 34 out of 100<jupyter_text>Local-DIFFI - Outliers on the y-axis<jupyter_code>diffi_yaxis, ord_idx_diffi_yaxis, exec_time_diffi_yaxis = local_diffi_batch(iforest, X_yaxis[np.where(y_yaxis_pred == 1)])
plot_ranking_syn(ord_idx_diffi_yaxis, 'Feature ranking syn dataset, ' + r'$f_2$'+ ' outliers - Local DIFFI')
print('Average computational time Local-DIFFI: {}'.format(round(np.mean(exec_time_diffi_yaxis),3)))<jupyter_output>Average computational time Local-DIFFI: 0.068<jupyter_text>SHAP - Outliers on the y-axis<jupyter_code>shap_yaxis, ord_idx_shap_yaxis, exec_time_shap_yaxis = local_shap_batch(iforest, X_yaxis[np.where(y_yaxis_pred == 1)])
plot_ranking_syn(ord_idx_shap_yaxis, 'Feature ranking syn dataset, ' + r'$f_2$'+ ' outliers - SHAP')
print('Average computational time SHAP: {}'.format(round(np.mean(exec_time_shap_yaxis),3)))<jupyter_output>Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.[...]<jupyter_text>Predictions - Outliers on the bisector<jupyter_code>y_bisec_pred = np.array(iforest.decision_function(X_bisec) < 0).astype('int')
print('Detected anomalies: {} out of {}'.format(int(sum(y_bisec_pred)), len(y_bisec_pred)))<jupyter_output>Detected anomalies: 94 out of 100<jupyter_text>Local-DIFFI - Outliers on the bisector<jupyter_code># Local-DIFFI
diffi_bisec, ord_idx_diffi_bisec, exec_time_diffi_bisec = local_diffi_batch(iforest, X_bisec[np.where(y_bisec_pred == 1)])
plot_ranking_syn(ord_idx_diffi_bisec, 'Feature ranking syn dataset, ' + r'$f_1$'+ ' and ' +r'$f_2$'+ ' outliers - Local DIFFI')
print('Average computational time Local-DIFFI: {}'.format(round(np.mean(exec_time_diffi_bisec),3)))<jupyter_output>Average computational time Local-DIFFI: 0.068<jupyter_text>SHAP - Outliers on the bisector<jupyter_code>shap_bisec, ord_idx_shap_bisec, exec_time_shap_bisec = local_shap_batch(iforest, X_bisec[np.where(y_bisec_pred == 1)])
plot_ranking_syn(ord_idx_shap_bisec, 'Feature ranking syn dataset, ' + r'$f_1$'+ ' and ' +r'$f_2$'+ ' outliers - SHAP')
print('Average computational time SHAP: {}'.format(round(np.mean(exec_time_shap_bisec),3)))<jupyter_output>Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
Setting feature_perturbation = "tree_path_dependent" because no background data was given.[...]
{'fraction_non_alphanumeric': 0.09738519818904186, 'fraction_numerical': 0.011457082139887277, 'mean_word_length': 7.231178707224335, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '287057', 'license': 'MIT', 'n_tokens_mistral': 3721, 'n_tokens_neox': 3516, 'n_words': 888, 'path': 'examples/Local_diffi_demo-V2.ipynb', 'repo_name': 'lucianolorenti/DIFFI', 'size': '10823'}