<jupyter_start><jupyter_text>Case studyIn the case study we demonstrate how to generate counterfactual explanations by using our library on Statlog (German Credit Data) Data Set from UCI ML repository.<jupyter_code>from tensorflow import keras
import numpy as np
import pandas as pd
import warnings
warnings.filterwarnings('ignore')
np.random.seed(44)<jupyter_output><empty_output><jupyter_text>Data and model loadingStatlog (German Credit Data) was gathered from UCI ML repository and consists of 20 features and 61 columns (most of them are represented by one-hot encoding). The dataset is wrapped in GermanData class, for easier use. Additionally, we load a pretrained keras model (simple logistic regression with 2 outputs) which will be used for prediction making.<jupyter_code>from data import GermanData

german_data = GermanData('data/datasets/input_german.csv', 'data/datasets/labels_german.csv')
model = keras.models.load_model('models/model_german')
german_data.input.sample(5)<jupyter_output><empty_output><jupyter_text>Test sample selectionWe select one instance, for which we want to calculate the counterfactual.<jupyter_code>X_test = german_data.input.iloc[0]
X_test<jupyter_output><empty_output><jupyter_text>Data scalingFor the model to work, the dataset has to be standardized. The GermanData scale function uses StandardScaler from sklearn.<jupyter_code>X_test_scaled = german_data.scale(X_test)
X_test_scaled<jupyter_output><empty_output><jupyter_text>Now, we evaluate the model on the test sample.<jupyter_code>model.predict(np.expand_dims(X_test_scaled, axis=0))<jupyter_output><empty_output><jupyter_text>These outputs  are interpreted as the model prediction of the testing instance to class 1. For this credit data it means a bad (not paying loans) class of customers.  Counterfactual explanations generationHere we demonstrate how to generate counterfactual explanations using CADEX, FIMAP and ECE methods implemented in our library.  FIMAP<jupyter_code>from counterfactuals.explainers import Fimap

model_predictions = model.predict(german_data.X_train)
model_predictions = np.argmax(model_predictions, axis=1)
fimap = Fimap()
fimap.fit(german_data.X_train, model_predictions)

cf_fimap = fimap.generate(X_test)
model.predict(german_data.scale(cf_fimap))<jupyter_output><empty_output><jupyter_text>The class predicted for the counterfactual is 0, meaning good credit score.<jupyter_code>from counterfactuals.visualization import show

show(X_test, cf_fimap)<jupyter_output><empty_output><jupyter_text>We can see two things wrong with the generated counterfactual: - the categorical variables, originally represented in one-hot encoding, were changed to values different than 0 or 1- the value of age variable decreased, resulting in poor quality of the counterfactual - we can't recommend someone to decrease their age in order to obtain creditTo fix it we can use constraints, which can be defined either in code or in spreadsheets (which can be used by users not familiar with programming).<jupyter_code>from counterfactuals.constraints import OneHot, ValueMonotonicity

constraints = [OneHot("account_status", 7, 10), 
               OneHot("credit_history", 11, 15),
               OneHot("purpose", 16, 25), 
               OneHot("savings", 26, 30), 
               OneHot("sex_status", 31, 34),
               OneHot("debtors", 35, 37), 
               OneHot("property", 38, 41),
               OneHot("other_installment_plans", 42, 44), 
               OneHot("housing", 45, 47), 
               OneHot("job", 48, 51),
               OneHot("phone", 52, 53), 
               OneHot("foreign", 54, 55), 
               OneHot("employment", 56, 60),
               ValueMonotonicity(['age'], "increasing")
              ]
fimap = Fimap(constraints=constraints)
fimap.fit(german_data.X_train, model_predictions)

cf_fimap_constraints = fimap.generate(X_test)
model.predict(german_data.scale(cf_fimap_constraints))<jupyter_output><empty_output><jupyter_text>As FIMAP is a method that doesn't guarantee finding a true counterfactual (with a change in prediction), we should always check whether the prediction changed. Here, we'd have to tune the hyperparameters to find a true counterfactual.<jupyter_code>show(X_test, cf_fimap_constraints, constraints=constraints)<jupyter_output><empty_output><jupyter_text>The constraints column shows what constraints have been placed on the given attribute. If these constraints were not met, it would be marked with an asterisk.  CADEXFor CADEX, we can either pass scaled instance and then unscale the obtained counterfactual or pass transform and inverse_transform parameters to the constructor.<jupyter_code>from counterfactuals.explainers import Cadex 

cadex = Cadex(model, transform=german_data.scale, inverse_transform=german_data.unscale)
cf = cadex.generate(X_test)
model.predict(german_data.scale(cf))
show(X_test, cf)<jupyter_output><empty_output><jupyter_text>For CADEX we can also use constraints:<jupyter_code>cadex = Cadex(model, n_changed=10, transform=german_data.scale, inverse_transform=german_data.unscale, constraints=constraints)
cf = cadex.generate(X_test)
model.predict(german_data.scale(cf))
show(X_test, cf, constraints=constraints)<jupyter_output><empty_output><jupyter_text>ECE We can use ECE to select the best counterfactuals - we'll run it using 10 explainers, 5 FIMAPs and 5 CADEXs with different parameter values.<jupyter_code>fimaps = []
fimap_hyperparameters = [
    (0.1, 0.001, 0.01),
    (0.1, 0.05, 0.5),
    (0.2, 0.01, 0.1),
    (0.2, 0.08, 0.8),
    (0.5, 0.001, 0.01)
]
for tau, l1, l2 in fimap_hyperparameters:
    fimap = Fimap(tau, l1, l2)
    fimap.fit(german_data.X_train, model_predictions, epochs=300)
    fimaps.append(fimap)
    
cadexs = []
n_list = [5, 8, 10, 15, 20]
for n_changed in n_list:
    cadex = Cadex(model, n_changed, transform=german_data.scale, inverse_transform=german_data.unscale)
    cadexs.append(cadex)<jupyter_output><empty_output><jupyter_text>Now, let's use ECE and generate up to 4 best counterfactuals:<jupyter_code>from counterfactuals.explainers import ECE
from counterfactuals.visualization import compare
pd.set_option("display.max_rows", None)

ece = ECE(4, columns=list(german_data.X_train.columns), bces=cadexs + fimaps, dist=2, h=5, lambda_=0.001, n_jobs=1)
cfs = ece.generate(X_test)<jupyter_output><empty_output><jupyter_text>First, let's see all 10 counterfactuals:<jupyter_code>compare(X_test, ece.get_aggregated_cfs())<jupyter_output><empty_output><jupyter_text>And now, those selected by ECE:<jupyter_code>compare(X_test, cfs)<jupyter_output><empty_output><jupyter_text>We can do the same with constraints:<jupyter_code>fimaps = []
fimap_hyperparameters = [
    (0.1, 0.001, 0.01),
    (0.1, 0.05, 0.5),
    (0.2, 0.01, 0.1),
    (0.2, 0.08, 0.8),
    (0.5, 0.001, 0.01)
]
for tau, l1, l2 in fimap_hyperparameters:
    fimap = Fimap(tau, l1, l2, constraints=constraints)
    fimap.fit(german_data.X_train, model_predictions, epochs=300)
    fimaps.append(fimap)
    
cadexs = []
n_list = [10, 14, 18, 20, 25]
for n_changed in n_list:
    cadex = Cadex(model, n_changed, transform=german_data.scale, inverse_transform=german_data.unscale, constraints=constraints)
    cadexs.append(cadex)
ece = ECE(4, columns=list(german_data.X_train.columns), bces=cadexs + fimaps, dist=2, h=5, lambda_=0.001, n_jobs=1)
cfs = ece.generate(X_test)
compare(X_test, ece.get_aggregated_cfs(), constraints=constraints)
compare(X_test, cfs, constraints=constraints)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09620488322717623, 'fraction_numerical': 0.026671974522292995, 'mean_word_length': 5.514261019878997, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '596236', 'license': 'MIT', 'n_tokens_mistral': 2675, 'n_tokens_neox': 2504, 'n_words': 665, 'path': 'evaluation/case study.ipynb', 'repo_name': 'LoGosX/counterfactuals', 'size': '7536'}