<jupyter_start><jupyter_text>Contrastive Explanation Method Table of contents:* [Imports](imports)* [Performance of the CEM on the MNIST dataset](potcotmd)* [Training the MNIST classifier](training_mnist_classifier)* [Training the MNIST autoencoder](training_mnist_ae)* [MNIST Contrastive Explanation Method](cem_mnist)* [Evaluate MNIST Contrastive Explanation Method](eval_cem_mnist)* [Performance of the CEM on the Fashion MNIST dataset](potcotfd)* [Training the Fashion MNIST classifier](training_fashion_mnist_classifier)* [Training the Fashion MNIST autoencoder](training_fashion_mnist_ae)* [Fashion MNIST Contrastive Explanation Method](cem_fashion_mnist)* [Evaluate Fashion MNIST Contrastive Explanation Method](eval_cem_fashion_mnist)  Imports<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import time
import os

import torch

from cem.datasets.mnist import MNIST
from cem.datasets.fashion_mnist import FashionMNIST

from cem.models.cae_model import CAE
from cem.models.conv_model import CNN

from cem.train import train_ae, train_cnn

from cem.cem import ContrastiveExplanationMethod

# set random seeds for reproducability (although the CEM is fully determininstic)
torch.manual_seed(0)
np.random.seed(0)
# Change the following variables to either include or skip over optional
# evaluation metrics for the CNN and autoencoder
EVALUATE_CNN = True
EVALUATE_AUTOENCODER = True

# Change these variables to force (re-)download of the datasets
DOWNLOAD_MNIST = True
DOWNLOAD_FASHION_MNIST = True

# Device on which experiments are run
DEVICE = 'cpu'<jupyter_output><empty_output><jupyter_text>Performance of the CEM on the MNIST dataset<jupyter_code># specify which dataset we want to train on, batch size is only used
# in training the CNN and AE.
dataset = MNIST(batch_size=64, download=DOWNLOAD_MNIST)<jupyter_output><empty_output><jupyter_text>Training the MNIST classifier This section trains the classifier for which the pertinent positives and negatives will be constructed.If a load path is given and exists a model will be loaded from said file and will not train.<jupyter_code># Initialise the CNN object
cnn = CNN()

# The train function attempts to load the model from the specified path
# if no path is specified (by empty string) the model will be trained
# using the specified dataset.
train_cnn(cnn, dataset, load_path="./cem/models/saved_models/mnist-cnn.h5")
# This cell evaluates the performance of the CNN on the specified test set.
if EVALUATE_CNN:

    total_acc = 0
    total_batches = 0
    for step, (batch_inputs, batch_targets) in enumerate(dataset.test_loader):

        predictions = cnn(batch_inputs)
        acc = (predictions.argmax(1).cpu().numpy() == batch_targets.cpu().numpy()).sum()/(predictions.shape[0])
        total_batches += 1
        total_acc += acc

    print("classifier accuracy on test set: {}".format(total_acc / total_batches))<jupyter_output>classifier accuracy on test set: 0.971437101910828<jupyter_text>Training the MNIST autoencoder This section trains the autoencoder which will be used as regularizer for the data space which the perturbations are found in.<jupyter_code># Train or load autoencoder
cae = CAE()

# The train function attempts to load the model from the specified path
# if no path is specified (by empty string) the model will be trained
# using the specified dataset.
train_ae(cae, dataset, load_path="./cem/models/saved_models/mnist-cae-no-rs.h5")
# This cell evaluates the performance of the autoencoder by visualising a batch of 
# in- and corresponding output images.
if EVALUATE_AUTOENCODER:
    images, _ = dataset.get_batch()
    
    # rescale images since our ae is trained on non-zero-mean images
    images += 0.5

    # get sample outputs
    output = cae(images)
    # prep images for display
    images = images.numpy()
    output = output.detach().numpy()

    # plot the first ten input images and then reconstructed images
    fig, axes = plt.subplots(nrows=2, ncols=4, sharex=True, sharey=True, figsize=(12,4))

    # input images on top row, reconstructions on bottom
    for images, row in zip([images, output], axes):
        for img, ax in zip(images, row):
            ax.imshow(np.squeeze(img))
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)<jupyter_output><empty_output><jupyter_text>MNIST Contrastive Explanation Method<jupyter_code># In this cell the CEM object is initialised. For each parameter a short
# description is given.

# used in experiment: kappa 10, gamma 1.0, beta 0.1, c 0.1, lr 0.01

# to understand the impact of the following parameters please refer
# to the paper in this github.
kappa = 10
gamma = 1.0
beta = 0.1
c = 0.1
lr = 0.01

# number of iterations to train for
iterations = 100
# number of searches (equal to the number of times c gets updated)
n_searches = 3
# initial c value, in the original implementation the value for c 
# was initiated at 10 to prevent unnecessary iterations in which
# no weight updates are performed.
c_init = 10.0

# print information during training
verbose = True
print_every = 50

# the shape of a single sample, this is used to reshape input
# and output of the classifier and autoencoder
input_shape = (1, 28, 28)

# Select either PP for pertinent postive or PN for pertinent negative
mode = "PP"

CEM = ContrastiveExplanationMethod(
    cnn,
    cae,
    iterations=iterations,
    n_searches=n_searches,
    kappa=kappa,
    gamma=gamma,
    beta=beta,
    learning_rate=lr,
    c_init=c_init,
    c_converge=c,
    verbose=verbose,
    print_every=print_every
)<jupyter_output><empty_output><jupyter_text>Example usage of MNIST Contrastive Explanation Method<jupyter_code>sample, _ = dataset.get_sample()
delta = CEM.explain(sample, mode=mode)

if delta is None:
    print("no solution found...")
    raise

before = torch.argmax(cnn(sample.view(-1, *input_shape)))

print("original image classified as: {}".format(before))

if mode == "PP":
    after = np.argmax(cnn(sample-delta.view(-1, *input_shape)).detach().cpu()).item()
    print("pertinent positive classified as: {}".format(after))

elif mode == "PN":
    after = np.argmax(cnn(delta.view(-1, *input_shape)).detach().cpu()).item()
    print("image with pertinent negative added classified as: {}".format(after))

fig, (ax1, ax2, ax3) = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=True, figsize=(10, 5))

ax1.imshow(sample.squeeze(), cmap="gray")
ax1.title.set_text("original image")

ax2.imshow(delta.view(28, 28), cmap="gray")
ax2.title.set_text("image with perturbation")

if mode == "PP":
    ax3.imshow(sample.squeeze() - delta.view(28, 28), cmap="gray")
    ax3.title.set_text("pertinent positive")
elif mode == "PN":
    ax3.imshow(delta.view(28, 28) - sample.squeeze(), cmap="gray")
    ax3.title.set_text("pertinent negative")<jupyter_output>search: 0 iteration: 50 c: 10.00 loss: 138.56 solution: False
search: 0 iteration: 100 c: 10.00 loss: 138.56 solution: False
new best delta found, loss: 81.19239807128906
new best delta found, loss: 73.32435607910156
new best delta found, loss: 71.2755126953125
search: 1 iteration: 50 c: 100.00 loss: 97.40 solution: True
new best delta found, loss: 70.82515716552734
new best delta found, loss: 70.43706512451172
search: 1 iteration: 100 c: 100.00 loss: 70.58 solution: True
new best delta found, loss: 70.24156188964844
new best delta found, loss: 66.9354019165039
new best delta found, loss: 64.95694732666016
search: 2 iteration: 50 c: 50.05 loss: 111.60 solution: True
search: 2 iteration: 100 c: 50.05 loss: 385.80 solution: True
original image classified as: 1
pertinent positive classified as: 1<jupyter_text>Performance of the CEM on the FashionMNIST dataset<jupyter_code>dataset = FashionMNIST(batch_size=64, download=DOWNLOAD_FASHION_MNIST)<jupyter_output><empty_output><jupyter_text>Training the Fashion MNIST classifier This section trains the classifier for which the pertinent positives and negatives will be constructed.If a load path is given and exists a model will be loaded from said file and will not train.<jupyter_code>cnn = CNN()

train_cnn(cnn, dataset, load_path="./cem/models/saved_models/fashion-mnist-cnn.h5")
images, _ = dataset.get_batch()

output = cnn(images)

images = images.numpy()
output = output.detach().numpy()
# This cell evaluates the performance of the CNN on the specified test set.
if EVALUATE_CNN:

    total_acc = 0
    total_batches = 0
    for step, (batch_inputs, batch_targets) in enumerate(dataset.test_loader):

        predictions = cnn(batch_inputs)
        acc = (predictions.argmax(1).cpu().numpy() == batch_targets.cpu().numpy()).sum()/(predictions.shape[0])
        total_batches += 1
        total_acc += acc

    print("acc: {}".format(total_acc / total_batches))<jupyter_output>acc: 0.8946058917197452<jupyter_text>Training the Fashion MNIST autoencoder<jupyter_code># Train or load autoencoder
cae = CAE(device="cpu")

train_ae(cae, dataset, iterations=10, save_fn="fashion-mnist-cae", device="cpu", load_path="./cem/models/saved_models/fashion-mnist-cae.h5")
# This cell evaluates the performance of the autoencoder by visualising a batch of 
# in- and corresponding output images.
if EVALUATE_AUTOENCODER:
    images, _ = dataset.get_batch()
    images += 0.5 # rescale images

    # get sample outputs
    output = cae(images)
    # prep images for display
    images = images.numpy()
    output = output.detach().numpy()

    # plot the first ten input images and then reconstructed images
    fig, axes = plt.subplots(nrows=2, ncols=4, sharex=True, sharey=True, figsize=(12,4))

    # input images on top row, reconstructions on bottom
    for images, row in zip([images, output], axes):
        for img, ax in zip(images, row):
            ax.imshow(np.squeeze(img))
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)<jupyter_output><empty_output><jupyter_text>Fashion MNIST Contrastive Explanation Method<jupyter_code># In this cell the CEM object is initialised. For each parameter a short
# description is given.

# used in experiment: kappa 10, gamma 1.0, beta 0.1, c 0.1, lr 0.01

# to understand the impact of the following parameters please refer
# to the paper in this github.
kappa = 10
gamma = 1.0
beta = 0.1
c = 0.1
lr = 0.01

# number of iterations to train for
iterations = 100
# number of searches (equal to the number of times c gets updated)
n_searches = 3
# initial c value, in the original implementation the value for c 
# was initiated at 10 to prevent unnecessary iterations in which
# no weight updates are performed.
c_init = 10.0

# print information during training
verbose = True
print_every = 50

# the shape of a single sample, this is used to reshape input
# and output of the classifier and autoencoder
input_shape = (1, 28, 28)

# Select either PP for pertinent postive or PN for pertinent negative
mode = "PP"

CEM = ContrastiveExplanationMethod(
    cnn,
    cae,
    iterations=iterations,
    n_searches=n_searches,
    kappa=kappa,
    gamma=gamma,
    beta=beta,
    learning_rate=lr,
    c_init=c_init,
    c_converge=c,
    verbose=verbose,
    print_every=print_every
)<jupyter_output><empty_output><jupyter_text>Example usage of Fashion MNIST Contrastive Explanation Method<jupyter_code>sample, _ = dataset.get_sample()
delta = CEM.explain(sample, mode=mode)

if delta is None:
    print("no solution found...")
    raise

before = torch.argmax(cnn(sample.view(-1, *input_shape)))

print("original image classified as: {}".format(before))

if mode == "PP":
    after = np.argmax(cnn(sample-delta.view(-1, *input_shape)).detach().cpu()).item()
    print("pertinent positive classified as: {}".format(after))

elif mode == "PN":
    after = np.argmax(cnn(delta.view(-1, *input_shape)).detach().cpu()).item()
    print("image with pertinent negative added classified as: {}".format(after))

fig, (ax1, ax2, ax3) = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=True, figsize=(10, 5))

ax1.imshow(sample.squeeze(), cmap="gray")
ax1.title.set_text("original image")

ax2.imshow(delta.view(28, 28), cmap="gray")
ax2.title.set_text("image with perturbation")

if mode == "PP":
    ax3.imshow(sample.squeeze() - delta.view(28, 28), cmap="gray")
    ax3.title.set_text("pertinent positive")
elif mode == "PN":
    ax3.imshow(delta.view(28, 28) - sample.squeeze(), cmap="gray")
    ax3.title.set_text("pertinent negative")<jupyter_output>new best delta found, loss: 264.59136962890625
new best delta found, loss: 13.138795852661133
new best delta found, loss: 12.135979652404785
new best delta found, loss: 7.94731330871582
new best delta found, loss: 6.095447540283203
new best delta found, loss: 5.2656450271606445
new best delta found, loss: 3.6525628566741943
search: 0 iteration: 50 c: 10.00 loss: 21.06 solution: True
new best delta found, loss: 1.5561892986297607
new best delta found, loss: 1.2803919315338135
search: 0 iteration: 100 c: 10.00 loss: 70.50 solution: True
search: 1 iteration: 50 c: 5.05 loss: 4.55 solution: True
new best delta found, loss: 1.0781699419021606
search: 1 iteration: 100 c: 5.05 loss: 55.91 solution: True
search: 2 iteration: 50 c: 2.57 loss: 14.05 solution: True
search: 2 iteration: 100 c: 2.57 loss: 28.52 solution: True
original image classified as: 3
pertinent positive classified as: 3
{'fraction_non_alphanumeric': 0.10063694267515924, 'fraction_numerical': 0.048182840014986884, 'mean_word_length': 4.655084745762712, 'pattern_counts': {'":': 8, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '586648', 'license': 'MIT', 'n_tokens_mistral': 4841, 'n_tokens_neox': 4266, 'n_words': 1412, 'path': 'usage-examples.ipynb', 'repo_name': 'davidvos/contrastive-explanation-method', 'size': '13345'}