<jupyter_start><jupyter_text>Getting started with Perceptual Adversarial RobustnessThis notebook contains examples of how to load a pretrained model, measure LPIPS distance, and construct perceptual and non-perceptual attacks.If you are running this notebook in Google Colab, it is recommended to use a GPU. You can enable GPU acceleration by going to **Runtime** > **Change runtime type** and selecting **GPU** from the dropdown. First, make sure you have installed the `perceptual_advex` package, either from GitHub or PyPI:<jupyter_code>try:
    import perceptual_advex
except ImportError:
    !pip install perceptual-advex<jupyter_output><empty_output><jupyter_text>Loading a pretrained modelFirst, let's load the CIFAR-10 dataset along with a pretrained model. The following code will download a model checkpoint and load it, but you can change the `checkpoint_name` parameter to load a different checkpoint. The checkpoint we're downloading here is trained against $L_2$ adversarial attacks with bound $\epsilon = 1$.<jupyter_code>import subprocess
import os

if not os.path.exists('data/checkpoints/cifar_pgd_l2_1.pt'):
    !mkdir -p data/checkpoints
    !curl -o data/checkpoints/cifar_pgd_l2_1.pt https://perceptual-advex.s3.us-east-2.amazonaws.com/cifar_pgd_l2_1_cpu.pt
from perceptual_advex.utilities import get_dataset_model

dataset, model = get_dataset_model(
    dataset='cifar',
    arch='resnet50',
    checkpoint_fname='data/checkpoints/cifar_pgd_l2_1.pt',
)<jupyter_output><empty_output><jupyter_text>If you want to experiment with ImageNet-100 instead, just change the above to    dataset, model = get_dataset_model(        dataset='imagenet100',         Change this to where ImageNet is downloaded.        dataset_path='/path/to/imagenet',        arch='resnet50',         Change this to a pretrained checkpoint path.        checkpoint_fname='/path/to/checkpoint',    )  Viewing images in the datasetNow that we have a dataset and model loaded, we can view some images in the dataset.<jupyter_code>import torchvision
import numpy as np
import matplotlib.pyplot as plt

# We'll use this helper function to show images in the Jupyter notebook.
%matplotlib inline
def show(img):
    if len(img.size()) == 4:
        img = torchvision.utils.make_grid(img, nrow=10, padding=0)
    npimg = img.detach().cpu().numpy()
    plt.figure(figsize=(18,16), dpi=80, facecolor='w', edgecolor='k')
    plt.imshow(np.transpose(npimg, (1,2,0)), interpolation='nearest')
import torch

# Create a validation set loader.
batch_size = 10
_, val_loader = dataset.make_loaders(1, batch_size, only_val=True)

# Get a batch from the validation set.
inputs, labels = next(iter(val_loader))

# If we have a GPU, let's convert everything to CUDA so it's quicker.
if torch.cuda.is_available():
    inputs = inputs.cuda()
    labels = labels.cuda()
    model.cuda()

# Show the batch!
show(inputs)<jupyter_output><empty_output><jupyter_text>We can also test the accuracy of the model on this set of inputs by comparing the model output to the ground-truth labels.<jupyter_code>pred_labels = model(inputs).argmax(1)
print('Natural accuracy is', (labels == pred_labels).float().mean().item())<jupyter_output><empty_output><jupyter_text>If the natural accuracy is very low on this batch of images, you might want to load a new set by re-running the two cells above.  Generating perceptual adversarial examplesNext, let's generate some perceptual adversarial examples using Lagrange perceptual attack (LPA) with AlexNet bound $\epsilon = 0.5$. Other perceptual attacks (PPGD and Fast-LPA) are also found in the `perceptual_advex.perceptual_attacks` module, and they mostly share the same options.<jupyter_code>from perceptual_advex.perceptual_attacks import LagrangePerceptualAttack

attack = LagrangePerceptualAttack(
    model,
    num_iterations=10,
    # The LPIPS distance bound on the adversarial examples.
    bound=0.5,
    # The model to use for calculate LPIPS; here we use AlexNet.
    # You can also use 'self' to perform a self-bounded attack.
    lpips_model='alexnet_cifar',
)
adv_inputs = attack(inputs, labels)

# Show the adversarial examples.
show(adv_inputs)

# Show the magnified difference between the adversarial examples and unperturbed inputs.
show((adv_inputs - inputs) * 5 + 0.5)<jupyter_output><empty_output><jupyter_text>Note that while the perturbations are sometimes large, the adversarial examples are still recognizable as the original image and do not appear too different perceptually.We can calculate the accuracy of the classifier on the adversarial examples:<jupyter_code>adv_pred_labels = model(adv_inputs).argmax(1)
print('Adversarial accuracy is', (labels == adv_pred_labels).float().mean().item())<jupyter_output><empty_output><jupyter_text>Even though this network has been trained to be robust to $L_2$ perturbations, there are still imperceptible perturbations found using LPA that fool it almost every time!  Calculating LPIPS distanceNext, let's calculate the LPIPS distance between the adversarial examples we generated and the original inputs:<jupyter_code>from perceptual_advex.distances import LPIPSDistance
from perceptual_advex.perceptual_attacks import get_lpips_model

# LPIPS is based on the activations of a classifier, so we need to first
# load the classifier we'll use.
lpips_model = get_lpips_model('alexnet_cifar')
if torch.cuda.is_available():
    lpips_model.cuda()

# Now we can define a distance based on the model we loaded.
# We could also do LPIPSDistance(model) for self-bounded LPIPS.
lpips_distance = LPIPSDistance(lpips_model)

# Finally, let's calculate the distance between the inputs and adversarial examples.
print(lpips_distance(inputs, adv_inputs))<jupyter_output><empty_output><jupyter_text>Note that all the distances are within the bound of 0.5! At this bound, the adversarial perturbations should all have a similar level of perceptibility to the human eye.Other distance measures between images are also defined in the `perceptual_advex.distances` package, including $L_\infty$, $L_2$, and SSIM.  Generating non-perceptual adversarial examplesThe `perceptual_advex` package also includes code to perform attacks based on other, narrower threat models like $L_\infty$ or $L_2$ distance and spatial transformations. The non-perceptual attacks are all in the `perceptual_advex.attacks` module. First, let's try an $L_2$ attack:<jupyter_code>from perceptual_advex.attacks import L2Attack

attack = L2Attack(
    model,
    'cifar',
    # The bound is divided by 255, so this is equivalent to eps=1.
    bound=255,
)
l2_adv_inputs = attack(inputs, labels)

show(l2_adv_inputs)
show((l2_adv_inputs - inputs) * 5 + 0.5)

l2_adv_pred_labels = model(l2_adv_inputs).argmax(1)
print('L2 adversarial accuracy is', (labels == l2_adv_pred_labels).float().mean().item())<jupyter_output><empty_output><jupyter_text>Here's an example of a spatial attack (StAdv):<jupyter_code>from perceptual_advex.attacks import StAdvAttack

attack = StAdvAttack(
    model,
    bound=0.02,
)
spatial_adv_inputs = attack(inputs, labels)

show(spatial_adv_inputs)
show((spatial_adv_inputs - inputs) * 5 + 0.5)

spatial_adv_pred_labels = model(spatial_adv_inputs).argmax(1)
print('Spatial adversarial accuracy is', (labels == spatial_adv_pred_labels).float().mean().item())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08342420937840785, 'fraction_numerical': 0.011450381679389313, 'mean_word_length': 5.45862676056338, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '17112', 'license': 'MIT', 'n_tokens_mistral': 2341, 'n_tokens_neox': 2161, 'n_words': 752, 'path': 'getting_started.ipynb', 'repo_name': 'kevinbro96/perceptual-advex', 'size': '7336'}