<jupyter_start><jupyter_text>In the earlier notebooks ([`01_1_simple_pgd.ipynb`](https://github.com/dipanjanS/adversarial-learning-robustness/blob/main/tutorials/attacks/01_1_simple_pgd.ipynb) and [`01_2_simple_pgd.ipynb`](https://github.com/dipanjanS/adversarial-learning-robustness/blob/main/tutorials/attacks/01_2_simple_pgd.ipynb)) we saw how to launch a simple attack. But what if we wanted to extend this idea and wanted to fool our model into predicting a class of our choice. In this notebook, we will learn how to fool a pre-trained ResNet50 model by launching a targeted PGD-based attack (Projected Gradient Descent).     Initial setup and imports<jupyter_code>from tensorflow.keras.applications.imagenet_utils import preprocess_input
from tensorflow.keras.applications.resnet50 import decode_predictions
import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import json
import cv2

print("TensorFlow version: ",tf.__version__)<jupyter_output>TensorFlow version:  2.3.0<jupyter_text>We first download a sample image which we will perturb in order to fool our pre-trained image classifier. We also download a dictionary that contains the mapping between the Imagenet dataset class labels and their ids (the ResNet50 model is pre-trained on ImageNet).<jupyter_code>!wget -q https://adversarial-ml-tutorial.org/introduction/pig.jpg
!wget -q https://raw.githubusercontent.com/USCDataScience/dl4j-kerasimport-examples/master/dl4j-import-example/data/imagenet_class_index.json<jupyter_output><empty_output><jupyter_text>A few early constants<jupyter_code>EPS = 2./255
IMAGENET_LABELS = "imagenet_class_index.json"

with open(IMAGENET_LABELS) as f:
    IMAGENET_CLASSES = {int(i):x[1] for i,x in json.load(f).items()}<jupyter_output><empty_output><jupyter_text>Utility functionsNext we define a couple of utility functions that we will be using throughout the course of this notebook.<jupyter_code># Function to load up an image from a local filepath and
# display it
def show_image(image_path):
    image_pixels = plt.imread(image_path)
    plt.imshow(image_pixels)
    plt.show()
    return image_pixels

# Function to preprocess an image for performing inference
def preprocess_image(image_pixels, preprocess=False):
    if preprocess:
        image_pixels = preprocess_input(image_pixels)
    image_pixels = cv2.resize(image_pixels, (224, 224))
    image_pixels = np.expand_dims(image_pixels, axis=0)

    return image_pixels

# Clipping utility to project delta back to our favorable pixel
# range ([0 ,1])
def clip_eps(delta_tensor):
    return tf.clip_by_value(delta_tensor, 
                            clip_value_min=-EPS, 
                            clip_value_max=EPS)

# Parse the label
def get_label(preds):
    print(IMAGENET_CLASSES[preds.argmax()])<jupyter_output><empty_output><jupyter_text>Loading our first image and pre-trained model<jupyter_code>sample_image = show_image("pig.jpg")
preprocessed_image = preprocess_image(sample_image, preprocess=True)
resnet50 = tf.keras.applications.ResNet50(weights="imagenet")<jupyter_output><empty_output><jupyter_text>Running inference with a pre-trained model<jupyter_code>preds = resnet50.predict(preprocessed_image)
print("Predicted:", decode_predictions(preds, top=3)[0])
print("Class idx:", preds.argmax())<jupyter_output>Predicted: [('n02395406', 'hog', 0.99968374), ('n02396427', 'wild_boar', 0.00031595054), ('n03935335', 'piggy_bank', 9.8273716e-08)]
Class idx: 341<jupyter_text>Let's now write our utility functions for launching a targeted PGD-based attack.  Utility functions for adversarial attacksIn this attack we will use PGD to simply maximize the loss for the given (true) class and at the same time _minimize the loss for the target class_ such that the visual semantics of our input image does not get hampered.<jupyter_code>def generate_adversaries_targeted(image_tensor, delta, model,
    true_index, target_index):
    # Loss and optimizer
    scc_loss = tf.keras.losses.SparseCategoricalCrossentropy()
    optimizer = tf.keras.optimizers.Adam(learning_rate=5e-3)

    for t in range(350):
        plt.show()
        with tf.GradientTape() as tape:
            tape.watch(delta)
            inp = preprocess_input(image_tensor + delta)
            predictions = model(inp, training=False)
            loss = (- scc_loss(tf.convert_to_tensor([true_index]), predictions) + 
	                    scc_loss(tf.convert_to_tensor([target_index]), predictions))
            if t % 20 == 0:
                print(t, loss.numpy())
            
        # Get the gradients
        gradients = tape.gradient(loss, delta)
        
        # Update the weights
        optimizer.apply_gradients([(gradients, delta)])

        # Project the delta back (l-infinite norm)
        delta.assign_add(clip_eps(delta))

    return delta

def perturb_image(image_path, true, target, model):
    # Load and preprocess image but a but without any `preprocess_input`
    sample_image = show_image(image_path)
    preprocessed_image = preprocess_image(sample_image)

    # Generate predictions before any adversaries
    unsafe_preds = model.predict(preprocess_input(preprocessed_image))
    print("Prediction before adv.:", decode_predictions(unsafe_preds, top=3)[0])

    # Initialize the perturbation quantity
    image_tensor = tf.constant(preprocessed_image, dtype=tf.float32)
    delta = tf.Variable(tf.zeros_like(image_tensor), trainable=True)

    # Get the learned delta and display it
    delta_tensor = generate_adversaries_targeted(image_tensor, delta, model, true, target)
    plt.imshow(50*delta_tensor.numpy().squeeze()+0.5)
    plt.show()

    # See if the image changes
    plt.imshow((image_tensor + delta_tensor).numpy().squeeze()/255)
    plt.show()

    # Generate prediction
    perturbed_image = preprocess_input(image_tensor + delta_tensor)
    preds = model.predict(perturbed_image)
    print("Prediction after adv.:", decode_predictions(preds, top=3)[0])<jupyter_output><empty_output><jupyter_text>**Note** that the hyperparameters like learning rate, number of epochs were determined after empirical experiments and also by referring to [this tremendous tutorial](https://adversarial-ml-tutorial.org/introduction/).   Launching a targeted attack!Let's try to make our model believe that the `hog` is actually a `Lakeland_terrier`. Recall that the class index of hog in the ImageNet dataset is 341 and for `Lakeland_terrier` it's 189.<jupyter_code>perturb_image("pig.jpg", 341, 189, resnet50)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09143819880533006, 'fraction_numerical': 0.021442793689692142, 'mean_word_length': 4.693112467306015, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '105327', 'license': 'Apache-2.0', 'n_tokens_mistral': 2178, 'n_tokens_neox': 2032, 'n_words': 531, 'path': 'tutorials/attacks/02_1_targeted_pgd.ipynb', 'repo_name': 'dipanjanS/adversarial-learning-robustness', 'size': '6529'}