<jupyter_start><jupyter_text>> Code to accompany **Chapter 6: Methods for Generating Adversarial Perturbation**  ResNet50 Foolbox Saliency AttackThe saliency attack focusses on permutating pixels in the image that are most salient to the adversarial misclassification or adversarial target classification.The Jacobian Saliency Map Approach (JSMA) is described in greater detail in chapter 6. The paper describing this approach is [N<NAME> et al., “The Limitations of Deep Learning in Adversarial Settings,” 1st IEEE European Symposium on Security & Privacy (2016)](https://arxiv.org/abs/1511.07528). To begin, import some libararies and dependencies.<jupyter_code># Import the model
import tensorflow as tf
from tensorflow import keras
from keras.applications.resnet50 import ResNet50
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Import helper function
import sys
sys.path.append('..')
from strengtheningdnns.image_utils import image_from_file<jupyter_output>Using TensorFlow backend.<jupyter_text>Select our non adversarial image and see what ResNet50 classifies it as<jupyter_code>original_image_path = '../images/koala.jpg'
x = image_from_file(original_image_path, [224,224])<jupyter_output>(?, ?, ?)
Re-scaling image to  [224, 224]<jupyter_text>We'll re-scale it and check that it is correctly classified. Get the ResNet50 model<jupyter_code>model = ResNet50(weights='imagenet', include_top=True)
from keras.applications.resnet50 import preprocess_input
from keras.applications.resnet50 import decode_predictions

x_preds = model.predict(np.expand_dims(preprocess_input(x), 0))
y = np.argmax(x_preds)
y_name = decode_predictions(x_preds, top=1)[0][0][1]
print("Prediction for image: ", y_name)

plt.imshow(x)<jupyter_output>Prediction for image:  koala<jupyter_text>Create the equivalent foolbox model from the Keras one The Foolbox model is able to perform pre-processing on the image to make it suitable to ResNet50. This involves normalising the data around the ImageNet mean RGB values on which the classifier was initially trained. The ```preprocessing``` variable defines the means for this pre-processing step. The equivalent normalisation is done in ```keras.applications.resnet50.preprocess_input``` - the function that we have called previously to prepare input for ResNet50. If you are interested in understanding this pre-processing step in greater detail, take a look at[the ResNet50Preprocessing.ipynb file on Github](https://github.com/katywarr/strengthening-dnns/blob/master/chapter04/resnet50_preprocessing.ipynb) or, if you are running the notebooks, [run the ResNet50Preprocessing.ipynb notebook](../chapter04/resnet50_preprocessing.ipynb).<jupyter_code>preprocessing = (np.array([103.939, 116.779, 123.68]), 1)<jupyter_output><empty_output><jupyter_text>ResNet50 was trained on images with the channels ordered BGR, rather than RGB. This step switches the channels of the image data to BGR. This is another step in ```keras.applications.resnet50.preprocess_input.```<jupyter_code>x_bgr = x[..., ::-1]<jupyter_output><empty_output><jupyter_text>Create the foolbox model passing the information required for pre-processing.<jupyter_code>import foolbox

# Temporary using Keras Model to get around bug that looks like: 
# https://github.com/bethgelab/foolbox/issues/164 
fmodel = foolbox.models.KerasModel(model, bounds=(0, 255), 
                                   preprocessing=preprocessing)
#fmodel = foolbox.models.TensorFlowModel.from_keras(model, 
#                                                   bounds=(0, 255), 
#                                                   preprocessing=preprocessing)<jupyter_output>WARNING:tensorflow:From C:\Users\katyw\Anaconda3\envs\strengthening-dnns\lib\site-packages\tensorflow\python\ops\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.<jupyter_text>Define the attack specificity[foolbox.criteria](https://foolbox.readthedocs.io/en/latest/modules/criteria.html) defines the attach specificity (targeted versus untargeted) and the probability threshold required.<jupyter_code>attack_criterion = foolbox.criteria.Misclassification()<jupyter_output><empty_output><jupyter_text>Define the attack method<jupyter_code>attack = foolbox.attacks.SaliencyMapAttack(fmodel, criterion=attack_criterion)<jupyter_output><empty_output><jupyter_text>Run the attack<jupyter_code>adversarial = attack(input_or_adv = x_bgr, label = y, unpack = False)
x_adv = adversarial.image[..., ::-1]

x_adv_preds = model.predict(preprocess_input(x_adv[np.newaxis].copy()))
y_adv = np.argmax(x_adv_preds)
y_adv_name = decode_predictions(x_adv_preds, top=1)[0][0][1]
print(print("Prediction for image: ", y_adv_name))

plt.figure()

# Plot the original image
plt.subplot(1, 3, 1)
plt.title(y_name)
plt.imshow(x)  
plt.axis('off')

# Plot the adversarial image
plt.subplot(1, 3, 2)
plt.title(y_adv_name)
plt.imshow(x_adv)  
plt.axis('off')

# Plot the difference
plt.subplot(1, 3, 3)
plt.title('Difference')
difference = x_adv - x
# Set differences that haven't changed to 255 so they don't show on the plot 
difference[difference == 0] = 255 
plt.imshow(abs(difference))
plt.xticks([])
plt.yticks([])

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.096186361074582, 'fraction_numerical': 0.024234454255119293, 'mean_word_length': 5.54054054054054, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '533721', 'license': 'MIT', 'n_tokens_mistral': 1803, 'n_tokens_neox': 1641, 'n_words': 462, 'path': 'chapter06/resnet50_foolbox_saliency.ipynb', 'repo_name': 'christian-westbrook/strengthening-dnns', 'size': '5332'}