<jupyter_start><jupyter_text>One Pixel Attack Tutorial Part 2 - ImageNet <NAME> March 22, 2018 [Part 1](1_one-pixel-attack-cifar10.ipynb)This time around, we will demonstrate the one pixel attack on ImageNet. The images are 224x224 px in size, with 1000 classes to pick from.  Imports Ensure that you have `numpy`, `pandas`, `scipy`, `matplotlib`, `tensorflow-gpu`, and `keras` installed.<jupyter_code># Python Libraries
%matplotlib inline
import pickle
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import keras
from keras import backend as K
from keras.preprocessing.image import load_img, img_to_array
from keras.applications.imagenet_utils import decode_predictions, preprocess_input

# Helper functions
import helper
from attack import PixelAttacker

matplotlib.style.use('ggplot')
np.random.seed(100)<jupyter_output>/home/hyper/Documents/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.<jupyter_text>Load Dataset  Download ImageNet Images To begin, we will demonstrate an attack on one image. The code below will download images from the specified URLs and save to the `data` directory. You may also add more ImageNet URLs, which can be obtained from http://image-net.org/download-imageurls.<jupyter_code>with open('data/imagenet_classes.pkl', 'rb') as f:
    class_names = pickle.load(f)
word_to_class = {w:i for i,w in enumerate(class_names)}
# Image URLs given by ImageNet
urls = [
    'http://farm4.static.flickr.com/3154/2585091536_78f528fdee.jpg',
]

# Where each file will be saved to
filenames = [
    'data/balloon.jpg'
]

# The labels of each corresponding image
labels = [
    word_to_class['balloon']
]
# Download all image files
for url, filename in zip(urls, filenames):
    print('Downloading', filename)
    helper.download_from_url(url, filename)<jupyter_output>0.00B [00:00, ?B/s]<jupyter_text>Let's see what the image looks like. The code below will load an image of a balloon into memory and convert it into a `numpy` array.<jupyter_code>originals = [load_img(filename, target_size=(224, 224)) for filename in filenames] # Load an image in PIL format
images = np.array([img_to_array(original) for original in originals]) # Convert the PIL image to a numpy array
helper.plot_image(images[0])<jupyter_output><empty_output><jupyter_text>Load Model Keras conveniently supplies pretrained weights for several ImageNet models. We will use `MobileNet` (~30 MB). Running the cell below will initiate a download of the model weights if it is not cached already.<jupyter_code>model = keras.applications.MobileNet()<jupyter_output><empty_output><jupyter_text>Now, predict the image's class.<jupyter_code>processed_images = preprocess_input(images.copy()) # Prepare the image for the model
predictions = model.predict(processed_images) # Get the predicted probabilities for each class
label = decode_predictions(predictions) # Convert the probabilities to class labels

label<jupyter_output><empty_output><jupyter_text>We see that the model correctly predicts `balloon` with 99.7% confidence.  Perturb Image Like before, we can perturb an image by changing the color of one or more pixels. The cell below will modify the center pixel to be yellow. As the images are much larger in size than in Cifar10, this change will be barely noticable.<jupyter_code>pixel = np.array([112, 112, 255, 255, 0]) # pixel = x,y,r,g,b
image_perturbed = helper.perturb_image(pixel, images)
helper.plot_image(image_perturbed[0])<jupyter_output><empty_output><jupyter_text>Let's see what the model now predicts.<jupyter_code>preprocessed_perturbed = preprocess_input(image_perturbed.copy())
predictions = model.predict(preprocessed_perturbed)
label = decode_predictions(predictions)

label<jupyter_output><empty_output><jupyter_text>The model still predicts balloon, but the perturbation did manage to change the confidence slightly.  Attack Finally, we use differential evolution to conduct the attack to see what percentage of images we can successfully cause the model to misclassify in a reasonable amount of time.It is recommended to use Keras with a GPU enabled. The code below can be used to check (if using TensorFlow).<jupyter_code># Should output /device:GPU:0
K.tensorflow_backend._get_available_gpus()
models = [model]

test = processed_images, np.array([labels])

attacker = PixelAttacker(models, test, class_names, dimensions=(224, 224))
result = attacker.attack(0, model, maxiter=20, verbose=True)<jupyter_output>Confidence: 0.99635327
Confidence: 0.99635327
Confidence: 0.996336
Confidence: 0.9962914
Confidence: 0.9962914
Confidence: 0.99625033
Confidence: 0.99625033
Confidence: 0.99625033
Confidence: 0.99625033
Confidence: 0.9962244
Confidence: 0.9962244
Confidence: 0.9962244
Confidence: 0.995893
Confidence: 0.995893
Confidence: 0.995893
Confidence: 0.9958299
Confidence: 0.9958299
Confidence: 0.9955239
Confidence: 0.9955239
Confidence: 0.9955239<jupyter_text>In this instance, the attack was not successful. The model still correctly predicts balloon. However, the above output shows that the attack does indeed lower the confidence of the model. If we have a larger dataset, we can see more easily how many of the images in the dataset can be attacked successfully.  TODO: Attack with more images<jupyter_code># predictions = model.predict(attack_image)
# label = decode_predictions(predictions)
# label<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08143839238498149, 'fraction_numerical': 0.04565485633703508, 'mean_word_length': 6.013597033374537, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '160805', 'license': 'MIT', 'n_tokens_mistral': 1926, 'n_tokens_neox': 1720, 'n_words': 606, 'path': '2_one-pixel-attack-imagenet.ipynb', 'repo_name': 'carlospolop-forks/one-pixel-attack-keras', 'size': '5681'}