Adversarial perturbations cause a shift in the salient features of an image, which may result in a misclassification. We demonstrate that gradient-based saliency approaches are unable to capture this shift, and develop a new defense which detects adversarial examples based on learnt saliency models instead. We study two approaches: a CNN trained to distinguish between natural and adversarial images using the saliency masks produced by our learnt saliency model, and a CNN trained on the salient pixels themselves as its input. On MNIST, CIFAR-10 and ASSIRA, our defenses are able to detect various adversarial attacks, including strong attacks such as C&W and DeepFool, contrary to gradient-based saliency and detectors which rely on the input image. The latter are unable to detect adversarial images when the L_2- and L_infinity- norms of the perturbations are too small. Lastly, we find that the salient pixel based detector improves on saliency map based detectors as it is more robust to white-box attacks. Adversarial examples highlight a crucial difference between human vision and computer image processing. Often computers fail to understand the relevant characteristics of an image for classification (Ribeiro et al., 2016) or fail to generalize locally, i.e., misclassify examples close to the training data (Szegedy et al., 2013) . Attacks exploit this property by altering pixels the classifier heavily relies on -pixels which are irrelevant to humans for object recognition. As a consequence, adversarial perturbations fool classifiers while the correct class remains clear to humans. Saliency maps identify the pixels an image classifier uses for its prediction; as such, they can be used as a tool to understand why a classifier is fooled. Building on this concept, researchers have shown qualitatively that adversarial perturbations cause a shift in the saliency of classifiers (Fong & Vedaldi, 2017; Gu & Tresp, 2019) . Figure 1 shows examples of a natural image and corresponding adversarial images, each above their respective saliency maps. The saliency maps corresponding to adversarial images show perceptible differences to that of the original image, even though adversarial images themselves often seem unperturbed. For the original image, the saliency map shows that the classifier focuses on the four (and a couple of random pixels on the left). We observe that for the adversarial images, the classifier starts focusing more on irrelevant aspects of the left side of the image. There is ample research into different techniques for finding saliency maps (see e.g. Zeiler & Fergus, 2014; Springenberg et al., 2014; Bach et al., 2015; Ribeiro et al., 2016; Shrikumar et al., 2017; Selvaraju et al., 2017; Zintgraf et al., 2017; Fong & Vedaldi, 2017) . However, not all saliency maps are equally informative (Fong & Vedaldi, 2017) . For example, the Jacobian 1 can be used to determine the saliency of a pixel in the classification of the image (Papernot et al., 2016b; Zhang et al., 2018) . As the Jacobian is often used to generate adversarial examples, intuitively, we expect that it can be used effectively to detect adversarial perturbations. Zhang et al. (2018) propose a defense to this effect: they determine whether an input is adversarial, given the Jacobian-based The top is the input image and the bottom shows the corresponding saliency map. In the second row, lighter colours correspond to higher saliency (black corresponds to a saliency of 0, the lowest possible value). The classifier predicts (from left to right) the images as: 4, 9, 9 , 8, 9, 9. Note the stark difference between the saliency masks of the original image and those of the adversarial examples. saliency map concatenated with the image. However, as shown qualitatively by Gu & Tresp (2019) , gradients are not always able to capture differences between adversarial images and natural images (for an example see Figures 7 and 8 in Appendix D). 2 Here we inspect the proposed Jacobian-based approach and show that only the concatenated input affects the technique's performance in detecting adversarial examples, with the Jacobian having no effect. While gradients may not be informative for detection, saliency should be an effective tool for detecting adversarial images. In our analysis, we use more powerful model-based saliency techniques and show that the magnitude of the shift of the saliency map due to adversarial perturbations often exceeds the L 2 distance between the saliency maps of different natural images. Building on this result, we consider two different possible effects adversarial perturbations might have on the classifier: 1. They might cause the classifier to focus on the wrong pixel locations 2. They might change the pixel values of salient pixels Based on these hypotheses, we employ two CNN classifier architectures to detect adversarial images. Claim (1) can be captured by shifts in saliency maps, as previously considered by Fong & Vedaldi (2017) . In this work, we extend on their analysis 3 by proving the defensive capability of our model-based saliency against difficult black-box attacks, such as C&W and DeepFool 4 , as well as white-box adversarial attacks. By considering claim (2), we demonstrate that incorporating pixel values improves the performance of the classifier when shifts in saliency maps do not suffice to capture adversarial perturbations. We also show that our salient-pixel based defense generalizes well (detecting stronger attacks when trained on weaker attacks) and is more robust than the saliency map defense against white-box attacks. Lastly, we demonstrate that saliency can be used to detect adversarial examples generated by small perturbations, contrary to other defenses, which exhibit threshold behavior: i.e., when the adversarial perturbation is too small, other defenses (specifically Gong et al., 2017; Zhang et al., 2018) are unable to detect the adversarial images. In our analysis, we ascertain that the saliency maps of adversarial images differ from those of natural images. Further, we show that salient pixel based defenses perform better than a saliency map defense. When trained on a single black-box attack, our method is able to detect adversarial perturbations generated by different and stronger attacks. We show that gradients are unable to capture shifts in saliency due to adversarial perturbations and present an alternative adversarial defense using learnt saliency models that is effective against both black-box and white-box attacks. Building on the work of Gong et al. (2017) , we further establish the notion of threshold behavior, showing that the trend depends on the L 2 and L ∞ -norms of the perturbations and therefore also prevails when using other methods (JSD) and across different attacks. Future work could further investigate the performance of the defense in different applications. For example, as our method runs in real-time, it could be used to detect adversarial perturbations in video to counter recent attacks Jiang et al., 2019) . A ARCHITECTURES, HYPER-PARAMETERS AND DATA Figure 3 : ASSIRA, CIFAR-10, and MNIST image classifier architecture and hyper-parameters. The first entry corresponds to the first layer, and the table proceeds chronologically until the last layer. Parameters f, k, p, s and n represent the number of filters, kernel size, pooling size, stride, number of filters, respectively. If stride is omitted, it is set to 1. All classifiers have a final softmax activation.
We show that gradients are unable to capture shifts in saliency due to adversarial perturbations and present an alternative adversarial defense using learnt saliency models that is effective against both black-box and white-box attacks.
{"src": "scitldr-AIC_train", "paper_id": "HJe5_6VKwS"}