Injecting adversarial examples during training, known as adversarial training, can improve robustness against one-step attacks, but not for unknown iterative attacks. To address this challenge, we first show iteratively generated adversarial images easily transfer between networks trained with the same strategy. Inspired by this observation, we propose cascade adversarial training, which transfers the knowledge of the end of adversarial training. We train a network from scratch by injecting iteratively generated adversarial images crafted from already defended networks in addition to one-step adversarial images from the network being trained. We also propose to utilize embedding space for both classification and low-level (pixel-level) similarity learning to ignore unknown pixel level perturbation. During training, we inject adversarial images without replacing their corresponding clean images and penalize the distance between the two embeddings (clean and adversarial). Experimental show that cascade adversarial training together with our proposed low-level similarity learning efficiently enhances the robustness against iterative attacks, but at the expense of decreased robustness against one-step attacks. We show that combining those two techniques can also improve robustness under the worst case black box attack scenario. Injecting adversarial examples during training (adversarial training), BID1 BID3 increases the robustness of a network against adversarial attacks. The networks trained with one-step methods have shown noticeable robustness against onestep attacks, but, limited robustness against iterative attacks at test time. To address this challenge, we have made the following contributions:Cascade adversarial training: We first show that iteratively generated adversarial images transfer well between networks when the source and the target networks are trained with the same training method. Inspired by this observation, we propose cascade adversarial training which transfers the knowledge of the end of adversarial training. In particular, we train a network by injecting iter FGSM images (section 2.1) crafted from an already defended network (a network trained with adversarial training) in addition to the one-step adversarial images crafted from the network being trained. The concept of using already trained networks for adversarial training is also introduced in BID9. In their work, purely trained networks are used as another source networks to generate one-step adversarial examples for training. On the contrary, our cascade adversarial training uses already defended network for iter FGSM images generation. Low level similarity learning: We advance the previous data augmentation approach by adding additional regularization in deep features to encourage a network to be insensitive to adversarial perturbation. In particular, we inject adversarial images in the mini batch without replacing their corresponding clean images and penalize distance between embeddings from the clean and the adversarial examples. There are past examples of using embedding space for learning similarity of high level features like face similarity between two different images BID8 BID7 ). Instead, we use the embedding space for learning similarity of the pixel level differences between two similar images. The intuition of using this regularization is that small difference on input should not drastically change the high level feature representation. We train ResNet models BID2 on MNIST BID6 and CIFAR10 dataset BID4 ) using the proposed adversarial training. We first show low level similarity learning improves robustness of the network against adversarial images generated by one-step and iterative methods compared to the prior work. We show that modifying the weight of the distance measure in the loss function can help control trade-off between accuracies for the clean and adversarial examples. Together with cascade adversarial training and low-level similarity learning, we achieve accuracy increase against unknown iterative attacks, but at the expense of decreased accuracy for one-step attacks. We also show our cascade adversarial training and low level similarity learning provide much better robustness against black box attack. One-step fast gradient sign method (FGSM), referred to as "step FGSM", generates adversarial image X adv by adding sign of the gradients w.r.t. the clean image X multiplied by ∈ as shown below BID1: DISPLAYFORM0 One-step target class method generates X adv by subtracting sign of the gradients computed on a target false label as follows: DISPLAYFORM1 We use least likely class y LL as a target class and refer this method as "step ll".Basic iterative method, referred to as "iter FGSM", applies FGSM with small α multiple times. DISPLAYFORM2 We use α = 1, number of iterations N to be min (+ 4, 1.25). Clip X, is elementwise clipping function where the input is clipped to the range [max(0, X −), min(255, X +)].Iterative least-likely class method, referred to as "iter ll", is to apply "step ll" with small α multiple times. DISPLAYFORM3 Carlini and Wagner attack BID0 ) referred to as "CW" solves an optimization problem which minimizes both an objective function f (such that attack is success if and only if f (X adv) < 0) and a distance measure between X adv and X.Black box attack is performed by testing accuracy on a target network with the adversarial images crafted from a source network different from the target network. Lower accuracy means successful black-box attack. When we use the same network for both target and source network, we call this as white-box attack. Adversarial training: is a form of data augmentation where it injects adversarial examples during training. In this method, k examples are taken from the mini batch B (size of m) and the adversarial examples are generated with one of step method. The k adversarial examples replaces the corresponding clean examples when making mini batch. Below we refer this adversarial training method as "Kurakin's". Figure 1: Correlation between adversarial noises from different networks for each. Shaded region shows ± 0.1 standard deviation of each line. Ensemble adversarial training BID9: is essentially the same with the adversarial training, but uses several pre-trained vanilla networks to generate one-step adversarial examples for training. Below we refer this adversarial training method as "Ensemble".3 PROPOSED APPROACH We first show transferability between purely trained networks and adversarially trained networks under black box attack. We use ResNet BID2 ) models for CIFAR10 classification. We first train 20-layer ResNets with different methods (standard training, adversarial training) and use those as target networks. We re-train networks (standard training and adversarial training) with the different initialization from the target networks, and use the trained networks as source networks. Experimental details and model descriptions can be found in Appendix A and B.In table 1, we report test accuracies under black box attack. We first observe that high robustness against one-step attack between defended networks (R20 K2 -> R20 K), and low robustness between undefended networks (R20 2 -> R20). This observation shows that error surfaces of neural networks are driven by the training method and networks trained with the same method end up similar optimum states. It is noteworthy to observe that the accuracies against step attack from the undefended network (R20 2) are always lower than those from defended network (R20 K2). Possible explanation for this would be that adversarial training tweaks gradient seen from the clean image to point toward weaker adversarial point along that gradient direction. As a , one-step adversarial images from defended networks become weaker than those from undefended network. Transferability (iterative attack): We observe "iter FGSM" attack remains very strong even under the black box attack scenario but only between undefended networks or defended networks. This is because iter FGSM noises (X adv -X) from defended networks resemble each other. As shown in figure 1, we observe higher correlation between iter FGSM noises from a defended network (R20 K) and those from another defended network (R20 K2).Difficulty of defense/attack under the black box attack scenario: As seen from this observation, it is efficient to attack an undefended/defended network with iter FGSM examples crafted from another undefended/defended network. Thus, when we want to build a robust network under the black box attack scenario, it is desired to check accuracies for the adversarial examples crafted from other networks trained with the same strategy. Inspired by the observation that iter FGSM images transfer well between defended networks, we propose cascade adversarial training, which trains a network by injecting iter FGSM images crafted from an already defended network. We hypothesize that the network being trained with cascade adversarial training will learn to avoid such adversarial perturbation, enhancing robustness against iter FGSM attack. The intuition behind this proposed method is that we transfer the knowledge of the end of adversarial training. In particular, we train a network by injecting iter FGSM images crafted from already defended network in addition to the one-step adversarial images crafted from the network being trained. We advance the algorithm proposed in by adding low level similarity learning. Unlike, we include the clean examples used for generating adversarial images in the mini batch. Once one step forward pass is performed with the mini batch, embeddings are followed by the softmax layer for the cross entropy loss for the standard classification. At the same time, we take clean embeddings and adversarial embeddings, and minimize the distance between the two with the distance based loss. The distance based loss encourages two similar images (clean and adversarial) to produce the same outputs, not necessarily the true labels. Thus, low-level similarity learning can be considered as an unsupervised learning. By adding regularization in higher embedding layer, convolution filters gradually learn how to ignore such pixel-level perturbation. We have applied regularization on lower layers with an assumption that low level pixel perturbation can be ignored in lower hierarchy of networks. However, adding regularization term on higher embedding layer right before the softmax layer showed best performance. The more convolutional filters have chance to learn such similarity, the better the performance. Note that cross entropy doesn't encourage two similar images to produce the same output labels. Standard image classification using cross entropy compares ground truth labels with outputs of a network regardless of how similar training images are. The entire training process combining cascade adversarial training and low level similarity learning is shown in figure 2. We define the total loss as follows: DISPLAYFORM0 are the ing embeddings from X i and X adv i, respectively. m is the size of the mini batch, k (≤ m/2) is the number of adversarial images in the mini batch. λ is the parameter to control the relative weight of classification loss for adversarial images. λ 2 is the parameter to control the relative weight of the distance based loss L dist in the total loss. Bidirectional loss minimizes the distance between the two embeddings by moving both clean and adversarial embeddings as shown in the left side of the figure 3. We tried N = 1, 2 and found not much difference between the two. We report the with N = 2 for the rest of the paper otherwise noted. When N = 2, L dist becomes L2 loss. DISPLAYFORM1 Pivot loss minimizes the distance between the two embeddings by moving only the adversarial embeddings as shown in the right side of the figure 3. DISPLAYFORM2 In this case, clean embeddings (E i) serve as pivots to the adversarial embeddings. In particular, we don't back-propagate through the clean embeddings for the distance based loss. The intuition behind the use of pivot loss is that the embedding from a clean image can be treated as the ground truth embedding. We first analyze the effect of low level similarity learning on MNIST. We train ResNet models BID2 with different methods (standard training, Kurakin's adversarial training and adversarial training with our distance based loss). Experimental details can be found in Appendix A. TAB2 shows the accuracy for MNIST test dataset for different types of attack methods. As shown in the table, our method achieves better accuracy than Kurakin's method for all types of attacks with a little sacrifice on the accuracy for the clean images. Even though adversarial training is done only with "step ll", additional regularization increases robustness against unknown "step FGSM", "iter ll", "iter FGSM" and CW L ∞ attacks. This shows that our low-level similarity learning can successfully regularize the one-step adversarial perturbation and its vicinity for simple image classification like MNIST. To visualize the embedding space, we modify 20-layer ResNet model where the last fully connected layer (64x10) is changed to two fully connected layers (64x2 and 2x10). We re-train networks with standard training, Kurakin's method and our pivot loss on MNIST. 1 In figure 4, we draw embeddings (dimension=2) between two fully connected layers. As seen from this figure, adversarial images from the network trained with standard training cross the decision boundary easily as increases. With Kurakin's adversarial training, the distances between clean and adversarial embeddings are minimized compared to standard training. And our pivot loss further minimizes distance between the clean and adversarial embeddings. Note that our pivot loss also decreases absolute value of the embeddings, thus, higher λ 2 will eventually in overlap between distinct embedding distributions. We also observe that intra class variation of the clean embeddings are also minimized for the network trained with our pivot loss as shown in the scatter plot in figure 4 (c).1 Modified ResNet models showed slight decreased accuracy for both clean and adversarial images compared to original ResNet counterparts, however, we observed similar trends (improved accuracy for iterative attacks for the network trained with pivot loss) as in table 2. We train 20-layer ResNet models with pivot loss and various λ 2 s for CIFAR10 dataset to study effects of the weight of the distance measure in the loss function. Figure 5 shows that a higher λ 2 increases accuracy of the iteratively generated adversarial images. However, it reduces accuracy on the clean images, and increasing λ 2 above 0.3 even in divergence of the training. This is because embedding distributions of different classes will eventually overlap since absolute value of the embedding will be decreased as λ 2 increases as seen from the section 4.2. In this experiment, we show that there exists clear trade-off between accuracy for the clean images and that for the adversarial images, and we recommend using a very high λ 2 only under strong adversarial environment.5 CASCADE ADVERSARIAL TRAINING ANALYSIS We further study the transferability of iter FGSM images between various architectures. To this end, we first train 56-layer ResNet networks (Kurakin's, pivot loss) with the same initialization. Then we train another 56-layer ResNet network (Kurakin's) with different initialization. We repeat the training for the 110-layer ResNet networks. We measure correlation between iter FGSM noises from different networks. Figure 6 (a) shows correlation between iter FGSM noises crafted from Kurakin's network and those from Pivot network with the same initialization. Conjectured from, we observe high corre- lation between iter FGSM noises from networks with the same initialization. Correlation between iter FGSM noises from the networks with different initialization, however, becomes lower as the network is deeper as shown in figure 6 (b). Since the degree of freedom increases as the network size increases, adversarially trained networks prone to end up with different states, thus, making transfer rate lower. To maximize the benefit of the cascade adversarial training, we propose to use the same initialization for a cascade network and a source network used for iterative adversarial examples generation. We first compare a network trained with Kurakin's method and that with pivot loss. We train 110-layer ResNet models with/without pivot loss and report accuracy in table 3. We observe our lowlevel similarity learning further improves robustness against iterative attacks compared to Kurakin's adversarial training. However, the accuracy improvements against iterative attacks (iter FGSM, CW) are limited, showing regularization effect of low-level similarity learning is not sufficient for the iterative attacks on complex color images like CIFAR10. This is different from MNIST test cases where we observed significant accuracy increase for iterative attacks only with pivot loss. We observe label leaking phenomenon reported in happens even though we don't train a network with step FGSM images. Additional analysis for this phenomenon is explained in Appendix D.Next, we train a network from scratch with iter FGSM examples crafted from the defended network, R110 P. We use the same initialization used in R110 P as discussed in 5.1. In particular, iter FGSM images are crafted from R110 P with CIFAR10 training images for = 1,2,..., 16, and those are used randomly together with step ll examples from the network being trained. We train cascade networks with/without pivot loss. We also train networks with ensemble adversarial training BID9 with/without pivot loss for comparison. The implementation details for the trained models can be found in Appendix B.We find several meaningful observations in table 3. First, ensemble and cascade models show improved accuracy against iterative attack although at the expense of decreased accuracy for onestep attacks compared to the baseline defended network (R110 K). Additional data augmentation from other networks enhances the robustness against iterative attack, weakening label leaking effect caused by one-step adversarial training. Second, our low-level similarity learning (R110 P,E, R110 P,C) further enhances robustness against iterative attacks including fully unknown CW attack (especially for =4). Additional knowledge learned from data augmentation through cascade/ensemble adversarial training enables networks to learn partial knowledge of perturbations generated by an iterative method. And the learned iterative perturbations become regularized further with our low-level similarity learning making networks robust against unknown iterative attacks. During this process, clean embeddings from other classes might also be moved toward the decision boundary which in decreased accuracy for the clean images. We finally perform black box attack analysis for the cascade/ensemble networks with/without pivot loss. We report black box attack accuracy with the source networks trained with the same method, but with different initialization from the target networks. The reason for this is adversarial examples transfer well between networks trained with the same strategy as observed in section 3.1. We re-train 110-layer ResNet models using Kurakin's, cascade and ensemble adversarial training with/without low-level similarity learning and use those networks as source networks for black-box attacks. Baseline 110-layer ResNet model is also included as a source network. Target networks are the same networks used in table 3. We found iter FGSM attack ed in lower accuracy than step FGSM attack, thus, report iter FGSM attack only in table 4.We first observe that iter FGSM attack from ensemble models (R110 E2, R110 P,E2) is strong ( in lower accuracy) compared to that from any other trained networks.2 Since ensemble models learn various perturbation during training, adversarial noises crafted from those networks might be more general for other networks making them transfer easily between defended networks. Second, cascade adversarial training breaks chicken and egg problem. (In section 3.1, we found that it is efficient to use a defended network as a source network to attack another defended network.) Even though the transferability between defended networks is reduced for deeper networks, cascade network (R110 K,C) shows worst case performance against the attack not from a defended network, but from a purely trained network (R110 2). Possible solution to further improve the worst case robustness would be to use more than one network as source networks (including pure/defended networks) for iter FGSM images generation for cascade adversarial training. Third, ensemble/cascade networks together with our low-level similarity learning (R110 P,E, R110 P,C) show better worst case accuracy under black box attack scenario. This shows that enhancing robustness against iterative white box attack also improves robustness against iterative black box attack. We performed through transfer analysis and showed iter FGSM images transfer easily between networks trained with the same strategy. We exploited this and proposed cascade adversarial training, a method to train a network with iter FGSM adversarial images crafted from already defended networks. We also proposed adversarial training regularized with a unified embedding for classification and low-level similarity learning by penalizing distance between the clean and their corresponding adversarial embeddings. Combining those two techniques (low level similarity learning + cascade adversarial training) with deeper networks further improved robustness against iterative attacks for both white-box and black-box attacks. However, there is still a gap between accuracy for the clean images and that for the adversarial images. Improving robustness against both one-step and iterative attacks still remains challenging since it is shown to be difficult to train networks robust for both one-step and iterative attacks simultaneously. Future research is necessary to further improve the robustness against iterative attack without sacrificing the accuracy for step attacks or clean images under both white-box attack and black-box attack scenarios. We perform 24x24 random crop and random flip on 32x32 original images. We generate adversarial images with "step ll" after these steps otherwise noted. We use stochastic gradient descent (SGD) optimizer with momentum of 0.9, weight decay of 0.0001 and mini batch size of 128. For adversarial training, we generate k = 64 adversarial examples among 128 images in one mini-batch. We start with a learning rate of 0.1, divide it by 10 at 4k and 6k iterations, and terminate training at 8k iterations for MNIST, and 48k and 72k iterations, and terminate training at 94k iterations for CIFAR10. Ensemble models Pre-trained models R20 E, R20 P,E, R110 E, R110 P,E R20 3, R110 3 R110 E2, R110 P,E2 R20 4, R110 4 Cascade models Pre-trained model R20 K,C, R20 P,C R20 P R110 K,C, R110 P,C R110 P R110 K,C2, R110 P,C2R110 P Figure 7: Argument to the softmax vs. in test time. "step ll", "step FGSM" and "random sign" methods were used to generate test-time adversarial images. Arguments to the softmax were measured by changing for each test method and averaged over randomly chosen 128 images from CIFAR10 test-set. Blue line represents true class and the red line represents mean of the false classes. Shaded region shows ± 1 standard deviation of each line. We draw average value of the argument to the softmax layer for the true class and the false classes to visualize how the adversarial training works as in figure 7. Standard training, as expected, shows dramatic drop in the values for the true class as we increase in "step ll" or "step FGSM direction. With adversarial training, we observe that the value drop is limited at small and our method even increases the value in certain range upto =10. Note that adversarial training is not the same as the gradient masking. As illustrated in figure 7, it exposes gradient information, however, quickly distort gradients along the sign of the gradient ("step ll" or "step FGSM) direction. We also observe improved (broader margins than baseline) for "random sign" added images even though we didn't inject random sign added images during training. Overall shape of the argument to the softmax layer in our case becomes smoother than Kurakin's method, suggesting our method is good for pixel level regularization. Even though actual value of the embeddings for the true class in our case is smaller than that in Kurakin's, the standard deviation of our case is less than Kurakin's, making better margin between the true class and false classes. We observe accuracies for the "step FGSM" adversarial images become higher than those for the clean images ("label leaking" phenomenon) by training with "step FGSM" examples as in. Interestingly, we also observe "label leaking" phenomenon even without providing true labels for adversarial images generation. We argue that "label leaking" is a natural of the adversarial training. To understand the nature of adversarial training, we measure correlation between gradients w.r.t. different images (i.e. clean vs. adversarial) as a measure of error surface similarity. We measure correlation between gradients w.r.t. clean vs. "step ll" image, clean vs. "step FGSM" image, clean vs. "random sign" added image, and "step ll" image vs. "step FGSM" image for three trained networks (a) R20, (b) R20 K and (c) R20 P (Ours) in TAB10 shows that black box attack between trained networks with the same initialization tends to be more successful than that between networks with different initialization as explained in. In table 9, our method (R20 P 2) is always better at one-step and iterative black box attack from defended networks (R20 K, R20 P) and undefended network (R20) than Kurakin's method (R20 B2). However, it is hard to tell which method is better than the other one as explained in the main paper. In table 10, we show black box attack accuracies with the source and the target networks switched from the table 4. We also observe that networks trained with both low-level similarity learning and cascade/ensemble adversarial training (R110 P,C2, R110 P,E2) show better worst-case performance than other networks. Overall, iter FGSM images crafted from ensemble model families (R110 E, R110 P,E) remain strong on the defended networks. such that X + δ ∈ n where, the function f is defined such that attack is success if and only if f (X + δ) < 0, δ is the target perturbation defined as X adv −X, c is the parameter to control the relative weight of function f in the total cost function, and τ is the control threshold used to penalize any terms that exceed τ.Since CW L ∞ attack is computationally expensive, we only use 100 test examples (10 examples per each class). We search adversarial example X adv with c ∈ {0.1, 0.2, 0.5, 1, 2, 5, 10, 20} and τ ∈ {0.02, 0.04, ..., 0.6} for MNIST and c ∈ {0.1, 0.3, 1, 3, 10, 30, 100} and τ ∈ {0.001, 0.002, ..., 0.01, 0.012, ..., 0.02, 0.024, ..., 0.04, 0.048, ..., 0.08} for CIFAR10. We use Adam optimizer with an initial learning rate of 0.01/c since we found constant initial learning rate for c · f (X + δ) term is critical for successful adversarial images generation. We terminate the search after 2,000 iterations for each X, c and τ. If f (X + δ) < 0 and the ing ||δ|| ∞ is lower than the current best distance, we update X adv. FIG5 shows cumulative distribution function of for 100 successful adversarial examples per each network. We report the number of adversarial examples with > 0.3*255 for MNIST and that with > 2 or 4 for CIFAR10. As seen from this figure, our approaches provide robust defense against CW L ∞ attack compared to other approaches.
Cascade adversarial training + low level similarity learning improve robustness against both white box and black box attacks.