Adversarial training is one of the strongest defenses against adversarial attacks, but it requires adversarial examples to be generated for every mini-batch during optimization.   The expense of producing these examples during training often precludes adversarial training from use on complex image datasets. 
 In this study, we explore the mechanisms by which adversarial training improves classifier robustness, and show that these mechanisms can be effectively mimicked using simple regularization methods, including label smoothing and logit squeezing.  
 Remarkably, using these simple regularization methods in combination with Gaussian noise injection, we are able to achieve strong adversarial robustness -- often exceeding that of adversarial training -- using no adversarial examples. Deep Neural Networks (DNNs) have enjoyed great success in many areas of computer vision, such as classification BID7 , object detection BID4 , and face recognition BID11 . However, the existence of adversarial examples has raised concerns about the security of computer vision systems BID16 BID1 . For example, an attacker may cause a system to mistake a stop sign for another object BID3 or mistake one person for another BID14 . To address security concerns for high-stakes applications, researchers are searching for ways to make models more robust to attacks.Many defenses have been proposed to combat adversarial examples. Approaches such as feature squeezing, denoising, and encoding BID19 BID13 BID15 BID10 have had some success at pre-processing images to remove adversarial perturbations. Other approaches focus on hardening neural classifiers to reduce adversarial susceptibility. This includes specialized non-linearities BID20 , modified training processes BID12 , and gradient obfuscation BID0 .Despite all of these innovations, adversarial training BID5 , one of the earliest defenses, still remains among the most effective and popular strategies. In its simplest form, adversarial training minimizes a loss function that measures performance of the model on both clean and adversarial data as follows DISPLAYFORM0 where L is a standard (cross entropy) loss function, (x i , y i ) is an input image/label pair, θ contains the classifier's trainable parameters, κ is a hyper-parameter, and x i,adv is an adversarial example for image x. BID9 pose adversarial training as a game between two players that similarly requires computing adversarial examples on each iteration.A key drawback to adversarial training methods is their computational cost; after every mini-batch of training data is formed, a batch of adversarial examples must be produced. To train a network that resists strong attacks, one needs to train with the strongest adversarial examples possible. For example , networks hardened against the inexpensive Fast Gradient Sign Method (FGSM, Goodfellow et al. (2014) ) can be broken by a simple two-stage attack BID17 . Current state-of-theart adversarial training results on MNIST and CIFAR-10 use expensive iterative adversaries BID9 , such as the Projected Gradient Descent (PGD) method, or the closely related Basic Iterative Method (BIM) BID8 . Adversarial training using strong attacks may be 10-100 times more time consuming than standard training methods. This prohibitive cost makes it difficult to scale adversarial training to larger datasets and higher resolutions.In this study, we show that it is possible to achieve strong robustness -comparable to or greater than the robustness of adversarial training with a strong iterative attack -using fast optimization without adversarial examples. We achieve this using standard regularization methods, such as label smoothing BID18 and the more recently proposed logit squeezing BID6 . While it has been known for some time that these tricks can improve the robustness of models, we observe that an aggressive application of these inexpensive tricks, combined with random Gaussian noise, are enough to match or even surpass the performance of adversarial training on some datasets. For example, using only label smoothing and augmentation with random Gaussian noise, we produce a CIFAR-10 classifier that achieves over 73% accuracy against black-box iterative attacks, compared to 64% for a state-of-the-art adversarially trained classifier BID9 . In the white-box case, classifiers trained with logit squeezing and label smoothing get ≈ 50% accuracy on iterative attacks in comparison to ≈ 47% for adversarial training. Regularized networks without adversarial training are also more robust against non-iterative attacks, and more accurate on non-adversarial examples.Our goal is not just to demonstrate these defenses, but also to dig deep into what adversarial training does, and how it compares to less expensive regularization-based defenses. We begin by dissecting adversarial training, and examining ways in which it achieves robustness. We then discuss label smoothing and logit squeezing regularizers, and how their effects compare to those of adversarial training. We then turn our attention to random Gaussian data augmentation, and explore the importance of this technique for adversarial robustness. Finally, we combine the regularization methods with random Gaussian augmentation, and experimentally compare the robustness achievable using these simple methods to that achievable using adversarial training. We studied the robustness of adversarial training, label smoothing, and logit squeezing through a linear approximation L that relates the magnitude of adversarial perturbations to the logit gap and the difference between the adversarial directions for different labels. Using this simple model, we observe how adversarial training achieves robustness and try to imitate this robustness using label smoothing and logit squeezing. The resulting methods perform well on MNIST, and can get results on CIFAR-10 and CIFAR-100 that can excel over adversarial training in both robustness and accuracy on clean examples. By demonstrating the effectiveness of these simple regularization methods, we hope this work can help make robust training easier and more accessible to practitioners.
Achieving strong adversarial robustness comparable to adversarial training without training on adversarial examples
{"src": "scitldr-AIC_train", "paper_id": "BJlr0j0ctX"}