Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss.
 We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step.
 We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with strong robustness to black-box attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks. Machine learning (ML) models are often vulnerable to adversarial examples, maliciously perturbed inputs designed to mislead a model at test time BID4 BID36 BID15 BID29 . Furthermore, BID36 showed that these inputs transfer across models: the same adversarial example is often misclassified by different models, thus enabling simple black-box attacks on deployed models BID23 .Adversarial training BID36 increases robustness by augmenting training data with adversarial examples. showed that adversarially trained models can be made robust to white-box attacks (i.e., with knowledge of the model parameters) if the perturbations computed during training closely maximize the model's loss. However, prior attempts at scaling this approach to ImageNet-scale tasks BID12 ) have proven unsuccessful BID20 .It is thus natural to ask whether it is possible, at scale, to achieve robustness against the class of black-box adversaries Towards this goal, BID20 adversarially trained an Inception v3 model BID38 on ImageNet using a "single-step" attack based on a linearization of the model's loss BID15 . Their trained model is robust to single-step perturbations but remains vulnerable to more costly "multi-step" attacks. Yet, BID20 found that these attacks fail to reliably transfer between models, and thus concluded that the robustness of their model should extend to black-box adversaries. Surprisingly, we show that this is not the case.We demonstrate, formally and empirically, that adversarial training with single-step methods admits a degenerate global minimum, wherein the model's loss can not be reliably approximated by a linear function. Specifically, we find that the model's decision surface exhibits sharp curvature near the data points, thus degrading attacks based on a single gradient computation. In addition to the model of BID20 , we reveal similar overfitting in an adversarially trained Inception ResNet v2 model BID37 , and a variety of models trained on MNIST BID22 .We harness this result in two ways. First, we show that adversarially trained models using single-step methods remain vulnerable to simple attacks. For black-box adversaries, we find that perturbations crafted on an undefended model often transfer to an adversarially trained one. We also introduce a simple yet powerful single-step attack that applies a small random perturbation-to escape the nonsmooth vicinity of the data point-before linearizing the model's loss. While seemingly weaker than the Fast Gradient Sign Method of BID15 , our attack significantly outperforms it for a same perturbation norm, for models trained with or without adversarial training.Second, we propose Ensemble Adversarial Training, a training methodology that incorporates perturbed inputs transferred from other pre-trained models. Our approach decouples adversarial example generation from the parameters of the trained model, and increases the diversity of perturbations seen during training. We train Inception v3 and Inception ResNet v2 models on ImageNet that exhibit increased robustness to adversarial examples transferred from other holdout models, using various single-step and multi-step attacks BID15 BID7 BID19 . We also show that our methods globally reduce the dimensionality of the space of adversarial examples BID40 . Our Inception ResNet v2 model won the first round of the NIPS 2017 competition on Defenses Against Adversarial Attacks BID21 , where it was evaluated on other competitors' attacks in a black-box setting. BID16 BID24 BID31 BID28 BID10 and many remain vulnerable to adaptive attackers BID7 b; BID3 . Adversarial training BID36 BID15 BID20 appears to hold the greatest promise for learning robust models. show that adversarial training on MNIST yields models that are robust to whitebox attacks, if the adversarial examples used in training closely maximize the model's loss. Moreover, recent works by BID34 , BID33 and BID18 even succeed in providing certifiable robustness for small perturbations on MNIST. As we argue in Appendix C, the MNIST dataset is peculiar in that there exists a simple "closed-form" denoising procedure (namely feature binarization) which leads to similarly robust models without adversarial training. This may explain why robustness to white-box attacks is hard to scale to tasks such as ImageNet BID20 . We believe that the existence of a simple robust baseline for MNIST can be useful for understanding some limitations of adversarial training techniques. BID36 found that adversarial examples transfer between models, thus enabling blackbox attacks on deployed models. showed that black-box attacks could succeed with no access to training data, by exploiting the target model's predictions to extract BID39 a surrogate model. Some prior works have hinted that adversarially trained models may remain vulnerable to black-box attacks: BID15 found that an adversarial maxout network on MNIST has slightly higher error on transferred examples than on white-box examples. further showed that a model trained on small perturbations can be evaded by transferring perturbations of larger magnitude. Our finding that adversarial training degrades the accuracy of linear approximations of the model's loss is as an instance of a gradient-masking phenomenon BID30 , which affects other defensive techniques BID31 BID7 BID28 BID5 BID2 .
Adversarial training with single-step methods overfits, and remains vulnerable to simple black-box and white-box attacks. We show that including adversarial examples from multiple sources helps defend against black-box attacks.
{"src": "scitldr-AIC_train", "paper_id": "rkZvSe-RZ"}