Recently, researchers have discovered that the state-of-the-art object classifiers can be fooled easily by small perturbations in the input unnoticeable to human eyes.  It is known that an attacker can generate strong adversarial examples if she knows the classifier parameters . Conversely, a defender can robustify the classifier by retraining if she has the adversarial examples . 
The cat-and-mouse game nature of attacks and defenses raises the question of the presence of equilibria in the dynamics .
In this paper, we present a neural-network based attack class to approximate a larger but intractable class of attacks, and 
 formulate the attacker-defender interaction as a zero-sum leader-follower game. We present sensitivity-penalized optimization algorithms to find minimax solutions, which are the best worst-case defenses against whitebox attacks. Advantages of the learning-based attacks and defenses compared to gradient-based attacks and defenses are demonstrated with MNIST and CIFAR-10. Recently, researchers have made an unsettling discovery that the state-of-the-art object classifiers can be fooled easily by small perturbations in the input unnoticeable to human eyes BID24 BID7 . Following studies tried to explain the cause of the seeming failure of deep learning toward such adversarial examples. The vulnerability was ascribed to linearity BID24 , low flexibility BID4 , or the flatness/curvedness of decision boundaries BID20 , but a more complete picture is still under research. This is troublesome since such a vulnerability can be exploited in critical situations such as an autonomous car misreading traffic signs or a facial recognition system granting access to an impersonator without being noticed. Several methods of generating adversarial examples were proposed BID7 BID19 BID2 , most of which use the knowledge of the classifier to craft examples. In response, a few defense methods were proposed: retraining target classifiers with adversarial examples called adversarial training BID24 BID7 ; suppressing gradient by retraining with soft labels called defensive distillation BID22 ; hardening target classifiers by training with an ensemble of adversarial examples BID25 .In this paper we focus on whitebox attacks, that is, the model and the parameters of the classifier are known to the attacker. This requires a more robust classifier or defense method than simply relying on the secrecy of the parameters as defense. When the classifier parameters are known to an attacker, existing attack methods are very successful at fooling the classifiers. Conversely , when the attack is known to the classifier, e.g., in the form of adversarial examples, one can weaken the attack by retraining the classifier with adversarial examples, called adversarial training. However, if we repeat adversarial sample generation and adversarial training back-to-back, it is observed that the current adversarially-trained classifier is no longer robust to previous attacks (see Sec. 3.1.) To find the classifier robust against the class of gradient-based attacks, we first propose a sensitivitypenalized optimization procedure. Experiments show that the classifier from the procedure is more robust than adversarially-trained classifiers against previous attacks, but it still remains vulnerable to some degrees. This raises the main question of the paper: Can a classifier be robust to all types of attacks? The answer seems to be negative in light of the strong adversarial examples that can be crafted by direct optimization procedures from or BID2 . Note that the class of optimization-based attack is very large, as there is no restriction on the adversarial patterns that can be generated except for certain bounds such as l p -norm bounds. The vastness of the optimization-based attack class is a hindrance to the study of the problem, as the defender cannot learn efficiently about the attack class from a finite number of samples. To study the problem analytically, we use a class of learning-based attack that can be generated by a class of neural networks. This class of attack can be considered an approximation of the class of optimization -based attacks, in that the search space of optimal perturbation is restricted to the parameter space of a neural network architecture, e.g., all perturbations that can be generated by fully-connected 3-layer ReLU networks. Similar to what we propose, others have recently considered training neural networks to generate adversarial examples BID21 BID0 . While the proposed learning-based attack is weaker than the optimization-based attack, it can generate adversarial examples in test time with only single feedforward passes, which makes real-time attacks possible. We also show that the class of neural-network based attacks is quite different from the the class of gradient-based attacks (see Sec. 4.1.) Using the learning-based attack class, we introduce a continuous game formulation for analyzing the dynamics of attack-defense. The game is played by an attacker and a defender/classifier 1 , where the attacker tries to maximize the risk of the classification task by perturbing input samples under certain constraints such as l p -norm bounds, and the defender/classifier tries to adjust its parameters to minimize the same risk given the perturbed inputs. It is important to note that for adversarial attack problems, the performance of an attack or a defense cannot be measured in isolation, but only in pairs of (attack, defense) . This is because the effectiveness of an attack/defense depends on the defense/attack it is against. As a two-player game , there may not be a dominant defense that is no less robust than all other defenses against all attacks. However, there is a natural notion of the best defense or attack in the worst case. Suppose one player moves first by choosing her parameters and the other player responds with the knowledge of the first player's move. This is an example of a leader-follower game BID1 for which there are two well-known states, the minimax and the maximin solutions if it is a constant-sum game. To find those solutions empirically, we propose a new continuous optimization method using the sensitivity penalization term. We show that the minimax solution from the proposed method is indeed different from the solution from the conventional alternating descent/ascent and is also more robust. We also show that the strength/weakness of the minimax-trained classifier is different from that of adversarially-trained classifiers for gradient-based attacks. The contributions of this paper are summarized as follows.• We provide a continuous game model to analyze adversarial example attacks and defenses, using the neural network-based attack class as a feasible approximation to a larger but intractable class of optimization-based attacks.• We demonstrate the difficulty of defending against multiple attack types and present the minimax defense as the best worst-case defense methods.• We propose a sensitivity-penalized optimization method (Alg. 1) to numerically find continuous minimax solutions, which is better than alternating descent/ascent. The proposed optimization method can also be used for other minimax problems beyond the adversarial example problem.The proposed methods are demonstrated with the MNIST and the CIFAR-10 datasets. For readability, details about experimental settings and the results with CIFAR-10 are presented in the appendix. In this paper, we present a continuous game formulation of adversarial attacks and defenses using a learning-based attack class implemented by neural networks. We show that this class of attacks is quite different from the gradient-based attacks. While a classifier robust to all types of attack may yet be an elusive goal, the minimax defense against the neural network-based attack class is well-defined and practically achievable. We show that the proposed optimization method can find minimax defenses which are more robust than adversarially-trained classifiers and the classifiers from simple alternating descent/ascent. We demonstrate these with MNIST and CIFAR-10.
A game-theoretic solution to adversarial attacks and defenses.
{"src": "scitldr-AIC_train", "paper_id": "ByqFhGZCW"}