Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness

The vulnerability of deep neural networks to adversarial examples has motivated an increasing number of defense strategies for promoting model robustness. However, the progress is usually hampered by insufficient robustness evaluations. As the de facto standard to evaluate adversarial robustness, adversarial attacks typically solve an optimization problem of crafting adversarial examples with an iterative process. In this work, we propose a Model-Agnostic Meta-Attack (MAMA) approach to discover stronger attack algorithms automatically. Our method learns the optimizer in adversarial attacks parameterized by a recurrent neural network, which is trained over a class of data samples and defenses to produce effective update directions during adversarial example generation. Furthermore, we develop a model-agnostic training algorithm to improve the generalization ability of the learned optimizer when attacking unseen defenses. Our approach can be flexibly incorporated with various attacks and consistently improves the performance with little extra computational cost. Extensive experiments demonstrate the effectiveness of the learned attacks by MAMA compared to the state-of-the-art attacks on different defenses, leading to a more reliable evaluation of adversarial robustness.


INTRODUCTION
Deep neural networks are vulnerable to the maliciously crafted adversarial examples (Biggio et al., 2013;Szegedy et al., 2014;Goodfellow et al., 2015), which aim to induce erroneous model predictions by adding small perturbations to the normal inputs. Due to the threats, a multitude of defense strategies have been proposed to improve adversarial robustness (Kurakin et al., 2017b;Madry et al., 2018;Wong & Kolter, 2018;Dong et al., 2020a;Pang et al., 2020). However, many defenses have later been shown to be ineffective due to incomplete or incorrect robustness evaluations (Athalye et al., 2018;Uesato et al., 2018;Carlini et al., 2019;Croce & Hein, 2020b;Dong et al., 2020b;Tramer et al., 2020), making it particularly challenging to understand their effects and identify the actual progress of the field. Therefore, developing methods that can evaluate adversarial robustness accurately and reliably is of profound importance.
Adversarial attacks are broadly adopted as an indispensable solution to evaluate adversarial robustness for different models. One of the most prominent attacks is the projected gradient descent (PGD) method (Madry et al., 2018), which generates an adversarial example by performing iterative gradient updates to maximize a classification loss (e.g., cross-entropy loss) w.r.t. the input. Although recent methods improve upon PGD by introducing different loss functions (Carlini & Wagner, 2017;Gowal et al., 2019;Croce & Hein, 2020b;Sriramanan et al., 2020) and adjusting the step size (Croce & Hein, 2020b), most of them adopt hand-designed optimization algorithms, such as vanilla gradient descent, momentum (Polyak, 1964), and Adam (Kingma & Ba, 2015). However, it has been shown that these attacks can be sub-optimal (Croce & Hein, 2020b;Tramer et al., 2020), arousing an overestimation of adversarial robustness for some defenses. It is thus imperative to develop more effective optimization algorithms for improving adversarial attacks. Nevertheless, designing a generic optimization algorithm in a hand-crafted manner is non-trivial, considering the different defense models with varying network architectures, defense strategies, and training datasets.
To overcome this issue and develop stronger adversarial attacks, in this paper, we propose a Model-Agnostic Meta-Attack (MAMA) approach to learn optimization algorithms in adversarial attacks automatically. In particular, we parameterize the optimizer with a recurrent neural network (RNN) to mimic the behavior of iterative attacks, which outputs an update direction at each time step during adversarial example generation. By learning to solve a class of optimization problems with different data samples and defense models, the RNN optimizer could benefit from long-term dependencies of iterative attacks and exploit common structures among the optimization problems. To improve and stabilize training, we then propose a prior-guided refinement strategy, which imposes a prior given by the PGD update rule on the outputs of the RNN optimizer, whose parameters are trained via the maximum a posteriori (MAP) estimation framework. Consequently, the RNN optimizer can learn to yield more effective update directions than hand-designed ones.
Despite the effectiveness, the learned optimizer may not generalize well to attacking unseen defenses due to potential overfitting to the defenses used for training. Although training a different optimizer for each defense is possible, the training process can be time-consuming and inconvenient in practice when it is utilized to benchmark adversarial robustness. To endow the learned optimizer with a better generalization ability, we develop a model-agnostic training algorithm with a gradient-based metatrain and meta-test process to simulate the shift from seen to unseen defenses. Therefore, the learned optimizer can directly be deployed to attack unseen defenses without retraining or fine-tuning.
Extensive experiments validate the effectiveness and generalization ability of the learned optimizers for attacking different defense models. We also demonstrate the flexibility of our method integrating with various attacks, including those with different loss functions (Carlini & Wagner, 2017;Gowal et al., 2019;Croce & Hein, 2020b) and initialization (Tashiro et al., 2020). Our method consistently improves the attacking performance over baselines, while introducing little extra computational cost. By incorporating our method with an orthogonal technique -output diversified initialization (ODI) (Tashiro et al., 2020), we achieve lower robust test accuracy on all 12 defense models that we study than other state-of-the-art attacks, including the MultiTargeted (MT) attack (Gowal et al., 2019) and AutoAttack (Croce & Hein, 2020b), leading to a more reliable evaluation of adversarial robustness. Moreover, MAMA provides a plug-and-play module in adversarial attacks, which may benefit future adversarial attacks if new improvements have been made in different aspects.

ADVERSARIAL ATTACKS
Let f (x) : x ∈ R D → z ∈ R K denote a classifier, which outputs the logits z := [z 1 , ..., z K ] over K classes for an input image x. The prediction of f is C f (x) = arg max i=1,...,K z i . Given the true label y of x, adversarial attacks aim to generate an adversarial examplex that is misclassified by f (i.e., C f (x) = y), while the distance between the adversarial examplex and the natural one x measured by the ∞ norm is smaller than a threshold as x − x ∞ ≤ . Although we introduce our approach based on the ∞ norm only, the extension to other p norms is straightforward. The adversarial examplex can be generated by solving a constrained optimization problem aŝ where L is a loss function on top of the classifier f (x). For example, L could be the cross-entropy loss as L ce (f (x), y) = − log p y where p y = e zy / K i=1 e z i denotes the predicted probability of class y, or the margin-based CW loss (Carlini & Wagner, 2017) as L cw (f (x), y) = max i =y z i − z y .
A lot of gradient-based methods (Goodfellow et al., 2015;Kurakin et al., 2017a;Carlini & Wagner, 2017;Madry et al., 2018; have been proposed to solve this optimization problem. The projected gradient descent (PGD) method (Madry et al., 2018) performs iterative updates aŝ where B (x) = {x : x − x ∞ ≤ } denotes the ∞ ball centered at x with radius , Π(·) is the projection operation, and α t is the step size at the t-th attack iteration. PGD initializesx 0 randomly by uniform sampling within B (x) and adopts a fixed step size α in every iteration.
To develop stronger attacks for reliable robustness evaluation, recent improvements upon PGD include adopting different loss functions (Gowal et al., 2019;Croce & Hein, 2020b;Sriramanan et al., 2020), adjusting the step size (Croce & Hein, 2020b), and designing sampling strategies of initialization (Tashiro et al., 2020). For the update rules, besides the vanilla gradient descent adopted in PGD, recent attacks Gowal et al., 2019) suggest to use the momentum (Polyak, 1964) and Adam (Kingma & Ba, 2015) optimizers. In contrast, we aim to enhance adversarial attacks by learning optimization algorithms in an automatic way.

ADVERSARIAL DEFENSES
To build robust models against adversarial attacks, numerous defense strategies have been proposed, but most defenses can be evaded by new adaptive attacks (Athalye et al., 2018;Tramer et al., 2020). Among the existing defenses, adversarial training (AT) is arguably the most effective defense technique, in which the network is trained on the adversarial examples generated by attacks. Based on the primary AT frameworks like PGD-AT (Madry et al., 2018) and TRADES , improvements have been made via ensemble learning (Tramèr et al., 2018;Pang et al., 2019), metric learning (Mao et al., 2019;Pang et al., 2020), semi-supervised learning (Alayrac et al., 2019;Zhai et al., 2019), and self-supervised learning (Chen et al., 2020a;Kim et al., 2020). Due to the high computational cost of AT, other efforts are devoted to accelerating the training procedure (Shafahi et al., 2019;Wong et al., 2020;. Recent works emphasize the training tricks (e.g., weight decay, batch size, etc.) in AT (Gowal et al., 2020;Pang et al., 2021). However, it has been shown that the robust test accuracy of numerous adversarially trained models can be degraded significantly by using stronger attacks (Croce & Hein, 2020b;Tramer et al., 2020), indicating that the reliable evaluation of adversarial robustness remains an imperative yet challenging task. In this paper, we focus on evaluating adversarial robustness of AT models due to their superior robustness over other defense techniques.

META-LEARNING
Meta-learning (learning to learn) studies how to learn learning algorithms automatically from a meta perspective, and has shown promise in few-shot learning (Finn et al., 2017) and learning optimizers (Andrychowicz et al., 2016). The latter is called learning to optimize (L2O). The first L2O method leverages a coordinate-wise long short term memory (LSTM) network as the optimizer model, which is trained by gradient descent on a class of optimization problems (Andrychowicz et al., 2016). The optimizer can also be trained by reinforcement learning (Li & Malik, 2017). Bello et al. (2017) propose to adopt an RNN controller to generate the update rules of optimization algorithms, which can be described as a domain specific language. L2O is then extended to efficiently optimize black-box functions (Chen et al., 2017). Recent methods focus on improving the training stability and generalization of the learned optimizers by designing better optimizer models and training techniques (Lv et al., 2017;Wichrowska et al., 2017;Chen et al., 2020b).
The general idea of meta-learning has been investigated in adversarial attacks. Most related methods learn to generate adversarial examples by using convolutional neural networks (CNNs), which usually take clean images as inputs and return adversarial examples/perturbations (Baluja & Fischer, 2017;Poursaeed et al., 2018). Differently, our method adopts an RNN model as the attack optimizer to mimic the behavior of iterative attacks. Empirical results validate the superiority of our approach compared to the CNN-based generators. Meta-learning has also been used in black-box attacks (Du et al., 2020) and AT defenses (Xiong & Hsieh, 2020;Jiang et al., 2021). It is noteworthy that Xiong & Hsieh (2020) propose a similar approach to ours that uses an RNN optimizer to learn attacks for AT. Although the RNN optimizer is also adopted in our approach, we improve the attacking performance and generalization ability of the RNN optimizer with new techniques, as presented in Sec. 3. More detailed comparisons between our work and Xiong & Hsieh (2020) are shown in Appendix A.

METHODOLOGY
In this paper, we propose a Model-Agnostic Meta-Attack (MAMA) approach to improve adversarial attacks by learning optimization algorithms automatically. MAMA permits learning to exploit common structures of the optimization problems (1) over a class of input samples and defense models. We first present the instantiation of the learned optimizer with a recurrent neural network (RNN) and the corresponding training procedure in Sec. 3.1. The learned optimizer is expected to not only perform well on the data samples and defenses for which it is trained but also possess an excellent generalization ability to other unseen data samples, defenses, and more optimization steps (i.e., attack iterations) than training. Therefore, we propose a model-agnostic training algorithm simulating train/test shift to ensure generalization of the learned optimizer, as detailed in Sec. 3.2. An overview of our proposed MAMA approach is provided in Fig. 1.

META-ATTACK WITH RECURRENT NEURAL NETWORKS
Different from the existing attack algorithms, we propose a meta-attack that learns an optimizer in adversarial attacks parameterized by an RNN model m φ with parameters φ to mimic the behavior of iterative attacks. The learned optimizer is capable of optimizing the loss function L in Eq. (1) for different data samples (x, y) and defense models f . At each step, m φ takes gradient information of the loss function (1) at the current adversarial example as input and outputs an update direction for generating the adversarial example. This procedure can be expressed aŝ where α t is the step size, ∇ t = ∇ x L(f (x t ), y) denotes the gradient of the loss function L atx t , and h t denotes the hidden state representation of the optimizer, which contains rich temporal gradient information to capture long-term dependencies. The output g t of the RNN model is used to update the adversarial examplex t . We apply a tanh transformation to the last layer of the RNN model to normalize and smoothen the update direction g t since the adversarial perturbations are required to be smaller than the threshold . In particular, we adopt the coordinate-wise long short term memory (LSTM) network (Andrychowicz et al., 2016) as the RNN model m φ , which can significantly reduce trainable parameters, making training much easier and faster.
Given a distribution F over defenses f and a data distribution D over (x, y) pairs, the parameters φ can be trained by maximizing the expected loss over the entire trajectory of optimization as wherex t is obtained by the optimizer model m φ through Eq. (3), w t ∈ R ≥0 is the weight for the loss at the t-th step to emphasize different importance on the loss at different steps, and T is the total number of optimization steps. For example, we could let w t = 1[t = T ] to focus on the loss at the final adversarial example since it would be used to attack the classifier, but using a different setting for w t can make training easier. For training the optimizer network m φ , we can maximize the objective (4) using gradient ascent on φ. The expectation in Eq. (4) is approximated by mini-batches of randomly sampled defense models f and data points (x, y). The gradient w.r.t. φ is calculated by truncated backpropagation. Besides the general framework of learning to optimize (Andrychowicz et al., 2016;Chen et al., 2020b;Xiong & Hsieh, 2020), there exist particular challenges in training the optimizer due to the optimization problem of adversarial attacks as illustrated below.
Vanishing gradient problem. Note that the optimization problem (1) is a constrained one, which is different from unconstrained problems commonly studied in previous works (Andrychowicz et al., 2016;Chen et al., 2020b;Lv et al., 2017;Wichrowska et al., 2017). As in Eq.
(3), we need to project the adversarial example onto the ∞ ball B (x) after every optimization step, which is essentially the clip function. However, optimizing Eq. (4) requires backpropagation through the clip function, which can result in a number of zero gradient values when the update exceeds the ∞ ball. This can consequently lead to the vanishing gradient problem for a long optimization trajectory. To address this problem, we apply the straight-through estimator (Bengio et al., 2013) to get an approximate gradient of the clip function as ∇ x Π B (x) (x ) = I, where I denotes the identity matrix. In this way, the optimizer can effectively be trained to optimize the constrained problem (1).
Training instability problem. Training the learned optimizer may suffer from an instability problem (Chen et al., 2020b), leading to unsatisfactory performance. To further stabilize training of the learned optimizer m φ and improve its performance, we propose a prior-guided refinement strategy which leverages an informative prior to refine the outputs of the optimizer. In particular, the prior is given by the PGD update rule (Madry et al., 2018) which performs generally well on many defenses. We specify the prior distribution of g t to be an isotropic Gaussian distribution as N (sign(∇ t ), 1 2λt ), whose mean is the PGD update direction sign(∇ t ). The variance of the distribution is 1 /2λt with λ t being a hyperparameter. Under the maximum a posteriori (MAP) estimation framework, the objective function (4) can be viewed as the likelihood while the prior is equivalent to minimizing the squared 2 distance as λ t g t − sign(∇ t ) 2 2 . We provide a formal derivation of MAP in Appendix B. Therefore, the overall objective function of training the optimizer becomes where {λ t } T t=1 can be seen as another set of weights corresponding to the regularization terms at each step. In this way, the optimizer can learn from the informative prior given by the PGD update rule and explore potential better directions simultaneously. To maximize the objective function (5), a basic technique is using gradient ascent on φ. We call it Basic Meta-Attack (BMA).

MODEL-AGNOSTIC META-ATTACK ALGORITHM
To make robustness evaluation more convenient after training, the learned optimizer should generalize well to new data samples, defense models, and more optimization steps that are not seen during training. In practice, we observe that the data-driven process of optimizing Eq. (5) (i.e., BMA) enables to obtain a generalizable optimizer on new data samples with more optimization steps, but the learned optimizer may not necessarily generalize well to new defenses, as shown in Sec. 4.3. This is because the defenses usually adopt different network architectures and defense strategies, making it hard for an attack optimizer to generalize well across a variety of defenses (Tramer et al., 2020).
To improve the generalization ability of the learned optimizer when attacking unseen defenses, we propose a model-agnostic training algorithm for meta-attack (i.e., MAMA). This method simulates the shift between defenses during training with a gradient-based meta-train and meta-test procedure, inspired by model-agnostic meta-learning (Finn et al., 2017;. After MAMA training, the learned optimizer can directly be used to attack unseen defenses, which will be more convenient and user-friendly for carrying out robustness evaluation once novel defense methods are presented. Specifically, assume that we have a set of N defenses (denoted as F) in training, where we reuse the notation F without ambiguity. To simulate the shift between defense models in training and testing, at each training iteration we split F into a meta-train setF of N − n defenses and a meta-test setF of the other n defenses. The model-agnostic training objective is to maximize the loss J (φ;F, D) on the meta-train defenses, while ensuring that the update direction of φ also leads to maximizing J (φ;F, D) on the meta-test defenses, such that the learned optimizer will perform well on true test defenses. During meta-train, the optimizer parameters φ are updated by the one-step gradient ascent where β is the meta-train learning rate. During meta-test, we expect the updated parameters φ to exhibit a good generalization performance on the meta-test defenses, which indicates that J (φ ;F, D) is also maximized. The meta-train and meta-test can be optimized simultaneously as Algorithm 1 Model-Agnostic Meta-Attack (MAMA) algorithm Require: A data distribution D over (x, y) pairs, a set of defense models F = {fj} N j=1 , the objective function J in Eq. (5), attack iterations T , batch size B, learning rates β, γ, balancing hyperparameter µ. Ensure: The parameters φ of the learned optimizer.
1: for iter = 1 to MaxIterations do 2: Randomly draw N − n defenses from F to getF, and letF = F\F consist of the other n defenses; 3: for fr inF do Meta-train 4: Randomly draw a mini-batch For (x r i , y r i ) ∈ Br, computex r i t from φ for defense fr by Eq. (3) for t = 1 to T ; 6: end for 7: Compute J (φ;F, D) using the defenses fr inF and the corresponding mini-batches Br; 8: for fe inF do Meta-test 10: Randomly draw a mini-batch For (x e i , y e i ) ∈ Be, computex e i t from φ for defense fe by Eq.
(3) for t = 1 to T ; 12: end for 13: Compute J (φ ;F, D) using the defenses fe inF and the corresponding mini-batches Be; 14: φ ← φ + γ∇ φ (J (φ;F, D) + µJ (φ ;F, D)); 15: end for where µ is a hyperparameter to balance the meta-train loss and the meta-test loss. The objective (6) is optimized by gradient ascent. After accumulating the gradients from meta-train and meta-gradients from meta-test, the optimizer is trained to perform well on attacking both meta-train and meta-test defenses. The gradient and meta-gradient flows in the computation graph are illustrated in Fig. 1(b). We summarize the overall MAMA algorithm in Alg. 1.

EXPERIMENTS
In this section, we present the experimental results to validate the effectiveness of BMA and MAMA for adversarial attacks. We follow a simple-to-complex procedure to conduct experiments. Sec. 4.2 first shows the advantages of the learned optimizer in BMA when considering one defense at a time. Sec. 4.3 then proves that the learned optimizer possesses good cross-data, cross-iteration, and cross-model generalization abilities, respectively. Sec. 4.4 demonstrates that BMA and MAMA can achieve state-of-the-art performance by comparing with various top-performing attacks on 12 defense models. More details about the baseline attacks and defenses are provided in Appendix C and D, respectively. We mainly present the results based on the ∞ norm in this section. The results based on the 2 norm and standard deviations of multiple runs are provided in Appendix E.

EXPERIMENTAL SETUP
The hyperparameters of baseline attacks are set according to the common practice or their original implementations. Specifically, in Sec. 4.2 and Sec. 4.3, we set the attack iterations as T = 20 and the fixed step size as α = 2/255 according to (Madry et al., 2018). The learned optimizer in our method also adopts the same setting for fair comparison. In Sec. 4.4, we adopt the official hyperparameter configurations of the baseline attacks since we assume that their hyperparameters have been tuned appropriately. Our BMA and MAMA attacks follow a similar setup with 100 iterations, 20 random restarts, and ODI initialization. In all experiments, we use a two-layer LSTM with 20 hidden units in each layer as the optimizer model. We set w t = 1 and λ t = 0.1 in Eq. (5) and study different λ t through an ablation study. We set the learning rates as β = 0.0001 and γ = 0.001 in MAMA with µ = 1.0. The mini-batch size is B = 32.

EFFECTIVENESS AND ADAPTIVENESS OF BMA
We first train a different attack optimizer for each defense by BMA. We consider four typical AT defenses on CIFAR-10 ( Comparison with hand-designed optimizers. Since most existing adversarial attacks adopt handdesigned optimizers, we first compare the performance of our learned optimizer with these handdesigned ones. Specifically, we adopt the CW loss (Carlini & Wagner, 2017) (Uesato et al., 2018;Gowal et al., 2019)). Besides, we also adopt a CNN-based adversarial example generation method (Adv-CNN) (Baluja & Fischer, 2017) for comparison. We report the classification accuracy of the four defenses against these attacks in Table 1. We can see that BMA with the learned optimizer leads to lower accuracy of the defense models than the existing attacks, demonstrating that BMA can discover stronger adversarial attacks automatically.
Ablation study of the prior-guided refinement strategy. We conduct an ablation study to investigate the effects of the prior-guided refinement strategy introduced in Sec. 3.1. Fig. 2 shows the accuracy of HYDRA along the training iterations of BMA with λ t := λ = 0.0, 0.01, 0.1, 0.5, and 1.0, respectively. After integrating the PGD update rule as a prior with λ > 0, the learned optimizer achieves faster convergence and better performance than that with λ = 0. The results indicate that the proposed prior-guided refinement strategy can benefit from the informative prior given by the PGD update rule and explore potential better update directions simultaneously. Moreover, as λ increases, to a certain extent the convergence tends to be faster, but the performance tends to be similar. Therefore, we select λ = 0.1 for all experiments in the training phase of BMA.
Adaptiveness. The learned optimizer is generally compatible with previous methods on improving adversarial attacks in different aspects. To verify this, we incorporate our method with PGD variants using the cross-entropy loss (PGD CE ) (Madry et al., 2018), the difference of logits ratio loss (PGD DLR ) (Croce & Hein, 2020b) and PGD DLR with varying step size (APGD DLR ) (Croce & Hein, 2020b), the MultiTarged (MT) attack (Gowal et al., 2019) that performs targeted attack for each class, and the output diversified initialization method (Tashiro et al., 2020) with 10 random restarts (ODI-R10). Note that these methods only change the loss function L or the starting pointx 0 , which do not affect the learning process of BMA. Table 2 shows the results of the baseline methods and their extensions by integrating with the learned optimizer in BMA. The results verify that integrating BMA can consistently enhance the performance of each method. The mechanism of combining BMA is concise and easy to implement, yet steadily improving the different attack methods.

GENERALIZATION EVALUATION
Cross-data generalization. We first evaluate the cross-data generalization ability of the learned optimizer in BMA on unseen data points. Specifically, we randomly select 1, 000, 2, 000, 5, 000, and all 10, 000 data points from the CIFAR-10 test set as the training set for BMA, respectively, and adopt the entire test set of CIFAR-10 for evaluation. The performance on the entire test set is shown in Fig. 3. We can see that the learned optimizers using different numbers of data points achieve similar attack performance on the four defense models, indicating that the data-driven procedure enables the learned optimizer to obtain a good generalization ability to new unseen data samples.
Cross-iteration generalization. We then evaluate the generalization ability of the learned optimizer in BMA to more attack iterations than training. Fig. 4 shows the curves of model accuracy w.r.t. the number of attack iterations ranging from 1 to 100. We also mark the number of iterations used for training BMA at 20. The results on four defense models verify that BMA can consistently outperform the baseline attacks with varying attack iterations. It demonstrates that the learned optimizers also have a good generalization ability across attack iterations.
Cross-model generalization. The training process may face inconvenience in practice when training a different optimizer for each defense. A better property of the learned optimizer is more generalizable for adversarial attacks on any unseen defenses. We thus consider the challenging and practical scenario -training the optimizer on several available defense models and evaluating its generalization ability to other unseen defense models. This setting requires the optimizer to learn more general knowledge for adversarial attacks. For choosing the defense models in training, we mainly consider using a set of diverse defenses. Thus we randomly sample four models to train the optimizer from TRADES (T)  , AWP (A) , Unlabeled (U) , Hydra (H) (Sehwag et al., 2020), and SAT (S) (Huang et al., 2020). MAMA randomly selects three defenses as meta-train and the other one as meta-test in each iteration. We compare MAMA with two BMA variants. The first one trains an optimizer for each training defense and chooses the best optimizer for evaluation (BMA bs ), and the second one optimizes Eq. (5) using the training defenses as an ensemble (BMA en ). Table 3 shows that MAMA is more effective than the two methods, indicating that the generalization ability of the optimizer can be improved by adopting the model-agnostic algorithm. The effect of meta-test components is listed in Appendix E.3.

COMPARISON WITH STATE-OF-THE-ART ATTACKS
In this section, we perform a large-scale experiment to compare the performance of our methods with various state-of-the-art attacks for robustness evaluation of 12 defense models on CIFAR-10. We consider both BMA that learns a different attack optimizer for each defense model and MAMA that only learns one generalizable optimizer adopting the model-agnostic training algorithm to test all defenses. We compare our methods with several typical and state-of-the-art attacks, including 1) PGD CE (Madry et al., 2018), 2) PGD CW , 4) APGD (Croce & Hein, 2020b) with the DLR loss, 3) MT, 5) AutoAttack (AA) (Croce & Hein, 2020b), and 6) ODI (Tashiro et al., 2020). PGD CE , Table 4: Classification accuracy (%) of different defenses on CIFAR-10 under various white-box attacks. The defenses marked with † adopt additional unlabeled datasets. The defenses marked with * adopt = 0.031 as originally reported, and the others adopt = 8/255. ∆ represents the difference between reported accuracy and testing accuracy by ours. We also present the complexity (forward times) of each attack method in blue.  PGD CW , and ODI use 100 iterations with step size 2/255 and 20 restarts. APGD also uses the same iterations and restarts. MT performs targeted attacks for 9 classes except the true one with 2 restarts and 100 attack iterations. We adopt the official implementation of AA. We integrate our methods with ODI to establish stronger attacks. Our methods also use 100 attack iterations with 20 ODI restarts. MAMA is trained on defense models {T,A,U,H} with the model-agnostic algorithm. Table 4 shows the classification accuracy of these defense models against different adversarial attacks. We also show the clean accuracy and the reported robust accuracy of these models in the corresponding papers. Our methods achieve lower robust test accuracy on all 12 defense models than the baseline attacks, including MT and AA, demonstrating that ours can discover stronger adversarial attacks automatically than the existing attacks with hand-designed optimizers. Meanwhile, our methods are not only stronger but also less complex than AA. Besides, the results also demonstrate that MAMA can achieve a comparable performance of BMA, showing that the learned optimizer has a good generalization ability to unseen defense models due to the involved model-agnostic training algorithm, and could be flexibly used to evaluate the robustness of future defenses. Robustness evaluation on other datasets. We directly apply the learned optimizer in MAMA on CIFAR-10 to evaluate adversarial robustness on CIFAR-100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009), without updating the optimizer by new datasets or defenses. We adopt = 8/255 on CIFAR-100 and = 4/255 on ImageNet. During testing, there is a significant shift between training and testing defense models for MAMA. Despite these difficulties, MAMA can consistently improve the performance on most defenses in Table 5. The results verify that MAMA obtains a good generalization ability for adversarial attacks on unseen datasets, benefiting from the data-driven learning procedure automatically.

Defense
Running time. Fig. 5 shows the running time of AA, ODI, and BMA (MAMA) based on a single RTX 2080-Ti GPU. BMA is slightly slower than the baseline ODI, but significantly faster than AA.

CONCLUSION
In this paper, we propose a Model-Agnostic Meta-Attack (MAMA) approach to automatically discover stronger attack algorithms. Specifically, we learn an optimizer in adversarial attacks parameterized by an RNN model, which is trained over a class of data samples and defense models to produce effective update directions for crafting adversarial examples. We further develop a modelagnostic learning approach with a gradient-based meta-train and meta-test procedure to improve the generalization ability of the learned optimizer to unseen defense models. Extensive experimental results demonstrate the effectiveness, adaptiveness, and generalization ability of MAMA on different defense models, leading to a more reliable evaluation of adversarial robustness.
A MORE DISCUSSIONS ON RELATED WORK Xiong & Hsieh (2020) propose a similar approach to ours that uses an RNN optimizer to learn attacks for adversarial training. The differences between our method and Xiong & Hsieh (2020) are three-fold. First, the motivations are different: we propose to discover stronger adversarial attack algorithms while they aim to learn robust models by adversarial training. Second, based on the different goals, the techniques are different: we develop a prior-guided refinement strategy and a model-agnostic training algorithm to improve the attacking performance and generalization ability of the RNN optimizer while they do not adopt these techniques and use a relatively smaller number of attack iterations (i.e., T = 10). Third, the empirical evaluations are different: we mainly conduct experiments to show the attack performance on various defense models while they mainly consider the defense performance of adversarial training. Overall, our method differs from Xiong & Hsieh (2020) in the aspects of motivation, technique, and experiment. Despite using a similar Learningto-Learn framework, our method has its own contributions in developing an effective attack method for evaluating adversarial robustness.

B EXPLANATION OF MAXIMUM A POSTERIORI
Assume that the output of the RNN optimizer is a random variable g t . The objective of the optimizer is to maximize the loss L atx t after applying g t , as in Eq. (3). By using the cross-entropy loss as L, we actually want to minimize p y (i.e., the probability of the true class) givenx t . We let p(y|g t ) = 1 py such that we want to maximize p(y|g t ). Thus p(y|g t ) can be viewed as the conditional likelihood. By assuming the prior distribution of g t to be a Gaussian one as p(g t ) ∼ N (sign(∇ t ), 1 2λt ), we can estimate the posterior distribution of g t using Bayes' theorem p(g t |y) ∼ p(y|g t )p(g t ).
Under the maximum a posteriori (MAP) framework, we would like to solve arg max log p(g t |y) = arg max{− log p y + λ t g t − sign(∇ t ) 2 2 } = arg max{L(f (x t ), y) − sign(∇ t ) 2 2 }. Be accumulating the loss at each step, we can obtain Eq. (5).

C TECHNICAL DETAILS OF ADVERSARIAL ATTACKS
In this section, we introduce more related backgrounds and technical details for reference.
PGD. One of the most commonly studied adversarial attacks is the projected gradient descent (PGD) method (Madry et al., 2018). PGD iteratively crafts the adversarial example aŝ where B (x) = {x : x − x ∞ ≤ } denotes the ∞ ball centered at x with radius , Π(·) is the projection operation, and α t is the step size at the t-th attack iteration. L could be the cross-entropy loss as L ce (f (x), y) = − log p y where p y = e zy / K i=1 e z i denotes the predicted probability of class y, or the margin-based CW loss (Carlini & Wagner, 2017) as L cw (f (x), y) = max i =y z i − z y .  propose to adopt the momentum optimizer (Polyak, 1964) in adversarial attacks, which can be formulated as

MI-FGSM.
where α is a fixed step size and µ is a hyperparameter.
NI-FGSM. Lin et al. (2019) propose to adopt the Nesterov accelerated gradient (Nesterov, 1983) in adversarial attacks, as tion. Specifically, we adopt the CW loss (Carlini & Wagner, 2017) and set = 0.1. We compare our learned optimizer (in BMA) with vanilla gradient descent (used in PGD (Madry et al., 2018)), momentum (Polyak, 1964) (used in MI-FGSM ), nesterov accelerated gradient (Nesterov, 1983) (used in NI-FGSM (Lin et al., 2019)), and Adam (Kingma & Ba, 2015) (used in (Gowal et al., 2019;Uesato et al., 2018)). For fair comparison, we set the attack iterations as T = 20 and the fixed step size as α = 0.025 in all methods. The learned optimizer in BMA is trained with 20 attack iterations on the entire test set of CIFAR-10. We report the classification accuracy of the two defenses including Overfitting (Rice et al., 2020) and Robustness (Engstrom et al., 2019) against these attacks in Table 7. BMA with the learned optimizer leads to lower accuracy of the defense models than the existing attacks with hand-designed optimizers, demonstrating the effectiveness of BMA based on the 2 norm.

E.2 RANDOMNESS EXPERIMENTS
To avoid the randomness caused by a single experiment, we report the standard deviations of multiple runs on CIFAR-10 in Table 8. We can see that the learned optimizers using different runs achieve similar attack performance on the four defense models including HYDRA (Sehwag et al., 2020), Pre-training , Overfitting (Rice et al., 2020) and FastAT (Wong et al., 2020). The results also demonstrate the stability of BMA to existing attacks. Effect of different µ. The meta-train and meta-test can be optimized simultaneously as Eq. (6), and µ is the hyperparameter to balance the meta-train and meta-test losses. To estimate the effect of different µ, we sample four models to train the optimizer from TRADES (T), AWP (A), Unlabeled (U), Hydra (H), and SAT (S), and test the performance on Pre-Training for different µ. We can see in Table 9, a proper value 1.0 gives the best performance, indicating that the meta-train and meta-test should be equally learned. Effect of different numbers of meta-train and meta-test. Apart from the results based on different hyparameters, we study the effectiveness for different numbers of meta-train and meta-test. In the experiments, we attempt to sample m domains as meta-train, and remaining n domains as meta-test, thus named Sm-Tn. We report the classification accuracy of the defense model on different numbers of domains in Table 10. We can see that the best performance can be obtained by setting m as 3 and n as 1, which is also adopted in all experiments of MAMA. We also attempt to apply the automatic step size scheduler (Croce & Hein, 2020b) to MAMA in Table 11, which adopts 100 iterations and 20 restarts based on ODI on the different defenses including TRADES , Pre-training , HYDRA (Sehwag et al., 2020). The results show that integrating the automatic step size does not further improve the overall performance for robustness evaluation. We think that combining MAMA with ODI can already converge to sufficiently good results for robustness evaluation and may not be improved by integrating other techniques such as automatic step size.