Pareto Adversarial Robustness: Balancing Spatial Robustness and Sensitivity-based Robustness

Adversarial robustness, which mainly contains sensitivity-based robustness and spatial robustness, plays an integral part in the robust generalization. In this paper, we endeavor to design strategies to achieve universal adversarial robustness. To hit this target, we firstly investigate the less-studied spatial robustness and then integrate existing spatial robustness methods by incorporating both local and global spatial vulnerability into one spatial attack and adversarial training. Based on this exploration, we further present a comprehensive relationship between natural accuracy, sensitivity-based and different spatial robustness, supported by the strong evidence from the perspective of robust representation. More importantly, in order to balance these mutual impacts of different robustness into one unified framework, we incorporate \textit{Pareto criterion} into the adversarial robustness analysis, yielding a novel strategy called \textit{Pareto Adversarial Training} towards universal robustness. The resulting Pareto front, the set of optimal solutions, provides the set of optimal balance among natural accuracy and different adversarial robustness, shedding light on solutions towards universal robustness in the future. To the best of our knowledge, we are the first to consider the universal adversarial robustness via multi-objective optimization.

against pixel-wise perturbations (normally within the constraints of an l p ball), and spatial robustness, i.e., robustness against multiple spatial transformations. To understand these two main robustness aspects, the computer vision and graphics literature provides a deeper insight. It shows that there are two main factors that determine the appearance of a pictured object [17], [18]: (1) the lighting and materials, and (2) geometry. Most previous works related to adversarial robustness only focused on the first factor [17] by examining pixelwise perturbations, e.g., Projected Gradient Descent (PGD) attacks [10], assuming that the underlying geometry stays the same after the adversarial perturbation. Only a small proportion of research works attempted to tackle the lessstudied second factor, such as Flow-based [17] and Rotation-Translation (RT)-based attacks [19], [20].
However, additional consideration about the spatial robustness is crucial for the universal robustness, the final goal we hope to accomplish in the robust generalization. One of the most important reasons is that sensitivity-based robustness, normally measured l p -distance, is not sufficient to maintain the perceptual similarity [17], [19]- [21]. Concretely, spatial attacks or geometric transformations result in small perceptual differences, but they yield large l p distances.
In order to achieve the universal adversarial robustness, we find that the key is a clear relationship among accuracy, sensitivity-based and spatial robustness. The trade-off between sensitivity-based robustness and accuracy has been revealed by a series of works [22]- [24], but the comprehensive relationships between spatial robustness and them are still unclear. Previous works [25], [26] ever explored this issue, but they only focused on Rotation-Translation spatial robustness, lacking considering Flow-based spatial robustness [17], [27]. Surprisingly, we find that Flow-based spatial robustness presents an opposite relationship compared with the one revealed previously, yielding the previous conclusion less comprehensive or reliable.
Based on this important finding, we start our exploration about clearer relationships between different robustness, and we eventually harmonize the conflicting relationships within them by the leverage of Pareto criterion [28]- [30], thus achieving an optimal balanced universal robustness. Recent work [24] owed the conflicting relationship within all to the overparametrization, while we uncovered it from the perspective of different shape-biased representations. Another work [31] examined the trade-off in the inference time, while we target on more comprehensive relationships between different robustness with a different methodology.
In this paper, we progressively unfold our research story arXiv:2111.01996v1 [cs.
LG] 3 Nov 2021 line. Firstly, we provide deeper insights into the robustness relationships by respectively investigating the two main spatial robustness branches, i.e., Flow-based spatial attack [17] and Rotation-Translation (RT) attack [20]. After uncovering their different impacts on the local and global spatial sensitivity, we propose the integrated spatial attack and spatial adversarial training, which can incorporate the comprehensive spatial vulnerabilities or robustness. Based on this understanding, we present a comprehensive relationship among accuracy, sensitivity-based robustness and two branches of spatial robustness by investigating their different saliency maps from the perspectives of shape-bias, sparse or dense representation. It turns out that the relationship between sensitivity-based and RT robustness is a fundamental trade-off, but sensitivitybased and Flow-based spatial robustness are highly correlated, providing a vital supplementary for previous conclusions. Thus, comprehensive relationships between accuracy and different robustness are not pure trade-offs, motivating us to introduce the Pareto criterion [28]- [30], the general multiobjective optimization principle, into the universal adversarial robustness analysis. The Paretor criterion develops an optimal balance between the interplay of natural accuracy and different adversarial robustness, leading to the universal adversarial robustness in a Pareto manner. By additionally incorporating the two-moment term that can capture the interaction between losses of accuracy and different robustness, we finally propose a bi-level optimization framework called Pareto Adversarial Training. The resulting Pareto front provides the set of optimal solutions that balance perfectly all the considered relationships, which outperforms other existing strategies. Our contributions are summarized as follows: • We reveal the existence of both local and global spatial robustness, and further propose the integrated spatial attack and spatial adversarial training, incorporating comprehensive spatial vulnerabilities. • We present comprehensive relationships within accuracy, sensitivity-based and different spatial robustness, supported by strong and intuitive evidence from the perspective of robust representation. • We incorporate the Pareto criterion into the adversarial robustness analysis, and the resulting Pareto Adversarial Training can optimally balance multiple adversarial ro-bustness, yielding universal adversarial robustness.

II. LOCAL AND GLOBAL SPATIAL ROBUSTNESS
To present the comprehensive relationships between accuracy and different adversarial robustness in the later part, we firstly provide a fine-grained understanding of the less studied spatial robustness. We summarize a flurry of related work about spatial robustness [17], [19], [20], [25]- [27] into two major branches: (1) Flow-based Attacks, and (2) Rotation-Translation (RT) Attacks. Specifically, we find that the former mainly focuses on the local spatial vulnerability while the latter tends to capture the global spatial sensitivity. Based on this finding, the integrated spatial attack and spatial adversarial training are further proposed, respectively.

A. Local Spatial Robustness: Flow-based Attacks
The most representative Flow-based Attack is Spatial Transformed Attack [17], in which a differentiable flow vector w F = (∆µ, ∆v) is defined in the 2D coordinates (µ, v) to craft adversarial spatial transformation. The vanilla targeted Flowbased attack [17] follows the optimization manner (κ = 0): is the classifier in the K-classification task. x w F is Flow-based adversarial example parameterized by flow vector w F . L f low measures the local smoothness of spatial transformation balanced by τ .
Interestingly, in our empirical study shown in the left part of Figure 1, it suggests Flow-based attack tends to yield local permutations among pixels in some specific regions irrespective of the option of τ , rather than a global spatial transformation based on their shapes. We analyze that this phenomenon is owing to two factors: 1) Local permutations, especially in regions where colors of pixels change dramatically, are already sufficiently sensitive to manipulate, demonstrated by our empirical results shown in Figure 1. 2) The optimization manner does not incorporate any sort of shape transformation information, e.g., a parametric equation of rotation, as opposed to vanilla Rotation-Translation attack, which we present in the following. Therefore, we conclude that Flow-based attacks tend to capture the local spatial vulnerability. Further, to design the integrated spatial attack, we transform Eq 1 into its untargeted version under cross entropy loss with flow vector bounded by an F -ball: where L CE θ (x, y) = log j exp f j θ (x) −f y θ (x). To maintain a uniform optimization form in our integrated spatial attack, we replace local smoothness term L f low in Eq. 1 with our familiar l p constraint, and leverage cross entropy loss instead of max operation suggested in [32]. Proposition 1 reveals the correlation between the two losses, indicating that the smooth approximation version of max operation in Eq. 1, denoted as L S θ , has a parallel updating direction with cross entropy loss regarding w F . Proof can be found in Appendix B.
Proposition 1. For a fixed (x w F , y) and θ, consider as the smooth version loss of Eq. 1 without local smoothness term, then we have

B. Global Spatial Robustness: Rotation-Translation Attacks
The original Rotation-Translation attack [19], [20] applies parametric equation constraints on 2D coordinates, thus capturing the global spatial information: To design a generic spatial transformation matrix that can simultaneously consider rotation, translation, cropping and scaling, we re-parameterize the transform matrix as a generic 6-dimensional affine transformation one, inspired by Spatial Transformer Networks [33]: where we denote A w RT as the generic 6-dimensional affine transformation matrix, in which each entry of w RT indicates the increment on different spatial aspects. For example, RT , w 23 RT ) determines the translation. Finally, the optimization form of the resulting generic and differentiable RT-based attack bounded by RT -ball is exhibited as: C. Integrated Spatial Attack The key to achieve integrated spatial robustness is to design an integrated parameterized sampling grid T w RT ,w F (G) that can wrap the regular grid with both flow and affine transformation, where G is the generated grid. We show our integrated approach as follows: Then we sample new x adv by T w RT ,w F (G) via the differentiable bilinear interpolation [33]. Note that w F has the same dimension as the grid G, which is different from the impact of two-dimensional translation parameters in w RT . Then the final loss function of the integrated spatial attack can be presented as: where η w is the crafted integrated spatial perturbation parameterized by w = [w F , w RT ] T , simultaneously considering both Flow-based and RT spatial sensitivity. Note that η w itself does not necessarily satisfy the l p constraint directly. For the implementation, we follow the PGD procedure [10], a common practice in sensitivity-based attacks. We consider the infinity norm of w and different learning rates for the two sorts of spatial robustness. Therefore, the updating rule of w in each iteration is: Figure 1, we can observe that our Integrated Spatial Attack can construct both local and global spatial transformations on images. Concretely, it can simultaneously yield local pixel-wise permutations and global shape transformation.
Then, we visualize the loss surface under this Integrated Spatial Attack leveraging "filter normalization" [34] as illustrated in Figure 2. We strictly follow the implementation from [34] to achieve the desirable visualization of loss landscape of our integrated adversarial attacks with respect to all the differentiable parameters w. Specifically, we view w F and w RT as two parameterized filters, which is analogical to the "filter normalization" technique proposed by [34]. In the left part of Figure 2, we adjust the initialization of variance of w, which then can provide a distant view of loss landscape before the optimization in Eq. 8. It exhibits a highly regular loss landscape, and its non-concavity w.r.t. only rotation and translation [20] has been tremendously improved. In the middle of Figure 2, we then provide a closer view of the loss landscape before the optimization and it shows a highly convex surface around the w to be optimized, facilitating the following optimization. In the right part of Figure 2, we also present the loss landscape around the maxima w * after the optimization in Eq. 8 of our integrated spatial attack, exhibiting a highly concave surface as well. In summary, the highly non-concave loss landscape with respect to only rotation and translation ever raised by [20], has been largely alleviated by considering both local and global spatial vulnerability. This integrated form smooths the optimization process, which guarantees the efficacy of our Integrated Spatial Attack.

D. Spatial Adversarial Training
As Eq. 9 incorporates the local and global spatial robustness simultaneously, it is natural to leverage it to construct Spatial Adversarial Training, which we deploy in Experiment IV-C.

III. RELATIONSHIP BETWEEN SENSITIVITY AND SPATIAL ROBUSTNESS
In this section, we will empirically investigate the relationships between different robustness, and then explain them from the perspective of shape-based representation by leverage of saliency map.

A. Relationships
We conduct our rigorous experiments on MNIST, CIFAR-10 and Caltech-256 datasets, in order to empirically examine the tendency of both local and global spatial robustness as the sensitivity-based robustness increases. Specifically, after adversarially training multiple PGD (sensitivity-based) robust models under different numbers of PGD iterations, we further compute their test accuracy under Flow-based and RT-based spatial attacks via methods proposed in Section II, respectively. The accuracy is computed on correctly classified test data for the considered model to mitigate the impact of the slightly different generalization of these PGD trained models. We fix both F and RT as 0.3 on MNIST, and choose F and RT as 0.3, 1.0 on CIFAR-10 and Caltech-256. Then by adjusting the number of iterations in Flow-based and RT-based spatial attacks, we can control their strength of perturbations.
In Figure 3, the X-axis implies adversarially PGD trained models under different numbers of PGD iterations, which can measure the different strength of a model's PGD (sensitivitybased) robustness. The Y-axis represents the computed test accuracy of corresponding PGD trained models under different spatial attacks, and a high-level test accuracy reflects a model's high spatial robustness. It turns out that Flow-based spatial 3*'7UDLQHG0RGHOVXQGHU'LIIHUHQW,WHUDWLRQV robustness (red lines) presents a steady ascending tendency across three datasets as the PGD sensitivity-based robustness increases, while the trend of RT-based spatial robustness (blue lines) fluctuates conversely. This result reveals that the sensitivity-based and RT-based spatial robustness is a tradeoff relationship, consistent with previous conclusion [25], [26]. However, this trade-off does not (even on the contrary) apply to the local spatial sensitivity, where sensitivity-based and Flow-based spatial robustness is positively correlated. We provide the strong and intuitive evidence from the perspective of shape-biased representation in the following.
Natural PGD AT Flow AT RT AT

B. Explanation from the Shape-bias Representation
We show first with our brief conclusion: the sensitivitybased robustness corresponds to the sparse and shape-bias representation [35], [36], indicating that sensitivity-based robust models rely more on the global shape in prediction rather than the local texture. Nevertheless, the local and global spatial robustness are associated with different representation manners, respectively.
To examine the shape-biased representation, we visualize the saliency maps of naturally trained, PGD, Flow-based and RT adversarially trained models on some randomly selected images on Caltech-256, which are exhibited in Figure 4. Specifically, visualizing the saliency maps aims at assigning a sensitivity value, sometimes also called "attribution", to show the sensitivity of the output to each pixel of an input image. Following [35], [36], we leverage SmoothGrad [37] to calculate saliency map S(x) of an image x, which alleviates the noises in gradient by averaging over the gradient of n noisy copies of an input: where x i = x + q i , and q i are noises drawn i.i.d from a Gaussian distribution N (0, σ 2 ). In our experiment, we set n = 100 and the noise level σ/(x max − x min ) = 0.1. Figure 4 manifests that PGD trained models tend to learn a sparse and shape-biased representation among all pixels of an image, while two types of spatially adversarially trained models suggest converse representation. In particular, the representation from the Flow-based training model presents a noisy and shape-biased one as it places extreme values, albeit being noisy, on the pixels around the shape of objects, e.g., the edge between the horse and the background shown in Flow AT in Figure 4. On the contrary, RT-based models have less reliance on the shape of objects, and at the meantime, the saliency values tend to be dense, smoothly scattering around more pixels of an image.
Quantitatively, we calculate the distance of saliency maps from different models across all test data on Caltech-256 dataset, and then compute their skewness in Figure 5. Specifically, we compute the pixel-wise distance between the saliency maps of two models, and then we calculate the median of skewness of the saliency map difference among all test data. Note that if two saliency maps have no statistical difference, then the difference values will follow a symmetric normal distribution with skewness 0. A negative skewness indicates that the original saliency map (representation) tends to be sparse compared with a considered model. We plot the tendency of skewness as the strength of some specific robustness increases in Figure 5. We summarize the observations into two conclusions: 1) Based on the first and forth sub-pictures, both PGD and Flow-based robust models tend to learn a sparse and shape-biased representation compared with the natural model, but the Flow-based trained model is less sparse (we call it noisy shape-biased) in comparison with the PGD trained one. 2) On the contrary, RT-based robust models have the trend to learn a dense representation, which is also intuitive as the RT trained model is expected to memorize broader pixel locations to cope with potential rotations and transformations in the test data. Overall, the divergent representation (sparse vs dense) between RT-based and sensitivity robustness verifies their tradeoff shown in Figure 3 is fundamental. More importantly, the positive correlation of sensitivity-based and local spatial robustness, which is exhibited in Figure 3, can also be explained by their similar shape-biased representation, although the latter tends to be noisy. We provide a sketch map in the Appendix C to illustrate their relationships.

IV. PARETO ADVERSARIAL ROBUSTNESS
A. Motivation Pareto Optimization. Based on the aforementioned analysis on the relationships between natural accuracy and different kinds of adversarial robustness, a natural question is how to design a training strategy that can perfectly balance their mutual impacts, which mainly sources from their different representation manners. In particular, in most cases their relationships exhibit trade-off ones, except the positive correlation between the sensitivity robustness and local spatial robustness. In the multi-objective optimization, we are normally required to involve multiple criteria or objective function to be optimized simultaneously, which has been applied in various fields in science, such as engineering and economics. Pareto optimization, serving as the key part in multi-objective optimization [38], can provide optimal solutions especially in the presence of multiple conflicting objectives. Therefore, in order to harmonize these competing optimization objectives in the context of adversarial robustness, we introduce Pareto Optimization [28], [39], [40], which has been successfully applied when optimal decisions need be taken in the presence of trade-offs between multiple conflicting objectives.
In particular, Pareto optimization endeavors to achieve Pareto Optimality, a balanced situation between all objectives, where none of the objective functions can be improved in value without degrading some of the other objective values. In addition, in the pareto optimization, a Pareto front will be developed, consisting all the optimal solutions given some specific restraints. For instance, in finance, after the Pareto optimization, Efficient Frontier will be constructed to show the best combinations of risk and expected return that are available. The resulting Pareto front after Pareto optimization is the set of Pareto optimal solutions, and each of them perfectly balances all objectives given a specific constraint.
Based on the knowledge of Pareto optimization and our issue of balancing multiple adversarial robustness, a natural choice is to leverage Pareto optimization into the adversarial training. The resulting Pareto Front can describe the optimal combinations of available losses, which can provide a promising solution to address the comprehensive adversarial robustness issue. Limitations of Existing Strategies. Given perturbation sets S i , i = 1, ..., m, and its corresponding adversarial risk R adv (f ; S i ) := E (x,y)∼D [max r∈Si L(f (x + r), y)], our goal is to find f θ that can achieve the uniform risk minimization across all S i as well as the minimal risk in the natural data. There are two common strategies to handle this issue. Intuitively, Max AT may overfit to one specific type of adversarial robustness if its adversarial attack used for training is too strong. Figure 6 demonstrates that as the strength of PGD attack used in Max AT increases, the comprehensive robustness of Max AT degenerates to a single PGD adversarial training, owing to the fact that the PGD loss tends to dominate among all losses.
In Figure 6, we draw the difference of robust accuracy between Max AT and single PGD adversarial training. It can be observed that as the attack for in PGD robustness in Max AT increases, the difference of three kinds of robust accuracy between Max AT and a single PGD AT tends to vanish. Further, we consider the generalization issue based on different risks, and then set the risk in Max AT and Ave AT as Proposition 2 illuminates that Max AT is closely associated with some form of Ave AT. This indicates that Max AT is likely to perform similarly with the specific form of Ave AT, which also suffers from unsatisfactory solutions when the strength of different attacks mixed in the training are imbalanced.
Proposition 2. Given KKT differentiability and qualification conditions, ∃λ i ≥ 0, such that the risk minimizer in Max AT, i.e., R max is a first-order stationary point of Si∈S λ i R Si regardless of the relationship of S i .
Although both Ave AT and Max AT may fail in certain scenarios, adversarial training with linear combinations between various adversarial losses still enjoys a more general form. Based on this form, adversarial training with an optimal combination has the potential to achieve better performance over Max AT and Ave AT, alleviating from overfitting issues. Our further proposed Pareto Adversarial Training provides strong evidence to support this intuition.

B. Pareto Adversarial Training
The key to achieve Pareto optimal adversarial robustness is to find optimal combinations α between natural accuracy, sensitivity-based and spatial robustness to perfectly balance their mutual impacts during the whole training process. We denote α = (α 0 , α 1 , α 2 , α 3 ) as the combination coefficients on different losses. We use L nat , L PGD , L Flow and L RT represent the natural loss, the PGD adversarial loss, the Flow-based and the RT-based adversarial loss, respectively. Our goal is to train a model f θ under the optimal combinations α * of different losses, and the computation of α * in the training is also associated with different losses determined by model parameters θ. This implies a bilevel optimization problem with θ as the upper-level variable and α as the lower-level variable. In the construction of low-level optimization regarding α, we apply a two-moment objective function with respect to all losses. We name this bi-level optimization as Pareto Adversarial Training, which is formulated as: where L θ 0 , L θ 1 , L θ 2 , L θ 3 represent L nat , L PGD , L Flow , L RT respectively for simplicity, sharing the same model parameter θ. r indicates the expectation of one-moment over all robust losses, i.e., spatial and sensitivity-based losses, which reflects the strength of comprehensive robustness we require after solving this quadratic lower-level optimization regarding α. In particular, given the model parameter θ in each training step, the larger r we require will push the resulting α i , i = 1, 2, 3 larger, thus putting more weights on the robust losses rather than the natural loss to pursue more robustness. Two-Moment Objective Function. The two-moment form is a common practice in the Pareto optimization. For example, in the financial portfolio theory, the mean-variance optimization is normally leveraged to compute the Pareto Efficient Front, where the risk of asset portfolio, measured by their variances, is minimized to balance different correlations of these assets given an expected return from the investor. Similarly, the square loss of the difference between each loss pair in Eq. 11 measures their mutual impacts. For instance, the descending of L PGD tends to increase L RT as they have a fundamental tradeoff relationship. We hope to mitigate all these mutual impacts, measured by the weighted quadratic difference, among all losses given an expected robustness level r we hope to achieve. In the implementation, as we regard all losses as random variables with its stochasticity arising from the mini-batch sampling from data, we leverage sliding windows technique to Given the current α, update f θ by descending its stochastic gradient of: Evaluate P in the quadratic form shown in Eq. 12. 10: Solve Eq. 12 via CVXOPT tool to obtain the α. 11: until Convergence compute their expectations. Our bi-level optimization within a batch is: (1) θ: update parameters θ via SGD (2) α: solve α via quadratic programming. Denote the random variables L 0 , L 1 , L 2 , L 3 with mean vector µ and covariance matrix Σ. We transform our lower-level optimization regarding α as the following standard quadratic form: where P = 8(diag(Σ) + diag(µµ T )) − 2(Σ + µµ T ). We utilize the CVXOPT tool to solve this quadratic optimization within each mini-batch training. We provide the proof about the quadratic formulation in Appendix E. A detailed algorithm description is given in Algorithm 1. In the lower level procedure of Pareto adversarial training, we solve the quadratic optimization regarding α given θ in each training step, to obtain the optimal combinations among natural loss, sensitivity-based and spatial adversarial loss. Then in the upper level optimization, we leverage our familiar SGD method to update θ based on α * calculated from the lower level problem. Note that the computation complexity of our method is similar to Ave AT and Max AT, which is still competitive in computation.

C. Pareto Front
Implementation Details. For MNIST comparison, we trained the Simple CNN in [22] on MNIST for 100 epochs. As for the CIFAR-10 dataset, we choose the widely used Pre-Act ResNet-18 with grouped normalization and trained the network for 76 epochs. The other details of our implementation on MNIST and CIFAR-10 are based on [22], while the implementation on Caltech-256 has refer to [36] with 10 epochs to finetune a pretrained ResNet-18.
• PGD Attack. We apply the widely accepted setting on these three datasets. We set step size as 0.01, as 0.3 on MNIST while the step size is 0.007 and is 0.031 on both CIFAR-10 and Caltech-256 datasets. To evaluate the different levels of robustness, we conduct the evaluation of PGD attack under 10, 20, 30, 40 iterations on MNIST, and 5, 10, 15, 20 iterations on CIFAR-10 and Caltech-256 datasets. • Flow-based and RT Attacks. On MNIST, we set step size α F and α RT as 0.01 and 0.1, and choose F , RT as 0.3. We select 5, 10, 15, 20 as the attack iterations for the evaluation of both two attacks. On CIFAR-10, we set step size α F as 1e−3 and α RT as 0.05, and choose F , RT as 0.3, 1.0. We select 3, 5, 10, 15 as the attack iterations for the evaluation of both two attacks. On Caltech-256, we set step size α F as 1e−5 and α RT as 0.1, and choose F , RT as 0.3 and 1.0 for the two attacks, respectively. We select 3, 5, 10, 15 as the attack iterations for the evaluation of both two attacks. • PGD Adversarial Training. We choose PGD iterations as 30, 3, 5 in the PGD adversarial training on MNIST, CIFAR-10 and Caltech-256, respectively. The adversarial attack strength is the same as PGD attacks for each dataset respectively. • Spatial Adversarial Training. Our integrated spatial adversarial training is based on our proposed integrated spatial attacks that unifies both Flow-based an RT-based attacks. We set the iterations as 20, 5, 10 and on MNIST, CIFAR-10 and Caltech-256, respectively. Other hyperparameters are same as those in their corresponding attacks. By adjusting the upper bound of expected adversarial robustness loss r, we can evenly generate Pareto optimal solutions where the obtained models will have different levels of robustness under optimal combinations. The set of all Pareto optimal solutions then forms the Pareto front.
Concretely, we train deep neural networks under different adversarial training strategies, i.e., PGD Adversarial Training (PGD AT), Spatial Adversarial Training (Spatial AT) proposed in Section II-D, Max AT, Ave AT and Pareto Adversarial Training (Pareto AT) under different r, in which we apply a proper iteration, respectively.  use the increment over the naturally trained model as the metric, which we call Universal Robustness Score. We report Robustness Scores of all models on CIFAR-10 in Table I and results on the other two datasets are similar. It manifests that Pareto AT (r = 4.0) has the best universal robustness score over the other models, although the highest specific robustness normally exists in the adversarial training model that only focuses on it. Finally, we plot universal robustness scores and sacrificed clean accuracy of all methods across three datasets in Figure 7, where multiple Pareto AT models (red points) are trained under different r. The Pareto criterion exhibited in Figure 7 provides an optimality principle, by which Pareto Adversarial Training can achieve the best universal robustness compared with other methods, given a certain level of sacrificed clean accuracy we can tolerate. By adjusting expected universal robustness r in Pareto Adversarial Training, we can develop the set of Pareto optimal solutions, i.e., the Pareto front. It manifests that all other methods are above our Pareto front, and thus they are less effective than our proposal.
Note that the perturbation strength adopted in Table I is mild, but we need to point out that the superiority of Pareto AT over Ave AT can be enlarged if the overfitting issue is severe. We demonstrate this claim in Table II where we apply a stronger PGD perturbation used in AT, and finally Ave AT overfits to sensitivity robustness more severely, achieving much less universal robustness and sacrificing more clean accuracy than Pareto AT. Pareto Adversarial Training can mitigate the overfitting issue regarding an overly strong perturbation in AT as Pareto AT can automatically adjust the weights α while training, which is the key advantage of Pareto AT over Ave AT. Overall, Pareto adversarial training perfectly balances mutual impacts of sensitivity-based and spatial robustness under Pareto criterion.

V. DISCUSSION AND CONCLUSION
The principal purpose of our work is to design a novel approach towards the universal adversarial robustness. To achieve this goal, we firstly analyze the two main branches of spatial robustness and then integrate them into one attack and adversarial training design. Furthermore, we investigate the comprehensive relationships between sensitivity-based and two distinct spatial robustness from the perspective of representation. Based on the understanding of the mutual impacts of different kinds of adversarial robustness, we introduce Pareto criterion into adversarial training framework, yielding the Pareto Adversarial Training. The resulting Pareto front provides optimal solutions over existing baselines given the universal robustness level we hope to attain. In the future, we hope to apply Pareto analysis into more general Out-of-Distribution generalization settings.
Robustness Score (%) Natural Model Ave AT Pareto AT (r = 3.7) Natural

APPENDIX A VISUALIZATION OF VARIOUS ATTACKS
To better present the visual effect of various kinds of adversarial attacks, we provide a high-resolution results on Caltech-256 in Figure 8. It turns out that Flow-based attack focus on local spatial vulnerability that mainly blur pixels in some local regions, while RT attacks cause a shape-based global spatial transformation. More importantly, our integrated spatial attacks are more comprehensive in the sense of spatial robustness, combining both local and local spatial sensitivity.

APPENDIX B PROOF OF PROPOSITION 1
Proof. Firstly, we have the following equations according to the definitions of loss function: Then, we compute their gradients with respect to flow vector x w F respectively. The gradient of L CE θ (x w F , y) is shown as follows: Similarly, the gradient of L S θ (x w F , y) is: Then we take the multiplication of ∇ w F L S θ (x w F , y) by a term K i =y exp(f i θ (xw F )) K i=1 exp(f i θ (xw F )) , finally we attain: as r(x w F , y).

APPENDIX C A SKETCH MAP ABOUT THE RELATIONSHIPS BETWEEN SPATIAL AND SENSITIVITY-BASED ROBUSTNESS
As shown Figure 9, we set the spatial robustness as the X-axis and sensitivity-based robustness as the Y-axis, respectively. As the fundamental trade-off arising from the representation occurs between RT-based spatial robustness and PGD robustness, we place them in a perpendicular position. By contrast, flow-based robustness is set in the intersection of two axis because this kind of local spatial robustness is highly overlapped with sensitivity-based robustness. More importantly, we point out that the comprehensive adversarial robustness is between the two directions of both spatial and sensitivity-based adversarial robustness, which will be the more important research direction we need to adjust in the future.

APPENDIX D PROOF OF PROPOSITION 2
Proof. Let f as the minimizer, e.g., the neural networks after the optimization: Then the optimization can be equivalent to a constrained version: We point out that our conclusion is made under the assumption that the KKT condition holds and the stationary point of f regarding the Lagrangian function can be attained, which normally requires the convexity condition. However, under these assumptions, we can still establish the close correlation between Max AT and Ave AT, indicating they are likely to perform similarly in many cases.

APPENDIX E OPTIMIZATION ANALYSIS ON THE PARETO ADVERSARIAL TRAINING AND ALGORITHM
We provide the proof of P in the following: