Dominant Patterns: Critical Features Hidden in Deep Neural Networks

In this paper, we find the existence of critical features hidden in Deep NeuralNetworks (DNNs), which are imperceptible but can actually dominate the outputof DNNs. We call these features dominant patterns. As the name suggests, for a natural image, if we add the dominant pattern of a DNN to it, the output of this DNN is determined by the dominant pattern instead of the original image, i.e., DNN's prediction is the same with the dominant pattern's. We design an algorithm to find such patterns by pursuing the insensitivity in the feature space. A direct application of the dominant patterns is the Universal Adversarial Perturbations(UAPs). Numerical experiments show that the found dominant patterns defeat state-of-the-art UAP methods, especially in label-free settings. In addition, dominant patterns are proved to have the potential to attack downstream tasks in which DNNs share the same backbone. We claim that DNN-specific dominant patterns reveal some essential properties of a DNN and are of great importance for its feature analysis and robustness enhancement.


Introduction
Deep Neural Networks (DNNs) are proved as susceptible to adversarial perturbations [29,7,15,17,2,27]. The basic procedure of attacking a well-trained DNN f is to generate an imperceptible perturbation δ from an image x, such that the perturbation could change the prediction of f . Mathematically, the perturbation is dependent on x and f , which could be denoted as δ(x, f ), and satisfies where the latter constraint ensures that the perturbation is imperceptible by restricting it in an p norm bound ξ. By contrast, the goal of defenders (also network trainers) is to train a network insensitive to any imperceptible perturbations, that where X is the dataset that f works on.
Typically, δ is regarded as a special noise, which has seldomly been treated as a meaningful signal. However, if we drop the prejudice that "the images are signal, while the perturbations are noise", we will discover that the output of the DNN is actually determined by x and δ jointly. In other words, although the amplitude of δ is too small to be perceived by human beings, it could still be meaningful and contain much information from the perspective of f , and sometimes even has a more significant effect compared to x. With this idea, corresponding to equation (2), we come up with a question:  Figure 1: An example of dominant pattern. The dominant pattern is generated from a given DNN and has been mapped from [-10,10] to [0,255] for better observation.
Given an f , is there a perturbation δ that can provoke a high response from f and make it insensitive to natural images x? That is, for a well-trained network f , can we find a δ(f ), satisfying f (δ(f ) + x) = f (δ(f )) and δ(f ) p ≤ ξ, ∀x ∈ X.
To some extent, equation (3) can be regarded as a "dual" version of equation (2). For a DNN f , it is actually pursuing a δ(f ) that can dominate DNN's output and thus we name it as dominant pattern.
The dominant pattern is small and could be ignored in magnitude but is a critical feature hidden in DNN. Theorically, dominant patterns are data-independent and DNN-specific which could reflect some essential properties of DNN. Therefore it is a very valuable things and is helpful for network analysis, feature analysis, and perhaps robustness enhancement.
In this paper, we design an algorithm to find the dominant pattern for a given DNN. The crucial idea is to keep the output insensitive to x ∈ X, so that make δ more relevant to the nature of DNN .
Here, we achieve such insensitivity by optimizing a designed Cosine-Similarity Loss (CoSLoss) to encourage the similarity between f (δ) and f (δ + x).
As an example, we demonstrate δ(f ) and its influence on the DNN in classification task in Figure  1, where δ is the dominant pattern found in a well-trained VGG19 [26] model on ImageNet. The pattern is predicted as "iguana", and added with it, the original images are amazingly all mispredicted as "iguana" by VGG19.
A direct application of dominant patterns is the universal attacks. Such attacks generate a set of adversarial examples by crafting only one perturbation, named the Universal Adversarial Perturbation (UAP), which could be realized by a dominant pattern. Comprehensive experiments show that dominant patterns outperform state-of-the-art (SOTA) UAP methods. Actually, a UAP pursues a perturbation to affect the output, i.e., to let the output be a function of both x and δ, while the dominant pattern pursues to determine the output, i.e., the output only depends on δ. This essential difference between them accounts for the gap in their performance.
Notably, the dominant pattern is a unique nature of DNNs instead of that for specific data. On one hand, the dominant pattern can be calculated from any unlabeled image and perform significantly better than previous label-free UAP methods. On the other hand, any downstream task that uses the DNN's backbone, i.e., the main part of this DNN, will also be controlled by the dominant pattern. The special properties of dominant patterns make them intrinsic and critical features hidden in DNNs, and provide a new viewpoint for understanding DNN's vulnerability and enhancing their robustness in the future.
The major contributions of our work can be summarized as: • We raise an idea that a well-trained DNN can be controlled by a special perturbation named dominant pattern. We verify the existence of dominant patterns by designing an algorithm based on cosine similarity to pursue insensitivity in the feature space.
• Since dominant patterns are independent of data, any unlabeled natural image set can be used in our algorithm. We substantiate that when dominant patterns are used as UAPs, they can outperform several SOTA methods, especially in label-free settings. 2 • We verify the dominance of these patterns when transferred to downstream tasks, indicating that dominant patterns are actually intrinsic features hidden in DNNs as a potential helper for model analysis and robustness enhancement.

Related Work
Recent studies have shown that DNNs are susceptible to maliciously constructed small noise called adversarial perturbations. In this section, we roughly classify them as image-specific attacks and image-agnostic ones as described below.

Image-specific attacks
Szegedy et al. [29] first demonstrates the existence of small perturbations that can fool DNNs. Since then, many attacks have been proposed, including Fast Gradient Sign Method (FGSM) [7], Projected Gradient Descent (PGD) [15], C&W [2], and DeepFool [17]. The general attack procedure is to optimize the input sample to adversarially maximize the loss, while restricting the magnitude of perturbations. It is also demonstrated that the adversarial examples generated by attacking a DNN may transfer to other DNNs as well [6,32,3]. These attacks are image-specific because they craft different perturbations for different samples.

Image-agnostic attacks
Image-agnostic attacks, in contrast, change a DNN's predictions on different inputs with a single imperceptible universal adversarial perturbation (UAP). The first UAP is proposed by Moosavi-Dezfooli et al. [16], where an iterative procedure based on Deepfool [17] is designed. Mopuri et al. [20] propose a Network for Adversary Generation (NAG) to model the distribution of adversarial perturbations. Omid Poursaeed et al. [25] present a framework named GAP which can craft universal perturbations for both classification and semantic segmentation tasks. Different to previous methods, GAP can achieve targeted attacks. The methods mentioned above require the knowledge of original training data, which may not be feasible in practice. Thus, some researchers try to train universal perturbations without using any training data [19,18,21,14]. In [19] and [18], the perturbations are generated via corrupting the extracted features at multiple layers. By contrast, [21] trains a generative model to craft UAPs by utilizing the class impressions, which is a generic representation of the samples belonging to certain categories, while [14] considers the model uncertainty to craft an insensitive universal perturbation. Recently, more types of UAPs have come out to fit different demands. Philipp Benz et al. [33] propose a framework tailored for the class discriminative universal attack, where the perturbation fools a target network to misclassify only a chosen group of classes. Targeted UAPs can also be found by exploiting a proxy dataset instead of the original training data [34].

Discovering the dominant pattern
In this section, we formally define dominant patterns and then propose an algorithm to find them.
As the name suggests, the dominant pattern is a small perturbation δ that can determine the output , where X can be any unlabeled natural image set with or without intersection on the target DNN's training set. We keep the output feature of DNN unchanged by solving the optimization problem as maximize where ξ is a small value ensuring the quasi-imperceptibility of δ, and L is a loss function used to measure the similarity between vectors as discussed in detail later.
Since the constraint function δ p ≤ ξ can be easily realized by clamping the variable's value during training, the key to the algorithm falls into how to quantify the similarity between f (δ) and f (δ + x). It is worth noting that feature f (·) is a high-dimensional vector. Typically, there are several metrics to measure the similarity between vectors, such as cosine similarity, Euclidean distance, and Kullback-Leibler divergence, which are defined as follows.

Cosine Similarity Loss (CoSLoss).
Cosine similarity is the cosine of the angle between vectors. It treats the input vectors a, b as normalized, which is a judgment of orientation and not magnitude, and its outcome is bounded in [0, 1]. The higher the cosine similarity is, the more similar the two vectors are. Therefore, the CoSLoss can be defined as where n denotes the dimension of a and b, a i and b i denote the i-th dimension of the vector a and b, respectively.

Euclidean Distance Loss (EDLoss)
. Euclidean distance is defined as the length of the line segment between the two points. The loss according to this similarity metric can be directly designed as 3. Kullback-Leibler Divergence Loss (KLDLoss). If we treat the input vectors a and b as two probability distributions, then Kullback-Leibler divergence is defined to measure the similarity between two probability distributions. KLD is an asymmetric measurement function used to measure how well a probability distribution a fits the target distribution b.
In our work, we regard f (δ) as the target distribution, since δ is the main part we optimized.
Theoretically, CoSLoss is a better choice, because EDLoss is easily affected by magnitude, while KLDLoss is an asymmetric metric thus is not easy to optimize. To verify our choice, we choose a classification task to compare the impact of different losses on feature similarity. We define a metric called dominance ratio, i.e. the ratio of perturbed images (δ + x) that are predicted as the dominant pattern's class, to measure the performance of these losses. We attack VGG19 [26] on ImageNet with three mentioned loss, which is calculated at the logit layer of DNNs since it is highly related to the dominance ratio. In Table 1, we list the dominance ratio of patterns found by minimizing different losses (the detailed setting could be found in Section 4). Because cosine similarity outperforms the others, we choose it as the objective in the algorithm.
The optimization is iteratively proceeded over training data to minimize the CoSLoss. Adam [10] optimizer is applied and the perturbation is clamped by ∞ norm in each iteration to satisfy the magnitude requirement. The details are summarized in Algorithm 1.

Algorithm 1 Finding dominant patterns for a given DNN
Input: DNN f , loss function L CoS , training set X, perturbation magnitude ξ, batch size b, maximum number of epochs m, learning rate for Adam optimizer lr.

Experiments
In this section, we first verify the existence of dominant patterns, showing that the proposed algorithm can indeed find patterns dominating DNNs. Further, the layer-wise output of a DNN is visualized to better understand the dominance process. After that, we demonstrate that dominant patterns can be applied to UAP tasks and defeat state-of-the-art methods. Moreover, we discuss its superiority over UAP methods in terms of their dominance and task transferability.
In our experiment, the public datasets we used include validation set of ImageNet ILSVRC2012 [4] (50, 000 images), MSCOCO 2017 [13] (118, 287 images) and CIFAR10 [11] (50, 000 images). Among them, the ImageNet validation set is not divided into training and test sets, therefore we randomly split it into two parts, where 40, 000 pieces are used for training, and the rest 10, 000 are used for testing. We also create an artificial dataset with 4, 000 images in four classes (1, 000 per class) by using BigGAN [1] of which the official model can only generate four classes.
In UAP tasks, comprehensive baselines are chosen for comparsion, including UAP [16], GAP [25], NAG [20], FT-UAP [5], TUAP [34], FFF [19], AAA [21], GD-UAP [18], and PD-UA [14]. For fair comparison, we inherit all hyperparameter settings from the respective papers, and the generated dominant patterns and UAPs are restricted with δ ∞ ≤ 10 for images in range [0, 255]. To evaluate fooling performance, the most common metric Fooling Rate is used, which denotes the proportion of test images for which the prediction of the target DNN is changed after adding the perturbation.

Dominant patterns
We first implement Algorithm 1 on four representative DNNs with ImageNet validation set and COCO 2017 training set. The calculated dominant patterns are shown in Figure 2. Since the max magnitudes of the results are restricted to ξ = 10 which is nearly invisible, we map the magnitudes from [-10,10] to [0,255] for better observation.
Visually, the found patterns are regular textures, which contain certain information. According to Table 2, although such a pattern is meaningless compared with natural images, the corresponding DNN identifies it as a certain class (named dominant class) with very high confidence. We repeat the experiments several times and find a stable dominant class for each setting. Quantitatively, their dominance ratio (defined in Section 3) are also reported. After adding a dominant pattern, more than three-quarters of images are misled to the dominant class, which originally belong to 1,000 classes. The high dominance ratio demonstrates that the generated dominant patterns successfully attract DNN's attention and take control of its output.
By taking a closer look at the respective performance of dominant patterns trained on ImageNet and COCO, we find that for VGG19 and RN152, their dominant classes have not changed when given different data prior information. However, that is not true for the other two DNNs. Actually, the training procedure of dominant patterns depends on training data. Since the distribution of training set varies from each other, dominant patterns trained with them may belong to different classes.
To investigate how these seemingly meaningless patterns dominate the DNNs, we visualize the layer-wise output of VGG19 corresponding to a randomly chosen natural image x, the dominant pattern δ, and the perturbed image δ + x in Figure 3. The horizontal axes denotes the ReLU layer index. Obviously, along with the forward propagation from input to output, the dominant pattern embedded in δ + x gradually seize the control role such that the features become less related to x and more similar to δ. As a numerical comparison, the Structural SIMilarity [31] (SSIM) of δ + x v.s. δ and that of δ + x v.s. x are given.  The above is not a cherry-picked case. We repeat this experiment for all the 10,000 images in the test set and report the average SSIM with standard deviations in Figure 4, from which one could observe a similar trend. This phenomenon indicates that the damage of dominant patterns permeates through the whole DNN, and has already been accomplished at the output of the backbone (the 16 th ReLU layer). That is to say, dominant patterns destroy DNNs by disabling their ability to extract meaningful features from natural images. Since feature extraction is a key in the computer vision, this inspires us that dominant patterns are attributes for DNNs and can serve as an all-in-one perturbation for DNNs with the same backbones in different tasks.

Performance in UAP task
In a universal attack task, one seeks a universal perturbation δ that can change the prediction of most images from dataset X, i.e., the attackers find a δ p ≤ ξ that satisfies Compare this equation with the dominant pattern's definition in equation (3), it can be easily found that the dominant patterns can also change DNN's prediction. Therefore, they can be applied to UAP tasks. We discuss its performance from two aspects according to the type of training set used.
Train with original data. The most attack-friendly situation is to train UAP on labeled data that is also used for training the DNNs, which is called label-dependent attack. In this case, UAPs could achieve high fooling performance. The fooling rates calculated by applying different attack methods to different DNNs are shown in Table 3, where the best fooling rates are highlighted in bold, and the second-best ones are underlined. For comparative experiments whose results are not provided in the corresponding papers, we mark them with '-'. From the table, it can be concluded that our dominant patterns defeat the SOTA method. Figure 5 shows some adversarial examples generated from dominant patterns for different DNNs.  use an unlabeled proxy dataset that is irrelevant to target models. Such data can provide data prior information, leading to label-free attacks. Some attackers also attempt to conduct data-free attacks, where no data is utilized during training.
According to Algorithm 1, finding dominant patterns is a self-supervised process thus does not requires any data labels, which satisfies a label-free setting. We use MSCOCO [13] and artificial images generated by BigGAN [1] as our proxy dataset. Details of these datasets are stated in Section 4.1. The fooling rates are reported in Table 4, where the performance of data-free UAPs (row 1-4) and label-free UAPs (row 5-7) are given for comparison. Combined with Table 3, it can be concluded that dominant patterns obtained from proxy data not only outperform all label-free and data-free attacks, but also are comparable to SOTA label-dependent attacks. The outstanding performance further enhances the practicality of dominant patterns. Notice that, the performance gap between COCO and BigGAN is mainly due to the lack of diversity in BigGAN pictures (with only four classes).   Table 5. Here, the top-k ratio means the overall proportion of top k most frequently occurred classes.
Although it has been discovered in [16] and [30] that UAPs tend to aggregate images to some dominant labels, dominant patterns gather perturbed images to several certain classes to a large extent that most UAPs cannot achieve, as shown in Table 5. This verifies our claim that "UAPs are not dominant patterns". The direction aggregation implies that DNNs' vulnerability is class-dependent and there are certain flaws that are extremely fragile. TUAP is a newly proposed method that can take into account the multiple dimensions of the logit vector to fully utilize information. It satisfies some of the conditions for obtaining dominant patterns, so that it can perform well on dominance ratio.
Dominant patterns can be transferred to downstream tasks. According to Figure 3 and dominant patterns' definition, even the features extracted by the backbone of DNNs can also be controlled by dominant patterns. Theoretically, this phenomenon will be kept if the backbone of a pre-trained DNN remains unchanged in the downstream tasks. However, when using the pre-trained network, it is common for trainers to freeze the parameters of the backbone and then train a head for downstream tasks. This is a potential risk and can be exploited by our dominant patterns.
To verify this assumption, we transfer the VGG19, which pre-trained on ImageNet, to CIFAR10 classification task by freezing the parameters in convolution layers (backbone) and fine-tune the ones in fully connected layers (classifier head). And then we use the dominant pattern obtained from the pre-trained VGG19 to attack the new DNN fine-tuned in CIFAR10. Consequently, the accuracy in CIFAR10 task is promoted to 87.02%, and the results are shown in Table 6. The dominant pattern still serves as a strong perturbation which shows extremely high confidence and dominance ratio. Note that since CIFAR10 is a 10-category classification task, the theoretical highest fooling rate is 90% because the clean samples which share the same prediction with the dominant pattern will not change their predictions after attacked.
More importantly, the reported CoSLoss remains at a very low level on the fine-tuned network. This indicates that the DNN is still in control of the dominant pattern, even though the task has already changed. This phenomenon further verifies that the proposed dominant pattern can attack any DNNs that use the same pre-trained backbone without any changes. It is also the reason why we call the dominant pattern "the unique nature of the DNN".

Conclusion
In this work, we provide a new perspective on perturbations, and propose that there are dominant patterns hidden in DNNs, which is a critical feature of DNN and can help understand and improve the DNN's robustness. Although small and invisible in the image space, dominant patterns have some meaningful patterns that have the ability to mislead the majority of natural images to their own classes. Through numerical experiments, we verify the existence of such pattern and visualize how its dominance takes place. Further, we demonstrate the outstanding performance of dominant patterns in UAP tasks, and analyze their particularity beyond the traditional attacks. In the future, it is possible for researchers to go deeper into the properties of dominant patterns, and apply it to realize better model defense in further study. Our proposal reveals the underlying threat hidden in DNNs, but since we focus only on the digital space and white-box attacks, it does not induce imminent threats to the applications of DNNs in the physical world.
A Error bars for experiment on Universal Adversarial Perturbation task    The confidence and dominance ratio are not so satisfying as the ones trained with ImageNet and COCO dataset, especially for VGG19 and DN121. We think this is a foreseeable result, considering that the BigGAN based dataset is not exactly a natural image set, but close to the natural image distribution. More importantly, due to the restriction of open source, the generated images can only cover 4 over 1,000 classes in ImageNet, which makes it a rather challenging task.

13
C Adversarial Examples generated by utilizing proxy image set.