Curved Representation Space of Vision Transformers

Neural networks with self-attention (a.k.a. Transformers) like ViT and Swin have emerged as a better alternative to traditional convolutional neural networks (CNNs) for computer vision tasks. However, our understanding of how the new architecture works is still limited. In this paper, we focus on the phenomenon that Transformers show higher robustness against corruptions than CNNs, while not being overconfident (in fact, we find Transformers are actually underconfident). This is contrary to the intuition that robustness increases with confidence. We resolve this contradiction by investigating how the output of the penultimate layer moves in the representation space as the input data moves within a small area. In particular, we show the following. (1) While CNNs exhibit fairly linear relationship between the input and output movements, Transformers show nonlinear relationship for some data. For those data, the output of Transformers moves in a curved trajectory as the input moves linearly. (2) When a data is located in a curved region, it is hard to move it out of the decision region since the output moves along a curved trajectory instead of a straight line to the decision boundary, resulting in high robustness of Transformers. (3) If a data is slightly modified to jump out of the curved region, the movements afterwards become linear and the output goes to the decision boundary directly. Thus, Transformers can be attacked easily after a small random jump and the perturbation in the final attacked data remains imperceptible, i.e., there does exist a decision boundary near the data. This also explains the underconfident prediction of Transformers. (4) The curved regions in the representation space start to form at an early training stage and grow throughout the training course. Some data are trapped in the regions, obstructing Transformers from reducing the training loss.


Introduction
Self-attention-based neural network architectures, including Vision Transformers [1], Swin Transformers [2], etc. (hereinafter referred to as Transformers), have shown to outperform traditional convolutional neural networks (CNNs) in various computer vision tasks. The success of the new architecture has prompted a question, how Transformers work, especially compared to CNNs, which would also shed light on the deeper understating of CNNs and eventually neural networks.
In addition to the improved task performance (e.g., classification accuracy) compared to CNNs, Transformers also show desirable characteristics in other aspects. It has been shown that Transformers are more robust to adversarial perturbations than CNNs [3,4,5]. In addition, Transformers are reported not overconfident in their predictions unlike CNNs [6] (and we show that Transformers are actually underconfident in this paper).
The high robustness, however, does not comport with underconfidence. Intuitively, a data that is correctly classified by a model with lower confidence is likely to be located closer to the decision boundary. Then, a smaller amount of perturbation would move the data out of the decision region, which translates into lower robustness of the model. However, the previous results claim the opposite.
To mitigate the contradiction of the robustness and the underconfidence, we explore the representation space of Transformers and CNNs. We examine the input-output relationship of the models, i.e., how the output feature of the penultimate layer (which is simply referred to as output in this paper) changes with respect to the linear change of the input data. We find that for certain data, Transformers move the output along a nonlinear trajectory within a arXiv:2210.05742v1 [cs.CV] 11 Oct 2022 certain range, whereas CNNs show fairly linear movements of the output. In other words, the representation space of Transformers is curved. We show that this curved representation space results in the aforementioned contradiction. Fig. 1 visualizes the representation spaces of CNNs and Transformers comparatively (see Appendix A for implementation details). An image data from ImageNet [7], marked with the black dot in Fig. 1a, is gradually modified by a fixed amount along two mutually orthogonal directions. The corresponding outputs of ResNet50 and Swin-T are obtained, which are shown after two-dimensional projection in Figs. 1b and 1c, respectively. While the gradual changes of the input produce almost linear changes in the output of ResNet50, the output trajectory of Swin-T is curved around the original output (and then becomes linear when the change of the output is large).
Our research questions and findings can be summarized as follows. 1. How does the representation space of Transformers look like? To answer this, we analyze the output movement of the penultimate layer with respect to the linear movement of the input. We find that for some input data, the directions of successive movements of the output significantly change in the case of Transformers unlike CNNs, indicating that the representation space of Transformers is locally curved. 2. What makes Transformers robust to input perturbation? We find that the curved regions in the representation space account for the robustness of Transformers. When a data is located in a curved region, the movement of the output by the perturbation of the input data follows a curved trajectory. This makes it hard to move the data out of its decision region with a small amount of perturbation, which explains high robustness of Transformers for the data. 3. Then, why is the prediction of Transformers underconfident? Although it takes many steps to escape from a curved decision region and reach a decision boundary, we find that a decision boundary is actually located closely to the original output. We demonstrate a simple trick to reach the decision boundary quickly. I.e., when a small amount of random noise is added to the input data, its output can jump out of the locally curved region and arrive at a linear region, from which a closely located decision boundary can be reached by adding only a small amount of perturbation. This reveals that the decision boundary exits near the original data in the representation space, which explains the underconfident predictions of Transformers. 4. When are the curved regions created and how do they influence the training process? To answer this, we examine the curvedness of the representation space during training. Interestingly, curved regions already start to form at an early training stage. This causes difficulty of training of Transformers, because once a training data is trapped in a curved region, it hardly escapes the region and its training loss is hardly reduced for the rest of the training process. This explains larger training loss values observed in training of Transformers in comparison to those of CNNs.

Related work
Since the first application of the self-attention mechanism to vision tasks [1], a number of studies have shown that the models built with traditional convolutional layers are outperformed by Transformers utilizing self-attention layers in terms of task performance [2,8,9,10,11,12,13,14,15,16]. There have been efforts to compare CNNs and Transformers in various aspects. Empirical studies show that Transformers have higher adversarial robustness than CNNs [5,4,17,18], which seems to be due to the reliance of Transformers on lower frequency information than CNNs [19,20]. [6] conclude that Transformers are calibrated better than CNNs yielding overconfident predictions [21,22,23]. However, there has been no clear explanation encompassing both higher robustness and lower confidence of Transformers.
Understanding how neural networks work has been an important research topic. A useful way for this is to investigate the input-output mapping formed by a model. Since models with piecewise linear activation functions (e.g., ReLU) implement piecewise linear mappings, several studies investigate the characteristics of linear regions, e.g., counting the number of linear regions as a measure of model expressivity (or complexity) [24,25,26,27,28,29] and examining local properties of linear regions [30]. Some studies examine the length of the output curve for a given unit-length input [29,31,32]. There also exist some works that relate the norm of the input-output Jacobian matrix to generalization performance [33,34]. However, the input-output relationship of Transformers has not been explored previously, which is focused in this paper.
3 On the ostensible contradiction of high robustness and underconfidence

Model calibration
It is desirable that a trained classifier is well-calibrated by making prediction with reasonable certainty, e.g., for data that a classifier predicts with confidence of 80%, its accuracy should also be 80% in average. A common measure to evaluate model calibration is the expected calibration error (ECE) defined as [35] where K is the number of bins of confidence, P (i) is the fraction of data falling into bin i, o i is the accuracy of the data in bin i, and e i is the average confidence of the data in bin i. A limitation of ECE is that it does not distinguish between overconfidence and underconfidence because the sign of the difference between the accuracy and the confidence is ignored. Therefore, we define a modified version of ECE, called signed ECE (sECE), as follows.
An overconfident model will have higher confidence than accuracy, resulting in a negative sECE value. An underconfident model, in contrast, will show a positive value of sECE.
We compare the calibration of CNNs, including ResNets [36], DenseNets [37], MobileNets [38,39], and Transformers, including ViT [1] and Swin [2], on the ImageNet validation set using ECE and sECE in Table 1 and Fig. 2. CNNs show negative ECE values in Table 1 and bar plots below the 45 • line in Fig. 2, indicating overconfidence in prediction, which is consistent with the previous studies [21]. On the other hand, Transformers are underconfident, showing positive sECE and bar plots over the 45 • line. This comparison result is interesting: Transformers reportedly show higher classification accuracy than CNNs, but in fact with lower confidence.

Passage to decision boundary
It is a common intuition that if a model classifies a data with low confidence, the data is likely to be located near a decision boundary. Based on the above results, therefore, the decision boundaries of Transformers are assumed to be formed near the data compared to CNNs. To validate this, we formulate a procedure to examine the distance to a decision boundary from a data through a linear travel. Concretely, we aim to solve the following optimization problem.
arg min C(x ) = y, where x is an input data, y is the true class label of x, C is the classifier, d is the travel direction, is a positive real number, and x is the traveled result of x. Here, we want to set the direction d in such a way that the travel reaches to a closely located decision boundary and becomes as small as possible. For this, we adopt the idea of the fast gradient sign method (FGSM) [40] used for adversarial attack. We find the direction to which the classification loss function J (i.e., cross-entropy) increases, i.e., d = sign(∇ x J(C(x), y)).
Note that d 2 = √ D, where D is the dimension of x. Refer to Alg. 1 in Appendix C for the detailed procedure to solve the optimization problem in Eq. 3. Fig. 3 shows the obtained values of with respect to the confidence values for the ImageNet validation data. It is observed that decision boundaries are farther from the data in the input space for Transformers than CNNs, which is the opposite to our expectation. This contradiction is resolved in the following section.

Resolving the contradiction 4.1 Shape of representation space
As an attempt to resolve the contradiction, we examine the input-output relationship of the models, i.e., how linear movements in the input space appear in the representation space of a model.
We divide the travel to the decision boundary into N steps as where x (0) = x and x (N ) are the initial and final data points, respectively. For each x (n) , we obtain its output feature at the penultimate layer, which is denoted as z (n) . Unlike the travel in the input space, the magnitude and direction of the travel appearing in the representation space may change at each step. Thus, the movement at step n is defined as from which the magnitude (ω (n) ) and relative direction (θ (n) ) are obtained as We consider four different ways to determine d x : • d FGSM (blue-colored trajectory in Fig. 4): FGSM direction for x (0) (as in Sec. 3.2), • d r : random direction, • d FGSM ⊥ (green-colored trajectory in Fig. 4): random direction that is orthogonal to the FGSM direction, • d r+FGSM (yellow-colored trajectory in Fig. 4): FGSM direction for the randomly perturbed data x and r controls the amount of the "random jump." Fig. 5 shows the direction changes in early steps of travel (i.e., θ (1) , · · · , θ (4) ) for ResNet50 and Swin-T when /N =.002 and r =.05 (see Appendix D.1 for the magnitude distributions, which shows a similar trend; the results for the whole travel are shown in Appendix D.2). The following observations can be made (also illustrated in Fig. 4). 1) For ResNet50, the direction does not change much, no matter how the travel direction in the input space is set (Figs. 5a, 5b, 5c). This indicates that the input-output relationship of CNNs is fairly linear around the data. 2) In contrast, Swin-T shows locally nonlinear input-output relationship; θ (n) is significantly large no matter how the travel direction in the input space is set (Figs. 5e, 5f, 5g). I.e., the representation space of Transformers is curved around the data. 3) Fig. 5h shows that the direction does not change much after the random jump. By moving x (0) in a random direction r, z (0) can pass over the curved region without meandering in the early steps and make linear movements afterwards (z r (n) in Fig. 4).

Robustness and underconfidence of Transformers
Figs. 6a and 6b show the distribution of the Euclidean distance from the original output to the decision boundary in the representation space (i.e., ||z (N ) − z (0) ||) for ResNet50 and Swin-T. Note that the distance scale is different between the models. Interestingly, the distance distribution for Swin-T is bimodal, i.e., the data are grouped into those having small distances (around zero) and those having large distances (around 10).
We examine this phenomenon further in Fig. 6c, which is a scatter plot between the confidence and the distance for Swin-T, where the colors represent θ (1) of the corresponding data. Note that θ (1) is highly correlated to the total direction change ( N −1 n=1 θ (n) ), and thus is used as a measure of curvedness of the representation space around the data (see Appendix D.3). It is clear that the curvedness dichotomizes the data: those associated with small values of θ (1) are located in linear regions (marked with yellowish colors), while those associated with large values of θ (1) are located in curved regions (marked with greenish colors). In particular, the data in the latter group show larger distances to the decision boundaries, and thus become more robust against adversarial attacks. In other words, since they are located in curved regions, an attack on them becomes challenging.
To validate this, we apply the iterative FGSM attack (I-FGSM) [41], which is one of the strong attacks, to the correctly classified ImageNet validation data for Swin-T. We set the maximum amount of perturbation to IFGSM =.001 or .002, the number of iterations to T =10, and the step size to IFGSM /T . Fig. 7 shows the classification accuracy after attack with respect to θ (1) . We can observe that the data having large values of θ (1) show high robustness (i.e., high accuracy after the attack), which makes the overall robustness of Swin-T (blue line) higher than that of ResNet50 (red line).
We hypothesize that the curved representation space also causes the underconfident prediction of Transformers. As shown as the blue-colored trajectory in Fig. 4b, the decision boundary is actually close to the data point but the curved travel reaches the decision boundary at a farther location. To validate this hypothesis, we add a small amount of noise to the input data in order to check if the decision boundary at a closer location can be found if the data jumps out of the curved region (i.e., reaching z r (N ) from z r (0) in Fig. 4b).
Fig . 8 shows the distribution of the distance to the decision boundary in the representation space. It can be observed that when the FGSM direction is used after random jump (orange line), the bimodality of the distribution is significantly reduced. As shown in Fig. 5h, the travel becomes less curved and thus the decision boundary can be reached effectively. The 2D projected movements after the jump in Fig. 9 also supports this. Furthermore, the random jump can be made even more effective by setting the jump direction to the FGSM direction found after the random jump (red-colored trajectory in Fig. 4), resulting in further reduction in the bimodality (red line in Fig. 8).   The reduced distance to the boundary by random jump implies that the jumped input data can be made misclassified by adding a smaller amount of perturbation than the original input data. Fig. 10 demonstrates that this actually works. The figure shows example images perturbed linearly in the FGSM direction (i.e., x (N ) ) and those first undergone random jump ( r =.05) and then perturbed linearly in the FGSM direction (i.e., x r (N ) ). It is clear that the images are easily misclassified with significantly reduced amounts of perturbation (smaller and higher PSNR) after the random jump passing over curved regions.

Curved space during training
For deeper understanding of the curved regions in the representation space, we look into the training process of Transformers. Each row of Fig. 11 shows the loss change of each of the 10,000 ImageNet training data at every 10 epochs during training of Swin-T (see Appendix F for training details). The rows are sorted in an ascending order of θ (1) after training. It is observed that for the data located in curved regions (showing large values of θ (1) ), the loss does not change much from the early training stage (bottom rows in the figure). Fig. 12 provides another view of this phenomenon in terms of the relationship between the loss at a certain epoch and the loss change from the epoch until the end of training. The loss values for the data residing in curved regions (dark-colored points in the figure) are hardly reduced already from 30 epochs. In other words, certain training data seem to be trapped in curved regions, which obstructs the training of the network.
When do curved regions start to form? We check the relationship of θ (1) at a certain training stage and θ (1) after training in Fig. 13. The data points are mostly above the 45 • line, indicating that once a data is trapped in a curved region, it hardly escapes the region. It can be also seen that θ (1) becomes larger during training, i.e., the curvedness gets severer.

Conclusion
We studied the input-output relationship of Transformers by examining the trajectory of the output in the representation space with respect to linear movements in the input space. The experimental results indicated that the representation space of Transformers is curved around some data, which explains high robustness and underconfident prediction of Transformers. It was also shown that the curved regions are formed at the early stage of training and can cause difficulty in training.
Based on our findings, many interesting research questions can follow towards better understanding of the working mechanism of neural networks, such as which architectural components and optimization techniques promote curved regions, how curved regions can be reduced or intensified during or after training, how curvedness can be quantified, etc. Fig. 1 is produced as follows. The FGSM direction d x is found for the given data x, and we choose a random direction d ⊥ x orthogonal to d x . Then, we move x in the input space using d x and d ⊥ x by

A Representation space visualization
where α is a positive real number, N determines the total number of points in the grid of the input space as (2N + 1) 2 (Fig. 1a), and i and j are integers within [−N, N ].
For each x ij , we obtain its output at the penultimate layer of a model, z ij , i.e., the feature in the representation space. Then, z ij is projected onto the two-dimensional plane determined by two arbitrary mutually orthogonal vectors d z and d ⊥ z , which produces Figs. 1b and 1c.

B Model calibration
Fig. 14 provides reliability diagrams of the models that are listed in Table 1 but not included in Fig. 2.

D.2 Whole travel trajectory
Fig . 17 shows the step-wise magnitude (ω (n) ) and direction change (θ (n) ) during the whole travel for N =50 steps to the decision boundary for Swin-T. x ← clip 0,1 (x + p) where clip a,b (z) = min(max(z, a), b).

5:
if C(x ) = y then 5: ← × d 6:    Figure 17: ω (n) and θ (n) during the whole travel of 50 ImageNet data to decision boundaries for Swin-T.   Table 2 shows the correlation coefficients between θ (1) (the first direction change) and N −1 n=1 θ (n) (the total direction change). While the latter is a measure of curvedness of the representation space in an overall sense, the two values are highly correlated. Thus, we use θ (1) instead; in addition, it can provide the information about the local curvedness near z 0 .

D.4 Curvedness in different directions
Fig . 18 shows the relationship of θ (1) in two different random directions, implying that no matter which random direction is chosen, the result remains consistent. Fig. 19 shows the relationship of the length of travel ( ) to decision boundaries for original images (x-axis) and random jumped images (y-axis) for Swin-T. The dark-colored points on the bottom part of the figure correspond to the images that escape curved regions with random jump and then can reach nearby located decision boundaries with imperceptible noise (Fig. 10).

F Model training
We follow the original training implementation code of the Swin-T Transformer for model training in Sec. 5, e.g., 300 training epochs, batch size of 128, learning rate of 5 × 10 −4 , weight decay parameter of .05, and AdamW [42] as the optimizer.