A Survey of Visual Transformers

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern Convolution Neural Networks (CNNs). In this survey, we have reviewed over one hundred of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where a taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, three promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.


I. INTRODUCTION
T RANSFORMER [1], which adopts an attention-based structure, has first demonstrated its tremendous effects on the tasks of sequence modeling and machine translation. As illustrated in Fig. 1, Transformers have gradually emerged as the predominant deep learning models for many NLP tasks. The most recent dominant models are the selfsupervised Transformers, which are pre-trained over sufficient datasets and then fine-tuned over a small sample set for a given downstream task [2]- [9]. The Generative Pre-trained Transformer (GPT) families [2]- [4] leverage the Transformer decoders to enable auto-regressive language modeling, while the Bidirectional Encoder Representations from Transformers (BERT) [5] and its variants [6], [7] serve as auto-encoder language models built on the Transformer encoders.
In the CV field, prior to the visual Transformers, Convolution Neural Networks (CNNs) have emerged as a dominant paradigm [10]- [12]. Inspired by the great success of such self-attention mechanisms for the NLP tasks [1], [13], some CNN-based models attempted to capture the long-range dependencies through adding a self-attention layer at either spatial level [14]- [16] or channel level [17]- [19], while others try to replace the traditional convolutions entirely with the global [20] or local self-attention blocks [21]- [27]. Although Ramachandr et al. have demonstrated the efficiency of selfattention block [24] without the help from CNNs, such pure attention model is still inferior to the State-of-The-Art (SoTA) CNN models on the prevailing benchmarks.
With the grateful achievements of linguistic Transformers and the rapid development of visual attention-based models, numerous recent works have migrated the Transformers to the CV tasks, and some comparable results have been achieved. Cordonnier et al. [28] theoretically demonstrated the equivalence between multi-head self-attention and CNNs, and they designed a pure Transformer by using patch downsampling and quadratic position encoding to verify their theoretical conclusion. Dosovitskiy et al. [29] further extended such a pure Transformer for large-scale pre-training, which has achieved SoTA performance over many benchmarks. Additionally, the visual Transformers have also obtained great performances for other CV tasks, such as detection [30], segmentation [31], [32], tracking [33], generation [34], and enhancement [35].
As illustrated in Fig. 1, following the pioneer works [29], [30], hundreds of Transformer-based models have been proposed for various vision applications within the last year. Thus, a systematic literature survey is strongly desired to identify, categorize, and evaluate these existing visual Transformers. Considering that the readers may come from different areas, we review all these visual Transformers according to three fundamental CV tasks (i.e., classification, detection, and segmentation) and different types of data streams (i.e., images, point clouds, multi-stream data). As illustrated in Fig. 3, this survey categorizes all these existing methods into multiple groups according to their dedicated vision tasks, data stream types, motivations, and structural characteristics.
Before us, several reviews on the Transformers have been published, where Tay et al. [46] reviewed the efficiency of the linguistic Transformers, Khan et al. [47] and Han et al. [48] summarized the early visual Transformers and attention-based models. The most recent review of the Transformers is introduced by Lin et al., which provides a systematic review of various Transformers, but they only mention vision applications sketchily [49]. Distinctively, this paper aims to provide arXiv:2111.06091v4 [cs.CV] 6 Dec 2022  [8]. (Bottom Right) Odyssey of visual Transformer backbone where the black [29], [36]- [40] is the SoTA with external data and the blue [41]- [45] refers to the SoTA without external data (best viewed in color). more comprehensive review of the most-recently visual Transformers and categorize them systematically: (1) Comprehensiveness & Readability. This paper comprehensively reviews over a hundred visual Transformers according to their applications on three fundamental CV tasks (i.e., classification, detection, and segmentation) and different types of data streams (i.e., image, point clouds, multi-stream data). We select more representative methods with detailed descriptions and analyses, but introduce other related works briefly. In addition to analysing each model independently, we also build their internal connections from certain senses such as progressive, contrastive, and multi-view analysis. (2) Intuitive Comparison. As these existing visual Transformers follow different training schemes and hyper-parameter settings for various vision tasks, this survey presents multiple lateral comparisons over different datasets and restrictions. More importantly, we summarize a series of promising components designed for each task, including: (a) shallow local convolution with hierarchical structure for backbone; (b) spatial prior acceleration with sparse attention for neck detector; and (c) general-purpose mask prediction scheme for segmentation. (3) In-depth Analysis. We further provide well-thought insights from the following aspects: (a) How visual Transformers bridge the traditional sequential tasks to the visual ones (Why does Transformer work effectively in CV); (b) the correspondence between the visual Transformers and other neural networks; (c) the double edges of the visual Transformers; and (d) the correlation of the learnable embeddings (i.e., class token, object query, mask embedding) adopted in different tasks and data stream types. Finally, we outline some future research directions. For example, the encoder-decoder Transformer backbone can unify multiple visual tasks and data stream types through query embeddings. The rest of this paper is organized as follows. An overview of the architectures and the critical components for the vanilla sequential Transformers are introduced in Sec. II. A comprehensive taxonomy for the Transformer backbones is summarized in Sec. III with a brief discussion of their applications for image classification. We then review contemporary Trans-former detectors, including Transformer necks and backbones in Sec. IV. Sec. V clarifies the mainstream and its variants for the visual Transformers in the segmentation field according to their embedding forms (i.e., patch embedding and query embedding). Sec. III-V also briefly analyzes a specific aspect of their corresponding fields with performance evaluation. In addition to 2D visual recognition, Sec. VI briefly introduces the recently-developed 3D visual recognition from the perspective of point clouds. Sec. VII further overviews the fusion approaches within the visual Transformers for multiple data stream types (e.g., multi-view, multi-modality, visual-linguistic pre-training, and visual grounding). Finally, Sec. VIII provides three aspects for further discussion and points out some promising research directions for future investment.

II. ORIGINAL TRANSFORMERS
The original Transformer [1] is first applied to the task for sequence-to-sequence auto-regression. Compared with previous sequence transduction models [50], [51], such original Transformer inherits the encoder-decoder structure but discards the recurrence and convolutions entirely by using multi-head attention mechanisms and point-wise feed-forward networks. In the following sub-sections, we will provide an architectural overview of the original Transformers and describe their four key components.

A. (Multi-Head) Attention Mechanism
The mechanism with one single head attention can be grouped into two parts: 1) A transformation layer maps the input sequences X ∈ R nx×dx , Y ∈ R ny×dy into three different vectors (query Q, key K, and value V ), where n and
d are the length and the dimension of the inputs, respectively. 2) An attention layer, as shown in Fig. 2, explicitly aggregates the query with the corresponding key, assigns them to the value and updates the output vector.
The formula for the transformation layer is defined as where W Q ∈ R dx×d k , W K ∈ R dy×d k , and W V ∈ R dy×d v are linear matrices. d k and d v are the dimension of the querykey pair and the value which are projected from Y and X, respectively. Such two sequence inputs are referred as the cross-attention mechanism. It can also be regarded as a selfattention when Y = X. In form, self-attention is applied to both Transformer encoder and decoder, while the crossattention severs as a junction within the decoder. Then, the scale-dot attention mechanism is formulated as where the attention weights are generated by a dot-product operation between Q and the K, a scaling factor √ d k and a softmax operation are supplied to translate the attention weights into a normalized distribution. The resulting weights are assigned to the corresponding value elements, thereby yielding the final output vector.
Due to the restricted feature subspace, the modeling capability of the single-head attention block is quite coarse. To tackle this issue, as shown in Fig. 2, a Multi-Head Self-Attention mechanism (MHSA) is proposed to linearly project the input into multiple feature sub-spaces and process them by using several independent attention heads (layers) parallelly. The resulting vectors are concatenated and mapped to the final outputs. The process of MHSA can be formulated as where h is the head number, W O ∈ R hdv×d model denotes the output projected matrix, Z i denotes the output vector of each head, W Qi ∈ R d model ×d k , W Ki ∈ R d model ×d k , and W Vi ∈ R d model ×dv are three different groups of matrices. Multi-head attention separates the inputs into h independent attention heads with d model /h-dimensional vectors, and integrates each head features dependently. Without extra costs, multi-head attention enriches the diversity of the feature subspaces.

B. Position-wise Feed-Forward Networks
The output of MHSA is then fed into two successive feedforward networks (FFN) with a ReLU activation as This position-wise feed-forward layer can be viewed as a point-wise convolution, which treats each position equally but uses different parameters between each layer.

C. Positional Encoding
Since the Transformer/Attention operates on the input embedding simultaneously and identically, the order of the sequence is neglected. To make use of the sequential information, a common solution is to append an extra positional vector to the inputs, hence term the "positional encoding". There are many choices for positional encoding. For example, a typical choice is cosine functions with different frequencies as where pos and d are the position and the length of the vector, respectively, and i is the index of each element within vector. Fig. 4 shows the overall Transformer models with the encoder-decoder architecture. Specifically, it consists of N successive encoder blocks, each of which is composed of two sub-layers. 1) An MHSA layer aggregates the relationship within the encoder embeddings. 2) A position-wise FFN layer extracts feature representations. For the decoder, it also involves N consecutive blocks that follow a stack of the encoders. Compared with the encoder, each decoder block appends to a multi-head cross-attention layer to aggregate both decoder embeddings and encoder outputs, where Y corresponds to the former, and X is the latter as shown in Eq. (1). Moreover, all of the sub-layers in both encoder and decoder employ a residual connection [11] and a Layer Normalization [163] to enhance the scalability of the Transformer. In order to record the sequential information, each input embedding is attached with a positional encoding at the beginning of the encoder stack and the decoder stack. Finally, a softmax operation are used for predicting the next word.

D. Transformer Model
In an auto-regressive language model, the Transformer is originated from the machine translation tasks. Given a sequence of words, the Transformer vectorizes the input sequence into the word embeddings, adds the positional encodings, and feeds the resulting sequence of the vectors into an  encoder. During training, as illustrated in Fig. 4, Vaswani et al. design a masking operation according to the rule for the autoregressive task [1], where the current position only depends on the outputs of the previous positions. Based on this masking, the Transformer decoder is able to process the sequence of the input labels parallelly. During the inference time, the sequence of the previously-predicted words is processed by the same operation to predict the next word.

III. TRANSFORMER FOR CLASSIFICATION
Following the prominent developments of the Transformers in NLP [2]- [5], recent works attempt to introduce visual Transformers for image classification. This section comprehensively reviews over 40 visual Transformers and groups them into six categories, as shown in Fig. 5. We start with introducing the Fully-Attentional Network [24], [28] and the Vision Transformer (ViT) [29], such Original Visual Transformer first demonstrates its efficacy on multiple classification benchmarks. Then we discuss Transformer Enhanced CNN methods that utilize Transformer to enhance the representation learning in CNNs. Due to the negligence of local information in the original ViT, the CNN Enhanced Transformer employs an appropriate convolutional inductive bias to augment the visual Transformer, while the Local Attention Enhanced Transformer redesigns patch partition and attention blocks to improve their locality. Following the hierarchical and deep structures in CNNs [164], the Hierarchical Transformer replaces the fixedresolution columnar structure with a pyramid stem, while the Deep Transformer prevents the attention map from oversmooth and increases its diversity in the deep layer. Moreover, we also review the existing visual Transformers with Self-Supervised Learning. Finally, we make a brief discussion based on intuitive comparisons for further investigation. More visual Transformers' milestones are introduced in App. A.

A. Original Visual Transformer
Inspired by the tremendous achievements of the Transformers in the NLP field [2]- [5], the previous technology trends for the vision tasks [14]- [17], [165] incorporate the attention mechanisms with the convolution models to augment the models' receptive field and global dependency.
Beyond such hybrid models, Ramachandran et al. contemplate whether the attention can completely replace the convolution, and then present a Stand-Alone self-attention network (SANet) [24], which has achieved superior performance on the vision tasks as compared with the original baseline. Given a ResNet [11] architecture, the authors straightforwardly replace the spatial convolution layer (3 × 3 kernel) in each bottleneck block with a locally spatial self-attention layer and keep other structures the same as the original setting in ResNet. Moreover, lots of ablations have shown that the positional encodings and convolutional stem can further improve the network efficacy.
Following [24], Cordonnier et al. pioneer a prototype design (called Fully-Attentional Network in their original paper) [28], including a fully vanilla Transformer and a quadratic positional encoding. The authors also theoretically prove that a convolutional layer can be approximated by a single MHSA layer with relative positional encoding and sufficient heads. With the ablations on CIFAR-10 [166], they further verify that such a prototype design does learn to attend a grid-like pattern around each query pixel, as their theoretical conclusion.
Different from [28] that only focuses on lite scale model, the Vision Transformer (ViT) [29] further explores the effectiveness of the vanilla Transformer with large-scale pre-trained learning, and such a pioneer work impacts the community significantly. Because the vanilla Transformers only accept the sequential inputs, the input image in ViT is firstly split into a series of non-overlapped patches and they are then projected into patch embeddings. Then a 1D learnable positional encoding is added on the patch embeddings to retain the spatial information, and the joint embeddings are then fed into the encoder, as shown in Fig. 6. Similar to BERT [5], a learned [class] token is attached with the patch embeddings to aggregate the global representation and it serves as the final output for classification. Moreover, a 2D interpolation complements the pre-trained positional encoding to maintain the consistent order of the patches when the feeding images are in arbitrary resolution. By pre-training with a large-scale private dataset (JFT-300M [167]), ViT has achieved similar or even superior results on multiple image recognition benchmarks (ImageNet [168] and CIFAR-100 [166]) as compared with the most prevailing CNNs methods. However, its generalization capability tends to be eroded with limited training data.

B. Transformer-enhanced CNNs
As described in Section II, the Transformer has two keys: MHSA and FFN. There exists an approximation between the convolutional layer and the MHSA [28], and Dong et al. suggest that the Transformer can further mitigate the strong bias of MHSA with the help of skip connections and FFN [169]. Recently, some methods attempt to integrate the Transformer into CNNs to enhance representation learning. VTs [52] decouples semantic concepts for the input image into different channels and relates them densely through the encoder block, namely VT-block. Such VT-block substitutes the last convolution stage to enhance the CNN model's ability on semantic  Figure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder. In order to perform classification, we use the standard approach of adding an extra learnable "classification token" to the sequence. The illustration of the Transformer encoder was inspired by Vaswani et al. (2017).

VISION TRANSFORMER (VIT)
An overview of the model is depicted in Figure 1. The standard Transformer receives as input a 1D sequence of token embeddings. To handle 2D images, we reshape the image x ∈ R H×W ×C into a sequence of flattened 2D patches x p ∈ R N ×(P 2 ·C) , where (H, W ) is the resolution of the original image, C is the number of channels, (P, P ) is the resolution of each image patch, and N = HW/P 2 is the resulting number of patches, which also serves as the effective input sequence length for the Transformer. The Transformer uses constant latent vector size D through all of its layers, so we flatten the patches and map to D dimensions with a trainable linear projection (Eq. 1). We refer to the output of this projection as the patch embeddings.
Similar to BERT's [class] token, we prepend a learnable embedding to the sequence of embedded patches (z 0 0 = x class ), whose state at the output of the Transformer encoder (z 0 L ) serves as the image representation y (Eq. 4). Both during pre-training and fine-tuning, a classification head is attached to z 0 L . The classification head is implemented by a MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time.
Position embeddings are added to the patch embeddings to retain positional information. We use standard learnable 1D position embeddings, since we have not observed significant performance gains from using more advanced 2D-aware position embeddings (Appendix D.3). The resulting sequence of embedding vectors serves as input to the encoder.
Hybrid Architecture. As an alternative to raw image patches, the input sequence can be formed from feature maps of a CNN (LeCun et al., 1989). In this hybrid model, the patch embedding projection E (Eq. 1) is applied to patches extracted from a CNN feature map. As a special case, the patches can have spatial size 1x1, which means that the input sequence is obtained by simply 3 Fig. 6. Illustration of ViT. The flatten image patches with an additional class token are fed into the vanilla Transformer encoder after positional encoding.
Only the class token can be predicted for classification. (from [29].) modelling. Unlike previous approaches that directly replace convolution with attention structure, Vaswani et al. propose a conceptual redefinition that the successive bottleneck blocks with MHSA can be formulated as the Bottleneck Transformer (BoTNet) [53] blocks. The relative position encoding [170] is adopted to further mimic the original Transformer. Based on ResNet [11], BoTNet outperforms the most CNN models with similar parameter settings on the ImageNet benchmark and further demonstrates the efficacy of hybrid models.

C. CNN-enhanced Transformer
Inductive bias is defined as a set of assumptions on data distribution and solution space, whose manifestations within convolution are the locality and the translation invariance [171]. As the covariance within local neighborhoods is large and tends to be gradually stationary across an image, CNNs can process an image effectively with the help of the biases. Nevertheless, strong biases also limit the upper bound of CNNs when sufficient data are available. Recent efforts attempt to leverage an appropriate CNN bias to enhance Transformer.
Touvron et al. propose a Data-efficient image Transformer (DeiT) [41] to moderate the ViT's dependence on large datasets. In addition to the existing strategies for data augmentation and regularization, a teacher-student distillation strategy is applied for auxiliary representation learning. The student model is the ViT, where a distilled token is attached to the patch embeddings and it is supervised by the pseudo labels from the teacher model. Extensive experiments have demonstrated that CNN is a better teacher model than the Transformer. Surprisingly, the distilled student Transformer even outperforms its teacher CNN model. These observations are explained in [172]: the teacher CNN transfers its inductive bias in a soft way to the student Transformer through knowledge distillation. Based on ViT's architecture, DeiT-B attains the top-1 accuracy of 85.2% without external data. ConViT [54] appends a parallel convolution branch with vanilla Transformer to impose inductive biases softly. The main idea of the convolution branch is a learnable embedding that is first initialized to approximate the locality as similar as the convolution and then explicitly gives each attention head freedom to escape the locality by adjusting a learned gating parameter. CeiT [55] and LocalViT [56] extract the locality by directly adding a depth-wise convolution in FFN.
As point-wise convolution is equal to position-wise FFN, they extend FFN to an inverted residual block [173] to build a depth-wise convolutional framework. Based on the assumption of positional encoding [58] and the observation in [174], ResT [58] and CPVT [57] try to adapt the inherent positional information of the convolution to the arbitrary size of inputs instead of interpolating the positional encoding. Including CvT [37], these methods replace the linear patch projection and positional encoding with the convolution stacks. Both methods benefit from such convolutional position embedding, especially for small model.
Besides the "internal" fusion, many approaches focus on an "apparent" combination according to different visual Transformer's structures. For standard columnar structure, Xiao et al. substitute the original patchify stem (single non-overlapped large kernel) with several stacked stride-2 3 × 3 kernels [59]. Such a Convolutional Stem significantly improves ViT by 1-2% on accuracy for ImageNet-1k and facilitates its stability and generalization for the downstream tasks. For hierarchical structures, Dai et al. [40] investigate an optimal combination of hybrid models to benefit the performance trade-off. By comparing a series of hybrid models, they propose a Convolution and Attention Network (CoAtNet) to leverage the strength of both CNNs and Transformer. The authors observe that depthwise convolution can be naturally integrated into the attention block, and stacking convolution vertically in the shallow layer is more effective than the original hierarchical methods. It has achieved the SoTA performance across multiple datasets.

D. Local Attention Enhanced Transformer
The coarse patchify process in ViT [29] neglects the local image information. In addition to adding CNNs, various local attention mechanisms are proposed to dynamically attend the neighbour elements and augment the local extraction ability.
One of the representative methods is the Shifted windows (Swin) Transformer [36]. Similar to TSM [175] (Fig. 7(a)), Swin utilizes a shifted window along the spatial dimension to model the global and boundary features. In detail, two successive window-wise attention layers can facilitate the crosswindow interactions ( Fig. 7(b)-(c)), similar to the receptive field expansion in CNNs. Such operation also reduces the computational complexity from O(2n 2 C) to O(4M 2 nC) in one attention layer, where n and M denote the patch length and the window size, respectively. Swin Transformer achieves 84.2% accuracy on ImageNet and the latest SoTA on multiple dense prediction benchmarks (see Sec. IV-B).
Inspired by [176], Han et al. leverage a Transformer-iN-Transformer (TNT) [60] model to aggregate both patch-and pixel-level representations. Each layer of TNT consists of two successive blocks, an inner block models the pixel-wise interaction within each patch, and an outer block extracts the global information. Twins [61] employs a spatially separable selfattention mechanism, similar to depth-wise convolution [173] or window-wise TNT [60], to model the local-global representation. Another separate form is ViL [62], which replaces the single class token with a series of local embeddings (termed as global memory). These local embeddings only perform an inner attention and an interaction with their corresponding 2D spatial neighbors. VOLO [45] proposes an outlook attention, which is similar to a patch-wise dynamic convolution, to focus on the finer-level features, including three operations: unfold, linear-wights attention, and refold. Based on [44], it achieves SoTA results on ImageNet without external data.

E. Hierarchical Transformer
Due to the columnar structure of ViT [29] with a fixed resolution across the entire Transformer layers, it loses the fine-grained features and suffers from heavy computational costs. Followed by the hierarchical models, Tokens-to-Token ViT (T2T-ViT) [64] first introduces a paradigm of hierarchical Transformer and employs an overlapping unfold operation for down-sampling. However, such operation brings heavy memory and computation costs. Therefore, Pyramid Vision Transformer (PVT) [42] leverages a non-overlapping patch partition to reduce feature size. Furthermore a spatial-reduction attention (SRA) layer is applied in PVT to further reduce the computational cost by learning low-resolution key-value pairs. Empirically, PVT adapts the Transformer to the dense prediction tasks on many benchmarks which demand large inputs and fine-grained features with computational efficiency. Moreover, both PiT [65] and CvT [37] utilize pooling and convolution to perform token downsampling, respectively. In detail, CvT [37] improves the SRA of PVT [42] by replacing the linear layer with a convolutional projection. Based on the convolutional bias, CvT [37] can adapt to arbitrary size inputs without positional encodings.

F. Deep Transformer
Empirically, increasing model's depth always strengthens its learning capacity [11]. Recent works apply a deep structure to Transformer and massive experiments are conducted to investigate its scalability by analyzing cross-patch [68] and cross-layer [38], [67] similarities, and the contribution of residual blocks [43]. In the deep Transformer, the features from the deeper layers tend to be less representative (attention collapse [67]), and the patches are mapped into the indistinguishable latent representations (patch over-smoothing [68]). To address such limitations mentioned above, current methods present the corresponding solutions from two aspects.
From the aspect of model's structure, Touvron et al. present efficient Class-attention in image Transformers (CaiT [43]), including two stages: 1) Multiple self-attention stages without class token. In each layer, a learned diagonal matrix initialized by small values is exploited to update the channel weights dynamically, thereby offering a certain degree of freedom for channel adjustment. 2) Last few class-attention stages with frozen patch embeddings. A later class token is inserted to model global representations, similar to DETR [30] with an encoder-decoder structure. This explicit separation is based on the assumption that the class token is invalid for the gradient of patch embeddings in the forward pass. With distillation training strategy [41], CaiT achieves a new SoTA on imagenet-1k (86.5% top-1 accuracy) without external data. Although deep Transformer suffers from attention collapse and oversmoothing problems,it still largely preserves the diversity of the attention map between different heads. Based on this observation, Zhou et al. propose Deep Vision Transformer (DeepViT) [67] that aggregates different head attention maps and re-generates a new one by using a linear layer to increase cross-layer feature diversity. Furthermore, Refiner [38] applies a linear layer to expand the dimension of the attention maps (indirectly increasing the head number) for diversity promotion. Then, a Distributed Local Attention (DLA) is employed to achieve better modeling of both the local features and the global ones, which is implemented by a head-wise convolution effecting on the attention map.
From the aspect of training strategy, Gong et al. present three Patch Diversity losses for deep Transformer that can significantly encourage patches' diversity and offset oversmoothing problem [68]. Similar to [177], a patch-wise cosine loss minimizes pairwise cosine similarity among patches. A patch-wise contrastive loss regularizes the deeper patches by their corresponding one in the early layer. Inspired by Cutmix [178], a patch-wise mixing loss mixes two different images and forces each patch to only attend to the patches from the same image and ignore unrelated ones. Compared with LV-ViT [44], their similar loss function is based on a distinctive motivation. The former focuses on the patch diversity, while the latter focuses on data augmentation about token labeling.

G. Transformers with Self-Supervised Learning
Following the grateful success of self-supervised in the NLP field [5], recent works also attempt to design various selfsupervised learning schemes for the visual Transformers in both generative and discriminative ways.
For the generative models, Chen et al. propose an image Generative Pre-training Transformer (iGPT) [69] for selfsupervised visual learning. Different from the patch embedding of ViT [29], iGPT directly resizes and flattens the image to a lower resolution sequences. The resized sequences are then input into a GPT-2 [4] for auto-regressive pixel prediction. iGPT demonstrates the effectiveness of the Transformer in the visual tasks without any help from image-specific knowledge, but its considerable computation cost is hard to be accepted (roughly 2500 V100-days for pre-training). Instead of generating such raw pixels directly, Bao et al. propose a BERTstyle [5] visual Transformer (BEiT) [71] by reconstructing the masked image in the latent space. Similar to the dictionary in BERT [5], dVAE [149] vectorizes the image into discrete visual tokens. The resulting visual token serves as pseudo label to pre-train ViT for masked patch construction.
For the discriminative models, Chen et al. [73] go back to basics and investigate the effects of several fundamental components for stabilized self-supervised ViT training. They observe that the unstable training process mildly affects the eventual performance, and extend MoCo series to MoCo v3, containing a series of training strategies such as freezing projection layer. Following DeiT [41], Caron et al. further extend the teacher-student recipe to self-supervised learning and propose DINO (2021) [74]. The core concepts of DINO can be summarized into three points. A momentum encoder inherited SwAV [179], serves as a teacher model that outputs the centered pseudo labels over a batch. An online encoder without the prediction head serves as a student model to fit the teacher's output. A standard cross-entropy loss connects selftraining with knowledge distillation. More interestingly, selfsupervised ViT can learn flourishing features for segmentation, which are normally unattainable by the supervised models.
H. Discussion 1) Algorithm Evaluation and Comparative Analysis: In our taxonomy, all the existing supervised models are grouped into six categories. Tab. I summarizes the performances of these existing visual Transformers on ImageNet-1k benchmarks. To evaluate them objectively and intuitively, we use the following three figures to illustrate their performances on ImageNet-1k under different configurations. Fig. 8(a) summarizes the accuracy of each model under 224 2 inputs size. Fig. 8(b) takes the FLOPs as the horizontal axis, which focuses on their performances under higher-resolution. Fig. 8(c) focuses on the pre-trained models with external datasets. From these comparison results, we briefly summarize several performance improvements on efficiency and scalability as follows.
• Compared with the most structure-improved methods, the basic training strategies like DeiT [41] and LV-ViT [44], are more universal for various models, tasks, and inputs.
• The locality is indispensable for the Transformer, which is reflected by the dominant of VOLO [45] and Swin [36] on classification and dense prediction tasks, respectively.
• The convolutional patchify stem (ViT c [59]) and early convolutional stage (CoAtNet [40]) can significantly boost the accuracy of the Transformers, especially for large models. We speculate the reason is because these designs introduce a more stringent high-level features than the non-overlapping patch projection in ViT [29].
• The deep Transformer, such as Refined-ViT [38] and CaiT [43], has great potential. As the model size grows quadratically with the channel dimension, the trade-off in deep Transformer is considered for further investigation.
• CeiT [55] and CvT [37] show significant advantages in training a small or medium model (0−40M), which suggests that such kinds of hybrid attention blocks for lightweight models are worth further exploring. BoTNet-S1-59-T2 BoTNet-S1-110-T4 ViT Comparisons of recent visual Transformers on ImageNet-1k benchmark, including ViT [29], DeiT [41], BoTNet [53], VTs [52], ConViT [54], CeiT [55], LocalViT [56], TNT [60], Swin [36], PiT [65], T2T-ViT [64], PVT [42], CvT [37], DeepViT [67], CaiT [43], Cross ViT [180] (best viewed in color).  2) Brief Discussion on Alternatives: During the development of the visual Transformers, the most common question is whether the visual Transformers can replace the traditional convolution completely. By reviewing the history of the performance improvements in the last year, there is no sign of relative inferiority here. The visual Transformers have returned from a pure structure to a hybrid form, and the global information has gradually returned to a mixed stage with the locality bias. Although the visual Transformers can be equivalent to CNN or even has a better modeling capability, such a simple and effective convolution operation is enough to process the locality and the semantic features in the shallow layer. In the future, the spirit of combining both of them shall drive more breakthroughs for image classification.

IV. TRANSFORMER FOR DETECTION
In this section, we review visual Transformers for object detection, which can be grouped into two folds: Transformer as the neck and Transformer as the backbone. For the neck detectors, we mainly focus on a new representation specified to the Transformer structure, called object query, that a set of learned parameters aggregate instance features from input images. The recent variants try to solve an optimal fusion paradigm in terms of either convergence acceleration or performance improvement. Besides these neck design, a proportion of backbone detectors also take specific strategies into consideration. Finally, we evaluate them and analyze some potential methods for these detectors.

A. Transformer Neck
We first review DETR [30] and Pix2seq [76], the original Transformer detectors that reformulate two different paradigms for object detection. Subsequently, we mainly focus on the DETR-based variants, improving such Transformer detector in accuracy and convergence from five aspects: sparse attention, spatial prior, structural redesign, assignment optimization, and pre-training model.
1) The Original Detectors: DEtection with TRansformer (DETR) [30] is the first end-to-end Transformer detector that eliminates hand-designed representations [181]- [184] and non-maximum suppression (NMS) post-processing, which redefines the object detection as a set prediction problem. In detail, a small set of learnable positional encodings, called object queries, that are parallelly fed into the Transformer decoder to extract the instance information from the image features. Then, these object queries are independently predicted to be a detection result. Instead of the vanilla k-class classification, a special class, no object label (∅) is added for k+1 class classification. During the training process, a bipartite matching strategy is applied between the predicted objects and the ground-truth to identify one-to-one label assignment, hence removing the redundant predictions at the inference time without NMS. In back propagation, a Hungarian loss includes a log-likelihood loss for all classification results and a box loss for all the matched pairs. More details about the Hungarian matching strategy are available in the App. B. Overall, DETR provides a new paradigm for end-to-end object detection. The object query gradually learns an instance representation during the interaction with image features. The bipartite matching allows a direct set prediction and easily joints to the one-to-one label assignment, hence eliminating traditional post-processing. DETR achieves competitive performance on the COCO benchmark but suffers from slow convergence as well as poor performance on small objects.
Another pioneered work is Pix2seq [76], treating generic object detection as a language modeling task. Given an image input, a vanilla sequential Transformer is executed to extract features and generate a series of object descriptions (i.e. class labels and bounding boxes) auto-regressively. Such a simplified but more elaborate image caption method is derived under the assumption that if a model learns about both location and label of an object, it can be taught to produce a description with specified sequence [76]. Compared with DETR, Pix2seq attains a better result on small objects. How to combine both kinds of concepts is worthy of further consideration.
2) Transformer with Sparse Attention: In DETR, the dense interaction across both object queries and image features costs unbearable resources and slows down the convergence of DETR. Therefore, the most recent efforts aim to design a datadependent sparse attention to address these issues.
Following [185], Zhu et al. develop Deformable DETR to ameliorate both training convergence and detection performance significantly via multi-scale deformable attention [77]. Compared with original DETR, the deformable attention module only samples a small set of key (reference) points for full features aggregation. Such sparse attention can be easily stretched to multi-scale feature fusion without the help of FPN [186]. Moreover, an iterative bounding box refinement and a two-stage prediction strategy (Sec. IV-A3) are developed to further enhance the detection accuracy. Empirically, Deformable DETR achieves a higher accuracy (especially for small objects) with 10× less training epochs and reduces the computing complexity to O(2N q C 2 +min(HW C 2 , N q KC 2 )) with 1.6× faster inference speed. Please see the App. C for more details of deformable attention mechanism.
By visualizing the attention map of DETR [30], Zheng et al. observe that the semantically similar and spatially close elements always have a similar attention map in the encoder [78]. Then they present an Adaptive Clustering Transformer (ACT), leveraging a multi-round sensitivity hashing to dynamically cluster the queries into different prototypes. The attention map of the prototype is then broadcast to their corresponding queries. Unlike the redesign of on sparse attention, Wang et al. introduce a Poll and Pool (PnP) sampling model [79] to extract the fine foreground features and condense the contextual back-ground features into a smaller one. Such fine-coarse tokens are then fed into DETR to generate the detection results. Instead of the input sparsification, Sparse DETR [80] applies a hysteretic scoring network (corresponding to the Poll operation in [79]) to update the expected tokens selectively within the transformer encoder, where the top-k selected tokens are supervised by pseudo labels from the binarized decoder crossattention map with BCE loss.
3) Transformer with Spatial Prior: Unlike anchor or other representations directly generated by content and geometry features [181], [187], object queries implicitly model the spatial information with random initialization, which is weakly related with the bounding box. The mainstream for spatial prior applications are the one-stage detector with empirical spatial information and the two-stage detector with geometric coordinates initialization or Region-of-Interest (RoI) features.
In one-stage methods, Gao et al. suggest Spatially Modulated Cross-Attention (SMCA) [81] to estimate the object queries' spatial prior explicitly. Specifically, a Gaussian-like weight map generated by object queries is multiplied with the corresponding cross-attention map to augment the RoI for convergence acceleration. Furthermore, both intra-scale and multi-scale self-attention layers are utilized in the Transformer encoder for multi-scale feature aggregation, and the scaleselection weights generated from object queries are applied for scale-query adaptation. Meng et al. [82] extract the spatial attention map from the cross-attention formulation and observe that the extreme region of such attention map has larger deviations at the early training. Consequently, they propose Conditional DETR where a new spatial prior mapped from reference points is adopted in the cross-attention layer, thereby attending to extreme regions of each object explicitly. The reference point is predicted by the object query or serves as a learned parameter replacing the object query. Following [82], Anchor DETR [83] suggests to explicitly learn the 2D anchor points ([cx, cy]) and different anchor patterns instead of the high-dimensional spatial embedding. Similar to [181], the pattern embeddings are assigned to the meshed anchor points so that they can detect different scale objects anywhere. DAB-DETR [84] then extends the 2D concept to a 4D anchor box ([cx, cy, w, h]) to explicitly provide proposal bounding box information during the cross-attention. With the auxiliary decoder loss of the coordinates offset [77], such a 4D box can be dynamically refined layer-by-layer in the decoder. However, the same reference boxes/points may severely deteriorate queries' saliency and confuses detector due to the indiscriminative spatial prior. By assigning queryspecific reference points to object queries, SAP-DETR [85] only predicts the distance from each side of the bounding box to these points. Such a query-specific prior discrepancies queries' saliency and pave the way for fast model convergency.
In two-stage methods, Zhe et al. empower the Top-K region proposals from the outputs of the encoder to initialize the decoder embedding instead of the learned content query [77]. Efficient DETR [86] also adopts a similar initialization operation for dense proposals and refines them in the decoder to get sparse prediction by using a shared detection head with the dense parts. More interestingly, it is observed that small stacking decoder layers bring slight performance improvement, but even more stacks yield even worse results. Dynamic DETR [87] regards the object prediction in a coarse-to-fine process. Different from the previous RoI-based initialization detectors, a cross-attention between the pool region features and the object embeddings is used to perform decoder embedding updates. The RoI features associated with the box embedding are then refined accordingly in the next layer. 4) Transformer with Redesigned Structure: Besides the modifications focusing on the cross-attention, some works redesign an encoder-only structure to avoid the problem of the decoder directly. TSP [88] inherits the idea of set prediction [30] and dismisses the decoder and the object query to accelerate convergence. Such encoder-only DETR reuses previous representations [181], [187], and generates a set of fixedsize Features of Interests (FoI) [187] or proposals [181] that are subsequently fed into the Transformer encoder. In addition, a matching distillation is applied to resolve the instability of the bipartite matching, especially in the early training stage. Fang et al. [89] combine the encoder-decoder neck of DETR and the encoder-only backbone of ViT into an encoder-only detector and develop YOLOS, a pure sequence-to-sequence Transformer to unify the classification and detection tasks. It inherits ViT's structure and replaces the single class token with fixed size learned detection tokens. These object tokens are first pre-trained on the transfer ability for the classification tasks and then fine-tuned on the detection benchmark.

5) Transformer with Bipartite Matched Optimization:
In DETR [30], the bipartite matching strategy forces the prediction results to fulfil one-to-one label assignment during the training scheme. Such a training strategy simplifies detection pipeline and directly builds up an end-to-end system without the help of NMS. To deeply understand the efficacy of the end-to-end detector, Sun et al. devote to exploring a theoretical view of one-to-one prediction [190]. Based on multiple ablation and theoretical analyses, they conclude that the classification cost for one-to-one matching strategy serves as the key component for significantly avoiding duplicate predictions. Even so, DETR is suffering from multiple problems caused by bipartite matching. Li et al. [92] exploit a denoising DETR (DN-DETR) to mitigate the instability of bipartite matching. Concretely, a series of objects with slight perturbation is supposed to reconstruct their actual coordinates and classes. The main ingredients of the denoising (or reconstruction) part are an attention mask that prevents information leakage between the matching and noised parts, and a specified label embedding to indicate the perturbation. Recently, Zhang et al. [93] present an improved denoising training model called DINO (2022) by incorporating a contrastive loss for the perturbation groups. Based on DN-DETR [92], DINO attaches a "no object" class for the negative example if the distance is far enough from the perturbation, which avoids redundant prediction due to the confusion of multiple reference points near an object. As a result, DINO attains the current SoTA on the COCO dataset.
6) Transformer Detector with Pre-Training: Inspired by the pre-trained linguistic Transformer [3], [5], Dai et al. devise an Unsupervised Pre-training DETR (UP-DETR) [90] to assist the convergence for supervised training. The objective of pretraining is to localize the random cropped patches from a given image. Specifically, each patch is assigned to a set of queries and predicted independently via the attention mask. An auxiliary reconstruction loss forces the detector to preserve the feature discrimination so as to avoid over-bias towards the localization in pre-training. FP-DETR [91] devotes to narrowing the gap between upstream and downstream tasks. During the pre-training, a fully encoder-only DETR like YOLOS [89] views query positional embeddings as a visual prompt to enhance target area attention and object discrimination. A task adapter implemented by self-attention is used to enhance object interaction during fine-tuning.

B. Transformer Backbone
We have reviewed numerous Transformer-based backbones for image classification [29], [41] in Sec. III. These backbones can be easily incorporated into various frameworks (e.g., Mask R-CNN [188], RetinaNet [183], DETR [30], etc.) to perform dense prediction tasks. For example, the hierarchical structure like PVT [42], [66], constructs the visual Transformer as a high-to-low resolution process to learn multi-scale features. The locally enhanced structure constructs the backbone as a local-to-global combination, which can efficiently extract both short-range and long-range visual dependencies and avoid quadratic computational overhead, such as Swin-Transformer [36], ViL [62], and Focal Transformer [63]. App. D includes more detailed comparisons of these models for the dense prediction tasks. In addition to the generic Transformer backbone, the Feature Pyramid Transformer (FPT) [94] combines the characteristics across both the spaces and the scales, by using self-attention, top-down cross-attention, and bottom-up cross channel attention. Following [191], HRFormer [95] introduces the advantages of multiresolution to the Transformer along with non-overlapping local self-attention. HRViT [96] redesigns a heterogeneous branch and a cross-shaped attention block to further optimize the trade-off between efficiency and accuracy.

C. Discussion
We summarize five folds of the Transformer neck detectors in Tab. II, and more details of Transformer backbone for dense prediction tasks are referred to in Tab. IV. The majority of Transformer neck promotions concentrate on the following five aspects: 1) The sparse attention model and the scoring network are proposed to address the problem of redundant feature interaction. These methods can significantly alleviate computational costs and accelerate model convergence.
2) The explicit spatial prior, which is decomposed into the selected feature initialization and the positional information extracted by learned parameters, would enable the detector to predict the results precisely. 3) Multi-scale features and layer-by-layer updating are extended in the Transformer decoder for small object refinement. 4) The improved bipartite matching strategy is beneficial to avoid redundant prediction as well as perform end-to-end object detection. 5) The encoder-only structure reduces the overall Transformer stack layers but increases the FLOPs excessively, while the encoder-decoder structure is a good trade-off between FLOPs and Parameters, but the deeper decoder layers may cause the problems of long training time and over-smooth.
Moreover, there are many Transformer backbones for improving classification performance, but few works are developed for the dense prediction tasks. In the future, we anticipate that the Transformer backbone would cooperate with the deep high-resolution network to solve dense prediction tasks.

V. TRANSFORMER FOR SEGMENTATION
Patch-Based and Query-Based Transformer are the two major ways for Segmentation. The latter can be further grouped into Object Query and Mask Embedding methods.

A. Patch-Based Transformer
Because of the receptive field expansion strategy [192], CNNs require multiple decoder stacks to map the high-level features into the original spatial resolution. Instead, patchbased Transformer can easily incorporate with a simple decoder for segmented mask prediction because of its global modelling capability and resolution invariance. Zheng et al. extend ViT [29] for semantic segmentation tasks, and present SEgmentation TRansformer (SETR) [97] by employing three fashions of the decoder to perform per-pixel classification: naive up-sampling, progressive up-sampling, and multi-level feature aggregation (MLA). SETR demonstrates the feasibility of the visual Transformer for the segmentation tasks, but it also brings unacceptably extra GPU costs. TransUNet [98] is the first for medical image segmentation. Formally, it can be viewed as either a variant of SETR with MLA decoder [97], or a hybrid model of U-Net [193] and Transformer. Thanks to the strong global modeling capability of Transformer encoder, Segformer [99] designs a lightweight decoder with only four MLP layers. Segformer shows superior performance as well as stronger robustness than CNNs when tested with multiple corrupted types of images.

B. Query-Based Transformer
Query embeddings are a set of scratch semantic/instance representations gradually learning from the image inputs. Unlike patch embeddings, queries can more "fairly" integrate the information from features and naturally join with the set prediction loss [30] for post-processing elimination. Existed query-based models can be grouped into two categories. One is driven by both the tasks of detection and segmentation, simultaneously (called object queries). The other is only supervised by the segmentation task (called mask embeddings). 1) Object Queries: There are three training manners for object query based methods ( Fig. 10(a)-(c)). With the success of DETR [30] for the object detection tasks, the authors extend it to panoptic segmentation (hence termed Panoptic DETR [30]) by training a mask head based on the pre-trained object queries (Fig. 10(a)). In detail, a cross-attention block between the object queries and the encoded features is applied to generate an attention map for each object. After an upsampling FPN-style CNN, a spatial argmax operation fuses the resulting binary masks to a non-overlapping prediction. Instead of using a multi-stage serial training process, Cell-  Figure 2: MaskFormer overview. We use a backbone to extract image features F. A pixel decoder gradually upsamples image features to extract per-pixel embeddings Epixel. A transformer decoder attends to image features and produces N per-segment embeddings Q. The embeddings independently generate N class predictions with N corresponding mask embeddings Emask. Then, the model predicts N possibly overlapping binary mask predictions via a dot product between pixel embeddings Epixel and mask embeddings Emask followed by a sigmoid activation. For semantic segmentation task we can get the final prediction by combining N binary masks with their class predictions using a simple matrix multiplication (see Section 3.4). Note, the dimensions for multiplication are shown in gray.
Note, that most existing mask classification models use auxiliary losses (e.g., a bounding box loss [21,4] or an instance discrimination loss [42]) in addition to Lmask-cls. In the next section we present a simple mask classification model that allows end-to-end training with Lmask-cls alone.

MaskFormer
We now introduce MaskFormer, the new mask classification model, which computes N probabilitymask pairs z = {(pi, mi)} N i=1 . The model contains three modules (see Fig. 2): 1) a pixel-level module that extracts per-pixel embeddings used to generate binary mask predictions; 2) a transformer module, where a stack of Transformer decoder layers [41] computes N per-segment embeddings; and 3) a segmentation module, which generates predictions {(pi, mi)} N i=1 from these embeddings. During inference, discussed in Sec. 3.4, pi and mi are assembled into the final prediction.
Pixel-level module takes an image of size H × W as input. A backbone generates a (typically) low-resolution image feature map F ∈ R CF × H S × W S , where CF is the number of channels and S is the stride of the feature map (CF depends on the specific backbone and we use S = 32 in this work). Then, a pixel decoder gradually upsamples the features to generate per-pixel embeddings Epixel ∈ R CE ×H×W , where CE is the embedding dimension. Note, that any per-pixel classificationbased segmentation model fits the pixel-level module design including recent Transformer-based models [37,53,29]. MaskFormer seamlessly converts such a model to mask classification.
Transformer module uses the standard Transformer decoder [41] to compute from image features F and N learnable positional embeddings (i.e., queries) its output, i.e., N per-segment embeddings Q ∈ R CQ×N of dimension CQ that encode global information about each segment MaskFormer predicts. Similarly to [4], the decoder yields all predictions in parallel.
Segmentation module applies a linear classifier, followed by a softmax activation, on top of the per-segment embeddings Q to yield class probability predictions {pi ∈ ∆ K+1 } N i=1 for each segment. Note, that the classifier predicts an additional "no object" category (∅) in case the embedding does not correspond to any region. For mask prediction, a Multi-Layer Perceptron (MLP) with 2 hidden layers converts the per-segment embeddings Q to N mask embeddings Emask ∈ R CE ×N of dimension CE . Finally, we obtain each binary mask prediction mi ∈ [0, 1] H×W via a dot product between the i th mask embedding and per-pixel embeddings Epixel computed by the pixel-level module. The dot product is followed by a sigmoid activation, i.e., mi[h, w] = sigmoid(Emask[:, i] T · Epixel[:, h, w]).
Note, we empirically find it is beneficial to not enforce mask predictions to be mutually exclusive to each other by using a softmax activation. During training, the Lmask-cls loss combines a cross entropy classification loss and a binary mask loss Lmask for each predicted segment. For simplicity we use the same Lmask as DETR [4], i.e., a linear combination of a focal loss [27] and a dice loss [33] multiplied by hyper-parameters λfocal and λdice respectively. 4 Fig. 11. Illustration of Maskformer. (from [32].) DETR and VisTR develop a parallel model for end-to-end instance segmentation (Fig. 10(b)). Based on DETR [30], Cell-DETR leverages a cross-attention block to extract instancewise features from the box branch and fuses the previous backbone features to augment the CNN decoder for accurate instance mask segmentation of biological cells. Another extension is VisTR [101] that directly formulates the video instance segmentation (VIS) task as parallel sequence prediction. Apart from the similar structure as Cell-DETR [100], the key of VisTR is a bipartite matching loss at the instance sequence level to maintain the order of outputs, so as to adapt DETR [30] to VIS for direct one-to-one predictions. Unlike prior works that treat detection and mask generation branches separately, QueryInst [102] builds a hybrid cascaded network (Fig. 10(c)), where the previous box outputs together with the shared queries serve as the inputs of the mask head for accurate mask segmentation. Notably, QueryInst leverages the shared queries to keep the instance correspondences across multistage, so that mitigating the problem of inconsistent objects in previous non-query based methods [189], [194]. QueryInst obtains the latest SoTA results on the COCO datasets.
2) Mask Embeddings: The other framework makes efforts to use queries to predict mask directly, and we refer to this learned mask-based query as mask embeddings. Unlike the object queries, mask embeddings are only supervised by the segmentation tasks. As shown in Fig. 10(d), two disjoint sets of queries are employed parallelly for different tasks, and the box learning is viewed as an auxiliary loss for further enhancement. For semantic and box-free instance segmentation, a series of query-based Transformers predict the mask directly without the help of the box branch ( Fig. 10(e)).
From the auxiliary training perspective, the core is how to enable 1D sequence outputs to be supervised by 2D mask labels directly. To this end, ISTR [103] empowers a mask precoding method to encode the ground-truth spatial mask into low-dimensional mask embedding for instance segmentation. Similarly, Dong et al. propose a more straightforward pipeline SOLQ [104] and explore three reversible compression encoding methods for mask embeddings. In detail, a set of unified queries is applied to perform multiple representation learning parallelly: classification, box regression, and mask encoding. Based on the original DETR [192], SOLQ adds a mask branch to produce mask embedding loss. Both ISTR and SOLQ obtain comparable results and outperform previous methods even with approximation-based suboptimal embeddings. However, there exists a huge gap between AP box and AP seg (Tab. III).
From the box-free perspective, Wang et al. pioneer a new paradigm Max-DeepLab [31] that directly predicts panoptic masks from the query without the help of the box branch. Specifically, it forces the query to predict the corresponding mask via a PQ-style bipartite matching loss and a dualpath Transformer structure. Given a set of mask embeddings and an image input, Max-DeepLab processes them separately in both Transformer and CNN path, and then generates a binary mask and a class for each query, respectively. Max-DeepLab achieves new SoTA with 51.3% PQ on COCO testdev set, but leads to heavy computational costs due to its dual-path high-resolution processing. Segmenter [105] views the semantic segmentation task as a sequence-to-sequence problem. In detail, a set of mask embeddings that represent different semantic classes are fed into the Transformer encoder together with image patches, and then a set of labeled masks are predicted for each patch via an argmax operation.
Unlike the conventional semantic segmentation methods that predict mask at the pixel level, Cheng et al. reformulate the semantic segmentation task as a mask prediction problem and enable this output format to the query-based Transformer, which is called Maskformer [32]. Different from Max-DeepLab [31], Maskformer leverages a simple Transformer decoder without redundant connection as well as a sigmoid activation for overlapping binary masks selection. It not only outperforms the current per-pixel classification SoTA on largeclass semantic segmentation datasets, but also generalizes the panoptic segmentation task with a new SoTA result (Tab. III).

C. Discussion
We summarize the aforementioned Transformers according to three different tasks. Table III(a) focuses on ADE20K (170 classes). It can be shown that when trained on the datasets with large numbers of classes, the segmentation performance of visual Transformers is improved significantly. Table III(b) focuses on COCO test dataset for instance segmentation. Clearly, the visual Transformers with mask embeddings surpass most prevailing models for both segmentation and detection tasks. However, there is a huge performance gap between AP box and AP seg . With the cascaded framework, QueryInst [102] attains the SoTA among various Transformer models. It is worthy of further study for combining the visual Transformers with the hybrid task cascade structures. Table III(c) focuses on panoptic segmentation. Max-DeepLab [31] is general to solve both foreground and background in the panoptic segmentation task via a mask prediction format, while Maskformer [32] successfully employs this format for semantic segmentation and unifies both semantic and instance segmentation tasks into a single model. Based on their performances in the panoptic segmentation field, we can conclude that the visual Transformers could unify multiple segmentation tasks into one box-free framework with mask prediction.

VI. TRANSFORMER FOR 3D VISUAL RECOGNITION
With the rapid development of 3D acquisition technology, stereo/monocular images and LiDAR (Light Detection And Ranging) point clouds become the popular sensory data for 3D recognition. Discriminated from the RGB(D) data, point cloud representation pays more attention to distance, geometry, and shape information. Notably, such a geometric feature is significantly suitable for Transformer on account of its characteristic on sparseness, disorder, and irregularity. Following the success of 2D visual Transformers, substantial approaches are developed for 3D visual analysis. This section exhibits a compact review for 3D visual Transformers following Representation learning, Cognition mapping, and Specific processing.

A. Representation Learning
Compared with conventional hand-designed networks, visual Transformer is more appropriate for learning semantic representations from point clouds, in which such irregular and permutation invariant nature can be transformed into a series of parallel embeddings with positional information. In view of this, Point Transformer [106] and PCT [107] firstly demonstrate the efficacy of the visual Transformer for 3D representation learning. The former merges a hierarchical Transformer [106] with the down-sampling strategy [200] and extends their previous vector attention block [25] to 3D point clouds. The latter first aggregates neighbour points and then processes such neighbour embeddings on a global off-set Transformer where a knowledge transfer from Graph Convolution Network (GCN) is applied for noise mitigation.
Notably, the positional encoding, a significant operation of the visual Transformer, is diminished in both the approaches because of points' inherent coordinate information. PCT directly processes on the coordinates without positional encodings, while Point Transformer adds a learnable relative positional encoding for further enhancement. Lu et al. leverage a localglobal aggregation module 3DCTN [108] to achieve local enhancement and cost-efficiency. Given the multi-stride downsampling groups, an explicit graph convolution with maxpooling operation are used to aggregate the local information within each group. The resulting group embeddings are concatenated and fed into the improved transformer [106], [107] for global aggregation. Park et al. present Fast Point Transformer [109] to optimize the model efficiency by using voxel-hashing neighbor search, voxel-bridged relative positional encoding, and cosine similarity based local attention.
For dense prediction, Pan et al. propose a customized pointbased backbone Pointformer [110] for attending the local and global interactions separately within each layer. Different from previous local-global forms, a coordinate refinement operation after the local attention is adopted to update the centroid point instead of the surface one. And a local-global cross attention model fuses the high-resolution features, followed by global attention. Fan et al. return to a Single-stride Sparse Transformer (SST) [111] rather than the down-sampling operation to address the problem for small scale detection. Similar to Swin [36], a shifted group in continuous Transformer block is adopted to attend to each group of tokens separately, which further mitigates the computation problem. In voxel-based methods, Voxel Transformer (VoTr) [112] separately operate on the empty and non-empty voxel positions effectively via local attention and dilated attention blocks. VoxSeT [113] further decomposes the self-attention layer into two crossattention layers, and a group of latent codes link them to preserve global features in a hidden space.
Following the mentioned methods in Sec. III-G, a series of self-supervised Transformers are also extended to 3D spaces [114]- [116]. Specifically, Point-BERT [114] and Point-MAE [115] directly transfer the previous works [71], [72] to point clouds, while MaskPoint [116] changes the generative training scheme by using a contrastive decoder as similar as DINO (2022) [93] for binary noise/part classification. Based on large experiments, we can conclude that such generative/contrastive self-training methods empower visual Transformers to be valid in either images or points.

B. Cognition Mapping
Given rich representation features, how to directly map the instance/semantic cognition to the target outputs also arouse considerable interests. Different from 2D images, the objects in 3D scenes are independent and can be intuitively represented by a series of discrete surface points. To bridge the gap, some existed methods transfer domain knowledge into 2D prevailing models. Following [30], 3DETR [117] extends an end-to-end module for 3D object detection via farthest point sampling and Fourier positional embeddings for object queries initialization. Group-Free 3D DETR [118] applies a more specified and stronger structure than [117]. In detail, it directly selects a set of candidate sample points from the extracted point clouds as the object queries and updates them in the decoder layerby-layer iteratively. Moreover, the K-closed inside points are assigned positive and supervised by a binary objectiveness loss in both sampler and decoder heads. Sheng et al. proposes a typical two-stage method that leverages a Channel-wise Transformer 3D Detector (CT3D) [119] to simultaneously aggregate proposal-aware embedding and channel-wise context information for the point features within each proposal.
For monocular sensors, both MonoDTR [120] and Mon-oDETR [121] utilize an auxiliary depth supervision to estimate pseudo Depth Positional Encodings (DPEs) during the training process. In MonoDETR [120], DPEs are first attached with the image features for Transformer encoder and then serve as the inputs of the DETR-like [30] decoder to initialize the object queries. In MonoDETR [121], both visual features and DPEs are first extracted by two different encoders parallelly and then interact with object queries via two successive cross-attention layers. Based on foreground depth supervision and narrow categorisation interval, MonoDETR obtains the SoTA result on the KITTI benchmark. DETR3D [122] introduces a multicamera 3D object detection paradigm where both 2D images and 3D positions are associated by the camera transformation matrices and a set of 3D object queries. TransFusion [123] further takes the advantages of both LiDAR points and RGB images by interacting with object queries through two Transformer decoder layers successively. More multi-sensory data fusion are introduced in Sec. VII-A.

C. Specific Processing
Limited by sensor resolution and view angle, point clouds are afflicted with incompletion, noise, and sparsity problems in real-world scenes. To this end, PoinTr [124] represents the original point cloud as a set of local point proxies and leverages a geometry-aware encoder-decoder Transformer to migrate the centre point proxies towards incomplete points direction. SnowflakeNet [125] formulates the process of completing point clouds as a snowflake-like growth, which progressively generates child points from their parent points implemented by a point-wise splitting deconvolution strategy. A skip-Transformer for adjacent layers further refines the spatialcontext features between parents and children to enhance their connection regions. Choe et al. unify various generation tasks (e.g. denosing, completing and super-resolution) into a Point cloud Reconstruction problem, hence termed PointRecon [126]. Based on voxel hashing, it covers the absolute-scale local geometry and utilizes a PointTransformerlike [106] structure to aggregate each voxel (the query) with its neighbours (the value-key pair) for fine-grained conversion from the discrete voxel to a group of point sets. Moreover, an amplified positional encoding is adapted to the voxel local attention scheme, implemented by using a negative exponential function with L1-loss as weights for vanilla positional encodings. Notably, compared with masked generative self-training, the completion task directly generates a set of complete points without the explicit spatial prior of incomplete points.

VII. TRANSFORMER FOR MULTI-SENSORY DATA STREAM
In the real world, multiple sensors are always used complementarily rather than a single one. To this end, recent works start to explore different fusing methods to cooperate multi-sensory data stream effectively. Compared with the typical CNNs, Transformer is naturally appropriate for multi-stream data fusion because of its nonspecific embedding and dynamically interactive attention mechanism. This section details these methods according to their data stream sources: Homologous Stream and Heterologous Stream.

A. Homologous Stream
Homologous stream is a set of multi-sensory data with similar inherent characteristics, such as multi-view, multidimension, and multi-modality visual stream data. They can be categorized into two groups: Interactive Fusion and Transfer Fusion, according to their fusion mechanism.
1) Interactive Fusion: The classical fusion pattern of CNN adopts a channel concatenation operation. However, the same positions from different modalities might be anisotropic, which is unsuitable for the translation-invariant bias of CNN. Instead, the spatial concatenation operation of Transformer enables different modalities to interact beyond the local restriction.
For the local interaction, MVT [127] spatially concatenates the patch embeddings from different views and strengthens their interaction via a modal-agnostic Transformer encoder. Considering the redundant features from different modalities, MVDeTr [128] projects each view of features onto the ground plane and extends the multi-scale deformable attention [77] to a multi-view design. TransFuser [129], COTR [130], and mmFormer [134] deploy a hybrid model. TransFuser models image and LiDAR inputs separately by using two different convolution backbones and links the intermediate feature maps via a Transformer encoder together with a residual connection. COTR shares the CNN backbone for each of view images and inputs the resulted features into a Transformer encoder block with a spatially expanded mesh-grid positional encoding. mmFormer exploits a modality-specific Transformer encoder for each sequence of MRI image and a modality-correlated Transformer encoder for multi-modal modeling.
For the global interaction, Wang et al. [131] leverage a shared backbone to extract the features for different views. Instead of pixel/patch wise concatenation in COTR [130], the extracted view-wise global features are spatially concatenated to perform view fusion within a Transformer. Considering the angular and position discrepancy across different camera views, TransformerFusion [133] first converts each view feature into an embedding vector with the intrinsics and extrinsics of their camera views. These embeddings are then fed into a global Transformer whose attention weights are used for a frame selection so as to compute efficiently. To unify the multisensory data in 3D detection, FUTR3D [132] projects the object queries in DETR-like decoder into a set of 3D reference points. These points together with their related features are subsequently sampled from different modalities and spatially concatenated to update the object queries.
2) Transfer Fusion: Unlike the interactive fusion implemented by the Transformer encoder with self-attention, the other fusing form is more like a transfer learning from the source data to the target one via a cross-attention mechanism. For instance, Tulder et al. [135] insert two cooperative cross-attention Transformers into the intermediate backbone features for bridging the unregistered multi-view medical images. Instead of the pixel-wise attention form, a tokenpixel cross-attention is further developed to alleviate arduous computation. Long et al. [136] propose a epipolar spatiotemporal Transformer for multi-view image depth estimation. Given a single video containing a series of static multi-view frames, the neighbour frames are first concatenated and the epipolar is then warped into the centre camera space. The resulted frame volume finally serves as the source data to perform fusion with the centre frame through a cross-attention block. With the spatially-aligned data streams, DRT [137] first explicitly models the relation map between different data streams by using a convolution layer. The resulting maps are subsequently fed into a dual-path cross-attention to build both local and global relationships parallelly, thereby it can collect more regional information for glaucoma diagnosis.

B. Heterologous Stream
Visual Transformers also perform excellently on heterologous data fusion, especially in visual-linguistic representation learning. Although different tasks may adopt different training schemes, such as supervised/self-supervised learning or compact/large-scale datasets, we here categorize them into two representative groups only according to their cognitive forms: 1) Visual-Linguistic Pre-Training including Vision-Language Pre-training (VLP) and Contrastive Language-Image Pretraining (CLIP), 2) and Visual Grounding such as Phrase Grounding (PG), Referring Expression Comprehension (REC). More comparisons please see Tab. V.
1) Visual-Linguistic Pre-Training: Due to limited annotated data, early VLP methods commonly rely on off-theshelf object detector [201] and text encoder [5] to extract data-specific features for joint distribution learning. Given an image-text pair, an object detector pre-trained on Visual Genome (VG) [202] first extracts a set of object-centric RoI features from the image. The RoI features serving as visual tokens are then merged with text embeddings for pre-defined tasks pre-training. Basically, these methods are grouped into dual-stream and single-stream fusion.
The dual-stream methods, including ViLBERT [139] and LXMERT [140], apply a vision-language cross-attention layer between two data-specific frameworks for multi-modal transferring fusion. Concretely, ViLBERT [139] is pre-trained through Masked Language Modeling (MLM), Masked Region Classification (MRC), and Image Text Alignment (ITA) on Conceptual Captions (CC) [203] with 3M image-text pairs. LXMERT [140] extends the pre-training datasets to a largescale combination and further indicates that the pre-trained task-specific (BERT [5]) weights initialization is harmful to the pre-training of multi-sensory data fusion.
VideoBERT [138] is the first single-stream VLP method, which clusters latent space features of each video frame as vi-sual tokens and organizes their corresponding text embeddings by using a captioning API. Subsequently, these features are together fed into a cross-modality self-attention layer for joint representation learning. Following [138], VisualBERT [141] extends such a single-stream framework for various imagetext tasks and adds a segment embedding to distinguish between textual and visual tokens. VL-BERT [142] suggests that unmatched image-caption pairs over the ITA pre-training may decrease the accuracy of downstream tasks. And the authors further introduce both text-only corpus and unfrozen detector strategies for pre-training enhancement. Instead, such a "harmful" pre-training strategy is refuted by UNITER [143], and the authors deploy an optimal transport loss to explicitly build Word-Region Alignment (WRA) at the instance level. To the same end, Oscar [144] uses shared linguistic semantic embeddings of a salient object class (called tag) as an anchor point to link both region and its paired words. Zhou et al. propose Unified VLP [145] to handle both generation and understanding tasks via a shared Transformer encoder-decoder with two customized attention masks. Without extra auxiliary training, Unified VLP only adopts MLM during pre-training and attains superior results on Visual Question Answering (VQA) [204] and Visual Captioning (VC) [205] tasks.
However, these methods rely heavily on the visual extractor or predefined visual vocabulary, leading to a bottleneck of the VLP expressive upper bound. To address this issue, VinVL [147] develops an improved object detector for VLP pre-training on multiple large-scale dataset combination. Instead of the object-centric RoI features, ViLT [146] initializes the interaction Transformer weights from a pre-trained ViT, and adopts whole word masking and image augmentation strategy for VLP pre-training. UniT [151] follows the architecture of DETR and applies a wide range of task for unified Transformer pre-training via different task-specific output heads simultaneously. SimVLM [152] adopts [40] to obtain image features and designs a Prefix Language Modeling as pretraining objective to generalize zero-shot image captioning.
Besides the conventional pre-training scheme with multitask supervision, another recent line has been developed for contrastive learning. The most representative work is CLIP [148]. Based on the 400M Internet image-text pairs datasets, both image and text encoder are jointly trained by a contrastive loss for ITA. Different from previous methods, CLIP enables the pre-trained model with a linear classifier to zero-shot transfer to the most visual downstream tasks efficiently by embedding the whole semantics of the objective dataset's classes. Based on extensive experiments on over 30 existing CV tasks (e.g., classification and action recognition), CLIP attains superior results to classical supervised methods, demonstrating that such task-agnostic pre-training is also generalized well in the CV field. ALIGN [150] further expands a noisy dataset of over one billion image alt-text pairs rather than the elaborate filtering or post-processing steps in CLIP [148]. Combining masked modeling and contrastive learning pre-training strategy, Data2Vec [153] proposes a selfdistilled network treating the masked features as a type of data augmentation, whose structure is analogous to DINO (2021) [74]. By testing on different sensory benchmarks (  Fig. 12. The overview of CILP (from [148]).
(voice, image, and language), it achieves competitive or better results compared with the existing self-supervised methods.
2) Visual Grounding: Compared with VLP, visual grounding has more concrete target signal supervision whose objective is to locate the target objects according to their corresponding descriptions. In the image space, Modulated DETR (MDETR) [154] extends its previous work [30] to phrase grounding pre-training that locates and assigns the bounding box to each instance phrase in one description. Based on the proposed combined dataset from many existing ones, MDETR is first pre-trained on the 1.3M aligned textimage pairs for PG and then fine-tuned on other downstream tasks. During pre-training, the image-text pair features are separately processed by two specific extractors, and fed into a DETR-like Transformer for salient object localization. Besides the box loss, two auxiliary losses are adopted to enforce network to model an alignment between image feature and their corresponding phrase tokens. With the large-scale imagetext pairs pre-training, MDETR can be easily generalized in few-shot learning, even on long-tail data. Different from MDETR [154] adding two auxiliary losses for box-phrase alignments, Referring Transformer [157] directly initializes object queries with phrase-specific embeddings for PG, which explicitly reserves an one-to-one phrase assignment for final bounding box prediction. VGTR [156] reformulates the REC as a task for single salient object localization from the language features. In detail, a text-guided attention mechanism encapsulates both self-attention block and text-image crossattention one to update the image features simultaneously. The resulted image features, which serve as the key-value pairs, interact with language queries when regressing bounding box coordinates in the decoder. Following ViT [29], TransVG [155] keeps the class token to aggregate the image and language features simultaneously for the mentioned object localization in REC. Pseudo-Q [158] focuses on REC for the unsupervised learning, where a pseudo-query generation module based on a pre-trained detector and a series of attributes&relationship generation algorithm is applied to generate a set of pseudo phrase descriptions, and a query prompt is introduced to match feature proposals and phrase queries for REC adaptation.
In the 3D spaces, LanguageRefer [159] redefines the multistream data reasoning as a language modeling problem, whose core idea is to omit point cloud features and infuse the predicted class embeddings together with a caption into a language model to get a binary prediction for object selection. Following the conventional two-stream methods, TransRe-fer3D [160] further enhances the relationship of the object features by using a cross-attention between asymmetric object relation maps and linguistic features. Considering the specific view for varied descriptions, Huang et al. present a Multi-View Transformer (MVT 2022) [161] for 3D visual grounding. Given a shared point cloud feature for each object, MVT first appends the converted bounding box coordinates to the shared objects in order to get specific view features. These multiview features are then fed into a stack of the Transformer decoders for text data fusion. Finally, the multi-view features are merged by an order-independent aggregation function and converted to the grounding score. MVT achieves the SoTA performance on Nr3D and Sr3D datasets [206]. In the video space, a specific 3D data (with temporal dimension), Yang et al. propose TubeDETR [162] to address the problem of Spatio-Temporal Video Grounding (STVG). Concretely, a slow-fast encoder sparsely samples the frames and performs crossmodal self-attention between the sampled frames and the text features in the slow branch, and aggregates the updated sample features into the full-frame features from fast branch via a broadcast operation. A learnable query attached with different time encodings, called time-specific queries in the decoder is then predicted as either a time-aligned bounding box or "no object". It attains SoTA results on STVG leaderboards.

VIII. DISCUSSION AND CONCLUSION
This section briefly conducts a summary of the performance improvements provided in Sec. VIII-A, some critical issues discussed in Sec. VIII-B, future research directions suggested in Sec. VIII-C, and final conclusion given in Sec. VIII-D.

A. Summary of Recent Improvements
We briefly summarize the major performance improvements for three fundamental CV tasks as follows.
(1) For classification, a deep hierarchical Transformer backbone is valid for decreasing the computational complexity [42] and avoiding the feature over-smooth [38], [43], [67], [68] in the deep layer. Meanwhile, the early-stage convolution [40] is enough to capture the low-level features, which can significantly enhance the robustness and reduce the computational complexity in the shallow layer. Moreover, both the convolutional projection [55], [56] and the local attention mechanism [36], [45] can improve the locality of the visual Transformers. The former [57], [58] may also be a new approach to replace the positional encoding.
(3) For segmentation, the encoder-decoder Transformer models may unify three segmentation sub-tasks into a mask prediction problem via a set of learnable mask embeddings [31], [105], [199]. This box-free approach has achieved the latest SoTA performance on multiple benchmarks [199]. Moreover, the specific hybrid task is cascaded with the model [102] of the box-based visual Transformers, which have demonstrated a higher performance for instance segmentation.
(4) For 3D visual recognition, the local hierarchical Transformer with a scoring network could efficiently extract features from the point clouds. Instead of the elaborate local design, the global modeling capability enables the Transformer to easily aggregate surface points. In addition, the visual Transformers can handle multi-sensory data in 3D visual recognition, such as multi-view and multi-dimension data.
(5) The mainstream approaches of visual-linguistic pretraining has gradually abandoned the pre-trained detector [146] and focused on the alignments [148] or similarities [153] among different data streams in the latent space based on the large-scale noised datasets [150]. Another concern is to adapt the downstream visual tasks to the pre-training scheme to perform zero-short transferring [148]. (6) The recent prevailing architecture for multi-sensory data stream fusion is the single-stream method, which spatially concatenates different data streams and performs interaction simultaneously. Based on the single-stream model, numerous recent works devote to finding a latent space to make different data streams semantically consistent.

B. Discussion on Visual Transformers
Despite that the visual Transformer models are evolved significantly, the "essential" understanding remains insufficient. Therefore, we will focus on reviewing some key issues for a deep and comprehensive understanding.
1) How Transformers Bridge the Gap Between Language and Vision: Transformers are initially designed for machine translation tasks [1], where each word of a sentence is taken as a basic unit representing the high-level semantics. These words can be embedded into a series of vector representations in the N-dimensional feature space. For visual tasks, each single pixel of an image is unable to carry semantic information, which is not full compliance with the feature embedding as done for the traditional NLP tasks. Therefore, the key for transferring such feature embeddings (i.e., word embedding) to image features and applying Transformer to various vision tasks is to build an image-to-vector transformation and maintain the image's characteristics effectively. For example, ViT [29] transforms an image into patch embeddings with multiple lowlevel information under strong slackness conditions. And its votarist [40], [59] leverages convolution to extract the lowlevel features and reduce the redundancy from patches.
2) The Relationship between Transformers, Self-Attention and CNNs: From the perspective of CNNs, its inductive bias is mainly shown as locality, translation invariance, weight sharing, and sparse connection. Such a simple convolutional kernel can perform template matching efficiently in lower-level semantic processing but its upper-bound tend to be lower than Transformers due to the excessive bias.
From the perspective of Transformers, as detailed in Sec. III-B and Sec. III-D, attention layer can theoretically express any convolution when a sufficient number of heads are adopted [28]. Such fully-attentional operation can combine both local-level and global-level attentions, and generate attention weights dynamically according to the feature relationships. Dong et al. demonstrate that the self-attention layer manifests strong inductive bias towards "token uniformity" when it is trained on deep layers without short connection or FFNs [169]. Yu et al. also argue that such an elaborate attention mechanism can be replaced by a pooling operation readily [207]. Therefore, it is concluded that Transformer must consist of two key components: a global token mixer (e.g., selfattention layer) aggregates the relationship of tokens, and a position-wise FFN extracts the features from the inputs. By comparison, the visual Transformer has a powerful global modelling capability, making it efficiently attend to high-level semantic features. CNNs can effectively process the low-level features [40], [59], enhance the locality of the visual Transformers [54], [82], and append the positional features via padding operations [57], [58], [174].
3) Double Edges of Visual Transformers: We conclude three double-edged properties of visual Transformers as follows. Global property enables Transformer to acquire capacious receptive fields and interact easily between various high-level semantic features, while it becomes inefficiency and debility during low-level processing because of quadratic computing and noised low-level features. Slack bias offers visual Transformer a higher upper bound than CNNs based on sufficient training data without sophistic assumptions but performs inferiority and slow convergency in small datasets [208]. Low-pass is also a significant property of visual Transformer showing excellent robustness, whereas it is insensitive to lowlevel features (e.g., complicated textures and edges) compared with CNN. Accordingly, it is concluded that Transformers at a high-level stage play a vital role in various vision tasks. 4) Learnable Embeddings for Different Visual Tasks: Various learnable embeddings are designed to perform different visual tasks, such as class token, object query, and mask embedding. These learnable tokens are mainly adopted into two different Transformer patterns, i.e., encoder-only and encoder-decoder ones, as illustrated in Fig. 13. On the quantity level, the number of learned tokens depends on the target prediction. For example, the visual Transformers [29], [41] in the classification task adopt only one class token, and the DETR's votarist in detection [30], [82] and segmentation [199] tasks employ multiple learned queries. On the position level, encoder-only Transformers capitalize on the initial token(s) [29], [89] and later token(s) [43], [105], while the learned positional encoding [30], [82], [199] and the learned decoder input embedding [77] are applied to the encoder-decoder structure. Different from the vanilla ViT with initial class token, CaiT [43] observes that the later class token can reduce FLOPs of Transformer and improve model performance slightly. Segmenter [105] also shows such strategy efficiency for the segmentation tasks. From the viewpoint of the encoder-decoder Transformer, the decoder input token is considered as a special case of the encoder-only Transformer with later token. It standardizes visual Transformers in the fields of detection [30] and segmentation [199] by using a small set of object queries (mask embeddings). By combing both later tokens and object queries (mask embeddings), the structure like Deformable DETR [77], which takes object queries and the learnable decoder embeddings (equivalent to the later tokens) as the inputs, may unify the learnable embeddings for different tasks into the encoder-decoder Transformer.

C. Future Research Directions
Visual Transformers have achieved significant progresses and obtained promising results. However, some key technologies are still insufficient to cope with complicated challenges in the CV fields. Based on the above analysis, we point out some promising research directions for future investigation.
1) Set Prediction: Touvron et al. found that multiple class tokens would converge consistently due to the same gradient from the loss function [41], whereas it does not emerge in dense prediction tasks [30], [199]. We conclude that their marked difference lies in the label assignment and the number of targets. Thus, it is natural to consider a set prediction design for the classification tasks, e.g., multiple class tokens are aligned to mix-patches via set prediction, like the data augmentation training strategy in LV-ViT [44]. Furthermore, the label assignment in the set prediction strategy leads to training instability during the early process, which degrades the accuracy of the final results. Redesigning the label assignments and set prediction losses may be helpful for the detection frameworks.
2) Self-Supervised Learning: Self-supervised pre-training of Transformers has standardized the NLP field and obtained tremendous successes in various applications [2], [5]. Because of the popularity of self-supervision paradigms in the CV field, the convolutional Siamese networks employ contrastive learning to perform self-supervised pre-training, which differs from the masked auto-encoders used in the NLP field. Recently, some studies have tried to design self-supervised visual Transformers to bridge the discrepancy of pre-training methodology between vision and language. Most of them inherit the masked auto-encoders in the NLP field or contrastive learning schemes in the CV field. There is no specific supervised method for the visual Transformers, but it has revolutionized the NLP tasks such as GPT-3. As described in Sec. VIII-B4, the encoder-decoder structure may unify the visual tasks by learning the decoder embedding and the positional encoding jointly. Thus it is worth of further investigating the encoderdecoder Transformers for self-supervised learning.

D. Conclusion
Since ViT demonstrated its effectiveness for the CV tasks, the visual Transformers have received considerable attentions and undermined the dominant of CNNs in the CV field. In this paper, we have comprehensively reviewed more than one hundred of visual Transformer models which have been successively applied to various vision tasks (i.e., classification, detection, and segmentation) and data streams (e.g., images, point clouds, image-text pairs, and other multiple data streams). For each vision task and data stream, a specific taxonomy is proposed to organize the recently-developed visual Transformers and their performances are further evaluated over various prevailing benchmarks. From our integrative analysis and systematic comparison of all these existing methods, a summary of remarkable performance improvements is provided in this paper, four essential issues for the visual Transformers are also discussed, and several potential research directions are further suggested for future investment. We do expect that this review paper can help readers have better understandings of various visual Transformers before they decide to perform deep explorations.

APPENDIX A OVERVIEW OF DEVELOPMENT TREND ON VISUAL
TRANSFORMERS Transformer backbones sprang up within the last year. When our systematics matches the timeline of these models, we can clearly trace the development tendency of Transformer for image classification (the Fig. 1 in main text). As a type of self-attention mechanism, visual Transformers are mainly redesigned according to either the vanilla structure in NLP (ViT [29] and iGPT [69]) or attention-based model in CV (VTs [52] and BoTNet [53]).

APPENDIX B MORE DETAIL FORMULA OF DETR
The bipartite matching loss L match is applied between predictionŷ σ(i) and ground-truth objects y i to identify one-to-one In this paper, we propose Deformable DETR, which mitigates the slow convergence and high complexity issues of DETR. It combines the best of the sparse spatial sampling of deformable convolution, and the relation modeling capability of Transformers. We propose the deformable attention module, which attends to a small set of sampling locations as a pre-filter for prominent key elements out of all the feature map pixels. The module can be naturally extended to aggregating multi-scale features, without the help of FPN (Lin et al., 2017a). In Deformable DETR , we utilize (multi-scale) deformable attention modules to replace the Transformer attention modules processing feature maps, as shown in Fig. 1.
Deformable DETR opens up possibilities for us to exploit variants of end-to-end object detectors, thanks to its fast convergence, and computational and memory efficiency. We explore a simple and effective iterative bounding box refinement mechanism to improve the detection performance. We also try a two-stage Deformable DETR, where the region proposals are also generated by a vaiant of Deformable DETR, which are further fed into the decoder for iterative bounding box refinement.
Extensive experiments on the COCO (Lin et al., 2014) benchmark demonstrate the effectiveness of our approach. Compared with DETR, Deformable DETR can achieve better performance (especially on small objects) with 10× less training epochs. The proposed variant of two-stage Deformable DETR can further improve the performance. Code is released at https://github. com/fundamentalvision/Deformable-DETR.

RELATED WORK
Efficient Attention Mechanism. Transformers (Vaswani et al., 2017) involve both self-attention and cross-attention mechanisms. One of the most well-known concern of Transformers is the high time and memory complexity at vast key element numbers, which hinders model scalability in many cases. Recently, many efforts have been made to address this problem (Tay et al., 2020b), which can be roughly divided into three categories in practice.
The first category is to use pre-defined sparse attention patterns on keys. The most straightforward paradigm is restricting the attention pattern to be fixed local windows. Most works (Liu et In back propagation, the Hungarian loss L H includes a negative log-likelihood loss for all label predictions (i = 1 · · · N ) and a box loss for all matched pairs (c i = ∅) as (7) APPENDIX C MORE DETAILED FORMULA OF DEFORMABLE DETR Given L feature maps X l ∈ R H l ×W l ×C and a query sequence z ∈ R Nq×C , MSDA samples offsets ∆p ∈ R 2 of each query for N k sample keys at each layer's head via two linear layers. While it is sampling the features of key points V i ∈ R L×N k ×Cv , a linear projection layer is applied to the query to generate an attention map A i ∈ R Nq×L×N k for key samples, where N q and C are query's length and dimension, respectively (see Fig. 14). The process is formulated as     ALIGN [150] • EfficientNet [12] Noise English al-text data [150] 1.8B Contrasive ICML 2021 SimVLM [152] • CoAtNet [40] Noise English al-text data [150] 1.8B PLM ICLR 2022 Data2Vec [153] • ViT [29] ImageNet-1k LS-960 Books Corpus & English Wikipedia data 1k 960h 1M Self-Distillation Arxiv 2022 datasets, data size, and objective task for each visual Transformer during the pre-training. The architecture (single-/dualstream) and the type of visual token inputs are also organized for each approach.