EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models. To facilitate future research, we release all the code and models at https://github.com/baaivision/EVA.


Introduction
Scaling up pre-trained language models (PLMs) [9,63,76] has revolutionized natural language processing (NLP) in the past few years. The key to this success lies in the simple and scalable self-supervised learning task of masked signal prediction [31,74], with which Transformer models [101] † Interns at Beijing Academy of Artificial Intelligence.  could be scaled up to billions of parameters using nearly unlimited unlabelled data, and generalize well to a wide range of downstream tasks with little tuning. With further scaling on compute, data, and model sizes, PLMs have led to not only continuous performance improvements [51,75,76], but also a surprising emergence of in-context learning capability [9,25,107,108].
pure MIM pre-trained model could not achieve favorable performance at billion-scale model sizes [117]. We regard this gap stems from the fact that natural images are raw and information-sparse. Meanwhile, an ideal vision pretext task needs the abstraction of not only the low-level geometry & structure information, but also high-level semantics, which is hardly captured by pixel-level recovery tasks [115].
In this work, we seek a suitable MIM pretext task for large scale vision representation learning and explore its limits at the scale of one billion parameters with tens of millions of unlabeled data. Recently, there are a few trials leveraging the semantic information from image-image or image-text contrastive learning [13,22,73] for MIM pretraining [44,109,130], which perform fairly well in vision downstream tasks. However, there remains a debate that (i) tokenized semantic features could provide better supervision signal for masked modeling in vision [5,70,104], and (ii) good performances could be also achieved via a simple postdistillation process without masked prediction tasks [110]. Through a pilot empirical study, we find that simply using image-text aligned (i.e., CLIP [73]) vision features as the prediction targets in MIM scales up well and achieves satisfactory performances on a broad range of downstream benchmarks. This pre-training task draws the benefits from both the high-level semantic abstraction of image-text contrastive learning as well as the good capture of geometry & structure in masked image modeling, which typically covers the information needed for most visual perception tasks. Via this MIM pretext task, we can efficiently scale up a vanilla ViT encoder [33], dubbed EVA, to one billion parameters with strong visual representations that transfers well to a wide range of downstream tasks (Fig. 1) Notably, different from other state-of-the-art billion-scale vision foundation models that demand tens of millions of or even billions of labeled images, such as SwinV2-G using ImageNet-21K-ext-70M [64] and ViT-g/G using JFT-3B [123], EVA does not need a costly supervised training stage and only leverage images from open-sourced datasets for academic reproducibility.
Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not observed in other smaller-scale models, e.g., EVA makes a significant breakthrough in the challenging large vocabulary object-level recognition task: our model achieves almost the same performance on LVISv1.0 [39] (55.0 AP mask on val), an instance segmentation benchmark with more than 1,200 categories, as COCO [61] (   but with only 80 categories annotated. This emergent ability well matches the expectation of model scaling [108], that larger capability of model results in not only predictable performance improvements on standard benchmarks, but also unpredictable phenomenons and capabilities for resolving more challenging tasks. Going beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot that builds a bridge between vision and language. We show that initializing the image encoder via pre-trained EVA in a 1.1 billion parameters CLIP model can outperform the training from scratch counterpart on a broad range of zero-shot image / video classification benchmarks with much fewer samples and less compute. Moreover, EVA can greatly stabilize the giant CLIP's training & optimization process. Since large CLIP models usually suffer from training instability and inefficiency issues [2, 48], we hope our solution opens up a new direction for scaling up and accelerating the costly training of multi-modal foundation models.
By scaling up vision-centric foundation models with MIM pre-training to achieve strong performance on broad downstream tasks, we hope EVA would bridge the gap between vision and language with masked signal modeling, and contributes to the big convergence across different modalities.

Fly EVA to the Moon
We first conduct a series of pilot experiments for choosing an ideal vision pretext task in §2.1, then we scale up EVA pre-training via the chosen pre-training objective in §2.2. Finally, we evaluate the pre-trained representation on various downstream tasks in §2.3. Detailed experimental settings and configurations are in Appendix A.

The Feature Instrumentality Project
In this section, we seek a MIM vision pretext task with compelling transfer performance. Based on previous literature on vision pre-training, we study two promising candidates: (i) recovering the masked out tokenized semantic vision features [5,70,104], and (ii) feature distillation from strong pre-trained representation as in [110]. Both of them exploit pre-trained image-text aligned vision features (i.e., CLIP [73] vision features). Via a series of pilot experiments shown in Table 2, we find that: (i) the (additional) CLIP feature tokenization process is unnecessary for achieving good downstream performance (ii) feature distillation fails to provide consistent performance gain as the pre-training becomes longer. Instead, we find that simply reconstructing the masked out CLIP vision features conditioned on visible image patches is highly performant, which is chosen for scaling up EVA.
We clarify that this MIM pretext task is not originally proposed by us. Regressing the masked out image-text aligned vision features for MIM pre-training has been studied in MVP [109] and recently has been revisited by MILAN [44]. In this work, we show that this pretext task can scale up to billion-scale parameters and tens of millions of unlabeled images for vision-centric representation learning without (i) semantic feature quantization / tokenization [5,70], and (ii) explicitly using image-text paired pre-training data and large corpora as in BEiT-3 [104].

Pre-training
Architecture. The architecture configurations of EVA are in Table 3a. EVA is a vanilla ViT [33] with 1.0B parameters. The shape of her follows ViT giant [123] and the vision encoder of BEiT-3 [104]. We do not use relative positional embeddings [89] and layer-scale [99] during pre-training.
Pre-training objective. EVA is pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. We corrupt the input patches with [MASK] tokens, and we use block-wise masking with a masking ratio of 40% following [5,70,104]. The target for MIM pre-training is from the publicly available 1 Ope-nAI CLIP-L/14 vision tower trained on 224×224 pixel images [73]. The output feature of EVA is first normalized [3] and then projected to the same dimension as the CLIP feature via a linear layer. We use negative cosine similarity as the loss function.
Pre-training data. The data we used for pre-training EVA are summarized in Table 3b. For CC12M [16] and CC3M [88] datasets, we only use the image data without captions. For COCO [61] and ADE20K [129] datasets, we only use the train set data. ImageNet-21K [30] and Ob-ject365 [87] image data are also used. All these data are publicly accessible. The merged dataset for pre-training has 29.6 million images in total. model #param. extra labeled data image size top-1 acc.  The CLIP features we used as MIM prediction targets are trained on a 400 million image-text dataset in a selfsupervised manner. So during pre-training EVA also implicitly exploits the knowledge from this dataset to some extent. Meanwhile, these CLIP features is also widely used in other state-of-the-art representation learning & pre-training works such as the BEiT family [70,104], AI generated content [78,83,84] and large scale dataset filtering [10,85,86]. Table 3c, EVA is optimized via Adam [53] with decoupled weight decay [67] of 0.05. The peak learning rate is 1e-3 and decays according to a cosine learning rate schedule. We employed stochastic depth [46] with a rate of 0.1 for regularization and RandResizeCrop (0.2, 1) for data augmentation. Color jitter is not used.

Pre-training settings & hyper-parameters. As shown in
Pre-training infrastructure and statistics. Some basic pre-training statistics are available in Table 3d. The GPU we use is NVIDIA A100-SXM4-40GB. Pre-training code is based on BEiT [5] written in PyTorch [69]. We also adopt DeepSpeed optimization library [80] with ZeRO stage-1 optimizer [77] to save memory. We find using fp16 format with dynamic loss scaling is stable enough during the whole course of pre-training while using bfloat16 format is unnecessary. Since we use fp16 precision, EVA can also be pre-trained using 16× NVIDIA 24GB (32GB) GPUs with (without) gradient checkpointing [20].

Evaluation on Downstream Tasks
In this section, we extensively evaluate pre-trained EVA on several representative benchmarks, such as image classification ( §2.3.1), video action recognition ( §2.3.2), object detection & instance segmentation ( §2.3.3), semantic segmentation ( §2.3.4), and contrastive image-text pre-training with zero-shot evaluation ( §2.3.5). EVA achieves state-ofthe-art performance on a broad range of downstream tasks.
Results. Table 4 compares EVA with some state-of-the-art models on ImageNet-1K validation set. EVA achieves 89.6% top-1 accuracy with 336 2 inputs, comparable to BEiT-3. Using a larger image resolution of 560 2 can further boost the top-1 accuracy to 89.7%. Notice that BEiT-3 treats image classification as an image-to-text retrieval task. Therefore they leverage an additional one billion parameters pre-trained language encoder along with 35 million image-text data (21M pairs from CC12M, CC3M, SBU, COCO, VG and 14M pairs from ImageNet-21K) as well as 160GB text data in total. Meanwhile, we simply use a linear classifier on top of EVA with only ImageNet-21K image-tag data used for additional finetuning. With only publicly available data, EVA creates a new state-of-the-art image classification result on ImageNet-1K with a much neater architecture.
Robustness & generalization ability evaluation. We evaluate the robustness and generalization capability of EVA trained with an image size of 336 2 on 6 different ImageNet-1K validation set variants. In Table 5, we compare EVA with some top open-sourced models collected by the timm library 2 [111]. Following the evaluation procedure in [40], all these models are first fine-tuned on the original ImageNet-1K training set and then evaluated on different validation sets using the same fine-tuned model without further hyper- parameter selection and specialized fine-tuning. As shown in Table 5, EVA is the most competitive one in terms of absolute top-1 accuracies. However, these model various in pre-train data (from ImageNet-1K, ImageNet-21K to JFT-300M), input resolutions (from 224 2 to 800 2 ), model sizes (from hundreds of millions to one billion parameters) as well as architectures (ConvNets, vanilla & hierarchical ViTs), etc. Therefore their absolute accuracies are not directly comparable. Instead, we are more interested in the gap between the averaged top-1 accuracy on 6 validation set variants and the original ImageNet-1K validation set top-1 accuracy (the lower the better), i.e., we care about whether a model along with its training settings biases towards the original validation set and generalize well on other variants. From this perspective, EVA not only achieves the highest averaged accuracy, but also has the smallest performance gap, which reflects the excellent robustness and generalization ability of EVA.

Video Action Recognition
Datasets. For video action recognition, we evaluate EVA on Kinetics-400 (K-400) [52], Kinetics-600 (K-600) [14] and Kinetics-700 (K-700) [15] benchmarks. We first conduct intermediate fine-tuning on a merged dataset coined Kinetics-722 (K-722) that integrates videos from K-400, K-600 and K-700. We remove leaked as well as repeated videos in both training and validation sets. After this data de-duplicating process, K-722 has 0.63M training videos in total with 722 action classes.
Training & evaluation settings. EVA processes video data simply via spatial-temporal attention as [35,97] with no specific architectural adaptation for video related tasks. We first train EVA using K-722 training set for 40 epochs with 8 frames and 224 2 resolution, then we fine-tune EVA on each dataset for only 1 or 2 epochs. We set frame×crop×clip to 16×3×4 for fine-tuning and evaluation for all datasets. The  Table 6. Video action recognition. With only publicly available K-400, K-600 and K-700 as video pre-training data, EVA is also quite performant in video action recognition tasks.
Results. As shown in Table 6, EVA achieves better performance compared with some recent video-specific or large foundation models in video recognition. For reference, directly adapting image-only pre-trained EVA to K-400 without K-722 intermediate fine-tuning can also achieve a very competitive top-1 accuracy of 88.4%.

Object Detection & Instance Segmentation
Datasets. We evaluate the object detection and instance segmentation performance of EVA on both COCO [61] and LVISv1.0 [39] benchmark. COCO is a widely used objectlevel recognition benchmark consisting of 118k train, 5k val, and 20k test-dev images respectively, with 80 common object categories. LVISv1.0 is an emerging largevocabulary object-level recognition benchmark, which has more than 1,200 object categories as well as more than 2 million high quality instance segmentation masks (nearly 2× of COCO instance masks).
Notably, COCO and LVISv1.0 almost use the same set of images, and both train and val split of LVISv1.0 have a huge overlap with COCO train and val split. Meanwhile, COCO has much fewer object categories than LVISv1.0 (i.e., 80 v.s.1,200+). Therefore it is interesting and meaningful to evaluate a model's performance on both COCO and LVIS.
Metrics. For COCO, we report the standard box AP (AP box ) and mask AP (AP mask ) on both val and test-dev split. For LVISv1.0, we evaluate EVA using AP box , AP mask and AP mask rare defined in [39] on the v1.0 val set. We also report the experimental "boundary, fixed AP"(AP boundary in Table 8a) used in the LVIS 2021 challenge 3 for reference.
Training & evaluation settings. EVA uses Cascade Mask R-CNN [12] as the detector and adopts the training settings (e.g., large scale jitter data augmentation [37]) & architecture configurations (e.g., interleaved window & global (a) Results of object detection and instance segmentation on LVISv1.0 val set. "AP box , AP mask , AP mask rare " are evaluated using the rule defined in [39]. We also report the experimental "boundary, fixed AP"(AP boundary ) used in the LVIS 2021 challenge for reference.  Table 8. Object detection and instance segmentation performance on LVISv1.0 dataset (Table 8a). EVA not only achieves the state-ofthe-art results on both LVISv1.0 and COCO with the same set of architectures and settings, but also largely closes the performance gap between them (Table 8b). (timestamp: Nov 10, 2022) attention) of ViTDet [59]. Following the common practice [64,104,126], we first conduct intermediate fine-tuning for the whole detector using Objects365 [87] dataset with a resolution of 1024 2 , then we fine-tune the detector on COCO and LVISv1.0 train split respectively with 1280 2 inputs.
We report single-scale evaluation and multi-scale evaluation / test-time augmentation (tta) results of EVA for comparison. For COCO, Soft-NMS [8] is also applied. For instance segmentation task, the classification score is calibrated [47] via maskness [105].
The model architecture as well as the hyper-parameters for COCO and LVISv1.0 are almost the same (i.e., the hyperparameters are nearly "zero-shot" transferred from COCO to LVISv1.0), expect we use federated loss [131] and repeat factor sampling [39] following ViTDet on LVISv1.0.
COCO Results. Perhaps COCO is the most fierce vision benchmark. Table 7 [126], EVA demonstrates that with sufficient model size, data and pre-training, better performance can be also achieved via the classic R-CNN framework [38]. On the other hand, FocalNet and Group DETRv2 are incapable of instance segmentation due to using DINO.
Compared with SwinV2-Giant [64] and FD-SwinV2-Giant [110] that also adopt a (stronger HTC++ [17]) detector from the R-CNN family but with ∼3× model size of EVA, our approach streamlines the pre-training processes and pulls off a "Giant-killing" act via better representations.
Compared with BEiT-3, EVA shows that is possible to build a state-of-the-art object-level recognition system without exploiting (i) semantic feature quantization / tokenization [5,70], and (ii) image-text paired pre-training data and large corpora during pre-training.
LVIS Results. Table 8a summarizes the results on LVISv1.0 val set. EVA achieves state-of-the-art performance under all metrics with single-scale evaluation, outperforming the previous best approaches by a large margin.
Analysis the LVIS-COCO gap: more is different. It is interesting and valuable to evaluate and analyze a model on both COCO and LVISv1.0 benchmarks, as they share almost the same image set but with different numbers of annotated object categories. Compared with COCO which has only 80 annotated categories, LVIS annotates more than 1,200 object categories and thus naturally features a longtail distribution, which is more close to the challenging real world scenario [39]. In general, LVIS is considered to be a much harder benchmark than COCO in object-level recognition, and conventional methods usually suffer from a large performance drop on LVIS compared with COCO.
In Table 8b, we analyze the performance gap between LVISv1.0 and COCO benchmark of EVA and other stateof-the-art approaches. For previous leading methods such as ViTDet, the performance gap between AP box is around 8, and the gap between AP mask is around 5. However, using the same detector (Cascade Mask R-CNN) and almost the same settings as ViTDet pre-trained via MAE-Huge (ViTDet-H), EVA not only achieves the state-of-the-art results on both LVIS & COCO benchmarks simultaneously, but also largely closes the performance gap between them, especially for the instance segmentation task, that EVA achieves the same performance on LVIS and COCO with single-scale evaluation. Compared with ViTDet-H, we show a little larger model and stronger representations can take a great leap in the challenging large vocabulary instance segmentation benchmark-with one caveat described next.
Notice that it is inaccurate to say EVA "solves" the LVIS large vocabulary instance segmentation task based on "zero AP mask gap", and it is possible to achieve even higher AP mask on LVIS than COCO (e.g., LVIS has higher quality mask annotations for both training and evaluation). Although the AP mask rare of EVA on LVIS is much better than previous approaches, there is still a big gap between the rare categories and common / frequent categories (i.e., AP mask rare = 48.3 v.s. AP mask comm = 55.5 / AP mask freq = 57.4). From another perspective, the gap of AP box between LVIS and COCO (1.9 for EVA) may better reflect the overall gap between the large vocabulary and common vocabulary object-level recognition, since this metric eliminates the confounder of instance mask annotation quality.
Nevertheless, EVA makes a significant breakthrough in the challenging large vocabulary object-level recognition task. We believe other large vision foundation models such as SwinV2-G and BEiT-3 also have similar properties on COCO and LVISv1.0 benchmark. We hope our efforts can encourage the community to pay more attention to the relationships between different tasks while scaling up models and chasing the state-of-the-art in each individual task.  Table 9. Semantic segmentation performance on ADE20K and COCO-Stuff-164K dataset. "mIoU ss ": mIoU of single-scale evaluation, "mIoU ms ": mIoU using multi-scale evaluation.

Semantic Segmentation
Dataset. We evaluate EVA on ADE20K [129] and COCO-Stuff-164K [11]  Training & evaluation settings. We follow the task transfer pipelines of ViT-Adapter [23]+mask2former [24] but with a weakened model adaptation processes due to GPU memory limitation (40GB of VRAM): (i) relative position biases [90] are not applied. (ii) We use 8× decoders in mask2former segmentation head instead of 9×. (iii) The feature dimension in mask2former head is ∼0.6× of EVA encoder.
Results. We compare EVA with other leading semantic segmentation methods in Table 9. EVA achieves strong results in both ADE20K and COCO-Stuff-164K datasets.
On the other hand, the segmentation performance of EVA is slightly lower compared with BEiT-3 on ADE20K, we suspect this is partially due to our weakened architectural configurations.

Contrastive Language-Image Pre-training with Zero-shot Classification Evaluation
CLIP (Contrastive Language-Image Pre-training) [48,50,72,73] is a type of multi-modal foundation model that connects vision and language via contrastive image-text pre-training. CLIP can be applied to any image classification benchmark by simply providing the names of the visual categories to be recognized [1]. Thus the introduction of CLIP essentially reshapes the landscape of visual recognition. Meanwhile, CLIP features also play a central role in representation leaning [70,104], AI generated content [78,83,84] and large dataset filtering [10,85,86] Table 10. EVA as a vision-centric, multi-modal pivot. We evaluate a billion-scale contrastive language-image pre-trained (CLIP) model with the vision tower initialized from pre-trained EVA, which largely accelerates the contrastive training efficiency and shows promising zero-shot classification performance across a wide range of image / video benchmarks. The statistics & performance of EVA's MIM teacher (OpenAI CLIP-L) are also presented for reference. A thorough evaluation of EVA CLIP on 35 zero-shot benchmarks (including 27 zero-shot image classification benchmarks + 4 zero-shot video classification benchmarks + 2×2 zero-shot retrieval benchmarks) is available at link1 and link2.
In this section and Table 10, we show that EVA is not only a strong encoder for a wide range of vision downstream tasks, but also a multi-modal pivot that builds a bridge between vision and language. To demonstrate that, we train & evaluate EVA as a billion-scale CLIP's vision tower in various zero-shot image / video classification benchmarks.
Baselines and major challenges in CLIP model scaling. We compare our CLIP (dubbed EVA CLIP) with other opensourced strong CLIP competitors that exploit publicly accessible data / academic resources only. Model configurations and statistics are detailed in Table 10a.
There are two well-known major challenges of CLIP model training and scaling: (i) Large-scale Open CLIP models (e.g., Open CLIP-H & Open CLIP-g [2, 48]) usually suffer from severe training instability issues [2] and have to use bfloat16 format for optimization. (ii) The training efficiency is low, which may hinder model scaling and downstream performance. For instance, Open CLIP-g is heavily under-trained due to its large compute requirement, and its performance is even worse than the sufficiently-trained Open CLIP-H with a smaller model size.
Compared with our CLIP model, Open CLIP-H & -g are trained from scratch with much more image-text pairs (∼2.9× and ∼1.1× of ours) sampled from a much larger dataset (∼5× of ours) on ∼3× of GPUs. While by leveraging EVA, billionscale CLIP model training can be accelerated with improved zero-shot classification performance, described next.
Training settings. For our CLIP model, we initialize the vision encoder via pre-trained EVA and the language encoder from OpenAI CLIP-L. The pre-training implementation is based on Open CLIP [48]. We also adopt DeepSpeed optimization library [80] with ZeRO stage-1 optimizer [77] to save memory. We find using fp16 format with dynamic loss scaling is stable enough during the whole course of training while using bfloat16 format is unnecessary. These modifications allow us to train a 1.1B CLIP with a batch size of 41k on 256× NVIDIA A100 40GB GPUs.
Evaluation settings. We evaluate zero-shot image / video classification performance of each CLIP model on 12 benchmarks and report top-1 accuracy for comparisons.
Results. Table 10b shows the comparison. Our EVA CLIP achieves the highest averaged accuracy, and performs the best in 10 out of 12 zero-shot classification benchmarks. Notably, the ImageNet-1K validation zero-shot top-1 accuracy is 78.2% without using any of its training set labels, matching the original ResNet-101 [41]. Moreover, our model is quite robust and suffers from the smallest performance drop when facing natural distribution shifts in ImageNet.
At last, in Table 11 we provide zero-shot, linear probing & end-to-end fine-tuning top-1 accuracy of EVA-CLIP on ImageNet-1K validation set for reference. Our approach creates the new state-of-the-art results among all existing self-supervised learning methods. Notice that EVA CLIP's vision branch learns from Ope-nAI CLIP-L, while language branch initialized from the same CLIP-L model. Therefore, starting from a CLIP-L with only 430M parameters, we progressively scale up a 1.1B EVA CLIP-g with large performance improvements. This implies that interleaved MIM & image-text contrastive pre-training could be an efficient and scalable CLIP training approach. To our knowledge, EVA CLIP-g is the largest performant CLIP model trained via publicly accessible data and resources. We hope our practice on scaling and improving CLIP can also inspire and transfer to the study of other large scale multi-modal foundation models.

Related Work
Masked image modeling (MIM) learns rich visual representations via predicting masked visual contents conditioned on visible context. ViT [33] and iGPT [18] report the first meaningful MIM pre-training results. The BEiT family [5,70,104] greatly improves MIM's performance via masked visual token prediction. Recent work [4,21,32,34,40,106,116,130] (re-)explore pixel / feature regression in MIM, but only in a relatively small model and data scales. In this work, we explore the limits of large scale MIM pre-training via masked image-text aligned feature prediction [44,109].
Vision foundation models. ConvNets [56] have long been the de-facto standard visual architecture ab initio. Since AlexNet [55], ConvNets have rapidly evolved and become deeper, wider and larger [41,45,65,91,93,96,114]. However, at sufficient model and data scales, ConvNets lag behind ViTs [33] due to a lack of scalable pre-training tasks and the built-in inductive biases. Entering the 2020s, large pretrained ViTs [33,123] such as SwinV2-G [64] with hierarchical architectures as well as BEiT-3 [104] with multi-modal representations started to demonstrate various vision benchmarks. In this work, we show by leveraging unlabeled images, vanilla ViT can be efficiently scaled up to billion-scale parameters, and stands out in various downstream tasks.
cleaned K-722 contains 0.63M training videos, covering 722 human action classes. Table 15 lists the detailed settings & hyper-parameters for fine-tuning on this dataset. In the second stage, we further fine-tune on each dataset using more input video frames of 16 with a resolution of 224 2 . For the frame sampling, we adopt the sparse sampling strategy [103]. During testing, we follow the common practice of multi-view inference [35,66,97,106] with 4 temporal clips and 3 spatial crops. The final prediction is the ensemble of all trials. Table 16 lists the detailed hyper-parameters for fine-tuning on K-400, K-600 and K-700.

A.3. Object Detection & Instance Segmentation
The detailed hyper-parameters are shown in Table 17 and  Table 18. For intermediate fine-tuning on Objects365 [87], the model is trained with a batch size of 128 for 380k iterations. To accelerate the training process, we use a smaller input resolution of 1024 2 for the first 320k iteration. Af-terward, the input resolution is lifted to 1280 2 for a better adaptation to the fine-tuning of COCO and LVIS.
For fine-tuning COCO and LVIS, the learning rate is initialized as 2.5e-5 and step by a factor of 10 for the last 5k iterations. As shown in Table 18, we use almost identical hyper-parameters for training COCO and LVIS. Except for the commonly used repeat factor sampling [39] and federated loss [131] that are specialized for long-tailed recognition, the only difference in training is that we train the model for 45k steps on COCO, while a longer 75k step on LVIS, since the tail classes generally take a longer schedule to converge [36].

A.4. Semantic Segmentation
Detailed configurations about semantic segmentation are available in Table 19. Our settings basically follow ViT-Adapter [23] with Mask2Former [24] as the segmentation head. For ADE20K, we use COCO-Stuff pre-trained weights as initialization.  Table 19. COCO-Stuff-164K and ADE20K semantic segmentation fine-tuning settings.