Despite the promising performance of ViT, it still suffers from several issues. For instance, ViT requires to be trained on a large dataset. As ViT is initially trained on a JFT dataset [1]} with \(\approx 300M\)  images before being fine-tuned on ImageNet (\(\approx 1.2M\)  images) [2]}. If the dataset is insufficient, the model may perform worse than CNN-based approaches. Although the pre-trained weights can be used on various tasks, many datasets are not transferable or even with an inferior performance. Moreover, ViT is not a general-purpose backbone since it is only suitable for image classification but not for the dense prediction, such as object detection and image segmentation, due to the patch partitions.
