Vision Transformers [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]} (ViTs) are emerging as an alternative to CNN for feature extractors and visual recognition. Despite the superior performance, they are generally more data-hungry than their CNN counterparts, and rely on large amounts of data and more epochs for training.
To tackle this problem, DeiT [9]} improves its data efficiency by knowledge distillation from pre-trained CNNs, coupled with a better training recipe.
Liu et al. propose a dense relative localization loss to improve ViTs’ data efficiency [10]}.
Unlike the prior works [9]}, [10]}, [13]} that focus on the data efficiency issue of transformer backbones on image classification tasks, we tackle the data efficiency issue of detection transformers on the object detection task.
