Establishing object detection baselines for ViT is challenging due to technical obstacles that include mitigating ViT's large memory requirements when processing detection-sized inputs (, \(\scriptstyle \sim \) 20\(\times \)  more patches than in pre-training), architectural incompatibilities (, single-scale ViT a multi-scale detector), and developing effective training formulae (, learning schedules, regularization and data augmentation methods, ) for numerous pre-trained initializations, as well as random initialization. We overcome these obstacles and present strong ViT-based Mask R-CNN baselines on COCO when initializing ViT from-scratch [1]}, with pre-trained ImageNet [2]} supervision, and with unsupervised pre-training using recent methods like MoCo v3 [3]}, BEiT [4]}, and MAE [5]}.
<FIGURE>