In this section we address two technical obstacles when using ViT as the backbone in Mask R-CNN: (1) how to adapt it to work with a feature pyramid network (FPN) [1]} and (2) how to reduce its memory footprint and runtime to make benchmarking large ViT backbones tractable.
