We perform extensive experiments with pretraining vision transformers for upstream classification and inpainting that are then finetuned for downstream segmentation. Compared with the state-of-the-art, our AFNO using the ViT-B backbone outperforms existing GFN, LS, and self-attention for few-shot segmentation in terms of both efficiency and accuracy, e.g, compared with self-attention, AFNO achieves slightly better accuracy while being \(30\%\)  more efficient. For Cityscapes segmentation with the Segformer-B3 backbone, AFNO achieves state-of-the-art and beats previous methods, e.g. AFNO achieves more than 2% better mIoU compared with efficient self-attention [1]}, and is also competitive with GFN and LS.
