In order to provide more extensive comparison with the state-of-the-art efficient transformers we have included experiments for different trunks at different scales. Since the primary motivation of this work is to deal with high resolution vision, we focus on the task of Cityscapes semantic segmentation a the benchmark that is a challenging task due to the high 1024\(\times \) 2048 resolution of images. For the trunks we adopt: i) the Segformer [1]} backbones B0, B1, B2, B3, under three different mixers namely AFNO, GFN and efficient self-attention (ESA); ii) Swin backbones [2]} (T, S, B), and iii) ResNet [3]} and MobileNetV2 [4]}. Results for LS and self-attention are not reported due to instability issues with half-precision training and quadratic memory usage with sequence length respectively. Note that efficient self-attention is self-attention but with a sequence reduction technique introduced in [5]}. It is meant to be a cheap approximation to self-attention. Numbers for ResNet and MobileNetV2 are directly adopted from [1]}. For training Segformer we use the same recipe as discussed in Section A.2 which consists of pretraining on ImageNet-1K classification for 300 epochs. For training Swin, we use pretrained classification checkpoints available from the original authors and then combine Swin with the Segformer head [1]}. We use the same training recipe as the Segformer models for Swin.
