To understand if our primitives will generalize to structured prediction tasks on larger images, we conduct initial investigations with the simple attention-convolutional hybrids on detection and instance segmentation, using the Mask R-CNN [1]} framework. These hybrids are also faster and consume less memory than pure attention models, enabling faster experimental cycles. We only replace the last 3 convolutional layers in the ResNet-50 and ResNet-101 backbones with two halo layers with block size, \(b=8\)  and halo size \(h=3\)  (Rows 3 and 6 in Table REF ). For ResNet-50, we also examine using \(b=32\)  and halo size \(h=3\)  to understand benefits from larger receptive fields. We also use squeeze-and-excitation with convolutions and pre-train them on \(512\times 512\)  images with the regularizations mentioned in Section REF : label smoothing, RandAugment, and stochastic depth. We train our models on the COCO dataset [2]} with \(1024\times 1024\)  size images for 32 epochs, using the Cloud TPU Detection Codebase https://github.com/tensorflow/tpu/tree/master/models/official/detection. We provide more training details in the Appendix REF .
