Global Context Vision Transformers
We propose global context vision transformer (GC ViT), a novel architecture
that enhances parameter and compute utilization. Our method leverages global
context self-attention modules, joint with local self-attention, to effectively
yet efficiently model both long and short-range spatial interactions, without
the need for expensive operations such as computing attention masks or shifting
local windows. In addition, we address the issue of lack of the inductive bias
in ViTs via proposing to use a modified fused inverted residual blocks in our
architecture. Our proposed GC ViT achieves state-of-the-art results across
image classification, object detection and semantic segmentation tasks. On
ImageNet-1K dataset for classification, the base, small and tiny variants of GC
ViT with $28$M, $51$M and $90$M parameters achieve $\textbf{83.2\%}$,
$\textbf{83.9\%}$ and $\textbf{84.4\%}$ Top-1 accuracy, respectively,
surpassing comparably-sized prior art such as CNN-based ConvNeXt and ViT-based
Swin Transformer by a large margin. Pre-trained GC ViT backbones in downstream
tasks of object detection, instance segmentation, and semantic segmentation
using MS COCO and ADE20K datasets outperform prior work consistently, sometimes
by large margins. Code available at https://github.com/NVlabs/GCViT.