In the following sections, we will focus on model studies to distinguish the advantages of self-attention over convolutions for vision and and understand how to best design self-attention vision architectures.
This knowledge is important since much of the progress in convolutional networks comes from improvements in architecture design while keeping the core convolution primitive the same [1]}, [2]}, [3]}. We believe our study is the first to explicitly examine the design of optimal self-attention vision architectures.
