Transformers have recently delivered tremendous success for representation learning in language and vision. This is primarily due to the attention mechanism that effectively mixes the tokens’ representation over the layers to learn the semantics present in the inputWe use the terms “attention", “token mixing", and “mixing" interchangeably throughout this paper.. After the inception of dot-product self-attention [1]}, there have been several efficient alternatives that scale nicely with the sequence size for large pretraining tasks; see e.g., [2]}, [3]}, [4]}, [5]}, [6]}. However, the learnable inductive bias of attention is not explored well. A strong theoretical understanding of attention's inductive bias can motivate designing more efficient architectures, and can explain the generalization ability of these networks.
