Self-attention was the fundamental building block in the first proposed vision transformer (ViT) [1]}. It consists of an outer product of two linear functions, followed by a non-linearity and a product with another linear function, which makes it non-convex and non-interpretable. One approach to understand attention has been to design new alternatives to self-attention which perform similarly well, which may help explain its underlying mechanisms. One set of work pertains to Multi-Layer Perceptron (MLP) based architectures, [2]}, [3]}, [4]}, [5]}, [6]}, while another line of work proposes Fourier based models [7]}, [8]}, [9]}, [10]}. Others have proposed replacing self-attention with matrix decomposition [11]}. While all of these works have appealing applications that leverage general concepts about the structure of attention, they lack any fine-tuned theoretical analysis on these architectures from an optimization perspective.
