To address this shortcoming, we leverage convex duality to analyze a single block of self-attention with ReLU activation. Since self-attention incurs quadratic complexity in the sequence, we alternatively analyze more efficient modules. As representatives for more efficient modules, we focus on MLP Mixers [1]} and Fourier Neural Operators (FNO) [2]}. MLP-mixer mixes tokens (purely) using MLP projections in both the token and feature dimensions. In contrast, FNO mixes tokens using circular convolution that is efficiently implemented based on 2D Fourier transforms.
<FIGURE>