Token mixers that replace the self-attention in transformers with fixed token mixing mechanisms, such as the Fourier transform (FNet), achieves comparable generalization with lower computational requirements [1]}. Other token-mixing architectures have also been proposed that use standard neural components, such as convolutional layers and multi-layer perceptrons (MLPs) for mixing visual tokens. MLP-mixer [2]} uses two MLP layers (cascade of \(1\times 1\)  convolutions) applied first to image patch sequence and then to the channel dimension to mix tokens. ConvMixer [3]} uses standard convolutions along image dimensions and depth-wise convolutions across channels to mix token information. These token mixing models perform well with lower computational costs compared to transformers without compromising generalization.
