MLP-Based Mixers relax the graph similarity constraints of the self-attention and spatially mix tokens using MLP projections. The original MLP-mixer [1]} achieves similar accuracy as self-attention. It is further accelerated by ResMLP [2]} that replaces the layer norm with the affine transforms. gMLP [3]} also uses an additional gating to weight tokens before mixing. This class of methods however lack scalability due to quadratic complexity of MLP projection, and their parameter inefficiency for high resolution images.
