We begin by analyzing the MLP-Mixer architecture, an all-MLP alternative to self-attention networks with competitive performance on image classification benchmarks [1]}. The proposal is simple–apply an MLP along one dimension of the input, followed by an MLP along the opposite dimension. The simplest form of this MLP-Mixer architecture can thus be written as
\(p_{MM}^* &:= \min _{\mathbf {W}_{1j}, \mathbf {W}_{2j}} \sum _{i=1}^n \mathcal {L}(\sum _{j=1}^m \sigma (\mathbf {W}_{1j} \mathbf {X}_i) \mathbf {W}_{2j}, \mathbf {Y}_i) \nonumber \\&+ \frac{\beta }{2}\sum _{j=1}^m \Vert \mathbf {W}_{1j}\Vert _F^2 + \Vert \mathbf {W}_{2j}\Vert _F^2,\) 
