
where \(\sigma \)  is an activation function. While [1]} use the GeLU non-linearity [2]}, we analyze the simpler linear and ReLU activation counterparts, which shed important insights into the underlying structure of the MLP-Mixer architecture.
