To delve into this, we examine GPT2\({}_{1/4}\), which is GPT2 with its MLP layer reduced from \(d\to 4d\to d\) to \(d\to d\to d\) (thus, \(1/4\) of its original size), and GPT2\({}_{0}\), which is GPT2 but without any MLP layer.