In Figure 6, we observe that perplexity starts to noticeably decrease when the rank of the language modeling head \(W\) is inferior to 1000, _regardless of the model size_. This hints that the head is not a major performance bottleneck for models with greater hidden dimensions, but that it may hurt performance for models with smaller ones independently of the quality of the output representations.