In this section, we aim at identifying a formal link between the inherent dimensionality of the contextual distribution and the performance bottleneck that can be attributed to the lower dimensionality of the output representations of a language model. To that end, we conceptualize a language modeling head optimized on _ideal_ contextual representations, and we explore the relationship between its spectral properties and the performance gap induced when training a low-rank head on the same representations.