These properties shed light on how the dimensionality of the ideal language modeling head impacts the performance when the LM head is low-rank. However, the relation obtained in Theorem 5.3 is not particularly strong, as discussed in Appendix A.2.