* We characterize the performance saturation of small language models through evaluation and extrapolation of the scaling laws;
* We find that the representations of smaller models degenerate concurrently with this saturation. We shed light on _rank saturation_, i.e. the explosion of the entropy of singular value distributions of small LM prediction heads;
* We empirically verify that the rank of the target contextual distribution is usually high. Moreover, we observe that regardless of the expressiveness of the output representations of a model, a linear head \(W\) substantially affects performance when \(rank(W)<1000\);
* We theoretically quantify the performance limitation induced by a low-rank linear language modeling head.