We also believe that further exploration of the specific nature of the singular components after the collapse we describe in Section 4.2 could improve our understanding of LM saturation. We hypothesize that the resulting dominating components are correlated with token frequency, based on previous works that link anisotropy with token frequency (Gao et al., 2019; Ethayarajh, 2019; Bis et al., 2021) and show the importance of token frequency in the LM head mechanism (Meister et al., 2023).