RepresentationDegenerationis a phenomenon in which pretrained models tend to adopt low-entropy singular value distributions (Jing et al., 2022). In language modeling, representation degeneration takes the form of anisotropy (Ethayarajh, 2019; Rajee & Pilehvar, 2021) and was proven to be related with the Zipfian shape of token distribution (Gao et al., 2019; Bis et al., 2021). We study this phenomenon along training and its relation with saturation.