The representation degeneration problem is a common phenomenon that affects self-supervised learning methods used for textual data (Gao et al., 2019; Lai et al., 2023), among other modalities (Jing et al., 2022; Godey et al., 2024). Many observations on the intermediate representations of Language Models (LMs) have shed light on their low angular variability (or _anisotropy_) (Zhou et al., 2021; Rajee & Pilehvar, 2022) or on outlier dimensions that emerged during training (Puccetti et al., 2022). However, these observations were mostly made on relatively small-scale models of dimensions comparable to BERT (Devlin et al., 2019) or models from the GPT-2 family (Radford et al., 2019).