(Shumailov et al., 2023) define model collapse and attribute it to two mechanisms: finite sampling when training a model (leading to cut off of low-probability data) and function approximation errors (the model is not sufficiently expressive to model the true distribution). They observe (and, for a single Gaussian, prove) that upon iteratively resampling finite "training data" the generated distribution becomes more and more peaked. Other models studied empirically are mixtures of (two) Gaussians and VAEs on MNIST. To study language models, (Shumailov et al., 2023) iteratively fine tune Meta's OPT-125M model on wikidata2. For generation of new text they use a 5-way beam search, which, by its nature, (approximatively) generates only low-perplexity data.