**Experiments on LLMs.** We finetune Llama2 with LoRA, generating synthetic AI data for the next finetuning iteration. Inspired by the setup in Shumailov et al. (2023), we use Wikidata-103, partitioned into approximately \(2.2\) million sequences of 128 tokens. AI data is generated through prompt completion, using the first 96 tokens from the original sequences as prompts. The model is trained only on the last 32 tokens to preclude information leakage, i.e. the model being trained on the ground truth of the same 32 tokens. The evaluations are conducted exclusively on the same 32 tokens. We use top-p 0.9 and temperature 0.9 across all generation. The results, depicted in Figure 4 (left), illustrate a scaling law decay over several generations. The first generated dataset still contain useful but limited information and the utility of the second generation's data markedly diminishes. These phenomena corroborate the anticipated loss of scaling law and model collapse, further indicating that model collapse is even more pronounced here, highlighting the challenges in training next generation LLMs. More details and results in Appendix H.