Figure 2: **Tails of AI-generated data: Top.** Perplexity diagram of the Wikitext-103 test set, measured with Llama2–7B as the anchor model. We query the Wikitext-finetuned Llama2–7B to generate AI data, which is compared to the original set. Perplexity is calculated solely for the generated positions in both the AI and original datasets. AI data is generated for various settings of \((p,T)\). **Bottom.** Distribution of greatest common divisors (GCD) of pairs of random integers (original data (blue) scaling as \(p(GCD=k)\propto k^{-2}\)). A transformer is trained on this task on \(300M\) samples and used as a generator on a test set of randomly sampled integer pairs, giving the truncated GCD distribution.