Figure 3: **Illustration of Our Main Results for Simplified LLMs. Left plot. Empirical confirmation of the double scaling law. The true distribution of the data is Zipf with exponent \(\beta=3/2\). Broken lines correspond to \(k^{-(\beta-1)}\), for varying \(T\) and different values of \(k\). Middle plot. Model collapse over multiple generations. Again \(\beta=3/2\), \(T_{0}=T\) across all generations with no additional tail-cutting, regeneration for 5 times. Right plot. Notice the grokking behavior, as perfectly predicted by the Theorem 3.2. For any given value \(\pi\) for the proportion of real data, the broken lines are asymptotes \(E_{test}\asymp(\pi T)^{-c}\) and each plateau has length of order \(k^{d}/\pi\), both predicted by the theorem. See Figure 10 for similar results with other values of \(k\).**of generations moves the loss scaling curve progressively to the right. This leads to eventual model collapse.