Figure 20: **Hutter LLM.** true distribution of the data is Zipf with exponent \(\beta=2\). Here, the scalable resource is either clean data or AI data-generated data, corresponding to a version of real data with its tail cut at rank \(k\) (here we use \(k=10\)). We either mix with a fixed amount (here \(T^{\prime}=10^{4}\) samples) of the other resource, or we don’t mix at all. Then we scale up the scalable resource by cranking up \(T\). As predicted by Corollary 3.3, the orange curve always grokks: AI synthesized data is helpful in the regime where real data is scarce; once more of real data becomes available the model grokks for a while and then forgets the AI synthesized data. Note that the green curve (only AI data) and red curve (AI + real data) don’t grokk because the optional resource (real data) is not being scaled; if it is also scaled, then green and red will provably grokk (as in Figure 3). The diagonal broken line corresponds to the standard Hutter scaling law \(E_{test}\asymp T^{-c}\), where \(c:=1-1/\beta\). The horizontal broken lines correspond to \(E_{test}\asymp k^{-(\beta-1)}\) and \(E_{test}\asymp T^{\prime-c}\), both predicted by Theorem 2.1.