**Transformers Learning the GCD.** Our first illustration of our theory "in the wild" is for sequence-to-sequence transformer models for an arithmetic task: predicting the greatest common divisor (GCD) of two integers, encoded as sequences of digits in some base \(B\) following Charton (2023). This setup is a perfect intermediate step between our toy models and large scale LLMs; it uses the transformer architecture and training algorithms on sizeable models, while the underlying data has a deterministic nature. Over the course of training the model progressively learns new GCDs and with them also their products with already learned GCDs. We can thus view each such learned group, usually learned in "bursts", as a new skill. For the purpose of this experiment, we use this model after \(300M\) samples as the generator of AI-data. In Figure 4 we validate the predicted scaling law for a single generation and observe 'un-learning' of skills when training exclusively with generated data, as well as a grokking effect when training with mixtures. See Appendix G for a full description and more details and Figures.