**Autoregressive Bigram Models with Perplexity Loss.** We move one step further towards "real" LLMs to investigate autoregressive bigram models. The dataset now comprises sequentially generated integers, adhering to Equation (16), with the model trained on all tokens. We use the averaged perplexity score of the test set as the test error metric. Our study encompasses a range of effects--such as top-p inference, temperature scaling, limited real data, and training on progressively larger AI datasets. Consistent with the findings in Section 4, we observe the same patterns of scaling loss and progressive model collapse across generations. Relevant figures are provided in Appendix F.