The phenomenon can be related to grokking, which describes the improvement of performance from the random chance level to perfect generalization [31]. [31] find that this improvement can occur well past the point of overfitting. In pre-training, the models are usually underfitting instead of overfitting overall. Since the pre-training corpus is a mixture of different documents, it is possible that the model already fits some patterns--such as numerical addition--in the data, while still underfitting the overall corpus.