Note that while \(\ell(y|x)\) has the same form as the pre-training loss \(L\), they are not equal. First, the pre-training loss is an average of all the tokens in all the documents pre-trained on. According to our empirical observation, the losses of different documents are not uniform. Second, if \(x\) and similar documents do not exist in the pre-training corpus, \(\ell(y|x)\) is the generalization loss, which is often related to other factors beyond the training loss, such as the model size. For example, in computer vision, a highly over-parameterized models often improve over an under-parameterized models in test performance when both models converge on the training data [9; 4].