**Relationship of Pre-training Loss and Task Performance.** In the transfer learning setting, [25; 43] find that models with the same pre-training loss can have different downstream performance after finetuning, due to inductive bias in model sizes, model architectures, and training algorithms. For the prompted performance of large language models, [49] claim that perplexity is a strong predictor of in-context learning performance, but the evidence is limited to the OPT model [54] and a subset of BIG-Bench [41]. Instead, [40] find that low perplexity does not always imply high in-context learning performance when the pre-training corpus changes.