We study the relationship of pre-training loss and performance on downstream tasks of language models, across model sizes, training tokens, tasks, languages, prompting types, and answer forms. Factors we have not considered are model architectures and training algorithms. We analyze the performance-loss curves of LLaMA, a language model with a slightly different architecture, and fine that the relationship holds for the model family. But there are fundamentally different model architectures, such as routed Transformers [11], and non-Transformer architectures [12; 30] beyond our consideration. Both our models and LLaMA use AdamW optimizer [27], while there are other optimizers for language model pre-training [39; 24].