We evaluate the performance of intermediate training checkpoints. The checkpoints are saved around every 43B tokens during pre-training. We plot the points of task performance (\(y\)-axis) and training loss (\(x\)-axis) in Figure 1. From the curves, we can see that the training loss is a good predictor of the performance on 12 downstream tasks.