Figure 7: **Pile validation loss when continuing to pre-train on Pile (a) and SlimPajama (b). Each curve starts from the same checkpoint pre-trained on 300B tokens of Pile but is trained with a different maximum learning rate. As we focus on the effect of re-warming the learning rate, we only show curves for the first 100B tokens.