Continuually Pre-trained ModelsTo ablate the performance of combining LR re-warming and re-decaying with replay, we opt to train one model that exclusively re-warms and re-decays the learning rate and another that combines both techniques. Given results from (Sec. 6.2) for the weak distribution shift, we select 5% replay for both model scales. For both models, we re-warm to the \(\eta_{max}\) of pre-training (\(3\cdot 10^{-4}\)) and re-decay using cosine annealing set to reach \(\eta_{min}\) by the end of continual pre-training. More hyperparameters are reported in Table 13 of the appendix.