Continually Pre-trained ModelsTo ablate the performance of combining LR re-warming and re-decaying with replay, we opt to train one model that exclusively re-warms and re-decays the learning rate and another that combines both techniques. Given results from the previous section for the weak distribution shift, we select 5% replay for the weak shift setting and 25% replay for the stronger shift setting. For both models, we re-warm to the \(\eta_{\textit{max}}\) of pre-training (\(3\cdot 10^{-4}\)) and re-decay it using a cosine decay schedule set to reach \(\eta_{\textit{min}}\) by the end of continual pre-training. More hyperparameters are reported in Table 13 of the appendix.