Figure 3 reports validation losses for the \(\mathcal{D}_{0}\) and \(\mathcal{D}_{1}\) datasets throughout the continual pre-training of all models. The top row of plots reports results for the weak distribution shift (300B Pile\(\rightarrow\)300B SP), while the bottom row reports results for the strong distribution shift (300B Pile\(\rightarrow\)200B Ger.). For both shifts, the \(\eta_{\text{min}}\) constant learning rate model achieves the least forgetting on \(\mathcal{D}_{0}\). It also adapts the least on \(\mathcal{D}_{1}\) for the strong shift, however, for the weak shift it adapts more than the \(\eta_{\text{max}}\) constant baseline. When comparing these baselines to the models that re-warm and re-decay on both datasets, we observe that the latter models adapt better to the new dataset by a significant margin for both distribution shifts. This shows that re-warming and re-decaying are necessary to maximize adaptation to the new dataset when continually pre-training LLMs. Among the models that re-warm and re-decay the LR, we observe that varying the learning rate causes small differences in adaptation and forgetting: higher values of \(\eta_{\text{max}}\) lead to more forgetting and more adaptation while the opposite is true for lower values. When comparing the baselines to the union-trained baseline, we observe that the final validation loss for \(\mathcal{D}_{1}\) is significantly higher than the union-trained model on both distribution shifts. This is also the case for \(\mathcal{D}_{1}\) on the weak distribution shift, but interestingly for the strong distribution shift, the constant baselines achieve lower \(\mathcal{D}_{1}\) validation loss than the union-trained model. We hypothesize that this is due to the stronger distribution shift generally enhancing adaptation and exacerbating forgetting in the context of LLMs. When comparing models continually pre-trained with re-warming and re-decaying to the union baseline, we note that these models adapt better (lower final validation loss) to \(\mathcal{D}_{1}\) than the union baseline. However, these models experience significant forgetting on \(\mathcal{D}_{0}\), showing the need for replay to make these models competitive with the union baseline.