Validation Loss ComparisonThe results in Fig. 4 (top and bottom) show the evolution of the validation loss during continual pre-training on the respective \(\mathcal{D}_{1}\) datasets. Table 2 reports the average final validation loss for each of these models. The final loss is averaged over the last 100 iterations of training sampled at intervals of 10 iterations. We consistently observe across both distribution shifts that even the lowest tested replay of 1% significantly reduces forgetting on Pile compared to the no-replay baselines. This effect is more pronounced in the strong-shift scenario due to the larger amount of forgetting in this setting. We observe little impact on downstream performance for 1%,5%, and 10% replay when compared to the 0% baseline, showing that the forgetting benefits of replay come at little cost in our setting. However, when using an extreme amount of replay (50%), we observe that the model adapts less to \(\mathcal{D}_{0}\). Interestingly, for both datasets, the 50% replay models attain or surpass the final average validation performance of the baseline training on \(\mathcal{D}_{1}\cup\mathcal{D}_{0}\). This is curious as these model have seen 150B and 100B fewer tokens of \(\mathcal{D}_{1}\) than their respective baselines.