Figure 5 reports the validation loss during continual pre-training of 405M parameter models for weak (top) and strong (bottom) shifts. Table 3 reports the average (over the last 100 iterations) final loss value for these models. Since the transition from English to German represents a starker distribution shift than Pile to SlimPajama, training on German leads to significantly more forgetting on Pile (\(\mathcal{D}_{0}\)) for the continually pre-trained model without replay (0.27 vs 1.39 for weak and strong shifts respectively). However, choosing 25% replay to handle the starker shift significantly reduces the amount of forgetting on Pile, a reduction of 1.23 in terms of final loss. When comparing continually pre-trained models to baselines trained exclusively on \(\mathcal{D}_{1}\), we observe that the continually pre-trained models always have lower validation loss across both distribution shifts. When comparing the continually pre-trained models with the \(\mathcal{D}_{0}\cup\mathcal{D}_{1}\) baselines we find that both models achieve nearly identical (weak shift) or identical (strong shift) average final validation losses. This shows that for strong and weak distribution shifts, a simple and scalable combination of LR re-warming, LR re-decaying, and replay can achieve similar performance to the \(\mathcal{D}_{0}\cup\mathcal{D}_{1}\) baseline.