Given the need to mitigate forgetting when re-warming and re-decaying, we move on to investigate the effects of replay in our weak and strong-shift continued pre-training scenarios. Specifically, we use compete equivalent replay (see Sec. 4.2 for details) where replay tokens from \(\mathcal{D}_{0}\) are added at the cost of removing the equivalent number of \(\mathcal{D}_{1}\) tokens from the budget. Following the same two dataset settings, the model is pre-trained on \(\mathcal{D}_{0}\) (Pile) for 300B tokens. This is followed by continual pre-training on a SlimPajama (weak shift) or German Common Crawl (strong shift). For more details regarding the setup, please see Section 5.2. Our continued pre-training is conducted for the full size of the respective datasets, which is 300B tokens for SlimPajama (weak shift) and 200B tokens for German Common Crawl (strong shift). We consider 1%, 5%, 10%, and 50% replay for both shifts and add 0.5% and 25% replay runs for the weak and strong distribution shifts respectively. We consider two baselines to put these results into a broader context. The first baseline is a model trained on \(\mathcal{D}_{1}\) without replay. The second baseline model is trained from random initialization on a union of \(\mathcal{D}_{0}\) and \(\mathcal{D}_{1}\) for 600B tokens (SlimPajama) and 500B tokens (German Common Crawl). The latter baseline reflects the practice of fully re-training the model to update it instead of continually pre-training the existing model. All models re-warm and re-decay the learning rate using a cosine decay schedule fit to their token budget with the same \(\eta_{\textit{max}}\) (\(3\cdot 10^{-4}\)) and \(\eta_{\textit{min}}\) (\(3\cdot 10^{-5}\)) values as during pre-training on \(\mathcal{D}_{0}\).