We first investigate the effect of linear warm-up duration on forgetting and adaptation in the **two datasets, weak shift** and **two dataset, stronger shift** settings (see Sec. 5.2 for details). The models are pre-trained on 300B tokens of Pile (Gao et al., 2020) (\(\mathcal{D}_{0}\)). We continue to pre-train the models on SlimPajama (weak shift) and German Common Crawl (strong shift) for the first 50B tokens of training.