* **Two datasets, weak shift**: In this variation, we consider \(\mathcal{D}_{0}\) to be the Pile (Gao et al., 2020) and \(\mathcal{D}_{1}\) to be pre-training on SlimPajama (Soboleva et al., 2023). SlimPajama is an extensively deduplicated version of RedPajama (Computer, 2023) which is built based on the LLaMA dataset (Touvron et al., 2023a). We consider this to be a weak but realistic distribution shift as both datasets are English-language, contain common domains(CommonCrawl, GitHub, Arxiv, Wikipedia, and StackExchange) as well as other non-overlapping domains. Additionally, SlimPajama (2023) is a newer dataset than Pile (2020) and is therefore likely to have newer data within the overlapping domains. Therefore, despite the potential overlap, we believe this transition is realistic and is likely to be of interest to many practitioners wishing to update an LLM on a similar distribution to pre-training (e.g., newly collected data of the same sources with higher quality filtering). * **Two datasets, stronger shift**: In this variation, we consider \(\mathcal{D}_{0}\) to be pre-training on the Pile (Gao et al., 2020) and \(\mathcal{D}_{1}\) to be pre-training on German Common Crawl. German Common Crawl is a \(\sim\) 200B token dataset taken from the Oscar dataset (Laippala et al., 2022). We note that this constitutes a stronger shift given the change of language. This setting is of particular interest for practitioners wishing to augment an LLM with a new natural language, programming language, or specific domain that is notably different in vocabulary from pre-training. We note, however, that as the domain strays farther and farther away from the tokenizer's training corpus, the tokenizer may become a key bottleneck to performance. We leave the treatment of the tokenizer to future work.