Existing work considers Domain Adaptive Continual Pre-training (DACPT), a setting where a series of unlabelled domains become available to the LM sequentially and practitioners wish to train on each domain in a self-supervised fashion while retaining performance across each of them. While the objective is similar to our own, we consider general-purpose pre-training datasets that mix many domains as opposed to domain-specific datasets. Ke et al. (2022) assume data from previous domains is not available when training on new domains and develop a new technique for this setting which involves an importance mask of parameters for all previous tasks to prevent forgetting when pre-training with a masked language modeling (MLM) objective. Gururangan et al. (2020) investigated domain and task adaptive pre-training of RoBERTa (also MLM) and contributed a sample selection strategy for efficient continual pre-training. Similarly, Xie et al. (2023) also propose a data selection strategy that reduces the computational cost of continual pre-training (shown for autoregressive LLMs). Qin et al. (2023) investigate re-cycling fine-tuned adapter layers of previous base LMs as the initialization of new adapters for adapting continually updated versions of the base LM to specific tasks. Recently, Wu et al. (2024) proposed LLaMA Pro, a method for the continual pre-training of LLMs that enables learning new tasks without forgetting previous knowledge. However, unlike our work which considers adapting all existing weights, LLaMA Pro requires growing the size of the model for each new update and only adjusting the new weights.