\(N=2\) **settings** - Here we assume a model is available (e.g. via hugging face or pre-trained in-house) that has been pre-trained for autoregressive language modeling on a dataset (\(\mathcal{D}_{0}\)) using a linear warmup and cosine decay LR schedule. We also assume that the schedule follows existing conventions in the literature (e.g. decaying to the token budget; see Sec. 4 for details) as it is the case for most performant pre-trained LLMs (Rae et al., 2021; Hoffmann et al., 2022; Touvron et al., 2023; 1). Given a model pre-trained on \(\mathcal{D}_{0}\), we now assume that a practitioner wants to update this model on a new dataset \(\mathcal{D}_{1}\) using the same self-supervised objective.