* **Three datasets, no shift** : We consider an \(N=3\) setting, where \(\mathcal{D}_{0},\mathcal{D}_{1},\mathcal{D}_{2}\) are each district 100B token splits of SlimPajama. This setting is primarily used to evaluate the ability of our techniques to scale to many future updates and to assess the performance of our proposed infinite learning rate schedules. * **Domain incremental continual pre-training**: This setting considers consuming the tokens of SlimPajama sequentially ordered by domain. That is, we train on a sequence of \(N\) future datasets \(\{\mathcal{D}_{0},\mathcal{D}_{1},\ldots,\mathcal{D}_{N-1}\}\) each of is a distinct domain of SlimPajama 300B. We note that this is similar to DACPT (Ke et al., 2022), however, we consider much larger datasets for each domain. This setting is particularly challenging due to the distribution shift experience at the transition between each domain. While it is certainly interesting, we believe it is unnecessarily difficult compare to mixing the SlimPajama data before training on it. The poor results in this setting (Sec. A.1 of the appendix) suggest that general-purpose LLMs should be continually pre-trained on a mixture of domains if possible, not updated per domain.