Our experimental results assume that continually pre-trained LLMs undergo two or more pre-training phases sequentially. That is, our results apply to situations where a continually pre-trained LLM is randomly initialized and pre-trained on datasets \(\mathcal{D}_{0},\mathcal{D}_{1},\ldots,\mathcal{D}_{N-1}\) in sequence where \(N\geq 2\) and \(\textit{tokens}(\mathcal{D}_{i})\geq 100\)B. We note that this includes situations where the LLM in question is an open-source model (Touvron et al., 2023; 20; Jiang et al., 2023; Gemma Team et al., 2024) which has already been pre-trained on \(\mathcal{D}_{0}\) and situations where organizations may wish to train an initial LLM with the intention of continually pre-training it on new data. The new data may be similar to the previous data, corresponding to a weak distribution shift (e.g., the latest web-scrape of different domains), or quite different from previous data, corresponding to a strong distribution shift (e.g., data from a completely new language).