undergo two or more pre-training phases sequentially. At the start of each phase, we reset the optimizer states, since optimizer states may not always be available, e.g. when using open-weight models from HuggingFace. That is, our results apply to situations where a continually pre-trained LLM is randomly initialized and pre-trained on datasets \(\mathcal{D}_{0},\mathcal{D}_{1},\ldots,\mathcal{D}_{N-1}\) in sequence where \(N\geq 2\). For the realistic settings we consider _tokens_(\(\mathcal{D}_{i}\)) \(\geq\) 100B.