Continued Training.Similar to recent work (Google, 2023), we find that switching the data distribution and learning rate decay schedule at the end of model training greatly improves model quality. Concretely, after having trained over the entirety of our 8T pre-training dataset, we use the same loss objective and perform continued training on small number of tokens in comparison to the pre-training tokens.