Figure 1: **Continual pre-training decreases computational costs of updating the model while maintaining similar final validation and average evaluation performance. We report results for the Pile \(\cup\) SlimPajama(SP)/German(Ger.) baseline model trained on the union of both datasets which we consider to be an upper bound on performance. We also report performance for two continually pre-trained models. “PT on Pile” starts from a pre-trained Pile checkpoint and only uses learning rate re-warming and re-decaying, while “Replay (PT on Pile)” re-warms the learning rate, re-decays it, and uses 5% replay for SlimPajama and 25% replay for German. We observe that the combination of LR re-warming, re-decaying, and replay allows our continually pre-trained model to attain similar average performance to the baseline model while requiring substantially less compute.