Figure 11: **Replay v.s. no replay when re-warming the learning rate and continually pre-training on different amounts of data. We train 405M parameter models with and without 5% replay on 100B, 200B, and 300B tokens of SlimPajama data.