Figure 6 reports the validation loss during continual pre-training for 405M and 10B models, while Table 4 reports the average (over the last 100 iterations) final loss value for each model. As expected, we observe that all baselines and continually pre-trained models consistently improve in perplexity on both datasets from increasing parameter count. For the 405M models, we observe that \(\text{Pile}\cup\text{SP}\) achieves identical validation loss on each dataset to the baselines trained individually on them. In contrast, the 10B parameter model trained on \(\text{Pile}\cup\text{SP}\) outperforms the models trained individually on each. We hypothesize that this happens due to larger models having more capacity, thus being capable of learning at a higher rate for longer. We observe that replaying 5% pile data when continuing to pre-train on \(\text{SlimPajama}\) reduces forgetting on \(\text{Pile}\) validation by 0.19 and 0.21 for 10B and 405M parameter models respectively.