SlimPajama as in Fig. 3. It is noteworthy that the higher the re-warming, the more pronounced this effect is, as seen with the \(\eta_{max}=6\cdot 10^{-4}\) curve when continuing to pre-train on Pile (with a peak loss increase of 0.2) vs continuing to pre-train on SlimPajama (peak loss increase of 0.45).