For instance, in our settings that span only two datasets \((\mathcal{D}_{0},\mathcal{D}_{1})\), we use replay of data from \(\mathcal{D}_{0}\) when training on \(\mathcal{D}_{1}\). We replay the data in the order it was seen when pretraining on \(\mathcal{D}_{0}\), as we did not observe noticeable differences when reshuffling the replay data in preliminary experiments. The use of methods for selecting replay samples is left as future work. We refer to models using replay as "\(\mathcal{D}_{1}\)\(x\)% Replay", where \(x\) is the percentage of data in each training batch that comes from \(\mathcal{D}_{0}\). Conversely, \((100\%-x)\)% of the samples in each training batch will be sampled from \(\mathcal{D}_{1}\). When comparing models trained with replay to other configurations, we ensure that the compute is _equivalent_ by reducing the number of \(\mathcal{D}_{1}\) tokens to accommodate replay tokens from \(\mathcal{D}_{0}\).