In many of our experiments, we compare models trained with replay to models trained without it. When making such comparisons, we keep the amount of compute constant for training both models. That is, we correspondingly reduce the number of tokens seen from the new dataset to accommodate the additional tokens seen from the replay buffer. We refer to this use of replay as _compute-equivalent replay_. For instance, suppose datasets \(\mathcal{D}_{0}\) and \(\mathcal{D}_{1}\) each contain 100B tokens. We wish to compare model (a) trained sequentially on \(\mathcal{D}_{0}\) and \(\mathcal{D}_{1}\) to model (b) trained sequentially on \(\mathcal{D}_{0}\) and \(\mathcal{D}_{1}\) with 5% compute equivalent replay. Model (a) will see all tokens from both datasets for a total of 200B unique tokens. Model (b) will see 100B unique tokens of \(\mathcal{D}_{0}\) and 95B unique tokens of \(\mathcal{D}_{1}\) plus 5B replayed tokens from \(\mathcal{D}_{0}\) for a total of 200B tokens. In this way, both compared models expend the same amount of compute.