Adapting replay to Domain Incremental continual pre-trainingIn settings that span more than two tasks, we use a form of reservoir sampling (Buzzega et al., 2020), where the replay buffer is updated at discrete intervals. We refer to this technique as _discrete reservoir sampling_. Specifically, given a sequence of \(N\) datasets \(\mathcal{D}_{0},\mathcal{D}_{1},\dots,\mathcal{D}_{N-1}\) of sizes \(s_{0},s_{1},\dots,s_{N-1}\) that are trained on in sequence, we update the replay buffer \(\mathcal{R}\) at each dataset transition. Let \(\mathcal{R}_{i}\) correspond to the state of the replay buffer as the start of training on the \(i\)-th dataset.