where \(i\) is the index of the dataset on which we are currently pretraining, and \(\gamma_{j}=0\) if \(j=0\), and \(1\) otherwise. The latter is because when pretraining on \(\mathcal{D}_{i}\), we only see \(s_{i}\cdot(1-\alpha)\) tokens of \(\mathcal{D}_{i}\) because we use compute-equivalent replay, except for the first dataset \(\mathcal{D}_{0}\) where replay is not used, and where we hence see all \(s_{0}\) tokens of \(\mathcal{D}_{0}\).