We consider a **domain incremental learning** setting, where we train on a sequence of \(N\) future datasets \(\{\mathcal{D}_{0},\mathcal{D}_{1},\dots,\mathcal{D}_{N-1}\}\) each of which comes from a distinct domain. This setting is particularly challenging due to the distribution shift experience at the transition between each domain. Concretely, we consider dataset \(\mathcal{D}_{0}\) to be pre-training on the Pile (Gao et al., 2020) and \(\mathcal{D}_{1}\) to be pre-training on SlimPajama. However, instead of consuming the data in \(\mathcal{D}_{1}\) using the sampling percentages from table 1, we process the dataset one domain at a time starting from the largest domain to the smallest. This simulates a situation where new data is received from different domains at different times and we wish to update our models on the sequence while being robust to all distribution shifts.