BaselinesWe also train several baselines. Two baselines are trained on \(\mathcal{D}_{0}\) and \(\mathcal{D}_{1}\) respectively while the third is trained on the union of each dataset \(\mathcal{D}_{0}\cup\mathcal{D}_{1}\). We consider the model trained on \(\mathcal{D}_{0}\cup\mathcal{D}_{1}\) to be an upper bound on performance as it represents an expensive full re-training. The baselines trained on individual datasets can be seen as compute-equivalent alternatives to continual pre-training (e.g., one could opt to train a model from random initialization on \(\mathcal{D}_{1}\) instead of continually pre-training it).