Note, 10% of domain data (2.39B) translates to less than 1% of the 300 billion tokens the original Pythia was trained on. These results demonstrate that being selective during the data curation process for continual pre-training can have large effects on domain performance at a small cost.