SetupsWe train on the mixtures of GitHub, Pile-CC, and Books3 subset from the Pile for a total of 30B tokens and evaluate the model on the three domains, respectively. For specific mixtures, we grid search from \(\{0,0.125,0.25,\ldots,0.875,1\}\)3 and retain valid ones in which three proportions sum up to 1 and do not use up all tokens in any of the domains5, which results in 32 mixtures in total.