Pretraining large language models.We consider the task of pretraining an autoregressive language model \(p_{\theta}\) via next-token predictions (Radford et al., 2018). The training dataset \(\mathcal{D}_{\textbf{train}}=\{\mathcal{D}_{i}\}_{i=1}^{M}\) composes \(M\) domains with mixture proportions \(\mathbf{r}\in\Delta^{M-1}\). In each training step, the task first samples a batch of domain indices according to the mixture proportions and then sample sequences of \(L\) tokens from the sampled domains.