Selected Token Loss Aligns Better with Downstream PerformanceWe use the reference model to filter tokens and explore the changes in validation loss after training on all/selected tokens, while observing their relationship with downstream loss. As shown in Figure 6, we pretrain approximately 4B tokens and displayed the variation curves of loss on different pretraining method and validation sets during the pretraining process. We can observe that on the tokens selected by the reference model, the decrease in average loss of the Rho-1 is more significant compared to regular pretraining. On the contrary, on the unselected tokens, the decrease in average loss of the regular pretraining is more significant. If we associate fig (a), fig(b) with fig(c), it is not difficult to find that the model trained on the selected tokens has a more significant decrease in downstream loss, while ordinary pretraining, although reducing the average loss of all tokens during the training phase, is difficult to have a significant decrease in downstream loss. Therefore, we expect that selecting tokens for pretraining is more efficient.