Figure 3: Comparing new tokens vs. repeated tokens for random data selection and D4 for fixed selection ratio \(R=0.25\) for 1.3B OPT pre-training. Each method chooses 25% of documents from the source dataset \(D_{source}\), and epochs over that subset until the target token budget of 40B is reached. We observe that repeating tokens via D4 outperforms baseline training (random, new tokens).