General ImprovementsAcross all tasks in Table 1, we observe that models trained on synthetic data combined with the C4 dataset (Synthetic+C4) exhibit an overall higher average performance of 49.4% as compared to those trained solely on the real C4 dataset with a 85B token split, which scored an average of 47.4%. This shows that the inclusion of synthetic data can enhance the general understanding capabilities of NLP models. Moreover, even the TinyLlama model trained for 10x compute and data, performs comparably to the other models trained on real data. This suggests that the gains from filtering out, or adding more real data are very low. As opposed to this, **WRAP** shows that pre-training on even small amounts of synthetic data can contribute to large performance gains.