Synthetic data scaling.The impressive performance of many over-trained small language models (e.g., Mistral series models (Jiang et al., 2023), and Gemma series models (Gemma-Team et al., 2024), _inter alia_) demonstrates the necessity of training with large amount of tokens (even passing the compute-optimal chinchilla law (Rae et al., 2021)). However, whether we have similar conclusions on the training with synthetic data is still an open question, as the quality of synthetic data may not be as consistent as real-world data (Yu et al., 2024). Future research should investigate the scaling laws for synthetic data and determine the optimal balance between the quantity and quality of synthetic samples. This exploration could help us understand the most effective strategies for leveraging synthetic data in training large-scale language models, potentially leading to more efficient and cost-effective approaches (Muennighoff et al., 2024).