Recent research, including studies by Hoffmann et al. (2022) and Muennighoff et al. (2024), highlights the importance of a balanced approach in scaling both model size and training data to enhance model capabilities. Specifically, enlarging pre-training datasets has been emphasized as a crucial step forward. However, challenges arise in addressing data limitations, such as the potential pitfalls of up-sampling. Hernandez et al. (2021) demonstrated that increasing only 0.1% of the training dataset by a factor of 100 significantly reduced model efficacy, pointing out the limitations of simple data amplification strategies. Meanwhile, Muennighoff et al. (2024) approach, which involved repeating the entire pre-training dataset across multiple epochs, showed promising results.