Math.Recent advancements in mathematical reasoning for language models (LMs) have led to the development of various approaches to improve performance on math-related tasks. One approach is to train on math-targeted pre-training data, such as Minerva (Lewkowycz et al., 2022), Llemina (Azerbayev et al., 2023), and DeepSeeKMath (Shao et al., 2024). Another mainstream method is to generate synthetic questions and answers to imitate the training or validation set of target benchmarks. For instance, WizardMath (Luo et al., 2023) leverages a series of operations to increase the complexity of questions and answers using GPT-3.5, while MetaMath (Yu et al., 2023) bootstraps the questions in MATH and GSM8K by rewriting them in different ways, such as semantic rephrasing, self-verification, and backward reasoning. GAIR-Abel (Chern et al., 2023) found that the format of the augmented answers is crucial to final performance, with answers that begin with a paraphrasing of the question followed by a step-by-step solution showing better performance than those in vanilla format. Xwin-Math (Li et al., 2024) further scaled up synthetic SFT data to one million examples and found that the LLaMA-2 7B model (Touvron et al., 2023) can still benefit from data scaling. MMIQC (Liu and Yao, 2024) composed a bundle of datasets that infuse SFT style data (via question-answer rephrasing or directly taken from MetaMath) with a subset of high-quality mathematical pre-training data, such as OpenWebMath (Paster et al., 2023).