Mathematical capabilities have long been considered so challenging that they are thought to emerge in common language models only at a very large scale. For instance, studies by (Wei et al., 2022, 2020) suggest that only models with size exceeding 50 billion parameters can attain meaningful accuracy or benefit from chain-of-thought processing on math problems. A strategy to equip smaller language models with mathematical abilities involves creating math-specific base models trained on hundreds of billions of math-related pre-training data (Lewkowycz et al., 2022; Azerbayev et al., 2023). However, the accuracy of such models remains modest; for example, Lemma-7B (Azerbayev et al., 2023) only achieves 36.4% on the GSM8K dataset (Cobbe et al., 2021) and 18.0% on the MATH dataset (Hendrycks et al., 2021).