In this paper, we demonstrate that common language models of small size, such as the LLaMA-2 7B model (Touvron et al., 2023), already possess strong mathematical capabilities without specific pre-training on math-related data.