This study reveals that common 7B language models, such as LLaMA-2 7B, already exhibit strong mathematical capabilities, challenging the previous belief that advanced mathematical reasoning is exclusive to larger, more extensively pre-trained models. By significantly scaling up SFT data, we have markedly improved the stability of the model's mathematical problem-solving skills. Our methodology has enabled the Xwin-Math models to reach performance levels comparable to, and in some instances surpassing, those of their larger counterparts. Our analysis also indicates that the enhancements are primarily attributable to heightened accuracy in single-step reasoning and a extra resampling of training data can improve the accuracy of harder questions. Additionally, we see more substantial reduction of calculation errors as opposed to logical reasoning errors. Our research contributes valuable insights into the mathematical capabilities of large language models.