We perform experiments on two mathematical reasoning benchmarks, namely GSM8k [42] and MATH [43], the results demonstrate that our _WizardMath_ outperforms all other open-source LLMs, achieving state-of-the-art performance. Specifically, _WizardMath_ observe a substantial improvement in pass@1 with an increase of +24.8 (81.6. vs. 56.8) on GSM8k, and +9.2 (22.7 vs. 13.5) on MATH. Notably, our model even also significantly surpasses OpenAI's ChatGPT-3.5 [5], Anthropic's Claude Instant-1 [39], and Google's PaLM-2 [44] in terms of pass@1 on GSM8k.