This paper introduces _WizardMath_, a mathematics model fine-tuned with _RLEIF_. The experimental results demonstrate that _WizardMath_ achieves SOTA performance surpassing all existing open-source LLMs on two widely recognized mathematical reasoning benchmarks: GSM8k and MATH. Furthermore, _WizardMath_ exhibits superior performance compared to some of the largest close-source LLMs, including ChatGPT, GPT-3.5, Claude Instant, PaLM-2, PaLM-1 and Minerva on the GSM8k benchmark.