Large language models (LLMs), such as GPT-4, have shown remarkable performance in natural language processing (NLP) tasks, including challenging mathematical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. In this paper, we present _WizardMath_, which enhances the mathematical reasoning abilities of Llama-2, by applying our proposed _Reinforcement Learning from Evol-Instruct Feedback_ (**_RLEIF_**) method to the domain of math. Through extensive experiments on two mathematical reasoning benchmarks, namely GSM8k and MATH, we reveal the extraordinary capabilities of our model. _WizardMath_ surpasses all other open-source LLMs by a substantial margin. Furthermore, our model even outperforms ChatGPT-3.5, Claude Instant-1, PaLM-2 and Minerva on GSM8k, simultaneously surpasses Text-davinci-002, PaLM-1 and GPT-3 on MATH. More details and model weights are public at [https://github.com/nlpxucan/WizardLM](https://github.com/nlpxucan/WizardLM) 3 and [https://huggingface.co/WizardLM](https://huggingface.co/WizardLM).