* We introduce _WizardMath_ model, which enhances the mathematical reasoning abilities for open-source pretrained large language model Llama-2 [20]. * We propose a new method, _Reinforcement Learning from Evol-Instruct Feedback_ (_**RLEIF**_)_, alongside _Evol-Instruct_ and Reinforcement Learning, for improving LLM reasoning performance. * _WizardMath_ surpasses all other open-source LLMs by a substantial margin in terms of mathematical reasoning, including Llama-2 70B [20], Llama-1 65B [4], Falcon-40B [21], MPT-30B [8], Baichuan-13B Chat [9] and ChatGLM2 12B [45] on both GSM8k [42] and MATH [43]. * _WizardMath_ significantly outperforms various main closed-source LLMs, such as ChatGPT [5], GPT-3.5, Claude Instant [39], PaLM-2 [44], PaLM-1 [7] and Minerva[15] on GSM8k.