Large language models (LLMs)  have pushed the limits of natural language understanding and exhibited excellent problem-solving ability. Despite the great success, most existing open-source LLMs (e.g., LLaMA-2)  are still far away from satisfactory for solving mathematical problems due to the complex reasoning procedures. To bridge this gap, we proposeMetaMath, a finetuned language model that specializes in mathematical reasoning. Specifically, we start by bootstrapping mathematical questions by rewriting the question from multiple perspectives, which results in a new dataset called MetaMathQA. Then we finetune the LLaMA-2 models on MetaMathQA.
Experimental results on two popular benchmarks (i.e., GSM8K and MATH)  for mathematical reasoning
demonstrate that
MetaMath outperforms a suite of open-source LLMs by a significant margin. Our MetaMath-7B model achieveson GSM8K andon MATH, exceeding the state-of-the-art models of the same size byand.
Particularly,
MetaMath-70B achieves an accuracy ofon GSM8K, slightly better than GPT-3.5-Turbo.
We release all the MetaMathQA dataset, the MetaMath models with different model sizes and the training code for public use.