In this paper, we present an innovative process-oriented math process reward model called\methodname, which assigns a reward score to each step of math problem solutions.
The training of\methodnameis achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work.
We explore the effectiveness of\methodnamein two scenarios:
1) Verification:\methodnameis utilized for reranking multiple outputs generated by Large Language Models (LLMs) ;
2) Reinforcement Learning:\methodnameis employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO) .
With\methodname, a series of open-source LLMs demonstrates exceptional performance.
For instance, the step-by-step PPO with\methodnamesignificantly improves the accuracy of Mistral-7B (77.9%84.1% on GSM8K and 28.6%33.0% on MATH) .
The accuracy can be further enhanced to 89.1% and 43.5% on GSM8K and MATH with the verification of\methodname, respectively.
We believe that automatic process supervision holds significant potential for the future evolution of LLMs.