Inspired by InstructGPT[2] and PRMs[36], we train two reward models to predict the quality of the instructions and the correctness of each step in the answer respectively:1. Instruction Reward Model (IRM): This model aims to judge the quality of the evolved instructions on three aspects: i) Definition, ii) Precision, and iii) Integrity. To produce the ranking list training data of IRM, for each instruction, we firstly use ChatGPT and Wizard-E 4 to generate 2-4 evolved instructions respectively. Then we leverage Wizard-E to rank the quality of those 4-8 instructions. Footnote 4: Wizard-E named Wizard-E-Generator, which is an Alpha version fine-tuned Llama model specifically used to execute _Evol-Instruct_ without APIs.
2. Process-supervised Reward Model (PRM): As there is no powerful open-source math reasoning LLMs before this work, there is no simple way to support highly precise process supervision without professional human-labelers and close-source ChatGPT. Therefore, we depend on ChatGPT to provide process supervision, and ask it to assess the correctness of each step in the solutions generated by our model. 3. PPO training. We evolve the original math (GSM8k + MATH) instructions by 8 turns, increasing the data size from 15k to 96k. We use IRM and PRM to generate the instruction reward (\(r^{I}\)) and the answer reward (\(r^{A}\)). Then apply a product as the final reward \(r=r^{I}\cdot r^{A}\).