**Process-Supervised RLHF.** In existing literature of RLHF [376], the supervision signals for RL training can be generally classified into two distinct categories: outcome-supervision signals and process-supervision signals. The outcome-supervised RLHF employs a quantitative score to assess the quality of the whole text generated by LLMs. In contrast, process-supervised RLHF offers an evaluation of each individual component (_e.g._, sentence, word, or reasoning step) within the generated content, which can provide fine-grained supervision signals to guide the training, helping LLMs refine the undesired generation contents [376, 377]. OpenAI has proposed a fine-grained annotation dataset named PRM800k [377] consisting of 12K process-annotated mathematical problems (_i.e._, MATH dataset [378]) and 75K solutions generated by LLMs of these problems, where each reasoning step of mathematical problems is labeled as _positive_, _negative_ or _neutral_ in PRM800k. This fine-grained dataset has been utilized in existing work [377, 379] to train the process-supervised reward models (PRM), and the probability from the prediction of each label can be considered as the supervision signals during RLHF procedure. To effectively leverage process-supervision signals from PRMs, existing work [376] has utilized expert iteration [380, 381], an effective RL algorithm to improve the base policy via learning from expert policy. Typically, expert iteration contains two main stages: policy improvement and distillation [376]. In the policy improvement stage, expert policy processes the systematic search procedure to produce the samples. PRMs provide process-supervision signals to guide expert policy in the search procedure and enhance the quality of samples. Subsequently, during the distillation stage, the samples generated by expert policy in the first stage are utilized to improve the base policy through supervised fine-tuning. In addition to expert iteration, PRMs can also be utilized to re-rank the candidates of the final answers generated by LLMs [377] or to select better intermediate reasoning steps during step by step reasoning [382, 379].