The differences between outcome-based and process-based reward modelling are further discussed by [40]. Outcome-supervised reward models (ORMs) undergo training exclusively utilizing the ultimate outcomes derived from the model's chain-of-thought process. Conversely, process-supervised reward models (PRMs) are designed to solicit feedback for each individual step within the chain-of-thought progression. In the domain of logical reasoning, ORMs frequently employ incorrect reasoning pathways yet yield the correct final answer [83, 41]. Notably, PRMs has been demonstrated to effectively alleviate this phenomenon of inconsistent behavior [40]. [36, 84, 85] amassed an expansive corpus of process-based supervised signals through meticulous manual annotation, which verified that PRMs and supervision with manual annotation yielded more pronounced advantages for LLMs as compared to ORMs.