Large Language Models For Reinforcement Learning.Nevertheless, even state-of-the-art models frequently manifest logical errors and a range of illusions [70, 71]. These anomalies become especially challenging within domains necessitating multi-step reasoning, where a singular logical misstep maybe precipitate the unraveling of an entire solution. An effective strategy involves the training of reward models aimed at discriminating between favorable and unfavorable outputs [36]. Early outcome-based approaches were mainly performed on algorithmic tasks [72, 73, 74, 75]. [42] demonstrated the significant benefits of reward models or validators, and [76] proposed a heuristic-based step-size-aware RM. [77, 78, 2, 79] proposed the use of reward models for a reinforcement learning pipeline. [20, 37, 38, 39, 42, 80, 81, 82] employed rejection sampling for searching to achieve alignment of LLMs with human preferences.