Eq. (3) shows a simplified policy gradient formulation of process supervision (omitting some techniques like the KL penalty used for regularization (Schulman et al., 2017)). For a standard online RL process, the model samples a reasoning path for a question, and then the reward model is used to compute the reward for each step.