**Differences in Sampling Compared to Online RL.** Standard auto-regressive sampling can bring more reasoning paths to one query. However, we find that the sampled reasoning paths have minor differences in Fig. 7, such as word substitution (orange part), rewriting of formulas (red part), rearrangement of syntax (brown part), but the correct intermediate steps are consistent (green part).