**Relation to RL Fine-Tuning.** Our method can be regarded as a special case of Eq. (3) with the following assumption: 1) We relax the reward computation in Eq. (3) and assume that every token in the ground-truth data can provide a positive process reward. 2) Sampling masked position has a similar effect with sampling new paths.