Therefore, for queries in the dataset, each of them already includes a ground truth response, which can provide rich process supervision signals of correct intermediate steps. MFT does not sample new correct reasoning steps but rather different preceding reasoning steps, allowing the model to predict the correct steps given different contexts. In short, the online RL sample new **actions** and **states** in a Markov Decision Process, while we sampled the **state** to make it partially observed. Adding noise to the **state** is verified to be helpful in generalizing for offline RL (Sinha et al., 2022), such as continuous noise on state embedding. We compare with other noise in Tab. 3, and we find that the discrete noise of MFT is more efficient.