In terms of fine-tuning, whether through Supervised Fine Tuning (SFT) or Reinforcement Learning (RL), the primary aim is to gather high-quality supervisory signals for model improvement. For SFT, the approach generally involves leveraging larger models or manual efforts to collect high-quality data (An et al., 2023; Yu et al., 2023; Yue et al., 2023; Luo et al., 2023). MetaMath(Yu et al., 2023; Gou et al., 2023) augments its dataset by rewriting mathematical queries and provide answers with the help of an LLM. It also leverages backward reasoning to enhance the model's capabilities. MAmmoTH(Yue et al., 2023) compiles diverse mathematical datasets for training, enriching them with Chains of Thought (CoT) and Programs of Thought (PoT) rationales for reasoning. RL-based methods strive to gather superior training signals through self-sampling and reward supervision. Lightman et al. (2023) proposes to use process supervision to give more precise signals than outcome supervision. Some strategies adopt the process supervision into the online RL training (Wang et al., 2023b; Xi et al., 2024; Luo et al., 2023). RFT(Yuan et al., 2023) is a simplified offline method of standard online RL. It enhances its training dataset with accurate reasoning paths by generating and collecting them via a SFT model. This approach is further extended by ReST (Singh et al., 2023) through an iterative process.