LLM can be self-improved by distilling reasoning steps to solve complex problems. Huang et al. (2022) employs an LLM with self-consistency to generate reasoning chains from unlabeled data. These chains are subsequently utilized to fine-tune the model, enhancing its generalized reasoning capabilities. Zelikman et al. (2022) proposes STaR, a few-shot learning approach to improve LM's reasoning capabilities using a self-loop bootstrap strategy. SECToR (Zhang and Parkes, 2023) uses chain-of-thought to obtain arithmetic answers, then fine-tune the model to generate the answer without CoT directly.