To leverage LLMs for addressing complex reasoning tasks, Wei et al. (2022) extends in-context learning with step-by-step reasoning processes, first introducing the concept of chain-of-thought (CoT) prompting. Kojima et al. (2022) finds that simply adding a magic phrase _Let's think step by step_ in prompts enables LLMs to perform zero-shot chain-of-thought reasoning without any human annotation. These studies have highlighted the significance of chain-of-thought in enhancing the model's capability for complex reasoning and improving its reasoning and planning abilities.