Chain-of-thought prompting Wei et al. (2022) elicits the complex reasoning ability of LLMs with task-specific exemplars in a few-shot setting, which limits the scalability and generalization. To reduce the cost of hand-crafted few-shot exemplars, Kojima et al. (2022) proposes zero-shot CoT by introducing a magic phrase _Let's think step by step_ after question, which enables LLMs to generate reasoning chains in a zero-shot manner. However, zero-shot CoT suffers from poor-quality reasoning paths, coming with many mistakes. Since the diversity of demonstration plays a vital role in reasoning chains generation, Auto-CoT Zhang et al. (2023)2023f) generates the demonstrations automatically via clustering and representative exemplars selection, which improves the diversity and consistently matches or exceeds the performance of Few-shot CoT. COSP (Wan et al., 2023) introduces the outcome entropy of the question to aid demonstration selection. Xu et al. (2023) proposes Repropting to find the effective CoT prompt by employing Gibbs sampling iteratively. Meanwhile, some mistakes in reasoning chains come from missing-step errors, Wang et al. (2023f) extend the zero-shot CoT into Plan-and-Solve (PS) Prompting via devising a plan to divide the entire task into smaller sub-tasks and carrying out the sub-tasks according to the plan with more detailed instructions. LogCoT (Zhao et al., 2023c) uses symbolic logic to validate the zero-shot reasoning process, thus reducing errors in reasoning. Besides, PoT (Chen et al., 2022a) also explore language models, such as Codex, to generate an executable Python program to solve math problems in zero-shot setting via adding _Let's write a Python program step by step..._, which mitigates errors in intermediate reasoning steps. Some work introduces agents to solve reasoning problems. For example, Agent Instruct (Crispino et al., 2023a) utilizes agents to generate task-related, informative instructions, which guides LLMs to perform zero-shot reasoning.