Pre-trained language models (PLMs) can automatically learn general representations from unlabeled text and achieve excellent performance through fine-tuning on downstream tasks. Devlin et al. (2019); Raffel et al. (2020); Radford and Narasimhan (2018). Recently, scaling up language models significantly improves performance and brings many surprises, such as emergent abilities Wei et al. (2022); Schaeffer et al. (2023). Therefore, the paradigm of natural language processing is shifting from pre-training with fine-tuning to pre-training with in-context learning. However, as of now, large-scale language models (LLMs) still have considerable room for improvement on complex reasoning tasks, such as mathematical reasoning Cobbe et al. (2021); Patel et al. (2021), commonsense reasoning Talmor et al. (2021); Mihaylov et al. (2018), etc.