For decades, researchers in Natural Language Processing (NLP) have been exploring the fundamental principles of intelligence [1]. The recent advances in large language models (LLMs) seem to have revealed a glimmer of hope. Benefitting from the unprecedented scales of model size and training data, many LLMs like ChatGPT [14], PaLM [15], LLaMA [16], and others have emerged strong capabilities in reasoning [17], planning [18], and learning from experience [19] at or surpassing human levels. These general capabilities also provide a foundation for LLMs to address intricate real-world tasks, such as successfully completing the entire Uniform Bar Examination (UBE) [14] or coding based on natural language instructions (StabilityAI 2023).