**Implementation Details.** For each task and dataset, we evaluate the compared LLMs using the same prompts and results parsing method provided by existing work (_i.e.,_ TruthfulQA, HotPotQA, Gorilla, HaluEval) or designed according to our empirical experience (_i.e.,_ TriviaQA, Natural Questions, Web Questions, ARC, WikiFact, GSM8k, MATH, C-Objects, Penguins, LAMBADA, WMT22, XSum, HumanEval, CrowS-Pairs, WinoGender, RealToxicityPrompt). Specifically, all the experiments about closed-source models are based on invoking their official APIs, while for open-source models, we utilize their publicly available code and model parameters, and perform the inference on 8 A800-80G GPUs. For TriviaQA, OpenbookQA, HellaSwag, and SocialIQA, we experiment on the development set since the test set is not publicly released. While for other datasets, we experiment on the test set. To reproduce our experiments, we also publicly release our experimental code and data in [https://github.com/RUCAIBox/LLMSurvey/tree/main/Experiments](https://github.com/RUCAIBox/LLMSurvey/tree/main/Experiments).