\(\bullet\)_Language generation_. As discussed before, for language generation, we consider evaluating three kinds of tasks, _i.e.,_ language modeling, conditional text generation, and code synthesis. Specially, we select four commonly-used datasets, namely LAMBADA [233] (language modeling), WMT'22 [545] (machine translation), XSum [549] (text summarization), and HumanEval [105] (code synthesis) for evaluation. In WMT'22, we construct a new evaluation set by selecting 1000 examples for each language pair from the original large-scale test set to examine the average performance of LLMs in machine translation. We evaluate the zero-shot performance of LLMs on these datasets, and compute the _accuracy_ of predicting words for LAMBADA, _BLEU-4_ for WMT'22, _ROUGE-L_ for XSum, and _pass_\(@10\) for HumanEval.