* **MMLU: We use the official MMLU** **(**Hendrycks et al.**,** 2021**)** **evaluation script and prompts available at [https://github.com/hendrycks/test](https://github.com/hendrycks/test), with modifications to allow for batch processing. We evaluate using 0 few-shot examples, following the original setup of MMLU. We report average accuracy across test examples.**
* **ToxiGen: We follow the setup in** **Touvron et al.** **(**2023b**)****, but use the original set of prompts from** **Hartvigsen et al.** **(**2022**)****, which are designed to elicit toxic generations for certain groups. We take only the prompts designed to produce toxic language ('hateful' prompts) and use 500 prompts per group to reduce evaluation costs. For base language models, we pass in the original ToxiGen prompts unchanged and greedily decode up to the first new line (or a maximum of 512 tokens). For instruction-tuned models, we place the prompt in the corresponding template, and ask the model to complete the prompt, until the model generates a stop token (or a maximum of 512 tokens). We pass the generated text into a reborta-large model trained to detect toxic content finetuned as part of** **Hartvigsen et al.** **(**2022**)****28**. We then report the percentage of generations deemed toxic by the classifier.** Footnote 28: [https://huggingface.co/tomh/toxic_roberta](https://huggingface.co/tomh/toxic_roberta)**
* **TruthfulQA: Following** **Touvron et al.** **(**2023b**)****, we mainly use the generation setting of TruthfulQA** **(**Lin et al.**,** 2022**)****. The TruthfulQA dataset contains 818 questions, which are used to prompt the tested model to generate answers. We use the default QA prompt format with 6 in-context QA examples. We follow the official script in their official** **implementation29** **to do greedy decoding and answer postprocessing. We train two LLaMA 2-based classifiers for judging the truthfulness and informativeness of the model response, due to the deprecation of GPT-3 making exact replication of the original TruthfulQA evaluation infeasible. We find that the LLaMA 2 judges are generally able to match the performance of the original GPT-3-based judges used by** **Lin et al.** **(**2022**)****. We report the rate of the responses being truthful and informative (% Informative and Truthful) following** **Touvron et al.** **(**2023b**)****. We only report the % Informative and Truthful as our primary metric.** Footnote 29: [https://github.com/sylinrl/TruthfulQA/](https://github.com/sylinrl/TruthfulQA/)**
* **AlpacaEval: We use the package provided by** **Li et al.** **(**2023**)****, following the default setup which asks the evaluated model to generate responses for 805 prompts and employ GPT-4 to compare the response with Davinci-003. We employ the "alpaca_eval_gpt4" annotator. We allow the evaluated model to generate up to 2048 tokens, without specifying special stop sequences.