To evaluate this ability, existing studies consider multiple criteria for human alignment, such as helpfulness, honesty, and safety [368, 46, 170]. For helpfulness and honesty, adversarial question answering tasks (_e.g._, TruthfulQA [556]) can be utilized to examine LLM's ability in detecting possible falsehood in the text [36, 81]. Furthermore, harmlessness can be also evaluated by several existing benchmarks, _e.g._, CrowS-Pairs [603] and Winogender [604]. Despite the automatic evaluation with the above datasets, human evaluation is still a more direct way to effectively test the human alignment ability of LLMs. OpenAI invites many experts in domains related to AI risks to evaluate and improve the behaviors of GPT-4 when encountering risky contents [46]. In addition, for other aspects of human alignment (_e.g._, truthfulness), several studies propose to use specific instructions and devise annotation rules to guide the annotation process [81]. Empirical studies have revealed that these strategies can greatly improve the human alignment ability of LLMs [170]. For instance, after alignment tuning on data collected through interactions with experts, the incorrect behavior rate of GPT-4 can be largely reduced when it deals with sensitive or disallowed prompts. In addition, high-quality pre-training data can reduce the effort required for alignment [46]. For instance, Galactic is potentially more harmless due to the less biased contents in the scientific corpus [35].