Apart from instruction tuning, it is important to construct high-quality datasets for aligning LLMs with human values and preferences (_e.g.,_ helpfulness, honesty, and harmlessness). In this section, we introduce several widely used datasets for alignment tuning, including HH-RLHF [170], SHP [177], PKU-SafeRLHF [181], Stack Exchange Preferences [178] and Sandbox Alignment Data [179]. We show their details in Table IV.