\(\bullet\)_Alignment tuning_. Since LLMs are trained to capture the data characteristics of pre-training corpora (including both high-quality and low-quality data), they are likely to generate toxic, biased, or even harmful content for humans. It is necessary to align LLMs with human values, _e.g.,_helpful, honest, and _harmless_. For this purpose, InstructGPT [66]designs an effective tuning approach that enables LLMs to follow the expected instructions, which utilizes the technique of _reinforcement learning with human feedback_[66, 79]. It incorporates human in the training loop with elaborately designed labeling strategies. ChatGPT is indeed developed on a similar technique to InstructGPT, which shows a strong alignment capacity in producing high-quality, harmless responses, _e.g._, rejecting to answer insulting questions.