\(\bullet\)_Human alignment_. The related research of human alignment can be dated back to the year 2017 (or earlier) for OpenAI: a blog article entitled "learning from human preferences"18 was posted on the OpenAI blog describing a work that applied reinforcement learning (RL) to learn from the _preference comparisons_ annotated by humans [79] (similar to the _reward training_ step in the aligning algorithm of InstructGPT in Figure 12). Shortly after the release of this RL paper [79], the paper of the Proximal Policy Optimization (PPO) [128] was published in July 2017, which now has been the foundational RL algorithm for learning from human preferences [66]. Later in January 2020, GPT-2 was fine-tuned using the aforementioned RL algorithms [79, 128], which leveraged human preferences to improve the capacities of GPT-2 on NLP tasks. In the same year, another work [129] trained a summarization model for optimizing human preferences in a similar way. Based on these prior work, InstructGPT [66] was proposed in January 2022 to improve the GPT-3 model for human alignment, which formally established a three-stage _reinforcement learning from human feedback (RLHF)_ algorithm. Note that it seems that the working of "_instruction tuning_" has seldom been used in OpenAI's paper and documentation, which is substituted by _supervised fine-tuning on human demonstrations_ (_i.e._, the first step of the RLHF algorithm [66]). In addition to improving the instruction following capacity, the RLHF algorithm is particularly useful to mitigate the issues of generating harm or toxic content for LLMs, which is key to the safe deployment of LLMs in practice. OpenAI describes their approach to alignment research in a technical article [130], which has summarized three promising directions: "training AI systems to use human feedback, to assist human evaluation and to do alignment research".