The concept of aligning models with human preference originally emerged in the context of training simple robots in virtual environments or Atari games (Christiano et al., 2017; Ibarz et al., 2018) and was subsequently applied to various Natural Language Processing tasks. For example, Ziegleret al. (2019) employed Proximal Policy Optimization (PPO) Schulman et al. (2017), an RL algorithm, to fine-tune GPT-2 Radford et al. (2019) based on human preferences, improving its performance across four NLP tasks. Building on these prior works, Ouyang et al. (2022) introduced InstructGPT, a model based on GPT-3 Brown et al. (2020), which they further fine-tuned using reinforcement learning from human feedback (RLHF). Additionally, Ouyang et al. (2022) formally outlined the RLHF algorithm, which encompasses SFT, reward model (RM) training, and reinforcement learning via Proximal Policy Optimization (PPO). The RLHF algorithm not only enhances the model's ability to follow instructions but also shows promising potential to mitigate the generation of toxic or harmful content.