To align LLMs with human values, reinforcement learning from human feedback (RLHF) [79, 367] has been proposed to fine-tune LLMs with the collected human feedback data, which is useful to improve the alignment criteria (_e.g._, helpfulness, honesty, and harmlessness). RLHF employs reinforcement learning (RL) algorithms (_e.g._, Proximal Policy Optimization (PPO) [128]) to adapt LLMs to human feedback by learning a reward model. Such an approach incorporates humans in the training loop for developing well-aligned LLMs, as exemplified by InstructGPT [66].