human labelers to annotate the preference for these pairs. The annotation process can be conducted in multiple forms, and a common approach is to annotate by ranking the generated candidate texts, which can reduce the inconsistency among annotators. Then, the RM is trained to predict the human-preferred output. In InstructGPT, labelers rank model-generated outputs from best to worst, and the RM (_i.e._, 6B GPT-3) is trained to predict the ranking. Note that, in recent work [371], the annotation of preference on response pairs has been conducted by an AI agent (usually an aligned LLM) instead of humans, which is called _"reinforcement learning from AI feedback (RLAIF)"_. LLMs trained with typical RLHF algorithms tend to generate harmless responses with less helpfulness, which is called _evasion problem_[371]. To guarantee both the harmlessness and helpfulness, RLAIF generates the AI feedback based on pre-set alignment principles in instructions [371, 372], which can also reduce the efforts of human annotation.