Aligning with shared human preference and values.Directly finetuning on value-aligned or human-preferred data is a straightforward method for aligning language models, but this method often requires substantial human annotation, which can be prohibitively expensive at scale. Additionally, such annotation frequently exhibits varying styles and inconsistent quality, particularly in the case of poorly annotated samples at the lower end of the quality spectrum (Gilardi et al., 2023b; Meta, 2023). To address these practical challenges, an advanced technique known as "reinforcement learning from human feedback (RLHF)" has been proposed (Christiano et al., 2017; Leike et al., 2018; Ouyang et al., 2022). This approach involves training a reward model with human data to act as a proxy of human judgment, which guides the optimization of the LM generation policy.