We propose Reinforcement Learning from Contrast Distillation (RLCD) , a method for aligning language models to follow natural language principles without using human feedback.
RLCD trains a preference model using simulated preference pairs that contain both a high-quality and low-quality example, generated using contrasting positive and negative prompts. The preference model is then used to improve a base unaligned language model via reinforcement learning. Empirically, RLCD outperforms RLAIFand context distillationbaselines across three diverse alignment tasks—harmlessness, helpfulness, and story outline generation—and on both 7B and 30B model scales for preference data simulation.