**Methods.** In addition to DPO, we evaluate several existing approaches to training language models to adhere to human preferences. Most simply, we explore zero-shot prompting with **GPT-J**[43] in the summarization task and 2-shot prompting with **Pythia-2.8B**[3] in the dialogue task. In addition, we evaluate the **SFT** model as well as **Preferred-FT**, which is a model fine-tuned with supervised learning on the chosen completion \(y_{w}\) from either the SFT model (in controlled sentiment and summarization) or a generic LM (in single-turn dialogue). Another pseudo-supervised method is **Unlikelihood**[44], which simply optimizes the policy to maximize the probability assigned to \(y_{w}\) and _minimize_ the probability assigned to \(y_{l}\); we use an optional coefficient \(\alpha\in[0,1]\) on the 'unlikelihood' term. We also consider **PPO**[37] using a reward function learned from the preference data and **PPO-GT**, which is an oracle that learns from the ground truth reward function available in the controlled sentiment setting. In our sentiment experiments, we use two implementations of PPO-GT, one of-the-shelf version [42] as well as a modified version that normalizes rewards and further tunes hyperparameters to improve performance (we also use these modifications when running 'normal' PPO with learned rewards). Finally, we consider the **Best of \(N\)** baseline, sampling \(N\) responses from the SFT model (or Preferred-FT in dialogue) and returning the highest-scoring response according to a reward function learned from the preference dataset. This high-performing method decouples the quality of the reward model from the PPO optimization, but is computationally impractical even for moderate \(N\) as it requires sampling \(N\) completions for every query at test time.