To address these issues, there have been proposals for supervised fine-tuning methodologies such as Rank Responses to align Human Feedback (RRHF) (Yuan et al., 2023), Reward rAnked Fine-Tuning (RAFT) (Dong et al., 2023), and Direct Preference Optimization (DPO) (Rafailov et al., 2023). These methods circumvent the intricacies inherent in reinforcement learning and have been shown to yield empirical results on par with RLHF. Notably, the DPO technique straightforwardly encourages the LLM to favor positive responses and discourage negative ones. DPO has been observed to yield performant learning outcomes, in spite of its uncomplicated training procedure.