Reinforcemnt Learning Another form of FT relies on RL or RL-inspired optimization strategies to better align the model after its pre-training phase. A few prominent examples are reinforcement learning from human feedback (RLHF) (OpenAI, 2023; Touvron et al., 2023), direct preference optimization (DPO) (Rafailov et al., 2023), and proximal policy optimization (PPO) (Schulman et al., 2017; Tunstall et al., 2023).