We review the RLHF pipeline in Ziegler et al. (and later [38; 1; 26]). It usually includes three phases: 1) supervised fine-tuning (SFT); 2) preference sampling and reward learning and 3) RL optimization.