\[\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(y\mid x)} \big{[}r_{\phi}(x,y)\big{]}-\beta\mathbb{D}_{\text{KL}}\big{[}\pi_{\theta}(y \mid x)\mid\mid\pi_{\text{ref}}(y\mid x)\big{]} \tag{3}\]where \(\beta\) is a parameter controlling the deviation from the base reference policy \(\pi_{\text{ref}}\), namely the initial SFT model \(\pi^{\text{SFT}}\). In practice, the language model policy \(\pi_{\theta}\) is also initialized to \(\pi^{\text{SFT}}\). The added constraint is important, as it prevents the model from deviating too far from the distribution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers. Due to the discrete nature of language generation, this objective is not differentiable and is typically optimized with reinforcement learning. The standard approach [49; 38; 1; 26] has been to construct the reward function \(r(x,y)=r_{\phi}(x,y)-\beta(\log\pi_{\theta}(y\mid x)-\log\pi_{\text{ref}}(y \mid x))\), and maximize using PPO [37].