We can also use our framework to diagnose instabilities with standard actor-critic algorithms used for the RLHF, such as PPO. We follow the RLHF pipeline and focus on the RL fine-tuning step outlined in Section 3. We can draw connections to the control as inference framework [20] for the constrained RL problem outlined in 3. We assume a parameterized model \(\pi_{\theta}(y\mid x)\) and minimize \(\mathbb{D}_{\text{KL}}[\pi_{\theta}(y|x)\mid\mid\pi^{*}(y\mid x)]\) where \(\pi^{*}\) is the optimal policy from Eq. 7 induced by the reward function \(r_{\phi}(y,x)\).