where \(\pi_{\text{ref}}\) is a pre-trained language (translation) model \(\sigma\) is the Sigmoid function, and \(\beta\) is a hyperparameter. DPO training can be conducted in a supervised fine-tuning style, as it relies exclusively on labeled preference data and does not require interaction between agents and their environment.