To address the complicated nature of reinforcement learning in LLM training, direct preference optimization (DPO) Rafailov et al. (2023), among other methods Yuan et al. (2023); Dong et al. (2023), have been popularized for its simplicity and effectiveness. DPO involves curating preference datasets using human or strong AI (_e.g.,_ GPT-4 (OpenAI, 2023)) judgement to select chosen and rejected responses to questions. These datasets are used to train LLMs by comparing log probabilities of chosen versus rejected answers. However, obtaining these probabilities can be challenging with proprietary models like GPT-4, since they do not offer log probabilities for inputs.