@inproceedings{
xiao2024caldpo,
title={Cal-{DPO}: Calibrated Direct Preference Optimization for Language Model Alignment},
author={Teng Xiao and Yige Yuan and Huaisheng Zhu and Mingxiao Li and Vasant G Honavar},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=57OQXxbTbY}
}
We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring
their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in
scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Teng Xiao Yige Yuan Huaisheng Zhu Mingxiao Li Vasant G Honavar