**DPO outline.** The general DPO pipeline is as follows: 1) Sample completions \(y_{1},y_{2}\sim\pi_{\text{ref}}(\cdot\mid x)\) for every prompt \(x\), label with human preferences to construct the offline dataset of preferences \(\mathcal{D}=\{x^{(i)},y_{w}^{(i)},y_{l}\}_{i=1}^{N}\) and 2) optimize the language model \(\pi_{\theta}\) to minimize \(\mathcal{L}_{\text{DPO}}\) for the given \(\pi_{\text{ref}}\) and \(\mathcal{D}\) and desired \(\beta\). In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using \(\pi^{\text{STF}}\), we initialize \(\pi_{\text{ref}}=\pi^{\text{SFT}}\) whenever available. However, when \(\pi^{\text{SFT}}\) is not available, we initialize \(\pi_{\text{ref}}\) by maximizing likelihood of preferred completions \((x,y_{w})\), that is, \(\pi_{\text{ref}}=\arg\max_{\pi}\mathbb{E}_{x,y_{w}\sim\mathcal{D}}\left[\log \pi(y_{w}\mid x)\right]\). This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and \(\pi_{\text{ref}}\) used by DPO. Further details related to the implementation and hyperparameters can be found in Appendix B.