After applying DPO for preference alignment, the model's score on MTBench (Zheng et al., 2024) increased from 6.89 after SFT to 7.25, surpassing even large models such as Llama2-70B-Chat (see Figure 13). However, we also noticed that the performance on benchmarks are slightly compromised, which is known as the alignment tax (Askell et al., 2021).