However, to align the model with human preferences, RLHF (Ouyang et al., 2022) presents a more complex challenge. It involves the development of alignment criteria, the acquisition of human feedback, and final learning adjustments based on this feedback. LLaMA2 (Touvron et al., 2023) is currently one of the publicly available models utilizing RLHF, with other models such as WebGPT (Nakano et al., 2021), InstructGPT (Ouyang et al., 2022), and GPT-4 (OpenAI, 2023) being proprietary. Implementing RLHF is intricate, stemming not only from the need for human annotations but also due to technical challenges. These include overfitting in reward models and instabilities during the Reinforcement Learning training phase (Gao et al., 2022). Additionally, the tedious procedure of training multiple LMs including the model being aligned, the reward model, and the inference model at the same time substantially amplifies memory and computational demands, particularly for larger models. Rafailov et al. (2023) proposed direct preference optimization (DPO) to align models with human preferences instead of complex and unstable RLHF. Nevertheless, one still needs to collect human-labeled preference data in the target language.