**Preference Datasets.** Our model incorporates a blend of publicly accessible datasets and synthetic data from the LLM. The open-source Chinese datasets consist of non-harmful and beneficial sections from _cvalues.rllf_, _comparison.gpt4_data_zh_ and _oast.rm_zh_ in LLama-factory (Zheng et al., 2024), huozi, and zhihu. For English, the dataset includes _comparison.gpt4_data_en_ from LLama-factory and beavertails (Ji et al., 2024). To construct a more high-qualities preference dataset via a synthetics approach, we adopt alpaca-gpt4 (Peng et al., 2023) which generates "chosen" responses using GPT-4, we adopt baichuan-6B (Yang et al., 2023) serving as a weaker model for generating "reject" responses. The dataset comprises 183k Chinese pairs and 46k English pairs in total.