After SFT, we employ DPO (Rafailov et al., 2024) for human preference alignment of the model. During this stage, UltraFeedback (Cui et al., 2023) is utilized as the primary alignment dataset, and an proprietary preference dataset is constructed to enhance the model's code and mathematical capabilities. We conduct one epoch of DPO training with a learning rate of \(1\times 10^{-5}\) and utilize a Cosine LRS since we have a pre-defined training steps.