Note that the above may not be feasible due to the absence of such aligned models, or the fact that it renounces control over the reference model, leading to safety concerns. Instead, we propose'step-wise DPO', named sDPO, where we use the preference datasets (or subsets of a preference dataset) in _a step-by-step manner_ when undergoing DPO training. The aligned model in the previous step is used as the reference model for the current step, which results in utilizing a more aligned reference model (_i.e._, a better lower bound).