The target model in each step of sDPO is also initialized with \(M_{t-1}\), the aligned model from the last step. This ensures that the final model in sDPO has undergone training with the same amount of data as the final model in DPO. On the other hand, one concern of such design choice is that it may become increasingly difficult to stabilize the training of the target model as the steps progress, since it has already undergone training with a decreasing learning rate schedule in the preceding steps. Thus, another option is to use the initial SFT base model \(S\) as the target model for all steps of sDPO.