Target model.For \(t>1\), the target model which is trained using the preference loss of DPO in each step of sDPO is also initialized as \(M_{t-1}\) instead of \(S\). This ensures that the final model trained with sDPO has been directly trained with the same amount data as a model trained with DPO.