However, as shown in Fig. 3, initializing the target model as \(S\) results in a much bigger initial loss than that of \(M_{t-1}\), which could lead to an unstable training. The main reason is that DPO training is usually done where the reference and target models are the same. In contrast, initializing the target model as \(S\) creates a differential in the reference and target models, which may be amplified as the steps progress. Thus, for stable training, initializing the target model as \(M_{t-1}\) was chosen for sDPO.