Comparing SOLAR 10.7B + SFT + DPO and SOLAR 10.7B + SFT + sDPO, we can see that using sDPO over DPO results in a higher H4 score overall, with noticeable improvements in ARC and TruthfulQA scores. Therefore, we believe sDPO could function as a drop-in replacement for DPO training with better performance. Looking at SOLAR 10.7B + SFT + sDPO and SOLAR 10.7B + SFT + sDPO Strat., we see that the specific way of splitting the available DPO data into multiple \(D_{t}\) can also impact performance. We find that the natural split of using different preference datasets as \(D_{t}\) works best in our experiments. We believe further exploration of how to define \(D_{t}\) is an interesting direction for future research.