Training details.We use a supervised fine-tuned SOLAR 10.7B (Kim et al., 2023) as our SFT base model \(S\) as it delivers excellent performance with its uncommon 10.7B size. Further, the scarcity of 10.7B sized models leads to the absence of open source models that can be adopted as reference models, making the usage of sDPO more necessary. We use OpenOrca (Mukherjee et al., 2023) (\(\sim 12K\) samples) and Ultrafeedback Cleaned (\(\sim 60K\) samples) (Cui et al., 2023; Ivison et al., 2023) as our preference datasets. The training hyper-parameters closely follow that of Tunstall et al. (2023). We use two steps in sDPO, where we use OpenOrca as dataset \(D_{1}\) in the first step and Ultrafeedback Cleaned as dataset \(D_{2}\) in the second step.