Evaluation results for applying sDPO to the SFT base model, along with results for other top-performing models are shown in Tab. 2. Comparing the pretrained-only 'SOLAR 10.7B' to the instruction-tuned 'SOLAR 10.7B + SFT', we can see an increase of \(+5.24\) in terms of H4. Applying sDPO on SOLAR 10.7B + SFT further increases the H4 score upto \(74.31\), an improvement of \(+4.80\). Notably, 'SOLAR 10.7B + SFT + sDPO' outperforms other larger models such as Mikral 8x7B-Instruct-v0.1, despite the smaller number of parameters. This highlights that effective alignment tuning could be the key to unlocking next level performance for smaller LLMs. Further, applying sDPO results in substantially higher score of \(72.45\) for TruthfulQA, which shows the effectiveness of the alignment tuning process.