We propose sDPO where we use the preference data in a stepwise way instead of all at once. We show that applying sDPO results in more performant models than DPO in terms of H4 score. We also empirically exhibit that sDPO results in more aligned reference models by comparing mean \(\gamma_{\pi_{ref}}\).