Reference model.The reference model is used to calculate the log probabilities of the preference dataset. For each step, only a subset of the total data is used and the reference model is initialized as \(M_{t-1}\), _i.e,_ the aligned model from the previous step. The initial reference model is set as \(S\), the SFT base model. This results in using a more aligned reference model than conventional DPO.