relies on an important assumption of \(\pi_{\text{ref}}=\pi_{w}\). Contrary to common practices where \(\pi_{\text{ref}}\) is set as the initial SFT checkpoint, our approach considers it as the ideal policy we aim to reach. Although the ideal policy \(\pi_{w}\) is unknown and unattainable during model training, it is not engaged in the loss after our approximation.