feedback (RLHF) based on the base model. The weights of these models are denoted as \(\theta_{PLM}\) and \(\theta_{chat}\), respectively, where \(\theta_{PLM},\theta_{chat}\in\mathbb{R}^{d}\), and \(d\) is the number of parameters.