**Training Settings.** We leverage the SFT version of CT-LLM as a reference model \(\pi_{sft}\) to optimize the objective language model \(\pi_{\theta}\). \(\pi_{\theta}\) is initialized by the model parameters of the \(\pi_{sft}\). We set the hyperparameters as follows: 1. The \(\pi_{\theta}\) is trained on 8 H800, 2. learning rate \(=1e-6\), 3. batch size \(=4\), 4. epoch numbers \(=2\), 5. weight decay \(=0.1\), 6. warmup ratio \(=0.03\), 7. \(\beta=0.5\) to control the deviation from \(\pi_{sft}\).