**Background.** LLMs have shown remarkable capabilities in a wide range of NLP tasks [55, 56, 67, 90]. However, these models may sometimes exhibit unintended behaviors, _e.g._, fabricating false information, pursuing inaccurate objectives, and producing harmful, misleading, and biased expressions [66, 366]. For LLMs, the language modeling objective pre-trains the model parameters by word prediction while lacking the consideration of human values or preferences. To avert these unexpected behaviors, human alignment has been proposed to make LLMs act in line with human expectations [66, 367]. However, unlike the original pre-training and adaptation tuning (_e.g._, instruction tuning), such an alignment requires considering very different criteria (_e.g._, helpfulness, honesty, and harmlessness). It has been shown that alignment might harm the general abilities of LLMs to some extent, which is called _alignment tax_ in related literature [368].