**Finetuning of LLMs** Recent years have seen rapid progress in developing large-scale language models (LLMs) Brown et al. (2020); OpenAI (2023); Workshop et al. (2022); Touvron et al. (2023); Anil et al. (2023). To adapt these foundation models to downstream tasks, fine-tuning Mishra et al. (2021); Sanh et al. (2021); Chung et al. (2022); Muenighoff et al. (2023); Zhou et al. (2023); Lin et al. (2023); Ji et al. (2024) has become a prevalent approach. Traditional supervised fine-tuning may be limited by the cost and compute required for adapating LLMs. Addressing these challenges, research in the realm of parameter-efficient fine-tuning Houlsby et al. (2019), such as Prompt Tuning Lester et al. (2021), Prefix-Tuning Li and Liang (2021), P-Tuning Liu et al. (2022) and Low-Rank based fine-tuning Hu et al. (2021), has gained traction. These methods enable LLMs to acquire domain-specific knowledge and adapt to specialized tasks such as question answering, summarization, and dialogue generation. Another branch of finetuning is through RLHF Ouyang et al. (2022); Rafailov et al. (2023); Liu et al. (2023); Zhang et al. (2023), which adopts RL to align LLM's preference with human.