Some works have studied continual learning for language models. Qin et al. (2022) focused on efficient lifelong pre-training on pre-trained language models for emerging data. Ke et al. (2022) proposed a continual domain-adaptive pre-training method on a masked language model. Song et al. (2023) introduced continual parameter-efficient tuning for the ongoing adaptation of LLMs to continual tasks. Xie et al. (2023) investigate an alternative approach to continual pre-training for developing domain-specific LLMs. Qi et al. (2023) suggests that fine-tuning compromises the safety alignment of LLMs. Zhai et al. (2023) evaluates the forgetting in fine-tuned multimodal LLMs.