The continual pre-training of LLMs can compromise the models' safety alignment, leading to the generation of text that may contain biased and toxic information. Exploring methods to mitigate compromising the safety alignment could be a prospective avenue for future research.