Our work examines the forgetting occurrence during continual pre-training on an existing fine-tuned LLM. Our paper primarily focuses on continual pre-training using the Traditional Chinese corpus. We evaluate the impact of continual pre-training across various dimensions, including output format, knowledge, and reliability. We show that more than straightforward methods are required for resolving this issue. Also, we observe an increased prominence of the repetition problem in models that tend to generate Traditional Chinese outputs. Lastly, despite continual pre-training, our findings suggest that the model's knowledge remains unaffected while its reliability declines.