One notable limitation arises from the resource-intensive nature of continual pre-training LLMs, making reproducing all the straightforward continual pre-training methods outlined in this work challenging. Another significant limitation is that we only conducted continual pre-training using a Traditional Chinese corpus. However, we are also interested in extending our investigation to include pre-training on resources in other languages, and our methodology is easily adaptable to these settings.