The Wiki2023 dataset provides a relatively clean testbed for studying continual knowledge acquisition. However, its scope is limited to Wikipedia, which restricts the trained models' adaptability to other sources like web pages from Common Crawl or scientific documents from arXiv. We focus on eliciting factual knowledge with instruction-tuning on QA data in this paper. The effectiveness of pre-instruction-tuning with different types of data for enhancing other skills like reasoning or comprehension is something that needs to be explored in future studies.