**Novelty** refers to the information that was unseen by the LLM before. We gauge the level of novelty in a document based on the **perplexity** recorded by LLM. Documents with higher perplexity are less represented in the original training corpus, thus being more likely to contain novel knowledge for the model. Such samples are also viewed as more difficult to learn [3]. Hence, these samples can be valuable in continual pre-training to help models acquire novel information.