We observe that most of these issues arise during the pre-training phase, with catastrophic forgetting being the notable exception. Hence, many LLMs will suffer from factual errors of this kind regardless of any post-training process.