For datasets, we mainly experiment with the Pile and RedPajama. For the Pile, we find duplicates in the raw data, so deduplication is performed before training with it. The Pile contains 5 coarse-grained domains, which are further decomposed into 22 fine-grained domains. Our experiment in Sec. 3.1 is on Github and Pile-CC domains while the experiment in Sec. 3.2 is on Github, Pile-CC, and the Books. All these are fine-grained domains. For our experiments with 5 domains in Sec. 3.3 we adopt the five coarse-grained domains, i.e., academic, internet, prose, dialogues, and misc, where misc include Github and the DeepMind Mathematics Dataset which are symbolic content. We use the coarse-grained domains because it is hard to find five fine-grained domains with sufficient tokens. For the RedPajama, we download the version produced and shared by Chen et al. (2024a).