**Heuristic Rules** We designed heuristic rules to conduct data filtering, which removes data of low quality. These rules represent an integrated framework of filtering strategies, inspired by methodologies derived from several datasets and models, notably RefinedWeb (Penedo et al., 2023) and CC-Net (Wenzek et al., 2020), along with some rules that are applied while training other language models, such as Gopher (Rae et al., 2022) and T5 (Raffel et al., 2020). We also developed a set of rules tailored to address characteristics inherent to our dataset.