To assess the impact of the tokenizers on the model downstream performance, we trained a decoder-only transformer model of size 2.6 B for each tokenizer. We trained our models for 52.6 B tokens following the scaling laws proposed by Hoffmann et al. (2022), based on the causal language modeling training objective. The hyper-parameters are described in Table 10 in the Appendix C.