Although it is expedient to leverage an existing tokenizer (_e.g.,_ OPT [90] and GPT-3 [55] utilize the tokenizer of GPT-2 [26]), using a tokenizer specially designed for the pre-training corpus can be highly beneficial [78], especially for the corpus that consists of diverse domains, languages, and formats. Therefore, recent LLMs often train the customized tokenizers specially for the pre-training corpus with the SentencePiece library [226], which includes Byte-level BPE and Unigram tokenization. A note is that normalization techniques in BPE, such as NFKC [227], may degrade the tokenization performance [34, 64, 78]. When extending existing LLMs (_i.e.,_ continual pre-training or instruction tuning), we should be also aware of the potential side effect with customized tokenizers. For example, LLaMA trains the BPE tokenizer based on a pre-training corpus mainly consisting of English texts, and the derived vocabulary might be less capable in processing non-English data, _e.g.,_ taking longer inference latency to generate Chinese texts.