When we aim to enhance the capabilities of a LLM in a specific language, vocabulary extension is an intuitively reasonable approach. In this section, we evaluate the impact of vocabulary extension through the LLM-Eval benchmark, and the experimental results are presented in table 1. Initially, we collected one million Chinese sentences from the internet (approximately 0.5 billion tokens) and further pretrain the original LLaMA without vocabulary extension. Surprisingly, we find that this model significantly outperform the vocabulary-extended Chinese LLaMA, across settings of 1K, 5K, and 950K instruction tuning. This discovery is thought-privoking, given that the Chinese LLaMA underwent further Chinese pretraining on 30 billion tokens, a much larger volume than our 0.5 billion tokens. Moreover, within the 950K setting, we include results from extending the vocabulary on original LLaMA and training it with the same 0.5 billion tokens, to mitigate the influence of training data discrepancy. The outcomes remain consistent. This indicates that vocabulary extension is not a favorable choice within training scales of tens of billions of tokens. While we don't negate the efficacy of vocabulary extension in settings involving larger-scale pretraining (such as trillions of tokens), as reported in other literatures [23], this already leans more towards retraining than mere language transfer.