In the previous section, we observed that extending the vocabulary had a negative impact on language transferability. A plausible hypothesis is the existence of cross-lingual semantic alignment within LLMs, which vocabulary expansion might disrupt. To validate this alignment hypothesis, we fine-tune LLaMA with a dataset of 1k instructions and examine the model's output. Excitingly, we observed a certain proportion of code-switching samples. As depicted in figure 3, these samples' model responses consist of tokens from multiple languages and are semantically coherent. We have observed that code-switching occurs not only in the transfer process when Chinese is the target language, but also when other 13 low-resource languages are target languages. As shown in figure 4, the proportion of samples with code-switching is approximately between \(2\%\) to \(5\%\). This indicates that LLaMA might have learned cross-lingual alignment relationships between concepts during the pretraining process.