Multilingual language models have demonstrated a high level of zero-shot or few-shot cross-lingual transferability across a wide range of tasks Wu and Dredze (2019); Pires et al. (2019); Winata et al. (2021). This means that they can acquire the language capability from supervised data in one language and apply it to another without or with few additional training data. The mechanism behind the strong cross-lingual performance has been investigated by the researchers. It has been shown that multilingual language models have inferred universal rules applicable to any language Artetxe et al. (2020); Chi et al. (2020); Conneau et al. (2020). Contrary to the common hypothesis that multilingual multilingual language models such as mBERT Devlin et al. (2019) rely on a shared subword vocabulary and joint pretraining across multiple languages Pires et al. (2019); Cao et al. (2020); Wu and Dredze (2019), researchers have developed new understandings on the models, emphasizing the models' ability to learn universal semantic abstractions Artetxe et al. (2020); Chi et al. (2020). In terms of the factors that influence cross-lingual performance, researchers have associated transferability with parameter sharing Conneau et al. (2020); Dufter and Schutze (2020); Wu et al. (2022) and language distance Conneau et al. (2020); Eronen et al. (2023). We here further investigate the cross-lingual transferability of language models with new LLaMA-based experiments, presenting outcomes from a different aspect.