Some studies explore the multilingual capabilities of language models, focusing on their cross-lingual performance or structural commonalities between languages K et al. (2020); Doddapaneni et al. (2021); Chai et al. (2022). More recently, to understand the relation of certain abilities with specific model architecture especially the dominant Transformer Vaswani et al. (2017) architecture, studies such as Hou et al. (2023); Stolfo et al. (2023); Friedman et al. (2023) explore the reasoning abilities of LLMs with self-attention layers. While others like Geva et al. (2021); Dai et al. (2022); Meng et al. (2022) focus on the feed-forward layers, regarding them as key-value memories for storing factual knowledge. However, these works often examine components in isolation and lack a comprehensive framework that encompasses the entire end-to-end process. More importantly, the working mechanism of multilingualism is largely unaddressed in these interpretations.