Recent advancements in the field of large language models (LLMs), such as GPT-4 [2], Gemini [14], and Claude [1], have demonstrated remarkable capabilities in processing and understanding multiple languages. On the other hand, though notable models in open source community, such as LLaMA [15, 16], MPT [15], Falcon [11], Mistral [18], Mistral [18], SOLAR [19], and Phi-1.5 [10] have set benchmarks in English tasks, these developments have predominantly favored English, leading to a performance gap in non-English languages.