Arabic is the 5th most widely spoken language in the world despite being a low-resource language. Perhaps, the most commonly used Arabic language model is BERT [1]} which is trained on a relatively small corpus, Arabic Wikipedia, compare to the LM trained for other languages. Thus, it is insufficient to represent a rich and complex language like Arabic.
