After the breakthrough of the attention mechanism [1]}, a new generation of pretrained models have appeared. They have achieved tremendous performances since the introduction of the Transformers [2]}. Besides, the Bidirectional Encoder Representations from Transformers (BERT) [3]} has been unleashed to become the state-of-the-art model for language understanding and gave new inspiration to further development in the Natural Language Processing (NLP) field. Accordingly, most languages have their own BERT-based language models. Specifically, the Arabic language has multiple language models: AraBERT [4]}, GigaBERT [5]}, and multilingual cased BERT model (hereafter mBERT) [6]} which was simultaneously pretrained on 104 languages.
