Compared to the increasing studies of contextualized word representations in Indo-European languages, similar research for Arabic language is still very limited. AraBERT [1]}, a BERT-based model, was released using a pre-training dataset of 70 million sentences, corresponding to 24 GB of text covering news from different Arab
media. AraBERT was pre-trained on a TPUv2-8
pod for 1,250K steps. It achieved state-of-the-art performances on three Arabic tasks: Sentiment Analysis, Named
Entity Recognition, and Question Answering. Nevertheless, the pre-trained dataset is mostly a MSA based. Authors concluded that there is a need for pretrained models that can tackle a variety of Arabic dialects. Lately, GigaBERT [2]} customized bilingual language model for English and Arabic has outperformed AraBERT in several downstream tasks.
