The model developed for Arabic, AraBERT [1]}, yields promising performance. However, it suffers from a number of issues some of which are serious. First, in spite of the prominently positive impact of data size on model performance, AraBERT does not make full use of available and/or easily accessible data. Second, it is pre-trained with only Modern Standard Arabic (MSA) data. This largely limits AraBERT's applicability to tasks involving dialects. Arabic dialects vary from MSA at various linguistic levels, thus rendering a model that only serves MSA significantly sub-optimal. The critical need for an Arabic LM that serves dialects also arise from the explosive use of these dialects on social media. Third, AraBERT was evaluated on only three tasks: named entity recognition, question answering, and named entity recognition. Hence it is not clear how much it would perform on more diverse datasets and a wider range of tasks. Fourth, AraBERT has not been compared to the current SOTA multilingual model XLM-R [2]}, making it hard to decide which model to choose when working on Arabic.
