A few Arabic LMs have been developed. The most notable among these is AraBERT [1]}. AraBERT is trained with the same BERT architecture [2]} and uses the same BERTBase configuration. The second version of AraBERT, AraBERTv1, employs the Farasa tool [3]} to segment input text before training. AraBERT is trained on 23GB of Arabic text, comprising more than 70M sentences and 3B words. To train AraBERT, antoun2020arabert use Arabic Wikipedia, the Open Source International dataset (OSIAN) [4]} (\(3.5\) M news articles from 24 Arab countries), and \(1.5\) B words Corpus from [5]} (5M articles extracted from 10 news sources). antoun2020arabert evaluate AraBERT on 3 Arabic downstream tasks. These are (1) sentiment analysis from 6 different datasets: HARD [6]}, ASTD [7]}, ArsenTD-Lev [8]}, LABR [9]}, and ArSaS [10]}. (2) NER, with the ANERcorp [11]}, and (3) Arabic QA, on Arabic-SQuAD [12]} and ARCD [12]} datasets. We now introduce our models.
