Data size. mBERT [1]} and XLM-R [2]} use a small Arabic text collection from Wikipedia (153M tokens) and CommonCrawl (\(2.9\) B tokens), respectively. AraBERT [3]} is BERTBase pretrained model for Arabic. As explained, antoun2020arabert use two newswire datasets to train AraBERT (zeroual2019osian and elkhair-2016) comprising more than 70M sentences, corresponding to 24GB of text (\(2.5\) B tokens). Our   and  are trained on 61GB of Arabic text (\(6.2\) B tokens) and 128GB of Arabic tweets (\(15.6\) B tokens), respectively.
