The recent paradigm shift is triggered by the introduction of BERT in 2018 by Google [1]}. BERT outperformed the previous state-of-the-art LSTM based RNN approaches with a significant improvement of upto 7.7% on typical NLP tasks. Indeed the major credit goes to a powerful state-of-the-art neural architecture known as Transformer with an attention mechanism [2]}. BERT has another variant which is trained on 104 different languages, including Arabic, known as mBERT. Besides, separate BERT models are only available for the English and Chinese languages. The Arabic model has trained on a small corpus and validated only on the sentence contradiction task. Though, Arabic mBERT was able to achieve state-of-the-art results in 6 text classification tasks with a few-short task-specific fine-tuning.
