In this paper, we describe the process of pretraining a Pytorch implementation of NVIDIA BERT language modelhttps://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling/BERT, called TunBERT (Tunisian BERT), trained on only 67.2 MB web-scraped dataset. We systematically compare our pre-trained model on three NLP downstream tasks;
that are different in nature: (i) Sentiment Analysis (SA),
(ii) Tunisian dialect identification (TDI), and (iii) Reading Comprehension Question-Answering (RCQA); against mBERT [1]}, AraBERT [2]}, GigaBERT [3]} and the state of the art performances when available.
Our contributions can be summarized as follows:
