In the recent years, contextual embedding shows the significant progress in the NLP research field. Consequently, according to the reported results in several researches (i.e. zhu-etal-2019-um) it's it's outperform the deep learning approaches. The transformer considered as an encoder-decoder architecture applied on attention mechanisms tasks. More particularly, Google has been released BERT [1]} which stands for Bidirectional Encoder Representations from Transformers. BERT is a deeply bidirectional architecture which means BERT during the training step can learn the important features from both sides of a word’s context. They applied masked language modeling this means the model depends on the position of the words to infer the information among them. The pre-trained BERT model can be fine-tuned.
