We compare CoCNN with four Transformer-based models (see Figure REF ). We use popular FastText word representation with all compared transformers. Our comparison starts with Vanilla_Tr, a single Transformer encoder (similar to the Transformer designed by [1]}). In BERT, we stack 12 transformers on top of each other where each Transformer encoder has more parameters than
the Transformer of Vanilla_Tr [2]}, [3]}. BERT with its large depth and enhanced encoders almost double the performance shown by Vanilla_Tr. We do not pretrain this BERT architecture. We follow the Transformer architecture designed by [4]} and introduce auxiliary loss after the Transformer encoders situated near the bottom of the Transformer stack of BERT to form BERT_Aux. Introduction of such auxiliary losses show moderate improvement of performance. BERT_Pre is the pretrained version of BERT. We follow the word masking based pretraining scheme of [5]}. The Bengali pretraining corpus consists of Prothom Alo [6]} news articles dated from 2014-2017 and BDNews24 [7]} news articles dated from 2015-2017. The performance of BERT jumps up more than double when such pretraining is applied. CoCNN without utilizing any pretraining achieves marginally better performance than BERT_Pre. Unlike Transformer encoders, conv imposes attention with a view to extracting important patterns from the input to provide the correct output (see Subsection REF ). Furthermore, VL of CoCNN is suitable for deriving semantic meaning of each input word in highly inflected and error prone settings.
