Self attention based Transformers have become the SOTA mechanism for sequence to sequence modeling in recent years [1]}. Some recent works have explored the use of such models in LM. Deep Transformer encoders outperform stacked LSTM models [2]}. A deep stacked Transformer model utilizing auxiliary loss was proposed by [3]} for character level language modeling. The multi-head self attention mechanism was replaced by a multi-linear attention mechanism with a view to improving LM performance and reducing parameter number [4]}. Bengali and Hindi language, having unique characteristics, remain open as to what strategy to use for model development in such domains.
