We compare CoCNN with four LSTM-based models (see Figure REF ). Two LSTM layers are stacked on top of each other in all four of these models. We do not compare with LSTM models that use Word2vec [1]} representation as this representation requires fixed size vocabulary. In spelling error prone setting, vocabulary size is theoretically infinite. We start with LSTM_FT, an architecture using sub-word based FastText representation [2]}, [3]}. Character aware learnable layers per LSTM time stamp form the new generation of SOTA LSTMs [4]}, [5]}, [6]}, [7]}. LSTM_CA acts as their representative by introducing variable size parallel conv filter output concatenation as word representation. The improvement over LSTM_FT in terms of PPL score is almost double. Instead of unidirectional many to one LSTM, we introduce bidirectional LSTM in LSTM_CA to form BiLSTM_CA which shows slight performance improvement. We introduce Bahdanu attention [8]} on BiLSTM_CA to form BiLSTM_CA_Attn architecture. Such measure shows further performance boost. CoCNN shows almost four times improvement in PPL score compared to BiLSTM_CA_Attn. If we compare Figure REF  and REF , we can see that CNNs perform relatively better than LSTMs in general for Bengali LM. LSTMs have a tendency of learning sequence order information which imposes positional dependency. Such characteristic is unsuitable for Bengali and Hindi with flexible word order.
