Sequence order information based statistical RNN models such as LSTM and GRU have been popular for LM tasks [1]}. [2]} showed the effectiveness of LSTM for English and French LM. The regularizing effect on LSTM was investigated by [3]}. SOTA LSTM models learn sub-word information in each time stamp. [4]} proposed a morphological information oriented character N-gram based word vector representation. It was improved by [5]} and is known as FastText. [6]} proposed a technique for learning sub-word level information from data, while such idea was integrated in a character aware LSTM model by [7]}. [8]} further improved word representation by combining ordinary word level and character-aware embedding. [9]} has shown that character-aware neural LMs outperform syllable-aware ones. [10]} evaluated such models on 50 morphologically rich languages.
