In general, neural language models require word embeddings as an input<cite> (Zaremba, Sutskever, and Vinyals 2014)</cite>. However, as described by<cite> (Verwimp et al. 2017)</cite> , this approach cannot make use of the internal structure of words although the internal structure is often an effective clue for considering the meaning of a word. For example, we can comprehend that the word 'causal' is related to 'cause' immediately because both words include the same character sequence 'caus'. Thus, if we incorporate a method that handles the internal structure such as character information, we can improve the quality of neural language models and probably make them robust to infrequent words.