As described in Section 3, <cite>Yang et al. (2018)</cite> interpreted training language modeling as matrix factorization and improved performance by computing multiple probability distributions. In this study, we generalized their approach to use the middle layers of RNNs.