3 Language Modeling as Matrix Factorization <cite>Yang et al. (2018)</cite> indicated that the training of language models can be interpreted as a matrix 2 Actually, we apply a bias term in addition to the weight matrix but we omit it to simplify the following discussion.