To solve the Softmax bottleneck, <cite>Yang et al. (2018)</cite> proposed Mixture of Softmaxes (MoS), which increases the rank of the matrix by combining multiple probability distributions computed from the encoded fixed-length vector. In this study, we propose Direct Output Connection (DOC) as a generalization of MoS. For stacked RNNs, DOC computes the probability distributions from the middle layers including input embeddings.