To solve the Softmax bottleneck, <cite>Yang et al. (2018)</cite> proposed Mixture of Softmaxes (MoS), which increases the rank of the matrix by combining multiple probability distributions computed from the encoded fixed-length vector.