If we consider traditional cluster encoded word representation, e.g., Brown clusters (Brown et al., 1992) , it only uses a small number of bits to track the path on a hierarchical tree of word clusters to represent each word. In fact, word embedding generalized the idea of discrete clustering representation to continuous vector representation in language models, with the goal of improving the continuous word analogy prediction and generalization ability (Bengio et al., 2003; <cite>Mikolov et al., 2013a</cite>; Mikolov et al., 2013b) .