A Survey on Context Learning <s> Speed-Up with Hierarchical Structures <s> In recent years, variants of a neural network architecture for statistical language modeling have been proposed and successfully applied, e.g. in the language modeling component of speech recognizers. The main advantage of these architectures is that they learn an embedding for words (or other symbols) in a continuous space that helps to smooth the language model and provide good generalization even when the number of training examples is insufficient. However, these models are extremely slow in comparison to the more commonly used n-gram models, both for training and recognition. As an alternative to an importance sampling method proposed to speed-up training, we introduce a hierarchical decomposition of the conditional probabilities that yields a speed-up of about 200 both during training and recognition. The hierarchical decomposition is a binary hierarchical clustering constrained by the prior knowledge extracted from the WordNet semantic hierarchy. <s> BIB001 </s> A Survey on Context Learning <s> Speed-Up with Hierarchical Structures <s> The supremacy of n-gram models in statistical language modelling has recently been challenged by parametric models that use distributed representations to counteract the difficulties caused by data sparsity. We propose three new probabilistic language models that define the distribution of the next word in a sequence given several preceding words by using distributed representations of those words. We show how real-valued distributed representations for words can be learned at the same time as learning a large set of stochastic binary hidden features that are used to predict the distributed representation of the next word from previous distributed representations. Adding connections from the previous states of the binary hidden features improves performance as does adding direct connections between the real-valued distributed representations. One of our models significantly outperforms the very best n-gram models. <s> BIB002 </s> A Survey on Context Learning <s> Speed-Up with Hierarchical Structures <s> Neural probabilistic language models (NPLMs) have been shown to be competitive with and occasionally superior to the widely-used n-gram language models. The main drawback of NPLMs is their extremely long training and testing times. Morin and Bengio have proposed a hierarchical language model built around a binary tree of words, which was two orders of magnitude faster than the non-hierarchical model it was based on. However, it performed considerably worse than its non-hierarchical counterpart in spite of using a word tree created using expert knowledge. We introduce a fast hierarchical language model along with a simple feature-based algorithm for automatic construction of word trees from the data. We then show that the resulting models can outperform non-hierarchical neural models as well as the best n-gram models. <s> BIB003 </s> A Survey on Context Learning <s> Speed-Up with Hierarchical Structures <s> The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. ::: ::: An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible. <s> BIB004 </s> A Survey on Context Learning <s> Speed-Up with Hierarchical Structures <s> We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. <s> BIB005
Another speed-up strategy is to replace the unstructured jV j-sized output layer with a hierarchical structure, which can provide an exponential reduction in time complexity compared to the NPLM model. Typically, the hierarchical structure is a binary tree whose leaf nodes correspond to the words in the vocabulary . Ref. BIB001 introduces a hierarchical probabilistic neural network language model where the binary tree structure is constructed based on the lexical resource WordNet  . To translate the WordNet tree to a binary tree, the K-means algorithm is used to split word clusters. The similarity scores between words for the K-means algorithm are calculated based on the ontological graph provided by WordNet (including IS-A, synonym etc. relations). After the binary tree is built, each word w in the vocabulary corresponds to a leaf node in the binary tree and can be uniquely specified by the path bðwÞ from the root to the leaf node w, bðwÞ ¼ ðb 1 ðwÞ; b 2 ðwÞ; . . . ; b m ðwÞÞ (where m is the length of this path depending on w). Thus given the path bðwÞ, the conditional probability of the current word can be computed as follows: This product can be interpreted as a series of binary decisions indicating which of the children of the current node to visit next. If the binary tree is balanced, the height of the tree should be log 2 jV j d e . As a result, the training time of this model is linear in Oðlog ðjV jÞÞ instead of jV j and the time complexity is exponentially reduced. However, such expert knowledge as WordNet is not always available, BIB003 introduces the hierarchical log-bilinear model (HLBL) whose hierarchical structure is built via a pure learning approach. The authors use log-bilinear (LBL) BIB002 as the foundation of the HLBL model for its simplicity and good performance. The LBL model adopts the inner product between the predicted feature vector and the feature vector for each word in the vocabulary as the unnormalized similarity. Here the predicted feature vector is computed as a linear combination of the word vectors in the context window. The HLBL model discusses four rules to construct a binary tree of words without expert knowledge: RANDOM, BALANCED, ADAPTIVE and ADAP-TIVE(). When training the HLBL model based on the latter three strategies, we need to train the HLBL model based on a random binary tree first and then use the learned word embeddings to build a more sophisticated binary tree. The HLBL model not only achieves an exponential reduction in time complexity, but also its performance is comparable to the foundation LBL model. Ref. BIB004 , BIB005 propose a Huffman tree based hierarchical neural network language model called Word2Vec. As a Huffman tree assigns short paths to frequent words, this results in even faster training.