A Survey on Context Learning <s> Discussion on Neural Network Methods <s> The word2vec software of Tomas Mikolov and colleagues (this https URL ) has gained a lot of traction lately, and provides state-of-the-art word embeddings. The learning models behind the software are described in two research papers. We found the description of the models in these papers to be somewhat cryptic and hard to follow. While the motivations and presentation may be obvious to the neural-networks language-modeling crowd, we had to struggle quite a bit to figure out the rationale behind the equations. ::: This note is an attempt to explain equation (4) (negative sampling) in "Distributed Representations of Words and Phrases and their Compositionality" by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean. <s> BIB001 </s> A Survey on Context Learning <s> Discussion on Neural Network Methods <s> The authors analyze three critical components in training word embeddings: model, corpus, and training parameters. They systematize existing neural-network-based word embedding methods and experimentally compare them using the same corpus. They then evaluate each word embedding in three ways: analyzing its semantic properties, using it as a feature for supervised tasks, and using it to initialize neural networks. They also provide several simple guidelines for training good word embeddings. <s> BIB002
As emerging techniques, both neural network and Restricted Boltzmann Machine based analysis have proven to be better than the conventional language models such as N-gram in terms of perplexity. Compared to RBM based analysis, neural network based analysis has garnered more attention and development, and the reason is threefold: First, neural network based analysis is more efficient in parameter inference. Most neural network based analysis can be trained via backpropagation. However backpropagation can not be directly applied to RBM based analysis because it takes exponential time to compute the gradient. This is due to exponentially many items involved in normalizing the probability distribution of each state based on its energy. Though, Contrastive Divergence provides an efficient estimation of the gradient using Gibbs sampling, the optimization of RBM based analysis is still more difficult than neural network based analysis. Second, the architecture of neural network based analysis is more flexible for extensions and modifications. By changing the neural network architecture, for example replacing the output layer with a scoring unit and using a recurrent structure, neural network based analysis can effectively incorporate prior knowledge, global context and extend to more application domains. Last, the rapid development of distributed representation makes a contribution. Word embeddings have two general usage: they can be directly used as the input to accomplish tasks such as part-of-speech tagging; they can also work as auxiliary features to improve the performance of existing models. There are several sets of popular trained word embeddings available on the Internet. Though it is hard to conclude which is the best as they were trained on different corpora and designed for different tasks, we can observe that a larger corpus of a suitable domain yields better word embeddings for specific tasks BIB002 . However, RBM based analysis has its own advantages as well. On one hand, RBM based analysis is based on global bag-of-word context, which means compared with neural network based analysis it can better utilize the global statistics. On the other hand, the training time consumption of RBM based analysis is much shorter than neural network based analysis. One big contribution made by neural network based analysis is the distributed representation. Compared with the one-hot representation, the distributed representation significantly improves the performance of context learning and is closer to how humans understand words. However, many researchers argue that the explanation of how the word embeddings are learned based on their context is hand-wavy and a more formal explanation is in need BIB001 . Therefore, possible future directions in neural network based analysis are: accelerating the training speed and trying to better understand the reason for the success of word embeddings.