A Survey on Context Learning <s> Discussion on Composite Models <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB001 </s> A Survey on Context Learning <s> Discussion on Composite Models <s> This paper introduces a new neural network language model (NNLM) based on word clustering to structure the output vocabulary: Structured Output Layer NNLM. This model is able to handle vocabularies of arbitrary size, hence dispensing with the design of short-lists that are commonly used in NNLMs. Several softmax layers replace the standard output layer in this model. The output structure depends on the word clustering which uses the continuous word representation induced by a NNLM. The GALE Mandarin data was used to carry out the speech-to-text experiments and evaluate the NNLMs. On this data the well tuned baseline system has a character error rate under 10%. Our model achieves consistent improvements over the combination of an n-gram model and classical short-list NNLMs both in terms of perplexity and recognition accuracy. <s> BIB002 </s> A Survey on Context Learning <s> Discussion on Composite Models <s> Title matching refers roughly to the following problem. We are given two strings of text obtained from different data sources. The texts refer to some underlying physical entities and the problem is to report whether the two strings refer to the same physical entity or not. There are manifestations of this problem in a variety of domains, such as product or bibliography matching, and location or person disambiguation. ::: ::: We propose a new approach to solving this problem, consisting of two main components. The first component uses Web searches to "enrich" the given pair of titles: making titles that refer to the same physical entity more similar, and those which do not, much less similar. A notion of similarity is then measured using the second component, where the tokens from the two titles are modelled as vertices of a "social" network graph. A "strength of ties" style of clustering algorithm is then applied on this to see whether they form one cohesive "community" (matching titles), or separately clustered communities (mismatching titles). Experimental results confirm the effectiveness of our approach over existing title matching methods across several input domains. <s> BIB003
As each individual category of context learning has its pros and cons, we would like to combine them in certain scenarios: a task is able to be divided into several sub-tasks that can be very effectively solved by the individual techniques; we need a compromise model between quality and efficiency which gains satisfying performance with acceptable time consumption; both local context and global context can work complementarily so that the composite model performs better than one of them working alone; useful external information can be incorporated as prior knowledge to enhance the original model. For instances, BIB003 divides the named entity disambiguation problem into explicit title enrichment and implicit community detection problems, and conquer them separately; BIB001 , BIB002 combine neural network language models with explicit n-gram models to achieve a trade-off between computational cost and model performance. Composite models can be used to solve many practical problems as they can make use of both methods. However, most of the existing composite models utilize each individual method separately-either each individual method works in different stage or the outcome of one method is used as prior knowledge. In the future, composite models with deep cooperation among its individual methods would be very helpful. 