A Survey on Context Learning <s> D <s> The description of a novel type of m-gram language model is given. The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data. This solution compares favorably to other proposed methods. While the method has been developed for and successfully implemented in the IBM Real Time Speech Recognizers, its generality makes it applicable in other areas where the problem of estimating probabilities from sparse data arises. <s> BIB001 </s> A Survey on Context Learning <s> D <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB002 </s> A Survey on Context Learning <s> D <s> We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model. <s> BIB003 </s> A Survey on Context Learning <s> D <s> Autonomous information agents alleviate the information overload problem on the Internet. The AGM belief revision framework provides a rigorous foundation to develop adaptive information agents. The expressive power of the belief revision logic allow a user's information preferences and contextual knowledge of a retrieval situation to be captured and reasoned about within a single logical framework. Contextual knowledge for information retrieval can be acquired via context sensitive text mining. We illustrate a novel approach of integrating the proposed text mining method into the belief revision based adaptive information agents to improve the agents' learning autonomy and prediction power. <s> BIB004 </s> A Survey on Context Learning <s> D <s> We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box. <s> BIB005 </s> A Survey on Context Learning <s> D <s> In this paper, we formally define the problem of topic modeling with network structure (TMN). We propose a novel solution to this problem, which regularizes a statistical topic model with a harmonic regularizer based on a graph structure in the data. The proposed method bridges topic modeling and social network analysis, which leverages the power of both statistical topic models and discrete regularization. The output of this model well summarizes topics in text, maps a topic on the network, and discovers topical communities. With concrete selection of a topic model and a graph-based regularizer, our model can be applied to text mining problems such as author-topic analysis, community discovery, and spatial text mining. Empirical experiments on two different genres of data show that our approach is effective, which improves text-oriented methods as well as network-oriented methods. The proposed model is general; it can be applied to any text collections with a mixture of topics and an associated network structure. <s> BIB006 </s> A Survey on Context Learning <s> D <s> ABSTRACT Community structure is a common attribute of many social networks, which would give us a better understanding of the networks. However, as the social networks grow lager and lager nowadays, the Pareto Principle becomes more and more notable which makes traditional community discovery algorithms no longer suitable for them. This principle explains the unbalanced existence of two different types of network actors. Specifically, the core actors usually occupy only a small proportion of the population, but have a large influence. In this paper, we propose a novel algorithm LCDN (Latent Community Discovery with Network Structure) for dividing the core actors. This is a hierarchical probabilistic model based on statistical topic model and regularized by network structure in data. We had experiments on three large networks which show that this new model performs much better than the traditional statistical models and network partitioning algorithms. <s> BIB007 </s> A Survey on Context Learning <s> D <s> Recommender systems have been researched extensively by the Technology Enhanced Learning (TEL) community during the last decade. By identifying suitable resources from a potentially overwhelming variety of choices, such systems offer a promising approach to facilitate both learning and teaching tasks. As learning is taking place in extremely diverse and rich environments, the incorporation of contextual information about the user in the recommendation process has attracted major interest. Such contextualization is researched as a paradigm for building intelligent systems that can better predict and anticipate the needs of users, and act more efficiently in response to their behavior. In this paper, we try to assess the degree to which current work in TEL recommender systems has achieved this, as well as outline areas in which further work is needed. First, we present a context framework that identifies relevant context dimensions for TEL applications. Then, we present an analysis of existing TEL recommender systems along these dimensions. Finally, based on our survey results, we outline topics on which further research is needed. <s> BIB008 </s> A Survey on Context Learning <s> D <s> A Language Model (LM) is a helpful component of a variety of Natural Language Processing (NLP) systems today. For speech recognition, machine translation, information retrieval, word sense disambiguation etc., the contribution of an LM is to provide features and indications on the probability of word sequences, their grammaticality and semantical meaningfulness. What makes language modeling a challenge for Machine Learning algorithms is the sheer amount of possible word sequences: the curse of dimensionality is especially encountered when modeling natural language. The survey will summarize and group literature that has addressed this problem and we will examine promising recent research on Neural Network techniques applied to language modeling in order to overcome the mentioned curse and to achieve better generalizations over word sequences. <s> BIB009 </s> A Survey on Context Learning <s> D <s> The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. ::: ::: An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible. <s> BIB010 </s> A Survey on Context Learning <s> D <s> The proliferation of digital cameras and the growing practice of online photo sharing using social media sites such as Flickr have resulted in huge volumes of geotagged photos available on the Web. Based on users' traveling preferences elicited from their travel experiences exposed on social media sites by sharing geotagged photos, we propose a new method for recommending tourist locations that are relevant to users i.e., personalization in the given context i.e., context awareness. We obtain user-specific travel preferences from his/her travel history in one city and use these to recommend tourist locations in another city. Our technique is illustrated on a sample of publicly available Flickr dataset containing photos taken in various cities of China. Results show that our context-aware personalized method is able to predict tourists' preferences in a new or unknown city more precisely and generate better recommendations compared to other state-of-the-art landmark recommendation methods. <s> BIB011 </s> A Survey on Context Learning <s> D <s> Title matching refers roughly to the following problem. We are given two strings of text obtained from different data sources. The texts refer to some underlying physical entities and the problem is to report whether the two strings refer to the same physical entity or not. There are manifestations of this problem in a variety of domains, such as product or bibliography matching, and location or person disambiguation. ::: ::: We propose a new approach to solving this problem, consisting of two main components. The first component uses Web searches to "enrich" the given pair of titles: making titles that refer to the same physical entity more similar, and those which do not, much less similar. A notion of similarity is then measured using the second component, where the tokens from the two titles are modelled as vertices of a "social" network graph. A "strength of ties" style of clustering algorithm is then applied on this to see whether they form one cohesive "community" (matching titles), or separately clustered communities (mismatching titles). Experimental results confirm the effectiveness of our approach over existing title matching methods across several input domains. <s> BIB012 </s> A Survey on Context Learning <s> D <s> Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperforms bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks. <s> BIB013 </s> A Survey on Context Learning <s> D <s> Rapid advances in bionanotechnology have recently generated growing interest in identifying peptides that bind to inorganic materials and classifying them based on their inorganic material affinities. However, there are some distinct characteristics of inorganic materials binding sequence data that limit the performance of many widely-used classification methods when applied to this problem. In this paper, we propose a novel framework to predict the affinity classes of peptide sequences with respect to an associated inorganic material. We first generate a large set of simulated peptide sequences based on an amino acid transition matrix tailored for the specific inorganic material. Then the probability of test sequences belonging to a specific affinity class is calculated by minimizing an objective function. In addition, the objective function is minimized through iterative propagation of probability estimates among sequences and sequence clusters. Results of computational experiments on two real inorganic material binding sequence data sets show that the proposed framework is highly effective for identifying the affinity classes of inorganic material binding sequences. Moreover, the experiments on the structural classification of proteins ( SCOP ) data set shows that the proposed framework is general and can be applied to traditional protein sequences. <s> BIB014 </s> A Survey on Context Learning <s> D <s> There is a growing interest in identifying inorganic material affinity classes for peptide sequences due to the development of bionanotechnology and its wide applications. In particular, a selective model capable of learning cross-material affinity patterns can help us design peptide sequences with desired binding selectivity for one inorganic material over another. However, as a newly emerging topic, there are several distinct challenges of it that limit the performance of many existing peptide sequence classification algorithms. In this paper, we propose a novel framework to identify affinity classes for peptide sequences across inorganic materials. After enlarging our dataset by simulating peptide sequences, we use a context learning based method to obtain the vector representation of each amino acid and each peptide sequence. By analyzing the structure and affinity class of each peptide sequence, we are able to capture the semantics of amino acids and peptide sequences in a vector space. At the last step we train our classifier based on these vector features and the heuristic rules. The construction of our models gives us the potential to overcome the challenges of this task and the empirical results show the effectiveness of our models. <s> BIB015 </s> A Survey on Context Learning <s> D <s> Epileptic seizure is a serious health problem in the world and there is a huge population suffering from it every year. If an algorithm could automatically detect seizures and deliver the patient therapy or notify the hospital, that would be of great assistance. Analyzing the scalp EEG is the most common way to detect the onset of a seizure. In this paper, we proposed the context-learning based EEG analysis for seizure detection (Context-EEG) algorithm. The proposed method aims at extracting both the hidden inherent features within EEG fragments and the temporal features from EEG contexts. First, we segment the EEG signals into EEG fragments of fixed length. Second, we learn the hidden inherent features from each fragment and reduce the dimensionality of the original data. Third, we translate each EEG fragment to an EEG word so that the EEG context can provide us with temporal information. And finally, we concatenate the hidden feature and the temporal feature together to train a binary classifier. The experiment result shows the proposed model is highly effective in detecting seizure. <s> BIB016
ATA mining is a research field wherein the objective is to identify and extract non-obvious patterns and information from the data. Having roots in statistical mathematics, many proposed methods perform computation and gather parameters using the data as independent individuals. This leads to poor inference ability and decision making in certain application domains, e.g., Sequence Analysis (SA), Named Entity Recognition (NER) and Disambiguation (NEN), Language Models (LM), etc. While the pioneering works in these fields focused only at one instance at a time and learning from its attributes, researchers have realized that looking at an instance in isolation or capturing global statistics need not provide the complete information BIB006 . However, effective capture and usage of contextual information require a significant computation capability-something that was lacking in early days of data mining. With the advent of more powerful hardware in last decade and a half, researchers are increasingly looking at effective means to capture the influence of neighbors of an instance and its semantic implications BIB002 . By providing either temporal or spatial dimension, this neighborhood information enhances the learning process. For instance in text domain, context defined as a bag of preceding and succeeding words surrounding the current word in a text document can be used to in pruning the search space of candidate named entities in NER BIB012 . The interpretation of context varies based on the problem domain and in some cases even with respect to the task within a domain-for instance in case of image and video processing it can be the neighbor pixels or the preceding or subsequent frame in a video, in case of peptide sequence analyses it may refer to the neighboring amino acids, whereas in case of continuous domain such as speech record and clinography it may refer to the peripheral signal fragments. By analyzing and learning the information and pattern within each neighborhood, context learning extracts contextual knowledge that assist us in a wide variety of application domains. For example, analyzing the context information in a text document provides us a way to establish its corresponding language model BIB002 , BIB001 ,  ,  , BIB003 . The learnt language model assist in many tasks such as machine translation BIB010 , text summarization BIB013 , entity resolution BIB012 , etc. Adopting such an approach ensures that both the meaning as well as the sequence of the words are conveyed in the correct order to make the output "linguistically" reasonable. As mentioned earlier, the context information is not restricted to only spatial dimension, but can also include in temporal aspects. In such cases, the context information is composed of data from nearby time slot, such as in cases of video and audio sequences processing domains like speech recognition  and video de-noising BIB005 . In bio-informatics as well, many problems can be better analyzed by incorporating context related information-such as peptide sequence analysis BIB014 , BIB015 , EEG and ECG signal processing BIB016 . An effective context discovery ensures that content within that context is usually semantically related. This allows us to not only learn the representation with semantic embeddings but also learn the componential structure of the instance and the context. As an example consider BIB010 , which is a popular distributed representation learning approach. A distributed representation is the process of representing a more generic model through the domain's componential structure. In other words, every token is mapped to a vector in a vector space according to their semantics in such a way that, semantically related words identified using the context are closer to each other. Because of the nature of the vector space, it can represent the word "King" as a combination of "Man" and "Ruler" and similarly "Queen" as "Woman" and "Ruler". Thus if we want to find the similarity of vectors between "King" and "Queen" an inference would yield a path like vector("King")-vector("Man") + vector("Woman"). Encoding semantic information in our model using the neighbors help better formulation of the solutions to a variety of problems, especially when the instance under consideration has sparse features, e.g., short length queries BIB004 . The practicality of leveraging context information is pervasive across domains-for instance context information can be found in cliques in social networks and physical locations BIB011 , BIB007 . In this survey, we attempt to present and discuss various context learning techniques. A majority of the context learning techniques can be categorized into explicit analysis (EA) (Section 2), implicit analysis (IA) (Section 3), neural network based (Section 4) and composite models (Section 5). This categorization also allows us to present techniques in a progressively increasing order of complexity and computational requirements. We analyze some of the influential methods in each of these categories, provide examples to better illustrate the nuances of the technique and then compare and contrast the methods in each of the categories. It should be noted that certain methods discussed in this paper within a category are aimed at particular application. It is also worthwhile to point that the computational complexity of the method increases from Sections 2 to 4). It is not our intention in this survey to enumerate a comprehensive list of applications for each method, but rather illustrate the underlying principle of a method through a widely understood application; however, we do attempt to point out some novel application of an approach in a non-standard domain. This is how our survey paper differs from those of BIB008 , BIB009 . These papers focus on a particular application or domain, like recommendation system, and enumerate different context learning strategies applicable in that domain. In contrast, our focus in this survey is to enumerate generic context learning strategies with applications being just a way to better convey the underlying principles.