A Survey on Context Learning <s> Neural Network Based Bi-gram Model <s> We are concerned with feed-forward non-linear networks (multi-layer perceptrons, or MLPs) with multiple outputs. We wish to treat the outputs of the network as probabilities of alternatives (e.g. pattern classes), conditioned on the inputs. We look for appropriate output non-linearities and for appropriate criteria for adaptation of the parameters of the network (e.g. weights). We explain two modifications: probability scoring, which is an alternative to squared error minimisation, and a normalised exponential (softmax) multi-input generalisation of the logistic non-linearity. The two modifications together result in quite simple arithmetic, and hardware implementation is not difficult either. The use of radial units (squared distance instead of dot product) immediately before the softmax output stage produces a network which computes posterior distributions over class labels based on an assumption of Gaussian within-class distributions. However the training, which uses cross-class information, can result in better performance at class discrimination than the usual within-class training method, unless the within-class distribution assumptions are actually correct. <s> BIB001 </s> A Survey on Context Learning <s> Neural Network Based Bi-gram Model <s> Currently, N-gram models are the most common and widely used models for statistical language modeling. In this paper, we investigated an alternative way to build language models, i.e., using artificial neural networks to learn the language model. Our experiment result shows that the neural network can learn a language model that has performance even better than standard statistical methods. <s> BIB002 </s> A Survey on Context Learning <s> Neural Network Based Bi-gram Model <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB003 </s> A Survey on Context Learning <s> Neural Network Based Bi-gram Model <s> Recommender systems have been researched extensively by the Technology Enhanced Learning (TEL) community during the last decade. By identifying suitable resources from a potentially overwhelming variety of choices, such systems offer a promising approach to facilitate both learning and teaching tasks. As learning is taking place in extremely diverse and rich environments, the incorporation of contextual information about the user in the recommendation process has attracted major interest. Such contextualization is researched as a paradigm for building intelligent systems that can better predict and anticipate the needs of users, and act more efficiently in response to their behavior. In this paper, we try to assess the degree to which current work in TEL recommender systems has achieved this, as well as outline areas in which further work is needed. First, we present a context framework that identifies relevant context dimensions for TEL applications. Then, we present an analysis of existing TEL recommender systems along these dimensions. Finally, based on our survey results, we outline topics on which further research is needed. <s> BIB004 </s> A Survey on Context Learning <s> Neural Network Based Bi-gram Model <s> There is a growing interest in identifying inorganic material affinity classes for peptide sequences due to the development of bionanotechnology and its wide applications. In particular, a selective model capable of learning cross-material affinity patterns can help us design peptide sequences with desired binding selectivity for one inorganic material over another. However, as a newly emerging topic, there are several distinct challenges of it that limit the performance of many existing peptide sequence classification algorithms. In this paper, we propose a novel framework to identify affinity classes for peptide sequences across inorganic materials. After enlarging our dataset by simulating peptide sequences, we use a context learning based method to obtain the vector representation of each amino acid and each peptide sequence. By analyzing the structure and affinity class of each peptide sequence, we are able to capture the semantics of amino acids and peptide sequences in a vector space. At the last step we train our classifier based on these vector features and the heuristic rules. The construction of our models gives us the potential to overcome the challenges of this task and the empirical results show the effectiveness of our models. <s> BIB005 </s> A Survey on Context Learning <s> Neural Network Based Bi-gram Model <s> Epileptic seizure is a serious health problem in the world and there is a huge population suffering from it every year. If an algorithm could automatically detect seizures and deliver the patient therapy or notify the hospital, that would be of great assistance. Analyzing the scalp EEG is the most common way to detect the onset of a seizure. In this paper, we proposed the context-learning based EEG analysis for seizure detection (Context-EEG) algorithm. The proposed method aims at extracting both the hidden inherent features within EEG fragments and the temporal features from EEG contexts. First, we segment the EEG signals into EEG fragments of fixed length. Second, we learn the hidden inherent features from each fragment and reduce the dimensionality of the original data. Third, we translate each EEG fragment to an EEG word so that the EEG context can provide us with temporal information. And finally, we concatenate the hidden feature and the temporal feature together to train a binary classifier. The experiment result shows the proposed model is highly effective in detecting seizure. <s> BIB006
Using artificial neural network to train language models was first introduced by BIB002 . The authors ask an enlightening question as indicated in the title of their work: Can Artificial Neural Networks Learn Language Models? To answer this question, this paper proposed an artificial neural network to form a bi-gram language model. The proposed neural network contains only two layers, the input layer of jV j input units and the output layer of jV j output units with jV j being the size of the vocabulary. The one-hot representation is adopted for words, therefore specifically the ith input unit is 1 if the previous word is the ith word in the vocabulary. The ith output unit denotes the probability of the ith word in the vocabulary being the current word. Hence, for each context of two adjacent words, a bi-gram conditional probability is derived from this two-layer neural network pðw t jw tÀ1 Þ ¼ countðw t ; w tÀ1 Þ=countðw tÀ1 Þ. When the vocabulary size jV j is large, the computational complexity can be very high, as the weight matrix is jV j Ã jV j. To overcome this problem, the model only updates its weights connected with the non-zero input unit during the backpropagation step. As a language model, the goal is to minimize the perplexity By minimizing this error function, the value of the ith output unit will converge to the condition probability pðw i jw j Þ given the previous word w j , which will lead this neural network to a model equivalent to a bi-gram language model without any smoothing. However, the lack of smoothing may result in over-fitting and a very poor performance on the test data set. To overcome the overfitting problem, the authors use an early-stopping technique in the training step, i.e., stop training when the model achieves the best performance on a small set of held-out data. To guarantee the values of the output units are probabilities (the sum is 1), the softmax activation function BIB001 is adopted for the output layer where y i is the unnormalized value of the the ith output unit y i ¼ w i;o þ P j w ij x ij , and x ij is the value of the jth input unit for the ith data. The empirical results show this neural network based bigram outperforms the standard n-gram model and even the Kneser-Ney smoothed n-gram model slightly with respect to perplexity (neural network based bi-gram: 11.16, n-gram: 11.99, KN n-gram: 11.17). However, due to the high dimensionality of the input and output layers, this model needs thousands times longer training time than the standard n-gram model. Many other researchers have proposed more sophisticated neural networks to address this problem ever since and much progress has been made. Ref. BIB003 is the first work that applied to a data set as large as millions of words. This work creatively introduces distributed representations for words into neural network based language models. Hence the dimensionality of input data is tremendously reduced and training time is shortened accordingly as we will discuss in the following section. By incorporating the distributed representation, the neural probabilistic language model BIB003 makes learning language model via a neural network structure practical and outperforms the conventional language models. A schematic illusion of the network architecture is shown in Fig. 5 . The basic idea is to predict the current word based on its historical context information, same as the n-gram model. Each word has its unique distributed representation, which is a real-valued feature vector and denoted by a row in matrix C. And C is a matrix of size jV j Â m, where jV j is the vocabulary size and m is the dimension of word feature vectors specified manually. Thus, w denotes a word and CðwÞ represents the corresponding word embedding, CðwÞ 2 R m . A three-layer neural network structure is adopted to implement the language model: The first layer is the projection layer which maps each word into its corresponding word embedding. w tÀnþ1 ; . . . ; w tÀ2 ; w tÀ1 on the bottom are the n À 1 preceding words as the historical context information. As with the n-gram model, this neural probabilistic language model is to predict the current word w t based on these n À 1 context words. Cðw tÀnþ1 Þ; . . . ; Cðw tÀ2 Þ; Cðw tÀ1 Þ are the word feature vectors corresponding to the n À 1 preceding words. By concatenating these word feature vectors, we obtain a context feature vector of length ðn À 1Þ Â m as the input for the second layer, and this context feature vector is denoted as x. The second layer is the hidden layer with tanh as the activation function where d is the bias and H is the hidden layer weight matrix. The third layer is the output layer. There are jV j units in the output layer, and the output value of the ith unit y i represents the unnormalized probability of the current word w t being the ith word in the vocabulary. At last, the output layer y is calculated as where b is the bias, W is the weight matrix for direct connections from input context features to the output layer (W can be set to 0 if there are no such direct connections), and U is the hidden output weight matrix. Like in BIB002 , a softmax normalization is adopted on the output layer to guarantee it is a probability distribution (summing up to 1). Thus, a context feature vector ðCðw tÀnþ1 Þ; . . . ; Cðw tÀ2 Þ; Cðw tÀ1 ÞÞ is mapped to a conditional probability of words in vocabulary for the current word w tP ðw t jw tÀ1 ; . . . ; w tÀnþ1 Þ ¼ e  where y i is the unnormalized probability in Equation BIB004 . The vocabulary size is normally very large for language models, therefore most computation happens at the output layer. Given the structure, training is achieved by searching for the parameter set that maximizes the training corpus penalized log-likelihood . . . ; w tÀnþ1 ; uÞ þ RðuÞ; where u ¼ ðb; d; W; U; H; CÞ is the parameter set consisting of weight matrices, biases and word embedding matrix, and RðuÞ is a regularization term (specifically, in the paper R is a weight decay penalty to prevent over-fitting by penalizing large weights which do not make a large contribution to the error reduction). Hence the neural network can be trained via stochastic gradient ascent, by iteratively updating the parameter set u as u u þ @ log ðP ðw t jw tÀ1 ; . . . ; w tÀnþ1 ; uÞÞ @u ; where is the learning rate. Note that unlike regular neural networks, this model updates both the weights and the input data (word embeddings) at each step. And only the word embeddings CðjÞ of words j inside the input window need to be visited and updated. This NPLM model outperforms the tri-gram language model by 24 percent in terms of perplexity. When dealing with sequential data, neural network based analysis can learn interesting features from the context information. BIB005 shows how to extract hidden features for amino acids and peptide sequences with a neural network based model. The main goal of this paper is to identify inorganic material affinity classes for peptide sequences. The task of peptide sequence analysis naturally shares some attributes in common with natural language processing, such as amino acids and words, peptide sequences and sentences. However, for this task, each peptide sequence carries more information than a text sentence: an unique binding property and a specific affinity class it belongs to. In order to incorporate the peptide sequence and affinity class knowledge, the neural network based context learning method also assigns feature vectors to the peptide sequences and affinity classes and takes them as part of the context input, as shown in Fig. 6 . In this way, this neural network structure turns into a semi-supervised method as the affinity class information is utilized . Based on the feature vectors extracted by this context learning method, an average improvement of 14 percent is achieved in terms of classification accuracy. Ref. BIB006 shows how to learn temporal information with a neural network based model when analyzing time series in bioinformatics.