Can advances in NLP help advance cognitive modeling? We examine the role of artificial neural networks, the current state of the art in many common NLP tasks, by returning to a classic case study. In 1986, Rumelhart and McClelland famously introduced a neural architecture that learned to transduce English verb stems to their past tense forms. Shortly thereafter, Pinker & Prince (1988) presented a comprehensive rebuttal of many of Rumelhart and McClelland's claims. Much of the force of their attack centered on the empirical inadequacy of the Rumelhart and McClelland (1986) model. Today, however, that model is severely outmoded. We show that the Encoder-Decoder network architectures used in modern NLP systems obviate most of Pinker and Prince's criticisms without requiring any simplication of the past tense mapping problem. We suggest that the empirical performance of modern networks warrants a re-examination of their utility in linguistic and cognitive modeling.
@cite_13 @cite_17 @cite_25 also solve an alternative problem---mapping semantic representations (usually one-hot vectors with one unit per possible word type, and one unit per possible inflection) to phonological outputs. As these networks use unstructured semantic inputs to represent words, they must act as memories---the phonological content of any word must be memorized. This precludes generalization to word types that were not seen during training.
@cite_13 @cite_25 @cite_17
2125076245 2493916176 2025032307 2546915671
Representing words into vectors in continuous space can form up a potentially powerful basis to generate high-quality textual features for many text mining and natural language processing tasks. Some recent efforts, such as the skip-gram model, have attempted to learn word representations that can capture both syntactic and semantic information among text corpus. However, they still lack the capability of encoding the properties of words and the complex relationships among words very well, since text itself often contains incomplete and ambiguous information. Fortunately, knowledge graphs provide a golden mine for enhancing the quality of learned word representations. In particular, a knowledge graph, usually composed by entities (words, phrases, etc.), relations between entities, and some corresponding meta information, can supply invaluable relational knowledge that encodes the relationship between entities as well as categorical knowledge that encodes the attributes or properties of entities. Hence, in this paper, we introduce a novel framework called RC-NET to leverage both the relational and categorical knowledge to produce word representations of higher quality. Specifically, we build the relational knowledge and the categorical knowledge into two separate regularization functions, and combine both of them with the original objective function of the skip-gram model. By solving this combined optimization problem using back propagation neural networks, we can obtain word representations enhanced by the knowledge graph. Experiments on popular text mining and natural language processing tasks, including analogical reasoning, word similarity, and topic prediction, have all demonstrated that our model can significantly improve the quality of word representations. Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models to learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram, words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks. Abstract A three-layer back-propagation network is used to implement a pattern association task in which four types of mapping are learned. These mappings, which are considered analogous to those which characterize the relationship between the stem and past tense forms of English verbs, include arbitrary mappings, identity mappings, vowel changes, and additions of a suffix. The degree of correspondence between parallel distributed processing (PDP) models which learn mappings of this sort (e.g., Rumelhart & McClelland, 1986, 1987) and children's acquisition of inflectional morphology has recently been at issue in discussions of the applicability of PDP models to the study of human cognition and language (Pinker & Mehler, 1989; Bever, in press). In this paper, we explore the capacity of a network to learn these types of mappings, focusing on three major issues. First, we compare the performance of a single-layered perceptron similar to the one used by Rumerlhart and McClelland with a multi-layered perceptron. The results suggest that it is unlikely that a single-layered perceptron is capable of finding an adequate solution to the problem of mapping stems and past tense forms in input configurations that are sufficiently analogous to English. Second, we explore the input conditions which determine learning in these networks. Several factors that characterize linguistic input are investigated: (a) the nature of the mapping performed by the network (arbitrary, suffixation, identity, and vowel change); (b) the competition effects that arise when the task demands simultaneous learning of distinct mapping types; (c) the role of the type and token frequency of verb stems; and (d) the influence of phonological subregularities in the irregular verbs. Each of these factors is shown to have selective consequences on both successful and erroneous performance in the network. Third, we outline several types of systems which could result in U-shaped acquisition, and discuss the ways in which learning in multi-layered networks can be seen to capture several characteristics of U-shaped learning in children. In general, these models provide information about the role of input in determining the kinds of errors that a network will produce, including the conditions under which rule-like behavior and U-shaped learning will and will not emerge. The results from all simulations are discussed in light of behavioral data on children's acquisition of the past tense and the validity of drawing conclusions about the acquisition of language from models of this sort. Recurrent neural networks (RNNs) have achieved state-of-the-art performances in many natural language processing tasks, such as language modeling and machine translation. However, when the vocabulary is large, the RNN model will become very big (e.g., possibly beyond the memory capacity of a GPU device) and its training will become very inefficient. In this work, we propose a novel technique to tackle this challenge. The key idea is to use 2-Component (2C) shared embedding for word representations. We allocate every word in the vocabulary into a table, each row of which is associated with a vector, and each column associated with another vector. Depending on its position in the table, a word is jointly represented by two components: a row vector and a column vector. Since the words in the same row share the row vector and the words in the same column share the column vector, we only need @math vectors to represent a vocabulary of @math unique words, which are far less than the @math vectors required by existing approaches. Based on the 2-Component shared embedding, we design a new RNN algorithm and evaluate it using the language modeling task on several benchmark datasets. The results show that our algorithm significantly reduces the model size and speeds up the training process, without sacrifice of accuracy (it achieves similar, if not better, perplexity as compared to state-of-the-art language models). Remarkably, on the One-Billion-Word benchmark Dataset, our algorithm achieves comparable perplexity to previous language models, whilst reducing the model size by a factor of 40-100, and speeding up the training process by a factor of 2. We name our proposed algorithm to reflect its very small model size and very high training speed.