Can advances in NLP help advance cognitive modeling? We examine the role of artificial neural networks, the current state of the art in many common NLP tasks, by returning to a classic case study. In 1986, Rumelhart and McClelland famously introduced a neural architecture that learned to transduce English verb stems to their past tense forms. Shortly thereafter, Pinker & Prince (1988) presented a comprehensive rebuttal of many of Rumelhart and McClelland's claims. Much of the force of their attack centered on the empirical inadequacy of the Rumelhart and McClelland (1986) model. Today, however, that model is severely outmoded. We show that the Encoder-Decoder network architectures used in modern NLP systems obviate most of Pinker and Prince's criticisms without requiring any simplication of the past tense mapping problem. We suggest that the empirical performance of modern networks warrants a re-examination of their utility in linguistic and cognitive modeling.
Of the proposals that map semantics to phonology, the architecture in @cite_13 is unique in that it uses an attractor network rather than a feed-forward network, with the main difference being training using Hebbian learning rather than the standard backpropagation algorithm. @cite_25 present an early use of a simple recurrent network @cite_4 to decode output strings, making their model capable of variable length output.
@cite_13 @cite_25 @cite_4
2172262882 2019342855 2110485445
This paper investigates the ability of a connectionist attractor network to learn a system analogous to part of the system of English verbal morphology. The model learned to produce phonological representations of stems and inflected forms in response to semantic inputs. The model was able to resolve several outstanding problems. It displayed all three stages of the characteristic U-shaped pattern of acquisition of the English past tense (early correct performance, a period of overgeneralizations and other errors, and eventual mastery). The network is also able to simulate direct access (the ability to create an inflected form directly from a semantic representation without having to ftrSt access an intermediate base form). The model was easily able to resolve homophonic verbs (such as ring Abstract One of the fundamental difficulties facing a child trying to acquire a language is that the association between meanings and sounds is for the most part an arbitrary one. In this work, we model this process using a recurrent neural network that is trained to map a set of plan vectors, representing meaning, to associated sequences of phonemes, representing the phonological structure of the surface forms. We evaluate the role of the similarity structure of the target forms (the adult vocabulary) and the similarity structure of the input forms (the semantic structure) on the evolution of the network's vocabulary. The model's performance offers a principled account of various phenomena associated with children's early vocabulary development including the difficulty of acquiring synonyms, the appearance of idiosyncratic forms and over-extension errors. The model makes several unexplored predictions for the developmental profiles of young children acquiring morphology. Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly context-dependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type token distinction.