Can advances in NLP help advance cognitive modeling? We examine the role of artificial neural networks, the current state of the art in many common NLP tasks, by returning to a classic case study. In 1986, Rumelhart and McClelland famously introduced a neural architecture that learned to transduce English verb stems to their past tense forms. Shortly thereafter, Pinker & Prince (1988) presented a comprehensive rebuttal of many of Rumelhart and McClelland's claims. Much of the force of their attack centered on the empirical inadequacy of the Rumelhart and McClelland (1986) model. Today, however, that model is severely outmoded. We show that the Encoder-Decoder network architectures used in modern NLP systems obviate most of Pinker and Prince's criticisms without requiring any simplication of the past tense mapping problem. We suggest that the empirical performance of modern networks warrants a re-examination of their utility in linguistic and cognitive modeling.
@cite_13 @cite_17 @cite_25 also solve an alternative problem---mapping semantic representations (usually one-hot vectors with one unit per possible word type, and one unit per possible inflection) to phonological outputs. As these networks use unstructured semantic inputs to represent words, they must act as memories---the phonological content of any word must be memorized. This precludes generalization to word types that were not seen during training.
@cite_13 @cite_25 @cite_17
2172262882 2019342855 2162108317
This paper investigates the ability of a connectionist attractor network to learn a system analogous to part of the system of English verbal morphology. The model learned to produce phonological representations of stems and inflected forms in response to semantic inputs. The model was able to resolve several outstanding problems. It displayed all three stages of the characteristic U-shaped pattern of acquisition of the English past tense (early correct performance, a period of overgeneralizations and other errors, and eventual mastery). The network is also able to simulate direct access (the ability to create an inflected form directly from a semantic representation without having to ftrSt access an intermediate base form). The model was easily able to resolve homophonic verbs (such as ring Abstract One of the fundamental difficulties facing a child trying to acquire a language is that the association between meanings and sounds is for the most part an arbitrary one. In this work, we model this process using a recurrent neural network that is trained to map a set of plan vectors, representing meaning, to associated sequences of phonemes, representing the phonological structure of the surface forms. We evaluate the role of the similarity structure of the target forms (the adult vocabulary) and the similarity structure of the input forms (the semantic structure) on the evolution of the network's vocabulary. The model's performance offers a principled account of various phenomena associated with children's early vocabulary development including the difficulty of acquiring synonyms, the appearance of idiosyncratic forms and over-extension errors. The model makes several unexplored predictions for the developmental profiles of young children acquiring morphology. Abstract An account is offered to change over time in English verb morphology, based on a connectionist approach to how morphological knowledge is acquired and used. A technique is first described that was developed for modeling historical change in connectionist networks, and that technique is applied to model English verb inflection as it developed from the highly complex past tense system of Old English towards that of the modern language, with one predominant “regular” inflection and a small number of irregular forms. The model relies on the fact that certain input-output mappings are easier than others to learn in a connectionist network. Highly frequent patterns, or those that share phonological regularities with a number of others, are learned more quickly and with lower error than low-frequency, highly irregular patterns. A network is taught a data set representative of the verb classes of Old English, but learning is stopped before reaching asymptote, and the output of this network is used as the teacher of a new net. As a result, the errors in the first network were passed on to become part of the data set of the second. Those patterns that are hardest to learn led to the most errors, and over time are “regularized” to fit a more dominant pattern. The results of the networks simulations were highly consistent with the major historical developments. These results are predicted from well-understood aspects of network dynamics, which therefore provide a rationale for the shape of the attested changes.