Can advances in NLP help advance cognitive modeling? We examine the role of artificial neural networks, the current state of the art in many common NLP tasks, by returning to a classic case study. In 1986, Rumelhart and McClelland famously introduced a neural architecture that learned to transduce English verb stems to their past tense forms. Shortly thereafter, Pinker & Prince (1988) presented a comprehensive rebuttal of many of Rumelhart and McClelland's claims. Much of the force of their attack centered on the empirical inadequacy of the Rumelhart and McClelland (1986) model. Today, however, that model is severely outmoded. We show that the Encoder-Decoder network architectures used in modern NLP systems obviate most of Pinker and Prince's criticisms without requiring any simplication of the past tense mapping problem. We suggest that the empirical performance of modern networks warrants a re-examination of their utility in linguistic and cognitive modeling.
@cite_56 @cite_0 @cite_15 @cite_46 map phonological strings to phonological strings using feed-forward networks, but rather than turning to Wickelphones to imprecisely represent strings of any length, they use fixed-size input and output templates, with units representing each possible symbol at each input and output position. For example, @cite_0 @cite_15 simplify the past-tense mapping problem by only considering a language of artificially generated words of exactly three syllables and a limited set of constructed past-tense formation patterns. @cite_56 @cite_46 additionally modify the input template to include extra units marking particular transformations (e.g., or ), enabling their network to learn multiple mappings.
@cite_0 @cite_15 @cite_46 @cite_56
2025032307 2082568609 2104162748 2060208675
Abstract A three-layer back-propagation network is used to implement a pattern association task in which four types of mapping are learned. These mappings, which are considered analogous to those which characterize the relationship between the stem and past tense forms of English verbs, include arbitrary mappings, identity mappings, vowel changes, and additions of a suffix. The degree of correspondence between parallel distributed processing (PDP) models which learn mappings of this sort (e.g., Rumelhart & McClelland, 1986, 1987) and children's acquisition of inflectional morphology has recently been at issue in discussions of the applicability of PDP models to the study of human cognition and language (Pinker & Mehler, 1989; Bever, in press). In this paper, we explore the capacity of a network to learn these types of mappings, focusing on three major issues. First, we compare the performance of a single-layered perceptron similar to the one used by Rumerlhart and McClelland with a multi-layered perceptron. The results suggest that it is unlikely that a single-layered perceptron is capable of finding an adequate solution to the problem of mapping stems and past tense forms in input configurations that are sufficiently analogous to English. Second, we explore the input conditions which determine learning in these networks. Several factors that characterize linguistic input are investigated: (a) the nature of the mapping performed by the network (arbitrary, suffixation, identity, and vowel change); (b) the competition effects that arise when the task demands simultaneous learning of distinct mapping types; (c) the role of the type and token frequency of verb stems; and (d) the influence of phonological subregularities in the irregular verbs. Each of these factors is shown to have selective consequences on both successful and erroneous performance in the network. Third, we outline several types of systems which could result in U-shaped acquisition, and discuss the ways in which learning in multi-layered networks can be seen to capture several characteristics of U-shaped learning in children. In general, these models provide information about the role of input in determining the kinds of errors that a network will produce, including the conditions under which rule-like behavior and U-shaped learning will and will not emerge. The results from all simulations are discussed in light of behavioral data on children's acquisition of the past tense and the validity of drawing conclusions about the acquisition of language from models of this sort. The traditional account of the acquisition of English verb morphology supposes that a dual architecture underlies the transition from early rote-learning processes (in which past tense forms of verbs are correctly produced) to the systematic treatment of verbs (in which irregular verbs are prone to error). A connectionist account supposes that this transition can occur in a single mechanism (in the form of a neural network) driven by gradual quantitative changes in the size of the training set to which the network is exposed. In this paper, a series of simulations is reported in which a multi-layered perceptron learns to map verb stems to past tense forms analogous to the mappings found in the English past tense system. By expanding the training set in a gradual, incremental fashion and evaluating network performance on both trained and novel verbs at successive points in learning, it is demonstrated that the network undergoes reorganizations that result in a shift from a mode of rote learning to a systematic treatment of verbs. Furthermore, we show that this reorganizational transition is dependent upon the number of regular and irregular verbs in the training set and is sensitive to the phonological sub-regularities characterizing the irregular verbs. The pattern of errors observed is compared to that of children acquiring the English past tense, as well as children's performance on experimental studies with nonsense verbs. It is concluded that a connectionist approach offers a viable alternative account of the acquisition of English verb morphology, given the current state of empirical evidence relating to processes of acquisition in young children. The acquisition of English noun and verb morphology is modeled using a single-system connectionist network. The network is trained to produce the plurals and past tense forms of a large corpus of monosyllabic English nouns and verbs. The developmental trajectory of network performance is analyzed in detail and is shown to mimic a number of important features of the acquisition of English noun and verb morphology in young children. These include an initial error-free period of performance on both nouns and verbs followed by a period of intermittent over-regularization of irregular nouns and verbs. Errors in the model show evidence of phonological conditioning and frequency effects. Furthermore, the network demonstrates a strong tendency to regularize denominal verbs and deverbal nouns and masters the principles of voicing assimilation. Despite their incorporation into a single-system network, nouns and verbs exhibit some important differences in their profiles of acquisition. Most importantly, noun inflections are acquired earlier than verb inflections. The simulations generate several empirical predictions that can be used to evaluate further the suitability of this type of cognitive architecture in the domain of inflectional morphology. Abstract In a recent issue of this journal, Pinker and Prince (1988) and Lachter and Bever (1988) presented detailed critiques of Rumelhart and McClelland's (1986) connectionist model of the child's learning of the phonological form of the English past tense. In order to address these criticisms, a new connectionist model was constructed using the back-propagation algorithm, a larger input corpus, a fuller paradigm, and a new phonological representation. This new implementation successfully addressed the criticisms of the phonological representation used by Rumelhart and McClelland. It did a much better job of learning the past tense using a fuller input set with realistic frequencies of occurrence. Ancillary simulations using the same network were able to deal with the homonymy problem and the generation of forms like “ated” from “ate”. The one feature not provided by the new model was a way of modeling early correct production of irregular forms. The success of the new model can be used to help clarify the extent to which the published critiques apply to a particular connectionist implementation as opposed to fundamental principles underlying the broader connectionist conceptualization.