Can advances in NLP help advance cognitive modeling? We examine the role of artificial neural networks, the current state of the art in many common NLP tasks, by returning to a classic case study. In 1986, Rumelhart and McClelland famously introduced a neural architecture that learned to transduce English verb stems to their past tense forms. Shortly thereafter, Pinker & Prince (1988) presented a comprehensive rebuttal of many of Rumelhart and McClelland's claims. Much of the force of their attack centered on the empirical inadequacy of the Rumelhart and McClelland (1986) model. Today, however, that model is severely outmoded. We show that the Encoder-Decoder network architectures used in modern NLP systems obviate most of Pinker and Prince's criticisms without requiring any simplication of the past tense mapping problem. We suggest that the empirical performance of modern networks warrants a re-examination of their utility in linguistic and cognitive modeling.
In this section, we first describe direct followups to the original 1986 model, employing various neural architectures. Then we review competing non-neural systems of context-sensitive rewrite rules in the style of the Sound Pattern of English (SPE) @cite_19 , as favored by Pinker and Prince.
@cite_19
2079402412 2044525914 2529089661 2952913664
Three neural network models were trained on the forward mapping from articulatory positions to acoustic outputs for a single speaker of the Edinburgh multi-channel articulatory speech database. The model parameters (i.e., connection weights) were learned via the backpropagation of error signals generated by the difference between acoustic outputs of the models, and their acoustic targets. Efficacy of the trained models was assessed by subjecting the models’ acoustic outputs to speech intelligibility tests. The results of these tests showed that enough phonetic information was captured by the models to support rates of word identification as high as 84 , approaching an identification rate of 92 for the actual target stimuli. These forward models could serve as one component of a data-driven articulatory synthesizer. The models also provide the first step toward building a model of spoken word acquisition and phonological development trained on real speech. We examined a variety of real-time responses evoked by a single piece of music, the organ Duetto BWV 805 by J S Bach. The primary data came from a concurrent probe-tone method in which the probe tone is sounded continuously with the music. Listeners judged how well the probe tone fit with the music at each point in time. The process was repeated for all probe tones of the chromatic scale. A self-organizing map (SOM) (Kohonen 1997 Self-organizing Maps (Berlin: Springer)) was used to represent the developing and changing sense of key reflected in these judgments. The SOM was trained on the probe-tone profiles for 24 major and minor keys (Krumhansl and Kessler 1982 Psychological Review 89 334^368). Projecting the concurrent probe-tone data onto the map showed changes both in the perceived keys and in their strengths. Two dynamic models of tonality induction were tested. Model 1 is based on pitch class distribu- tions. Model 2 is based on the tone-transition distributions; it tested the idea that the order of tones might provide additional information about tonality. Both models contained dynamic components for characterizing pitch strength and creating pitch memory representations. Both models produced results closely matching those of the concurrent probe-tone data. Finally real- time judgments of tension were measured. Tension correlated with distance away from the predominant key in the direction of keys built on the dominant and supertonic tones, and also correlated with dissonance. Classical translation models constrain the space of possible outputs by selecting a subset of translation rules based on the input sentence. Recent work on improving the efficiency of neural translation models adopted a similar strategy by restricting the output vocabulary to a subset of likely candidates given the source. In this paper we experiment with context and embedding-based selection methods and extend previous work by examining speed and accuracy trade-offs in more detail. We show that decoding time on CPUs can be reduced by up to 90 and training time by 25 on the WMT15 English-German and WMT16 English-Romanian tasks at the same or only negligible change in accuracy. This brings the time to decode with a state of the art neural translation system to just over 140 words per seconds on a single CPU core for English-German. Neural sequence-to-sequence models have provided a viable new approach for abstractive text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text). However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves. In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways. First, we use a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator. Second, we use coverage to keep track of what has been summarized, which discourages repetition. We apply our model to the CNN Daily Mail summarization task, outperforming the current abstractive state-of-the-art by at least 2 ROUGE points.