Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> Abstract We describe a model called the TRACE model of speech perception. The model is based on the principles of interactive activation. Information processing takes place through the excitatory and inhibitory interactions of a large number of simple processing units, each working continuously to update its own activation on the basis of the activations of other units to which it is connected. The model is called the TRACE model because the network of units forms a dynamic processing structure called “the Trace,” which serves at once as the perceptual processing mechanism and as the system's working memory. The model is instantiated in two simulation programs. TRACE I, described in detail elsewhere, deals with short segments of real speech, and suggests a mechanism for coping with the fact that the cues to the identity of phonemes vary as a function of context. TRACE II, the focus of this article, simulates a large number of empirical findings on the perception of phonemes and words and on the interactions of phoneme and word perception. At the phoneme level, TRACE II simulates the influence of lexical information on the identification of phonemes and accounts for the fact that lexical effects are found under certain conditions but not others. The model also shows how knowledge of phonological constraints can be embodied in particular lexical items but can still be used to influence processing of novel, nonword utterances. The model also exhibits categorical perception and the ability to trade cues off against each other in phoneme identification. At the word level, the model captures the major positive feature of Marslen-Wilson's COHORT model of speech perception, in that it shows immediate sensitivity to information favoring one word or set of words over others. At the same time, it overcomes a difficulty with the COHORT model: it can recover from underspecification or mispronunciation of a word's beginning. TRACE II also uses lexical information to segment a stream of speech into a sequence of words and to find word beginnings and endings, and it simulates a number of recent findings related to these points. The TRACE model has some limitations, but we believe it is a step toward a psychologically and computationally adequate model of the process of speech perception. <s> BIB001 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> Abstract Models of word recognition differ with respect to where the effects of sentential-semantic context are to be located. Using a crossmodal priming technique, this research investigated the availability of lexical entries as a function of stimulus information and contextual constraint. To investigate the exact locus of the effects of sentential contexts, probes that were associatively related to contextually appropriate and inappropriate words were presented at various positions before and concurrent with the spoken word. The results show that sentential contexts do not preselect a set of contextually appropriate words before any sensory information about the spoken word is available. Moreover, during lexical access, defined here as the initial contact with lexical entries and their semantic and syntactic properties, both contextually appropriate and inappropriate words are activated. Contextually effects are located after lexical access, at a point in time during word processing where the sensory input by itself is still insufficiently informative to disambiguate between the activated entries. This suggests that sentential-semantic contexts have their effects during the process of selecting one of the activated candidates for recognition. <s> BIB002 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> Abstract Previous work has shown a back-propagation network with recurrent connections can successfully model many aspects of human spoken word recognition (Norris, 1988, 1990, 1992, 1993). However, such networks are unable to revise their decisions in the light of subsequent context. TRACE (McClelland & Elman, 1986), on the other hand, manages to deal appropriately with following context, but only by using a highly implausible architecture that fails to account for some important experimental results. A new model is presented which displays the more desirable properties of each of these models. In contrast to TRACE the new model is entirely bottom-up and can readily perform simulations with vocabularies of tens of thousands of words. <s> BIB003 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> We present a new distributed connectionist model of the perception of spoken words. The model employs a representation of speech that combines lexical information with abstract phonological information, with lexical access modelled as a direct mapping onto this single distributed representation. We first examine the integration of partial cues to phonological identity, showing that the model provides a sound basis for simulating phonetic and lexical decision data from Marslen-Wilson and Warren (1994). We then investigate the time course of lexical access, and argue that the process of competition between word candidates during lexical access can be interpreted in terms of interference between distributed lexical representations. The relation between our model and other models of spoken word recognition is discussed. <s> BIB004 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> Since the publication of Oldfield's (1966) seminal article, “Things, Words and the Brain,” a great deal of attention has been devoted to the structural organization of words in the mental lexicon. Most of this research, however, has focused on the structure of higher level aspects of lexical representations, namely the semantic and conceptual organization of lexical items in memory (e.g., Miller & Johnson-Laird, 1976; Smith, 1978). As a consequence, little attention has been directed to the structural organization of the representations of sensory and perceptual information used to gain access to these higher level sources of information. The goal of the present investigation was to explore in detail this structure and its implications for perception of spoken words by normal and hearing-impaired listeners. ::: ::: In the present set of studies, structure will be defined specifically in terms of similarity relations among the sound patterns of words. Similarity will serve as the primary means by which the organization of acoustic-phonetic representations in memory will be investigated. We assume that similarity relations among the sound patterns of spoken words represent one of the earliest stages at which the structural organization of the lexicon comes into play. The precise aim of the present investigation was to gain a detailed understanding of the lower level relations between stimulus input, activation of phonetic representations, and, subsequently, recognition of spoken words. ::: ::: The identification of structure with similarity relations among sound patterns of words raises the difficult problem of defining similarity. Similarity, although crucial to the present investigation, is an ill-defined concept in research on speech perception and spoken word recognition, and one that deserves considerably more work in these areas of research (see Mermelstein, Reference Note 7). However, similarity can be approximated by both computational and behavioral predictors of confusion, the approach taken here. Thus, similarity will be defined in terms of a specific computational metric for predicting confusions among phonetic patterns as well as a behavioral, or operational, metric based on the results of a series of perceptual experiments. ::: ::: Having defined structure as the similarity relations among the sound patterns of words, the question arises: Should the structural organization of representations in memory have consequences for spoken word recognition? Consider a content-addressable memory system in which there is no noise either in the signal or the listener (Kohonan, 1980). In such a system, encoding the acoustic-phonetic information in the stimulus word is tantamount to locating the word in memory. In this case, the structural organization of acoustic-phonetic representations in memory would have no consequences for word identification. Instead, the task of spoken word recognition would be identical to phonetic perception, and one need only study phonetic perception to understand how words are recognized. (By phonetic perception we are referring to the perception of individual segments such as consonants and vowels.) ::: ::: It is undeniable that phonetic perception is important in spoken word recognition. It is not undeniable, however, that the human word recognition system operates as a noiseless content-addressable system or that the acoustic-phonetic signal itself is devoid of noise. To begin, the signal is very often less than ideal for the purposes of the listener. Words are typically perceived against a background of considerable ambient noise, reverberation, and the voices of other talkers. In addition, coarticulatory effects and segmental reductions and deletions substantially restructure the phonetic information in a myriad of ways (Luce & Pisoni, 1987). Although such effects may indeed be useful to the listener (Church, Reference Note 3; Elman & McClelland, 1986), they also undoubtedly produce considerable ambiguities in the speech signal, making a strictly content-addressable word recognition system based on phonetic encoding unrealistic. In short, both the noise inherent in the signal as well as the noise against which the signal is perceived make it unlikely that word recognition is accomplished by direct access, based solely on phonetic encoding, to acoustic-phonetic representations in memory. ::: ::: Not only is the speech signal noisy, so too is the recognition system of the listener. Although the human is clearly well adapted for the perception of spoken language, the system by which language is perceived is by no means a perfect one. In normal listeners, encoding, attentional, and memory demands frequently result in the distortion, degradation, or loss of important acoustic-phonetic information. The data on misperceptions alone attest to the fact that spoken word recognition is less than perfect (Bond & Garnes, 1980). In hearing-impaired listeners, the problems faced by normal listeners are exacerbated by impoverished input representations. Thus, again, a strictly content-addressable system does not suffice as a model of human spoken word recognition. ::: ::: The alternative to a noiseless content-addressable system is one in which the stimulus input activates a number of similar acoustic-phonetic representations or candidates in memory, among which the system must choose (Marslen-Wilson, 1989; Marslen-Wilson & Welsh, 1978; Triesman, 1978a, b). In this system, a considerable amount of processing activity involves discriminating among the lexical items activated in memory. Indeed, many current models of word recognition subscribe to the view that word recognition is to a great degree a process of discriminating among competing lexical items (Forster, 1979; Luce, Pisoni, & Goldinger, 1990; Marslen-Wilson, 1989; Marslen-Wilson & Welsh, 1978; McClelland & Rumelhart, 1981; McQueen, Cutler, Briscoe, & Norris, 1995; Morton, 1979; Norris, 1994). ::: ::: Given that one of the primary tasks of the word recognition system involves discrimination among lexical items, the study of the structural organization of words in memory takes on considerable importance, especially if structural relations can be shown to influence the ease or difficulty of lexical discrimination, and, subsequently, word recognition and lexical access. By the same token, under the assumption that word recognition involves discrimination and selection among competing lexical items, variations in the ease or difficulty of discriminating among items in memory can enlighten us as to the structural organization of the sound patterns of words. In short, lexical discrimination and structure are so inextricably tied together that the study of one leads to a further understanding of the other. ::: ::: Assuming, then, that structural relations among words should influence spoken word recognition via the process of discrimination, it is important to determine that structural differences among words actually exist. Previous research by Landauer and Streeter (1973) has demonstrated that words vary substantially not only in the number of words to which they are similar, but also in the frequencies of these similar words. These findings suggest that both structural and frequency relations among words may mediate lexical discrimination. Investigation of the behavioral effects of these relations should help us to understand further not only the process of lexical discrimination, but also the organization of the sound patterns of spoken words in memory. ::: ::: The issue of word frequency takes on an important role in the investigation of the structural organization of the sound patterns of words. Numerous studies over the years (Howes, 1954, 1957; Newbigging, 1961; Savin, 1963; Soloman & Postman, 1952) have demonstrated that the ease with which spoken words are recognized is monotonically related to experienced frequency, as measured by some objective count of words in the language. However, little work has been devoted to detailing the interaction of word frequency and structural relations among words (see, however, Triesman, 1978a, b). If word frequency influences the perceptibility of the stimulus word, it should also influence the degree of activation of similar words in memory. Frequency is important, then, in further specifying the relative competition among activated items that are to be discriminated among. ::: ::: The goal of the present investigation was, therefore, to examine the effects of the number and nature of words activated in memory on auditory word recognition. Throughout the ensuing discussion, the term similarity neighborhood will be employed. A similarity neighborhood is defined here as a collection of words that are phonetically similar to a given stimulus word. (The term stimulus word will be used to refer to the word for which a neighborhood is computed.) Similarity neighborhood structure refers to two factors: 1) the number and degree of confusability of words in the neighborhood, and 2) the frequencies of the neighbors. This first factor will be referred to as neighborhood density or neighborhood confusability; the second factor will be called neighborhood frequency. In addition to neighborhood structure, the frequency of the stimulus word itself will be of interest. <s> BIB005 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> This study demonstrates that listeners use lexical knowledge in perceptual learning of speech sounds. Dutch listeners first made lexical decisions on Dutch words and nonwords. The final fricative of 20 critical words had been replaced by an ambiguous sound, between [f] and [s]. One group of listeners heard ambiguous [f]-final words (e.g., [witlo?], from witlof, chicory) and unambiguous [s]-final words (e.g., naaldbos, pine forest). Another group heard the reverse (e.g., ambiguous [na:ldbo?], unambiguous witlof). Listeners who had heard [?] in [f]-final words were subsequently more likely to categorize ambiguous sounds on an [f]–[s] continuum as [f] than those who heard [?] in [s]-final words. Control conditions ruled out alternative explanations based on selective adaptation and contrast. Lexical information can thus be used to train categorization of speech. This use of lexical information differs from the on-line lexical feedback embodied in interactive models of speech perception. In contrast to on-line feedback, lexical feedback for learning is of benefit to spoken word recognition (e.g., in adapting to a newly encountered dialect). <s> BIB006 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> We conducted four experiments to investigate the specificity of perceptual adjustments made to unusual speech sounds. Dutch listeners heard a female talker produce an ambiguous fricative [?] (between [f] and [s]) in [f]- or [s]-biased lexical contexts. Listeners with [f]-biased exposure (e.g., [witlo?]; fromwitlof, “chicory”;witlos is meaningless) subsequently categorized more sounds on an [ef]-[es] continuum as [f] than did listeners with [s]-biased exposure. This occurred when the continuum was based on the exposure talker’s speech (Experiment 1), and when the same test fricatives appeared after vowels spoken by novel female and male talkers (Experiments 1 and 2). When the continuum was made entirely from a novel talker’s speech, there was no exposure effect (Experiment 3) unless fricatives from that talker had been spliced into the exposure talker’s speech during exposure (Experiment 4). We conclude that perceptual learning about idiosyncratic speech is applied at a segmental level and is, under these exposure conditions, talker specific. <s> BIB007 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> A Bayesian model of continuous speech recognition is presented. It is based on Shortlist (D. Norris, 1994; D. Norris, J. M. McQueen, A. Cutler, & S. Butterfield, 1997) and shares many of its key assumptions: parallel competitive evaluation of multiple lexical hypotheses, phonologically abstract prelexical and lexical representations, a feedforward architecture with no online feedback, and a lexical segmentation algorithm based on the viability of chunks of the input as possible words. Shortlist B is radically different from its predecessor in two respects. First, whereas Shortlist was a connectionist model based on interactive-activation principles, Shortlist B is based on Bayesian principles. Second, the input to Shortlist B is no longer a sequence of discrete phonemes; it is a sequence of multiple phoneme probabilities over 3 time slices per segment, derived from the performance of listeners in a large-scale gating study. Simulations are presented showing that the model can account for key findings: data on the segmentation of continuous speech, word frequency effects, the effects of mispronunciations on word recognition, and evidence on lexical involvement in phonemic decision making. The success of Shortlist B suggests that listeners make optimal Bayesian decisions during spoken-word recognition. <s> BIB008 </s> Non-native speech perception in adverse conditions: A review <s> Levels of processing <s> Dutch listeners were exposed to the English theta sound (as in bath), which replaced [f] in /f/-final Dutch words or, for another group, [s] in /s/-final words. A subsequent identity-priming task showed that participants had learned to interpret theta as, respectively, /f/ or /s/. Priming effects were equally strong when the exposure sound was an ambiguous [fs]-mixture and when primes contained unambiguous fricatives. When the exposure sound was signal-correlated noise, listeners interpreted it as the spectrally similar /f/, irrespective of lexical bias during exposure. Perceptual learning about speech is thus constrained by spectral similarity between the input and established phonological categories, but within those limits, adjustments are thorough enough that even nonnative sounds can be treated fully as native sounds. <s> BIB009
Although the incoming speech information is processed in a continuous manner, so that initial contact to stored lexical forms involves no conversion into any intermediate representation, listeners do draw on abstract representations of the phonemes of their language. They can, for instance, learn that a particular talker has a deviant pronunciation of a certain phoneme, and they can adjust their phonemic categorisations for that talker's speech very rapidly and on the basis of very little experience BIB006 BIB007 . The adjusted categorisations then apply to tokens of the phoneme in any word, not just to the words already heard from that talker; so if an ambiguous sound between /f/ and /s/ has been heard in words like gira[f/s], it will be identified as that talker's version of /f/, and if it is then presented in the context [naif/s], the word will be understood immediately to be knife and not nice . The phonemic difference for this talker between the adjusted /f/ and a real /s/ is exactly as effective in distinguishing between words in a cross-modal priming test as another talker's normal /f/ and /s/ BIB009 . Again, the point is that listeners are using phoneme information in the way that best enables them to recognise speech rapidly (in this case, learning from their experience of a talker's pronunciation so far, and using what has been learned to facilitate future processing of that person's speech). The forms that participate in the competition process, the earliest stage of word recognition, are phonological only; they are separate from the conceptual representations associated with the same word . This too supports efficiency; the speech information may activate lots of alternative words, but this does not lead in turn to consideration in parallel of lots of alternative semantic interpretations of the message. Instead, the most well-supported alternatives are passed on to be conceptually evaluated in the context so far. Sentence context does not rule out alternatives in advance, but as soon as one word is better supported by the input, its mismatched rivals can be ruled out by contextual evidence too. Thus in The crew mourned their capt-, the continuation captain is far more likely than captive, but nevertheless both remain available until just a bit more vowel support for the contextually appropriate alternative arrives and rules out the alternative BIB002 . The picture of native spoken-word recognition that arises from these decades of research is thus one of maximal efficiency. No decisions are made too early but the decision-making is cascaded so that the information from speech processing flows on from stage to stage as rapidly as possible. There are many models of spoken-word recognition, and they differ in particular in whether they allow higher-level processing to exercise control over processing at lower levels (e.g., BIB001 , or whether, as the evidence summarised here suggests, higher-level information does not rule out anything in advance but rather allows rapid selection between the best-supported alternatives BIB003 BIB004 BIB005 . The most recent model BIB008 allows various sources of information to be evaluated probabilistically, and this is also the first model which has fully incorporated frequency information, known to play a significant role in speech processing. All models agree on the necessary central features of the spoken-word recognition process, namely the simultaneous availability of, and competition between, multiple lexical alternatives.