Preben Wik
email: preben@speech.kth.seannah@speech.kth.se

Anna Hjalmarsson

Embodied conversational agents in Computer Assisted Language Learning

Keywords: Second langue learning, dialogue systems, embodied conversational agents, pronunciation training, CALL, CAPT

This paper describes two systems using embodied conversational agents (ECAs) for language learning. The first system, called Ville, is a virtual language teacher for vocabulary and pronunciation training. The second system, a dialogue system called DEAL, is a role-playing game for practicing conversational skills. Whereas DEAL acts as a conversational partner with the objective of creating and keeping an interesting dialogue, Ville takes the role of a teacher who guides, encourages and gives feedback to the students.

Introduction

An important aspect of research on computer assisted language learning (CALL) and computer assisted pronunciation training (CAPT) at the Centre for Speech Technology (CTT), KTH, is its focus on using embodied conversational agents (ECA) for language learning. Using the person metaphor rather than the desktop metaphor as an instructional interface could be beneficial in CAPT for several reasons:

• Users interacting with animated agents have been shown to spend more time with the system, think that it performs better, and enjoy the interaction more compared to interaction with a desktop interface [START_REF] Walker | Using a human face in an interface[END_REF][START_REF] Koda | Agents with Faces: The Effects of Personification of Agents[END_REF][START_REF] Lester | Increasing believability in animated pedagogical agents[END_REF][START_REF] Van Mulken | The Persona effect: How substantial is it?[END_REF]).

• Speech is multimodal and we communicate more than just verbally through our facial expression. It is well established that visual information supports speech perception [START_REF] Sumby | Visual Contribution to Speech Intelligibility in Noise[END_REF]. Since acoustic and visual speech are complementary modalities, introducing an ECA could make the learning more robust and efficient.

• Subjects listening to a foreign language often make use of visual information to a greater extent than subjects listening to their own language [START_REF] Burnham | The integration of auditory and visual speech information with foreign speakers: The role of expectancy[END_REF][START_REF] Granström | Prosodic cues in multi-modal speech perception[END_REF].

• The efficiency of ECAs for language training of deaf children has been demonstrated by [START_REF] Massaro | Using Visible Speech to Train Perception and Production of Speech for Individuals With Hearing Loss[END_REF][START_REF] Bosseler | Development and Evaluation of a Computer-Animated Tutor for Vocabulary and Language Learning for Children with Autism[END_REF] have also shown that using an ECA as an automatic tutor for vocabulary and language learning is advantageous for children with autism.

• ECAs are able to give feedback on articulations that a human tutor cannot easily demonstrate. Augmented reality display of the face that shows the position and movement of intra-oral articulators together with the speech signal may improve the learner's perception and production of new language sounds by internalizing the relationships between speech sounds and the gestures [START_REF] Engwall | Can audio-visual instructions help learners improve their articulation? -an ultrasound study of short term changes[END_REF].

We believe that using ECAs for language learning holds great promise for the future of CALL and CAPT. The challenge of making a virtual complement to a human tutor, or classroom teacher, that is infinitely patient, always available, and yet affordable, is an intriguing prospect.

This article describes two systems in which ECAs or computer-animated talking heads are used. Both systems are designed for language learning, but because of their very different roles and agendas, they behave differently, and have different design criteria.

The first system, called Ville, is a virtual teacher, guiding, encouraging and giving corrections and feedback on a student's pronunciation and language use. Ville and the underlying design criteria for the system are described in Section 3. A version of Ville without pronunciation analysis, but with logging and data collection abilities, has also been used by foreign students at KTH; this version is described in Section 4.

The feedback a learner of a new language (L2) receives when talking to a language teacher differs dramatically from the feedback one usually gets when talking to a native speaker. When a student makes a pronunciation error, a teacher or a proficient CAPT system should give explicit feedback on the L2 learner's utterance. In a communicative context however, when two people meet in a real dialogue, the pragmatic content of the exchange is what matters, and if there are pronunciation errors, they are usually not commented on.

The second system, called DEAL, is a role-play dialogue system for conversation training. The ECA in DEAL does not comment on a user's performance, but acts as a conversational partner, with the objective of creating and maintaining an interesting conversation. The feedback one can expect from a conversational exchange is backchannels and, when necessary, clarification questions. Mutual intelligibility is what is sought, and when it fails, communication breakdown is the result.

DEAL can also be seen as a meta-task, within the framework of Ville, of being a diagnostic gate-keeper to the next level of training. (A gate-keeper is a gaming term referring to entities protecting, or guarding gates to new levels in the game that the player must overcome in order to pass to the next level). If the student is able to successfully interact with the ECA in DEAL, it is a sign both to the student and to the system that the student is able to use the content of the previous lesson in a communicative context. The DEAL system is described in the end of the article in sect. 6.

Expressive abilities of the ECAs

The ECAs, or talking heads, that are developed at KTH [START_REF] Beskow | Talking heads -Models and applications for multimodal speech synthesis[END_REF], have the ability to link phonemes to visemes, thus synchronizing speech with lip movements. The architecture supports both synthetic speech from text (TTS) and pre-recorded utterances. TTS still has some limitations that may affect the CAPT program in a negative way, considering the fact that Ville is supposed to be a pronunciation model for the students. Ville's voice is therefore created from pre-recorded utterances. However, the ECA in DEAL has a TTS voice, since its utterances need to be generated in the course of the dialogue.

The ECAs can also move other parts of the head. Non-verbal signals such as head, eye, and eyebrow movements are used to signal stress, prominence, encouragement, or discourse changes such as turn-taking. The role-plays developed for DEAL should also include an aspect of drama, and the agent in DEAL should thus be able to display emotions such as surprise, anger, or joy.

In order to achieve varied and natural movements of the ECAs, a library of head and face movements has been developed. A sequence of movements (such as raising and lowering the eyebrows) is stored as a 'gesture', and a sequence of gestures can be stored as an 'event' or in a 'state'. An event is something that happens during a specified time frame, with a start and an end. A nod of the head together with a smile, as a confirmation that the student has done something correct, is an example of an event. A state is a loosely connected chain of gestures, without a defined start or end.

The ECA is always in a state and will stay in that state until some external event causes another state to begin. The state 'idle' for example, contains several types of blinking with the eyes, slight puckering of the mouth, tilting of the head, slightly turning the head left or right, where every such gesture has a weighted chance of occurring. Unless the student is actively interacting with the software, the agent is in the state 'idle'. In Ville, there is also a higher level collection of re-occurring 'conversational acts' (for example 'give praise', correct, incorrect) which are gestures and pre-recorded utterances with a common semantic meaning. A feedback expression like, for example, 'Correct' contains several gestures where Ville nods his head in various ways, and several pre-recorded utterances like 'Correct', 'Ok', 'Good', 'Yes'. Because gestures and utterances are selected independently of one another, it creates the impression of a larger and more natural variability in Ville's expressive repertoire through combinatorics. Should a version of Ville be created in a new target language, the library of gestures, events, states, and conversational acts can be kept intact, and only the recordings of actual words need to be replaced with equivalent recordings in the new target language.

Ville -The virtual language teacher

Ville is a virtual language teacher, who guides, encourages and gives feedback to students who wish to develop or improve their language skills. Ville takes on the teacher's role, selecting the words the students should say. This is a great advantage for the analysis stage in Ville, facilitating the task of correcting pronunciation errors, by having an implicit hypothesis of what kind of pronunciation errors a student is likely to make. The focus of Ville is on pronunciation and perception exercises designed to raise the awareness of particular aspects of the language that are known to be difficult for many L2 learners. Ville also helps with vocabulary training, providing a model pronunciation of new words, and drilling students in memorization exercises.

The first implementation of our virtual language teacher teaches Swedish to foreign university students at KTH, but our aim is to create a more general language tutor, with placeholders for language specific modules and user specific applications [START_REF] Engwall | Design strategies for a virtual language tutor[END_REF][START_REF] Wik | Designing a virtual language tutor[END_REF].

3.1.Contrastive analysis

Our vision is that Ville should be able to detect and give explicit feedback on all types of pronunciation errors that a language student is likely to make. Rather than giving a numerical score for how native-like a student's pronunciation is, which is common in current state-of-the-art CAPT, we want the system to be able to pin-point what type of pronunciation error the student makes in linguistic/phonetic terms. A numerical score will give a quantitative measure of the statistical similarity to the acoustic target, but such measures are difficult for the student to understand, and have been deemed insufficient to help students improve (Neri et al., 2002a). Knowledge-based rules that describe pronunciation difficulties will enable us to give feedback that is more similar to what teachers do, which will be more instructive and easier for the students to understand.

In order to help L2 learners' improve their pronunciation, insight into the types of error they make must first be obtained, and the implementation of detectors for specific pronunciation errors must be based on such insights. This strategy requires an interdisciplinary approach based on language-instruction pedagogy, phonetics, and speech technology. From the experience of language teachers we know that many difficulties L2 learners have are predictable, often based on the influence of their native language (L1). More specifically, difficulties are likely to occur for the learner of a new language if a distinction that carries meaning in the new language (L2) does not exist in the learner's native language. For example, L2 features not used to signal phonological contrast in L1 will be difficult to produce and perceive for the L2 learner [START_REF] Flege | Second-language Learning: The Role of Subject and Phonetic Variables[END_REF][START_REF] Mcallister | Perceptual foreign accent: L2 users' comprehension ability[END_REF].

By comparing the inventory of languages and using some form of contrastive phonetic, phonotactic, or prosodic analysis, a list of potential difficulties can be obtained [START_REF] Ellis | The Study of Second Language Acquisition[END_REF][START_REF] Meng | Deriving salient learners' mispronunciations from cross-language phonological comparisons[END_REF]. Not all features and contrasts in a language are, however, a source for pronunciation errors, and not all pronunciation errors are equally detrimental. A list of priorities is needed when deciding what detectors to include, and where to put the initial focus: Some errors are more common than others, and can be given priority based upon their frequency of occurrence. Some errors are easier for the student to correct than others. Some error-detectors are easier to build than others, and some pronunciation errors are perceived by native speakers as more serious than others, resulting in misunderstandings and communication breakdown.

Common pronunciation errors in Swedish as L2

Bannert, 2004 has investigated pronunciation difficulties in second language learners from 25 L1 languages, with Swedish as target language. Some of the most serious errors according to Bannert are:

• lexical stress (insufficient stress marking, or stress on the wrong syllable) • consonant deletion in a consonant cluster before a stressed vowel

3.2.Pronunciation error detectors in Ville

The sound processing in Ville is to a large extent done using Snack, an open source sound processing tool developed at KTH1 , in conjunction with N-align, the CTT aligner tool [START_REF] Sjölander | An HMM-based system for automatic segmentation and alignment of speech[END_REF]. Our current detectors are built on top of these components.

Vowel quantity detector

Students practice on carefully selected word-pairs, where the vowel duration of the stressed syllable changes the meaning of the word. Vowel quantity errors (failing to make a distinction between long and short vowels), are detected using forcedalignment, by identifying and time-marking phones, based on a transcription of what is being said and the speech wave of the student recording. The time segments are then normalized, and compared with a reference recording of the word. The Swedish language has complementary distribution, (i.e. a long vowel is followed by a short consonant and vice versa), so the relative duration of the following consonant must also be taken into consideration.

Lexical stress detector

Lexical stress is manifested by giving one of the syllables in a word more prominence. Commonly measured acoustic correlates to stress are pitch, intensity and duration. In Swedish, duration has been considered the most important correlate, which has also been noted in Dutch and English [START_REF] Sluijter | Phonetic correlates of stress and accent HIL dissertations 15[END_REF]. Pedagogically, it therefore makes sense to encourage L2 learners to use duration as the predominant variable. Since different phones have different intrinsic durations, one syllable may be longer than another without being perceived as prominent. This intrinsic duration must be normalized away before syllable length can be used as a measure for lexical stress. The lexical stress detector uses average phone durations measured from a number of speakers [START_REF] Carlson | Boundaries and groupings -the structuring of speech in different communicative situations: a description of the GROG project[END_REF], a syllable divided transcription, and forcedalignment to estimate which syllable is the most prominent in a student recording.

This estimate is then compared with a reference in order to determine if the student has placed the stress on the correct syllable or not.

Insertion and deletion detectors

Insertion and deletion errors are predictable in the sense that a mispronunciation hypothesis can be created in conjunction with certain consonant clusters. For example, many native Spanish speakers will produce a consonant cluster with an initial /s/ in Swedish by inserting a vowel in front of the /s/: 'Stockholm' thus becomes 'Estockholm'. Both our insertion and our deletion detectors employ forced-alignment in parallel on both the original transcription and on a modified transcription, where the hypothesised insertion or deletion is included. The aligner returns a distance measure (using viterbi search). By comparing the distance of the original transcription from the hypothesis transcription, we can determine the likelihood that an insertion or deletion was made.

3.3.Perception exercises

The first step in learning a new sound contrast is to be able to perceive it. If someone is unable to perceive a linguistic contrast, they are not likely to be able to produce it correctly either. The perception component of Ville does not contain speech technology components, but is still a very important part of the system from a pedagogical point of view. The exercises are designed to raise the awareness of the specific sound contrasts that are known to be difficult for many L2 learners. Minimal pairs are very useful for intuitively exposing learners to contrasts that exist in the target language (L2) but not in their native language (L1). In minimal pair exercises for vowel quality for example, a pair like /bita/-/byta/ ('bite' vs. 'swap') is presented on the screen, and Ville will randomly say one of the words. The student's task is to identify which word was uttered, and click on it. Ville will then give verbal feedback on the student's choice. Lexical stress exercises are performed in a similar fashion, by Ville saying a word and the student selecting the word type that was uttered. The classes are in this case not binary pairs, but a three by three grid with symbols representing different stress patterns and number of syllables, c.f. Figure 1.

Figure 1. A grid for lexical stress perception exercise: u is an unstressed syllable, -is a stressed.

3.4.Vocabulary

The central importance of vocabulary knowledge for language competence is clear.

Exercises for vocabulary training are therefore implemented as part of the virtual language teacher. 'Flashcards' are a popular and efficient vocabulary acquisition tool, where a deck of physical cards is normally used. A word is written in the L1 on one side of the card and in the L2 on the other. The card can be used two ways: Look at the L1 word and translate it to L2, or look at the L2 word and translate it to L1. The cards the student has learned can be discarded, so that they focus on the words they find difficult. The same paradigm is utilized in Ville, but with voice and pictures added to every word. Students can click on a card and hear Ville pronounce the word, make their own recordings of the words, and get feedback on their pronunciation.

3.5.Feedback and when to intervene

One of the most important aspects of being a teacher, virtual or real, is to give correct and relevant feedback. It is not easy for a virtual language teacher to know when it is appropriate to give feedback, how verbose it should be, and when it is better to refrain from talking. Having the opportunity to give verbal, multimodal feedback, does not in itself mean that it is always the best thing to do.

In an early version of Ville, pronunciation exercises included verbal feedback, in which Ville commented on the results from the detectors. In a vowel length exercise for example, Ville could say "Good, but your /a/ was a bit too long -try again, say: badda". Reactions from beta testers of the system revealed that such interventions were at first perceived as good, but that they soon became irritating and tiresome. This is similar to findings of [START_REF] Eskenazi | Using automatic speech processing for foreign language pronunciation tutoring: Some issues and a prototype[END_REF], who stated that "Interventions can appear to users as being either timely or irritating. Bothersome interventions tend to be caused by either recognition errors or by a system that intervenes too frequently and is too verbose."

We have hence implemented this part of the system so that, rather than using verbal feedback, the result from the pronunciation detectors is shown as iconic 'traffic lights'. Red or green circles will light up for the active detectors after a student recording Figure 3 .

Figure 3. Feedback on pronunciation is displayed with red (large) and green (small) lights.

The advantage with this type of visual feedback is that many lights can be shown in parallel, and the student will quickly be able to get an overview of how his performance was rated. If the student wishes to know why one of the circles was red, he can click on the circle, and a new page will appear with more detailed information such as graphs, or spectrograms, accompanied by the verbal feedback (Figure 4). If, on the other hand, this is a recurring error, and the student feels that he has already understood the information, he can simply note that the visual feedback indicates that the error is still occurring, and move on. Giving correct and relevant feedback is, however, a skill that many real teachers will spend years to develop, and that requires a great deal of intelligence, creativity, and sensitivity to master. Some teachers are clearly more successful than others at it and much more research is needed on how to successfully apply successful feedback didactics in CALL and CAPT [START_REF] Engwall | Pronunciation feedback from real and virtual language teachers[END_REF][START_REF] Neri | Feedback in Computer Assisted Pronunciation Training: Technology Push or Demand Pull?[END_REF].

Ville 1.0

Foreign students at KTH can study Swedish as a second language at the Unit for Language and Communication. The growing demand for classes, due to the large increase in foreign Master students in recent years, have inspired the language unit to seek new methods in their language training strategies. As a complementary selfstudy resource the unit has created a web-based beginner course in Swedish called SWELL (Swedish for Elementary Learners). For pronunciation and vocabulary practice, a version of Ville has also been offered since May 1 st 2008, in a first attempt to test Ville on real students, with real needs to learn a new language.

The Unit for Language and Communication wanted initial focus to be on vocabulary and provided pedagogical expertise to provide a training vocabulary of approximately 750 words and pictures divided into 27 lessons. This version of Ville contains no speech analysis, and no feedback is given on pronunciation (since analysis and identification of errors is a prerequisite for giving that type of feedback). Students are offered three types of exercises in this version.

Perception exercises: Ville says a word and the student must identify the corresponding picture (as described in Section 3.3)., Listen-and-click vocabulary exercises divided into perception tests, perception exercises, and perception games are offered (see Figure 5).

4.1.The need for a data collection

In order to develop new detectors for the pronunciation analysis part of the program, (as described in Section 3.1) we need to obtain speech data which contains typical pronunciation errors. For future development, we also see the need for a student profile, that is, a way to structure and store a student's performance and improvements, in order to create a lesson manager that can tailor lessons to a student's individual performance. These program enhancements are, however, difficult to develop without first having data on which to model the analysis, and such data is difficult to obtain without having the environment in which it occurs. Offering Ville to students within SWELL gives us the opportunity to collect real data from real students, and a way to resolve this dilemma.

Our approach is inspired by Lois Von Ahn's "Human Computation", and "Games with a purpose". von Ahn (2006) builds games that, when played by humans, produce useful computation as a side-effect. Through online games such as, for example, the ESP game2 , Verbosity3 , and Tag a tune4 , people are collectively solving large-scale computational problems in diverse areas such as image labelling, security, computer vision, Internet accessibility, adult content filtering, and Internet search. His players are offered entertainment, and provide researchers with brain power in return. Similarly, our users are offered education (hopefully with some entertainment value), and provide recordings and perception data in return. In both cases we are able to obtain user generated data for free, once the system has been built, in exchange for entertainment or education.

4.2.The Swedish L2 database

A data collection tool has been implemented in the version of Ville being offered to KTH students. All users were informed that they were taking part in a research project, and that, in order to use the program, they needed to give their consent that all data collected could be freely used for research purposes at KTH.

Every time a user makes a recording using the flashcards described in Section 3.4, the recording is saved locally on his computer, but it is also sent to our server. In addition to the actual speech files, an XML-file is created in which the text of the word spoken, the transcription, recording-style, and other information. are also stored. Since the user-generated data is stored in the same format as the content of the program, the student's own recordings can be retrieved and played back within the program, either by the student himself, or later by a researcher or a teacher.

The database currently being created by L2 learners of Swedish will eventually contain many instances of the same words spoken by different users. Many of the words are selected to highlight particular difficulties, and will be a good testbed for future pronunciation-error detectors. Recordings of the same word spoken by the same user at different times will further permit comparative studies of the user's improvement over time. Since all recordings are automatically annotated and saved, together with biometric data such as L1, sex, and age, this speech corpus could also in the future become a useful resource for other types of research. It could for example be used for comparative analysis between users of different groups (sex, age, country of origin, etc), or for ASR robustness or adaptation to non-native speech.

Perception exercises are also logged, and will be a valuable resource for determining which aspects of language learning the students are having difficulty with, and where to put the focus in future tool development. We can also determine which words are most difficult with regard to spelling or listening comprehension. We can also look at the distribution of exercises and recordings over time, per student, or per chapter. A synergy effect can be achieved when developing tools to analyze student performance on this level, since they will be similar to what is required for the lesson manager described in Section 4.1.

As Table1 shows that, although a substantial amount of users have tried the program, the number of exercises per user is in many cases very low. (170 students from Asia have for example only completed 79 writing exercises, which amounts to less than ½ exercise per student) This may in part be explained by users for whom the aim was not to learn Swedish, but who were merely curious to see what types of exercises and interaction that Ville offered. It may however also indicate that some users who did intend to use Ville to learn Swedish abandoned the program after a short time. The graphs in Figure 6 show that the total number of perception exercises (left) and recordings (right) are not evenly distributed among the students. Quite the contrary, it is reminiscent of a Zipfian distribution, where some students have been very active, while the contribution of the majority of users is quite small. The writing exercises follow a similar pattern. Only the 'active' students, i.e. those who have completed at least one exercise or recording are shown in the graphs. Note that, out of the 402 users, not more than 210 users have done any perception exercise, and only 90 users have made recordings.

4.3.Questionnaire

In January 2009 we also sent out a request to all the students at KTH who had used Ville, asking them to fill out a web-questionnaire in order to get some qualitative feedback. The questionnaire was anonymous, so we are not able to link student opinions to their performance. It focused instead on general questions that could give us indications of how students perceived the system, in order to help us improve it in the future.

Table 2 gives a summary of the questionnaire. All questions were multiple choice questions with the range 1-5 where 1 means "I do not agree at all" and 5 means "I totally agree". 59 students filled out the questionnaire.

Question

Mean St.Dev.

1. Ville is an effective tool for practicing Swedish at home.

3,91

1,09 2. I feel more at ease practicing pronunciation with Ville than with a real teacher.

2,91

1,13 3. I felt at ease doing recordings, even though I knew that my recordings were logged for research purposes.

3,36 There was also a free comment field where students could include their comments: Apart from some informative 'bug-reports' on misspellings and so on, we also got some great reviews. Here is a selection of comments:

• "Generally i see Ville is very useful tool in learning Swedish words writing and pronunciation , but the only thing needs attention is the user interface , it is not so user friendly when choosing between tests and perception and other options"

• "It is really great, and helps a lot in learning the Swedish language."

• "I have mostly used Ville to learn vocabulary and it was really useful. Maybe there should be something like saying the word in English and we have to write it correctly after that."

• "This is an extremely helpful program for beginners to get a know-how of Swedish language, especially pronunciation, vocabulary etc."

We can conclude that the system has been well received, at least by the students who filled out the questionnaire. It is interesting to note that the lowest score was on question 2 "I feel more at ease practicing pronunciation with Ville than with a real teacher" (reminding us of the programs role as a complement, not a replacement for a human teacher). Also note that people thought the program had helped them improve their pronunciation skills more than their listening and writing skills, even if many students made no, or very few recordings.

The number of Ville users and the amount of data is continuously increasing, but still limited. In order to get more data, we need both to increase the number of users, and to look at new ways to motivate the users we have to use the program more frequently.

Motivations for language learning

Learning a language requires a substantial effort, and the motivation for doing so varies both over time and between individuals. People learn a language for different reasons: A wish to be like the speakers of the language (integrative motivation) is often a strong motivating factor for younger learners, whereas the utility of what is learnt (instrumental motivation) is often a stronger motivator for adults. Motivation can also come from the pleasure of learning (intrinsic motivation), or from the task itself (task motivation) to mention some sources. Since the time spent in training is one of the key factors for language learning, increasing motivation and the time students spend practicing are crucial for a successful CALL application.

Game designers focus on finding ways to keep the players engaged and have in their pursuit of success, developed several effective design strategies both to get and to keep players engaged and motivated throughout a game. This is known as good gameplay. According to [START_REF] Prensky | The motivation of gameplay[END_REF] "Gameplay is all the doing, thinking and decision making that makes a game either fun or not". Good gameplay is what makes games addictive, and what makes millions of people spend a significant amount of their time and money on playing games. The pleasure of engagement is the motivation to play.

DEAL -A role-playing dialogue system for L2 learners

The same design principles that are used by game developers are starting to find their way into other fields as well. The notion of 'serious games' is an initiative focusing on using game design principles for purposes other than solely to entertain, for example training, advertising, simulation, or education [START_REF] Iuppa | Story and simulations for serious games : tales from the trenches[END_REF]. Good gameplay adds to any existing motivation to learn if there is one, and may otherwise create motivation by itself. The idea of transforming education and creating more engaging educational material by looking at the games industry has been suggested and described by several authors, for example [START_REF] Gee | What video games have to teach us about literacy and learning[END_REF] and [START_REF] Prensky | Digital game-based learning[END_REF]. To make our CALL and CAPT applications more effective we aim to look at new ways to motivate our users. The DEAL dialogue system has been developed to serve as a multidisciplinary research platform in the areas of human-like utterance generation [START_REF] Hjalmarsson | Dealing with DEAL: a dialogue system for conversation training[END_REF], game dialogue [START_REF] Brusk | DEAL -Dialogue Management in SCXML for Believable Game Characters[END_REF], and language learning [START_REF] Wik | DEAL A Serious Game For CALL Practicing Conversational Skills In The Trade Domain[END_REF].

Our aim is to allow language learners to practice conversational skills in a fun and challenging context. Our objective is similar to that of the Tactical Language Training System (TLTS) [START_REF] Johnson | The DARWARS Tactical Language Training System[END_REF], in the sense that both systems are simulation games for the acquisition of language and cultural skills. However, where TLTS places focus on realism (teaching US military appropriate manners and phrases to be used on foreign ground), we wish to focus more on entertainment. In that respect our objective is closer to Façade, a one act interactive drama where the player's interaction affects the outcome of the drama, and where the goal of the interaction is to create a good story [START_REF] Mateas | Façade: An experiment in building a fully-realized interactive drama[END_REF]. Our objective is also similar to that of the Nice project [START_REF] Gustafson | The NICE Fairy-tale Game System[END_REF], in that we wish to create a game in which spoken dialogue is not just an add-on, but is the primary means for game progression. Our ultimate goal is however on language learning.

6.1.Reasons for talking affects the design

Dialogue systems have traditionally been used mostly for information-seeking applications, such as train-schedule or weather-information systems. DEAL however, is a different type of system that is aimed at tutoring and entertainment. While the information-seeking type of system needs to be polite, unobtrusive and efficient, DEAL needs to be none of these things. Since the reason for talking to a dialogue system such as DEAL differs from an information-seeking application, the design criteria, and evaluation criteria will also be dramatically different. As an analogy we might consider some different reasons for walking. 1. One reason for walking might be to get somewhere, that is, the walking in itself is a hindrance for the real act, which is to arrive at the destination. Taking a shortcut or taking the car could be strategies to make it more efficient.

2. One could also choose to walk for health reasons, i.e. as a way to exercise. Then the walking in itself would be the reason for being on the road. Whether one enjoyed the walking or not, taking a shortcut would not be an option, since it would defeat the purpose of the walk.

3. Finally, one might wish to take a walk just for the pleasure of walking. A stroll in the park or a walk in the forest simply because it is an enjoyable and fun thing to do. Taking the car would be out of the question, since that would make one miss all the fun.

The first example is the walking equivalent of an information-seeking dialogue system. In such a system the user is not interested in the talking part, but wants to get an answer as quickly as possible. The talking equivalent of the second example could be to practice talking for language learning reasons. In a truly successful language learning application, the interaction would be so entertaining that the user would consider it to be a combination of 2 and 3: A game where the reason for playing would be just because it is fun (and at the same time getting useful exercise).

6.2.Evaluation of dialogue systems

As for evaluation, an information-seeking spoken dialogue system will be judged by factors such as efficiency in reaching task completion. A good system will thus try to minimize the number of turns needed (shorten the path). For DEAL the aim would be the opposite. The longer the conversation takes, and the more turns between the user and the system, the better. The interaction between the agent and the user ---if successful, will take the form of a role-play, and user satisfaction will depend on things other than efficiency in task completion. Apart from creating a good story, the social competence and personality of the character may be considered important factors, as well as the response time of the system, and how well the system handles errors. The criteria a non-native speaker (NNS) has for judging a dialogue system are different compared to a native speaker (NS). When a misunderstanding between a user and a spoken dialogue system occurs, a NS knows he has done nothing wrong, and will ascribe the misunderstanding to a weakness in the system. A NNS on the other hand will often be critical of his own ability in the new language, and might instead ascribe the misunderstanding to his own pronunciation, or incorrect use of vocabulary and grammar. A NNS will in a similar way be able to reason that if he is able to communicate with the system without communication breakdown, it can be seen as a confirmation of his abilities to communicate in the new language. This is a situation where the difficulties of ASR in combination with foreign accent could be re-interpreted as a difficulty within the gameplay. Rather than trying to adapt the ASR to be able to handle the strong accent, it becomes a part of the challenge of passing the gate-keeper. This way of motivating students to adapt their pronunciation is reminiscent of what happens in the real world, and may serve as an implicit learning factor. The limitations of the ASR can then be read as a measure of the student's communicative skills, where the challenge is to be able to negotiate with an ECA. If the student fails, he will not be allowed to move to the next level, and must go back to Ville -the pronunciation teacher, who will provide the student with the exercises and feedback she needs in order to try again, and hopefully pass the gatekeeper at a later time. In DEAL the agent can also choose to avoid acknowledging non-understandings or possible misunderstandings by tactical maneuvers, such as counter-questions or changing the subject.

6.3.Domain

DEAL serves as an important complement to Ville; whereas Ville provides exercises on isolated speech segments, i.e. phone, syllable, word, and sentence level, DEAL adds the possibility of practicing these segments in the context of a conversation. Ville has the role of a teacher who gives you feedback and help when you encounter problems. DEAL on the other hand has the role of a native speaker, for example, a person with a service occupation, whom you need to communicate with using your new language. There are many potential everyday situations in which one may want to use a new language. Good choices of domains include situations that follow familiar schemas or scripts, i.e. episodic knowledge structures that guide us in our daily interactions. The choice of domain for the first implementation of DEAL is a trading domain. Our first trading scenario is in a simulated flea market. This domain was chosen for several reasons:

• A trading situation is a fairly restricted and universally well-known domain. It is something everyone is conceptually familiar with, regardless of cultural and linguistic background. • It is a very useful domain to master in the new language • The flea market allows for, and almost invites characters who are eccentric or otherwise out-of-the-ordinary in an interesting way. • A flea market is a place where it is common to negotiate about price and to trade items. This type of negotiation is a complex process which includes both rational and emotional elements. • The shop can include almost any type of item. (In a larger framework vocabulary just learned in Ville can easily become items in the shop). • Second-hand items may have rich interesting characteristics or be defective and thus invite another type of conversation.

In summary it is a domain in which the user can engage in a dialogue that is well known but still includes elements of surprise, social commitment and competition (i.e. getting a good price).

6.4.Scenario

The basic teaching plan is for the language student is to use Ville in conjunction with DEAL. First Ville will teach the rudimentary vocabulary that is associated with the trade domain -that is: The numbers, some colors, a few objects like a clock and a teddy-bear, and a few phrases like "Do you have..." "How much does that cost" and so on. The student is then given a task to go to the nearby flea-market and use his newly acquired vocabulary in order to buy a given set of items from the shop-keeper in DEAL. The student is given a certain amount of money, but the money will not be enough to buy all the items on the student's list, unless he is creative. The stingy shopkeeper in the flea-market will try to get as much as possible for his goods. This scene can then unfold in different ways depending on what the student says. The willingness of the ECA to reduce the price of an item for example, may be affected by how the user gives praise or criticizes an item of interest, as, for example, in the dialogue below.

DEAL architecture

The dialogue system component in DEAL is based on Higgins, a spoken dialogue system developed at KTH [START_REF] Skantze | Galatea: a discourse modeller supporting concept-level error handling in spoken dialogue systems[END_REF]). Higgins includes modules for semantic interpretation and analysis. Pickering, a modified chart parser, supports continuous and incremental input from a probabilistic speech recognizer. Speech is unpredictable and chunking a string of words into utterances is difficult since pauses and hesitations will probably be incorrectly interpreted as end of utterance markers. This will be even more evident for second language learners whose conversation skills are not yet good and whose language contains disfluencies such as hesitations and false starts.

Pickering uses context-free grammars and builds deep semantic tree structures.

Grammar rules are automatically relaxed to handle unexpected, ungrammatical and misrecognized input robustly. The discourse modeler (DM), Galatea, interprets utterances in context and keeps a list of communicative acts (CA) in chronological order. Galatea resolves ellipses, anaphora, and has a representation of grounding status which includes information about who added a concept, in which turn a concept was introduced, and the concept's ASR confidence score. The system also contains an action manager (AM) described in 6.7, a communicative manager (CM), modules for text and text-to-speech generation (all described in 6.8) and a user interface (described in 6.6). structures that guide us when we interact with a shopkeeper in a shop in order to buy a product. Communicative acts used in DEAL include OBJECT-REQUEST, PROPERTY-REQUEST, PRICE-REQUEST, SUGGEST-PRICE, DEAL and so on. Consequently, the user can request objects, ask about object properties, give price offers, and make deals.

The haggling algorithm is a set of simple heuristics. These are based on the amount of the offer the user is suggesting in relation to the ''retail price'' which is stored in the system database. All objects in the database have obvious visual defects (e.g. the missing ear in Figure 8) and if detected and pointed out by the student, the agent reduces the price as shown in Dialogue example 1. The goal in DEAL is to build an emotional agent who is able to take initiative in the dialogue, if the student should fail to do so. As a first step in this development the agent looks angry or happy based on how the dialogue progresses. The agent responds with a smile to greetings and closings of deals. However, after long sequences of haggling or price offers from the student that are too low (less than 10% of the agent's initial price suggestion), the agent looks angry. The agent also takes initiative if no user input is provided, trying to bring the dialogue to a close. The action taken is based on the dialogue state; for example, if an object is in focus (on the table), the agent suggests a price for that object, and if no such object exists a new object is presented. An important characteristic of the system is that the goals of the agent and the student partly differ.

Both have the goal to complete a successful interaction; however the agent wants to sell objects for as much as possible while the student wants to buy them for the lowest possible price. In terms of gameplay, buying an object for a certain price must be challenging. To make the bargaining trickier, the agent easily gets ''fed up''. After a fixed set of speaker turns haggling about the price of a certain object the agent claims to be bored and refuses to discuss that object anymore. Instead, he suggests a new one.

6.8.Human-like language generation

DEAL is still under development and yet to be evaluated. Since the agent's behavior is crucial for how the system is perceived, the main effort so far has been to try to build a system that can generate utterances in a human-like manner. To encourage the user to talk to the system as if talking to another human being the agent needs to be responsive and flexible. Long response times and simple non-flexible utterances using templates or pre-recorded speech are not acceptable, if we are aiming for a system with a dialogue that is diverse and engaging.

Humans produce speech incrementally and on-line as the dialogue progresses using information from several different sources in parallel [START_REF] Brennan | Processes that shape conversation and their implications for computational[END_REF][START_REF] Aist | Software Architectures for Incremental Understanding of Human Speech[END_REF]. We anticipate what the other person is about to say in advance and start planning our next move while this person is still speaking. When starting to speak, we typically do not have a complete plan of how to say something, or even what to say. Yet, we manage to rapidly integrate information from different sources in parallel and simultaneously plan and realize new dialogue contributions. To keep the response times constant and without unnaturally long delays, the dialogue system needs to be capable of grabbing the turn, hold it while the system is producing the rest of the message, and release it after completion. A corpus of human-human dialogues in the DEAL domain was collected in order to study different human strategies on how to rapidly grab and maintain the floor while not knowing exactly what to say [START_REF] Hjalmarsson | Speaking without knowing what to say... or when to end[END_REF]. The data collection revealed a frequent use of linguistic devices often referred to as cue phrases or discourse markers. The function of these devices are to signal how new segments of speech relate to previous segments both within and between speaker turns. 81% of all speaker turns in the corpus contained at least one cue phrase and 21% of all words were labelled as cue phrases. Strategies that were frequently used to buy time while processing input was to generate filled pauses (e.g. eh, ehm), repetitions, or prosodic phrase-final lengthening. Incremental language production also requires that the system know how to alter, repair or refine previous utterances, such as when the system generates a response that was committed to too early and needs to revise it.

The generation task in DEAL is distributed over different modules. The communicative manager (CM) (see Figure 7) is responsible for detecting when it is appropriate for the system to speak and immediately initiates a new turn based on an early hypothesis from the input, or, if no such hypothesis is available, a grounding fragment or a filled pause. The CM also acts as an error-handling filter if the system is uncertain of some part of the incoming message. If the ASR confidence score for a particular entity is below a certain threshold, the CM generates a clarification CA without passing the message on to the action manager (AM), asking the user to clarify the entity before proceeding with the CA analysis any further. The system also utilizes the time it takes to make a complete analysis of input to ground the user's previous utterance (e.g. "ok a green watch"). This is done regardless of the fact that the availability of this object is unknown (i.e. the object could already be sold or not exist in the database). If the object for some reason turns out to be unavailable, the system revises its previous grounding segment and suggests another object (see dialogue example 2, s1a and b).

U1: I want to buy a green watch. S1a:

Ok, a green watch… S1b:

… I'm sorry there is no green watch but I do have a red one. U2: Do you have a yellow one? S3a: mm a yellow watch… S3b:

… here is one Dialogue example 2

If the user input contains no reference to a particular entity, the CM generates neutral feedback such as "yes" or "ok". Since the DM not only keeps track of the user's CAs but also its own previous CAs the CM can modify new responses from the AM based the type of feedback used in the first turn segment. An object that has already been grounded with a full noun phrase is referred to with a pronoun in the second part of the system response (see dialogue example 2, S3a and b).

The AM is responsible for deciding which action to take based on new input from the user, or, if no input is detected, it initiates an action based on the previous dialogue state. When the AM has generated a response it is passed on to the CM, which is responsible for modifying the response based on the previous dialogue context. For example, the CM decides how entities should be referred to, e.g. determines whether to use referring expressions or full noun phrases, as well as turning full propositions into elliptical constructions. The decisions are based on how well the entities are grounded in the dialogue, based on the confidence scores from the ASR and on whether these entities have been previously mentioned.

The CM forwards its message to Ovidius, the module responsible for realising the textual representation. Ovidius takes a system CA as input and generates a text that is subsequently realised acoustically by a speech synthesiser. Ovidius uses a set of template rules, working much like inverted Pickering grammar rules -they match the semantic tree structures and produce text strings. The acoustic realisation in the current version of DEAL is a combined set of pre-synthesized prompts and on-line text-to-speech generation. Feedback and other cue phrases as well as filled pauses are pre-synthesized prompts while the rest of the dialogue is synthesized speech generated online. The pre-synthesised elements have prosodic features, including F0 contour, speaker rate and energy, automatically extracted from the DEAL corpus. Since these elements are so frequently used, variation is essential or else the agent will sound too monotonous. An instance is randomly selected from a library of pre-synthesized prompts with the corresponding functional labelling (e.g. neutral-feedback, filled pause and so on). (For a more detailed description of the different functional classes see [START_REF] Hjalmarsson | Speaking without knowing what to say... or when to end[END_REF]. Whether the functions of these elements are interpreted by users as intended is, however, still to be evaluated.

Conclusions

Both the Ville and the Deal systems described in this article are ongoing projects, under constant revision and development. We intend them to serve as platforms for research in narrower sub-disciplines such as utterance generation, error handling and turn-taking in dialogue systems, and as testbeds for research on feedback strategies, pronunciation detectors, and other CALL-related research areas.

The release of Ville to real students has been quite successful. Although it has officially only been tried within the KTH campus so far, more than 400 students from 66 different countries have downloaded and tried the program, and most of the students who filled out the questionnaire were satisfied with the program. The method of collecting data (by offering education in exchange for student data) has proven very effective. A substantial amount of data has already been gathered, and new data is being generated continuously.

We have so far only scratched the surface of a large inter-disciplinary research area which has great potential to converge into something with a big impact on a large population group.

For those interested to read more about Ville and DEAL, the web sites www.speech.kth.se/ville and www.speech.kth.se/deal provide additional information and links to other publications.

thank Björn Granström, Rolf Carlson, Olov Engwall, Jens Edlund, and Julia Hirschberg for their valuable comments.

Figure 2 .

 2 Figure 2. Picture of the flashcards and recording section of the program

Figure 4 .

 4 Figure 4. Detailed feedback on duration analysis appears only if the student actively chooses to see it.

  Pronunciation exercises: Students are offered flashcards (as described in Section 3.4) and can record and play back their own recordings for self-evaluation.Writing exercises: Spelling exercises are also offered, where Ville says a word and the students should write down what they hear.

Figure 5 .

 5 Figure 5. A vocabulary perception exercise in Ville for SWELL

Figure 6 .

 6 Figure 6. The distribution of perception exercises (left) and recordings (right) done per student.

Figure 7 .

 7 Figure 7. The DEAL architecture

Figure 8 .

 8 Figure 8. The DEAL interface with a stingy shopkeeper trying to sell a teddy-bear with a missing ear.

Table 1 :

 1 of January 1 st 2009, 402 students from 66 different countries have used Ville. The distribution of students grouped by continents is shown in Table1. The users include both foreign students at KTH, (who have tried the software as part of the SWELL course) and external interested web users, who have downloaded the program without following a Swedish course at KTH. The number of exercises and recordings made by the students (grouped by continent)

				Perception	Writing
	Continent	Students Countries	Exercises	Exercises Recordings
	Asia	170	20	523	79	1277
	Europe	156	28	1345	574	4897
	Africa	18	7	189	17	61
	North America	7	1	77	8	92
	South America	10	6	128	24	203
	Central America	3	2	7	0	0
	Oceania	4	2	8	0	0
	Unknown	34	-	130	8	29
	Total	402	66	2407	710	6559

Table 2 .

 2 Questionnaire and mean opinion, in terms of mean score and standard deviation.

Snack: downloadable at http://www.speech.kth.se/snack/

http://www.gwap.com/gwap/gamesPreview/espgame

http://www.gwap.com/gwap/gamesPreview/verbosity

http://www.gwap.com/gwap/gamesPreview/tagatune

Acknowledgements

This research was carried out at the Centre for Speech Technology, KTH. The research is also supported by the Graduate School for Language Technology (GSLT). Many thanks to Jenny Brusk for her work in the DEAL project. We would also like to


Helmer Strik
email: h.strik@let.ru.nl

Khiet Truong
email: khiet.truong@tno.nl

Febe De Wet

Catia Cucchiarini
email: c.cucchiarini@let.ru.nl

Comparing different approaches for automatic pronunciation error detection

Keywords: pronunciation error detection, acoustic-phonetic classification, computer assisted pronunciation training, computer assisted language learning

One of the biggest challenges in designing computer assisted language learning (CALL) applications that provide automatic feedback on pronunciation errors consists in reliably detecting the pronunciation errors at such a detailed level that the information provided can be useful to learners. In our research we investigate pronunciation errors frequently made by foreigners learning Dutch as a second language. In the present paper we focus on the velar fricative /x/ and the velar plosive /k/. We compare four types of classifiers that can be used to detect erroneous pronunciations of these phones: two acoustic-phonetic classifiers (one of which employs lineardiscriminant analysis (LDA)), a classifier based on cepstral coefficients in combination with LDA, and one based on confidence measures (the so-called Goodness Of Pronunciation score). The best results were obtained for the two LDA classifiers which produced accuracy levels of about 85-93%.

Introduction

Computer assisted language learning (CALL) applications and, more specifically, computer assisted pronunciation training (CAPT) applications for language learning that make use of automatic speech recognition (ASR) have received considerable attention in recent years. Most of the literature on pronunciation assessment in language learning has focused on pronunciation grading (or scoring), while less attention has been paid to error detection (or localization). Pronunciation grading usually refers to a procedure used to calculate a global pronunciation score at the speaker or utterance level, which, for that matter, could also be a weighted average of local, phoneme scores. Error detection, on the other hand, requires calculating a score at a local (e.g. phoneme) level, for each individual realization of a given phone. Although this explanation might suggest that error detection is simply a specific sub-task of pronunciation grading, in fact these are two different tasks, with different goals and different outcomes. The distinction between pronunciation scoring and error detection becomes clear when we consider the specific goals for which they are employed. Pronunciation scoring is typically used in pronunciation testing applications to calculate global scores (whether or not obtained by averaging local scores) to provide an indication of the candidate's proficiency. Such global scores are usually not informative enough for applications like pronunciation training where students usually prefer to have more specific information on the nature of their pronunciation mistakes. Therefore, in pronunciation training, information should at least be provided at phoneme level for individual realizations of the various phones, so that learners can focus their attention on the most problematic sounds.

Automatic error detection at the phoneme level is often done by calculating so-called confidence measures (CMs) that can be derived from an ASR system based on Hidden Markov Models (HMM). Various types of CMs have been studied (e.g., [START_REF] Neumeyer | Automatic scoring of pronunciation quality[END_REF]. One of the best known examples is probably the Goodness Of Pronunciation (GOP) algorithm [START_REF] Witt | Use of speech recognition in Computer-assisted Language Learning[END_REF][START_REF] Witt | Phone-level Pronunciation Scoring and Assessment for Interactive Language Learning[END_REF], which has also been adopted by other authors (e.g. [START_REF] Mak | PLASER: Pronunciation Learning via Automatic Speech Recognition[END_REF]Neri et al., 2006a). CMs indicate how certain the recognizer is that a given target sound was pronounced correctly: the lower the confidence, the higher the chance that the sound (or utterance) was mispronounced. These CMs have the advantage that they can be obtained fairly easily with an ASR system, and they can be calculated in similar ways for all speech sounds.

To be of value in computer assisted pronunciation training systems, this type of evaluation should reflect human judgments of the same speech. However, correlations between CMs and human judgments are usually lower when they are computed for individual sounds than when they are computed over relatively long stretches of speech, e.g. a few utterances (e.g., [START_REF] Neumeyer | Automatic scoring of pronunciation quality[END_REF]. The smaller correlations observed at phoneme level may be explained by the lack of specificity that characterizes CMs, which are calculated using similar procedures for the same set of features for all the sounds, while this might not be the optimal approach to detect all pronunciation errors.

In a study of pronunciation errors in Dutch as a second language (L2), we found that learners of Dutch have problems with vowel length, and fricatives vs. plosives [START_REF] Neri | Segmental errors in Dutch as a second language: How to establish priorities for CAPT[END_REF](Neri et al., , 2006b)). This study was aimed at selecting the pronunciation errors that should be addressed in an ASR-based pronunciation training program for Dutch L2 called Dutch-CAPT. The aim of this training program was to help correct salient pronunciation problems that might hamper communication. The program addressed pronunciation errors that were selected on the basis of the following six criteria:

1.

Common across speakers of various L1's 2.

Perceptually salient 3.

Potentially hampering communication 4.

Frequent 5. Persistent 6.

Suitable for robust automatic detection Pronunciation errors that result in the realization of a Dutch phoneme different from the intended one and which lead to a different word being uttered are likely to hamper communication as they might result in a different message being understood. This might be the case when a long vowel is realized instead of a short one, thus producing the word /m n/ (moon) instead of /m n/ (man), or, analogously, when a velar plosive is realized instead of a velar fricative, for instance in words such as / s/ (gauze) and / s/ (gas), thus leading to the realization of the words /k s/ (cheese) and /k s/ (cash desk), respectively.

It is well known that HMMs are not very powerful devices to discriminate between sounds that are spectrally similar and differ mainly in duration; and HMMs are also not very well suited to discriminate between fricatives and plosives, since the difference between plosives and fricatives is coded in subtle differences in the amplitude envelope of the sound [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF], a feature that does not figure prominently in HMMs. One could think that by focusing more on these specific acoustic-phonetic details it might be possible to train classifiers that better discriminate mispronounced sounds from the correct, target sounds. This is where algorithms aimed at discrimination between specific phonetic classes, e.g. fricatives vs. plosives [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF], can come in handy.

Within the framework of our research on automatic pronunciation training for Dutch as a second language (DL2), we decided to explore the potential of specific acoustic phonetic feature analysis for automatic pronunciation error detection. We studied classifiers to discriminate long vs. short vowels, and fricatives vs. plosives [START_REF] Truong | Automatic pronunciation error detection in Dutch as a second language: an acoustic-phonetic approach[END_REF]). Here we focus on the discrimination of a typical pronunciation error in DL2 which is the substitution of the Dutch velar fricative /x/ with the velar plosive /k/. To select the specific acoustic phonetic features needed for this discrimination task, we draw on a previous study by [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF] which describes an algorithm that was able to discriminate voiceless fricatives from voiceless plosives with high accuracy.

We implemented and tested four different classifiers: [1] the GOP method; [2] a method based on [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF]; [3] the LDA-APF method in which acoustic-phonetic features (APF) based on [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF] are combined with linear discriminant analysis (LDA); and

[4] the LDA-MFCC method in which LDA is combined with Mel-Frequency Cepstrum Coefficients (MFCCs). The MFCCs are standard features which are generally used in ASR systems, e.g. for computing GOP-scores. Methods 1 and 4 are thus based on MFCCs, while methods 2 and 3 are based on acoustic-phonetic features; and methods 3 and 4 use LDA, while this is not the case for methods 1 and 2.

These four classifiers are described in section 2.3, which is the main part of section 2 'Material and method'. The classifiers were compared in a series of experiments of which the results are presented in section 3. These results open up new avenues of research and new possibilities of applications of these types of algorithms to automatic error detection in computer assisted pronunciation training. These and other issues are touched upon in section 4 entitled, 'Discussion and conclusions'.

Material and method

Material

All four classifiers used the same material for training and testing (see Table 1). We trained with native speech data taken from the Polyphone database (Den [START_REF] Os | The Dutch Polyphone corpus[END_REF]: read sentences, sampled at 8 kHz (telephone speech). For testing, we used two different conditions: [A] native speech from the same Polyphone database, and [B] non-native speech from the DL2N1 corpus (Dutch as Second Language, Nijmegen corpus 1). The DL2N1 corpus was collected in a previous study [START_REF] Cucchiarini | Quantitative assessment of second language learners' fluency by means of automatic speech recognition technology[END_REF]. In this corpus, 60 non-native speakers called from their home and read 10 Dutch phonetically rich sentences over the telephone. The subjects were sufficiently varied with respect to their mother tongue and proficiency level in Dutch. The speech (sampled at 8 kHz) was orthographically transcribed and automatically segmented by a speech recognizer.

The speech of 31 low-proficient non-native speakers of Dutch was annotated on pronunciation errors by a trained linguist. Because annotations of pronunciation errors are labourintensive and costly, we decided to have one expert carry out the task. Since it is well known that phonetic transcriptions may contain an element of subjectivity [START_REF] Cucchiarini | Assessing transcription agreement: Methodological aspects[END_REF], we tried to minimize the problem by choosing a qualified annotator who had previously carried out similar tasks following a consensus procedure with other experts. To determine whether the degree of subjectivity in our annotations was acceptable, we asked this annotator and five other Dutch experts to transcribe errors for a subset of the speech fragments and then compared the annotations pairwise using the transcription alignment program described in [START_REF] Cucchiarini | Assessing transcription agreement: Methodological aspects[END_REF]. The degree of agreement between our annotator and the five experts was generally very high: M = 89.3%, SD = 2.57. More details on the procedure adopted to check the reliability of the annotations are provided in Neri et al., (2006b) The phonemes /x/ and /k/ were automatically extracted from the signals based on a segmentation that was obtained by carrying out a forced alignment with the automatic speech recognition (ASR) system described in Section 2.2. The same automatic segmentation was used for all experiments.

Automatic speech recognition system

The ASR system consists of 37 context independent monophone hidden Markov phone models, and was implemented in the Hidden Markov Models Toolkit (HTK; [START_REF] Young | The HTK Book Version 3.0[END_REF]. The phone set includes a general speech model to account for unintelligible speech as well as a silence and a short pause model. Except for the short pause model, each hidden Markov model (HMM) has three states and 32 Gaussian mixtures per state. The single state of the short pause model is tied to the central state of the silence model. The HMMs are gender-dependent and were trained on 15.000 (7.500 male and 7.500 female) phonetically rich sentences from the Polyphone corpus (Den [START_REF] Os | The Dutch Polyphone corpus[END_REF]. The sentences were chosen such that the training material included at least 1.000 tokens for each phone.

Classifiers

In this section we describe four different methods to perform pronunciation error detection.

Method GOP: Goodness-Of-Pronunciation

The CMs that are derived from the ASR system, are obtained by a method suggested by [START_REF] Witt | Phone-level Pronunciation Scoring and Assessment for Interactive Language Learning[END_REF]. Gender-dependent HMMs are trained on native training data and are used to perform free phone and forced recognition for each utterance. The Goodness-Of-Pronunciation (GOP) score is subsequently derived at phone level. The GOP score for each phone corresponds to the frame normalized ratio between the log-likelihood score of its forced and free phone recognition. If the GOP score of a specific phone falls below a certain threshold, the pronunciation of the phone is accepted as a correct instance of the phone and vice versa. The GOP score is calculated in the same way for each phone, without taking specific properties of the phone into account.

According to the method proposed in [START_REF] Witt | Phone-level Pronunciation Scoring and Assessment for Interactive Language Learning[END_REF], the mean and variance of the GOP scores obtained from the training data can be used to choose an appropriate threshold for each phone. However, the resulting range of threshold values did not yield meaningful thresholds for our data. We therefore carried out an exhaustive search for values in the range between 0 and 10. We used the same optimization criterion that was used by [START_REF] Witt | Phone-level Pronunciation Scoring and Assessment for Interactive Language Learning[END_REF], i.e. maximizing the Scoring Accuracy (Correct Acceptance + Correct Rejection) for a given maximum level of False Acceptances, 10% in our case (see section 2.4 on evaluation).

Ideally, one should have enough non-native data available to derive the thresholds from real target data. Unfortunately, we did not have such a corpus available and therefore had to use artificial data that were generated on the basis of information obtained from the studies on pronunciation errors made by L2 learners of Dutch reported on in Neri et al. (2006b). These studies indicated that, for instance, / /, / /, / / and / / are often replaced by: /k/, / /, / / and / /, respectively. The artificial data were created by introducing errors into the pronunciation dictionary: for the present study incorrect pronunciations were "created" by changing the transcriptions of the phonemes /x/ and /k/.

Method Weigelt: Decision-tree

In [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF], an algorithm is described that successfully discriminates voiceless fricatives from voiceless plosives. We adopted this algorithm in our study to discriminate the voiceless velar fricative /x/ from the voiceless velar plosive /k/. Since the gross spectral shapes of plosives and fricatives are similar, several other cues to discriminate between these classes have been investigated, most of them based on the amplitude envelope. Weigelt's algorithm is based on three measures that can be obtained relatively easily and quickly: log rms (root-mean-square) energy, the first-order derivative of log rms energy, and zero-crossing rate. The main feature is the so-called Rate Of Rise (ROR) which is calculated by determining the derivative of the logarithm of rms energy. A 24 ms window n is shifted every 1 ms over the acoustic signal and for each window n, the energy is measured by computing the logarithm of the rms over window n:

× = 00002 . 0 10 log 20 n n rms E
ROR, the energy derivative, is then computed as:

( ) t E E ROR n n n ∆ - = -1
where t is the time step in which the window n is shifted, in our case 1 ms. The zero-crossing rate is defined as the number of times the signal crosses the axis. Since the release of the burst of the plosive causes an abrupt rise in amplitude, the ROR values of plosives are usually much higher than those of fricatives (see Figure 1). Consequently, the magnitude of the largest peak in the ROR contour can be used to discriminate plosives from fricatives. An ROR threshold can be set to classify sounds that have an ROR peak value above this threshold as plosives, and those sounds that are characterized by an ROR peak value under this threshold, as fricatives. In [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF], this threshold is heuristically set at 2240 dB/s. However, large peaks in the ROR contour can also be the result of other speech (e.g., vowel onset) or non-speech sounds (e.g., lip smacks). Therefore, the algorithm by [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF] uses four criteria to distinguish non-significant ROR peaks (i.e. those belonging to other speech/non-speech sounds) from significant ROR peaks (i.e. those that do belong to the release of the burst of the plosive). Starting with the largest peak, each criterion is examined: 1) for the 49-ms period following the peak, the value of E n must never fall below the value of E at the peak, 2) the maximum value of E n for the following 49 ms must be at least 12 dB above the value of E at the peak, 3) the maximum zero-crossing rate over the 49-ms period after the peak must be higher than 2000 zero crossings per second, and 4) the zero-crossing rate at exactly 49 ms after the peak must be no more than 100 crossings per second below the zero-crossing rate 20 ms before the peak. The derivative of log rms energy is a natural candidate for a measure that may capture differences in abrupt and slow rises of energy. Zero-crossing rates are used to discard peaks related to other speech/non-speech sounds. The duration of an energy pulse is examined to ensure that it is large enough in both amplitude and duration to be considered a speech sound. If any of these criteria fails, then the peak is labeled as non-significant and the phoneme is classified as a voiceless fricative. If the peak is significant and its ROR value is above the predetermined threshold, then the phoneme is classified as a voiceless plosive. All criteria were set and tuned by training and testing the algorithm using all possible combinations of values for the criteria. During this tuning stage, criterion four appeared to be too strict and was therefore not used in our adapted algorithm. The algorithm was optimized by adopting the same criterion that we applied in the GOP method, i.e. maximize the scoring accuracy for a False Acceptance level of maximally 10%. All acoustic measurements were carried out automatically in the program Praat [START_REF] Boersma | Praat, a system for doing phonetics by computer[END_REF] and were based on the same automatic segmentation that was used to compute the GOP measures. Figure 2 summarizes the adopted Weigelt binary decision-tree used in our study:

Method LDA-APF: Linear Discriminant Analysis with Acoustic-Phonetic Features

The relatively simple algorithm by [START_REF] Weigelt | Plosive/fricative distinction: the voiceless case[END_REF] proved to be successful in discriminating plosives from fricatives with energy as the primary acoustic-phonetic feature. However, in our adapted Weigelt's binary decision-tree, all criteria have the same weights. If any of the proposed criteria fails, then the phoneme is classified as a fricative, regardless of the importance of that criterion. In Linear Discriminant Analysis (LDA), weights are assigned to each feature in order to find the linear combination of features which best separates two or more classes. We use ROR and log rms energy, the main features in Weigelt's algorithm, as discriminative features in LDA to discriminate /x/ from /k/. The magnitude of the highest ROR peak, irrespective of its significance, is used, including 4 rms energy measurements that are made around that peak at 5 ms before (i1) and at 5 ms (i2), 10 ms (i3) and 20 ms (i4) after the peak (see Figure 1). In addition to these energy features, duration, either raw (dur) or normalized (normdur) is optionally added as a feature. Duration could be a discriminative cue since plosives are typically short acoustic events, while fricatives tend to be somewhat longer in duration. As non-natives tend to have lower articulation rates and longer segment durations [START_REF] Cucchiarini | Quantitative assessment of second language learners' fluency by means of automatic speech recognition technology[END_REF] duration was normalized for articulation rate (articulation rate is defined as the number of sounds divided by the total duration of the utterance without internal pauses). Duration normalization per speaker was achieved by computing the product of the articulation rate per speaker and segment duration:

duration segment rate on articulati duration normalized × =
All acoustic measurements were made in Praat and were based on the same automatic segmentation that was used for the other methods. The LDA models were trained using the same optimization criterion as in the GOP method.

Method LDA-MFCC: Linear Discriminant Analysis with Mel-Frequency Cepstrum Coefficients

The CMs calculated in method GOP are based on Mel-Frequency Cepstrum Coefficients (MFCCs). It is standard practice in ASR systems to use MFCCs, together with their first and second-order derivatives (which are often referred to as delta and delta-delta, respectively). In order to enable a fairer comparison between the LDA-APF and GOP classifiers, we decided to develop another classifier, i.e. LDA-MFCC, in which LDA is combined with MFCCs. Twelve MFCCs and one energy feature are calculated at the same moments that i1, i3 and i4 were extracted in method LDA-APF, together with their first and second-order derivatives. These 39 MFCC features were used to train (again using the same optimization criterion) and test the classifiers.

Evaluation

A classification algorithm like the GOP can produce four types of outcomes: 1) correctly accepted (CA) phone realizations, i.e. phones that were pronounced correctly and were also judged as correct; 2) correctly rejected (CR) phone realizations, i.e. phones that were pronounced incorrectly and were also judged as incorrect; 3) mispronunciations that were falsely judged as being correct (FA: False Accept) and 4) correct pronunciations that were falsely flagged as mispronunciations (FR: False Reject). To achieve optimal performance the algorithm should detect the mispronunciations and, at the same time, it should not flag as mispronunciations those realizations that were actually correct. For this reason both the amount of correctly rejected (CR) and correctly accepted (CA) realizations are important in the performance calculation.

The performance levels of the classifiers are evaluated by calculating the scoring accuracy, which is defined as: All four classifiers were optimized using the same criterion, i.e. maximizing the Scoring Accuracy (Correct Acceptance + Correct Rejection) while trying to keep the level of False Acceptances below 10%.

+ × =

Results

The four different classifiers, all trained on native speech, were tested on native (condition A) and non-native speech (condition B), for data of male and female speakers. This makes a total of 16 different combinations: 4 classifiers x 2 test conditions x 2 genders. The scoring accuracy (SA) values of these 16 different combinations are presented in Figure 3. The statistical significance of the differences between the different test conditions was determined by performing a z-test for the difference between independent proportions. Bonferroni correction for multiple testing was incorporated in the test. These results are presented in Table 2.

In Figure 3 it can be observed that all SA scores are quite high. In all four cases (male and female, test condition A & B) the classifiers can be ordered according to decreasing SA in the following way: two LDA methods, GOP, and Weigelt. The scores for the two LDA methods are similar. In condition B (mismatch: trained on native speech, tested on non-native speech of another corpus) the scores for LDA-APF are somewhat higher than those of LDA-MFCC, while in condition A (no mismatch) it is the other way around. However, these differences are not statistically significant. For test condition A, the SA scores are always higher for the data of the female speakers; these differences (between the male and female results) are significant for all the methods except GOP. Mixed results are obtained for condition B: higher SA scores for female data for the methods Weigelt and LDA-MFCC. Although the differences in SA are quite big, they are not significant in this case. This observation can be explained by the fact that the number of samples in the test population is much smaller in condition B than in condition A. Absolute differences in SA therefore need to be much bigger in condition B in order to be significant. Table 2. Statistical significance of the differences observed in Figure 3. Terminology: '=': difference is not significant, '>' or '<': significantly (p < 0.05) smaller or larger, respectively.

f m f m f Weigelt < < < < < < GOP = < < < LDA-APF = = Table 2b. Test condition B. GOP LDA-APF LDA-MFCC m f m f m f Weigelt < = < < < < GOP = < = < LDA-APF = =

Discussion & conclusions

Discussion

Several classifiers for automatic pronunciation error detection were developed and compared. These classifiers were developed for cases that are known to be problematic for foreigners that have to learn Dutch [START_REF] Neri | Segmental errors in Dutch as a second language: How to establish priorities for CAPT[END_REF](Neri et al., , 2006b)): vowel length (e.g., /A/ vs /a:/, /Y/ vs /u,y/), and pronunciation of fricatives such as the /x/ (which is often mispronounced as the plosive /k/). In the present study, we focus on the mispronunciation of /x/ as /k/; results for other types of pronunciation errors can be found in [START_REF] Truong | Automatic pronunciation error detection in Dutch as a second language: an acoustic-phonetic approach[END_REF]. Four different methods to discriminate between /x/ and /k/ were compared to each other: two methods based on commonly used MFCC features (GOP and LDA-MFCC), and two other methods based on acoustic-phonetic features (Weigelt and LDA-APF).

In practice, the amount of non-native speech data, and especially the amount of nonnative pronunciation errors, is not sufficient for reliable training and testing of classifiers. Witt proposed a method to introduce pronunciation errors in native data artificially, and used these artificial errors for developing the GOP classifier [START_REF] Witt | Use of speech recognition in Computer-assisted Language Learning[END_REF]. In this study we also artificially introduced errors in native speech. These artificially introduced errors were based on errors that are really made by non-natives, e.g. the incorrect pronunciation of /x/ as /k/. The test on native speech (condition A) was carried out to examine how well the discrimination between /x/ and /k/ could be made in "normal" native speech: the outcomes could serve as an upper-limit in performance. The test on non-native speech (condition B) was carried out to examine how well this distinction was made in non-native speech and to investigate whether the thresholds obtained with artificial errors in native speech were also suitable for non-native speech. The expectation was that the discrimination between /x/ and /k/ would be more difficult to make in non-native speech since non-native speakers usually utter speech sounds with a certain foreign accent that does not sound native-like, thus making the distinction between /x/ and /k/ less clear-cut. However, the results in Fig. 3 indicate that this is not the case, which suggests that foreign pronunciation errors can be detected using models that are trained with artificial errors in native speech. This is a considerable advantage given that data sparseness is a common problem in research on ASR-based L2 pronunciation error detection.

The results (Fig. 3) indicate that both LDA-APF and LDA-MFCC perform better than GOP and Weigelt. In Linear Discriminant Analysis (LDA), weights are assigned to each feature in order to find the linear combination of features which best separates the classes, while in the two other classifiers (that do not use LDA) all criteria have the same weights. For instance, in the LDA-MFCC classifier the largest weights are those of the energy features; LDA is thus capable of selecting those features that are most relevant. Apparently, this is an important advantage of the LDA-based classifiers compared to the other classifiers. When we look more closely at the differences between LDA-APF and LDA-MFCC we can observe that 1) LDA-APF performs better than LDA-MFCC in the condition B, but not in the condition A, and 2) the differences in performance between LDA-APF and LDA-MFCC are rather small and not statistically significant. Although LDA-APF and LDA-MFCC appear to achieve similar performance, from an efficiency and informative point of view, we prefer LDA-APF to LDA-MFCC: LDA-APF uses fewer features than LDA-MFCC, the features of LDA-APF are more intuitive (compared to MFCC's), and they can be useful for both learner (to provide meaningful feedback) and teacher (to make clear what the problematic pronunciation aspects are).

Furthermore, note that in condition B the test data were taken from a different corpus, and although this corpus also contains telephone speech, the (acoustic) properties of the signals can be slightly different. Since the APF features are more specific for a given speech sound, while the MFCC features are more general in nature, it is to be expected that when there is larger mismatch between training and test data/conditions, the APF features should perform better. Being more robust to training-test mismatches is another advantage of the APF-based classifiers.

In general, the advantage of an LDA-APF classifier is that one can focus on specific aspects of the error it has to detect, using features that are most suitable, making use of existing knowledge. However, an important disadvantage is that a specific classifier has to be developed for every pronunciation error. On the other hand, MFCCs are already available in the ASR system, and GOP scores can easily be obtained for all phones using similar procedures. However, the LDA-MFCC and GOP methods are also phone/error specific in that a phone-specific LDA matrix needs to be derived and a GOP threshold determined for each new pronunciation error that needs to be investigated. To increase the applicability of APF-based classifiers to other pronunciation errors, a more generic method to obtain acoustic-phonetic classifiers for these errors is needed. One of the directions that can be followed in future research is to develop training schemes that can automatically select the features that are most suitable for detecting each specific error.

Conclusions

The results of this study have shown that, for the correct classification of the velar fricative /x/ and the velar plosive /k/, both the LDA-APF and the LDA-MFCC methods yield better results than classification based on GOP scores and the decisions tree in Weigelt's method.

Although the LDA-APF method does not consistently outperform the LDA-MFCC method, it is more appealing from an efficiency and informative point of view because LDA-APF uses fewer features, the features that are used are relatively easy to extract and, in contrast to MFCCs, there is a clear link between feature properties and pronunciation errors.

Figure 1 .

 1 Figure 1. Log rms (top) and ROR (bottom) contours of /k/ (left) and /x/ (right).

Figure 3 .

 3 Figure 3. Results of classification experiments, /x/ vs. /k/, for male (top) and female (bottom) speakers

Table 1 . Number of tokens used to train and test the models Training (native, Polyphone) Testing condition A (native, Polyphone)

 1

	Testing condition B
	(non-native, DL2N1)

Table 2a

 2a

	. Test condition A.
	GOP	LDA-APF	LDA-MFCC
	m

Figure 2. Summary of decision-tree Weigelt, each criterion has to pass a certain threshold T.

Acknowledgements

We would like to thank the two anonymous reviewers for their useful comments.