Computer Science
ComputerScience,1970-2008-Computer Science-2004.jsonl.gz.gz
Computer Science
991a1372f89e2ad88548614772d4f0ae1fdd72cd
Comparative Evaluations in the Domain of Automatic Speech Recognition

The goal of this contribution is threefold: (1) to present the results of a comparative evaluation of different, academic and commercial, speech recognitions engines; (2) to study relative performances of Hidden Markov Model and hybrid technologies, as used in stateof-the-art systems; and (3) to study the impact of different linguistic resources, such as simple word spotting, statistical and grammarbased language models, on the speech recognition accuracy. All the evaluations were made on the basis of the same test data sets and conclusions derived from the obtained Word Error Rate scores. The evaluated speech recognition engines are all speaker independent, continuous speech recognition engines, either academic systems widely used in the research community or commercial tools currently available on the market. In this work, we considered three academic systems (HTK, Sirocco, and Strut/DRSpeech) and two commercial ones (for the confidence reasons, we name these systems SRE1 and SRE2). The main obtained results are that (1) the Hidden Markov Model (HMM) based technology performs better than the hybrid approach in the case of unconstrained continuous speech, and (2) the academic systems perform better in the case of continuous speech in French, while the commercial systems show better recognition accuracy for continuous speech in German.


Introduction
The growing availability of different algorithms, technologies and systems in the domain of automatic speech recognition leads to a quite natural question: among the available options, what is the most adequate for a given application. Different evaluation campaigns, for example, NIST 1 , can be mentionned, aiming among others at the evaluation of conversational telephone speech or the one of speaker recognition. As to the presented work, the main interest was not to try to provide an exhaustive overview of all available academic and commercial speech recognition engines (SREs) and state-of-the-art technologies. The aim was to present the results of the evaluation work that we had to carry out to choose the SRE to be used in the Inspire project 2 aiming at the developpement of a dialogue-based control of different home appliances in the framework of a SmartHome environnement.
The rest of the paper is organized as follows: section 2. describes the evaluation framework; section 3. the used SREs, section 4. the used train and test data; the results of evaluation are reported in section 5.; and finally, section 6. proposes the discussion of the obtained results.

Evaluation framework
The framework used for the evaluation of the different SREs was similar to the one proposed by NIST: all the considered SREs are used to recognize the same test data set, in our case conversational telephone speech acquired during the two field tests of the InfoVox project (Rajman et al., 2003). The obtained Word Error Rate (WER) scores (Gillick, 1989;Pallett, 1990) are then used as a measure of the global recognition performance of the SREs. For these evaluation performances, the acoustic models (AM) used by the all academic SREs were trained on audio data of the same quality as test data, the Swiss French Polyphone database of conversational telephone speech. The AMs of the commercial systems were used as provided. As to the language model (LM) training, the same data (transcriptions of dialogues recorded over the phone during Wizardof-Oz experiments in the InfoVox project) was used for all the SREs.
The second task considered in this work was the comparison of HMM based and hybrid 3 technologies. For this purpose, the academic SREs using HMM on one hand and hybrid AM on the other hand were all used to recognize digits. Digits recognition was selected because no language model is necessary, and the obtained WER scores can thus be used as direct measures of the recognition performances of the underlying technologies. Data used to train the AM, as well as the testing data, were both extracted from the Swiss French Polyphone database.
Finally, the impact of the use of different linguistic resources (such as word spotting grammar, statistical and grammar based language models) on the recognition performances was studied. This was done for only one SRE, the commercial system Loquendo, and the same data set was used to produce the required language models.
All the results presented hereafter were computed with sclite evaluation tool (http://www.nist.gov/speech/tools).

Speech Recognition Engines
The systems used for the evaluations were the following: HTK : (http://htk.eng.cam.ac.uk) an academic system using the HMM technology; supports the whole speech recognition process; includes numerous algorithms for features extraction and the possibility to train AM and LM. The version 3.1 of the software was used; DRSpeech/STRUT : academic systems using a hybrid technology; support the whole speech recognition process; include numerous algorithms for features extraction and the possibility to train AM. The implementation of the softwares available at IDIAP 4 was used, and the version 2.9p1 of the Noway acoustic decoder was used; Sirroco : (http://www.irisa.fr/sirocco) an academic acoustic decoder integrating estimation of phonemes probabilitiy distribution on the basis of feature vectors extracted by another system, HTK for example. The version 1.2.1 of the software was used in the combination with HTK; Nuance : (http://www.nuance.com) a commercial system, supports the whole speech recognition process, the version 7.1 of the software was used; Loquendo : (http://www.loquendo.com/en) a (HMM based) commercial system, supports the whole recognition process; includes the possibility to train LM; the versions 4.1.0 and 5.9 of the software were used.

Test databases
Details about the used test data are presented in the tables 1 and 2 hereafter. In these tables, column "Type" indicates the type of audio input data, "Volume" provides the size of the databases in recordings and the size of the phonetic vocabularies in words, and "Quality" characterizes the coding of audio input data.  InfoVox 70 telephone dialogues recorded in French speaking Switzerland during the two field-tests of the In-foVox project. 236 sentences were recorded during the first (internal) field-test, 640 sentences were recorded during the second (external) field-test.

Acoustic Models
The table 3 presents the AMs used by the SREs. In this table, the column "System" indicates the system used to prepare the AM, the column "Language" the language of the AM, and the column "Model" the type of audio input the model can recognize.

Language Models
The Loquendo SRE is provided with a specific tool to train language models in its proprietary formats, SATCA. For the other systems, the CMU Toolkit 5 was used to produce one language model that has then been converted in the formats supported by each of the concerned systems. Table 4 presents the data used to train the LMs. The column "Database" indicates the sources, and "Volume" the size of the database in sentences and words.

Comparative evaluation of the SREs
The results of evaluation of the SREs are presented in the tables 5 for the InfoVox data and 6 for the Inspire data.   The first conclusion that can be drawn from the obtained results is that the academic systems using the HMM technology have the similar performances as the commercial ones. As to the hybrid technology, it performs worth than the HMM based technology. The poor performance of HTK on the Inspire data is due to the fact that the used AM is of weak recognition quality. This is shown in the next section.
The huge difference in the recognition of the InfoVox and Inspire databases observed for the Loquendo SRE can be explained by the quality of the used LMs: in the case of the InfoVox database, the train corpus is composed of only 22'790 words, in the case of the evaluation on the Inspire data, the size of train corpus is more than 290'000 words. 5 http://mi.eng.cam.ac.uk/prc14/toolkit.html

Comparative evaluation of the acoustic models
The results of evaluation are presented in the   The obtained results show that the HMM techonology performs better than the hybrid one. Then, the substantial difference between the performances of digits recognition with commercial and academic SREs may be due to the fact that the commercial SREs have often the possibility either to switch between several AMs tuned to specific applications, or to tune the used AM online during the recognition process.
Finally, the huge difference in the recognition of Swiss French Polypphone and SpeechDat (II) databases observed for the Loquendo SRE can be explained by the complexity of the tasks: the complexity of the first task (from 2 to 15 digits per recording) is much higher than in the case of SpeechDat (II) (only 1 digit per recording).

Comparative evaluation of the linguistic resources
The linguistic resources we wanted to evaluate are those that can be used by any state-of-the-art speech recognition engine. In our experiments, we concentrated on the LM and the grammar (either a Context-Free Grammar (CFG) or a Word-Spotting Grammar (WSG)). The goal of this evaluation was to measure the impact on the recognition performance of the different linguistic resources and their varying adaptation to the domain of application. To achieve a controlable variability of the resource adaptation, the resources were derived from a corpus containing a set of sentences only characteristic for the application (the Inspire WoZ database) and progressively extended with a growing number of sentences not characteristic for the application (extracted from the SpeechDat (II) database). The derived CFGs were of course not very realistic for real life applications, but they allowed to keep comparable evaluation conditions. For the CFGs, the sentences contained in the training corpora were directly used as rules in the grammar, therefore yielding a grammar of the following form: To produce the LMs, the sentences of the training corpora were used in the traditional way, i.e. for each training corpus, SATCA estimated the 3-grams LM.
For the WSG, we used a regular grammar accepting any sequence of any of the phonetic words extracted from Inspire WoZ (this grammar is also sometimes referred to as the "ergodic language model"). Formally, the definition of such a grammar is the following: The results of the evaluation of the linguistic resources are reported in the table 9 for LMs evaluation, table 10 for CFGs evaluation, and table 11 for WSG evaluation. Discussion of the obtained results is provided in the next section. In these tables, column "Training" explains the data used to train the models: the lines '10K + Inspire WoZ', '5K + Inspire WoZ', '1K + Inspire WoZ' and '500 + Inspire WoZ' means that 10'000, 5'000, 1'000 and 500 sentences were randomly chosen in the German SpeechDat (II) database and added to the Inspire WoZ database.
Performance variation associated with the different resources is displayed in the figure 1.

Discussion
Evaluation shows that the academic SREs using the HMM based techonology have the recognition perfor- Concerning the evaluation of the linguistic resources, a very interesting observation is that the LMs act in the opposite way as CFGs (figure 1). The reason for this can be the following: the perplexity of the trained CFGs increases with the increase of the train material. This is not the case for LM models: more train data leads to more accurate estimation of the same N-grams structures.
Finally, one interesting surprising conclusion concerns the fact that the use of a simple word spotting grammar can lead to better recognition performances than "traditional" LMs trained on huge corpora (table 11 and the first line in tables 6, 9, 10). That means that if no sufficiant corpora are available to train the LMs, one can always use the WSGs.

Acknowledements
We would like to acknowledge the anonymous reviewers that provided us with the clever indications about the content of the contribution.