The present invention generally pertains to speech recognition applications and systems. More specifically, the present invention pertains to methods of identifying and analyzing performance problems during development of speech recognition applications.
Although great progress has been made in speech recognition technology during the past two decades, adoption of speech technology has not gone as smoothly as might be desired. This is evidenced by the relatively limited deployment of speech applications. The main barrier to wider adoption of speech recognition technology stems from the cost incurred in developing the speech applications. A typical speech application includes application logic, dialog, one or more grammars, a speech recognition engine, etc. Since it is unlikely that the system will be available right at the very beginning (i.e., without significant development), constructing a quality speech application usually involves four steps, namely, (1) design; (2) prototyping; (3) evaluation and tuning; and (4) deployment. Typically, the prototyping step and the evaluation and tuning step are performed iteratively, fine tuning the speech application through each iteration to improve performance.
Among these four steps, evaluation and tuning during test and pilot stages is one of the most important phases and usually costs most. The tuning phase can take many months and requires a team of developers, testers, and speech technology experts.
Significant efforts have been directed toward the goal of reducing the total cost needed to develop and deploy speech applications. For example, a series of development tools have recently been released by companies working in speech recognition. The development tools significantly improved the development process. However, some critical functionality is not available in these tools. For instance, speech application authors usually don't know what to do when the success rate of their application is not satisfactory, even though they have access to large quantities of application logs. It would therefore be very valuable to automatically (or semi-automatically) determine what callers are struggling with, or which parts of their application need the most work.
The present invention provides solutions to one or more of the above-described problems and/or provides other advantages over the prior art.