Lipreading, i.e. speech recognition from visual-only recordings of a speaker's face, can be achieved with a processing pipeline based solely on neural networks, yielding significantly better accuracy than conventional methods. Feed-forward and recurrent neural network layers (namely Long Short-Term Memory; LSTM) are stacked to form a single structure which is trained by back-propagating error gradients through all the layers. The performance of such a stacked network was experimentally evaluated and compared to a standard Support Vector Machine classifier using conventional computer vision features (Eigenlips and Histograms of Oriented Gradients). The evaluation was performed on data from 19 speakers of the publicly available GRID corpus. With 51 different words to classify, we report a best word accuracy on held-out evaluation speakers of 79.6 using the end-to-end neural network-based solution (11.6 improvement over the best feature-based solution evaluated).
Lipreading has been used as a complementary modality for speech recognition from noisy audio data @cite_27 @cite_29 , as well as for purely visual speech recognition @cite_26 @cite_25 @cite_35 . The latter gives rise to a , which is defined as a system enabling speech communication to take place when an audible acoustic signal is unavailable'' @cite_5 . Silent Speech technology has a large number of applications: It allows persons with certain speech impairments ( laryngectomees, whose voice box ( larynx ) has been removed) to communicate, as well as enabling confidential and undisturbing communication in public places @cite_5 . Further uses of lipreading have been proposed, automatic speech extraction from surveillance videos and its interpretation for forensic purposes @cite_22 . Lipreading has been augmented with images of the tongue and vocal tract @cite_19 @cite_34 @cite_12 . Furthermore, there are Silent Speech interfaces based on very different principles, like speech recognition from electromyography @cite_20 @cite_21 @cite_8 @cite_33 or (electro-)magnetic articulography @cite_31 .
@cite_35 @cite_26 @cite_22 @cite_33 @cite_8 @cite_29 @cite_21 @cite_19 @cite_27 @cite_5 @cite_31 @cite_34 @cite_20 @cite_25 @cite_12
2106137268 2060756806 2157190406 2168861131 2145442746 190036406 2039640471 2129160496 2126584718 2017280922
We have designed and implemented a lipreading system that recognizes isolated words using only color video of human lips (without acoustic data). The system performs video recognition using "snakes" to extract visual features of geometric space, Karhunen-Loeve transform (KLT) to extract principal components in the color eigenspace, and hidden Markov models (HMM's) to recognize the combined visual features sequences. With the visual information alone, we were able to achieve 94 accuracy for ten isolated words. We improve the performance of a hybrid connectionist speech recognition system by incorporating visual information about the corresponding lip movements. Specifically, we investigate the benefits of adding visual features in the presence of additive noise and crosstalk (cocktail party effect). Our study extends our previous experiments by using a new visual front end, and an alternative architecture for combining the visual and acoustic information. Furthermore, we have extended our recognizer to a multi-speaker, connected letters recognizer. Our results show a significant improvement for the combined architecture (acoustic and visual information) over just the acoustic system in the presence of additive noise and crosstalk. > It has been determined that the myoelectric signals (MES) from muscles associated with speech that are obtained using surface electrodes can be used to recognize speech at approximately five times a priori. The results of ten experiments have been evaluated using a maximum-likelihood recognition scheme and have consistently yielded similar recognition accuracy among several subjects. Individual word recognition from a ten-word set has approached 60 . Parameters studied have included energy, magnitude, and trial deviation. The results suggest that accuracies at a level suitable for use in a vocal prosthesis will require the development of a hybrid recognition algorithm utilizing either statistical and heuristic word separation or neural nets. > A machine learning technique is used to match reconstructed tongue contours in 30 frame per second ultrasound images to speaker vocal tract parameters obtained from a synchronized audio track. Speech synthesized using the learned parameters and noise as an activation function displays many of the time and frequency domain characteristics of the original audio, and, for isolated passages, is remarkably clear - although no articulators other than the tongue are included. Abstract Surgical voice restoration post-laryngectomy has a number of limitations and drawbacks. The present gold standard involves the use of a tracheo-oesophageal fistula (TOF) valve to divert air from the lungs into the throat, which vibrates, and from this, speech can be formed. Not all patients can use these valves and those who do are susceptible to complications associated with valve failure. Thus there is still a place for other voice restoration options. With advances in electronic miniaturization and portable computing power a computing-intensive solution has been investigated. Magnets were placed on the lips, teeth and tongue of a volunteer causing a change in the surrounding magnetic field when the individual mouthed words. These changes were detected by 6 dual axis magnetic sensors, which were incorporated into a pair of special glasses. The resulting signals were compared to training data recorded previously by means of a dynamic time warping algorithm using dynamic programming. When compared to a small vocabulary database, the patterns were found to be recognised with an accuracy of 97 for words and 94 for phonemes. On this basis we plan to develop a speech system for patients who have lost laryngeal function. The article compares two approaches to the description of ultrasound vocal tract images for application in a "silent speech interface," one based on tongue contour modeling, and a second, global coding approach in which images are projected onto a feature space of Eigentongues. A curvature-based lip profile feature extraction method is also presented. Extracted visual features are input to a neural network which learns the relation between the vocal tract configuration and line spectrum frequencies (LSF) contained in a one-hour speech corpus. An examination of the quality of LSFs derived from the two approaches demonstrates that the Eigemongues approach has a more efficient implementation and provides superior results based on a normalized mean squared error criterion. A speech prosthesis has been developed based on the following idea. When a handicapped person such as a laryngectomee tries to speak in vain, the movements of the mouth, tongue, etc., are elicited. By detecting the movements, what he or she is trying to say can be determined. Then a speech synthesizer is driven to produce a voice of good quality. A recent trend in law enforcement has been the use of Forensic lip-readers. Criminal activities are often recorded on CCTV or other video gathering systems. Knowledge of what suspects are saying enriches the evidence gathered but lip-readers, by their own admission, are fallible so, based on long term studies of automated lip-reading, we are investigating the possibilities and limitations of applying this technique under realistic conditions. We have adopted a step-by-step approach and are developing a capability when prior video information is available for the suspect of interest. We use the terminology video-to-text (V2T) for this technique by analogy with speech-to-text (S2T) which also has applications in security and law-enforcement.