Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79 and 73 , respectively, as compared to the baseline of 74 on cross-validation.
In general three types of features are used: texture-based features, shape-based features, or a combination of both @cite_17 @cite_10 . Texture-based features exploit the pixel values in a ROI --- usually closely around the mouth or including the jaws @cite_17 . Typically, this is done by applying a transformation such as the discrete cosine transform (DCT) and or a dimensionality reduction technique such as the linear discriminant analysis (LDA) to the ROI, possibly in combination with a principle component analysis (PCA) or a maximum-likelihood linear transform (MLLT) @cite_17 . A common feature post-processing technique involves a chain of LDAs and MLLTs on concatenated frames, the so-called HiLDA @cite_21 .
@cite_21 @cite_10 @cite_17
1929897159 2122992400 1991904384 2281723898
We propose the use of a hierarchical, two-stage discriminant transformation for obtaining audio-visual features that improve automatic speech recognition. Linear discriminant analysis (LDA), followed by a maximum likelihood linear transform (MLLT) is first applied to MFCC based audio-only features, as well as on visual only features, obtained by a discrete cosine transform of the video region of interest. Subsequently, a second stage of LDA and MLLT is applied to the concatenation of the resulting single modality features. The obtained audio-visual features are used to train a traditional HMM based speech recognizer. Experiments on the IBM ViaVoice sup TM audio-visual database demonstrate that the proposed feature fusion method improves speaker-independent, large vocabulary, continuous speech recognition (LVCSR) for both clean and noisy audio conditions considered. A 24 relative word error rate reduction over an audio-only system is achieved in the latter case. Linear discriminant analysis (LDA) is a popular feature extraction technique for face recognition. However, It often suffers from the small sample size problem when dealing with the high dimensional face data. Fisherface and null space LDA (N-LDA) are two conventional approaches to address this problem. But in many cases, these LDA classifiers are overfitted to the training set and discard some useful discriminative information. In this paper, by analyzing different overfitting problems for the two kinds of LDA classifiers, we propose an approach using random subspace and bagging to improve them respectively. By random sampling on feature vector and training samples, multiple stabilized Fisherface and N-LDA classifiers are constructed. The two kinds of complementary classifiers are integrated using a fusion rule, so nearly all the discriminative information is preserved. We also apply this approach to the integration of multiple features. A robust face recognition system integrating shape, texture and Gabor responses is finally developed. In this work, a new method is proposed to the widely neglected problem of Arabic font recognition, it uses global texture analysis. This method is based on fractal geometry, and the feature extraction does not depend on the document contents. In our method, we take the document as an image containing some specific textures and regard font recognition as texture identification. We have combined both techniques BCD (box counting dimension) and DCD (dilation counting dimension) to obtain the main features. The first expresses texture distribution in 2-D image. The second makes possible to take on the human vision system aspect, since it makes it possible to differentiate one font from another. Both features are expressed in a parametric form; then four features were kept. Experiments are carried out by using 1000 samples of 10 typefaces (each typeface is combined with four sizes). The average recognition rates are of about 96.2 using KNN (K nearest neighbor) and 98 using RBF (radial basic function). Experimental results are also included in the robustness of the method against written size, skew, image degradation (e.g., Gaussian noise) and resolution, and compared with the existing methods. The main advantages of our method are that (1) the dimension of feature vector is very low; (2) the variation sizes of the studied blocks (which are not standardized) are robust; (3) less samples are needed to train the classifier; (4) finally and the most important, is the first attempt to apply and adapt fractal dimensions to font recognition. A key issue regarding feature extraction is the capability of a technique to extract distinctive features to represent facial expressions while requiring a low computational complexity. In this study, the authors propose a novel approach for appearance-based facial feature extraction to perform the task of facial expression recognition on video sequences. The proposed spatiotemporal texture map (STTM) is capable of capturing subtle spatial and temporal variations of facial expressions with low computational complexity. First, face is detected using Viola–Jones face detector and frames are cropped to remove unnecessary background. Facial features are then modelled with the proposed STTM, which uses the spatiotemporal information extracted from three-dimensional Harris corner function. A block-based method is adopted to extract the dynamic features and represent the features in the form of histograms. The features are then classified into classes of emotion by the support vector machine classifier. The experimental results demonstrate that the proposed approach shows superior performance compared with the state-of-the-art approaches with an average recognition rate of 95.37, 98.56, and 84.52 on datasets containing posed expressions, spontaneous micro-expressions, and close-to-real-world expressions, respectively. They also show that the proposed algorithm requires low computational cost.