Note: Accepted at NIPS Deep learning Workshop Reference EPFL-REPORT-192559 Record created on 2013-12-19, modified on 2017-05-10
In speech, one of the first phoneme recognition system based on neural network was the Time Delay Neural Network @cite_2 . It was extended to isolated word recognition @cite_5 . At the same time, the hybrid HMM ANN architecture approach @cite_4 @cite_26 was developed, leading to more scalable systems. Recently, the deep belief network @cite_13 approach has been found to yield good performance in phone recognition @cite_12 . It was also extend to context-dependent phonemes in @cite_16 . However, these systems used complex hand-crafted features, such as MFCC. Later, there has been growing interests in using short-term spectrum as features. These intermediate'' representations (standing between raw signal and classical'' features such as cepstral-based features) have been successfully used in speech recognition applications, for example in unsupervised feature learning @cite_15 , acoustic modeling @cite_1 and large vocabulary speech recognition @cite_18 @cite_25 . Convolutional neural networks also yield state-of-the-art results in phoneme recognition @cite_23 . Although these systems are able to learn efficient representations, these features are still used as input for hybrid systems.
@cite_18 @cite_26 @cite_4 @cite_1 @cite_23 @cite_2 @cite_5 @cite_15 @cite_16 @cite_13 @cite_25 @cite_12
2963175699 2117671523 1666984270 2076462394
In hybrid hidden Markov model artificial neural networks (HMM ANN) automatic speech recognition (ASR) system, the phoneme class conditional probabilities are estimated by first extracting acoustic features from the speech signal based on prior knowledge such as, speech perception or and speech production knowledge, and, then modeling the acoustic features with an ANN. Recent advances in machine learning techniques, more specifically in the field of image processing and text processing, have shown that such divide and conquer strategy (i.e., separating feature extraction and modeling steps) may not be necessary. Motivated from these studies, in the framework of convolutional neural networks (CNNs), this paper investigates a novel approach, where the input to the ANN is raw speech signal and the output is phoneme class conditional probability estimates. On TIMIT phoneme recognition task, we study different ANN architectures to show the benefit of CNNs and compare the proposed approach against conventional approach where, spectral-based feature MFCC is extracted and modeled by a multilayer perceptron. Our studies show that the proposed approach can yield comparable or better phoneme recognition performance when compared to the conventional approach. It indicates that CNNs can learn features relevant for phoneme classification automatically from the raw speech signal. The authors present a time-delay neural network (TDNN) approach to phoneme recognition which is characterized by two important properties: (1) using a three-layer arrangement of simple computing units, a hierarchy can be constructed that allows for the formation of arbitrary nonlinear decision surfaces, which the TDNN learns automatically using error backpropagation; and (2) the time-delay arrangement enables the network to discover acoustic-phonetic features and the temporal relationships between them independently of position in time and therefore not blurred by temporal shifts in the input. As a recognition task, the speaker-dependent recognition of the phonemes B, D, and G in varying phonetic contexts was chosen. For comparison, several discrete hidden Markov models (HMM) were trained to perform the same task. Performance evaluation over 1946 testing tokens from three speakers showed that the TDNN achieves a recognition rate of 98.5 correct while the rate obtained by the best of the HMMs was only 93.7 . > Abstract Automaticspeechrecognitionsystemstypicallymodeltherela-tionship between the acoustic speech signal and the phones intwo separate steps: feature extraction and classier training. Inourrecentworks, wehaveshownthat, intheframeworkofcon-volutionalneuralnetworks(CNN),therelationshipbetweentheraw speech signal and the phones can be directly modeled andASR systems competitive to standard approach can be built. Inthis paper, we rst analyze and show that, between the rst twoconvolutional layers, the CNN learns (in parts) and models thephone-specic spectral envelope information of 2-4 ms speech.Given that we show that the CNN-based approach yields ASRtrends similar to standard short-term spectral based ASR sys-tem under mismatched (noisy) conditions, with the CNN-basedapproach being more robust.Index Terms: automatic speech recognition, convolutionalneural networks, raw signal, robust speech recognition. 1. Introduction State-of-the-art automatic speech recognition (ASR) systemstypically model the relationship between the acoustic speechsignal and the phones in two separate steps, which are op-timized in an independent manner [1]. In a rst step, thespeech signal is transformed into features, usually composed ofa dimensionality reduction phase and an information selectionphase, based on the task-specic knowledge of the phenomena.These two phases have been carefully hand-crafted, leading tostate-of-the-art features such as Mel frequency cepstral coef-cients(MFCCs)orperceptuallinearpredictioncepstralfeatures(PLPs). In a second step, the likelihood of subword units suchas, phonemes is estimated using generative models or discrimi-native models.In recent years, in the hybrid HMM ANN framework [1],there has been growing interests in using intermediate rep-resentations instead of conventional features, such as cepstral-based features, as input for neural networks-based systems.ANNs with deep learning architectures, more precisely, deepneural networks (DNNs) [2, 3], which can yield better systemthan a single hidden layer MLP have been proposed to addressvarious aspects of acoustic modeling. More specically, useof context-dependent phonemes [4, 5]; use of spectral featuresas opposed to cepstral features [6, 7]; CNN-based system withMel lter bank energies as input [8, 9, 10]; combination of dif-ferent features [11], to name a few. Features learning from therawspeechsignalusingneuralnetworks-basedsystemshasalsobeen investigated in [12]. In all these approaches, the features Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.