Deep Learning has been applied successfully to Automatic
Speech Recognition (ASR) [1]}. A variety of neural network architectures for acoustic modeling have been explored. For example, DNNs [2]}, CNNs [3]}, RNNs [4]} and end-to-end models[5]}, [6]}, [7]}, [8]}.
Currently, there are mainly three E2E models: Neural Transducer (NT) [9]}, [4]}, [11]}, [12]} models, Attention-Based Encoder-Decoder (AED) models [6]}, [7]}, [15]} and Connectionist Temporal Classification [16]} (CTC) models [5]}, [18]}. These E2E models treat ASR as a sequence-to-sequence task that directly learns speech to text mapping with a neural network.
The NT model consists of an encoder, which maps input acoustic frames into a higher-level representation, and a prediction and joint network which together correspond to the decoder network[12]}. The decoder is conditioned on the history of previous predictions. The NT training is unstable and takes more memory which may limit the training speed.
The AED model is composed of an encoder, which encodes acoustic features, and a decoder, which generates a sentence. The architectures of many state-of-the-art ASR systems [20]} are based on the AED models. However, the AED model outputs token by token, where each token depends on previously generated tokens and acoustic context, causing recognition delays.
On the other hand, the CTC model contains only an encoder and outputs all tokens independently. Although its decoding speed is faster than the AED model, in term of recognition accuracy it is generally inferior due to the conditional independence assumption between output tokens. However, in real environment speech recognition tasks, the attention model performs poorly because the alignment estimated in the attention mechanism is easily corrupted by the noise.
