We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during training. We apply a slightly modified sequence-to-sequence with attention architecture that has previously been used for speech recognition and show that it can be repurposed for this more complex task, illustrating the power of attention-based models. A single model trained end-to-end obtains state-of-the-art performance on the Fisher Callhome Spanish-English speech translation task, outperforming a cascade of independently trained sequence-to-sequence speech recognition and machine translation models by 1.8 BLEU points on the Fisher test set. In addition, we find that making use of the training data in both languages by multi-task training sequence-to-sequence speech translation and recognition models with a shared encoder network can improve performance by a further 1.4 BLEU points.
Early work on (ST) @cite_21 -- translating audio in one language into text in another -- used lattices from an ASR system as inputs to translation models @cite_29 @cite_20 , giving the translation model access to the speech recognition uncertainty. Alternative approaches explicitly integrated acoustic and translation models using a stochastic finite-state transducer which can decode the translated text directly using Viterbi search @cite_24 @cite_13 .
@cite_29 @cite_21 @cite_24 @cite_13 @cite_20
2113106066 2136530135 2139647714 1537859740
In speech translation, we are faced with the problem of how to couple the speech recognition process and the translation process. Starting from the Bayes decision rule for speech translation, we analyze how the interaction between the recognition process and the translation process can be modelled. In the light of this decision rule, we discuss the already existing approaches to speech translation. None of the existing approaches seems to have addressed this direct interaction. We suggest two new methods, the local averaging approximation and the monotone alignments. Spoken language translation (SLT) is of great relevance in our increasingly globalized world, both from a social and economic point of view. It is one of the major challenges in automatic speech recognition (ASR) and machine translation (MT), driving an intense research activity in these areas. Speech translation is useful to assist person-to-person communication in limited domains like tourism and traveling and to translate foreign parliamentary speeches and broadcast news. Speech translation is based on a suitable combination of two independent technologies, namely ASR and MT of written language. Thus, the important question is how to pass on the ASR ambiguities to the MT process. A unifying framework for this ASR-MT interface is provided by applying the Bayes decision rule to the speech translation tasks as whole rather than to each task individually. Depending on the MT approaches used, such as finite-state transducers or phrase-based modeling, various types of ASR-MT interfaces have been studied, ranging from N-best lists through word lattices to confusion networks. We have discussed experimental results on various tasks, ranging from limited to unrestricted domains. Despite the significant advances and the large number of experimental studies, it is still an open question what type of interface provides a suitable compromise between translation accuracy and computational cost. A fully integrated approach to speech input language translation in limited domain applications is presented. The mapping from the input to the output language is modeled in terms of a finite state translation model which is learned from examples of input output sentences of the task considered. This model is tightly integrated with standard acoustic phonetic models of the input language and the resulting global model directly supplies, through Viterbi search, an optimal output language sentence for each input language utterance. Several extensions to this framework, recently developed to cope with the increasing difficulty of translation tasks, are reviewed. Finally, results for a task in the framework of hotel front desk communication, with a vocabulary of about 700 words, are reported. This paper focuses on the interface between speech recognition and machine translation in a speech translation system. Based on a thorough theoretical framework, we exploit word lattices of automatic speech recognition hypotheses as input to our translation system which is based on weighted finite-state transducers. We show that acoustic recognition scores of the recognized words in the lattices positively and significantly affect the translation quality. In experiments, we have found consistent improvements on three different corpora compared with translations of single best recognized results. In addition we build and evaluate a fully integrated speech translation model.