We apply sequence-to-sequence model to mitigate the impact of speech recognition errors on open domain end-to-end dialog generation. We cast the task as a domain adaptation problem where ASR transcriptions and original text are in two different domains. In this paper, our proposed model includes two individual encoders for each domain data and make their hidden states similar to ensure the decoder predict the same dialog text. The method shows that the sequence-to-sequence model can learn the ASR transcriptions and original text pair having the same meaning and eliminate the speech recognition errors. Experimental results on Cornell movie dialog dataset demonstrate that the domain adaption system help the spoken dialog system generate more similar responses with the original text answers.
Different approaches are explored to address such problem in distinct applications. @cite_7 harnessed speech translation task by jointly learning ASR and machine translation to optimize bilingual evaluation understudy scores @cite_11 directly. This method alleviates the issue that the best ASR parameters on minimizing the traditional word error rate will only lead to sub-optimal performance. The same idea works for spoken content retrieval tasks. By modifying ASR training systems @cite_3 , the performance of retrieval-based models will be improved. However, all of these methods required modifying ASR modules. In this paper, we focus on mitigating the ASR errors based on a given ASR system. Without modifying ASR modules, interactive error recovery is applied to deal with errors in speech-to-speech translation system. @cite_9 employed conditional random field (CRF) models to detect ASR errors and attempted to resolve them by eliciting user feedback. In abstractive headline generation task for spoken content, @cite_18 proposed a method about considering ASR errors as a probability distribution. The work applied an attentive RNN to incorporate ASR error parameters into the attention mechanism.
@cite_18 @cite_7 @cite_9 @cite_3 @cite_11
2952465655 399167303 1970987322 2949174760
We explore the problem of translating speech to text in low-resource scenarios where neither automatic speech recognition (ASR) nor machine translation (MT) are available, but we have training data in the form of audio paired with text translations. We present the first system for this problem applied to a realistic multi-speaker dataset, the CALLHOME Spanish-English speech translation corpus. Our approach uses unsupervised term discovery (UTD) to cluster repeated patterns in the audio, creating a pseudotext, which we pair with translations to create a parallel text and train a simple bag-of-words MT model. We identify the challenges faced by the system, finding that the difficulty of cross-speaker UTD results in low recall, but that our system is still able to correctly translate some content words in test data. We propose a novel technique for adapting text-based statistical machine translation to deal with input from automatic speech recognition in spoken language translation tasks. We simulate likely misrecognition errors using only a source language pronunciation dictionary and language model (i.e., without an acoustic model), and use these to augment the phrase table of a standard MT system. The augmented system can thus recover from recognition errors during decoding using synthesized phrases. Using the outputs of five different English ASR systems as input, we find consistent and significant improvements in translation quality. Our proposed technique can also be used in conjunction with lattices as ASR output, leading to further improvements. We report insights from translating Spanish conversational telephone speech into English text by cascading an automatic speech recognition (ASR) system with a statistical machine translation (SMT) system. The key new insight is that the informal register of conversational speech is a greater challenge for ASR than for SMT: the BLEU score for translating the reference transcript is 64 , but drops to 32 for translating automatic transcripts, whose word error rate (WER) is 40 . Several strategies are examined to mitigate the impact of ASR errors on the SMT output: (i) providing the ASR lattice, instead of the 1-best output, as input to the SMT system, (ii) training the SMT system on Spanish ASR output paired with English text, instead of Spanish reference transcripts, and (iii) improving the core ASR system. Each leads to consistent and complementary improvements in the SMT output. Compared to translating the 1-best output of an ASR system with 40 WER using an SMT system trained on Spanish reference transcripts, translating the output lattice of a better ASR system with 35 WER using an SMT system trained on ASR output improves BLEU from 32 to 38 . Attention-based sequence-to-sequence models for speech recognition jointly train an acoustic model, language model (LM), and alignment mechanism using a single neural network and require only parallel audio-text pairs. Thus, the language model component of the end-to-end model is only trained on transcribed audio-text pairs, which leads to performance degradation especially on rare words. While there have been a variety of work that look at incorporating an external LM trained on text-only data into the end-to-end framework, none of them have taken into account the characteristic error distribution made by the model. In this paper, we propose a novel approach to utilizing text-only data, by training a spelling correction (SC) model to explicitly correct those errors. On the LibriSpeech dataset, we demonstrate that the proposed model results in an 18.6 relative improvement in WER over the baseline model when directly correcting top ASR hypothesis, and a 29.0 relative improvement when further rescoring an expanded n-best list using an external LM.