Neural Networks trained with gradient descent are known to be susceptible to catastrophic forgetting caused by parameter shift during the training process. In the context of Neural Machine Translation (NMT) this results in poor performance on heterogeneous datasets and on sub-tasks like rare phrase translation. On the other hand, non-parametric approaches are immune to forgetting, perfectly complementing the generalization ability of NMT. However, attempts to combine non-parametric or retrieval based approaches with NMT have only been successful on narrow domains, possibly due to over-reliance on sentence level retrieval. We propose a novel n-gram level retrieval approach that relies on local phrase level similarities, allowing us to retrieve neighbors that are useful for translation even when overall sentence similarity is low. We complement this with an expressive neural network, allowing our model to extract information from the noisy retrieved context. We evaluate our semi-parametric NMT approach on a heterogeneous dataset composed of WMT, IWSLT, JRC-Acquis and OpenSubtitles, and demonstrate gains on all 4 evaluation sets. The semi-parametric nature of our approach opens the door for non-parametric domain adaptation, demonstrating strong inference-time adaptation performance on new domains without the need for any parameter updates.
Tools incorporating information from individual translation pairs, or translation memories @cite_30 @cite_36 , have been widely utilized by human translators in the industry. There have been a few efforts attempting to combine non-parametric methods with NMT @cite_39 @cite_12 @cite_32 , but the key difference of our approach is the introduction of local, sub-sentence level similarity in the retrieval process, via n-gram level retrieval. Combined with our architectural improvements, motivated by the target encoder and gated attention from @cite_32 and the extended transformer model from @cite_46 , our semi-parametric NMT model is able to out-perform purely neural models in broad multi-domain settings.
@cite_30 @cite_36 @cite_32 @cite_39 @cite_46 @cite_12
2187442798 2276085145 2891713103 2618463334 2962712961 2795933031
Translation Memory (TM) systems have been under the spotlight of translation technology research led by both software developers and academic institutions. Both ends try to find ways to maximize the benefits deriving from the use of these tools, whether those translate into productivity enhancements or cost savings. The involvement of the user in these efforts has always been problematic. It is usually too costly, it delays the development of the product because it takes time, and it requires a well designed mechanism to be in place that facilitates the communication between the user and the developer. Naturally, many developers cannot afford to set up such capability, thus they risk producing TM tools that fail to correspond to the needs of translation professionals. The Translation Memories Survey 2006 (abbr. TM Survey 2006), reported in this paper, was initiated with a view to acting as this very channel of information deriving from users (or potential users) of TM systems. The main purpose behind it is to present the users perspective about TM systems and to supply data on the application domain, that is, information on the procedural aspects of the translation activity, on frequent work practices and on the tasks related to TM systems. It reports on the factors that affect TM use and offers an evaluation of the most commonly used systems according to functional and non-functional criteria. The results also reveal a range of future directions in TM research as those are envisioned by translation professionals. Commercial Translation Memory systems (TM) have been available on the market for over two decades now. They have become the major language technology to support the translation and localization industries. The following paper will provide an overview of the state of the art in TM technology, explaining the major concepts and looking at recent trends in both commercial systems and research. The paper will start with a short overview of the history of TM systems and a description of their main components and types. It will then discuss the relation between TM and machine translation (MT) as well as ways of integrating the two types of translation technologies. After taking a closer look at data exchange standards relevant to TM environments the focus of the paper then shift towards approaches to enhance the retrieval performance of TM systems looking at both non-linguistic and linguistic approaches. In this paper, we extend an attention-based neural machine translation (NMT) model by allowing it to access an entire training set of parallel sentence pairs even after training. The proposed approach consists of two stages. In the first stage--retrieval stage--, an off-the-shelf, black-box search engine is used to retrieve a small subset of sentence pairs from a training set given a source sentence. These pairs are further filtered based on a fuzzy matching score based on edit distance. In the second stage--translation stage--, a novel translation model, called translation memory enhanced NMT (TM-NMT), seamlessly uses both the source sentence and a set of retrieved sentence pairs to perform the translation. Empirical evaluation on three language pairs (En-Fr, En-De, and En-Es) shows that the proposed approach significantly outperforms the baseline approach and the improvement is more significant when more relevant sentence pairs were retrieved. One of the difficulties of neural machine translation (NMT) is the recall and appropriate translation of low-frequency words or phrases. In this paper, we propose a simple, fast, and effective method for recalling previously seen translation examples and incorporating them into the NMT decoding process. Specifically, for an input sentence, we use a search engine to retrieve sentence pairs whose source sides are similar with the input sentence, and then collect @math -grams that are both in the retrieved target sentences and aligned with words that match in the source sentences, which we call "translation pieces". We compute pseudo-probabilities for each retrieved sentence based on similarities between the input sentence and the retrieved source sentences, and use these to weight the retrieved translation pieces. Finally, an existing NMT model is used to translate the input sentence, with an additional bonus given to outputs that contain the collected translation pieces. We show our method improves NMT translation results up to 6 BLEU points on three narrow domain translation tasks where repetitiveness of the target sentences is particularly salient. It also causes little increase in the translation time, and compares favorably to another alternative retrieval-based method with respect to accuracy, speed, and simplicity of implementation.