We introduce the metric using BERT (Bidirectional Encoder Representations from Transformers) (, 2019) for automatic machine translation evaluation. The experimental results of the WMT-2017 Metrics Shared Task dataset show that our metric achieves state-of-the-art performance in segment-level metrics task for all to-English language pairs.
As shown in Figure , RUSE encodes an MT hypothesis and an reference translation by a sentence encoder, respectively. Then, following InferSent @cite_12 , a features are extracted by combining sentence embeddings of the two sentences, and the evaluation score is estimated by the regression model based on multi-layer perceptron (MLP).
@cite_12
2612953412
We investigate the problem of predicting the quality of sentences produced by machine translation systems when reference translations are not available. The problem is addressed as a regression task and a method that takes into account the contribution of different features is proposed. We experiment with this method for translations produced by various MT systems and different language pairs, annotated with quality scores both automatically and manually. Results show that our method allows obtaining good estimates and that identifying a reduced set of relevant features plays an important role. The experiments also highlight a number of outstanding features that were consistently selected as the most relevant and could be used in different ways to improve MT performance or to enhance MT evaluation.