Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese--English news translation task. We empirically test this claim with alternative evaluation protocols, contrasting the evaluation of single sentences and entire documents. In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences. Our findings emphasise the need to shift towards document-level evaluation as machine translation improves to the degree that errors which are hard or impossible to spot at the sentence-level become decisive in discriminating quality of different translation outputs. Neural machine translation (Kalchbrenner and Blunsom, 2013; BID10 BID0 has become the de-facto standard in machine translation, outperforming earlier phrasebased approaches in many data settings and shared translation tasks BID7 BID9 Cromieres et al., 2016) . Some recent results suggest that neural machine translation "approaches the accuracy achieved by average bilingual human translators [on some test sets]" (Wu et al., 2016) , or even that its "translation quality is at human parity when compared to professional human translators" (Hassan et al., 2018) . Claims of human parity in machine translation are certainly extraordinary, and require extraordinary evidence. 1 Laudably, Hassan et al. (2018) have released their data publicly to allow external validation of their claims. Their claims are further strengthened by the fact that they follow best practices in human machine translation evaluation, using evaluation protocols and tools that are also used at the yearly Conference on Machine Translation (WMT) BID2 , and take great care in guarding against some confounds such as test set selection and rater inconsistency.However, the implications of a statistical tie between two machine translation systems in a shared translation task are less severe than that of a statistical tie between a machine translation system and a professional human translator, so we consider the results worthy of further scrutiny. We perform an independent evaluation of the professional translation and best machine translation system that were found to be of equal quality by Hassan et al. (2018) . Our main interest lies in the evaluation protocol, and we empirically investigate if the lack of document-level context could explain the inability of human raters to find a quality difference between human and machine translations. We test the following hypothesis:A professional translator who is asked to rank the quality of two candidate translations on the document level will prefer a professional human translation over a machine translation.Note that our hypothesis is slightly different from that tested by Hassan et al. (2018) , which could be phrased as follows:A bilingual crowd worker who is asked to directly assess the quality of candidate translations on the sentence level will prefer a professional human translation over a machine translation.As such, our evaluation is not a direct replication of that by Hassan et al. (2018) , and a failure to reproduce their findings does not imply an error on either our or their part. Rather, we hope to indirectly assess the accuracy of different evaluation protocols. Our underlying assumption is that professional human translation is still superior to neural machine translation, but that the sensitivity of human raters to these quality differences depends on the evaluation protocol. Our results emphasise the need for suprasentential context in human evaluation of machine translation. Starting with Hassan et al.'s (2018) finding of no statistically significant difference in translation quality between HUMAN and MT for their Chinese-English test set, we set out to test this result with an alternative evaluation protocol which we expected to strengthen the ability of raters to judge translation quality. We employed professional translators instead of crowd workers, and pairwise ranking instead of direct assessment, but in a sentence-level evaluation of adequacy, raters still found it hard to discriminate between HUMAN and MT: they did not show a statistically significant preference for either of them.Conversely, we observe a tendency to rate HU-MAN more favourably on the document level than on the sentence level, even within single raters. Adequacy raters show a statistically significant preference for HUMAN when evaluating entire documents. We hypothesise that document-level evaluation unveils errors such as mistranslation of an ambiguous word, or errors related to textual cohesion and coherence, which remain hard or impossible to spot in a sentence-level evaluation. For a subset of articles, we elicited both sentence-level and document-level judgements, and inspected articles for which sentence-level judgements were mixed, but where HUMAN was strongly preferred in document-level evaluation. In these articles, we do indeed observe the hypothesised phenomena. We find an example of lexical coherence in a 6-sentence article about a new app "微信挪 车", which HUMAN consistently translates into "WeChat Move the Car". In MT, we find three different translations in the same article: "Twitter Move Car", "WeChat mobile", and "WeChat Move". Other observations include the use of more appropriate discourse connectives in HU-MAN, a more detailed investigation of which we leave to future work.To our surprise, fluency raters show a stronger preference for HUMAN than adequacy raters FIG0 . The main strength of neural machine translation in comparison to previous statistical approaches was found to be increased fluency, while adequacy improvements were less clear (Bojar et al., 2016b; Castilho et al., 2017b) , and we expected a similar pattern in our evaluation. Does this indicate that adequacy is in fact a strength of MT, not fluency? We are wary to jump to this conclusion. An alternative interpretation is that MT, which tends to be more literal than HUMAN, is judged more favourably by raters in the bilingual condition, where the majority of raters are native speakers of the source language, because of L1 interference. We note that the availability of document-level context still has a strong impact in the fluency condition (Section 3). In response to recent claims of parity between human and machine translation, we have empirically tested the impact of sentence and document level context on human assessment of machine translation. Raters showed a markedly stronger preference for human translations when evaluating at the level of documents, as compared to an evaluation of single, isolated sentences.We believe that our findings have several implications for machine translation research. Most importantly, if we accept our interpretation that human translation is indeed of higher quality in the dataset we tested, this points to a failure of current best practices in machine translation evaluation. As machine translation quality improves, translations will become harder to discriminate in terms of quality, and it may be time to shift towards document-level evaluation, which gives raters more context to understand the original text and its translation, and also exposes translation errors related to discourse phenomena which remain invisible in a sentence-level evaluation.Our evaluation protocol was designed with the aim of providing maximal validity, which is why we chose to use professional translators and pairwise ranking. For future work, it would be of high practical relevance to test whether we can also elicit accurate quality judgements on the document-level via crowdsourcing and direct assessment, or via alternative evaluation protocols. The data released by Hassan et al. (2018) could serve as a test bed to this end. One reason why document-level evaluation widens the quality gap between machine translation and human translation is that the machine translation system we tested still operates on the sentence level, ignoring wider context. It will be interesting to explore to what extent existing and future techniques for document-level machine translation can narrow this gap. We expect that this will require further efforts in creating document-level training data, designing appropriate models, and supporting research with discourse-aware automatic metrics. TAB1 shows detailed results, including those of individual raters, for all four experimental conditions. Raters choose between three labels for each item: MT is better than HUMAN (a), HUMAN is better than MT (b), or tie (t). TAB3 lists interrater agreement. Besides percent agreement (same label), we calculate Cohen's kappa coefficient DISPLAYFORM0 where P (A) is the proportion of times that two raters agree, and P (E) the likelihood of agreement by chance. We calculate Cohen's kappa, and specifically P (E), as in WMT (Bojar et al., 2016b, Section 3.3) , on the basis of all pairwise ratings across all raters.In pairwise rankings of machine translation outputs, κ coefficients typically centre around 0.3 (Bojar et al., 2016b) . We observe lower inter-rater agreement in three out of four conditions, and attribute this to two reasons. Firstly, the quality of the machine translations produced by Hassan et al. FORMULA0 is high, making it difficult to discriminate from professional translation particularly at the sentence level. Secondly, we do not provide guidelines detailing error severity and thus assume that raters have differing interpretations of what constitutes a "better" or "worse" translation. Confusion matrices in TAB4 indicate that raters handle ties very differently: in document-level adequacy, for example, rater E assigns no ties at all, while rater F rates 15 out of 50 items as ties (Table 4g). The assignment of ties is more uniform in documents assessed for fluency TAB1 , leading to higher κ in this condition TAB3 .Despite low inter-annotator agreement, the quality control we apply shows that raters assess items carefully: they only miss 1 out of 40 and 5 out of 128 spam items in the document-and sentence-level conditions overall, respectively, a very low number compared to crowdsourced work BID5 . All of these misses are ties (i. e., not marking spam items as "better", but rather equally bad as their counterpart), and 5 out of 9 raters (A, B1, B2, D, F) do not miss a single spam item.A common procedure in situations where interrater agreement is low is to aggregate ratings of different annotators BID2 . As shown in TAB2
Raters prefer adequacy in human over machine translation when evaluating entire documents, but not when evaluating single sentences.
{"src": "scitldr-AIC_train", "paper_id": "Hygfmc5U-7"}