A Smorgasbord of Features for Automatic MT Evaluation

This document describes the approach by the NLP Group at the Technical University of Catalonia (UPC-LSI), for the shared task on Automatic Evaluation of Machine Translation at the ACL 2008 Third SMT Workshop.


Introduction
Our proposal is based on a rich set of individual metrics operating at different linguistic levels: lexical (i.e., on word forms), shallow-syntactic (e.g., on word lemmas, part-of-speech tags, and base phrase chunks), syntactic (e.g., on dependency and constituency trees), shallow-semantic (e.g., on named entities and semantic roles), and semantic (e.g., on discourse representations). Although from different viewpoints, and based on different similarity assumptions, in all cases, translation quality is measured by comparing automatic translations against human references. Extensive details on the metric set may be found in the IQMT technical manual .
Apart from individual metrics, we have also applied a simple integration scheme based on uniformly-averaged linear metric combinations (Giménez and Màrquez, 2008a).

What is new?
The main novelty, with respect to the set of metrics presented last year , is the incorporation of a novel family of metrics at the properly semantic level. DR metrics analyze similarities between automatic and reference translations by comparing their respective discourse representation structures (DRS), as provided by the the C&C Tools (Clark and Curran, 2004). DRS are essentially a variation of first-order predicate calculus which can be seen as semantic trees. We use three different kinds of metrics: DR-STM Semantic Tree Matching, a la Liu and Gildea (2005), but over DRS instead of over constituency trees.

DR-O rp -⋆ Morphosyntactic overlapping on DRS.
Further details on DR metrics can be found in (Giménez and Màrquez, 2008b).

Improved Sentence Level Behavior
Metrics based on deep linguistic analysis rely on automatic processors trained on out-domain data, which may be, thus, prone to error. Indeed, we found out that in many cases, metrics are unable to produce a result due to the lack of linguistic analysis. For instance, in our experiments, for SR metrics, we found that the semantic role labeler was unable to parse 14% of the sentences. In order to improve the recall of these metrics, we have designed two simple variants. Given a linguistic metric x, we define: • x b → by backing off to lexical overlapping, O l , only when the linguistic processor is not able to produce a linguistic analysis. Otherwise, x score is returned. Lexical scores are conveniently scaled so that they are in a similar range to scores of x. Specifically, we multiply them by the average x score attained over all other test cases for which the parser succeeded.
• x i → by linearly interpolating x and O l scores for all test cases, via the arithmetic mean.
In both cases, system scores are calculated by averaging over all sentence scores. Currently, these variants are applied only to SR and DR metrics.

Uniform Linear Metric Combinations
We have simulated a non-parametric combination scheme based on human acceptability by working on uniformly averaged linear combinations (ULC) of metrics (Giménez and Màrquez, 2008a). Our approach is similar to that of Liu and Gildea (2007) except that in our case the contribution of each metric to the overall score is not adjusted.
Optimal metric sets are determined by maximizing the correlation with human assessments, either at the document or sentence level. However, because exploring all possible combinations was not viable, we have used a simple algorithm which performs an approximate search. First, metrics are ranked according to their individual quality. Then, following that order, metrics are added to the optimal set only if in doing so the global quality increases.

Experimental Work
We use all into-English test beds from the 2006 and 2007 editions of the SMT workshop (Koehn and Monz, 2006;Callison-Burch et al., 2007). These include the translation of three different language-pairs: German-to-English (de-en), Spanish-to-English (es-en), and French-to-English (fr-en), over two different scenarios: in-domain (European Parliament Proceedings) and out-of-domain (News Commentary Corpus) 1 . In all cases, a single reference translation is available. In addition, human assessments on adequacy and fluency are available for a subset of systems and sentences. Each sentence has been evaluated at least by two different judges. A brief numerical description of these test beds is available in Table 1. Metrics are evaluated in terms of human acceptability, i.e., according to their ability to capture the degree of acceptability to humans of automatic translations. We measure human acceptability by computing Pearson correlation coefficients between automatic metric scores and human assessments of translation quality both at document and sentence level. We use the sum of adequacy and fluency to simulate a global assessment of quality. Assessments from different judges over the same test case are averaged into a single score.

Individual Performance
In first place, we study the behavior of individual metrics. Table 2 shows meta-evaluation results, over into-English WMT 2007 test beds, in-domain and out-of-domain, both at the system and sentence levels, for a set of selected representatives from several linguistic levels.
At the system level (columns 1-6), corroborating previous findings by , highest levels of correlation are attained by metrics based on deep linguistic analysis (either syntactic or semantic). In particular, two kinds of metrics, respectively based on head-word chain matching over grammatical categories and relations ('DP-  Table 2: Meta-evaluation results based on human acceptability for the WMT 2007 into-English translation tasks HWC c -4', 'DP-HWC r -4'), and morphosyntactic overlapping over discourse representations ('DR-O rp -⋆'), are consistently among the top-scoring in all test beds. At the lexical level, variants of ROUGE and METEOR attain the best results, close to the performance of syntactic and semantic features. It can also be observed that metrics based on semantic roles and named entities have serious troubles with the German-to-English in-domain test bed (column 1). At the sentence level, the highest levels of correlation are attained by metrics based on lexical similarity alone, only rivaled by lexical overlapping over dependency relations ('DP-O r -⋆') and discourse rep-resentations ('DR-O r -⋆'). We speculate the underlying cause might be on the side of parsing errors. In that respect, lexical back-off strategies report in all cases a significant improvement.
It can also be observed that, over these test beds, metrics based on named entities are completely useless at the sentence level, at least in isolation. The reason is that they capture a very partial aspect of quality which may be not relevant in many cases. This has been verified by computing the 'NE-O e -⋆⋆' variant which considers also lexical overlapping over regular items. Observe how this metric attains a much higher correlation with human assessments.

Metric Combinations
We also study the behavior of metric combinations under the ULC scheme. Last 5 rows in Table 2 shows meta-evaluation results following 3 different optimization strategies: Optimal: the metric set is optimized for each test bed (language-pair and domain) individually.
Optimal⋆: the metric set is optimized over the union of all test beds.
Optimal h : the metric set is heuristically defined so as to include several of the top-scoring representatives from each level: We present results optimizing over the 2006 and 2007 data sets. Let us provide, as an illustration, Optimal⋆ 07 sets. For instance, at the system level, no combination improved the isolated global performance of the 'DR-O rp -⋆ b ' metric (R=0.94). In contrast, at the sentence level, the optimal metric set contains several metrics from each linguistic level: A similar pattern is observed for all test beds, both at the system and sentence levels, although with different metrics.
The behavior of optimal metric sets is in general quite stable, except for the German-to-English indomain test bed which presents an anomalous behavior when meta-evaluating WMT 2006 optimal metric sets at the system level. The reason for this anomaly is in the 'NE-M e -⋆' metric, which is included in the 2006 optimal set: { 'NE-M e -⋆', 'SR-O ri ' }. 'NE-M e -⋆' is based on lexical matching over named entities, and attains in the 2006 German-to-English in-domain test bed a very high correlation of 0.95 with human assessments. This partial aspect of quality seems to be of marginal importance in the 2007 test bed. We have verified this hypothesis by computing optimal metrics sets without considering NE variants. Correlation increases to more reasonable values (e.g., from 0.01 to 0.66 and from 0.34 to 0.91. This result suggests that more robust metric combination schemes should be pursued. For future work, we plan to apply parametric combination schemes based on human likeness classifiers, as suggested by Kulesza and Shieber (2004). We must also further investigate the impact of parsing errors on the performance of linguistic metrics.