Automated text summarisation and evidence-based medicine: A survey of two domains <s> Automatic Evaluation <s> In this paper we discuss manual and automatic evaluations of summaries using data from the Document Understanding Conference 2001 (DUC-2001). We first show the instability of the manual evaluation. Specifically, the low inter-human agreement indicates that more reference summaries are needed. To investigate the feasibility of automated summary evaluation based on the recent BLEU method from machine translation, we use accumulative n-gram overlap scores between system and human summaries. The initial results provide encouraging correlations with human judgments, based on the Spearman rank-order correlation coefficient. However, relative ranking of systems needs to take into account the instability. <s> BIB001 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Automatic Evaluation <s> Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results. <s> BIB002 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Automatic Evaluation <s> ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes measures to automatically determine the quality of a summary by comparing it to other (ideal) summaries created by humans. The measures count the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans. This paper introduces four different ROUGE measures: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S included in the ROUGE summarization evaluation package and their evaluations. Three of them have been used in the Document Understanding Conference (DUC) 2004, a large-scale summarization evaluation sponsored by NIST. <s> BIB003 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Automatic Evaluation <s> ParaEval is an automated evaluation method for comparing reference and peer summaries. It facilitates a tiered-comparison strategy where recall-oriented global optimal and local greedy searches for paraphrase matching are enabled in the top tiers. We utilize a domain-independent paraphrase table extracted from a large bilingual parallel corpus using methods from Machine Translation (MT). We show that the quality of ParaEval's evaluations, measured by correlating with human judgments, closely resembles that of ROUGE's. <s> BIB004 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Automatic Evaluation <s> We present a fully automatic approach for summarization evaluation that does not require the creation of human model summaries. Our work capitalizes on the fact that a summary contains the most representative information from the input and so it is reasonable to expect that the distribution of terms in the input and a good summary are similar to each other. To compare the term distributions, we use KL and Jensen-Shannon divergence, cosine similarity, as well as unigram and multinomial models of text. Our results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons can be very effective. They can be used to rank participating systems very similarly to manual model-based evaluations (pyramid evaluation) as well as to manual human judgments of summary quality without reference to a model. Our best feature, Jensen-Shannon divergence, leads to a correlation as high as 0.9 with manual evaluations. <s> BIB005 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Automatic Evaluation <s> In this paper we propose a novel information-theoretic metric for automatic summary evaluation when model summaries are available as in the setting of the AESOP task of the Update Summarization track of the Text Analysis Conference (TAC). The metric is based on the concept of information content operationalized by using a taxonomy. Hereby, we present and discuss the results obtained at TAC 2009. <s> BIB006
5.4.1 ROUGE. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) BIB002 BIB003 ] is one of the most widely used tools for automatic summary evaluation, and consists of a number of metrics that evaluate summaries based on distinct criteria. The various ROUGE measures attempt to compute similarities between system generated summaries and gold standard summaries at a lexical level. One of the metrics, for example, called ROUGE-N, is an n-gram based recall-oriented measure, and is calculated using the following formula: ROU GE − N (s) = r∈R < φ n (r), φ n (s) > r∈R < φ n (r), φ n (r) > where R = {r 1 , ..., r m } denote the group of gold-standard summaries, s denotes an automated system summary, and φ n (d) denotes a binary vector contained in a document d BIB002 BIB003 . This metric, therefore, simply attempts to measure the extent to which an automatically generated summary contains the same information as the reference summary. Other ROUGE metrics apply different techniques with the same primary intent. For example, ROUGE-L attempts to find the longest 5.4.2 BLEU. BLEU  was originally designed for evaluating machine translation, and has been shown to be promising for automatic summary evaluation as well BIB001 . In this technique, automatically computed accumulative n-gram matching scores (NAMS) between a model unit (MU) and a system summary (S) are used as performance indicators of the system  . A number of combinations of n-grams are used to compute NAMS, and the technique is shown to have satisfactory correlation with human scores. 5.4.3 Information-theoretic Evaluation. introduce an information-theoretic approach to the automatic evaluation of summaries based on the divergence of distribution of terms between an automatic summary and model summaries. For a set of documents D, the authors assume that there exists a probabilistic distribution with parameters specified by θ R that generates reference summaries from D, and the task of summarisation is to estimate θ R . Similarly, the authors assume that every system summary is generated from a probabilistic distribution with parameters specified by θ A . The process of summary evaluation then becomes the task of estimating the distance between θ R and θ A . The authors present a number of variants of divergence measures (e.g., Jensen-Shannon divergence (JS), Kullback-Leibler divergence (KL)) for this and show that this technique is comparable to ROUGE for the evaluation of single-document summarisation, and better than ROUGE in evaluating multi-document summarisation systems. BIB005 and BIB006 present similar information theoretic evaluation approaches, which incorporate other divergence measures and also attempt to capture complex phoenomena such as synonymy. 5.4.4 ParaEval. The motivation behind this evaluation technique is the lack of semantic matching of content in automatic evaluation. The authors of this evaluation technique BIB004 explain that an essential part of semantic matching involves paraphrase matching and this evaluation system attempts to perform that. ParaEval applies a three-level comparison strategy. At the top level, an optimal search via dynamic programming to find multi-word to multi-word paraphrase matches between generated and reference summaries is used. In the second level, a greedy algorithm is used to find single-word paraphrase matches among non-matching fragments in the first level. Finally, literal lexical uni-gram matching is performed on the remaining text at the third level. The authors show that the quality of ParaEval's evaluations closely resembles that of ROUGE. 