Automated text summarisation and evidence-based medicine: A survey of two domains <s> Manual Evaluation <s> In this paper we discuss manual and automatic evaluations of summaries using data from the Document Understanding Conference 2001 (DUC-2001). We first show the instability of the manual evaluation. Specifically, the low inter-human agreement indicates that more reference summaries are needed. To investigate the feasibility of automated summary evaluation based on the recent BLEU method from machine translation, we use accumulative n-gram overlap scores between system and human summaries. The initial results provide encouraging correlations with human judgments, based on the Spearman rank-order correlation coefficient. However, relative ranking of systems needs to take into account the instability. <s> BIB001 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Manual Evaluation <s> The combination of recent developments in question-answering research and the availability of unparalleled resources developed specifically for automatic semantic processing of text in the medical domain provides a unique opportunity to explore complex question answering in the domain of clinical medicine. This article presents a system designed to satisfy the information needs of physicians practicing evidence-based medicine. We have developed a series of knowledge extractors, which employ a combination of knowledge-based and statistical techniques, for automatically identifying clinically relevant aspects of MEDLINE abstracts. These extracted elements serve as the input to an algorithm that scores the relevance of citations with respect to structured representations of information needs, in accordance with the principles of evidence-based medicine. Starting with an initial list of citations retrieved by PubMed, our system can bring relevant abstracts into higher ranking positions, and from these abstracts generate responses that directly answer physicians' questions. We describe three separate evaluations: one focused on the accuracy of the knowledge extractors, one conceptualized as a document reranking task, and finally, an evaluation of answers by two physicians. Experiments on a collection of real-world clinical questions show that our approach significantly outperforms the already competitive PubMed baseline. <s> BIB002
Due to the difficulty associated with evaluating automatic summaries, manual evaluation is still a common practice. There is more confidence in manual evaluation (compared to automatic evaluation) since humans can infer, paraphrase and use world knowledge to relate to text units with similar meanings but worded differently. In such evaluations, domain experts (often several for a single summary) read and grade summaries, usually using some chosen scale. Most of the systems mentioned in the last subsection of the previous section have undergone some form of manual evaluation, or at least involved human experts for the preparation of a gold standard (e.g., BIB002 ). However, agreement among human summarisers is generally quite low, and the process of manual evaluation is quite expensive. Human judgements have been shown to be unstable and inconsistent as well BIB001 . As a result, alternative automatic evaluations having high correlation with human scores are usually used.  present the pyramid approach for the manual evaluation of summaries. Summary content is divided into summarisation content units (SCU), and SCUs representing the same semantic information are manually annotated in each source document. Once annotation is complete, all the SCUs are assigned weights based on the number of summaries each of them appear in. Next, the SCUs are partitioned into a pyramid in which each tier contains SCUs of the same weight and higher tiers contain SCUs of higher score (i.e., SCUs appearing in more human summaries). Summaries that contain more top-tier SCUs are ranked higher than those that contain lower tier SCUs. An optimal summary consists of all the top-tier SCUs combined, within the summary length limits. The score for an automatically generated summary is calculated by summing the weights the constituent SCUs, and dividing this number by the sum of the SCUs of the corresponding optimal summary. An earlier approach presented by  is similar: atomic semantic units, called factoids, are used to represent the meanings of sentences. The approach requires the generation of a large number of summarised articles from which the gold standard can be obtained by identifying the most frequently occurring factoids. As an example, the authors show that to generate a news article summary of a 100 words (from 50 sample summaries), all factoids appearing in at least 30% of the summaries had to be included. Hence, gold standards of different lengths can be generated by varying the factoid threshold. Although this approach ensures very high agreement among the humans, its requirement of a large number of sample summaries makes it quite an expensive approach.