Automated text summarisation and evidence-based medicine: A survey of two domains <s> 5.2.2 <s> Abstract As electronic information access becomes the norm, and the variety of retrievable material increases, automatic methods of summarizing or condensing text will become critical. This paper describes a system that performs domain-independent automatic condensation of news from a large commercial news service encompassing 41 different publications. This system was evaluated against a system that condensed the same articles using only the first portion of the texts (the lead), up to the target length of the summaries. Three lengths of articles were evaluated for 250 documents by both systems, totalling 1500 suitability judgements in all. The outcome of perhaps the largest evaluation of human vs machine summarization performed to date was unexpected. The lead-based summaries outperformed the “intelligent” summaries significantly, achieving acceptability ratings of over 90%, compared to 74.4%. This paper briefly reviews the literature, details the implications of these results, and addresses the remaining hopes for content-based summarization. We expect the results presented here to be useful to other researchers currently investigating the viability of summarization through sentence selection heuristics. <s> BIB001 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> 5.2.2 <s> The combination of recent developments in question-answering research and the availability of unparalleled resources developed specifically for automatic semantic processing of text in the medical domain provides a unique opportunity to explore complex question answering in the domain of clinical medicine. This article presents a system designed to satisfy the information needs of physicians practicing evidence-based medicine. We have developed a series of knowledge extractors, which employ a combination of knowledge-based and statistical techniques, for automatically identifying clinically relevant aspects of MEDLINE abstracts. These extracted elements serve as the input to an algorithm that scores the relevance of citations with respect to structured representations of information needs, in accordance with the principles of evidence-based medicine. Starting with an initial list of citations retrieved by PubMed, our system can bring relevant abstracts into higher ranking positions, and from these abstracts generate responses that directly answer physicians' questions. We describe three separate evaluations: one focused on the accuracy of the knowledge extractors, one conceptualized as a document reranking task, and finally, an evaluation of answers by two physicians. Experiments on a collection of real-world clinical questions show that our approach significantly outperforms the already competitive PubMed baseline. <s> BIB002 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> 5.2.2 <s> As the number of electronic biomedical textual resources increases, it becomes harder for physicians to find useful answers at the point of care. Information retrieval applications provide access to databases; however, little research has been done on using automatic summarization to help navigate the documents returned by these systems. After presenting a semantic abstraction automatic summarization system for MEDLINE citations, we concentrate on evaluating its ability to identify useful drug interventions for 53 diseases. The evaluation methodology uses existing sources of evidence-based medicine as surrogates for a physician-annotated reference standard. Mean average precision (MAP) and a clinical usefulness score developed for this study were computed as performance metrics. The automatic summarization system significantly outperformed the baseline in both metrics. The MAP gain was 0.17 (p<0.01) and the increase in the overall score of clinical usefulness was 0.39 (p<0.05). <s> BIB003
Baselines. Baselines can be considered to be the opposite of gold standards in that they indicate the minimum level of required performance by a summarisation system. For extractive summarisation, various baselines such as n random sentences have been used. A more appropriate baseline for news summarisation was introduced by BIB001 who used the first n sentences. With these baselines, the minimum performance required by a system is to select n sentences that better summarise the source than the baseline. Similar baselines have been established for summarisation in various other domains. For example, BIB002 propose an outcome extractor that is compared against a baseline of 'last n sentences' (since outcomes presented in a medical paper usually appear towards the end of the abstract). Baseline measures of tf.idf type have also been used in the literature and even for EBM BIB003 ]. In such baselines, the summarisation units (sentences, words, n-grams) are chosen based on their tf.idf values. A standard baseline for summarisation systems across domains and even within specific domains, however, still does not exist.