Automated text summarisation and evidence-based medicine: A survey of two domains <s> Extrinsic and Intrinsic Evaluation <s> The optimal amount of information needed in a given decision-making situation lies somewhere along a continuum from "not enough" to "too much". Ackoff proposed that information systems often hinder the decision-making process by creating information overload. To deal with this problem, he called for systems that could filter and condense data so that only relevant information reached the decision maker. The potential for information overload is especially critical in text-based information. The purpose of this research is to investigate the effects and theoretical limitations of extract condensing as a text processing tool in terms of recipient performance. In the experiment described here, an environment is created in which the effects of text condensing are isolated from the effects of message and individual recipient differences. The data show no difference in reading comprehension performance between the condensed forms and the original document. This indicates that condensed forms can be produced that are equally as informative as the original document. These results suggest that it is possible to apply a relatively simple computer algorithm to text and produce extracts that capture enough of the information contained in the original document so that the recipient can perform as if he or she had read the original. These results also identify a methodology for assessing the effectiveness of text condensing schemes. The research presented here contributes to a small but growing body of work on text-based information systems and, specifically, text condensing. <s> BIB001 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Extrinsic and Intrinsic Evaluation <s> In this paper we discuss manual and automatic evaluations of summaries using data from the Document Understanding Conference 2001 (DUC-2001). We first show the instability of the manual evaluation. Specifically, the low inter-human agreement indicates that more reference summaries are needed. To investigate the feasibility of automated summary evaluation based on the recent BLEU method from machine translation, we use accumulative n-gram overlap scores between system and human summaries. The initial results provide encouraging correlations with human judgments, based on the Spearman rank-order correlation coefficient. However, relative ranking of systems needs to take into account the instability. <s> BIB002 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Extrinsic and Intrinsic Evaluation <s> We describe our work in progress on natural language analysis in medical question-answering in the context of a broader medical text-retrieval project. We analyze the limitations in the medical domain of the technologies that have been developed for general question-answering systems, and describe an alternative approach whose organizing principle is the identification of semantic roles in both question and answer texts that correspond to the fields of PICO format. <s> BIB003 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Extrinsic and Intrinsic Evaluation <s> Knowing the polarity of clinical outcomes is important in answering questions posed by clinicians in patient treatment. We treat analysis of this information as a classification problem. Natural language processing and machine learning techniques are applied to detect four possibilities in medical text: no outcome, positive outcome, negative outcome, and neutral outcome. A supervised learning method is used to perform the classification at the sentence level. Five feature sets are constructed: unigrams, bigrams, change phrases, negations, and categories. The performance of different combinations of feature sets is compared. The results show that generalization using the category information in the domain knowledge base Unified Medical Language System is effective in the task. The effect of context information is significant. Combining linguistic features and domain knowledge leads to the highest accuracy. <s> BIB004 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Extrinsic and Intrinsic Evaluation <s> Multiple pieces of text describing various pieces of evidence in clinical trials are often needed in answering a clinical question. We explore a multi-document summarization approach to automatically find this information for questions about effects of using a medication to treat a disease. Sentences in relevant documents are ranked according to various features by a machine-learning approach. Those with higher scores are more important and will be included in the summary. The presence of clinical outcomes and their polarity are incorporated into the approach as features for determining importance of sentences, and the effectiveness of this is investigated, along with that of other textual features. The results show that information on clinical outcomes improves the performance of summarization. <s> BIB005 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Extrinsic and Intrinsic Evaluation <s> We propose an approach to automatic clinical question answering based on UMLS relations. Rules are defined to identify the medical concepts and their relations in the questions and documents with the help of Metamap transfer and SemRep. The phrase-level answers are generated through matching concepts and relations between question and documents. Experiment shows that our approach obtains high precision and recall. <s> BIB006
Techniques for automatic summary evaluation can be divided into two broad classes-extrinsic and intrinsic. Extrinsic evaluation techniques focus on the purpose of summaries. The objective is to measure the usefulness of a summary for a specific task. The single feature that plays the most important role in determining the usefulness of a summary is its content, and numerous evaluation techniques use this feature as an indication of the qualities of summaries. Recall, precision, F-score, coverage and other similar measures BIB002 have been frequently used in the past as simple evaluations of content relevance (e.g., BIB006 BIB003  BIB004 BIB005 ). More sophisticated techniques have also been applied for extrinsic evaluation. An example is the assessment of the answerability of questions, which can be answered from original texts, from summarised texts BIB001 ;  ;  . These evaluation techniques have been employed in open domain QA, but have not been explored for the medical domain until 41 The system does not perform information retrieval. Intrinsic methods concentrate on the summary itself, trying to measure features such as coherence, cohesion, grammaticality, readability and other important features. Intrinsic methods for extractive summaries assess features such as discourse well-formedness, while those for non-extractive approaches must also assess sentence well-formedness. Although in some domains, such as news, intrinsic evaluation plays an important role, for a query-focused summarisation system for EBM, we believe it to be much less important. The reasons are discussed later in the section.