Automated text summarisation and evidence-based medicine: A survey of two domains <s> Discussion of Evaluation Techniques for EBM <s> As the number of electronic biomedical textual resources increases, it becomes harder for physicians to find useful answers at the point of care. Information retrieval applications provide access to databases; however, little research has been done on using automatic summarization to help navigate the documents returned by these systems. After presenting a semantic abstraction automatic summarization system for MEDLINE citations, we concentrate on evaluating its ability to identify useful drug interventions for 53 diseases. The evaluation methodology uses existing sources of evidence-based medicine as surrogates for a physician-annotated reference standard. Mean average precision (MAP) and a clinical usefulness score developed for this study were computed as performance metrics. The automatic summarization system significantly outperformed the baseline in both metrics. The MAP gain was 0.17 (p<0.01) and the increase in the overall score of clinical usefulness was 0.39 (p<0.05). <s> BIB001 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> Discussion of Evaluation Techniques for EBM <s> Display Omitted We present an approach for query-focused extractive summarisation of medical text.Our approach is data-centric and utilises a specialised annotated corpus.We introduce target-sentence-specific summarisation to improve summarisation.We incorporate query information and domain-specific knowledge in various ways.Our approach performs significantly better than existing benchmark approaches. BackgroundEvidence-based medicine practice requires medical practitioners to rely on the best available evidence, in addition to their expertise, when making clinical decisions. The medical domain boasts a large amount of published medical research data, indexed in various medical databases such as MEDLINE. As the size of this data grows, practitioners increasingly face the problem of information overload, and past research has established the time-associated obstacles faced by evidence-based medicine practitioners. In this paper, we focus on the problem of automatic text summarisation to help practitioners quickly find query-focused information from relevant documents. MethodsWe utilise an annotated corpus that is specialised for the task of evidence-based summarisation of text. In contrast to past summarisation approaches, which mostly rely on surface level features to identify salient pieces of texts that form the summaries, our approach focuses on the use of corpus-based statistics, and domain-specific lexical knowledge for the identification of summary contents. We also apply a target-sentence-specific summarisation technique that reduces the problem of underfitting that persists in generic summarisation models. ResultsIn automatic evaluations run over a large number of annotated summaries, our extractive summarisation technique statistically outperforms various baseline and benchmark summarisation models with a percentile rank of 96.8%. A manual evaluation shows that our extractive summarisation approach is capable of selecting content with high recall and precision, and may thus be used to generate bottom-line answers to practitioners' queries. ConclusionsOur research shows that the incorporation of specialised data and domain-specific knowledge can significantly improve text summarisation performance in the medical domain. Due to the vast amounts of medical text available, and the high growth of this form of data, we suspect that such summarisation techniques will address the time-related obstacles associated with evidence-based medicine. <s> BIB002
If summarising for EBM is a hard task, evaluating summaries is even harder. A summarisation system for EBM should be capable of extracting evidence from medical articles and additionally assess the grade of the evidence. The evaluation should be able to determine if the evidence is correctly extracted and also if the extracted information correctly answers the practitioner's query. Therefore, the single most important aspect of the summary is its content and a strong focus on extrinsic evaluation is required. Intrinsic evaluation to assess aspects such as coherence and readability are perhaps not very important, and a stronger focus needs to be on the purpose of summaries  . For the automatic evaluation of summaries, approaches based on n-gram co-occurrence such as ROUGE are the most frequently used. Although these approaches are very robust and efficient, their main drawback is that if two summaries were produced using non-or almost non-overlapping vocabulary, yet conveying the same information, the similarity score such summaries would be assigned by purely n-gram based metrics would be too low and, hence, unrepresentative of the actual information they share. This is definitely not desirable for evaluation, particularly in the medical domain where relations such as synonymy and hyponymy play important roles. Furthermore, making relative comparisons between different summarisers is difficult using automatic evaluation approaches such as ROUGE. Thus, approaches that incorporate domain knowledge, semantic similarities, and comparisons between summarisation systems are required. Research on the evaluation of summarisers for EBM is still very much in its infancy. However, recent works in this area such as that of BIB001 and BIB002 have made useful contributions by ensuring that evaluation techniques assess the performance of the summariser in the light of its goals.