Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> Doctors within the NHS are confronting major changes at work. While we endeavour to improve the quality of health care, junior doctors' hours have been reduced and the emphasis on continuing medical education has increased. We are confronted by a growing body of information, much of it invalid or irrelevant to clinical practice. This article discusses evidence based medicine, a process of turning clinical problems into questions and then systematically locating, appraising, and using contemporaneous research findings as the basis for clinical decisions. The computerisation of bibliographies and the development of software that permits the rapid location of relevant evidence have made it easier for busy clinicians to make best use of the published literature. Critical appraisal can be used to determine the validity and applicability of the evidence, which is then used to inform clinical decisions. Evidence based medicine can be taught to, and practised by, clinicians at all levels of seniority and can be used to close the gulf between good clinical research and clinical practice. In addition it can help to promote self directed learning and teamwork and produce faster and better doctors. <s> BIB001 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> Primary care physicians have many questions about optimal care while they are seeing patients, but they pursue only about 30% of their questions. The authors designed a study to determine the factors that motivate physicians to pursue answers to some of their ques tions, while leaving the majority of their questions unanswered. They interviewed 49 non- academic primary care physicians during office hours to record clinical questions as they arose in the course of patient care. The physicians then recorded their perceptions of each question with respect to 12 factors expected to motivate information seeking. Two to five days after the interview, each physician was telephoned to determine which questions had been pursued. In a multiple logistic-regression model only two factors were significant pre dictors of pursuit of new information: the physician's belief that a definitive answer existed, and the urgency of the patient's problem. Other factors, including the difficulty of finding the answer, potential m... <s> BIB002 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> An apparatus for providing a signal in response to a force producing physical condition comprises a laterally flexible tubular enclosure having two opposite ends, and a housing for that tubular enclosure. Bothe ends Both the tubular enclosure are maintained relatively stationary and the above-mentioned force is picked up outside the tubular enclosure. The picked-up force is applied to the tubular enclosure between the relatively stationary ends and the tubular enclosure is laterally flexed with that applied force between its relatively stationary ends. A device for generating the above mentioned signal is located inside the tubular enclosure for generating that signal during the flexing of the tubular enclosure. <s> BIB003 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> IN RESPONSE: Dr. Castro and colleagues report the minimal overlap between journals indexed in LILACS and those indexed in MEDLINE. This suggests that the former may be a rich source of primary research as well as reviews. For persons who are concerned about comprehensive searching and have the skills and resources to use literature published in different languages, LILACS may be very valuable. The terminology issue raised by Dr. Atkins highlights confusion in the field: Both the test of homogeneity and the test of heterogeneity are used in the literature. We agree with Dr. Atkins's interpretation of this test. We disagree, however, that examination of study results should be the focus when one is deciding whether to statistically pool data. Before the test of homogeneity or heterogeneity is even considered, clinical judgment is required to evaluate study characteristics. One of the most important steps in a systematic review is assessment of the similarities or differences among studies, focusing on populations, interventions, and outcomes. At this point, on clinical grounds, reviewers may abandon the idea of statistically pooling study results in favor of a more descriptive and interpretive summary. If synthesizing study results does seem reasonable, the next step is testing whether the differences in results are greater than that which would be expected on the basis of chance alone. A significant positive test result can correlate with a clinically trivial degree of heterogeneity, and a negative test result does not rule out a clinically important degree of heterogeneity. Nevertheless, reviewers may decide not to statistically pool data if the test result is positive, or they may proceed to conduct sensitivity analyses or subgroup analyses (ideally on the basis of a priori issues identified as possible sources of variability). For further discussion about issues of homogeneity and heterogeneity, we refer readers to other articles in the Annals' systematic review series [1, 2]. <s> BIB004 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> This is the last in a series of five articles on narrative based medicine ::: ::: In a widely quoted riposte to critics who accused them of naive empiricism, Sackett and colleagues claimed that “the practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence By individual clinical expertise we mean the proficiency and judgment that individual clinicians acquire through clinical experience and clinical practice.”1 Sackett and colleagues were anxious to acknowledge that there is an art to medicine as well as an objective empirical science but they did not attempt to define or categorise the elusive quality of clinical competence. This article explores the dissonance between the “science” of objective measurement2 and the “art” of clinical proficiency and judgment,3–5 and attempts to integrate these different perspectives on clinical method. ::: ::: ### Summary points ::: ::: Even “evidence based” clinicians uphold the importance of clinical expertise and judgment ::: ::: Clinical method is an interpretive act which draws on narrative skills to integrate the overlapping stories told by patients, clinicians, and test results ::: ::: The art of selecting the most appropriate medical maxim for a particular clinical decision is acquired largely through the accumulation of “case expertise” (the stories or “illness scripts” of patients and clinical anecdotes) ::: ::: The dissonance we experience when trying to apply research findings to the clinical encounter often occurs when we abandon the narrative-interpretive paradigm and try to get by on “evidence” alone ::: ::: Science is concerned with the formulation and attempted falsification of hypotheses using reproducible methods that allow the construction of generalisable statements about how the universe behaves. Conventional medical training teaches students to view medicine as a science and the doctor as an impartial investigator who builds differential diagnoses as if they were scientific theories and who excludes competing possibilities in a manner … <s> BIB005 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> Extrinsic safety devices, such as trigger locks, have been central in the recent state discussions on how to reduce firearm injuries. The actual prevalence of handgun locking devices in the consumer market, however, is unknown. This study catalogued the extrinsic safety devices available from handgun dealers and discount retail chains in Milwaukee, WI. We found that all locations studied (n = 13) stock at least 1 type of extrinsic safety device. A total of 21 unique models of safety devices were stocked by the 13 locations, with trigger locks being the most common (n = 9). Other types of devices included lock-boxes (n = 5), cable locks (n = 4), hammer locks (n = 1), barrel locks (n = 1), and a rubber slide strap (n = 1). Handgun owners in the Milwaukee metro area have a selection of extrinsic handgun safety devices available from handgun dealers and discount retailers. However, there does not appear to be a consistent availability on type of device. <s> BIB006 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> Objectives: To explore whether structuring a literature search request form according to an evidence-based medicine (EBM) anatomy elicits more information, improves precision of search results, and is acceptable to participating librarians. ::: ::: Methods: Multicenter before-and-after study involved six different libraries. Data from 195 minimally structured forms collected over four months (Phase 1) were compared with data from 185 EBM-structured forms collected over a further four-month period following a brief training intervention (Phase 2). Survey of librarians' attitudes toward using the EBM-structured forms was conducted early during Phase 2. ::: ::: Results: 380 request forms, EBM-structured and minimally structured, were analyzed using SPSS. A statistically significant Pearson correlation was found between use of the EBM-structured form and complexity of the search strategy (P = 0.002). The correlation between clinical requests handled by the EBM-structured form and fewer items retrieved was also statistically significant (P = 0.028). However, librarians rated minimally structured forms more highly than EBM-structured forms against all dimensions except informativeness. ::: ::: Conclusions: Although use of the EBM-structured forms is associated with more precise searches and more detailed search strategies, considerable work remains on making these forms acceptable to both librarians and users. Nevertheless, with increased familiarity and improved training, information retrieval benefits could be translated into more effective search practice. <s> BIB007 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> CONTEXT ::: Systematic reviews of the literature are an important resource for clinicians. Unfortunately, the few published strategies for identifying these articles involve MEDLINE interfaces not widely available outside of academic medicine. In addition, the performance of these strategies is unknown. ::: ::: ::: OBJECTIVE ::: To develop and evaluate a search strategy for identifying systematic reviews by using a publicly available MEDLINE interface (PubMed). ::: ::: ::: DESIGN ::: Diagnostic test assessment. DEFINITION OF SENSITIVITY: The proportion of recognized systematic reviews (indexed in the Cochrane Library's Database of Abstracts of Reviews of Effectiveness [DARE] or in ACP Journal Club) that are identified by the search strategy. DEFINITION OF POSITIVE PREDICTIVE VALUE: The proportion of articles identified in one of three sample searches (screening for colorectal cancer, thrombolytic therapy for venous thromboembolism, and treatment of dementia) that meet a minimum definition of systematic review. ::: ::: ::: RESULTS ::: Our PubMed search strategy identified 93 of 100 DARE-indexed systematic reviews, a sensitivity of 93% (95% CI, 86% to 97%). For the sample of systematic reviews drawn from ACP Journal Club (n = 103), the PubMed strategy achieved a sensitivity of 97% (CI, 91% to 99%). When the three sample search strings were used, approximately 50% of retrieved articles met our minimum definition of systematic review. In contrast, the similar precision of a PubMed search restricted to review-type articles (as indexed by MEDLINE) was less than 10%. ::: ::: ::: CONCLUSIONS ::: This search strategy identified most systematic reviews without over-whelming users with numerous false-positive results. A "single-click" filter based on this strategy is now available as part of the Clinical Queries feature of PubMed. <s> BIB008 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> OBJECTIVE ::: To describe the obstacles encountered when attempting to answer doctors' questions with evidence. ::: ::: ::: DESIGN ::: Qualitative study. ::: ::: ::: SETTING ::: General practices in Iowa. ::: ::: ::: PARTICIPANTS ::: 9 academic generalist doctors, 14 family doctors, and 2 medical librarians. ::: ::: ::: MAIN OUTCOME MEASURE ::: A taxonomy of obstacles encountered while searching for evidence based answers to doctors' questions. ::: ::: ::: RESULTS ::: 59 obstacles were encountered and organised according to the five steps in asking and answering questions: recognise a gap in knowledge, formulate a question, search for relevant information, formulate an answer, and use the answer to direct patient care. Six obstacles were considered particularly salient by the investigators and practising doctors: the excessive time required to find information; difficulty modifying the original question, which was often vague and open to interpretation; difficulty selecting an optimal strategy to search for information; failure of a seemingly appropriate resource to cover the topic; uncertainty about how to know when all the relevant evidence has been found so that the search can stop; and inadequate synthesis of multiple bits of evidence into a clinically useful statement. ::: ::: ::: CONCLUSIONS ::: Many obstacles are encountered when asking and answering questions about how to care for patients. Addressing these obstacles could lead to better patient care by improving clinically oriented information resources. <s> BIB009 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> Users of clinical practice guidelines and other recommendations need to know how much confidence they can place in the recommendations. Systematic and explicit methods of making judgments can reduce errors and improve communication. We have developed a system for grading the quality of evidence and the strength of recommendations that can be applied across a wide range of interventions and contexts. In this article we present a summary of our approach from the perspective of a guideline user. Judgments about the strength of a recommendation require consideration of the balance between benefits and harms, the quality of the evidence, translation of the evidence into specific circumstances, and the certainty of the baseline risk. It is also important to consider costs (resource utilisation) before making a recommendation. Inconsistencies among systems for grading the quality of evidence and the strength of recommendations reduce their potential to facilitate critical appraisal and improve communication of these judgments. Our system for guiding these complex judgments balances the need for simplicity with the need for full and transparent consideration of all important issues. <s> BIB010 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> A large number of taxonomies are used to rate the quality of an individual study and the strength of a recommendation based on a body of evidence. We have developed a new grading scale that will be used by several family medicine and primary care journals (required or optional), with the goal of allowing readers to learn one taxonomy that will apply to many sources of evidence. Our scale is called the Strength of Recommendation Taxonomy. It addresses the quality, quantity, and consistency of evidence and allows authors to rate individual studies or bodies of evidence. The taxonomy is built around the information mastery framework, which emphasizes the use of patient-oriented outcomes that measure changes in morbidity or mortality. An A-level recommendation is based on consistent and good quality patient-oriented evidence; a B-level recommendation is based on inconsistent or limited quality patient-oriented evidence; and a C-level recommendation is based on consensus, usual practice, opinion, disease-oriented evidence, or case series for studies of diagnosis, treatment, prevention, or screening. Levels of evidence from 1 to 3 for individual studies also are defined. We hope that consistent use of this taxonomy will improve the ability of authors and readers to communicate about the translation of research into practice. <s> BIB011 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> To encourage high-quality patient care guided by the best evidence, many medical schools and residencies are teaching techniques for critically evaluating the medical literature. While a large step forward, these skills of evidence-based medicine are necessary but not sufficient for the practice of contemporary medicine. Incorporating the best evidence into the real world of busy clinical practice requires the applied science of information management. Clinicians must learn the techniques and skills to focus on finding, evaluating, and using information at the point of care. This information must be both relevant to themselves and their patients as well as being valid. The authors discuss the need to teach the applied science of information management along with, or perhaps even instead of, teaching the basic science of evidence-based medicine. All students, residents, and practicing physicians need three skills to practice the best medicine: the ability to select foraging—“keeping up”—tools that filter information for relevance and validity, the skill to select and use a <s> BIB012 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> OBJECTIVE ::: To develop optimal search strategies in Medline for retrieving systematic reviews. ::: ::: ::: DESIGN ::: Analytical survey. ::: ::: ::: DATA SOURCES ::: 161 journals published in 2000 indexed in Medline. ::: ::: ::: MAIN OUTCOME MEASURES ::: The sensitivity, specificity, and precision of retrieval of systematic reviews of 4862 unique terms in 782,485 combinations of one to five terms were determined by comparison with a hand search of all articles (the criterion standard) in 161 journals published during 2000 (49,028 articles). ::: ::: ::: RESULTS ::: Only 753 (1.5%) of the 49,028 articles were systematic reviews. The most sensitive strategy included five terms and had a sensitivity of 99.9% (95% confidence interval 99.6% to 100%) and a specificity of 52% (51.6% to 52.5%). The strategy that best minimised the difference between sensitivity and specificity had a sensitivity of 98% (97% to 99%) and specificity of 90.8% (90.5% to 91.1%). Highest precision for multiterm strategies, 57% (54% to 60%), was achieved at a sensitivity of 71% (68% to 74%). The term "cochrane database of systematic reviews.jn." was the most precise single term search strategy (sensitivity of 56% (52% to 60%) and precision of 96% (94% to 98%)). These strategies are available through the "limit" screen of Ovid's search interface for Medline. ::: ::: ::: CONCLUSIONS ::: Systematic reviews can be retrieved from Medline with close to perfect sensitivity or specificity, or with high precision, by using empirical search strategies. <s> BIB013 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> OBJECTIVE ::: To identify the most frequent obstacles preventing physicians from answering their patient-care questions and the most requested improvements to clinical information resources. ::: ::: ::: DESIGN ::: Qualitative analysis of questions asked by 48 randomly selected generalist physicians during ambulatory care. ::: ::: ::: MEASUREMENTS ::: Frequency of reported obstacles to answering patient-care questions and recommendations from physicians for improving clinical information resources. ::: ::: ::: RESULTS ::: The physicians asked 1,062 questions but pursued answers to only 585 (55%). The most commonly reported obstacle to the pursuit of an answer was the physician's doubt that an answer existed (52 questions, 11%). Among pursued questions, the most common obstacle was the failure of the selected resource to provide an answer (153 questions, 26%). During audiotaped interviews, physicians made 80 recommendations for improving clinical information resources. For example, they requested comprehensive resources that answer questions likely to occur in practice with emphasis on treatment and bottom-line advice. They asked for help in locating information quickly by using lists, tables, bolded subheadings, and algorithms and by avoiding lengthy, uninterrupted prose. ::: ::: ::: CONCLUSION ::: Physicians do not seek answers to many of their questions, often suspecting a lack of usable information. When they do seek answers, they often cannot find the information they need. Clinical resource developers could use the recommendations made by practicing physicians to provide resources that are more useful for answering clinical questions. <s> BIB014 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> Abstract Evidence-based practice (EBP) is increasingly being advocated as the preferred approach to practice in augmentative and alternative communication (AAC). The EBP process involves multiple steps. The asking of a well-built question is the first step in the quest for answers. At the same time it is also often the first stumbling block for practitioners. To facilitate the asking of well-built questions it may be helpful to follow a template. The most frequently used template is PICO, which stands for patient, intervention, comparison, and outcome [Richardson, W., Wilson, M., Nishikawa, J., & Hayward, R. (1995). The well-built clinical question: A key to evidence-based decisions. ACP Journal Club , 123 , A12–A13]. In this article, we examine the suitability of the PICO template for AAC in terms of the representativeness of the components, and the appropriateness of its subcomponents, and their terminology. Based on this analysis, we propose the PESICO template, which stands for person, environments, stakeholders, intervention, comparison, and outcome. This template is then illustrated with examples representing a range of decision-making areas in AAC. Finally, directions for future research are provided. Learning outcomes: The reader will be able to: (1) appreciate the importance of asking well-built questions, (2) name the shortcomings of the PICO template, and (3) describe the components of the proposed PESICO template for asking well-built questions. <s> BIB015 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> AimGiven a set of pre-defined medical categories used in Evidence Based Medicine, we aim to automatically annotate sentences in medical abstracts with these labels.MethodWe constructed a corpus of 1,000 medical abstracts annotated by hand with specified medical categories (e.g. Intervention, Outcome). We explored the use of various features based on lexical, semantic, structural, and sequential information in the data, using Conditional Random Fields (CRF) for classification.ResultsFor the classification tasks over all labels, our systems achieved micro-averaged f-scores of 80.9% and 66.9% over datasets of structured and unstructured abstracts respectively, using sequential features. In labeling only the key sentences, our systems produced f-scores of 89.3% and 74.0% over structured and unstructured abstracts respectively, using the same sequential features. The results over an external dataset were lower (f-scores of 63.1% for all labels, and 83.8% for key sentences).ConclusionsOf the features we used, the best for classifying any given sentence in an abstract were based on unigrams, section headings, and sequential information from preceding sentences. These features resulted in improved performance over a simple bag-of-words approach, and outperformed feature sets used in previous work. <s> BIB016 </s> Automated text summarisation and evidence-based medicine: A survey of two domains <s> EVIDENCE-BASED MEDICINE <s> Discharge summaries and other free-text reports in healthcare transfer information between working shifts and geographic locations. Patients are likely to have difficulties in understanding their content, because of their medical jargon, non-standard abbreviations, and ward-specific idioms. This paper reports on an evaluation lab with an aim to support the continuum of care by developing methods and resources that make clinical reports in English easier to understand for patients, and which helps them in finding information related to their condition. This ShARe/CLEFeHealth2013 lab offered student mentoring and shared tasks: identification and normalisation of disorders 1a and 1b and normalisation of abbreviations and acronyms 2 in clinical reports with respect to terminology standards in healthcare as well as information retrieval 3 to address questions patients may have when reading clinical reports. The focus on patients' information needs as opposed to the specialised information needs of physicians and other healthcare workers was the main feature of the lab distinguishing it from previous shared tasks. De-identified clinical reports for the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Task 3 were from the Internet and originated from the Khresmoi project. Task 1 annotations originated from the ShARe annotations. For Tasks 2 and 3, new annotations, queries, and relevance assessments were created. 64, 56, and 55 people registered their interest in Tasks 1, 2, and 3, respectively. 34 unique teams 3 members per team on average participated with 22, 17, 5, and 9 teams in Tasks 1a, 1b, 2 and 3, respectively. The teams were from Australia, China, France, India, Ireland, Republic of Korea, Spain, UK, and USA. Some teams developed and used additional annotations, but this strategy contributed to the system performance only in Task 2. The best systems had the F1 score of 0.75 in Task 1a; Accuracies of 0.59 and 0.72 in Tasks 1b and 2; and Precision at 10 of 0.52 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development. <s> BIB017
The phrase 'evidence-based medicine' was defined initially as "a process of turning clinical problems into questions and then systematically locating, appraising, and using contemporaneous research findings as the basis for clinical decisions" BIB001 . A more concrete and widely accepted definition of EBM was coined by  who explained it as "the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients." As the practice has evolved, medical practitioners increasingly integrate, as required by the latest clinical guidelines, the latest clinical evidence with their own expertise. In addition, practitioners are also required to incorporate the informed patients' choices. The long-term objective of EBM is to enhance the standard of healthcare, promote practices that are proven by evidence to work, and to identify elements of the practice that do not work. A patient-oriented approach is key to the practice-combining the best patient-oriented evidence with patient-centred care, placing the evidence in perspective with the needs and desires of the patient BIB012 . In the following sections, we provide an overview of EBM, discuss the problems associated with EBM practice, the resources and tools available for text processing in this domain and the solutions that NLP can offer. Incorporation of research evidence when making clinical decisions involves "the practice of assessing the current problem in the light of the aggregated results of hundreds or thousands of comparable cases in a distant population sample, expressed in the language of probability and risk" BIB005 ]. EBM therefore involves the efficient use of information search strategies to locate reliable and up-to-date information from varying sources and extraction strategies to efficiently collect and analyse retrieved information. The practice includes a process formally known as the Critical Appraisal Exercise which involves the following steps according to  : (1) clearly defining what the problem of a patient is, and what evidence is needed to address the patient's problems; (2) performing searches for the relevant literature efficiently; (3) selecting the best of the performed studies, and applying the guidelines of EBM to ascertain the validity of the studies; (4) appraising the quality of the evidence effectively; and (5) extracting and synthesising relevant evidences and applying them to the problem at hand. Defining a Patient Problem as a Clinical Question. Formulating the patient problem forms the basis for the clinical question, which is used to search resources for an evidence-based answer. A well formulated question includes information about a patient (symptoms, signs, test results and knowledge of previous treatments), the particular values and preferences of the patient, and other factors that could be relevant  ]. All that information should be summarised into a succinct question defining the problem and the specific additional items of information needed to solve the problem. There has been substantial research in the area of medical question formulation and query-focused summarisation, and, in recent years, particularly in the field of EBM (e.g., a recent example of research drive in this area is the CLEF eHealth shared tasks BIB017 ). This is because it has been shown that the answerability of questions can be largely increased by better query formulation among other things BIB002 . The PICO format, which has four components, has become the accepted framework for formulating patient-specific clinical queries BIB003 . The four components are: "primary Problem/Population, main Intervention, main intervention Comparison, and Outcome of intervention." The PICO components of queries represent key aspects of patient care, and have become very much the standard for formulating EBM queries. Although this query formulation framework was originally designed for treatment-like queries, it was later extended to other kinds of medical queries BIB006 ]. Research has shown that this mechanism of framing questions helps in the better formulating of clinical problems in queries, resulting in more accurate information retrieval results BIB007  . Although the PICO format has gained popularity with time, it is well known that not all clinical questions, because of the complex nature of the information needs, can be mapped in terms of PICO elements, and this is particularly true for non-therapy questions  . There is also evidence that even doctors find it difficult to formulate the questions in terms of the PICO format BIB009 . Variants of the framework have been suggested (e.g., PESICO BIB015 , PIBOSO BIB016 ); they offer more flexibility and comprehensiveness and have applications beyond query formulation. Conducting Literature Search. The medical domain has a massive amount of available published literature, scattered over many databases (e.g., MEDLINE 1 indexes over 24 million articles), and searching for relevant literature requires expertise in this area. Searching for appropriate literature can include searching from raw databases, databases with search filters, databases of pre-appraised articles, databases of synthesised articles and even personal contact with human sources. Some research has been carried out on strategies for retrieving high quality medical articles BIB004 BIB008 BIB013 . Searching for the correct literature is a rather tedious task, and it is in fact one of the major problems associated with EBM practice BIB014 . Selecting the Best Resources. The selected articles must be closely relevant to the problem at hand, and, simultaneously, they must have a good level of evidence. The level of evidence of a medical publication can be influenced by a number of factors such as the publication type and the sample size of the study. Checking the relevance of the papers requires a thorough analysis, and a ranking system is usually employed for examining the level of evidence of different sources. Medical publication types include Systematic Reviews (SR), Randomised Controlled Trials (RCT), Meta-Analyses (MA), single Case Studies, Tutorial Reviews and many more, including even personal opinions. Although all of them provide evidence of some form, their levels of evidence vary significantly. Figure 1 , slightly modified from the one provided by Gilbody [1996] , provides a ranking of some of the common medical article types (highest ranked on top). Fig. 1 : Quality of evidence with respect to publication types, adapted from Gilbody [1996] . A detailed analysis of these studies is outside the scope of this survey. Guidelines are available for healthcare practitioners to obtain information from each of these types of studies and evaluate their levels of evidence. Examples of guidelines available online include ASSERT (A Standard for the Scientific and Ethical Review of Trials) 2 , PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 3 , and EQUATOR (Enhancing the QUAlity and Transparency Of health Research) 4 . Identifying Strengths and Weaknesses. Once relevant papers are identified, each paper must be studied in detail to extract the evidence it contains with respect to the problem at hand. Practitioners are particularly interested in studying in detail what type of study was conducted, on the number of subjects, demographic information about the subjects,the kind of intervention, the length of the follow-up period, and the outcome measures  ]. For quantitative tests, analysing the results of statistical tests is also an important task. Identification of the strengths and weaknesses provides a clear indication of the quality of the evidence provided by each article, and hence aids the practitioner to make the final decision. Applying Evidence to Patient Care. Practitioners make their final judgment considering the outcomes presented in the article(s) and the relevance of the article(s) to the problem at hand. Often, a number of articles suggest the same solution or a similar one, making it easier for the practitioner to make a decision. However, there are also cases when chosen articles provide contradictory outcomes. In such cases, practitioners have to choose between outcomes based on personal experience, the quality of evidence of the articles, the closeness of the articles to the given problem or other sources of evidence. Note that the quality of evidence here refers to the overall quality of the evidence obtained from all the articles combined, whereas the level of evidence of a single article refers to the reliability of that particular article only. • 0:5 Generally, countries have mandatory clinical practice guidelines that must be adhered to when performing EBM. Practitioners primarily navigate through these guidelines during practice. As such, it is actually often the providers/developers of clinical guidelines who are required to perform elaborate literature searches and follow the approaches mentioned above for the preparation of the appropriate practice guidelines. There has been research to identify a grading system that is suitable for the practice of EBM, and numerous requirements for the systems have been specified, some more important than others in typical EBM settings. BIB010 , for example, suggested two aspects-simplicity and clarity-so that identifying the quality of a body of evidence is fast and can be easily projected on to some grading scale. BIB011 mentioned the comprehensiveness of evidence appraisal systems as a key aspect so that the same grading system may be applied to different types of clinical studies such as treatment, diagnosis and prognosis.