Treatment recommendations for the same condition from different guideline bodies often disagree, even when the same randomized controlled trial (RCT) evidence is cited. Guideline appraisal tools focus on methodology and quality of reporting, but not on the nature of the supporting evidence. This study was done to evaluate the quality of the evidence  underlying therapy recommendations in evidence-based clinical practice guidelines.A cross-sectional analysis of cardiovascular risk management recommendations was performed for three different conditions  from three pan-national guideline panels . Of the 338 treatment recommendations in these nine guidelines, 231 (68%) cited RCT evidence but only 105 (45%) of these RCT-based recommendations were based on high-quality evidence. RCT-based evidence was downgraded most often because of reservations about the applicability of the RCT to the populations specified in the guideline recommendation  or because the RCT reported surrogate outcomes .The results of internally valid RCTs may not be applicable to the populations, interventions, or outcomes specified in a guideline recommendation and therefore should not always be assumed to provide high-quality evidence for therapy recommendations.  From an analysis of cardiovascular risk-management recommendations in guidelines produced by pan-national panels, McAlister and colleagues concluded that fewer than half were based on high-quality evidence. Until recently, doctors largely relied on their own experience to choose the best treatment for their patients. Faced with a patient with high blood pressure (hypertension), for example, the doctor had to decide whether to recommend lifestyle changes or to prescribe drugs to reduce the blood pressure. If he or she chose the latter, he or she then had to decide which drug to prescribe, set a target blood pressure, and decide how long to wait before changing the prescription if this target was not reached. But, over the past decade, numerous clinical practice guidelines have been produced by governmental bodies and medical associations to help doctors make treatment decisions like these. For each guideline, experts have searched the medical literature for the current evidence about the diagnosis and treatment of a disease, evaluated the quality of that evidence, and then made recommendations based on the best evidence available.The recommendations made in different clinical practice guidelines vary, in part because they are based on evidence of varying quality. To help clinicians decide which recommendations to follow, some guidelines indicate the strength of their recommendations by grading them, based on the methods used to collect the underlying evidence. Thus, a randomized clinical trial (RCT)—one in which patients are randomly allocated to different treatments without the patient or clinician knowing the allocation—provides higher-quality evidence than a nonrandomized trial. Similarly, internally valid trials—in which the differences between patient groups are solely due to their different treatments and not to other aspects of the trial—provide high-quality evidence. However, grading schemes rarely consider the size of studies and whether they have focused on clinical or so-called “surrogate” measures.  Most guidelines also do not consider how generalizable (applicable) the results of a trial are to the populations, interventions, and outcomes specified in the guideline recommendation. In this study, the researchers have investigated the quality of the evidence underlying recommendations for cardiovascular risk management in nine evidence-based clinical practice guides using these additional criteria.The researchers extracted the recommendations for managing cardiovascular risk from the current US, Canadian, and European guidelines for the management of diabetes, abnormal blood lipid levels (dyslipidemia), and hypertension. They graded the quality of evidence for each recommendation using the Canadian Hypertension Education Program (CHEP) grading scheme, which considers the type of study, its internal validity, its clinical relevance, and how generally applicable the evidence is considered to be. Of 338 evidence-based recommendations, two-thirds were based on evidence collected in internally valid RCTs, but only half of these RCT-based recommendations were based on high-quality evidence. The evidence underlying 64 of the guideline recommendations failed to achieve a high CHEP grade because the RCT data were collected in a population of people with different characteristics to those covered by the guideline. For example, a recommendation to use spironolactone to reduce blood pressure in people with hypertension was based on an RCT in which the participants initially had congestive heart failure with normal blood pressure. Another 59 recommendations were downgraded because they were based on evidence from RCTs that had not focused on clinical measures of effectiveness.These findings indicate that although most of the recommendations for cardiovascular risk management therapies in the selected guidelines were based on evidence collected in internally valid RCTs, less than one-third were based on high-quality evidence applicable to the populations, treatments, and outcomes specified in guideline recommendations. A limitation of this study is that it analyzed a subset of recommendations in only a few guidelines. Nevertheless, the findings serve to warn clinicians that evidence-based guidelines are not necessarily based on high-quality evidence. In addition, they emphasize the need to make the evidence base underlying guideline recommendations more transparent by using an extended grading system like the CHEP scheme. If this were done, the researchers suggest, it would help clinicians apply guideline recommendations appropriately to their individual patients.http://dx.doi.org/10.1371/journal.pmed.0040250.Please access these Web sites via the online version of this summary at evidence-based medicine and on clinical practice guidelines • Wikipedia contains pages on US national guidelines• The National Guideline Clearinghouse provides information on The Guidelines International Network promotes the systematic development and application of clinical practice guidelines• Canadian Hypertension Education Program (CHEP) (in French and English)• Information is available on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group, an organization that has developed an grading scheme similar to the CHEP scheme • See information on the  There has been a rapid expansion in the number of clinical practice guidelines over the past decade and, as a result, clinicians are frequently faced with several guidelines for treatment of the same condition. Unfortunately, recommendations may differ between guidelines ,2, leaviHowever, few studies have addressed the issue raised in the second interpretation of an evidence-based guideline—that is, the quality of the evidence underpinning “evidence-based” guidelines. Given the widespread availability of electronic databases to search the literature, one would expect that evidence-based guidelines would usually cite the same evidence. However, an analysis of 15 guidelines for type 2 diabetes mellitus revealed little overlap—only ten studies  were cited in at least six of these guidelines, and the most frequently cited study in these guidelines  was conducted exclusively in patients without type 2 diabetes mellitus .http://www.gradeworkinggroup.org/, last accessed 29 January 2007)—consistency between studies is not explicitly evaluated in the CHEP scheme  and, instead, CHEP places primacy on the study achieving the highest evidence grading for a particular recommendation  –19 10–19idelines ,14,17 prThe standardized scores for each domain of the AGREE instrument (averaged across guidelines within the same topic areas) are outlined in Of the 338 recommendations that endorsed particular cardiovascular risk management therapies and cited evidence, 231 (68%) cited an RCT or systematic review of RCTs in support of that recommendation . Of thesThe most frequent reason for downgrading RCT-based therapy recommendations (64 [51%] of the 126 cases) were concerns about the need to extrapolate from a highly selected RCT population to the scenario and/or the target population specified in the guideline. As an illustration, consider the differences in quality of evidence underlying two antihypertensive recommendations that were both based on RCT evidence. The first recommendation was for the use of thiazides in individuals with hypertension. It cited a meta-analysis  of 30 RC[r] = 0.13, p = 0.78), number of references , or score on the “rigor of development” domain of the AGREE instrument .The quality of the evidence base cited for cardiovascular risk management therapies was not related to guideline length . For example, the 2003 Kidney Disease Outcome Quality Initiative guidelines  recommenWe do not mean to imply that recommendations should not be made in the absence of high quality evidence or that RCT evidence should not be extrapolated beyond the limits of trial eligibility criteria. Indeed, we recognize that trialists design RCTs with relatively homogenous populations in order to maximize internal validity , and there are published guides on how and when to extrapolate RCT evidence to individual patient situations . HoweverOur findings that only some guidelines linked their recommendations to citations and that only some used explicit grading systems to communicate the quality of the evidence echo earlier reviews ,32–34. SDespite a number of strengths, our study has some limitations. First, we did not systematically search for different guidelines, but instead examined only a small sample of guidelines; future studies should expand our work to explore the quality of evidence underlying guidelines in other topic areas. For example, we believe a systematic examination of all guidelines produced by a particular organization  would provide useful additional insights. To do so, we advocate the use of explicit grading schemes such as those of CHEP or GRADE . HoweveIn conclusion, our finding that less than one-third of treatment recommendations  were based on high-quality evidence in national evidence-based guidelines for common conditions should sound a note of caution to consumers of clinical practice guidelines who assume that the sobriquet “evidence based” means that all recommendations contained therein are derived from high-quality evidence. In particular, we have documented that even evidence arising from internally valid RCTs may not be directly applicable to the populations, interventions, and outcomes specified in a guideline recommendation. As a recent editorial noted, “external validity is the neglected dimension in evidence ranking” . Indeed,


The Levels of Evidence Rating System is widely believed to categorize studies by quality, with Level I studies representing the highest quality evidence. We aimed to determine the reporting quality of Randomised Controlled Trials (RCTs) published in the most frequently cited general orthopaedic journals.Two assessors identified orthopaedic journals that reported a level of evidence rating in their abstracts from January 2003 to December 2004 by searching the instructions for authors of the highest impact general orthopaedic journals. Based upon a priori eligibility criteria, two assessors hand searched all issues of the eligible journal from 2003–2004 for RCTs. The assessors extracted the demographic information and the evidence rating from each included RCT and scored the quality of reporting using the reporting quality assessment tool, which was developed by the Cochrane Bone, Joint and Muscle Trauma Group. Scores were conducted in duplicate, and we reached a consensus for any disagreements. We examined the correlation between the level of evidence rating and the Cochrane reporting quality score.We found that only the Journal of Bone and Joint Surgery – American Volume (JBJS-A) used a level of evidence rating from 2003 to 2004. We identified 938 publications in the JBJS-A from January 2003 to December 2004. Of these publications, 32 (3.4%) were RCTs that fit the inclusion criteria. The 32 RCTs included a total of 3543 patients, with sample sizes ranging from 17 to 514 patients. Despite being labelled as the highest level of evidence (Level 1 and Level II evidence), these studies had low Cochrane reporting quality scores among individual methodological safeguards. The Cochrane reporting quality scores did not differ significantly between Level I and Level II studies. Correlations varied from 0.0 to 0.2 across the 12 items of the Cochrane reporting quality assessment tool (p > 0.05). Among items closely corresponding to the Levels of Evidence Rating System criteria assessors achieved substantial agreement .Our findings suggest that readers should not assume that 1) studies labelled as Level I have high reporting quality and 2) Level I studies have better reporting quality than Level II studies. One should address methodological safeguards individually. The International Society of Medical Editors emphasises the importance of effective reporting in medical literature ,2. HowevSince January 2003, all clinical scientific articles published in the American Volume of The Journal of Bone and Joint Surgery (JBJS-A) have included a level of evidence rating ,6. The LThe Levels of Evidence Rating System causes readers to infer that Level I evidence RCTs are of better methodological quality than Level II evidence RCTs . The EdiThe assessment of the true quality of published studies remains challenging -11. One The most developed criteria for guiding clinicians in their assessment of study reporting quality have been proposed for RCTs, since RCTs are a study design that yield the lowest chance of bias ,13. The Given the upcoming use of the Levels of Evidence Rating System in orthopaedic literature, we aimed to evaluate the reporting quality of RCTs published in the JBJS-A from 2003 to 2004 (Level I and Level II evidence ratings). We, therefore, extracted the level of evidence rating as published in each RCT and compared this rating with the well-established Cochrane Bone, Joint and Muscle Trauma Group's reporting quality assessment tool. We chose the JBJS-A because it was the most frequently sited general orthopaedic journal (ISI web of science), and the only journal that used this Levels of Evidence Rating System in the eligible time period.Our hypotheses were twofold: 1) Level I evidence studies in a high impact general orthopaedic journal would not necessarily have high quality reporting and 2) the reporting quality of RCTs would not differ among trials labelled as Level I or Level II evidence.We conducted a methodological study. We assessed the level of evidence rating assigned to a series of RCTs with the Cochrane reporting quality score.Two assessors  identified orthopaedic journals that reported a level of evidence rating in their abstracts from January 2003 to December 2004 by searching the instructions for authors of the highest impact general orthopaedic journals . Within the eligible journal, two assessors  hand searched all issues from 2003–2004. The eligibility criteria were determined and set a priori. Eligible studies included those reported as RCTs involving a therapeutic intervention and using human subjects. We conducted searches in duplicate, and the consensus of three authors  resolved any disagreements.The relevant demographic information was extracted from each eligible study by one investigator (RWP) and rechecked for accuracy by a second investigator (PAAS). The extracted data included (1) first author , (2) cited statistical support or methodological support by a department of clinical epidemiology or public health, (3) year of publication, (4) total sample size, (5) number of centres, (6) name of intervention, (7)category of intervention , (8) body region ,(9) financial support (yes or no), (10) direction of results, and (11) trial reported according to the CONSORT statement (yes or no).One of the authors (RWP) extracted the level of evidence from each abstract of the included RCTs. A second author (INS) double-checked the evidence rating to ensure that it was correctly extracted from the paper.Two authors , blinded to study author and institution, graded the reporting quality of the included RCTs using the Cochrane reporting quality assessment tool, which was devised by the Cochrane Bone, Joint and Muscle Trauma Group, formally known as the Musculoskeletal Injuries Group. This scoring scheme covers aspects of internal and external validity for the assessment of methodological quality . We usedStudies that randomly allocated patients (Item D), concealed randomisation (Item A), blinded participants  and documented study withdrawals (Item B) were reported to reflect higher quality ,20. We sWe used Intraclass Correlation Coefficients (ICC) to measure the agreement between the assessors' assessment of study reporting quality. We used Landis and Koch's suggested criteria for the interpretation of the agreement: 0 to 0.2 represented slight agreement, 0.21 to 0.40 fair agreement, 0.41 to 0.60 moderate agreement, and 0.61 to 0.80 substantial agreement. A value above 0.80 was considered almost perfect agreement , the foot and ankle [6 (19%)], and the knee [9 (28%)]. Four (13%) RCTs were reported according to the CONSORT statement  studies. Level II studies were also subgrouped into 1 Level II-1 and 2 level II-2 studies as extracted from the included papers' abstract.Only 12 38%) of the 32 included RCTs clearly described allocation concealment (Item A). Seven (22%) clearly described an intention to treat analysis (Item B). Thirteen (41%) clearly described the blinding of outcome assessors (Item C). Twenty-three (72%) clearly described the comparability of the treatment and control group at entry (Item D). Six (19%) of the 32 RCTs clearly described the blinding of participants (Item E). Only 2 (6%) of the studies clearly described the blinding of treatment providers (Item F). Seventeen (53%) clearly described identical care programmes other than the trial options (Item G). Of the 32 RCTs, 25 (78%) adequately described the inclusion and exclusion criteria (Item H). Of all the items, I and J were described best in all 32 RCTs: 31 (97%) clearly described the interventions and 31 (97%) clearly described the outcome measures used. Twenty-two (69%) clearly described a useful diagnostic test in the outcome assessment (Item K). Only 10 (31%) described an appropriate duration of follow-up (Item L).Table 8% of theAmong items closely corresponding to the Levels of Evidence Rating System criteria , assessors achieved substantial agreement . Across each of the 12 items, however, agreement varied (Range of ICC = 0 to 0.80). In all cases, assessors achieved consensus, either alone or with a third, intervening reviewer.We compared the mean score in each item of the Cochrane reporting quality assessment tool separately (Items A through L) with each level of evidence Table . Mean quThe results of our methodological study demonstrated two key findings 1) Level I evidence studies revealed important limitations in their quality of reporting and 2) non- significant difference in the quality of reporting between studies labelled as Level I or Level II evidence.Our study is strengthened by the use of a well-described and commonly used quality assessment tool from the Cochrane Collaboration that identifies the relevant methodological aspects of trials as reported and assesses these aspects individually. Furthermore, all assessors  were well trained in quality assessments. Our decision to conduct assessments in duplicate (and triplicate when assessors disagreed) further strengthened the rigor of our assessments . The pauA previous review of published studies in The Journal of Bone and Joint Surgery 1988 through 2000 revealed a similar proportion (3%) of randomised trials compared with our current study 3.4%) [% [4]. ThDespite the widely held belief that the Levels of Evidence Rating system categorizes studies by quality ,6, our sThis study was underpowered to explore the influence of reported statistical support, adherence to CONSORT guidelines, multi-centre studies, and sources of funding on the quality of reporting, direction of results, and magnitude of treatment effect size. Future studies are needed to explore any associations.Our findings suggest that readers should not assume that 1) studies labelled as Level I have high quality of reporting and 2) Level I studies have better reporting quality than Level II studies. Methodological safeguards should be addressed individually.RWP is supported, in part, by a Stichting Wetenschappelijk Onderzoek Orthopaedische Chirurgie Fellowship, Biomet Netherlands, Anna Fonds, Zimmer The Netherlands, Stryker The Netherlands, MSD The Netherlands, and a Nederlandse Vereniging voor Orthopedische Traumatologie Fellowship. MB is supported, in part, by a Canada Research Chair from the Canadian Institutes of Health Research.The funding bodies were not involved in study design; not in the collection, analysis, and interpretation of data; not in the writing of the manuscript; and not in the decision to submit the manuscript for publication. The other authors did not receive funding.RWP and MB generated the study question. RWP, MB, and PAAS participated in the study design. RWP, RK, INS, and PAAS abstracted the data. RWP, MB, and INS participated in the data analysis, interpretation, and performed the statistical analysis. RWP and MB drafted the first version of the manuscript.RWP, MB, PAAS, INS, RK, and KHL participated in drafting the subsequent versions of the manuscript and participated in the critical review of manuscript. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:The Cochrane Bone, Joint and Muscle Trauma Group tool for reporting of methodological quality. The reporting quality of the included RCTs was scored with the Cochrane reporting quality assessment tool, which was devised by the Cochrane Bone, Joint and Muscle Trauma Group, formally known as the Musculoskeletal Injuries Group. This scoring scheme covers aspects of internal and external validity for the assessment of methodological quality.Click here for fileReferences to included studies. Thirty-two RCTs that fit the eligibility criteria.Click here for file


Over the past years concerns are rising about the use of Evidence-Based Medicine (EBM) in health care. The calls for an increase in the practice of EBM, seem to be obstructed by many barriers preventing the implementation of evidence-based thinking and acting in general practice. This study aims to explore the barriers of Flemish GPs  to the implementation of EBM in routine clinical work and to identify possible strategies for integrating EBM in daily work.We used a qualitative research strategy to gather and analyse data. We organised focus groups between September 2002 and April 2003. The focus group data were analysed using a combined strategy of 'between-case' analysis and 'grounded theory approach'. Thirty-one general practitioners participated in four focus groups. Purposeful sampling was used to recruit participants.A basic classification model documents the influencing factors and actors on a micro-, meso- as well as macro-level. Patients, colleagues, competences, logistics and time were identified on the micro-level , commercial and consumer organisations on the meso-level  and health care policy, media and specific characteristics of evidence on the macro-level . Existing barriers and possible strategies to overcome these barriers were described.In order to implement EBM in routine general practice, an integrated approach on different levels needs to be developed. Over the past years concerns are rising about the use of Evidence-Based Medicine (EBM) in health care. The calls for an increase in the practice of EBM, seem to be obstructed by many barriers preventing the implementation of evidence-based thinking and acting in general practice. There is a growing need to develop strategies that are able to address the barriers towards evidence-based thinking and acting. We searched Medline, the Cochrane Library, ACP-Journal Club, Dare, Sociological abstracts, Psychinfo and the Campbell Library to reveal studies addressing barriers towards the implementation of EBM table  as well An inductive qualitative study was conducted (September 2002 – June 2003) using four focus groups with a total of 31 GPs. Purposeful sampling was used to recruit participants. Two major criteria were used to select the participants: (1) variability in interest towards EBM and (2) variability in expertise with respect to EBM. One group of 7 academics was chosen because of their status of being good informants on EBM. Group 2 consisted of 7 GPs recruited from different local peer groups in a major Flemish city. Group 3 consisted of 6 GPs recruited from an interuniversity postgraduate course in EBM. Group 4 consisted of 11 GPs from a local peer group in a small town.A professional moderator was hired to facilitate the focus group discussion using a semi-structured interview guide. One researcher took notes on non-verbal actions of participants. Four general topics were discussed: 1. Knowledge and understanding of EBM; 2. Applicability of EBM in general practice; 3. Specific barriers to implement EBM; 4. Suggestions for bridging the barriers. At the end of the focus group (1.5–2 hours) GPs were asked to complete a short questionnaire collecting demographic data . GPs state that the emancipation process of patients has blurred the autocratic position of doctors and demands for different behavioural strategies. Despite the fact that doctors consider their patients as well-informed, they lack the knowledge of EBM (6:30.2). Many GPs refer to the difficulties of patients to understand the rapid changes and insights in medicine (1:53.2). To dissolve this problem, most of the GPs are willing to take responsibility to create a safe environment in which the evidence-based story can be taught to the patient, for example by searching for evidence together through an evidence-based internet approach during consultation. Consumers need education, in which they can be taught to behave critically towards medication and media. According to several GPs consumer organisations could play an important role here.Conflicts with colleagues seem to be common. GPs clearly express the opinion that specialists' thinking is too commercial (4:148.5). This implies they are promoting and using products based on a commercial rather than an evidence-based approach. There is a general feeling that specialists who do act evidence-based should be protected. GPs also feel strongly for building networks with reliable colleagues. The fact that patients still consider the advice of a specialist more valuable is also brought forward. GPs are very much concerned about the fact that their role will become reduced to treating those patients who do not fit the criteria of specialists (1:65.6). Furthermore, in educational programmes more lectures and workshops should be given by colleagues who are not explicitly connected to universities, making EBM more of a real-life business (1:56.1).Lack of time, EBM-resources, knowledge and skills are the barriers that are most recognised in existing literature. They recur during the focus group interviews. Many GPs think it is worthwhile to create a helpdesk system or a central on-line information system that provides answers to specific questions (4:180.3). On-line sources should be easy to navigate and free of charge. Guidelines and Critical Appraised Topics would be a great help. A select team of GPs should be paid to develop them. Educational programs should focus on an integration of EBM-principles in medical school and further training courses. Specialised EBM-teams should be established. The idea of outreach visits by independent educators, paid by government, gains approval by many GPs (4:181.2). As a lot of existing EBM material does not reach the GPs, they suggest to put more efforts into getting this material under their attention. Furthermore, it will be important to change GPs attitudes towards EBM. Many GPs state that EBM remains too theoretical and authority based (1:126.4). It restricts clinical practice, limiting the choices. Resistance should be broken down, for instance by finding inspiration in change management techniques, as indicated by some participants.The EBM movement is relatively new in Flanders. At this stage different organisations undertook efforts to stimulate EBM, for example universities, the Belgian Centre for Evidence-Based Medicine and other occupational groups. Some GPs criticise the fragmentation of initiative, which does lead to an organisational chaos in the field. GPs do see a role for government in providing a network which tightens all different institutions together (6:99.9).Several GPs criticise the fact that educational programmes and the development of guidelines is sponsored by commercial organisations, for example pharmaceutical firms and their representatives (1:152.5). Questions do rise about the honesty of their message, in which EBM is used as a fashion term. Moreover, GPs receive a lot of information from different sources. Only a small part of the information is independent and unfortunately not easily identified in the information overload (5:73.3). Suggestions are made to label independent information, distinguishing it from less relevant information. Conflicts of interest must be clearly stated. Many participants feel that spreading a uniform EBM message by all actors in the field of health care is very important and that this specific approach should also become a necessity for the commercial organisations.GPs in the sample worry about the quality of the messages spread by consumer organisations, for example self help groups. In order to stimulate EB-practice more attention should be paid to the content and form of the messages and the methods used (1:101.4). According to several GPs consumer organisations could play a major role in distributing evidence-based messages to patients that are screened for quality and in measuring their satisfaction with evidence-based treatments.Belgium has a particular way of financing the GPs. GPs are most of the time independent entrepreneurs, who are financed by the number of medical acts defined in the nomenclature. Several GPs raise their voice about 'the policy system' that works counterproductive for an evidence-based approach. The time spent on research and information seeking behaviour is not paid for. One focuses necessarily on the amount of consultations, rather then evidence-based acting, and doctors adapt to the protocols they are forced to follow (1:188.6). Consultations lead to a higher income for a doctor but, as indicated by several GPs, often also to unnecessary practices (4:102.2). According to some participants a fixed income could be a solution, as well as financial incentives for extra time that is invested in acting evidence-based. Reimbursement systems should be consequent with what is promoted regarding to EBM (4:132.6). Some GPs criticise the fact that EBM seems to develop as a reductionist model to judge results of clinical acting. Most GPs think that broader outcome-measures should be taken into account, for example quality of life (1:212.5). Some concerns are expressed about the coaching of GPs to implement evidence. Too easily, government assumes that all doctors are comfortable with the evidence-based paradigm. GPs are given the noble task of being the messengers, without having enough structured information and without being enough knowledgeable for that 'new' role. Therefore many voices are raised to improve the distribution of evidence-based information.The mass media are identified as a counterproductive force, because products or services are being promoted that are not always supported by evidence (1:98.2). Patients seek answers to their questions on the internet. The quality of this information is doubtable. This changing information seeking behaviour has an impact on the relation between GPs and patients. Some GPs state that patients become negotiating partners (6:27.8). Starting a social debate could be a first step to promote coherence between different parties of interest involved in the production, interpretation, use, dissemination and promotion of evidence. In defining a rational policy towards EBM GPs find it important to be involved in the process and to integrate late-adapters in the evidence-based movement (1:149.3). The media could play an important role in representing their voice in the economic choices made to support an evidence-based health care system, since these choices are often made by governmental institutions only.The sub-optimality of the evidence provided is frequently stated as a barrier to the implementation of EBM. Contradictions between similar evidence-based studies are common. Some of the studies that are promoted as 'evidence-based' are not up-to-date. Several GPs point towards the lack of evidence for many of their problems (5:56.3). There is also a time-delay between the research process and the actual adaptation of the new insights in the field (6:11.8). No explicit suggestions to overcome these barriers are given by the participants.The results presented in previous sections give an overview of issues mentioned by different GPs during the focus group sessions. Compared to what already was known from previous qualitative research (table 1.2) this study adds two more 'actors' that influence the implementation of EBM in general practice: consumer organisations at the meso-level and health care policy at the macro level. Providing logistic and educational support seems to be important to participants in previous research programmes, as well as for Flemish practitioners. Recent reviews on the effects of decision support systems to support evidence-based practice conclude that these strategies improve practitioners' performance ,26. PracApart from these insights the focus group research reveals that interventions should also focus on communication, rational policy making and the building of networks between different 'actors' to impact on an actual implementation of EBM in practice. The analytic classification proposed in this paper has the advantage to identify the complexity of problems and by consequence to identify and focus on interventions. To our knowledge no other study in the field of difficulties in the implementation of EBM in general practice has proposed a theoretical framework wherein barriers can easily be located and which naturally leads to the development of strategies to tackle them at the right level. We feel the framework is of relevance to the international community of GPs and those who promote EBM in this context. The classification model as presented seems to be a useful tool to orient change management processes.Some specific barriers for GPs are directly linked to the specific characteristics of the Belgian health care system. These characteristics should be taken into account in other countries or regions. Considering this aspect we want to warn for generalisations even in Belgium. Although the organisation of the health care system is a federal issue, the results presented could be biased since health care traditions are culturally diverging in Flanders, the Walloon provinces and Brussels. However, these contextual issues are fundamental to address when developing policies or interventions.The issue of generalisation brings us to some other methodological considerations. The 31 general practitioners who participated in this study formed a small sample of Flemish GPs. The sample was neither random nor representative. Given the basis on which most of the GPs joined the study we assume that their interest in EBM would be greater than that of their colleagues in the field. However, we are convinced that precisely this positive attitude and experience revealed knowledgeable results to contemplate further about interventions facilitating an evidence-based approach. The focus group methodology was considered appropriate to identify barriers and suggestions to overcome them. Methodological rigour was assured by using a combined approach of 'between-case analyses' and an inductive strategy based on 'grounded theory approach'. The first two focus groups were coded by two independent researchers and the analysis was checked and discussed with two senior researchers. However, when assessing the outcomes of the focus groups it was noticed that saturation was not reached. The last focus group interview still generated seven new ideas: five new barriers and two more interventions. Therefore we cannot guarantee that all factors, actors and suggestions were identified. More methodological research on predictive models for saturation points in qualitative research is necessary.Another interesting methodological issue is that we found a high level of 'agreement' between participants. A possible explanation for the lack of 'extreme cases' could be that the topic under study (the implementation of EBM) was relatively new to many participants. Conversation in the focus groups were about 'recognition' and 'explanation' rather than 'debate' and 'opposing opinions' about suggestions. This probably explains why we inventoried more barriers than suggestions on interventions and strategies.The classification model can be used as a frame of reference to initiate change management and policy interventions. For a successful implementation of EBM different 'actors' should be mobilised at different levels, as they are able to prevent GPs from acting evidence-based. However, potential conflicts between 'actors' can be bridged by 1) creating the necessary communication platforms between actors at different levels and within different disciplines , (2) communicating a consistent and uniform message on the principles of EBM and (3) modifying the current health care system towards a more evidence-based approach opening the discussion on a fixed income vs. an income based on consultations). Future research projects should focus on the changing relations between GPs and other 'actors'. Furthermore, (4) qualitative educational programmes should be offered, ideally within working hours, with a reasonable compensation for the income lost and organised by independent organisations, (5) access to EBM-resources should be free of charge, easy accessible and whenever possible provided in the mother tongue of the practitioners and (6) gaps in knowledge need to be addressed by researchers world wide. A better understanding of the complex relationships as proposed in the conceptual framework (figure  creatingThe author(s) declare that they have no financial competing interests. However, answers to questions could have been influenced by a convenience bias, as the participating GPs were aware that the focus groups were carried out by a research team known to be in favour of EBM.KH set up the focus groups, coded the manuscripts, analyzed the qualitative data and drafted the manuscript. BA coordinated the research process and participated in the design of the study. EV assisted in the coding process and participated in the data-analysis. ML helped developing the conceptual framework. AD and FB interpreted the results and commented on the draft version of the manuscript. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:


Electronic health resources are helpful only when people are able to use them, yet there remain few tools available to assess consumers’ capacity for engaging in eHealth. Over 40% of US and Canadian adults have low basic literacy levels, suggesting that eHealth resources are likely to be inaccessible to large segments of the population. Using information technology for health requires eHealth literacy—the ability to read, use computers, search for information, understand health information, and put it into context. The eHealth Literacy Scale (eHEALS) was designed (1) to assess consumers’ perceived skills at using information technology for health and (2) to aid in determining the fit between eHealth programs and consumers. The eHEALS is an 8-item measure of eHealth literacy developed to measure consumers’ combined knowledge, comfort, and perceived skills at finding, evaluating, and applying electronic health information to health problems. The objective of the study was to psychometrically evaluate the properties of the eHEALS within a population context. A youth population was chosen as the focus for the initial development primarily because they have high levels of eHealth use and familiarity with information technology tools. Data were collected at baseline, post-intervention, and 3- and 6-month follow-up using control group data as part of a single session, randomized intervention trial evaluating Web-based eHealth programs. Scale reliability was tested using item analysis for internal consistency  and test-retest reliability estimates. Principal components factor analysis was used to determine the theoretical fit of the measures with the data.r = .51 to .76. Test-retest reliability showed modest stability over time from baseline to 6-month follow-up (r = .68 to .40). Principal components analysis produced a single factor solution (56% of variance). Factor loadings ranged from .60 to .84 among the 8 items. A total of 664 participants  aged 13 to 21  completed the eHEALS at four time points over 6 months. Item analysis was performed on the 8-item scale at baseline, producing a tight fitting scale with α = .88. Item-scale correlations ranged from  The eHEALS reliably and consistently captures the eHealth literacy concept in repeated administrations, showing promise as tool for assessing consumer comfort and skill in using information technology for health. Within a clinical environment, the eHEALS has the potential to serve as a means of identifying those who may or may not benefit from referrals to an eHealth intervention or resource. Further research needs to examine the applicability of the eHEALS to other populations and settings while exploring the relationship between eHealth literacy and health care outcomes. An often unmentioned assumption underlying the deployment of eHealth resources intended for public consumption is that consumers have the skills to use such resources to their optimal level. Yet, with over 40% of US and Canadian adults having basic (or prose) literacy levels below what is considered necessary to optimally participate in civil society ,2, it isliteracy . This coInformed decision making requires that people can adequately access, understand, and process health information to meet their needs. Access refers both to the literal ability to access information resources like health websites, but also the quality of this access. This includes the quality of the technology  and the conditions of use, such as whether people have the privacy or time to properly engage eHealth resources. Access in the Internet age also requires an ability to derive meaning from text. As basic literacy skills rise, so does the ability to use computers effectively to solve problems, regardless of age, income, or education .Given issues of access and literacy, health practitioners in clinical and public health settings require an understanding of what abilities their patients/clients have before recommending eHealth resources. This article describes the development and psychometric evaluation of a measure of eHealth literacy designed for broad use in supporting consumer eHealth in public health and clinical care.Health literacy has been identified as a public health goal for the 21st century and a significant challenge facing health care globally -7. The reHealth literacy is comprised of six core skills, or literacies: (1) traditional literacy, (2) health literacy, 3) information literacy, (4) scientific literacy, (5) media literacy, and (6) computer literacy  informat, which pThe eHealth Literacy Scale (eHEALS) has been developed to address the need to assess eHealth literacy for a wide range of populations and contexts. The eHEALS is a self-report tool that can be administered by a health professional and is based on an individual’s perception of her or his own skills and knowledge within each measured domain. The instrument is designed to provide a general estimate of consumer eHealth-related skills that can be used to inform clinical decision making and health promotion planning with individuals or specific populations.It is not unreasonable to assume a link between eHealth literacy and technology use in general. The more an individual uses technology, the more likely they are to develop skills in using that technology as a tool. For that reason, youth can serve as an ideal group to test a measure of eHealth literacy given this population’s high familiarity with technology. In Canada, 99% of adolescents have access to the Internet, and the majority of Canadian teens report using the Internet for health in some capacity . AlthougRegardless of the population of interest, the need to navigate the Internet with confidence is particularly important for health issues in which the consequences for using low quality, misleading, or false information are great . By provA review of the literature was undertaken on each of the six key literacies that comprise Norman and Skinner’s eHealth literacy model  in the MA total of 89 youth (ages 14-24) completed the initial, larger version of the eHEALS as part of a pilot test and provided comments on the readability and item wording in focus groups immediately following completion of the instrument in paper form. The eHEALS was subsequently reviewed and modified to create the final battery of 8 items based on the qualitative and response feedback from participants, theoretical fit, and comprehensiveness. This study represents its first full psychometric assessment.This study was conducted as part of another larger evaluation project looking at eHealth smoking prevention and behavior change using a randomized controlled trial. The study described here involved participants from one arm of this trial given that the other arm was intended to promote eHealth literacy, thus potentially confounding the results of the psychometric review. The study recruited 664 adolescents from 14 secondary schools in a large Canadian city. Students in grades 9, 10, and 11 were sampled from a variety of class types encompassing different subject areas  and formats . An attempt was made to involve a cross-section of schools in the study through active recruitment directly with school administrators and teachers. Schools were offered a modest stipend for their involvement, but no direct incentives were provided to individual students as the study was considered a part of classroom activities due to a fit with the curriculum. Ethical approval for the study was obtained from the ethical review boards or committees from the University of Toronto, Toronto Public Health (a project partner), and both of the participating school boards.Age of the participants ranged from 13 to 21 , which included recent immigrants who may have been older than typical students in a particular grade. Sex was unevenly distributed within the sample due in part to the involvement of many single-sex classes involved in the study . The sample reflected the ethno-cultural diversity of the community, with the most commonly identified ethno-cultural groups being of Eastern European , East Asian , and Central Asian origin (N = 78), while 16% (N = 106) of participants did not identify with a particular cultural group. Ethno-cultural identity was determined using categories modified from Statistics Canada [Participants reported being regular users of various forms of information technologies: 71% of participants (N = 468) reported using email at least once a week, with 37% (N = 544) using it daily; 79% (N = 522) reported using the Web each week, with 35% (N = 232) using it daily; and 71% (N = 473) of participants were regular (weekly) users of text messaging, with 42% (N = 280) reporting using it daily. Most participants reported that their primary access point for the Internet was at home ; school  and friends’ homes  were identified as the most common secondary access points for the Internet.The eHEALS was administered within a larger battery of measures as part of a combined randomized trial evaluation of an eHealth literacy promotion intervention and a Web-based smoking cessation program . For theInternal consistency reliability was assessed using SPSS version 11.5 using thr = .51 to .76. Principal components analysis was performed and produced a single factor solution as expected . Factor loadings ranged from .60 to .84 among the 8 items.The internal consistency reliability and factor analysis results are presented in r = .49 to .68  using a standard regression model (SPSS REGRESSION) and using the intra-class correlation coefficient (SPSS RELIABILITY ICC MODEL (MIXED)). eHealth literacy scale scores were modestly correlated between administrations of the eHEALS, ranging from 9 to .68  . The int                    726 = 2.236, P = .026); however, no statistically significant differences were detected in scores at post-intervention and 3 and 6 month follow-up administrations of the eHEALS. Age did not predict eHealth literacy scores at any time point. No significant relationship was found between eHealth literacy and use of information technology overall or with respect to any individual forms of technology surveyed  (P = .05). eHealth literacy levels were also not related to overall self-evaluations of health and were not a significant predictor of perceived health status over time in this sample.Baseline levels of eHealth literacy were higher among males (The eHEALS has shown promise as a measure of the concept of eHealth literacy as defined as a set of skills required to effectively engage information technology for health. The eHEALS’ high levels of internal consistency and modest test-retest reliability suggests that it has utility in examining eHealth literacy over time to both assess natural histories and evaluate eHealth literacy intervention outcomes. While tools exist that enable consumers to critically evaluate eHealth resources , there reHealth literacy promotion takes place within a larger learning context, thus it makes sense to develop partnerships with other groups working within other literacy sectors in validating the eHEALS in relation to other measures of literacy, social functioning, health, and well-being. Two examples of such multi-sectoral partnerships include the National Literacy and Health Program sponsored by the Canadian Public Health Association  and the Conducting this study as part of a larger trial did pose problems for test-retest reliability; therefore, these results should be interpreted with caution. The lower than expected test-retest correlations between administrations of the eHEALS is attributed to a rise in eHealth literacy scores from baseline to post-intervention follow-ups, attributed to the smoking prevention intervention used in collecting the data . AlthougAdditional studies are required to longitudinally examine the eHEALS in study conditions that are not susceptible to influence of the characteristics of a specific intervention. Testing the eHEALS with a population that has high rates of information technology presents a limitation; however, it also provides an opportunity to understand the robustness of the measure within a specific population. Further research needs to consider the eHEALS’ application to other populations as well as groups with highly variable levels of technology familiarity.perceived skills and comfort with eHealth, not the skills directly. The eHealth literacy model includes six types of literacy, and thus each skill would require independent measurement, such as rigorous usability tests of standard computer equipment for computer literacy and reading aloud text passages to assess basic prose literacy. For health practitioners and consumers alike, such detailed assessment would be problematic in practice; however, it is worthwhile considering ways to conduct such measures in the future.The eHEALS measures consumers’ The need for skills in seeking, appraising, and applying lessons learned through use of eHealth resources is common across ages, genders, and cultural groups, and thus the potential applicability of the eHEALS as a standard assessment tool for gauging eHealth literacy in health care is high. Assessing consumers’ comfort in using eHealth allows for the identification of skill gaps and can better assist those with low comfort levels in taking advantage of the potential benefits that eHealth can afford. Doing so may foster development of tools that can meet these needs and aid in creating appropriate strategies for bridging the digital divide in consumer health care quality. Only by increasing the understanding of the disparities between available eHealth tools and consumers’ abilities to use them can the necessary steps towards eliminating them be taken.


To examine the rates of evidence-supported care provided in an obstetrics-gynaecology unit.The main diagnosis-intervention set was established for a sample of 325 consecutive inpatient admissions in 1998–99 in a prospective study in a UK tertiary care centre. A comprehensive literature search was conducted to obtain the evidence supporting the intervention categorised according to the following hierarchy: Grade A, care supported by evidence from randomised controlled trials; Grade B, care supported by evidence from controlled observational studies and convincing non-randomised evidence; and Grade C, care without substantial research evidence.Of the 325 admissions, in 135 (42%) the quality of care was based on Grade A evidence, in 157 (48%) it was based on Grade B evidence, and in 33 (10%) it was based on Grade C evidence. The patterns of care were not different amongst patients sampled in 1998 and 1999.A significant majority (90%) of obstetric and gynaecological care was found to be supported by substantial research evidence. The practice of medicine is underpinned by clinical experience, intuitive clinical reasoning and applied medical research. In recent times, there has been a shift towards evidence-based medicine (EBM) defined as the conscientious, explicit and judicious use of contemporary best research evidence in making decisions about the care of individual patients . To whatAll inpatient admissions at a tertiary care centre in 1998 and 1999 during the first week of October were reviewed. Our approach replicated previous study designs -7. We seMedical literature was searched and examined by two of us (NM and ATK) to determine whether or not the main diagnosis-intervention set for each admission was supported by research evidence using a systematic hierarchical approach . Any disKey references from the search were used to assign each diagnosis-intervention set to one of the following three categories of evidence: Grade A, care supported by evidence from randomised controlled trials (RCTs); Grade B, care supported by evidence from controlled observational studies as well as conditions where there were no controlled data in support of the interventions but there were convincing biological or basic research evidence such that controlled trials would be unnecessary or unethical; Grade C, care without substantial research evidence. Grade A evidence was presumed to be of the highest quality. Grade B evidence including non-randomised prospective cohort studies and large retrospective comparative studies, was second best. Where Grade B was based on inherent validity of interventions we sought consensus from two to three consultants. Both Grade A and B were considered substantial supportive evidence. We computed rates of evidence-based practice separately for the two sampling periods and sought for examples of methodological pitfalls, which we discuss while examining the validity of our findings.Of the complete list of 325 consecutive inpatient admissions Table  the careMedical practice has been criticised as not being based on solid evidence . No publThe validity of our findings depends on the methodological robustness of research designs used to evaluate evidence-based practice -7. The sEBM requires application of knowledge of medical informatics  and clinical epidemiology  along with intuition and experience to improve decision-making for individual patients . ProvisiKSK designed the study, supervised the work, analysed and interpreted the data, and helped draft the manuscript; ATK and MNM extracted the data, transferred it to a database designed by MB, and drafted the first manuscript; MNM extracted the data and transferred it to a database designed by MB; A-MG helped design the study and coordinated the work; and MB helped design the study, developed and maintained the study database, and coordinated and supervised extraction of information from case notes and literature. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:


The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the thirteenth of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.We reviewed the literature on applicability, transferability, and adaptation of guidelines.We searched five databases for existing systematic reviews and relevant primary methodological research. We reviewed the titles of all citations and retrieved abstracts and full text articles if the citations appeared relevant to the topic. We checked the reference lists of articles relevant to the questions and used snowballing as a technique to obtain additional information. We used the definition "coming from, concerning or belonging to at least two or all nations" for the term international. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.We did not identify systematic reviews addressing the key questions. We found individual studies and projects published in the peer reviewed literature and on the Internet.Should WHO develop international recommendations?• Resources for developing high quality recommendations are limited. Internationally developed recommendations can facilitate access to and pooling of resources, reduce unnecessary duplication, and involve international scientists.• Priority should be given to international health problems and problems that are important in low and middle-income countries, where these advantages are likely to be greatest.• Factors that influence the transferability of recommendations across different settings should be considered systematically and flagged, including modifying factors, important variation in needs, values, costs and the availability of resources.What should be done centrally and locally?• The preparation of systematic reviews and evidence profiles should be coordinated centrally, in collaboration with organizations that produce systematic reviews. Centrally developed evidence profiles should be adaptable to specific local circumstances.• Consideration should be given to models that involve central coordination with work being undertaken by centres located throughout the world.• While needs, availability of resources, costs, the presence of modifying factors and values need to be assessed locally, support for undertaking these assessments may be needed to make guidelines applicable.• WHO should provide local support for adapting and implementing recommendations by developing tools, building capacity, learning from international experience, and through international networks that support evidence-informed health policies, such as the Evidence-informed Policy Network (EVIPNet).How should recommendations be adapted?• WHO should provide detailed guidance for adaptation of international recommendations.• Local adaptation processes should be systematic and transparent, they should involve stakeholders, and they should report the key factors that influence decisions, including those flagged in international guidelines, and the reasons for any modifications that are made. The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the thirteenth of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.Adaptation involves modification according to different circumstances or environmental conditions . In the Article II of the World Health Organization (WHO) Constitution defines "setting, validating, monitoring and pursuing the proper implementation of norms and standards" as core functions of the WHO . AccordiDeveloping guidelines internationally poses challenges to ensure and monitor that WHO's guidelines are locally applicable or adaptable across different settings. The needs for adapting guidelines identified in the US survey referred to above are even greater for international guidelines. Organisations such as WHO that develop international guidelines need to consider variations in the contexts in which the guidelines will be applied, including differences in needs, values and the availability of resources. In this paper we addressed the following questions:• Should WHO develop international recommendations?• What should be done centrally and locally?• How should recommendations be adapted?Questions related to guideline implementation and dissemination are specifically addressed in another paper in this series .The Guidelines for WHO Guidelines (GWG) note that "Governments have as their main responsibility the health of the population, rather than the disease of the individual, and must consider other factors in addition to the traditional concern for maximizing the benefit to individual patients. WHO needs to assess the implications for population health of any recommendation as well. This requires explicit recognition that resources to provide health interventions  are limited. This involves considering the cost-effectiveness of alternative interventions, the opportunity costs of investing in one intervention versus another, the affordability of the interventions, and the feasibility of applying a set of recommendations in different settings ."The GWG also recognizes that "WHO takes a global perspective in addressing the needs of (192) member states. Differences in outcome will not only be due to transferring results from a research to a field setting, but also from the different cultural, economic, socio-demographic contexts present in the member-states," and that for "WHO guidelines, the traditional approach of reviewing and reporting evidence on efficacy and safety is certainly crucial but not sufficient. It can be regarded as the first step, but it is also necessary to examine the implications of applying each possible set of recommendations on a population basis. The initial body of evidence to be considered in WHO guidelines will be identical to that of traditional guidelines, but WHO guidelines will need to go further, to take the second step of spelling out the implications of adopting recommendations on costs and on population health. If done adequately, this will allow decision makers in different settings to take the third step of "localizing" the guidelines to their settings, and deciding where the trade-off between additional benefit and additional costs should be set. It will also be useful in determining what is acceptable for the end-users." However, one of the limitations of the GWG is that they have not been operationalized or implemented consistently. Only few WHO guideline processes have followed the GWG .Indeed, the need for international development with local adaptation is expressed in WHO statements such as "The strategy recommends a prevention-oriented approach that emphasizes the need for countries to develop coherent, multi-sectoral national strategies with a long-term, sustainable perspective, to make the healthy choices the preferred alternatives at both the individual and community level. We welcome the commitment shown by Member States to the strategy and will be working closely with them to help them implement its recommendations ."For example, the Global Strategy on Diet, Physical Activity and Health states that "the purpose of the Regional Consultations with Member States is for countries in each region to provide information on the extent of the problem associated with diet, physical activity and chronic disease, and appropriate prevention strategies for their particular countries. The consultation will focus on the discussion of national, regional and international interventions that will be effective within individual countries and that will take account of national, social, cultural and economic realities. Regional differences, common concerns, or international consensus, will be noted and serve as the basis of the development of the Global Strategy. This consultation process will build on past and current activities and programmes on the issue carried out by WHO Regional Offices and by Member States ."WHO also provides funding and support for specific guideline adaptation efforts. Specific WHO guideline adaptation projects exist in the area of HIV that have been supported by international workshops organized by regional offices with country involvement ,12. AnotWe are not aware of published surveys that address what other organisations do to ensure appropriate adaptation of guidelines. The items that are part of the AGREE instrument  include • The potential organisational barriers in applying the recommendations should be discussed.• The potential cost implications of applying the recommendations should be considered.• The guideline should present key review criteria for monitoring and audit purposes.The Conference on Guideline Standardization (COGS) checklist for reporting clinical practice guidelines suggests that guidelines should: "Describe the intended users of the guideline  and the settings in which the guideline is intended to be used" . The cheThere is, however, a growing interest in considerations of how to adapt guidelines -22. ThisIn this background section we provide selected examples of organizations that have specifically provided information relevant to the key questions we posed although the WHO is unique in that its mandate includes the more complex task of providing international guidance.The Scottish Intercollegiate Guideline Network (SIGN) asks guideline panels to consider issues of applicability when guideline groups summarize their view of the total body of evidence . The gui• Generalisability of study findings• Directness of application to the target population for the guideline.• Clinical impact • Implementability SIGN guideline development groups are provided with a form in which to record the main points from their considered judgement . Once thThe process recommended by the New Zealand Guideline Group includes the following steps figure  when ada• Appraise the guidelines (using AGREE instrument) for quality and process• Analyse the content for scope and applicability∘ Same health settings, professional groups?∘ Same patients, consumers?∘ Same interventions?∘ Same outcomes?• Look at the gaps in the issues covered in the overseas guidelines∘ Any clinical questions not covered?∘ Look at the sources of evidence∘ Is the search strategy available?∘ Are there any evidence tables?• Are the evidence statements and recommendations referenced?• Re-run the search strategy to include the questions selected∘ To include literature at least one year prior to the date of publishing∘ Check if any large study would radically change the recommendations• Implementation planning∘ Redesign the implementation plan to meet local circumstancesGuide should not be viewed as the sole source for informed public health decision making because local contextual information is also important. Many issues not addressed in the Guide will affect which interventions are implemented . However, the Guide provides systematically collected and detailed information on several issues of importance to public health practitioners and decision makers; information which is difficult or inefficient to develop locally. Guide reviews and recommendations will be most useful in conjunction with a participatory community planning process that clarifies needs and goals and that considers the Guide's evidence reviews and recommendations in conjunction with additional applicable community specific information."The Guide to Community Preventive Services states the following in a discussion of its methods : "The GuThe methods used to prepare this review are described in the introduction to this series . BrieflyWe did not identify systematic reviews addressing the key questions. We found individual studies and projects published in the peer reviewed literature and on the Internet that we will use to illustrate the responses to the key questions.Threats of new emerging diseases (e.g. severe acute respiratory syndrome (SARS) and avian influenza A (H5N1) infection) as well as pandemics of chronic diseases such as obesity and heart disease have prompted international action and are clear examples of the existence of international health problems and the need for international recommendations. Given the international orientation of WHO and the advantages of large international organizations , there is an important role for international recommendations . InternaAn illustration of the need for adaptation is provided by Rhinehart and colleagues who attempted to implement a nosocomial infection control program based on the US Centers for Disease Control (CDC) guidelines in an urban Indonesian public hospital . AdoptioGlobal recommendations should apply to most settings yet allow for adaptation to local circumstances. The factors that influence recommendations should be laid out explicitly. If differences in context are likely to lead to different recommendations or decisions, these should be flagged ,35. ContThe research evidence on what reflects the best distribution of responsibilities during the development of international guidelines is sparse. Global evidence (i.e. the best evidence from around the world) is the best starting point for judgements about effects and likely modifying factors. Synthesizing and making available this evidence should be coordinated centrally, although the actual work can be done anywhere. For example, the Agency for Health Care and Quality (AHRQ) funds Evidence-based Practice Centres throughout the US and Canada with the methodological competency to undertake systematic reviews, the UK National Institute for Health and Clinical Excellence (NICE) funds National Collaborating Centres responsible for guideline development , and the Cochrane Collaboration has 50 Collaborative Review Groups spread around the world that are responsible for preparing and updating systematic reviews . WHO could adapt, commission or prepare systematic reviews that are required for guideline development in collaboration with organizations such as these that conduct systematic reviews and follow suggestions to make these reviews more useful for policymakers . Lavis aSimilarly, adaptable evidence profiles  based onApparently successful examples of collaboration between central and country level groups exist. Wabitsch and colleagues described that during the adaptation of the WHO global HIV/AIDS guidelines  standardIn addition to supporting appropriate adaptation of its own guidelines, WHO should consider adapting guidelines developed by other organizations, given the potential value of WHO endorsement and savings, if high quality guidelines already exist. Detailed guidance on appropriate methods for adapting guidelines would help WHO guideline groups to adapt existing guidelines, when this is appropriate. Adaptation of recommendations is required because several judgments influence recommendations. Therefore, recommendations dealing with identical questions may differ between developers despite reliance on the same evidence. Implementation, which follows the process of adaptation, is topic of another paper in this series and tools for the evaluation of implementation of guidelines have been developed ,40.Decisions during local adaptation processes should be transparent and follow procedures that are similar to those used in developing the guidelines, including reporting the key factors that influence any modifications. Two fairly similar approaches have recently appeared that produced frameworks for identifying candidate guidelines for local adaptation. The Practice Guideline Evaluation and Adaptation Cycle (PGEAC) is a 10 step approach figure  17,41,4,441,42. . Whatever adaptation process is chosen the process should be made explicit, undergo review by peers, and involve consumers, policymakers and other stakeholders who may provide input about policy guidance.The other approach has been developed by the international working group ADAPTE ,19,22 anBalance sheets or evidence profiles are designed to assist decision-makers regarding outcomes in their practice setting ,44. For Given WHO's mandate; limited resources that are available to develop high quality guidelines that are informed by the best available evidence, particularly in LMIC; and the potential to reduce unnecessary duplication, WHO should continue to develop international guidelines. However, these guidelines will often require adaptation and tailoring to local contexts and WHO should, so far as possible, provide support to help ensure that international guidelines are adapted appropriately to local circumstances. To do this WHO must ensure that it systematically considers needs for local adaptation when developing guidelines; and that it has sufficient capacity to support both developing and supporting the adaptation of high priority guidelines. To do this as effectively and efficiently as possible WHO should collaborate with other organizations both in developing guidelines and, importantly, in developing capacity in LMIC and supporting appropriate adaptation in countries that lack resources. We provide other recommendations about how WHO can improve the implementation of organizational changes to guideline development in other articles in this series ,46-50.WHO has ample experience in adapting guidelines but this effort should be coordinated and disseminated among WHO guideline groups. Through coordinating guideline development within WHO and collaborating with other organizations, WHO could capitalise on this experience, improve the quality of its guidelines, and help to ensure that its guidelines are appropriately adapted and result in appropriate actions and health improvements. Similar suggestions have been made previously by investigators involved with WHO guideline projects. For example, "Countries should discuss and find ways of collaboration and formation of linkages and support with National HIV/AIDS program in order to enhance the implementation of IMCI algorithm which includes HIV/AIDS" .A systematic review of studies evaluating methods for adaptating guidelines is unlikely to retrieve high quality evidence given the paucity of research in this area. Further development and evaluation of frameworks and tools to support the appropriate adaptation of guidelines is needed. Given the limited capacity for this in many countries, comparisons of simpler processes that require fewer resources should be compared with more rigorous processes to determine the most efficient methods for ensuring that guidelines support well-informed decisions and actions appropriate for the specific contexts in which they are taken.ADO and AF work for the Norwegian Knowledge Centre forthe Health Services, an agency funded by the Norwegian government that produces systematic reviews and health technology assessments. All three authors are contributors to the Cochrane Collaboration. ADO and HJS are members of the GRADE Working Group. HJS is documents editor and chair of the documents development and implementation committee for the American Thoracic Society and senior editor of the American College of Chest Physicians' Antithrombotic and Thrombolytic Therapy Guidelines.HJS prepared the first draft of this review. AF and ADO contributed to drafting and revising it. All authors read and approved the final manuscript.


Since health information on the World Wide Web is of variable quality, methods are needed to assist consumers to identify health websites containing evidence-based information. Manual assessment tools may assist consumers to evaluate the quality of sites. However, these tools are poorly validated and often impractical. There is a need to develop better consumer tools, and in particular to explore the potential of automated procedures for evaluating the quality of health information on the web.This study (1) describes the development of an automated quality assessment procedure (AQA) designed to automatically rank depression websites according to their evidence-based quality; (2) evaluates the validity of the AQA relative to human rated evidence-based quality scores; and (3) compares the validity of Google PageRank and the AQA as indicators of evidence-based quality.The AQA was developed using a quality feedback technique and a set of training websites previously rated manually according to their concordance with statements in the Oxford University Centre for Evidence-Based Mental Health’s guidelines for treating depression. The validation phase involved 30 websites compiled from the DMOZ, Yahoo! and LookSmart Depression Directories by randomly selecting six sites from each of the Google PageRank bands of 0, 1-2, 3-4, 5-6 and 7-8. Evidence-based ratings from two independent raters (based on concordance with the Oxford guidelines) were then compared with scores derived from the automated AQA and Google algorithms. There was no overlap in the websites used in the training and validation phases of the study.P<.001). Addition of a quadratic component improved the fit, the combined linear and quadratic model explaining 82 percent of the variance. The correlation between Google PageRank and the evidence-based score was lower than that for the AQA. When sites with zero PageRanks were included the association was weak and non-significant . When sites with zero PageRanks were excluded, the correlation was moderate .The correlation between the AQA score and the evidence-based ratings was high and significant (r=0.85, Depression websites of different evidence-based quality can be differentiated using an automated system. If replicable, generalizable to other health conditions and deployed in a consumer-friendly form, the automated procedure described here could represent an important advance for consumers of Internet medical information. At least 50% of households in the United States, the United Kingdom and Australia are connected to the Internet -3. In adTo date, however, few health information websites have been subjected to rigorous assessment of their effectiveness in improving health outcomes. In the absence of such evidence, a minimum requirement for a high quality health information website should be that its content is evidence-based; that is, that its content and recommendations are consistent with evidence from a systematic review of the available medical literature. A non-evidence-based site would recommend treatments that are not supported by the evidence as effective, oppose the use of beneficial treatments of demonstrated benefit or fail to mention some effective treatments at all. For example, a depression webpage on the official site of the Church of Scientology claims that Dianetics is “the only proven effective technology of the human mind”  and failEvidence-based health information is routinely disseminated to health professionals with the aim of assisting clinical decision making, improving healthcare and thereby improving health outcomes. Such evidence, when provided to consumers, has the potential to improve health outcomes by assisting consumers to select effective, rather than ineffective self-help techniques and by supporting shared decision making and consumer-provider collaborative care . UnfortuVarious mechanisms have been proposed for assisting consumers to access high quality health websites . These iA potential solution to these problems is to develop assessment tools based on algorithms that automatically evaluate the quality of health information websites. To date there has been little work directed to this possibility. There is some evidence of a relationship between Google Page Rank and evidence-based quality from two recent studies , 20. HowThis paper describes the development of a computer algorithm, the Automatic Quality Assessment procedure (AQA), designed to automatically rank depression websites according to the evidence-based quality of their treatment information. In addition, it describes the results of an evaluation of the validity of the AQA as an indicator of human-rated evidence-based quality of treatment content. It also compares the validity of the AQA and Google PageRank as indicators of evidence-based quality.The study focused on the evaluation of treatment information in depression websites since depression is a primary source of disability burden  and it hThis section comprises two parts. The first describes the AQA and its development. The second describes the methodology used for the validation of the AQA as an indicator of evidence-based treatment quality. The evidence-based rating scale  employedIn the following we present the procedure for calculating AQA scores and note its dependence upon two learned queries and three numerical parameters. We then describe the development phase during which the queries were learned and the parameter values chosen. The development phase employed websites/webpages not in the validation set but for which we had collected human-rated relevance or evidence-based quality measures from our previous studies , 19, 24.The AQA assumes the availability of search engine software that incorporates a web crawler and has the ability to effectively score the relevance of documents to a query. The current study employed the Panoptic search engine for this purpose. However, we believe that other similar search engines could be substituted with similar results. A set of computer scripts were written to learn queries, to set values of tuning parameters and to collect and analyze output from the search engine. These scripts are not part of the Panoptic search engine.The AQA procedure comprised six steps as follows:The target websites were downloaded using web crawler software;These downloaded pages were aggregated with a large set of arbitrarily chosen general English language web pages and the resulting collection indexed using the search engine. This was necessary to avoid the extremely biased term frequency distribution of a depression-only collection;r) computed.A previously learned relevance query (see below) was processed over the collection created in Step 2 using the search engine to produce relevance scores for all documents. The relevance query consisted of many words and phrases, each with a numerical importance weighting. Documents with non-zero scores were not retrieved. For each site to be evaluated, the number of retrieved documents |R| were counted and the mean relevance score  was processed in the same fashion as in Step 3, yielding |Q| and r became 1.0 and the highest Sq was also 1.0.Site relevance and site quality scores were computed using Equations (1) and (2). These scores were normalized such that the highest SAn overall site score was computed using Equation (3). Gamma is a scaling parameter designed to make scores comparable with those from the human rating scale.rS=α × r + (1-α)× |R|Equation 1: qS= α × q + (1-α)× |Q|Equation 2: S = γ × (β × qS + (1-β)×rS)Equation 3: α, β and γ chosen.The following sections describe how relevance and quality queries were learned and the values of Relevance and quality queries were learned using an extension and novel application of the relevance feedback technique from the field of information retrieval. In the relevance feedback approach, a complex query consisting of weighted terms (words and phrases), is automatically generated by comparing the term frequency distributions of sets of relevant and irrelevant documents. Good terms occur frequently in relevant text but seldom otherwise. The resulting query is used by a text retrieval system to derive relevance scores for documents. We extended this method to learn a 'quality' query from sets of high and low quality webpages.Relevance query: During development of the relevance query, query terms were selected by computing Term Selection Values (TSVs) St John's Wort appears to be as effective as tricyclic antidepressants and causes fewer side effects, but little is known about any long term adverse effects.Cognitive therapy can be an effective treatment for depression.Cognitive behaviour therapy is at least as effective as drug treatment in mild-to-moderate depression.Cognitive behaviour therapy may be valuable for people who respond to the concept of Cognitive behaviour therapy, prefer psychological to antidepressant treatment, have not responded to antidepressant therapy. [Score 1 if mention at least one of these]Problem-solving may be effective for depression.[Generic] counselling is probably no more effective than treatment as usual from the GP for depression.Written information  can improve mild-to-moderate depression. [Score 1 if cognitive model]Exercise can be effective - alone or as an adjunct to other treatments.For each item, score 1 if the site information is consistent with the statement. Cumulate item scores across the scale to yield a total evidence-based score for the site.AQA scores were computed for the test websites by following Steps 1 to 6 of the AQA procedure using the relevance and quality queries and the values of α and β that were derived during training.The Google PageRank was recorded for each home page. Google PageRank is a measure employed by the Google Search Engine company to evaluate the reputation of a webpage. The PageRank is based on a computer algorithm that computes iteratively the number and importance of links to a webpage and in turn the number and importance of links to these linking pages . As noteThe sample size was considered sufficient to justify meaningful parametric analysis of the data. Intercorrelations between variables were computed using Pearson correlation tests. The validity of the automatic measure as an indicator of evidence-based quality was evaluated using hierarchical multiple regression. These analyses were performed using SPSS 12.0.1 . Tests oThe characteristics of the 30 depression test sites used in the validation are summarized in The mean (and standard deviation) of the evidence-based, Google PageRank and AQA scores were 5.92 , 3.67  and 8.07  respectively. AQA scores were available for 29 test sites only as one website included a robots.txt exclusion, an indicator that the administrator of the site prohibited external crawlers from accessing the website.P < .001).The relationship between the AQA score and the evidence-based ratings is shown in 2 = 0.10, F∆  = 14.3, P = .001), the combined linear and quadratic model explaining 82 percent of the variance.Visual inspection of the scatterplot in P = .22, n = 30; see P = .0001).By contrast, the correlation between Google PageRank and the evidence-based score was small and non-significant , but remained significantly lower than the association between the AQA and evidence-based quality scores (r(difference) (df = 20) = 0.22, t = 2.61, P = .02).Since it has been argued by some members of the search engine optimisation community that PageRanks of 0 may constitute a special subset of PageRank values (see Comment below), the above analyses were recomputed after excluding sites with PageRank of 0. The association between the PageRank and evidence-based scores for remaining sites was significant . In addition, the system may prove useful as a screening device for the use of web developers interested in maintaining quality health portals or links of high quality. Once filtered by the automatic quality evaluator, developers could evaluate the remaining sites manually to confirm accuracy of content, and to assess sites according to other dimensions of quality  and according to the needs of the organization and its users.Consistent with previous studies , 20 we hThe current tool is relevant only for identifying the quality of depression websites. However, the relevance feedback method used in the AQA is likely to generalize to other areas of mental health and medicine. To apply the AQA in alternative health domains requires that the procedure is trained to learn new terms and parameter values specific to the new domain. The validity of the technique in these other domains is a matter for empirical investigation.There are some limitations of the system described here. First, some minor changes to the AQA scaling procedure are necessary before the system is used in practice . Applying a non-linear transformation to the raw AQA scores  might result in a better prediction of the values of the evidence-based scores. However, given that the base (linear) correlation coefficient is already very high (0.85), the scope for improvement is limited. Secondly, the AQA scores could be compromised if publishers use “spamming” methods for optimizing their automatic quality scores. General search engines are faced with a similar problem when website developers attempt to artificially inflate the relevance rankings for their websites. This problem is not peculiar to automated methods for processing websites. It is likely that website developers also use strategies to maximize their scores on manually applied quality evaluation tools while leaving unchanged the substantive content. Public search engines incorporate algorithms for detecting attempts to distort rankings. It is likely that the AQA could be refined to do the same. The third limitation of the AQA system is that it is focused on treatment information (as indeed is DISCERN) and does not necessarily reflect the quality of other information on a site. In addition, the system may not adequately rate sites which present only one treatment type. The system will be most useful in identifying sites containing high quality, comprehensive treatment information. Another limitation relates to the gold standard employed for rating the evidence-based quality of the sites. It might be argued that to be considered valid, the rating system should be validated against health outcomes or another scale that has been thus validated. No such scales exist, and given the paucity of efficacy studies of websites, such validation exercises are not currently practical. Finally, one of the evidence-based raters had been involved in the initial selection of the sites which may have led to bias in the findings from the validity studies. However, the pattern of results and conclusions is identical if the findings are recomputed using the blinded rater’s data only. This is not surprising given the higher interrater reliability in this study for the evidence-based scale.The time has come to acknowledge that consumers do and will continue to use the Internet as a source of health information. We need to provide them with convenient, effective tools that optimize the usefulness of this process. This study demonstrates that automated methods offer considerable promise in this respect. The task before us now is to refine these methods, and to test the usability, robustness and generalizability of the systems we develop. In the process we need to test alternative strategies for quality filtering, to identify if multiple methods when combined improve the validity of the automatic algorithm, and to evaluate whether the techniques generalize across health domains. We must also construct a user interface for the procedure, and conduct consumer user and satisfaction studies on the resulting system.