Optimizing pain care requires ready access and use of best evidence within and across different disciplines and settings. The purpose of this randomized trial is to determine whether a technology-based “push” of new, high-quality pain research to physicians, nurses, and rehabilitation and psychology professionals results in better knowledge and clinical decision making around pain, when offered in addition to traditional “pull” evidence technology. A secondary objective is to identify disciplinary variations in response to evidence and differences in the patterns of accessing research evidence.Physicians, nurses, occupational/physical therapists, and psychologists (n = 670) will be randomly allocated in a crossover design to receive a pain evidence resource in one of two different ways. Evidence is extracted from medical, nursing, psychology, and rehabilitation journals; appraised for quality/relevance; and sent out (PUSHed) to clinicians by email alerts or available for searches of the accumulated database (PULL). Participants are allocated to either PULL or PUSH + PULL in a randomized crossover design. The PULL intervention has a similar interface but does not send alerts; clinicians can only go to the site and enter search terms to retrieve evidence from the cumulative and continuously updated online database. Upon entry to the trial, there is three months of access to PULL, then random allocation. After six months, crossover takes place. The study ends with a final three months of access to PUSH + PULL. The primary outcomes are uptake and application of evidence. Uptake will be determined by embedded tracking of what research is accessed during use of the intervention. A random subset of 30 participants/ discipline will undergo chart-stimulated recall to assess the nature and depth of evidence utilization in actual case management at baseline and 9 months. A different random subset of 30 participants/ discipline will be tested for their skills in accessing evidence using a standardized simulation test . Secondary outcomes include usage and self-reported evidence-based practice attitudes and behaviors measured at baseline, 3, 9, 15 and 18 months.The trial will inform our understanding of information preferences and behaviors across disciplines/practice settings. If this intervention is effective, sustained support will be sought from professional/health system initiatives with an interest in optimizing pain management.Registered as NCT01348802 on clinicaltrials.gov. Acute and chronic pain affects a large number of Canadians. The prevalence of chronic pain in Canadian adults has been estimated to range from 18–29% , with 60There is considerable variation in practice and widespread patient dissatisfaction with the way pain is managed across the spectrum, from relatively benign conditions to terminal illness. For example, in low back pain , unresolMany clinicians are not aware of recent pain research, and misconceptions are common ,10. In fPain management is central to health disciplines. Physicians, nurses, and rehabilitation professionals are commonly involved in managing acute and chronic pain . Pain isEvidence-based practice (EBP) can narrow the gap between research knowledge and practice . StudiesProfessional associations and individual clinicians in rehabilitation are highly supportive of EBP ,32,38-45Traditional approaches to knowledge translation (KT) in EBP have focused on providing knowledge and technical EBP “skills” to practitioners, leaving them with the burden of searching and appraising evidence ,52-56. SEvidence-based decision making originated within medicine and evolved to other professions. Limited studies, mostly in the United Kingdom, have addressed differences in attitude or awareness across professions ,62. FamiSystematic reviews suggest technology-based EBP can enhance care ,64,65, aThe primary objective of this randomized crossover trial is to evaluate the incremental effect of a PUSH evidence resource (Pain PLUS) across professions  as compared to a standard (PULL) approach. The primary outcomes are uptake and application of evidence, and they will be assessed at baseline, 3, 9, 15 and 18 months.The trial is a crossover randomized controlled trial (RCT) with repeated baseline and follow-up measurements Figure . The intTraditional RCT methodology supports a single primary outcome measure, but a theoretical framework for outcome evaluation was based on the conceptual framework of technology-based KT ,69. Ther1. Use of technology : Embedded tracking will record the retrieval behavior .2. Perceived usefulness : Users will respond to randomly embedded judgments about the usefulness of their session experience.3. Skills/behaviors in accessing information: An observational test of information-searching behaviors measuring how different professions access evidence for specific clinical questions and the type, quality, and usefulness of evidence retrieved.4. Application: Evidence-based decision makinga. Participants’ EBP behaviors and decision making will be measured by a standardized self-report scale.b. Chart-stimulated recall will measure competency for using an evidence-based approach to address problems within participants’ own clinical practice.Our secondary objective is to evaluate professional differences in access/use of research. Therefore, we will evaluate whether the primary outcome measures—either at baseline or in response to the intervention—are modified by professional group. Further, we will perform a structured classification of the types of studies accessed by different professions to look at variations in the type of evidence valued.We expect that (1) retrieval and application of pain evidence and satisfaction with evidence resources will improve more with push-out intervention and (2) there will be differences between professions in attitudes towards EBP, response to evidence tools/resources, and types of evidence most accessed.All participants will receive unique IDs and passwords and will be able to log into an evidence resource, Pain PLUS Figure , to searpain content of the service but can also select email frequency and the cut-off for relevance and quality to control the degree of relevance and frequency of evidence they receive. Their experience will be a customized notification of new research . All study participants will be signed into the h Figure .Users will “see” a similar format when they access the pull intervention, but this “placebo control” website will not include any individualized information, nor will they receive alert emails. This group can access new research but must search for it.Computerized randomization will allocate individuals to intervention group at three months; crossover is automatic at nine months.Eligible practitioners must (1) be physicians, nurses, occupational therapists (OTs), physical therapists (PTs), or psychologists who are currently working in clinical practice at least one day/week; (2) be fluent in English; (3) have access to a computer at home or at work that has unrestricted access to the World Wide Web; and (4) have an active email account. Clinicians who meet the eligibility criteria are enrolled in the study by the research assistant.We will use an evaluation framework developed specifically for technology-enabled knowledge translation -70 to evStructural usability analysis will have some embedded questions on functionality but will primarily be based on tracking of how the technology is used. We will have data on the number of logins and number of pain and other  research studies in the database that are accessed. These data will be summarized by the average frequency of use per month and mean number of studies accessed. Other utilization outcomes that will be tracked by the PLUS software include the change in the average frequency of use, average number of keystrokes per month, and the average number of times practitioners use certain specific electronic resources. Collection of these outcomes will be software regulated and, thus, not require any participant burden.The subjective elements will focus on perceived usefulness of individual sessions. Participants will be asked at the end of each use to answer a single question with a 7-point scale, indicating the extent to which the service provided useful information. If some users fail to log off the system and invoke the final question, we will send them a query by email.There are few well-validated outcomes assessments of EBP, knowledge acquisition, or attitude/behavior change. In this study, cognitive elements will be assessed using a previously validated scale containing cognitive and behavioral questions specifically related to EBP . This quBehavior change is the ultimate goal of KT. It will be assessed through self-report, performance tests, and observer-based competency assessments. EBP behaviors will be self-reported using the Behavior Subscale of the Knowledge Attitudes and Behaviour Questionnaire (KABQ) validated in different professions . Similare.g., percent of relevant articles, time to find), and effectiveness of their search . Volunteers will rate the usefulness of the research located. Our previous study determined that Clinical Queries was more efficient for physicians [We will directly measure respondents’ skill in accessing information by providing a clinical scenario where a clinical question needs to be addressed and asking participants to search for research information to address that request. We will monitor their approach to searching and their satisfaction and intention to act upon the information acquired. This test will replicate previous methods used to compare information retrieval by physicians on Clinical Queries versus PubMed . A subseysicians . In thisWe will obtain scores and a more detailed description of actual application of pain research evidence use before and after the intervention by performing chart-stimulated recall (CSR). This approach yields a rich and deep understanding of how evidence is used in practice based on semistructured interviews that probe how/why clinicians make choices about cases retrieved from their own records. A predetermined set of competencies are probed during the interview, and the performance is rated on a 7-point Likert scale. The score provides a quantitative estimate of competency . We have successfully coded previous CSR interviews using content-analysis techniques. We have successfully used CSR in a KT study  to evaluThe KABQ will be administered at baseline, 3, 9, 15, and 18 months. Other measures are embedded and distributed over time. CSR will be performed in a subset of 30/group during the first randomization cycle (at baseline and nine months). The skill assessment will be conducted in the final three months of the trial to allow maximum skill development to occur so that our comparisons of professional differences will be less affected by learning curves.It is important to monitor and potentially adjust for important covariates that influence study outcomes. We will measure the following variables for each clinician: baseline knowledge/attitudes towards EBP, years of practice, highest degree, and comfort with technology. Organizational variables will also be collected at baseline, including rural/urban location, clinic size, access to online journals, peer support, administrative support, and management support. These variables will be entered into an analysis of covariance examining pre- and post-intervention effects. Our sample size is sufficiently large that we will be able to assess the primary effects of these variables, as well as interactions between organizational variables and participant variables.R2 of 30% corresponds to an effect size of 0.42; 600 practitioners provides more than 99% power to detect this effect size with 12 predictors [To determine the appropriate sample size for this trial, the following factors were considered: (1) the primary and secondary outcomes in the trial, (2) clinically significant differences for the outcomes based on our previous RCT, (3) Type I  and Type II (beta) errors, (4) estimates of variation from our pilot data, (5) study design, and (6) dropouts. These calculations represent sample size for the overall comparisons between the four professional groups and two intervention groups. Sample-size estimation was based on detecting an effect size of 0.40 between groups on any of the three aspects of outcome . Assuming Type I error = 0.05 (two-tailed), Type II error = 0.80, and the effect size = 0.40, the sample size required per group is 80. The sample size required for two comparison groups across four professions = 80 × 2 × 4 = 640. To account for our dropout rate (8/431), we have conservatively added 30 to recruit a sample of 670 practitioners. For analysis of response mediators, an edictors .This study is organized locally but recruits participants internationally to inform practitioners of the study and the potential for free access to evidence resources. Our multimodal recruitment strategy will use professional associations, websites/conferences, and social media to inform practitioners of the study and the potential for free access to evidence resources.Participants will complete a baseline screen with the study research assistant, which will be independently reviewed by a trainee for violations. These will be reported (as a protocol diligence indicator), but no participants will be removed. No further rechecks of eligibility or exclusions will be included after this screening.We have attempted to create a blinded control by creating a similar front-face to the website and blinding participants to the nature of the difference in the two resources (push versus pull), although we acknowledge differences will become apparent to users. Since usage data are embedded and automated, they are not influenced by data collectors or social bias. Research assistants and data analysts will be blinded to group allocation for the duration of the study and data analysis. After crossover, subjects will be aware of the difference and, therefore, no longer be blinded. Thus, the overall potential for bias due to lack of blinding has been minimized.Inside the study, participant crossovers are impossible since we control access to the intervention. Contamination that happens by participants accessing open access content for physicians by signing up under an alias for service is possible, but not probable. As part of enrollment, we will ask participants to refrain from using any new evidence access tools/services or EBP training during the study trial and will re-evaluate their reports of services used at the end of each intervention cycle. Co-intervention will be discouraged, including attendance at intensive EBP training .We will use reliable and valid measures of attitudes to EBP that have been used with the studied clinical disciplines ,76,80,81We will use the following measures to ensure minimal loss to follow-up: (1) enrollment/consent will emphasize committing to the entire trial period and will exclude clinicians who cannot commit to the study for 18 months, (2) contact information will be updated at each study interval, and (3) response burden has been kept low. Burden is reduced since brief measures are embedded in the intervention; the total time required for extra questionnaires is relatively low and will be time distributed. The system will initiate regular reminders until the data collection is complete (or the participant withdraws consent).Compliance is one of our monitored variables. When participants fail to use an intervention, they are classified in some studies as noncompliant. We view lack of use as a failure of the intervention to meet participant needs. Thus, participants who do not use this service will not be classified as noncompliant or dropouts. Dropouts will be defined as participants who inform us that they no longer wish to participate in the study follow-up process. These participants will not be denied access to the intervention, and we will request permission to continue to use data from the embedded tracking of usage. Each participant will receive a $45 gift card upon completion of at least three survey assessments: baseline, one follow-up , final assessment. Participants who complete the CSR or the search skills assessment will receive an additional $20 gift card as honorarium for time volunteered.Post hoc pairwise comparisons of the two treatment arms at different time points will be done using multiple comparisons with adjustment to the Type I error rate. Type I error will be set to 5%  for all the calculation of the confidence intervals and performing the various hypothesis-testing procedures. Type I error will be adjusted accordingly for multiple comparisons. Standard diagnostic tools will be used to assess any of the model fittings. An intent-to-treat analysis will be employed—those who do not participate will be included in the analysis in their original group. No interim analyses are planned since there are no safety issues and we wish to ensure full power for both primary and secondary analyses.Quality checking of primary and secondary outcome data will be performed by re-entry of 100 cases and checking concordance. Descriptive statistics, including frequencies, means, and standard deviations, will be calculated and data displayed graphically for all study variables as a means of exploring data distributions. The data for all outcome measures will be analyzed to ensure that they meet the statistical assumptions necessary to use parametric statistics. Transformations may be considered if necessary to meet test assumptions. To test our main research question, we will use general, linear mixed-effects models  for repeWe also plan to analyze difference in professional attitudes, evidence access, and use. We have powered our study to allow this analysis. Our subgroup analyses will also describe differences in types of evidence accessed. Our group previously determined that physicians prefer systematic reviews to primacy studies and that access was better through peer-reviewed journals than through Cochrane Collaboration ; we willThe Steering Committee will consist of the four investigators. The study coordinator will prepare a monthly report, detailing the accrual rate, loss to follow-up, withdrawals, and compliance with study protocols. Since this trial does not directly involve patients, there is no need for a Data Safety Monitoring Committee.We have enrolled more than 70% of the sample target. More than 60% of the participants have reached the nine-month milestone (crossover).This trial should provide unique information on evidence-access skills and professional differences in use of evidence. We expect to demonstrate the effectiveness of push-out targeted evidence (Pain PLUS) and to move the intervention into open access upon trial completion. We expect that some differences in evidence access and application will occur across professional groups, both in terms of types of pain interventions and types of research that are accessed.It is our intent to ensure this intervention becomes open access to have ongoing impact. We believe this trial will inform our understanding on how to best deliver evidence to clinicians across a variety of professions dealing with pain. Sustainability of KT interventions is a critical issue area. As our findings emerge, we will deal with sustainability issues. Since evidence updates have been created for specific disciplines, including physicians, nurses, and rehabilitation professionals, it may be that folding in a pain resource into these existing resources would be the optimal method to ensure sustainability. Conversely, pain stakeholders may prefer a customized resource. A variety of pain resources are being developed, and strategies to share these across stakeholders are just emerging in stakeholder discussions. This may form an opportunity to tailor the knowledge from this study to emerging resources. Using the knowledge about what pain evidence exists, is accessed, and valued across professions will allow us to develop a pain knowledge strategy. The sustainability plan will be designed to maximize uptake of pain knowledge in a sustainable manner.One of the benefits of this study is the ability to compare the evidence accessed by different professions when dealing with the same issue. We will use a descriptive approach to convey these differences by classifying the type of pain, the type of clinical questions, and the type of research design of these studies to convey if different types of evidence are valued by different professions. We anticipate, for example, that nurses and therapists might access more qualitative research, whereas physicians might access more RCTs on drug interventions. Psychologists are being included for the first time in this type of intervention, and their information behaviors are relatively unstudied. We expect that this information might help us understand if evidence scanning and filtering needs to be adjusted across professions. For example, decisions about how to filter qualitative studies may be more challenging than are such decisions in quantitative research.We also acknowledge that the information resources that are available to knowledge users are rapidly expanding. Resources vary in their delivery, quality, customization, source content, and format of information. Knowledge users may become overwhelmed with information resources or stick with ones they have become comfortable with, irrespective of the quality of the information. It may not be clear to end users that the evidence-filtering process used in Pain PLUS protects them against low-quality information. Further, as clinicians often value advice about clinical decision making in actual cases, or implementation issues, they may be drawn to information resources that focus on these rather than the primary or synthesized research available. Since information is always in competition with other information, we cannot be certain that Pain PLUS will win in this competition.JCM, ML, and RBH were involved in the development of the Pain PLUS service. NB has no competing interests to declare.JCM drafted the protocol. All authors participated in the study design and provided feedback on the protocol. All authors read and approved the final manuscript.


This study aimed to evaluate patents as well as high level researches including systematic reviews/meta-analyses and randomized controlled clinical trials (RCT) published in scientific journals by Iranians endodontic.The study started with targeted searches of PubMed as well as World Intellectual Property Organization and United State Patent and Trademark Office.There were 4 filed/granted patents, 2 systematic reviews/meta-analyses and 25 RCTs. Patents were related to endodontic/dental (bio)materials. Performing a topic sorting, 15 RCTs were about vital pulp therapy and 8 about anesthesia and pain. More than 55% of these articles originated from three University of Medical Sciences: Shahid Beheshti (22.2%), Kerman (18.5%) and Mashad (14.8%).Vital pulp therapy was the most important topic amongst endodontic high level evidence articles. Evidence-based practice aims to apply the best available evidence gained from scientific methods to clinical decision making. It appraises the strength of evidence of the risks/benefits of treatments as well as diagnostic tests. Evidence-based dentistry (EBD), as a subcategory, is an approach to oral health-care that needs a careful amalgamation of clinically relevant scientific evidence, relating to the patient’s oral condition/history, with the clinician’s skill as well as the patient’s treatment needs/preferences. EBD is about providing personalized dental care based on the current best evidence. The highest level of evidence available represents the current best evidence for a specific clinical question  EvidenceThe Oxford Centre for Evidence-based Medicine suggests level A of evidence LOE1) for Randomized Controlled Clinical Trial (RCT) . RCT is  for RandScientometrics has been employed broadly for assessing the progress of science/technology. It uses articles and patents as important tools to map the development. A recent quantitative scientometric study showed that endodontic publications from different universities in Iran have considerably increased in PubMed index journals . In addiHowever, a science system report from Iran shows some important qualitative weak points such as low expected levels of citations for Life sciences . On the In this study, we aimed to evaluate systematic reviews and RCTs as the highest level of evidence which were published in PubMed index journals by Iranian authors as well as patent counting in the field of endodontics.MESH words searching method without time limitation was utilized for identifying PubMed-indexed endodontic articles . AbstracThe World Intellectual Property Organization (WIPO) were searched as the main international and United State Patent and Trademark Office (USPTO) as the most important national agency were searched for Iranian inventors who filed or granted the patents.There was only one patent in WIPO, whereas 1 filed and 2 granted patents in USPTO . Two sysSystematic Review is the gold standard for evidence; it provides a summary of individual studies that have answered the same question; as well as provides a method for managing large quantities of data. It has a clear criteria for retrieval/appraisal/synthesis of evidence from individual RCTs as well as other well-controlled trials. A recent study demonstrated that there were 49 systematic review and meta-analysis in the field of endodontics worldwide . Our resRCTs are regarded as the best study designs to test the efficacy of medical/dental treatments. In RCT the subjects are assigned by chance to separate groups that evaluate new and standard treatments; neither the operators nor the patients can choose which group. At the time of the trial, it is not known which treatment protocol is worse/best. The number of RCTs in the field of dentistry or endodontics has not been reported; however it is estimated that ~4-6% of published articles classified as RCT. Our results showed that RCTs from Iran are ~8% of PubMed-indexed published endodontic articles which seems at or even above the global level.The aim of our study was to assess the quantity of high-level evidences in endodontics; therefore, we did not appraise the quality of 25 RCTs. It was reported that the quality of RCT in the field of dentistry is still poor, and more efforts for progress are needed. The trials that are not well designed provide biased estimates of the treatment effects; moreover a journal’s impact factor (IF) is not related to the quality of results . Within Vital pulp therapy (VPT) as well as anesthesia and pain were two main subjects of published Iranian RCTs. VPT in pediatric dentistry is a well established treatment modality. Our data showed that 7 VPT/RCTs were carried out in the field of pediatric to test Formocresol, MTA, Electrosurgery, calcium hydroxide and CEM cement as pulpotomy agents 20]23][[23][20][[[23][20]It was reported that Mashad, Tabriz and Tehran dental schools were the top three institutions when looking at the number of published PubMed-indexed endodontic articles in Iran ; howeverRelevant literature regarding patents scientometrics in endodontics was found not. However, we found 4 patent documents in the field of endodontics originated from Iran in WIPO and USPTO. There are no patents from other dental subspecialties from Iran.It seems that endodontics rank at the first place for science production as well as technology in Iran.


All randomized clinical trials (RCTs) require a protocol; however, numerous studies have highlighted protocol deficiencies. Reporting guidelines may improve the content of research reports and, if developed using robust methods, may increase the utility of reports to stakeholders. The objective of this study was to systematically identify and review RCT protocol guidelines, to assess their characteristics and methods of development, and to compare recommendations.We conducted a systematic review of indexed literature  and a targeted search of supplementary sources, including a survey of major trial funding agencies in six countries. Records were eligible if they described a content guideline in English or French relevant to RCT protocols. Guidelines were excluded if they specified content for protocols for trials of specific procedures or conditions or were intended to assess trial quality. We extracted guideline characteristics and methods. Content was mapped for a subset of guidelines that described development methods or had institutional endorsement.Forty guidelines published in journals, books and institutional reports were included in the review; seven were specific to RCT protocols. Only eight (20%) described development methods which included informal consensus methods, pilot testing and formal validation; no guideline described all of these methods. No guideline described formal consensus methods or a systematic retrieval of empirical evidence to inform its development. The guidelines included a median of 23 concepts per guideline (interquartile range (IQR) = 14 to 34; range = 7 to 109). Among the subset of guidelines (n = 23) for which content was mapped, approximately 380 concepts were explicitly addressed ; range = 16 to 150); most concepts were addressed in a minority of guidelines.Existing guidelines for RCT protocol content varied substantially in their recommendations. Few reports described the methods of guideline development, limiting comparisons of guideline validity. Given the importance of protocols to diverse stakeholders, we believe a systematically developed, evidence-informed guideline for clinical trial protocols is needed. All randomized clinical trials (RCTs) require a protocol describing the rationale, methods, proposed analysis plan and organizational/administrative details from trial inception to reporting of results. Throughout a trial, diverse groups use the trial protocol, including investigators, participants and personnel, funding agencies, research ethics committees/institutional review boards (REC/IRB), journal editors and systematic reviewers. Transparent and clearly written protocols are important to guide trial conduct. They enable thorough assessment of adherence to scientific and ethical standards prior to trial inception -6 and moUnfortunately, a high proportion of trial protocols do not adequately describe important methodological details, decreasing their utility for trial implementation and critical appraisal of trials. For example, protocols often fail to designate primary outcomes  or detaiCONsolidated Standards Of Reporting Trials) ..83].Very few concepts were recommended consistently across guidelines, including several whose importance is supported by empirical evidence. For example, only half of the more recent guidelines ,75,80,81The reasons for the variation and omissions are unclear. Few of the guideline reports in our sample described their development methods, preventing assessment of the validity of the recommendations. If not properly developed, guidelines could potentially ultimately be of limited use and may not improve the reporting of elements that are important to key users of protocols. Of the eight guidelines that did detail methodology, four seem relatively comprehensive ,46,59,70For a guideline to be widely acceptable, we believe it should be developed using robust methodology that engages key stakeholders during development and is guided by empirical evidence, where possible. In addition, the methodology should be clearly reported and accessible to enable understanding of the process and assessment of its validity. Recommendations for reporting guideline development have recently been proposed  and inclThis review has some limitations. Although comprehensive in searching indexed periodicals, our review was not exhaustive in the search for institutional guidelines or books. However, our main findings would not likely substantively change with the inclusion of guidelines from these sources, as most guidelines available outside of journal articles did not describe development methods. Our results are also based on the methodology stated in included reports; we did not contact authors for additional information. Finally, the process of mapping and comparing concepts across guidelines was challenging due to the varied terminology used and the many sub-concepts of general headings that were recommended. To decrease bias we employed a systematic method and a second reviewer verified the process.Standard Protocol Items: Recommendations for Interventional Trials) has convened to produce such guidance by systematically developing recommendations for minimum content of clinical trial protocols [Our systematic review highlights some potential limitations of existing clinical trial protocol content guidelines. Given the evidence of protocol deficiencies -14 and trotocols . The priThis review identified many guidelines for clinical trial protocols; the recommendations provided by these guidelines varied substantially and potentially important concepts were often not recommended. Most guidelines did not describe their methods of development and none of the reports described replicable methods of development including formal consensus of key stakeholders or a thorough search for relevant empirical evidence. Given the importance of trial protocols to diverse stakeholders and evidence of protocol deficiencies, we believe that development of a guideline meeting such standards is needed.CDISC: Clinical Data Interchange Standards Consortium; CIOMS: Council for International Organizations of Medical Sciences; CONSORT: CONsolidated Standards Of Reporting Trials; EQUATOR: Enhancing the QUAlity and Tranparency Of Health Research; ICH E6: International Conference on Harmonization Tripartite Guideline for Good Clinical Practice E6; IQR: interquartile range; IRB: Institutional Review Board; PRACTIHC: Pragmatic Trials In Health Care; RCT: Randomized Clinical Trial; REC: Research Ethics Committee; SAE: Serious adverse event; SPIRIT: Standard Protocol Items, Recommendations for Interventional Trials.The authors have declared that no competing financial interests exist. Three members of the review team  are involved in the SPIRIT initiative.JT prepared the protocol with guidance from DM, AWC, ACT, and MS. MS and JT developed the search strategies. JT and JK selected relevant studies and extracted data; ACT participated in screening/extraction for the initial unpublished version of the review. JT carried out the analysis and prepared the manuscript with input from all authors. All authors read and approved the final manuscript.No direct funding was received for this study. Some authors were personally salaried by their institutions during the period of writing though no specific salary was set aside or given for the writing of this paper. Dr. Moher is supported, in part, by a University (of Ottawa) Research Chair. No funding bodies had any role in the study design, data collection, analysis, decision to publish or preparation of the manuscript.Appendix A. Systematic review protocol. Appendix B. Search strategy for Ovid MEDLINE® (including in-process and other non-indexed citations) 1948 to September Week 4 2010. Appendix C. Results of contact with nominated trial funding agencies. Appendix D. Details of guideline development methods described in reports (N = 40 guidelines).Click here for file


GRADE was developed to address shortcomings of tools to rate the quality of a body of evidence. While much has been published about GRADE, there are few empirical and systematic evaluations.To assess GRADE for systematic reviews (SRs) in terms of inter-rater agreement and identify areas of uncertainty.Cross-sectional, descriptive study.We applied GRADE to three SRs  with 29 comparisons and 12 outcomes overall. Two reviewers graded evidence independently for outcomes deemed clinically important a priori. Inter-rater reliability was assessed using kappas for four main domains  and overall quality of evidence.For the first review, reliability was: κ = 0.41 for risk of bias; 0.84 consistency; 0.18 precision; and 0.44 overall quality. Kappa could not be calculated for directness as one rater assessed all items as direct; assessors agreed in 41% of cases. For the second review reliability was: 0.37 consistency and 0.19 precision. Kappa could not be assessed for other items; assessors agreed in 33% of cases for risk of bias; 100% directness; and 58% overall quality. For the third review, reliability was: 0.06 risk of bias; 0.79 consistency; 0.21 precision; and 0.18 overall quality. Assessors agreed in 100% of cases for directness. Precision created the most uncertainty due to difficulties in identifying “optimal” information size and “clinical decision threshold”, as well as making assessments when there was no meta-analysis. The risk of bias domain created uncertainty, particularly for nonrandomized studies.As researchers with varied levels of training and experience use GRADE, there is risk for variability in interpretation and application. This study shows variable agreement across the GRADE domains, reflecting areas where further guidance is required. One of the motivations for developing the tool was to address shortcomings of other approaches to rating the strength or quality of a body of evidence. The GRADE tool offers a “common, sensible, and transparent approach to grading quality of evidence and strength of recommendations”. It represents an important tool for decision-makers as it provides a mechanism to bridge the gap from evidence synthesis to application of the evidence for informed decision-making.The GRADE tool  has been developed and refined over recent years by an international working group (www.gradeworkinggroup.org/publications/index.htm). One of the first citations regarding the GRADE tool appeared in the Canadian Medical Association Journal in 2003 British Medical Journal soon after Journal of Clinical Epidemiologywww.ims.cochrane.org/revman/gradepro) Much has been published about the GRADE tool , as well as overall quality of evidence  . The GRAFour individuals were involved in assessing the quality of evidence. All individuals had substantial experience and/or training in systematic review methods. LH has doctoral level training in research methods, more than 10 years of SR experience, and clinical training and experience in rehabilitation medicine. DD has doctoral level training in epidemiology and more than 5 years of SR experience. RF is a practicing pediatrician currently completing a doctoral degree with 4 years of SR experience. AA-S is a physician with specialty training, a doctoral degree in research methods, and 6 years of SR experience. The reviews were assessed by LH and RF (bronchiolitis), LH and DD (rotator cuff), and DD and AA-S (hip fracture). All reviewers read and were familiar with guidance on applying the GRADE tool Within reviews, we calculated inter-rater reliability between reviewers for each domain and overall quality of evidence using kappa statistics. Agreement was categorized as poor (0.00), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00) using one of the standard approaches Overall results are presented in We included 48 RCTs in this review. We graded the evidence for six comparisons and four outcomes . FurtherThis review included 27 trials  and 39 cohort studies. We graded evidence for 11 comparisons and four outcomes . There wThis review included 65 RCTs and 10 cohort studies. We graded evidence for 12 comparisons and four outcomes  for a toOne of the key challenges that arose during assessment of this domain was how to integrate risk of bias or quality assessments of the individual studies. This was more straightforward for RCTs than NRS. For instance, if the majority of evidence came from RCTs assessed as low risk of bias based on the Cochrane Risk of Bias tool Another question that arose was whether there should be pre-specified starting points for different study designs. For instance, it was questioned whether RCTs should always start out as low risk of bias overall whereas all NRS start out as high risk of bias overall. One restriction that we noted within the GRADE recommendations is that NRS are considered as a homogeneous body and start at high risk of bias. Moreover, they are all referred to as “observational studies.” This does not distinguish between studies that may be more or less prone to bias based on design features, such as prospective comparative studies versus a single sample before-after study. We questioned whether there should be a distinction among “observational studies” where some would start at high risk of bias because of key design limitations whereas others would start at moderate risk of bias based on more rigorous methodological approaches. For example, in the review of operative and nonoperative interventions for rotator cuff injuries, we considered prospective cohort studies that controlled for confounding to be less prone to bias and graded these higher than retrospective studies that failed to control for key confounders.A final key question that arose for the risk of bias domain was whether there should be different thresholds for different study designs. For instance, should RCTs always be higher in terms of risk of bias compared with NRS. Further, should the minimum threshold for RCTs be moderate risk of bias because the use of an appropriate randomized design always ensures less risk of bias compared with NRS. Alternatively, is it ever appropriate to rate NRS as low risk of bias given that they do not employ randomization and are always prone to bias due to potential imbalance between groups being compared, even when rigorous design or analytic methods are used to attempt to control for confounding.2 statistic, which provides an indication of the extent of statistical heterogeneity across a set of studies 2 was greater than 50%, the results were considered inconsistent. Similarly if I2 was less than 20%, the results were considered consistent. However, these were general guides and were not consistently applied; there were also areas where judgment was required. More disagreement arose when there was no meta-analysis. In general, the reviewers took a qualitative approach to assessing consistency across studies. For example, they considered whether the estimates were similar in terms of magnitude and direction of effect, as well as statistical significance. The somewhat different approach with and without meta-analysis raises the issue of whether assessments are made with differential standards depending on whether a meta-analysis is done. Often the reason for not conducting a meta-analysis, or pooling data, is due to heterogeneity across studies; hence, one option is that the default is “inconsistent” or “unknown” when there is no meta-analysis.When a meta-analysis was available for the comparison of interest, we relied heavily on the IThe two main criteria we used to assess this domain were whether the outcomes were intermediate or surrogate rather than the final health outcome, and whether the evidence came from direct head-to-head comparisons. We found that determining whether an outcome was intermediate or surrogate was somewhat context-specific and dependent on the research question posed in the review. For example, pain may be considered a surrogate or an intermediate outcome related to function. In the hip fracture review, pain itself was a primary outcome of interest, whereas in the rotator cuff review, pain was intermediate and function was among the primary outcomes of interest. We felt that directness with respect to whether the outcome is intermediate or surrogate could be determined at the outset of the review, thus eliminating disagreement at the stage of grading the evidence.We did not assess for applicability within this domain as there are discrepancies in current published guidance for grading evidence. One approach restricts the assessment to the two main criteria above This domain consistently had low reliability. Assessments were particularly problematic when no meta-analysis had been conducted as there was no confidence interval around a single estimate to determine the inclusion or exclusion of no effect or important benefit/harm. Similarly, assessment of precision was difficult when multiple meta-analyses were available for a given outcome due to differences in how an outcome was measured across studies. For instance, precision may have varied between a meta-analysis of pain measured on a continuous scale and a meta-analysis reporting the proportion of patients experiencing pain. In these cases, we relied heavily on the minimum thresholds for number of events (dichotomous outcomes) or subjects (continuous outcomes) that are specified in the GRADEPro Handbook We found it critical to set, a priori, a clinical decision threshold Much of the guidance in the GRADEPro Handbook is directed toward assessing precision around evidence of effect We found inconsistencies in our approach to upgrading and downgrading as well as the extent of upgrading and downgrading. For the most part, we used study design as our starting point and downgraded by one step for each domain that was not met; this approach appears to be consistent with the GRADE recommendation As we gained experience with applying GRADE for various review topics which had different questions and different types of evidence, we questioned whether we should aim for consistency across different bodies of evidence. For instance, should the evidence for the rotator cuff project, which was generally based on weak study designs with high risk of bias, ever be rated the same or higher than topics where the evidence comes only from RCTs. For instance, in the bronchiolitis review, some of the evidence was graded as low because it was based on a single trial, therefore the consistency was unknown and the confidence interval just crossed the null. However, the trial was at low risk of bias and was adequately powered to detect differences in the outcome of interest. Moreover, it was the largest trial conducted to date in the field, in some cases exceeding the sample size of all the other trials combined. Yet, the grade of evidence was comparable or in some cases lower than the body of evidence for the rotator cuff review which was driven by NRS.Two additional points were raised during the course of our work. First, we questioned whether we should develop strict rules for application of the GRADE tool in the work at our centre with the intent of increasing agreement between reviewers and across reviews; or, alternatively provide general guidance and reach consensus within the different reviews or clinical/topic areas. The latter approach may yield more inconsistency across reviews although this may provide important information for the end-user. For instance, if the reviewers report their agreement, the reader can judge the extent to which the specific issue is open to judgment and interpret the evidence more or less cautiously based on this information.The second additional point was challenges with terminology. We found it confusing for the risk of bias domain to use the same terminology as the process of assessing risk of bias of individual studies. Similarly, the overall quality of evidence was at times confused with quality of studies. Moreover, guidance documents differentially referred to the same overall assessment as quality of evidence Rating a body of evidence using the GRADE tool is becoming an important and recommended step in systematic reviews and other evidence synthesis initiatives. Rating the quality of a body of evidence is valuable for the end-users of evidence syntheses as it provides an indication of the confidence they can place in the results. Moreover, it is a key step in translating the body of evidence into clinical practice guidelines. We undertook this study to identify areas of uncertainty in applying the GRADE tool in order to inform the evidence synthesis efforts at our centre. Our research identified key areas where further guidance is needed to inform the application of the tool. Specifically, the domains of precision and risk of bias created the most inconsistency and challenges, whereas directness and consistency were more straightforward . Further, we encountered challenges when there were no meta-analyses upon which to base assessments, when outcomes were measured in different ways, and when evidence included nonrandomized studies.Journal of Clinical EpidemiologyApplication of the GRADE tool is complex as evidenced by the series of 20 articles that are currently being published in the This is one of the first systematic and empirical evaluations of applying the GRADE tool in systematic reviews. We recognize that the tool is not meant to eliminate disagreements nor the need for judgments As the GRADE developers acknowledge, “refinements are inevitable” for new and innovative tools such as GRADE


The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach has been adopted by many national and international organisations as a systematic and transparent framework for evidence-based guideline development. With reference to an ongoing debate in the literature and within public health organisations, this study reviews current experience with the GRADE approach in rating the quality of evidence in the field of public health and identifies challenges encountered.We conducted semi-structured interviews with individuals/groups that have applied the GRADE approach in the context of systematic reviews or guidelines in the field of public health, as well as with representatives of groups or organisations that actively decided against its use. We initially contacted potential participants by email. Responses were obtained by telephone interview or email, and written interview summaries were validated with participants. We analysed data across individual interviews to distil common themes and challenges.Based on 25 responses, we undertook 18 interviews and obtained 15 in-depth responses relating to specific systematic reviews or guideline projects; a majority of the latter were contributed by groups within the World Health Organization. All respondents that have used the GRADE approach appreciated the systematic and transparent process of assessing the quality of the evidence. However, respondents reported a range of minor and major challenges relating to complexity of public health interventions, choice of outcomes and outcome measures, ability to discriminate between different types of observational studies, use of non-epidemiological evidence, GRADE terminology and the GRADE and guideline development process. Respondents’ suggestions to make the approach more applicable to public health interventions included revisiting terminology, offering better guidance on how to apply GRADE to complex interventions and making modifications to the current grading scheme.Our findings suggest that GRADE principles are applicable to public health and well-received but also highlight common challenges. They provide a starting point for exploring options for improvements and, where applicable, testing these across different types of public health interventions. Several public health organisations are currently testing GRADE, and the GRADE Working Group is eager to engage with these groups to find ways to address concerns. Public health interventions affect large population groups and can generate significant health benefits at individual and population levels. Even though many public health approaches are preventive in nature, intervening in people’s lives may nevertheless do harm as well as good. In addition, they consume both financial and human resources, and may compromise individual freedom of choice. Public health interventions range from programmatic activities that initiate direct, proximal changes in a specific technology or behaviour to those that bring about more distal changes in multi-sectoral policies with indirect impacts on health . These iEvaluating public health interventions is far from straightforward and there is much discussion as to how evidence should be gathered, synthesised and used in decision making -9. DevelGuide to Community Preventive Services (Community Guide) issued by the United States Community Preventive Services Task Force [Public health organisations in different countries have developed distinct approaches to convey the quality of the evidence .These insk Force ,12, Publsk Force  and the sk Force ,15. Whilsk Force .The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group, a network of primarily clinical guideline developers and systematic reviewers, has attempted to meet this challenge by developing and testing a rigorous, systematic and transparent framework for evidence-based guideline development -19. In ahttp://www.gradeworkinggroup.org/society/). While the GRADE Working Group promotes the use of the framework across clinical and non-clinical health evidence, there has been much debate in the literature [More than 65 national and international organisations have adopted the GRADE approach  health policy, (ii) health system (iii) behavioural, (iv) nutrition, (v) environmental, (vi) vaccination and (vii) screening interventions. Post hoc we also added the category clinical interventions, as several answers provided by respondents related to treatment guidelines.Public health interventions were classified We primarily identified potential participants through the recommendations of members of the GRADE Working Group and the Guidelines Review Committee at the World Health Organization. We also approached the Cochrane Collaboration’s Public Health Group, the Cochrane Collaboration’s Effective Practice and Organisation of Care Group, the Campbell Collaboration, the SUPPORT collaboration through the Norwegian Knowledge Centre for the Health Services, the Public Health Agency of Canada, the Swedish National Institute of Public Health, the United States Community Preventive Services Task Force, the European Centre for Disease Prevention and Control and the NICE Centre for Public Health Excellence in the United Kingdom.We initially contacted potential participants by email with up to two email reminders as needed, inviting them to share their experience with applying the GRADE approach. We asked respondents the following three open-ended questions, and encouraged them to provide specific examples to illustrate their responses:• What are the challenges encountered in applying the GRADE approach to rating evidence for public health interventions?• Does the evidence grade obtained through GRADE adequately reflect the confidence in the evidence?• Could fine-tuning of the existing rating of evidence make the GRADE approach more applicable to public health interventions?In most cases, answers were obtained by telephone interview; in the remaining cases, answers were summarised based on email correspondence. We established experience with the use of GRADE for each interview as (i) applied without significant challenges, (ii) applied with minor challenges, (iii) applied with major challenges, (iv) currently being considered/tested, and (v) not currently applied. Specific challenges encountered and suggestions to change the existing rating of the evidence were initially identified by ER from written, validated summaries of individual interviews. These were then analysed across individual interviews by ER and EA to distil broader common themes.In order to validate our findings, we asked each participant to review and correct written summaries of the interviews and our rating of her/his experience with the application of GRADE. As necessary, we approached participants up to three times for validation. At a later stage, we also shared the draft paper with all participants and a wide range of interested parties to obtain feedback on our interpretation of findings. The study was acknowledged by the ethics committee at the University of Munich; ethics approval was not required.Of 29 individuals/groups in 12 organisations that were approached, 4 did not respond. Table Detailed responses to this question among the individuals/groups who had applied GRADE in one or more systematic review or guideline projects , the other half suggested that these were major . Even among the groups reporting challenges general feedback about the GRADE approach was positive. Users emphasised that GRADE principles apply to public health interventions, appreciated the systematic and transparent process of assessing the quality of evidence and stated that the GRADE approach has improved the quality of guideline development.Specific challenges reported can be grouped as follows:Complexity of public health interventions: Public health interventions tend to be complex with several components. In principle, studies assessing these interventions may look at one, all or different combinations of these components although, in practice, most evaluations focus on the impact of the whole package. Guideline developers and systematic reviewers developing a PICO  question for a specific intervention need to either consider the intervention as a whole or focus on a presumed active component. Either decision presents challenges regarding which studies to include or exclude, how to interpret heterogeneity, and how to make a careful judgement about the degree of indirectness.• Choice of outcomes and outcome measures: Most respondents emphasised that the choice of outcomes and outcome measures tends to be complicated in systematic reviews of public health interventions due to (i) multiple outcomes for most interventions, (ii) outcomes at individual and group levels, (iii) reliance on short-term surrogate rather than morbidity and mortality outcomes that are significantly delayed into the future, (iv) inconsistencies associated with the use of competing measures or assessment scales and (v) the need to group heterogeneous measures under “umbrella outcomes” to make the review policy-relevant. The choice of outcome and outcome measures has implications for indirectness of evidence and needs to be decided on and judged carefully as part of the GRADE process.• Ability to discriminate between different types of observational studies: Many groups reported difficulty using GRADE to rate public health evidence derived from observational studies. With the quality of evidence for all types of non-randomised studies starting as low, the GRADE approach is perceived as lacking the ability to distinguish between those public health interventions that are reasonably well-supported by evidence (e.g. by interrupted time series studies) and those that are less supported by evidence (e.g. by cohort studies). This may lead to misinterpretations of the evidence when communicating the message to policy-makers, and may even discourage the conduct of “best possible” studies. The GRADE approach should encourage people to develop and make the best use of high-quality evidence and not lead to an unrealistic restriction of the evidence, even if unintended.• Use of non-epidemiological evidence: Several groups also noted that the GRADE approach does not allow users to integrate non-epidemiological evidence, such as laboratory, mechanistic or animal studies, the principles of other disciplines  and evidence on implementation and context. All of these can be critical when assessing the effectiveness of a public health intervention , as well as when making a recommendation concerning implementation. As one respondent put it, ”relying solely on epidemiological evidence to assess intervention effectiveness is imprudent, as there are many factors influencing the results beyond whether or not the principle of a technology works.”• GRADE terminology: Several groups stated that some of the GRADE terminology and definitions were not appropriate for public health interventions . They also expressed concern about possible misinterpretations by policy-makers of “low quality evidence” and/or “weak recommendations”, with such wording potentially being used to justify inaction. Likewise, GRADE uses the term “observational study” to refer to all studies that are not randomised. In the public health world, the major distinction tends to be between observational studies where the assignment of individuals/clusters to an exposed/intervention group vs. unexposed/control group is outside of the control of the investigator  and those where the assignment of individuals/clusters to an intervention vs. control group is undertaken by the investigator in a randomised or non-randomised fashion . Possibly as a result of this confusion in terminology some guideline projects have quasi-experimental designs begin as high followed by downgrading rather than having these begin as low followed by upgrading, as recommended by the GRADE approach.• GRADE and guideline development process: Several respondents mentioned that GRADE is often applied to systematic reviews that do not consider all relevant data, in part because including non-randomised study designs makes the systematic review process methodologically challenging and much more time-consuming. Where the results of a systematic review are very heterogeneous and effect estimates cannot be pooled, applying GRADE criteria  to narrative estimates of impact becomes challenging. Some respondents also noted that the GRADE approach was not sufficiently objective and reproducible. Many respondents suggested that the GRADE process for going from evidence to recommendations was both difficult and ad hoc in terms of taking into consideration values and preferences, cost effectiveness and feasibility issues . Finally, there were concerns about the major financial and human resources required.• The United States’ Community Guide and the Public Health Guidance offered by the United Kingdom’s NICE Centre for Public Health Excellence are often referred to as models for evidence-based public health decision-making. Both organisations explicitly decided against using the GRADE approach:"· We perceive challenges in applying GRADE to the types of evidence we commonly use in the development of recommendations – these often include observational studies for questions of effectiveness and other types of studies for answering a range of other questions, such as qualitative research about the views of service users. We also use observational studies to help identify potential barriers to implementation. Further, as well as considering the quality, strength and applicability of the evidence, our committees take into account cost effectiveness, ethics, equity and other issues in developing recommendations. We think our current system gives us the flexibility we need, given the breadth of public health topics, the types of questions and the range of outcomes considered.” ""· At the Community Guide we do not use GRADE because we do not believe it works well when applied to population-level interventions. Specifically, it does not adequately differentiate between quasi-experimental designs with differing degrees of protection against threats to internal validity, and hence undervalues the evidence from the strongest of these designs. It is particularly problematic when there are many studies for which the most plausible threats to validity are expected to be randomly distributed, rather than systematically biasing the results in one direction.“ "Similarly, having explored its use, the Center for Psychiatric Rehabilitation at Boston University actively rejected the GRADE approach:"· Interventions in the field of psychiatric disability and mental illness are characterised by complexity, as they critically rely on interactions between two or more human beings, as well as social groups. In view of small sample sizes available, the typical course of exacerbations and remissions for many mental illnesses, and issues associated with role performance and societal integration outcomes, RCTs are very difficult to conduct. Therefore, the majority of studies in our field, with some notable exceptions, have been correlational, observational or quasi-experimental to this point. Given that GRADE, from the outset, determines observational studies to be of low quality, we felt that we had to begin from the question – what standards would properly assess such studies and their findings in terms of scientific rigor and meaning, rather than assessing them against a standard that they would rarely be able to meet. As a result, we developed our own grading systems for such studies.” "Most respondents felt that the evidence grade obtained by applying GRADE to their systematic review or guideline project was appropriate. They stated that the application of the GRADE criteria had opened their eyes to important limitations in the underlying evidence base. At the same time, they expressed the following concerns:Meaning of confidence in the evidence: GRADE defines quality of evidence as confidence in the pooled effect estimate. As the effectiveness of public health interventions is critically influenced by modes of delivery and contextual issues, a more relevant interpretation would be “confidence that the effect is meaningful across a range of plausible implementation contexts” • Selection of the appropriate body of evidence for rating can be difficult: Several groups stated that it is challenging to grade the quality of evidence, in the presence of both a single or few randomised studies  and a number of non-randomised studies . In the extreme case, can one well-conducted and large RCT conducted in one setting and rated as high-quality be sufficient to decide on the effectiveness of a complex intervention that critically relies on context?• All observational study designs begin as low-quality evidence: A number of participants stated that the current approach does not appropriately discriminate between stronger and weaker observational study designs. They suggested that quasi-experimental designs, interrupted time series studies and self-controlled case series should begin as moderate quality. In fact, some respondents felt that applying the GRADE criteria resulted in multiple discriminations against observational studies, where these studies started off low and were at risk of being penalised once more for threats to validity inherent to their design.•There are insufficient possibilities for upgrading observational studies: Different groups suggested a broader interpretation of existing criteria, such as extending the concept of dose–response to the population level, and the addition of new criteria for upgrading, in particular consistency and analogy. While GRADE allows for the downgrading of a body of evidence for inconsistency, consistency in findings across different settings, study designs and research groups actually increases our confidence in the evidence, as already stated by Sir Austin Bradford Hill. Analogy extends the concept of consistency to parallel evidence from related risk factors, interventions or population groups.•Non-epidemiological evidence can only be rated as very low-quality: Judging the effectiveness of a public health intervention sometimes relies on sources of evidence outside of epidemiology. Physiological, physical or engineering principles and the insights gained through laboratory or animal studies can only be brought into the rating exercise as a separate very low-quality piece of evidence rather than lending additional credibility to epidemiological evidence. GRADE also lacks a structured approach for integrating evidence on implementation and context, issues that often directly impact the effectiveness of complex interventions.•As has been apparent in previous sections, respondents did not limit their comments to rating the quality of evidence but provided feedback on the GRADE approach overall. Here, we focus on suggestions for fine-tuning the rating step; more general feedback has already been reported above. Suggestions by respondents largely fall into two categories and reflect comments discussed above: pragmatic guidance on the application of the GRADE approach for complex interventions and modifications to the existing GRADE criteria.Respondents expressed uncertainty about and requested pragmatic guidance on:• How to reflect intervention complexity in rating quality of evidence, in view of much of the available primary studies and systematic reviews simplifying intervention complexity;vs. group level outcomes, short term vs. longer term outcomes and surrogate vs. morbidity and mortality outcomes, as differences in prioritisation are likely to impact the evidence rating;• How to prioritise individual • How to decide whether very limited randomised evidence is sufficient or not, as including non-randomised evidence will have an impact on how the quality of evidence is rated;• How to ensure reproducibility in rating the quality of evidence, both in relation to individual GRADE criteria and their integration; and• How to apply the GRADE criteria to narrative summaries.Respondents also proposed specific modifications to rating the quality of evidence:• That selected observational study designs start off as moderate in order to discriminate more accurately between different levels of internal validity;• That additional criteria for upgrading be developed and operationalised, as these would allow confidence in a body of observational evidence to be reflected more accurately;• That a systematic approach for incorporating non-epidemiological evidence that directly relates to intervention effectiveness be developed; and• That a systematic approach for assessing the quality of modelling studies be considered.The objective of this study was to review the current use of and accumulating experience with the GRADE approach in rating the quality of evidence in the field of public health, and to identify challenges encountered to date. Despite the challenges reported, it is important to note that all respondents that have employed GRADE in a systematic review or guideline project appreciated the systematic and transparent process of assessing the quality of evidence. This highlights that the general principles of the approach are both suitable to public health and well-received.Common themes emerged with respect to aspects of the GRADE approach in its current form lacking applicability, reproducibility and clarity in the field of public health. Specific challenges related to (i) complexity of public health interventions, (ii) choice of outcomes and outcome measures, (iii) ability to discriminate between different types of observational studies, (iv) use of non-epidemiological evidence, (v) GRADE terminology and (vi) the GRADE and guideline development process. Respondents also made suggestions for additional pragmatic guidance and specific modifications to rating the quality of evidence; these are integrated in the below proposal towards making the GRADE approach more applicable to public health interventions.per se but relate more to the process of systematically reviewing and synthesising complex public health evidence; by requiring a transparent and structured approach for rating the quality of evidence and grading the strength of a recommendation, GRADE makes these challenges salient. In particular, the guideline development process often relies on systematic reviews that do not consider all relevant data, that are based on very heterogeneous data and that conclude with narrative syntheses rather than meta-analyses. Some of these issues are revisited in more detail below. Also, while it is argued that use of the GRADE approach substantially increases financial and human resource costs, a majority of these additional resources is likely to be required for conducting systematic reviews rather than for the application of GRADE.Several of the challenges reported are not specific to GRADE It is noteworthy that the majority of specific GRADE applications in the field of public health to date appear to have taken place at the World Health Organization where the use of GRADE has been mandatory since 1 January 2009. GRADE is also used as part of the SUPPORT Collaboration and at the Norwegian Knowledge Centre for the Health Services, which acts as one of the satellites for the Cochrane Collaboration’s Effective Practice and Organisation of Care Group. This rather limited experience with GRADE in the public health field begs the question whether this research was conducted ahead of its time, or whether the added value of the GRADE approach to public health beyond current practices is not sufficiently apparent. The fact that several groups are currently testing GRADE  or not appropriate where an intervention has multiple or indirect health impacts, where impacts materialise in a gradual or delayed fashion, and where an intervention is so complex that RCT results will be unacceptably artificial ,32. IndeEvidence-based public health guidance also needs to draw on sources of evidence beyond the traditional hierarchy of epidemiological study designs. First, public health evidence is never context-free , and theThis study is not representative of all GRADE applications in the field of public health to date. While we approached a large number of organisations and individuals, we may have overlooked others, and therefore potentially missed relevant insights. Although we obtained responses from all organisations approached, some groups or individuals within those organisations did not respond to our three emails to establish initial contact.Moreover, the characteristics of those interviewed vary greatly, for example in relation to level of epidemiological training, depth and breadth of the GRADE experience (i.e. number of specific GRADE applications and across different GRADE versions) and the level of support received from the GRADE Working Group. As a result, the classification of challenges as minor or major may not be directly comparable. On the other hand, our sample encompasses a range of organisations, health issues  and types of public health interventions, offering the full breadth of experiences to date. Therefore, we believe that this study offers important insights into issues that several organisations and groups are currently struggling with.While interview summaries and their qualitative interpretation are inherently subjective we had summaries validated and provided all participants with an opportunity to review the draft manuscript. Also, the fact that our empirical approach and the group of experts convened by the European Centre for Disease Prevention and Control  identifi• The GRADE approach is limited to assessing the level of evidence for the efficacy/effectiveness of an intervention, while prioritisation of topics, selection of experts and dealing with potential conflicts of interest should also be handled in a systematic, explicit and transparent manner.• The GRADE approach employs nomenclature to describe the confidence in the level of evidence that may sound pejorative, such as “low quality”; more neutral terminology would be more appropriate.• The GRADE approach appears to have too few categories to capture different levels of evidence represented by study designs beyond randomised controlled trials.• The GRADE approach is very limited in its applicability to other types of evidence, such as microbiological investigations, health economic models, mathematical models of the spread of infectious diseases and incidence/prevalence studies.• The GRADE approach includes too few criteria for upgrading the quality of observational studies.• The GRADE approach is limited in its applicability to issues beyond the assessment of efficacy/effectiveness (and possibly safety) of interventions, such as risk assessment, disease causation and spread of infectious diseases.• The GRADE approach is time-consuming and may thus not be suitable for a rapid grading of evidence under time pressure.• The GRADE approach does not appear to perform well in situations of scarce evidence that often characterise the field of public health.• The GRADE approach does not appropriately address going from evidence to recommendations, which should also include considerations of context, law, ethics, economic and political considerations.Some groups have shown that finding solutions to many of these challenges within the current version of GRADE is possible , others vs. beneficiaries and practitioners/policy-makers, the definition and classification of study designs, and the terminology used to communicate quality of evidence and strength of recommendation.We propose that the GRADE Working Group refine and revise, where appropriate, current terminology and definitions to make these more applicable to public health interventions. In particular, this relates to the interpretation of confidence in the evidence, the use of terms such as patients and clinicians vs. important vs. of limited importance, and alternative ways of communicating quality of evidence using symbols such as ⊕⊕⊕Ο [Efforts underway to address some of the issues raised include, for example, public health-compatible GRADE guidance on the meaning of outcomes that are critical  as ⊕⊕⊕Ο  or focus as ⊕⊕⊕Ο ,24. The We propose that the GRADE Working Group develop pragmatic guidance on how to apply the GRADE approach to complex interventions, illustrated with public health examples. In relation to rating the quality of evidence, guidance should address how to ensure reproducibility and internal consistency, when to consider non-randomised evidence in addition to randomised evidence, and how to apply GRADE criteria to narrative summaries. In relation to going from evidence to recommendations, guidance should include public health-specific considerations that influence the strength of a recommendation.Current efforts include the DECIDE project, which aims to improve the dissemination of evidence-based recommendations by building on the work of the GRADE Working Group . For exaWe propose that the GRADE Working Group review whether selected observational study designs should start off as moderate and develop and test additional criteria for upgrading, provided these can be made meaningful and applicable across public health and clinical medicine. The Working Group should also explore how other relevant sources of evidence that do not follow the traditional hierarchy of study designs  can be integrated with GRADE at the level of rating quality of evidence and/or at the level of grading a recommendation. GRADE should provide more explicit guidance about the use of different types of evidence.During regular meetings of the GRADE Working Group, specific examples of problems encountered can be raised and discussed, facilitating the continuous evolvement of the GRADE approach. Such a discussion led, for example, to agreement that the dose–response relationship criterion can also be applied at the population level ,24. SystThis empirical study reviewed current experience with the GRADE approach in the field of public health. It suggests that GRADE principles are applicable to public health, and that the systematic, transparent and rigorous nature of the process is well-received by those groups who have applied it. It also identified several common challenges, particularly in relation to evidence grading, which was the focus of this study. Contributions from respondents suggest that a combination of revisiting concepts and terminology, providing better guidance on how to apply GRADE to complex interventions and making some modifications to the existing scheme for rating quality of evidence could improve the applicability of the GRADE approach to public health interventions. Our proposal is intended as the starting point for a research and development agenda to explore options for improvements and, where applicable, to test these in systematic reviews and guideline projects across different types of public health interventions. Independent of whether the specific challenges identified here are perceived or real, they merit attention, if the GRADE approach is to be more widely adopted in the public health world.Both authors are members of the GRADE Working Group.ER conceived the study, approached study participants, conducted interviews and assumed primary responsibility for the analysis of findings. EA contributed to the design of the study and the analysis of findings. ER prepared the first draft of the manuscript, which was iteratively completed and revised by both authors. Both authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/13/9/prepubIn-depth responses, classified by health issue. This table provides the validated written summaries of 15 in-depth interviews conducted with individuals that have applied the GRADE approach in the context of systematic reviews or guidelines in the field of public health.Click here for file


Several studies have reported that clinical practice guidelines (CPGs) in a variety of clinical areas are of modest or variable quality. The objective of this study was to evaluate the quality of an international cohort of CPGs that provide recommendations on pharmaceutical management of glycemic control in patients with type 2 diabetes mellitus (DM2).We searched the National Guideline Clearinghouse (NGC) on February 15th and June 4th, 2012 for CPGs meeting inclusion criteria. Two independent assessors rated the quality of each CPG using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) instrument. Twenty-four guidelines were evaluated, and most had high scores for clarity and presentation. However, scope and purpose, stakeholder involvement, rigor of development, and applicability domains varied considerably. The majority of guidelines scored low on editorial independence, and only seven CPGs were based on an underlying systematic review of the evidence.The overall quality of CPGs for glycemic control in DM2 is moderate, but there is substantial variability among quality domains within and across guidelines. Guideline users need to be aware of this variability and carefully appraise and select the guidelines that they apply to patient care. High quality clinical practice guidelines (CPGs) provide recommendations based on a systematic review of the evidence, an assessment of balance of benefits and harms, and a transparent process for translating evidence to recommendations The global burden of diabetes is enormous. Of the estimated 346 million people worldwide with diabetes, 90% have type 2 diabetes mellitus (DM2) A number of studies have reported that CPGs in a variety of clinical areas are of modest or variable quality www.guideline.gov) on February 15th and June 4th, 2012 for all guidelines that provided recommendations on pharmacotherapy for glycemic control in persons with DM2. We searched for CPGs on two separate dates because guidelines are continually being revised, updated, or archived in the NGC and we wanted to ensure that we identified all guidelines relevant to our topic and that we did not exclude a guideline because it was archived during the time of our review process.We searched the National Guideline Clearinghouse (NGC) . For CPGs to be included in the NGC, guidelines must meet the following criteria: 1) the clinical practice guideline contains systematically developed statements that include recommendations, strategies, or information that assists physicians and/or other health care practitioners and patients to make decisions about appropriate health care for specific clinical circumstances; 2) the clinical practice guideline was produced under the auspices of medical specialty associations; relevant professional societies, public or private organizations, government agencies at the Federal, State, or local level; or health care organizations or plans; 3) corroborating documentation can be produced and verified that a systematic literature search and review of existing scientific evidence published in peer reviewed journals was performed during the guideline development; 4) the full text guideline is available upon request in the English language; 5) the guideline was developed, reviewed, or revised within the last 5 years In addition to meeting the NGC inclusion criteria, our study required that CPGs provided recommendations for glycemic control in any population with DM2, including adults, children, pregnant women, and persons with DM2 and any comorbid condition. If the full guideline was not available in the public domain, we purchased a copy.Two coauthors with experience in quality assessment of CPGs independently scored each guideline using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) instrument CPGs were considered to be based on a systematic review if there was either reference to a review or a review was contained within the guideline document, the review reported a search of one or more bibliographic databases, and a defined cohort of studies derived from the search was used to formulate recommendations.Twenty-four guidelines met our inclusion criteria . Ten of The overall quality of the included CPGs varied considerably, both within and across AGREE II domains . No guidScores for the stakeholder involvement (mean 52% of the maximum possible score) and applicability (mean 43% of the maximum possible score) were variable across guidelines. Seven guidelines scored greater than 60% on stakeholder involvement Scores for rigor of development were generally between 30–60%, with a few CPGs scoring very high and a few scoring very low . The ScoEditorial independence was the domain with the lowest scores across guidelines . CPGs infrequently described how the views of the funding body may or may not have influenced the content, and eight guidelines (33%) did not provide any information on conflicts of interest for the CPG developers. Of the 16 (66%) CPGs that did provide information on competing interests, only one guideline reported that they discussed and resolved their conflicts In the overall assessment, 13 guidelines (54%) were recommended, seven (29%) were recommended with modifications, and four (17%) were not recommended . The fouThe overall quality of the 24 guidelines for glycemic control in DM2 was highly variable, and no guideline scored well in all domains of quality. There was also significant variability across domains within guidelines. Guidelines consistently scored well in the domain of clarity and presentation, suggesting that this component of guideline development may be easier to achieve or more highly valued by guideline development organizations. On the other hand, editorial independence was poorly addressed by almost all guidelines (the only exceptions were the CPGs developed by the American College of Physicians (ACP) Our assessment also suggests that guideline developers do not pay sufficient attention to the applicability of their recommendations to their target audiences and to implementation issues. Lack of attention to these issues has been noted in other studies examining the quality and usefulness of clinical practice guidelines Several studies have examined the quality of various cohorts of CPGs in diabetes, and findings vary. Bennett and colleagues Systematic reviews should form the basis for all high quality CPGs There are several important issues with regards to using AGREE II to appraise the quality of CPGs. First, the AGREE II domain of rigor of development does not encompass all important aspects of the quality of a systematic review, as does a quality assessment instrument developed specifically for that purpose, such as AMSTAR The overall assessment in AGREE II of whether the CPG was recommended, recommended with modifications, or not recommended This study has limitations, in addition to those imposed by AGREE II. Our cohort may not be representative of all diabetes guidelines, as we selected only those examining glycemic control for type 2 diabetes included in the NGC. Guidelines on other aspects of diabetes and those not in the NGC  may differ in quality from those that we examined. In addition, the NGC does not contain all guidelines on diabetes: organizations choose to submit their guidelines to the NGC, and we did not search other sources for additional relevant guidelines.We purposefully chose a low threshold for defining whether a systematic review was used to develop recommendations in the CPG. If we had imposed a more stringent definition such as one requiring a search of multiple bibliographic databases, assessment of quality of individual studies and of the body of evidence, and an explicit framework for developing recommendations from the body of evidence, the number of CPGs in our cohort that were considered to base recommendations on an underlying systematic review would have been far fewer.In view of the potential impact of CPGs on health care delivery and patient outcomes, it is imperative that guidelines be of optimal quality. It is clear from this cohort of CPGs on glycemic control in DM2 that only a small minority of guidelines fulfill most criteria for a high quality guideline. The guideline user needs to beware, to critically appraise guidelines before use and to weigh the relative importance of the criteria for quality, starting with an assessment of whether a high quality systematic review underpins each recommendation.Figure S1PRISMA flow diagram.From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097.(TIF)Click here for additional data file.