Methods for describing one's confidence in the available evidence are useful for end-users of evidence reviews. Analysts inevitably make judgments about the quality, quantity consistency, robustness, and magnitude of effects observed in the studies identified. The subjectivity of these judgments in several areas underscores the need for transparency in judgments.stability ratings, and qualitative conclusions are tied to strength ratings. Our system emphasizes extensive a priori criteria for judgments to reduce the potential for bias. Further, the system makes explicit the impact of heterogeneity testing, meta-analysis, and sensitivity analyses on evidence ratings. This article provides details of our system, including graphical depictions of how the numerous judgments that an analyst makes can be combined. We also describe two worked examples of how the system can be applied to both interventional and diagnostic technologies.This paper introduces a new system for rating medical evidence. The system requires explicit judgments and provides explicit rules for balancing these judgments. Unlike other systems for rating the strength of evidence, our system draws a distinction between two types of conclusions: quantitative and qualitative. A quantitative conclusion addresses the question, "How well does it work?", whereas a qualitative conclusion addresses the question, "Does it work?" In our system, quantitative conclusions are tied to a priori criteria for making judgments, and the direct impact of analytic results on evidence ratings. These attributes form the basis for a logically consistent system that can improve the usefulness of evidence reviews.Although explicit judgments and formal combination rules are two important steps on the path to a comprehensive system for rating medical evidence, many additional steps must also be taken. Foremost among these are the distinction between quantitative and qualitative conclusions, an extensive set of  Systematic reviews, technology assessments, and clinical practice guidelines all incorporate evidence-based conclusions. The multifaceted nature of evidence, however, leads to varying degrees of confidence in how well the evidence supports conclusions drawn from it. For example, one is more confident in conclusions drawn from several well-designed randomized controlled trials that find similar effects than in conclusions drawn from a few poorly designed trials with disparate results. Consequently, methods for describing one's confidence in the available evidence are useful for end-users of evidence-based documents. This confidence is embodied in strength-of-evidence ratings.stability of evidence as distinct from the strength of evidence. We identify the many judgments inherent in the process of performing evidence reviews, and then note how such judgments are incorporated within two prominent rating systems: the U.S. Preventive Services Task Force (USPSTF) system and the Grades of Recommendation Assessment, Development and Evaluation (GRADE) system [In this article, we introduce a structured and transparent system for rating the strength of a body of evidence pertaining to a medical technology see Note 1). We also define the concept of the . We also) system . Next, wIn performing an evidence review, an analyst must make numerous judgments about the available evidence. For many of these judgments, analysts at different centers could reasonably disagree about the status of the evidence. For example, one analyst may view a certain methodological flaw as fatal, whereas another analyst may view that same flaw as minor. This problem may be partially addressed through the use of standardized quality instruments. However, different centers tend to use different instruments, which can lead to different assessments of the quality of trials . In the 2 . Other components of the evidence analysis also require judgments, including quantity, robustness, and magnitude of effect. Definitions of these terms appear in Table The challenge of conflicting judgment is magnified when several studies are available for review. With multiple studies, the analyst must also consider the degree of consistency among studies' results. Again, judgments must be made and different analysts may reasonably disagree. For example, different analysts may define "inconsistency" using different threshold values for IIn addition to these individual components , still another layer of judgment is required: how to combine each of these five components to produce a rating of the overall "strength" of the evidence. Different analysts may disagree about the relative importance of each component and their interplay. For example, if there is one small but well-conducted randomized controlled trial, how should one reconcile the high quality but low quantity of this evidence base to produce an overall strength rating? Or, how should one interpret very large effects observed in suboptimal study designs?With all the judgments necessary, two different analysts, when faced with exactly the same clinical question and exactly the same evidence base, may have different degrees of confidence in their conclusions. In some instances, this discrepancy could be so large that the two analysts reach different overall conclusions. In a pilot study of inter-reviewer agreement, the GRADE group asked 17 experienced reviewers to rate the strength of the evidence for each of 46 outcomes within 12 systematic reviews [transparency. With transparent judgments, the end users of the review  can decide for themselves whether the judgments are reasonable.The subjectivity of judgments at several points in the systematic review process underscores the need for Numerous rating systems designed to assess the strength of a body of evidence have been proposed. Several of these were reviewed in a 2002 report from the Research Triangle Institute-University of North Carolina Evidence-based Practice Center (RTI-UNC EPC) . This reOne prominent system is the Third U.S. Preventive Services Task Force (USPSTF) system . This syOne major effort to create a formal point system is the GRADE system -5 The GRWithin the various increments and decrements in GRADE, several implicit judgments are necessary. For example, what is a "serious" limitation in study quality or an "important" inconsistency among effect sizes? Such judgment is an inevitable part of performing systematic reviews, as discussed earlier, and the GRADE group readily acknowledges the need for judgment. Further, a key motivation behind the GRADE effort was a need for "explicit definitions" and "sequential, explicit judgments," and the An explicit system for combining these judgments  is a particular strength of the GRADE approach. This system defines precisely how the various aspects of the evidence are combined to arrive at an overall grade of the evidence for each outcome. Thus, if a user made different judgments than the analyst, then the user could apply the GRADE point system accordingly, and possibly arrive at a different rating of the evidence. For example, suppose an analyst judged that there were important inconsistencies in the data for a certain outcome, but a user examined the same data and believed that the inconsistency was unimportant. Using the GRADE system, the user could then increase the final grade by one level.In addition to USPSTF and GRADE, several other prominent systems for rating the strength of evidence have been proposed by the American College of Chest Physicians (ACCP) , the Ausa priori criteria for judgments; and 3) the direct impact of meta-analysis and sensitivity analysis on evidence ratings. We then provide details on the system itself, including graphical depictions of how judgments can be combined  the distinction between quantitative and qualitative conclusions; 2) extensive use of d Figure .overall evidence for the technology , and the strength of a recommendation about the technology. The first involves assessing the overall net benefit by balancing the benefits and harms. The second entails more judgments (listed in the USPSTF methods article) about the importance and impact of cost, ethics, law, patient expectations, and societal expectations [internal validity of the evidence. Questions about generalizability can be addressed outside the scope of the rating system. We are currently examining how well our evidence rating system can be applied within the overall GRADE framework.Our system is designed to rate the stability and strength of evidence for each outcome an analyst chooses to evaluate (see Note 2), and is not intended to produce overall recommendations for (or against) a technology. By contrast, both USPSTF and GRADE rate the strength of the ctations . Also, uOur system draws a distinction between two types of conclusions: quantitative and qualitative , and subsequent meta-regression analyses did not explain this heterogeneity. Consequently, we refrained from presenting an estimate of the size of treatment effect. However, all trials found that TLR rates were lower after implantation of a drug-eluting stent than after a bare metal stent, and the random-effects meta-analytic confidence interval demonstrated this clear direction of effect  associated with a technology. Other users are simply interested in whether the technology provides any benefit at all. If a systematic review provides both kinds of conclusions, then both needs are met.No other system for rating medical evidence distinguishes explicitly between quantitative and qualitative conclusions. We believe the distinction is critical to ensure that systematic reviews provide a full picture of the evidence. The GRADE group defines their ratings as the likelihood that further evidence will change one's confidence in the size of the effect, which appears to be a purely quantitative definition. The USPSTF system does not state whether its ratings refer to quantitative conclusions, qualitative conclusions, or both.In our system, stability and strength ratings are not independent. Logically, evidence that permits a "highly stable" estimate of treatment effect  must also permit a "strong" conclusion about the direction of effect . Thus, one built-in feature of our system is that the stability rating sets a lower bound on the strength rating. This means that "moderate" stability can be accompanied by a strength rating no lower than "moderate", and "low" stability can be accompanied by a strength rating no lower than "weak."bA1c in treatments for diabetes). Then, clinical importance can be addressed as a qualitative question: "Is the difference clinically important?" This question is addressed analytically via a comparison of effect sizes to an effect size predefined as clinically important.Crucial to understanding the results of a systematic review is understanding whether the results are clinically important; statistically significant results do not necessarily represent a clinically important effect. This has been mentioned in many systematic reviews -18, and a priori inclusion criteria to reduce the potential for bias in judgments about which studies to include. Some also make an a priori judgment about which instrument will be used to assess study quality. However, many other judgments are still susceptible to bias. To reduce this potential, our system specifies the use of a priori judgments wherever possible. For example, the system requires that one specify a priori quantitative definitions of "consistent" and "robust" effects. Also, the analyst must pre-specify the minimum percentage of included studies that reported the outcome of interest in order to permit a meta-analytic estimate of effect size. If only a small percentage of included studies reported the outcome, selective outcome reporting may have occurred, thereby biasing the meta-analytic summary statistic. For study quality, the analyst must identify not only the instrument to be used, but also the scoring system (if used) and the thresholds that define study quality categories . Even the threshold for statistical significance  must be specified beforehand. To address the question of clinical importance, the minimum level considered to be clinically important must also be determined a priori. These definitions, and others, are discussed further in the section entitled "How the System Works".Most systematic reviews use Many systematic reviews report the results of meta-analyses, and some also describe sensitivity analyses. Often, however, the results of these statistical analyses are not explicitly tied to ratings of the evidence. In this section, we describe how our system links analytic results to both stability and strength ratings.2 [single summary estimate is not warranted when the evidence demonstrates the existence of multiple estimates.The purpose of meta-analysis is not just to obtain a summary estimate of treatment effect, but also to test the data for consistency (heterogeneity testing). This latter purpose is typically accomplished using the Q-statistic, and more recently, I2 ,20 If imqualitative conclusion. Even if there is substantial unexplained heterogeneity, the evidence may still indicate a consistent direction of effect. The confidence interval (CI) around the random effects summary statistic, which incorporates both within-study and between-study variance, may lie fully above 0 or below 0 (see Note 4). This CI, therefore, is suitable for determining whether the data suggest a clear direction of effect.Although our system precludes the use of random-effects models in determining a single summary estimate of treatment effect, the use of these models does have an important role. This role involves a summary of the evidence to support a Other systems, such as GRADE and USPSTF, are largely silent on the role of meta-analysis in systematic reviews. Our system uses meta-analysis and meta-regression  to increase statistical power and employ precise study weights; furthermore, the system is unique in incorporating the results of these analyses into evidence ratings.We now turn to the role of sensitivity analysis in our system. In this context, consider that the goal of rating evidence is to assess the likelihood that future evidence will indicate something different than what current evidence indicates. If a large amount of consistent evidence has already accumulated, then future evidence is unlikely to alter the overall strength or stability. Conversely, conclusions based on only a small amount of accumulated evidence may easily change when a single new study is published.Considered from this perspective, we argue that sensitivity analysis (see Note 5) can substitute for certain judgments about quantity. The idea is that if the conclusion from a meta-analysis depends critically on only one or a few studies in that analysis , then the conclusion may not be robust. Such dependence suggests that a future study may alter conclusions based on currently available studies. Consequently, our system downgrades the stability or strength ratings accordingly. Although there is a widespread sense that sensitivity analysis should be incorporated into an analysis, the system is unique in offering explicit rules for how to gauge the impact of the results of sensitivity analyses on one's confidence in the available evidence.Sensitivity analysis can obviate the need for certain subjective judgments about the magnitude of effect. Some rating systems  employ such judgments, and if the observed effect is very large, the evidence receives a higher strength rating. Presumably, this is because a very large effect is less likely to be overturned by future evidence and is therefore more robust. However, if there are sufficient studies to perform direct robustness tests via sensitivity analyses, then we advocate doing so, in lieu of making judgments about effect sizes. A meta-analytic sensitivity analysis incorporates effect sizes and confidence intervals from all studies, so that the test is empirically-based.As with consistency, the quantitative/qualitative distinction helps clarify two notions of robustness. Quantitative robustness concerns the degree to which the summary effect size from meta-analysis tends to change based on relatively small alterations in the data. To assess quantitative robustness, one can perform successive meta-analyses and observe the relative changes in the summary estimate. If the changes in the estimate exceed a predetermined tolerance level, then the original summary estimate is not quantitatively robust. Qualitative robustness refers to whether the evidence base yields the same qualitative general conclusion upon alterations of the data. To assess it, one can again perform successive meta-analysis, but in this case the issue is whether the confidence intervals around summary statistics consistently indicate the same direction of effect.A priori, we had defined a qualitative robust evidence base as one where each of the last three cumulative meta-analyses yielded the same qualitative conclusion. Therefore, we deemed this evidence base to be qualitatively robust.For example, one qualitative robustness test we have employed utilized cumulative meta-analysis . In a reThe system is shown graphically in five figures:• Entry into system Figure • Overview of the high quality arm Figure • Homogeneous data Figure • Heterogeneous data Figure • Small evidence base Figure a priori criteria. Nearly all of these a priori criteria are operational definitions that are quantitative. The use of a priori criteria helps to reduce bias and subjectivity, as discussed above, and the use of quantitative definitions increases transparency. This system assumes that the assessor has already applied appropriate inclusion/exclusion criteria and has excluded from the analysis any study with fatal flaws.An important feature of this system is that every question illustrated in the figures requires a set of The initial entry into the system occurs with an assessment of the quality of the evidence for a specific outcome Figure , which wFigure are not met, then one proceeds to Figure are met, then one tests the quantitative consistency of the data using a heterogeneity measure such as Q or I2. Under homogeneity, one proceeds to Figure At the top of these pathways, one first considers whether the evidence base is sufficient to provide a single quantitative estimate of the effect size. We generally require at least three studies, but other investigators may wish to set this criterion higher . Additionally, the system requires that a certain percentage of the studies  must have calculable effect sizes (that can be determined without imputation). If these criteria and the effect is clinically important ; B) the treatment is beneficial but the effect may or may not be clinically important ; C) the treatment is beneficial but the effect is not clinically important ; or D) the treatment is not beneficial  (see Note 6). By contrast, example E in Figure Before detailing the steps in Figures In the homogeneous pathway Figure , one perContinuing within Figure Figure Figure If there are only two studies and both have calculable effect sizes, one performs a random-effects meta-analysis which, if informative, allows a qualitative conclusion. Of note, meta-analysis of a two-study evidence base is not required in the moderate quality arm, where a conclusion would require both studies to have a statistically significant effect. In the low quality arm, a minimum of three studies is required to reach any conclusion. A qualitative conclusion is also possible for two studies with imprecise effect sizes (that cannot be combined) if both studies are informative and show qualitatively consistent results. If two studies are qualitatively inconsistent or not informative when combined, the findings are inconclusive. If there is only one study, a large effect size is required to allow a weak qualitative conclusion .In this section, we provide two example applications of our evidence rating system. In addition to illustrating various aspects of the system, we show how the system can be used in conjunction with simple declarative conclusions that are tied to the stability and strength ratings. The first example involves drug-eluting stents (DESs) for the treatment of coronary artery disease, and the second example involves positron emission tomography (PET) in the staging of lymphoma.In a 2006 report, we examined the evidence comparing the safety and efficacy of drug-eluting stents (DESs) and bare-metal stents for the treatment of angina .We included 14 randomized trials that compared drug-eluting stents to bare metal stents and reported the percentage of patients in each group who underwent target lesion revascularization (TLR) after stent implantation. The trials enrolled a total of 7,006 patients. We addressed the quantitative issue of the size of the difference in overall TLR rates and also the qualitative issue of whether there was any difference in overall TLR rates between the two types of stents.a priori definitions of high, medium, and low quality) that the evidence base was of high quality.To assess the quality of the studies, we applied a quality rating scale, and determined  and I2 < 50% . We used a p value of 0.10 because of the known low power of Q. We used a threshold of 50% for I2 because this value represents moderate heterogeneity [2 = 78%, Q = 59, p value for Q was less than 0.000001).ogeneity ,20 For tGiven the heterogeneity of effect sizes, we performed meta-regressions in an attempt to explain why study results differed. Not all potential covariates could be examined because not all studies reported some covariates. The three covariates we could examine were drug type (paclitaxel or sirolimus), mean target vessel diameter, and mean target lesion length. None of these three factors were sufficient to explain the observed heterogeneity. Therefore, we did not draw a quantitative conclusion for the outcome of TLR rates. However, we proceeded to a qualitative analysis using a random-effects model to determine whether the evidence permitted a qualitative conclusion.We rated the stability of the evidence as Unstable, due to the unexplained heterogeneity among effect sizes.a priori as any statistically significant effect). This meant that the evidence was informative.We performed a random-effects meta-analysis using Cohen's h. The summary statistic was statistically significant (lower TLR rates among patients who received DESs) and clinically important  removal of the study with the smallest weight , and, 2) after the additional removal of the study with the second smallest weight in the meta-analysis. Because the evidence base met both of these criteria, we deemed the evidence base to be qualitatively robust.The strength rating of the evidence for a qualitative difference in TLR rates was Strong, because the meta-analysis was of high quality studies, and was informative and qualitatively robust.The conclusions for this outcome were phrased in the following manner:The use of DESs (Cypher and TAXUS stents) leads to lower overall TLR rates than use of bare-metal stents in patients with angina at 6 to 12 months following stent implantation. (Strength of evidence: Strong)• Due to unexplainable differences among the findings of different trials, one cannot accurately determine how much lower these rates are at 6 to 12 months following implantation of a DES.In a 2006 report prepared by ECRI's Health Technology Assessment Information (HTAIS) under contract to TRICARE Management Activity (see Note 7), we assessed the use of positron emission tomography (PET) in the staging of lymphoma. The reference standard for determining whether lymphoma has reached Stage IV is a bone marrow biopsy typically taken from the iliac crest. PET may potentially help patients avoid the invasiveness of bone marrow biopsy, depending on how accurately it detects bone marrow infiltration.We included five diagnostic cohort studies that performed PET as well as bone marrow biopsy and also reported sufficient information for the calculation of sensitivity and specificity. The studies reported data on a total of 243 patients. Because our research question was to determine a quantitative estimate of diagnostic accuracy, we did not attempt to draw a qualitative conclusion. Therefore, the text below refers only to the stability rating, not to a strength rating.a priori definitions of high, moderate and low quality) that the evidence base was of moderate quality. To rate the strength and stability of this evidence, we used the moderate quality branch of the system, which is not included in this paper, but is structurally very similar to the high quality arm. The key difference is an across-the-board decrease in both stability and strength ratings .To assess the quality of the studies, we applied a quality rating scale, and determined . We computed the diagnostic odds ratio for each of the five studies, and the heterogeneity test revealed no heterogeneity (I2 = 0%). Therefore, based on the definition of quantitative consistency, we deemed these findings to be quantitatively consistent. There was a threshold effect in the data, however, as evidenced by a plot of the data in ROC space and a strong negative correlation between sensitivity and specificity. We used the method of Littenberg and Moses [nd Moses  to constWhen evaluating this diagnostic, we attempted to reach only quantitative conclusions, not qualitative conclusions. Therefore, we did not consider informativeness, and we automatically proceeded to quantitative robustness testing for the summary diagnostic odds ratio (DOR).A priori, we defined a quantitatively robust evidence base as one that met both of the following two conditions: 1) the confidence interval around the summary DOR did not contain a DOR 50% higher or lower than the summary point estimate, and 2) a cumulative meta-analysis in which studies were entered by precision (highest precision study first) found that all of the last three analyses produced summary effects that were within 5% of the overall summary diagnostic odds ratio. In this case, the evidence base met neither of these conditions; therefore, we deemed the estimate to be not robust.Using the system, we assigned a stability rating of Low to the estimated diagnostic odds ratio of 12.7 (95% CI 5.5 to 29.7). This rating was based on the fact that the studies were of moderate quality, and the estimate was not quantitatively robust.The conclusion was phrased in the following manner:For the detection of bone marrow infiltration, at mean threshold PET has a sensitivity of 64% (95% CI: 43% to 80%), and a specificity of 88% (95% CI: 76% to 95%). Stability of estimate: Low.a priori judgments to avoid bias. Third, it makes explicit the consequences of the results of meta-analysis and sensitivity analyses for both stability ratings and strength ratings. We have linked these attributes, and many others, into a logically consistent system intended to improve the process of systematic review.The rating system described in this paper has several unique attributes. First, it distinguishes quantitative stability ratings from qualitative strength ratings. Second, it makes extensive use of Our system recognizes the fundamental role of judgment in summarizing evidence. The system is best viewed as a logical way to organize one's own judgments, not a predefined list of required judgments. Because different analysts can have different judgments, however, there is a critical need for transparency. This allows the user to trace the path of judgments that lead to a stability or strength rating. In fact, to use our system without making judgments explicit would constitute a misuse of the system.a priori and should be empirically-based. A priori judgments, a natural extension of a priori inclusion criteria, may help reduce bias by avoiding the influence of observed data on operational definitions. Also, basing judgments on empirical analyses, rather than opinion alone, may reduce bias. For example, judgments about quantity can be made empirically-based via formal sensitivity analysis.Explicitness in judgments has been advocated by the GRADE group , but expWhile our system allows different analysts to make different judgments, it does place certain boundaries on judgments. If there are only one or two studies, for example, the evidence base must be judged insufficient to permit a quantitative estimate of the effect size. Further, if these one or two studies are of low quality, then no conclusion is possible. If there is only one study, then it must have been a high quality study observing a large effect in order to permit any conclusion. These limitations for small evidence bases are grounded in the principle of independent replication of scientific findings.The GRADE system also places certain boundaries on the kinds of judgments permitted. In GRADE, one must give high priority to randomization: RCTs are initially judged as two levels stronger than observational studies, and three levels stronger than other study designs. Also, GRADE defines a "strong" association as a relative risk between 2 and 5 (or between 0.2 and 0.5) and a "very strong" association as a relative risk greater than 5 (or less than 0.2) (see Note 8). Other judgments are left up to the analyst, although GRADE recommends that all judgments be made explicit.a priori judgments. The analyst is responsible for making reasonable judgments before analyzing the data. For example, the a priori definition of a "clinically significant effect" is a critical judgment that can greatly impact the conclusions of the review. One strategy we have employed is to have additional clinicians and methodologists examine the analyst's a priori judgments for their reasonableness, and then to resolve disagreements in conference. These consensus judgments are then employed throughout the step-by-step system to produce evidence ratings. Whatever judgments are agreed upon, however, should be made fully explicit in the review. Such transparency, as emphasized throughout this paper, can enhance the flexibility and usefulness of the review.We next note some potential limitations of the system described in this paper. First, the system is complex. Understanding how it works, and more importantly why it works that way, requires careful study. Second, the system is resource-intensive, mostly because of the numerous a priori judgments to avoid bias, and directly linking analytic results to evidence ratings.In summary, explicit judgments and formal combination rules, as advocated by the GRADE group, represent two important steps on the path to a fully reliable system for rating medical evidence. Throughout this paper, we have argued that many additional steps must be taken. Foremost among these are distinguishing between quantitative and qualitative conclusions, making Considered as a whole, our system constitutes a flexible tool for incorporating the full complexity of the evidence. Despite this complexity, the system outputs straightforward conclusions and ratings for medical decision makers to employ as they encounter difficult evidence-based decisions.• Systematic reviewers inevitably make judgments about the quality, quantity, consistency, robustness, and magnitude of effects observed in the studies identified• This paper introduces a formal system for combining these judgments in a logical, consistent frameworka priori criteria for judgments, and the direct impact of meta-analysis and sensitivity analysis on evidence ratings• Unique aspects of the system include the distinction between quantitative and qualitative conclusions, extensive • Stability ratings refer to the likelihood that future evidence will indicate a different size of effect• Strength ratings refer to the likelihood that future evidence will overturn conclusions about whether a device, drug or procedure is effective (or harmful)The author(s) declare that they have no competing financial interests. JRT, SJT, and JTR are each employed by the ECRI Health Technology Assessment Information Service, which produces systematic reviews. ECRI paid the article processing charge for this manuscript.All authors made substantial contributions to the ideas presented in this manuscript. JT contributed extensively to the development of the rating system, devised the manuscript outline, and wrote the manuscript. CT supervised the project and contributed extensively to developing and testing the system. ST originated the idea for the rating system and contributed extensively to its development. JR contributed through editing and testing the system, and wrote the section entitled "How the System Works". All authors participated in the editing process, and all authors read and approved the final manuscript.1 – Throughout this paper, the word "technology" is used generically to refer to drugs, devices, or procedures.2 – The specific outcome could be a surrogate outcome or a patient-oriented outcome.3 – The presence of heterogeneity is a violation of the assumption of fixed-effects models: that the available data are sampled from a single distribution that describes a single summary estimate of treatment effect.4 – The use of 0 assumes that the effect size metric is centered around 0 . Other effect size metrics  are centered around 1, and for these one would determine whether the random-effects confidence interval were fully above 1 or below 1.5 – We refer to an entire family of procedures designed to assess the sensitivity of one's conclusions to certain aspects of the data or certain analytic assumptions. Many of these were listed in a paper by Olkin, and some6 – Although not illustrated in the figure, there are possible effect sizes corresponding to A through D that would fall on the left side of the graph (below zero). Some of these possibilities would lead to a conclusion that the treatment was either harmful or inferior to a comparison treatment.7 – TRICARE is the government agency under the Department of Defense responsible for administering the health benefits for the U.S. armed forces and their dependants.8 – Note that the GRADE system's treatment of effect magnitude does not specifically mention the confidence interval around the effect. Instead the system addresses imprecise or sparse data in a separate manner. Our system considers an effect size and its confidence interval simultaneously.The pre-publication history for this paper can be accessed here:


Hemangiopericytoma is a soft tissue vascular neoplasm arising from capillary pericytes and is found throughout the body in soft tissues and bone. It was first described in 1942. Primary vascular neoplasm of the spleen constitutes the majority of nonhaematolymphoid splenic tumors like haemangioma, lymphangioma, hemangioendothelioma, hemangiopericytoma etc. Splenic hemangiopericytoma is a rare tumor and probably first case was described in 1989. Uptill now only eight cases are reported in the English literature.A-35-year old male presented with fever and dull aching pain in left hypochondriac region. Radiological evaluation showed presence of multiple abscesses in spleen. Investigations were done to rule out common causes of abscess in spleen. After failure of medical management, he was subjected to elective splenectomy. There were dense adhesions between the spleen and the adjacent structures and the diaphragm. The histopathology of the resected specimen showed hemangiopericytoma of spleen.The present case illustrate that the hemangiopericytoma of spleen can mimic as multiple abscess. Splenectomy is the treatment of choice. It waterature ,6-13. Itterature -13. Yilmtionship .A 35-year-old male presented with complaint of mild grade, intermittent fever without chills and rigor and diurnal variation for last 2 months along with occasional dull aching pain in left hypochondriac region. Ultrasonography revealed multiple abscesses in spleen. He was managed on antibiotics for 21 days but fever continued. Computed tomographic (CT) scan of abdomen showed multiple abscesses in spleen without any other significant findings in abdominal organs. Around 200 ml of pus was aspirated under guidance, which was negative for bacteriological examination. He was kept on parenteral antibiotics but did not have any relief.Color Doppler of heart excluded any possibility of infective endocarditis. X-ray chest was normal. Montoux test was positive up to 11 × 11 mm and he was put on antitubercular treatment to which he did not respond in 6 weeks. Surgical consultation was done and it was decided to proceed with splenectomy.At laparotomy, there were dense adhesions between the spleen and the adjacent structures and the diaphragm. The patient went into shock after 2 hours of surgery and was resuscitated and taken back to operation theatre for reexploration. Exploration revealed generalized ooze from the area and hemostasis was achieved but again after 2 hours, the patient went into shock and was reexplored again but the patient succumbed to the continued hemorrhage.Gross specimen measured 18 × 12 × 6 cm and cut surface showed multiple lobulated, solid, well-circumscribed nodules of variable sizes 1 cm to 7 cm diameter with areas of hemorrhage and cystic degeneration replacing most of splenic parenchyma  being a recurrent translocation. Hallen suggested that karyotypic pattern of hemangiopericytoma of spleen may differ from hemangiopericytoma of soft tissue . It is dThis case is of multiple abscesses of spleen showing unusual histopathology. Till date, hemangiopericytoma of spleen was described with recurrent massive upper gastrointestinal bleeding or serosal cavernous hemangiomatosis of the adjacent colon or pain or rupture or lump but no case was reported with presentation as multiple abscesses in spleen. -13.The author(s) declare that they have no competing interests.All the authors equally participated in preparation of the manuscript.None


IARC Monographs on the Evaluation of the Carcinogenic Risk of Chemicals to Humans (IARC Monographs) on formaldehyde, 2-butoxyethanol, and 1-tert-butoxy-2-propanol. Each IARC Monograph includes a critical review of the pertinent scientific literature and an evaluation of an agent’s potential to cause cancer in humans. After a thorough discussion of the epidemiologic, experimental, and other relevant data, the working group concluded that formaldehyde is carcinogenic to humans, based on sufficient evidence in humans and in experimental animals. In the epidemiologic studies, there was sufficient evidence that formaldehyde causes nasopharyngeal cancer, “strong but not sufficient” evidence of leukemia, and limited evidence of sinonasal cancer. The working group also concluded that 2-butoxyethanol and 1-tert-butoxy-2-propanol are not classifiable as to their carcinogenicity to humans, each having limited evidence in experimental animals and inadequate evidence in humans. These three evaluations and the supporting data will be-published as Volume 88 of the IARC Monographs.An international, interdisciplinary working group of expert scientists met in June 2004 to develop  IARC Monographs on the Evaluation of the Carcinogenic Risk of Chemicals to Humans (IARC Monographs) on formaldehyde, 2-butoxyethanol, and 1-tert-butoxy-2-propanol . This is the fourth IARC evaluation of formaldehyde and the first of the glycol ethers.Twenty-six scientists from 10 countries met at the International Agency for Research on Cancer (IARC) in June 2004 to develop Formaldehyde is widely used in resins that bind wood products, pulp and paper, and glasswool and rockwool insulation. It is also used in plastics and coatings, textile finishing, and chemical manufacturing and as a disinfectant and preservative. High concentrations can be found in some work environments, and much lower concentrations in homes.2-Butoxyethanol is a glycol ether widely used as a solvent in paints, paint thinners, glass-cleaning and surface-cleaning products , and personal-care and other personal products and as a chemical intermediate. General-population exposure can occur through the use of consumer products, particularly cleaning agents.tert-Butoxy-2-propanol is a glycol ether that has found increasing use as a solvent in coatings, glass-cleaning and surface-cleaning products, inks, adhesives, and nail-polish lacquers.1-IARC Monographs. The working group writes a critical review of the pertinent scientific literature  and a consensus evaluation of each agent’s potential to cause cancer in humans.IARC convenes an international, interdisciplinary working group of expert scientists to develop each volume of the IARC Monographs are developed during an 8-day meeting whose objectives are review and consensus. Before the meeting, each member of the working group writes a portion of the critical review. At the meeting, four subgroups  review these drafts and develop consensus subgroup drafts. Then the working group meets in plenary session to review the subgroup drafts and develop a consensus evaluation. After the meeting, IARC scientists review the final draft for accuracy and clarity before publication.The The evaluation is developed in steps . The subWhen the epidemiologic evidence is sufficient, the final evaluation is carcinogenic to humans, regardless of the experimental evidence. In other cases, the mechanistic and other relevant data are considered to determine whether the default evaluation should be modified upward or downward. A subgroup of experts in cancer mechanisms assesses the strength of the mechanistic data and whether the mechanisms of tumor formation in experimental animals can operate in humans. The overall evaluation is a matter of scientific judgment, reflecting the combined weight of the evidence.Working groups are selected to invite the best-qualified experts and to avoid real or apparent conflicts of interests. Consideration is given also to demographic diversity and a balanced representation of all scientific views. Each potential participant submits a Declaration of Interests , which IThere was a statistically significant excess of deaths from nasopharyngeal cancer in the largest and most informative cohort study of industrial workers , with stExcess mortality from leukemia, primarily of the myeloid type, has been observed relatively consistently in six of seven studies of embalmers, funeral parlor workers, pathologists, and anatomists , 1984. AThere was no excess mortality from leukemia when the industrial workers were compared with the general U.S. population, but a comparison with the general population may be biased. In another study, excess mortality from leukemia was found in the recent update of garment workers in the United States . This exSeveral case–control studies have investigated the relationship between formaldehyde exposure and sinonasal cancer. A pooled analysis of 12 studies showed an increased risk of adenocarcinoma in men and women thought never to have been exposed to wood dust or leather dust, with an exposure–response trend for an index of cumulative exposure . One othIn experimental animals, several studies have shown that inhalation exposure induces squamous cell carcinomas of the nasal cavities in rats , althoug14C-formaldehyde resulted in some excretion in urine and feces within 12 hr , based on sufficient evidence in humans and sufficient evidence in experimental animals. Based on the information now available, this classification is higher than those of previous IARC evaluations , 1995.2-Butoxyethanol was tested for carcinogenicity by inhalation exposure in male and female mice and rats . Clear iRegarding mechanisms of carcinogenesis, the working group considered that hemolysis and associated oxidative stress in the liver have been proposed to be linked to the induction of mouse liver neoplasia. They also considered that, in view of lower sensitivity to hemolysis of human erythrocytes and higher human liver concentrations of the antioxidant vitamin E, the induction of liver tumors in humans would be improbable through this pathway, but it was noted that other potential mechanisms have not been investigated. The working group observed that the mouse forestomach tumors are associated with high local exposure to 2-butoxyethanol and high local concentrations of the toxic metabolite 2-butoxyacetic acid.The working group concluded that 2-butoxyethanol is not classifiable as to its carcinogenicity to humans (group 3), with limited evidence in experimental animals and inadequate evidence in humans.tert-Butoxy-2-propanol was tested for carcinogenicity by inhalation exposure in male and female mice and rats , with limited evidence in experimental animals and inadequate evidence in humans.The working group concluded that 1-A theme common to these three evaluations is the consideration of mechanistic information to develop and evaluate hypotheses about the sequence of steps leading to the induction of tumors in experimental animals. The hypothesized mechanisms described in these evaluations provide an interesting set of cases that range from a vast literature on respiratory-tract tumors in rats induced by inhalation of formaldehyde to some more tentative hypotheses about the various tumors observed in animals after exposure to glycol ethers. Both types of mechanistic data sets were of use in the evaluation process.The evaluation of formaldehyde as carcinogenic to humans shows the importance of mechanistic information in the classification of carcinogens. For the nasopharyngeal tumors, the working group discussed the convergence of the epidemiologic, experimental, and mechanistic evidence. If the evidence in humans had been less than sufficient, the strong mechanistic evidence in exposed humans and sufficient evidence in experimental animals might still have led to classification as group 1. The extensive mechanistic data for formaldehyde-induced respiratory cancer provide strong support for the empirical observation of nasopharyngeal cancer in humans, although computer models that predict an anterior-to-posterior gradient of formaldehyde deposition in the upper respiratory tract would predict that formaldehyde would cause cancer in the nose as well as the nasopharynx in humans.On the other hand, the lack of information on possible mechanisms by which formaldehyde might increase the risk of leukemia in humans tempered the interpretation of the epidemiologic data on that cancer type. The entire working group discussed at length this divergence between the epidemiologic and mechanistic conclusions for leukemia. Information to support a biologically plausible mechanism could have supported a stronger conclusion about the evidence of leukemia in humans.tert-butoxy-2-propanol. This view emphasized the dose-related induction of hepatoblastoma in male and female mice, considering hepatoblastoma as a rare neoplasm with low spontaneous incidence in mice, especially in females. Most of the working group, nevertheless, considered the evidence to be limited, based on the interpretation of hepatoblastoma being a variant of hepatocellular carcinoma.In the evaluations of the glycol ethers, the working group grappled with questions of interpretation and scientific judgment. A recurring issue was the criterion for characterizing a rare tumor or an unusual set of observations that can carry greater weight than a typical bioassay result. A related matter was how to bring in additional information to resolve difficult questions—for example, how to consider the results of historical controls or alternative statistical tests. When the working group tried to, but could not, reach consensus on a question of interpretation or scientific judgment, the evaluation presented the differing positions favored by its members. For example, after thorough discussion, several members of the working group favored an evaluation of the carcinogenicity in experimental animals as sufficient for 1-It is important to note that the evaluation of an agent as not classifiable as to its carcinogenicity to humans is not a determination of safety, with respect to both cancer and effects other than cancer. It indicates that the data did not meet the minimum standards developed by the IARC for sufficient evidence in experimental animals and suggests that further testing is needed, particularly when there is widespread human exposure or another reason for public health concern.


The future is expected to involve partnering with other groups to maximize scarce resources. Resources should be directed to facilitating implementation of existing guidelines rather than to developing new documents. The full spectrum of cancer care includes prevention, screening, diagnosis, primary treatment, follow-up, treatment of recurrent disease, and palliation. High-quality evidence is available in some areas, but gaps exist where guideline panels could provide guidance. Development of a pan-Canadian gynecologic oncology process could provide an opportunity to influence access to care at the political and policy levels.The themes influencing practice are consistent with goc will develop linkages such that the toolbox available through cscc-cpg-ag can be incorporated into future collaboration.The  Several provinces develop practice guidelines, and additional guidelines are available from other groups within and outside of Canada. Unfortunately, existing guidelines that address a given clinical situation may vary with regard to the methodologic quality of their development and to the recommendations made. These inconsistencies can cause confusion for clinicians and patients. Additional challenges can arise from suboptimal dissemination and implementation strategies. Guidelines are meant to enhance care, and the barriers that hamper the translation of new knowledge and guideline recommendations into clinical practice can also hinder effective uptake of other types of systematic clinical advice, such as targeted messaging and clinical leadership.Not all institutions, provinces, and professional associations have the resources necessary to develop, disseminate, implement, and evaluate guidelines. And in any case, reviewing a common evidence base and developing parallel guidelines represents a duplication of effort that does not make sense in the face of numerous competing demands on scarce resources. Where high-quality guidelines exist, adoption—with or without adaptation of the recommendations to reflect local circumstances and values—may be preferable to developing new guidelines. Canada has people with acknowledged expertise in both guideline development and guideline adaptation; many of those experts participated in the workshop.goc not only can add credibility to guidelines by endorsing them, but they can also provide vital networks for guideline dissemination, implementation, and evaluation. As education influentials in the gynecologic oncology community, goc members can also make a unique contribution to interpretation of evidence, provision of clinical context, identification of gaps in the evidence, and provision of well-reasoned expert opinion to fill the gaps until further research is completed.Groups such as With those issues in mind, the workshop agenda focused on three sets of objectives:goc wants to actively influence practice in the care of women with gynecologic cancer, and if so, the methods to be used. Specifically, to determine if goc should take an active leadership role in guideline development, adaptation, implementation, or evaluation, and to determine the extent to which such activities are to be evidence-based.To determine if To explore a collaborative model for designing a pan-Canadian strategy to develop and implement evidence-based clinical practice guidelines for gynecologic oncology.cpg Evaluation and Adaptation Cycle To investigate the utility of an established methodology, the cpg Evaluation and Adaptation Cycle will be described in a separate report.The present report describes discussions related to the first two sets of objectives. Methods, results, and conclusions associated with the sogc) with national guidelines, and an overview of cpg adaptation and adoption models and methods As a preliminary step for all workshop activities, participants were provided with common background information through a review of three different provincial approaches  for developing practice guidelines and treatment policies, a summary of the experience of the Society of Obstetricians and Gynecologists of Canada  addressed all three key questions related to goc want to influence practice in women with gynecologic cancer?Does goc use to inform practice at a pan-Canadian level?What methods should What role should an evidence-based process play in influencing practice?Following small-group discussion of each question, the individual groups reported their thoughts for further discussion by all participants before moving to the next question.Issues emerging from discussion during the breakout and plenary sessions related to four key themes:gocThe current and future role of Making the most of limited resourcesProviding guidance across the full continuum of careInfluencing funding for cancer care with a view to facilitating equity of access across provincesgoc’s mandate and is supported by its mission statement. Moreover, goc may be the group most likely to effectively influence practice across Canada. Currently, goc is viewed as having an advisory and educational role. The group could build on that experience by developing a more “outward looking” approach that would move their influence beyond goc’s membership to other specialty groups and trainees and that would solidify goc’s role as a knowledge broker for other clinicians. This process could start by dealing with the large unfilled demand from gynecologists outside cancer centres for leadership and high-quality, implementable guidelines related to cancer care.Workshop participants agreed that influencing practice is consistent with goc should consider, leading to the conclusion that oncology topics (such as systemic therapy for ovarian cancer) should be tackled first. Strengths on which goc can draw include its interdisciplinary approach and its experience with continuing medical education over the Internet.Some debate occurred about the range of guideline topics that goc should consider developing partnerships with other groups to maximize scarce resources and to explore funding opportunities. One of these partners might be the sogc, but the relationship between goc and the sogc with respect to guideline development and dissemination would need to be clearly defined. For example, goc guidelines would likely be developed independently of sogc, but sogc members could provide feedback on draft guidelines, and the sogc may wish to endorse goc guidelines for use by gynecologists across Canada.In addition to aiding in clinical decision-making, practice guidelines can have educational value. Plans to endorse information for patients with gynecologic cancer could also be incorporated into a program for influencing practice. In mapping out future activities, The number of oncologists across Canada is limited. To address the need for additional resources, the health care system should engage and educate a broad range of practitioners  to provide some aspects of care to women with gynecologic cancers.The human resource issues for providing care are also reflected in the limited number of individuals available to develop guidelines and other mechanisms to influence practice. In many cases, the same clinicians who deliver care to patients and act as practice leaders would be the key players in guideline development. In light of limited infrastructure and resources for influencing practice, workshop participants wondered if such resources should be directed to facilitating implementation of existing guidance rather than to developing new guidance documents. Broad participation in guideline- and policy-development activities may require a cultural shift for some practice communities so that time is allocated for this work. Such a shift may be more challenging in some settings  than in others. However, successful involvement of clinicians across Canada in guideline development, endorsement, dissemination, and implementation has the potential to build capacity, reduce duplication of effort, and improve patient care and outcomes. Evaluating the impact of these new activities on practice and patient outcomes will be critical.The full spectrum of cancer care includes prevention, screening, diagnosis, primary treatment , follow-up, treatment of recurrent disease, and palliation. The availability of high-quality evidence varies across this continuum, as does the need for guidance to clinicians and patients. In some cases, guidance is needed in circumstances where evidence of effectiveness is absent, inconsistent, or unclear. The result can be clinical uncertainty and variation in practice. Evidence-based guidelines help to inform clinical decisions by making recommendations based on a systematic review and interpretation of the available evidence. Where gaps in the evidence exist, guideline panels could provide guidance by using a formalized process to make recommendations based on expert opinion.cpd), patient advocacy, lobbying, treatment policies, algorithms, care paths, consensus statements, patient information, and clinical research. Many of these activities can be informed by evidence-based practice guidelines.Activities other than practice guidelines that may influence care include continuous professional development  was endorsed as an approach to providing advice to clinicians.For the time being, efforts should concentrate on guideline implementation issues, because developing practice guidelines requires infrastructure and resources not currently available to goc members could also respond to emerging evidence.The Society is in an ideal position to identify common issues in practice as priorities for guidance documents across the entire spectrum of care in gynecologic oncology, from prevention through to palliation. The Society’s role is to promote a culture of evidence-based practice in gynecologic oncology. As clinical experts, goc could consider partnering with groups that develop evidence-based guidelines to work on new guidelines or to update existing guidelines. Such work should be based on a defined process or structure, with involvement of appropriate experts in guideline and clinical research methods. Criteria—for example, need, potential impact on care, quality of the evidence base, availability of an existing systematic review or guideline—should be defined for choosing topics for guideline development or adaptation. Guidelines and other advice documents should include the patient perspective and an economic evaluation. When available, existing systematic reviews and good-quality guidelines could be the starting point for an evidence-based approach. Where evidence is insufficient, a formalized consensus process could be used to develop advice.In addition to endorsing or adapting evidence-based guidelines from other groups, goc should develop explicit implementation and evaluation strategies based on evidence of effectiveness.The Society could also contribute by identifying gaps in the evidence and advocating for appropriate clinical trials to fill those gaps. With its partners, goc and cpg-ag have common objectives with respect to influencing clinical practice.Overall, participants concurred that the workshop was a positive experience. Based on the presentations and ensuing discussion, they concluded that effective tools exist for guideline development, adaptation, and adoption, but that delivering guidelines to practitioners during the patient encounter presents a significant challenge. It became clear during the workshop that ag was still determining its mandate, the scope of its activities, and a model for a pan-Canadian approach. Representatives of the ag found the workshop to be useful as a pilot test for developing beneficial partnerships in the future. The goc representatives concluded that goc has a mandate to use clinical education to influence care and that this mandate may extend to other activities such as providing evidence-based advice to practitioners.When the workshop took place, the


In the United Kingdom, the National Health Service (NHS) has embarked on the formation of health action zones for the delivery of good quality community-centric healthcare. It is recognised that the evidence-based healthcare practice is critical in this venture, and the optimistic projection is that community-centric healthcare encompassing medical, dental, nursing, health visiting, optical and pharmaceutical services will be brought under the evidence-based healthcare practice framework. The evidence-based medicine and its practice in its 'first wave' with its six steps provided a paradigm shift in the clinical practice by integrating experiential and external evidence with clinical practice to arrive at clinical decisions and establish disease management framework. The internet was rightly considered as the key player in the evidence-based healthcare practice. The shortcomings of this global system has constrained the wide-spread application of the evidence-based practice.We will first take an in-depth look at the relevance, quality and accessibility issues, by presenting examples from respiratory and metabolic disease domains, particularly from the classes of paediatric metabolic disorders. While analysing the information presented, we will also assess to what degree the information needs of clinicians as well as patient/home-carers are met. We will examine what is required in terms of structure, protocols, information storage and access to meet the two often disparate needs.We will argue that the information should be in a form that enables their effective assimilation thereby enhancing the usage of the evidence-based practice. Concepts of componentware, model-based information- bayesian causal graphs will presented with examples. Examples of emerging database implementations will be referred to in these examples. Our on-going research work on MEDINFONET, and models of domain-specific software agents for knowledge management will be presented.The service-centric model of the next generation internet presents a bright perspective. From the evidence-based healthcare practice point-of-view, its differential service (diff-serv) facility with emphasis on classes of services will largely meet the speed of access requirement. The 'object web' with data integration facility, digital libraries, domain-specific search agents etc.. will provide a better information quality. We argue that models and metrics for information systems and its environment are needed, and in this connection present a five-layer maturity model for evidence-based healthcare practice.


Systems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.A GRADE evidence profile consists of two tables: a quality assessment and a summary of findings. Twelve evidence profiles were used in this pilot study. Each evidence profile was made based on information available in a systematic review. Seventeen people were given instructions and independently graded the level of evidence and strength of recommendation for each of the 12 evidence profiles. For each example judgements were collected, summarised and discussed in the group with the aim of improving the proposed grading system. Kappas were calculated as a measure of chance-corrected agreement for the quality of evidence for each outcome for each of the twelve evidence profiles. The seventeen judges were also asked about the ease of understanding and the sensibility of the approach. All of the judgements were recorded and disagreements discussed.There was a varied amount of agreement on the quality of evidence for the outcomes relating to each of the twelve questions (kappa coefficients for agreement beyond chance ranged from 0 to 0.82). However, there was fair agreement about the relative importance of each outcome. There was poor agreement about the balance of benefits and harms and recommendations. Most of the disagreements were easily resolved through discussion. In general we found the GRADE approach to be clear, understandable and sensible. Some modifications were made in the approach and it was agreed that more information was needed in the evidence profiles.Judgements about evidence and recommendations are complex. Some subjectivity, especially regarding recommendations, is unavoidable. We believe our system for guiding these complex judgements appropriately balances the need for simplicity with the need for full and transparent consideration of all important issues. Reviewers and users of reviews draw conclusions about the overall quality of the evidence that is reviewed. Similarly, people making recommendations and users of those recommendations draw conclusions about the strength of the recommendations that are made. Systematic approaches to doing this can help protect against errors by both doers and users, and can facilitate critical appraisal and communication of the conclusions that are made.The GRADE Working Group began as an informal collaboration of people with an interest in addressing shortcomings in systems for grading evidence and recommendations. We report elsewhere a critical appraisal of six prominent systems for grading evidence and recommendations . Based oThe aims of the pilot study were to test whether the approach is sensible relative to diverse examples of evidence and recommendations, and to agree on necessary changes to the approach, decision rules, and changes in how the evidence profiles used in the pilot study were constructed. The revised approach is described elsewhere .Seventeen people independently judged the quality of evidence, the balance between benefits and harms, and the formulation of a recommendation for 12 examples. The 17 judges all had experience using other approaches to grade evidence and recommendations.For each example we prepared an evidence profile. Each evidence profile was made based on information available in a systematic review and consists of two tables, one for quality assessment of the available information and one table that presents a summary of the findings Table . For theInstructions and a form for recording each judgement were included with each example see . The judThe 12 examples were selected to include a variety of health care interventions, types of evidence and types of recommendations. The questions that were posed in the 12 examples were:• Should depressed patients in primary care be treated with SSRIs or tricyclics? • Should patients with atrial fibrillation be treated with warfarin or aspirin for prevention of stroke? • Should patients with pain believed to be due to degenerative arthritis be treated with non-steroidal anti-inflammatory drugs (NSAIDs) or paracetamol? • Should patients who have had a myocardial infarction be given antiplatelet therapy to reduce all cause mortality? • Should patients who have had a myocardial infarction be offered exercise rehabilitation? • Should patients with deep venous thrombosis be treated with Low Molecular Weight Heparin (LMWH) or IV unfractionated heparin for prevention of pulmonary embolism? • Should antibiotics be used to treat acute maxillary sinusitis? • Should BCG vaccine be used to prevent tuberculosis? • Should surgical discectomy be recommended for patients with sciatica due to lumbar disc prolapse? • Should community water fluoridation be used to reduce dental caries? ,11• Should distribution of child safety seats and education programs be used to increase correct use of child safety seats? • Should hormone replacement therapy be given to prevent cardiovascular heart disease in healthy post menopausal women? For each example each person made judgements about;• the quality of evidence for each outcome, scored as high, intermediate, low, or very low;• the relative importance of each outcome, scored as critical to the decision (7–9), important but not critical to the decision (4–6), or not important to the decision (1–3);• the overall quality of all the critical outcomes, scored as high, intermediate, low, or very low;• the balance between benefits and harms, scored as net benefit, trade offs, uncertain net benefit, or not net benefit; and• the recommendation, scored as do it, probably do it, toss up, probably don't do it, or don't do it.For each example the judgements made by all 17 people were collected and summarised as illustrated in Table After grading all 12 examples, the judges were asked 16 questions regarding the sensibility and understandability of the approach. Each question consisted of a statement and five response options: strongly disagree, disagree, not sure, agree, and strongly agree. Eleven people completed this questionnaire. The questionnaire was adapted from Feinstein  and the 1. The approach is applicable to different types of interventions, including drugs, surgery, counselling, and community-based interventions.2. The approach is clear and simple to apply3. The information that is needed is generally available.4. Subjective decisions are generally not needed.5. All of the components included in each of the five types of judgements should be included6. There are not important components that are missing for any of the five types of judgements.7. The ways in which the components are aggregated for each of the five types of judgements are clear and simple.8. The ways in which the included components are aggregated are appropriate for each of the five types of judgements.9. The categories are sufficient to discriminate between different grades for each of the five types of judgements.10. The approach successfully discriminates between different grades of evidence.11. The approach successfully discriminates between different grades of recommendations.12. The overall quality of evidence is clear and understandable.13. The balance between the benefits and harms is clear and understandable.14. The recommendation is clear and understandable.15. The way in which the overall quality of evidence was graded is better than other ways of doing this with which I am familiar.16. The way in which the recommendation was graded is better than other ways of doing this with which I am familiar.The quality of evidence for each outcome as assessed by the 17 graders are shown in Table We found that in addition to study design, quality, consistency and directness, other quality criteria also influenced judgements about evidence. These additional criteria were sparse data, strong associations, publication bias, dose response, and situations where all plausible confounders strengthened rather than weakened our confidence in the direction of the effect. Concequently, the consistency with which we considered these additional issues were affected and disagreements regarding the quality of evidence for each outcome were reduced.Specification of outcomes in the question that each example addressed resulted in some confusion regarding the relative importance of each outcome and the overall quality of evidence across outcomes. We therefore agreed that outcomes should not be included in the questions and that all important outcomes should be considered. There was good agreement about the relative importance of the 48 outcomes that were considered. We reached a consensus about the relative importance of all but two of the outcomes. This was due to uncertainty and true disagreement about the importance of these two outcomes, dental fluorosis and bone fractures, in relation to the question about water fluoridation.There was a lack of agreement about the overall quality of evidence across the critical outcomes for each question Table . This poThe kappa statistics for each question are shown in Table The graders assessments about the balance between benefits and harms are shown in Table The graders individual considerations about the recommendations are shown in Table Eleven raters provided feedback on the sensibility and understandability of the GRADE system for grading evidence and formulating recommendations. Nine of the 11 respondents agreed or strongly agreed that the judgements about the overall quality of evidence were clear and understandable, and that the judgements about the balance between benefits and harms were clear and understandable using the GRADE approach. Everyone agreed or strongly agreed that the judgements about recommendations were clear and understandable. Eight of the judges agreed or strongly agreed that the GRADE approach to judging the overall quality of evidence was better than other grading systems with which they were familiar. Two disagreed and one was not sure. Eight also agreed that the GRADE approach to formulating recommendations was better than approaches with which the raters were familiar. Three raters were not sure about whether the GRADE approach was superior to other approaches of formulating recomendations.Nine of the 11 respondents agreed or strongly agreed that the GRADE approach was applicable to different types of interventions, and that the approach was clear and simple to apply. Five judges disagreed that the information that is needed is generally available, two were not sure and four agreed. Six of the eleven judges disagreed or strongly disagreed that subjective decisions were generally not needed, four were not sure and one agreed. Ten of the eleven judges agreed or strongly agreed that all the components included in each of the four types of judgements should be included; one judge was not sure. Five of the judges were unsure if there were not important components that were missing from any of the four types of judgements, one disagreed and three agreed or strongly agreed. Eight judges agreed or strongly agreed that the ways in which the components were aggregated for each of the four types of judgements were clear and simple; three were unsure. Seven judges agreed or strongly agreed that the ways in which the included components were aggregated were appropriate for each of the four types of judgements, two were unsure and two disagree. Ten of the eleven judges agreed or strongly agreed that the categories were sufficient to discriminate between different grades for each of the four types of judgements; one disagreed. All the eleven judges agreed or strongly agreed that the GRADE approach successfully discriminated between different quality of evidence, and between different grades of recommendations.This pilot study of the GRADE approach to grading the quality of evidence and strength of recommendations helped to identify problems with the approach and enabled us to address these. We found that it was possible to resolve most of the disagreements we had when making judgements independently and there was agreement that this approach warrants further development and evaluation.Many of the disagreements were a direct result of a lack of information. We concluded that there is a need for detailed additional information in evidence profiles, and have modified the evidence profiles accordingly. When we have found an empirical basis or compelling arguments, we have also provided precise definitions. For example, we have agreed on a basis for defining strong and very strong associations. However, in many cases we continue to rely on judgement. We have addressed this by always including the rationale for such judgements in footnotes attached to the evidence profile.The evidence profiles used in the pilot study were based on systematic reviews. -13 Much An important benefit of the approach to grading evidence and recommendations that we used in this study is that it clarifies the source of true disagreements, as well as helping to resolve disagreements through discussing each type of judgement sequentially. Judgements about the relative importance of different outcomes and about trade-offs, as well as about the quality of evidence, are made explicitly, rather than implicitly. This facilitates discussion and clarification of these judgements. It may be helpful to guideline panels and others to use this approach before making decisions and recommendations.The most common source of disagreement that we encountered was differences in what we consider to be sparse data. We have not reached a consensus on a definition of sparse data, but have acknowledged that we have different thresholds and now recognize this when we make judgements about the quality of evidence .We have as a result of this pilot study been able to make considerable improvements to our system for grading the quality of evidence and strength of recommendations. The evidence profiles used in the pilot study have been modified and now include information that was missing and was found to be an important source of disagreement, as illustrated in Table Judgements about confidence in evidence and recommendations are complex. The GRADE system represents our current thinking about how to reduce errors and improve communication of these complex judgements. Ongoing developments include:• Exploring the extent to which the same system should be applied to public health and health policy decisions as well as clinical decisions• Developing guidance for when and how costs (resource utilisation) should be considered• Developing guidance for judgements regarding sparse data• Adapting the approach to accommodate recommendations about diagnostic tests when these are based on evidence of test accuracy• Incorporating considerations about equity• Preparing tools to support the application of the GRADE systemPlans for further development include studies of the reliability and sensibility of this approach and a study comparing alternative ways of presenting these judgements . We inviBased on the results of this pilot study we have been able to considerably improve our system for grading the quality of evidence and strength of recommendation .DA has competing interests with the US Preventive Services Task Force (USPSTF), PAB has a competing interest with the US Task Force on Community Preventive Services (USTFCPS), ME has and JM had competing interests with the National Institute for Clinical Excellence (NICE), GHG, RJ and HS have competing interests with the American College of Chest Physicians (ACCP), RTH has competing interests with the Scottish Intercollegiate Guidelines Network (SIGN), SH and DO'C have competing interests with the Australian National Health and Medical Research Council (ANHMRC), BP has competing interests with the Oxford Centre for Evidence-Based Medicine (OCEBM). Most of the other members of the GRADE Working Group have experience with the use of one or more systems for grading evidence and recommendations.DA, PAB, ME, SF, GHG, RTH, SH, RJ, AL, NM, JM, DO'C, ADO, BP, HS, TTTE, GEV & JWJ Jr as members of the GRADE Working Group have contributed to the preparation of this manuscript and the development of the ideas contained herein, participated in the pilot study, and read and commented on drafts of this article. All authors except GEV judged the quality of the evidence and strength of recommendation based on information presented in the evidence profiles. GEV prepared the first draft of this article, had primary responsibility for preparing the evidence profiles used in the study, and coordinated the study.The pre-publication history for this paper can be accessed here:Instructions and form for judgements used in the pilot studyClick here for file


In an effort to ensure that physicians have access to reliable evidence on drug effectiveness and safety, the Italian Drug Agency launched a program to disseminate independent information. Interest in evidence-based medicine (EBM) is growing in Italy, although its impact upon health policies and clinical practice is unclear. Rather than getting health information from unbiased evidence-based sources, doctors in Italy still rely heavily upon the pharmaceutical industry for their information needs. For example, a recent survey showed that general practitioners receive 11 visits per week by pharmaceutical sales representatives , and observational studies where appropriate, and if there is no good evidence it says so.”According to its Web site, Clinical Evidence, and to speed up the diffusion of EBM, AIFA sponsored a free-access e-learning system, based on Clinical Evidence, called ECCE .A compulsory system of continuing medical education (CME) for all health professionals was introduced in Italy in 1998, based on credits awarded for time spent on educational activities. The more traditional form of acquiring CME is to attend lectures and conferences; it is much more rare for doctors to be exposed to small group interactive events. In order to maximise the effectiveness of the financial commitment for disseminating Clinical Evidence and a predefined sequence of questions. The vignettes reflect real-life circumstances as seen by an ordinary general practitioner in everyday practice. Whilst the primary target group of ECCE is general practitioners, many vignettes are also relevant to specialists.ECCE is an e-learning CME tool that uses interactive clinical vignettes based on chapters in Each vignette has a narrative with events presented in chronological order: the history evolves with new insights from diagnostic tests or additional information reported by the patient. Users progress from step to step by overcoming a question/answer decision system. We present the first step of the headache (chronic tension-type) vignette and the related question. The vignette was developed from the chapter Headache (chronic tension-type), in Clinical Evidence .Margaret says to her family doctor: “This time I didn't come for me, but to talk about Rachel, my 25-year-old daughter. As you probably remember, she got married last year; unfortunately, she doesn't seem to get along well with her husband... Anyway, the other day she told me that in the past few months she has often suffered from headache; I'm quite worried about that, you know what I have been through...” The doctor remembers very well that many years before, Margaret, one of his first patients, was always complaining about her headache, a pain that tormented her all the time and was not relieved by analgesics. He tells Margaret to come back with her daughter. After a few days the two women are in the doctor's office. Rachel says: “This headache is killing me. I have it every day now, sometimes with nausea. And I don't want to take analgesics any more, they don't do me any good”. She describes her pain as a bilateral, tight, “band-like” discomfort: “My head feels as if it is in a vice. Following the advice of a friend of mine I have also tried to take some drops of a benzodiazepine; but the headache didn't go away, and I felt drowsy and light headed”. Rachel looks very pale, tired and tense. Examining her, the doctor doesn't find anything abnormal; he suspects a chronic tension-type headache, and prescribes a battery of blood tests.Question: According to the studies identified by Clinical Evidence, in Rachel's case benzodiazepines:may be very effective, but only in the short termmay be effective, but only in the long termmay be useful only in people with very severe headachemay induce a modest short term improvement, but are often associated with adverse effectsare contraindicatedThe correct answer is: In Rachel's case, benzodiazepines may induce a modest short term improvement, but are often associated with adverse effects.The response option includes one or more correct answers for each question, and other possible responses as distractors. Users gain credits upon completing all steps as long as they reach a score above 80% of the maximum total score. Vignettes provide one or two credits depending on the number of questions in the vignette. During the first year we posted 120 vignettes.Clinical Evidence can be read on screen or printed and interactively managed along the steps of each vignette. The system tracks learning content and the learner's progress. Eight vignettes translated in English are available at http://ecce-gb.fad-ecm.it (registration required).ECCE has all the standard advantages of e-learning. Users select what they wish to learn, when and at what pace. The system is easy to use and works with basic computer requirements . The contents of ECCE became accessible to all physicians in March 2005 after a pilot period. Here we present the results of the first year of usage (until February 2006).In one year, 19,340 doctors voluntarily subscribed to ECCE  and around 93% logged in completing at least one vignette . Almost Doctors used ECCE throughout the entire day, with early morning and late evening being the most popular time slots. Altogether 215,412 vignettes have been completed and 292,058 credits awarded. The average number of completed vignettes for a single user was 13.75 , with a corresponding average credit of 16.22. There were 3,468 doctors who obtained 25 or more credits (30.2% of those who acquired at least one credit), exceeding the mandatory ministerial requirement (24 credits in 2005 for distance learning).Clinical Evidence (which is assumed to take five minutes per page).The top five accessed vignettes were appendicitis, atrial fibrillation, herpes zoster, paracetamol poisoning, and acute low back pain. The easiest vignette to be solved had a success rate of 93.91% at first attempt, while the average success rate was 79.19% and the hardest  case had a 39.10% success rate. Mean time to interactively complete a vignette was nine minutes for one credit history and 18 minutes for two credit histories. Before completing the vignette, doctors are supposed to peruse the related chapter in Clinical Evidence, translation of the fourth edition into Italian, development of the e-learning CME system and its management and technological platform). The 2.4 Euros excludes the costs of printing Clinical Evidence as well as indirect costs. All costs are tax free. Discounting the costs related to translation of Clinical Evidence and to the start-up of the platform, the cost per credit was 0.7 Euros.During the first year the cost per credit supplied through ECCE was 2.4 Euros . ECCE's vignettes were well received : more thClinical Evidence focuses on treatment, not diagnosis). The clinicians complained that reading the full chapter of Clinical Evidence was tedious, but appreciated the format of the clinical vignette, which allowed them to switch from passive reading to an active learning exercise.Overall, these opinions were largely consistent with what emerged from three focus groups run in northern, southern, and central Italy. The 26 clinicians who participated felt that ECCE was well representative of their clinical practice and that the information conveyed was reliable and relevant, despite the absence of evidence on how to make diagnoses  of all practising doctors. But there has been limited promotion of ECCE, involving only a small number of medical journal adverts, and the program is still in its infancy.This study has several limitations. The first limitation is selection bias: there may have been an over-representation in our sample of doctors with a generally positive attitude toward EBM and with greater computer skills. ECCE has probably filled the gap that still exists between a growing demand for reliable and independent critical appraisal and a limited offering . Thus thA second limitation relates to the CME structural factor : we cannot rule out the possibility that some doctors intentionally used ECCE with the aim of collecting as many credits as possible. This “opportunistic use” was also highlighted during the focus groups. However, at least 30.2% of doctors exceeded the CME mandatory requirement, after which they were presumably using ECCE to challenge their own knowledge and competence.The last two limitations deal with the innovation itself . Does the high success rate of the vignettes mean that ECCE was unable to discriminate poor performers from better ones? The average success rate was close to 80% (considered a moderately easy test). The test discrimination capability could be biased towards middle achievers, an attribute which could be seen negatively. We think that this is not negative itself: innovations that are not difficult to do (low complexity) and easy to try  maximise their perceived attraction and adoption .Clinical Evidence was boredom. The use of written case simulation seemed to transform the passive reading into a more interactive experience in which doctors searched for the right piece of information to be applied in specific situations.A fourth limitation is that positive results may reflect physician competence more than appropriate clinical practice. While changing provider behaviour could be seen as the final step of the innovation decision process , it shouClinical Evidence, a user-friendly medical resource free of conflicts of interest. This approach is different from that of other countries, which supported, for example, free access to Cochrane systematic reviews [Clinical Evidence and the Cochrane Library are related [Clinical Evidence was thought by policy makers to be more useful in daily practice by answering common and important clinical questions using clear summaries. Recognising the value of independent information, AIFA is attempting to balance a context where pharmaceutical companies have been the only information drivers for many years. The adoption of high quality information on EBM enhances doctors' mastery of information, emphasising sources that are high in relevance and validity, and that do not require a huge amount of time to access [AIFA supported the diffusion of  reviews . Althoug related , Clinicao access .In 2005 the cost for each credit provided through ECCE was less then 2.5 Euros. In 2006, maintaining the same patterns of usage, the estimated cost will be 0.7 Euros/credit, a reasonable effort considering that in 2004 the average cost of each CME credit in Emilia-Romagna, a region that monitored this expenditure, was 144.0 Euros . FurtherClinical Evidence, and the metaRegister of Controlled Trials would go some way to reducing the inequities in access to information for improving health care” [Cultural and policy changes are brought about by a complex interaction of social, economic, and political factors, including the influence of thought leaders. In 2001 Smith and Chalmers wrote: “Universal free access to an integrated information resource built from the Cochrane Library, th care” . Italy i