Systematic reviews provide evidence for clinical questions, however the literature suggests they are not used regularly by physicians for decision-making. A shortened systematic review format is proposed as one possible solution to address barriers, such as lack of time, experienced by busy clinicians. The purpose of this paper is to describe the development process of two shortened formats for a systematic review intended for use by primary care physicians as an information tool for clinical decision-making.We developed prototypes for two formats (case-based and evidence-expertise) that represent a summary of a full-length systematic review before seeking input from end-users. The process was composed of the following four phases: 1) selection of a systematic review and creation of initial prototypes that represent a shortened version of the systematic review; 2) a mapping exercise to identify obstacles described by clinicians in using clinical evidence in decision-making; 3) a heuristic evaluation (a usability inspection method); and 4) a review of the clinical content in the prototypes.After the initial prototypes were created (Phase 1), the mapping exercise (Phase 2) identified components that prompted modifications. Similarly, the heuristic evaluation and the clinical content review (Phase 3 and Phase 4) uncovered necessary changes. Revisions were made to the prototypes based on the results.Documentation of the processes for developing products or tools provides essential information about how they are tailored for the intended user. One step has been described that we hope will increase usability and uptake of these documents to end-users. Systematic reviews are one tool available to clinicians that provide the current best evidence. Ideally, authors of systematic reviews employ rigorous methods to select credible and relevant information to generate summative reports -3. AlthoSeveral clinical information tools currently exist that present information from systematic reviews in a shortened or summarized manner (such as the BMJ PICO abridged research articles). We completed two comprehensive reviews of the literature that examined the impact of interventions for seeking, appraising, and applying evidence from systematic reviews in decision-making by clinicians or policymakers ,10 and sThe development of the shortened systematic review formats is informed by the Knowledge-to-Action Cycle Figure  proposedTo describe the development process of two shortened formats for a systematic review intended for use by primary care physicians as an information tool for clinical decision-making.We employed a series of strategies to create two shortened formats, case-based and evidence-expertise, to aid in decision-making for clinicians before seeking input from users on their preferences. The components of the process included:1. selection of a systematic review and creation of initial prototypes that represent a shortened version of the systematic review;2. a mapping exercise to identify obstacles described by clinicians in using clinical evidence in decision making;3. a heuristic evaluation (a usability inspection method); and4. a review of the clinical content in the prototypes.We chose a full-length systematic review to be used for developing prototypes by having four generalist clinicians select from a list of systematic reviews that were drawn from 120 medical journals published in the last five years on topics relevant to primary care . These pTwo shortened formats were developed in collaboration with a human factors engineer using the selected systematic review. Human factors is the application of what is known about human capabilities and limitations to the design of tools in order to enable more effective use . Guidingi.e., specific to the information tool or document) to answering doctors’ questions about patient care with evidence; and to identify at least one attribute within each shortened systematic review format (case-based and evidence-expertise) that addresses these obstacles. The mapping exercise was not intended to provide a guarantee that each obstacle had been eliminated from the prototypes, but served as a methodical inspection of the documents, with the intrinsic obstacles as guidance for identifying at least one instance where they had been addressed.The aim of the mapping exercise was: to identify the intrinsic obstacles , and intrinsic factors . In our study, two people , an information specialist and a family physician, independently reviewed each obstacle and identified if it was an intrinsic factor of an information tool or document, or an extrinsic factor. The intrinsic obstacles are the elements that have the potential to be addressed in the development of an information tool or document. Discrepancies were resolved by discussion until consensus was reached.Ely and colleagues extensively studied the information needs of family physicians ,28-34. TWe reviewed both formats (case-based and evidence-expertise) to determine if they addressed obstacles identified as intrinsic factors. If intrinsic obstacles were not addressed, we changed the documents. For example, if the obstacle ‘resource not authoritative or trusted’ was not addressed, the citation  for the systematic review would be added.i.e., cosmetic, minor, moderate, major, or catastrophic. The severity estimates are based on frequency, impact, and persistence of errors.Heuristic evaluation is a usability inspection method that involves the examination of the prototypes by comparing them to recognized usability principles (the ‘heuristics’) . It is uClinical content was reviewed by a family physician (NP). The role of the clinical content reviewer was to ensure that the information was transferred from the original document to the shortened versions accurately  . The cliWe developed and refined a summary of a systematic review on rosacea in two formats, case-based and evidence-expertise, which addressed many obstacles clinicians encounter while searching for evidence-based answers to questions. As reported in the Methods section, we selected a systematic review of rosacea treatments and developed summaries in case-based and evidence-expertise formats.Thirty-two of 59 factors from Ely’s framework were indicated as intrinsic to an information tool. The strength of agreement between the two reviewers  was very good  . Ely and.’ This prompted the addition of a definition of odds ratio to the case-based format, given the supporting evidence that statistics commonly found in medical journals are not readily understood by clinicians [Eight items from Ely’s framework could not be addressed, as they were not applicable to the mapping. For the 24 items that were applicable, five were identified as being absent from one of the formats. For instance, both formats did not address the obstacle ‘failure to define important termsinicians -48. The inicians -52. Table.g., an evidence rating appearing in different columns of tables), or omissions that could potentially confuse readers . Minor issues concerned the small size of text and layout for the case-based format. We used all feedback to modify the prototype formats.The heuristic evaluation indicated that there were no major usability problems. Several moderate usability issues were identified, including wording that could potentially confuse readers, the placement of information (The clinical content review revealed that the evidence-expertise format accurately reflected the information in the full-length review. One issue was detected in the case-based prototype, and the reviewer recommended modifications to the case that included not focusing on iatrogenic rosacea, removing references to prednicarbate, and using the term ‘family physician’ instead of ‘general practitioner.’ All of these changes were made to the case-based prototype. Additional file We have described the components of the development process for two shortened formats of systematic reviews. Aside from the first phase of selecting and creating the initial prototype, each component of the development process stimulated alterations within the two formats. The second phase was mapping items within the prototypes to obstacles identified by doctors they encountered while searching for evidence-based information, as described by Ely and colleagues . Most obAlthough shortened formats may be familiar and currently available to clinicians, no formal evaluations of these formats have been published. This was confirmed when the 8,104 relevant records of the published and gray literature from our systematic reviews were also examined for studies that described alternate formats ,10. No aThe development process for the prototypes described in this paper needs to be considered within the context of certain limitations. It may not be possible for all groups to collaborate directly with a human factors engineer when developing information tools. One consideration is to hire consultants for this expertise and include this cost into the budget of research grants. Alternatively, online resources can be used to provide guidance . A singlReporting these steps and the outcomes has made the process for the development of the two prototypes transparent for users and publishers. As well, it encompasses one step of developing a viable document, which we hope will increase its usability and uptake to end-users.In the next step in the development of these prototypes, we plan to conduct focus groups with primary care physicians to gain their input on the format, presentation and layout of the revised prototypes after all revisions had been made. The purpose of the focus groups is to generate essential components of the shortened systematic reviews, and to seek reactions to these prototypes and their potential for clinical decision making. This activity will provide the opportunity to hear from users about their requirements when using such tools, as well as to make changes and correct problems as they emerge. Iterative focus groups allow the chance to take results and quickly incorporate them into the new design. This is an important step of the Knowledge-to-Action cycle that facilitates the tailoring of information tools to the needs of potential users . FollowiSharon E. Straus is an Associate Editor for Implementation Science. All editorial decisions regarding this manuscript were made independently by other editors. None disclosed for all other authors.SES conceived of the idea. JG and KAM provided guidance and advice on the methods of the study. LP, MK and NP participated in the mapping exercise. NP conducted the clinical content review. LP and AK prepared the original prototypes. LP wrote the manuscript, and all authors provided editorial advice. All authors read and approved the final manuscript.Obstacles to answering doctors’ questions about patient care with evidence.Click here for fileAssessment criteria for heuristic evaluation: modified for analysis of print materials.Click here for fileMapping exercise: Obstacles and descriptions of how they were addressed [.ddressed ,54.Click here for fileShortened format: Initial versions of two prototypes.Click here for fileShortened format: Revised versions of two prototypes.Click here for file


Clinical practice guidelines are an important element of evidence-based practice. Considering an often complicated body of evidence can be problematic for guideline developers, who in the past may have resorted to using levels of evidence of individual studies as a quasi-indicator for the strength of a recommendation. This paper reports on the production and trial of a methodology and associated processes to assist Australian guideline developers in considering a body of evidence and grading the resulting guideline recommendations.In recognition of the complexities of clinical guidelines and the multiple factors that influence choice in health care, a working group of experienced guideline consultants was formed under the auspices of the Australian National Health and Medical Research Council (NHMRC) to produce and pilot a framework to formulate and grade guideline recommendations. Consultation with national and international experts and extensive piloting informed the process.The FORM framework consists of five components  which are used by guideline developers to structure their decisions on how to convey the strength of a recommendation through wording and grading via a considered judgement form. In parallel (but separate from the grading process) guideline developers are asked to consider implementation implications for each recommendation.The framework has now been widely adopted by Australian guideline developers who find it to be a logical and intuitive way to formulate and grade recommendations in clinical practice guidelines. Best practice in health care should be guided by the results of research on the safety and effectiveness of different courses of clinical action. This evidence needs to be assembled, justified and presented in the form of health advice for multiple stakeholders including health professionals, decision makers and consumers of health care. Clinical practice guidelines are recognised as one of the best ways to present recommended courses of action based on research evidence, although recommendations are often presented inconsistently . Where sThe National Health and Medical Research Council (NHMRC) of Australia has been a world leader in developing and supporting the development of evidence-based health advice, including clinical practice guidelines. As early as 1999, the NHMRC commissioned and published 'Guidelines for Guideline Development' , anticipAustralian guideline developers must comply with NHMRC standards in order to gain NHMRC approval. These standards  have resulted in NHMRC approved guidelines being of higher quality than those developed outside NHMRC processes .By 2004, it had become clear that the NHMRC standards required expansion and revision in response to the rapid growth and diversification of clinical practice guidelines in Australia and elsewhere. There were two main areas where a need for revision was identified. The first was the need to develop a set of levels (or hierarchy) of evidence which would cover the different individual study designs used to address the different types of questions formulated by guideline development panels. This work  is outlined in Merlin et al . The secMany guideline recommendations have been rated solely according to the level of evidence of the individual studies contributing to that recommendation. In the late 1990 s and early 2000 s, NHMRC prepared a series of handbooks to assist clinical practice guideline developers. These handbooks stated that other elements such as study quality, size and precision of study results, and relevance to local practice were also important ,4. They In recognition of this need, and in response to requests from methodological experts that consult for the NHMRC on guideline development (Guidelines Assessment Register [GAR] consultants) (see Appendix 1), the NHMRC undertook to revise and update its methodological approaches. This paper reports on the production and trial of a methodology and associated processes to assist Australian guideline developers in considering a body of evidence and grading the resulting guideline recommendations.In 2004, the NHMRC commissioned a review of existing frameworks for assessing evidence internationally . This inThe report identified nine possible systems for use in developing clinical practice guidelines. Of these, three were considered to be most useful for informing the development of an Australian guideline recommendation process. These frameworks were the Scottish Intercollegiate Guidelines Network (SIGN) system and considered judgement statement ; and theThese systems were discussed at a face-to-face meeting of the working party with respect to their advantages and disadvantages and compatibility with the existing advice in the NHMRC 'Guidelines for Guideline Development' handbooks. A consensus was reached about how these frameworks could be adapted in the new process. From the three systems, we combined elements to achieve our objectives, which were: to have a system that matched and complemented the current NHMRC evidence dimensions and documents as closely as possible; simplicity and clarity of approach; and to provide transparent method/s of formulating and documenting judgments to give a graded set of recommendations. The working party drafted a new framework for grading recommendations and this was refined by extensive email consultation and iteration within the group.The resulting draft framework was piloted by GAR consultants working with guideline developers between 2005 and 2009. There were five main methods to gather feedback:• Known experts in the international guideline field were approached by NHMRC directly for comment on the draft system - this was a formal request and responses were semi-structured in that the experts were free to review in their own style,• Key evidence-based assessment organisations in Australian and New Zealand were invited to register feedback on the website where the system was posted,• All guideline development groups working within the NHMRC endorsement framework during this period used FORM under the guidance of the GARs - they were all invited to offer feedback during and after the process.• The draft process was presented at key conferences and interactive workshops ,• The website was open for the 5 years (passive seeking) and included a structured feedback form.Following this initial period of consultation (up until 2007) the FORM's framework was further refined, taking account of the feedback received, and the public consultation period was extended to June 2009. During the development, trialing and refinement period from 2004 to 2009, the international guideline community continued to debate and evolve other systems of guideline production - these developments were monitored and helped to inform the Australian process. The revised version of FORM was subsequently endorsed by the Council of the NHMRC.level of evidence to each study that reflects the risk of bias in its design, is only one small part of assessing evidence for a guideline recommendation. FORM provides a framework for assessing all the studies relevant for a recommendation against five criteria: the evidence base ; the consistency of findings between studies; the clinical impact suggested by the evidence base; the generalisability of the results to the population for whom the guideline is intended; and the applicability of the results to the Australian  health care setting. Under FORM, these five key components are individually assessed for each clinical question giving a picture of both the internal and external validity of the evidence base under consideration.The new FORM framework was loosely based on the SIGN considered judgement form . It provhttp://www.nhmrc.gov.au/guidelines/developers.htm)The evidence base is assessed in terms of the quantity and quality of the studies identified by a systematic literature review for the clinical question concerned ('included studies'). Study quality relates to an assessment of the risk of bias inherent in the conduct, design and reporting of results in the included studies. The guideline developers are free to choose the most relevant process or tool to assess risk of bias. To ensure that consideration is given to the full range of study designs required to assess the breadth of clinical questions in a guideline, the GAR consultants also developed levels of evidence to address different clinical questions . This has been comprehensively addressed by Merlin et al  . This allows users to assess whether the results are likely to be replicable or only likely to occur under certain conditions. Consistency may be assessed where appropriate as statistical heterogeneity (applying an I-squared statistic for example) or more likely will require the users to make a judgment about the overall direction of effects across multiple studies with reference to clinical heterogeneity. Possible sources of inconsistency (heterogeneity) in the results of studies may be differences in the study design, the quality of the studies (risk of bias), the population studied, and varying definitions of outcomes being assessed. Should results differ for certain subpopulations, this could then be reflected in the development of the recommendation.a substantial reduction in risk of cardiovascular events. In this case a qualifying recommendation could be made to differentiate the small group of people who may experience adverse events as a result of statin therapy.Clinical impact is a measure of the likely benefit that application of the guideline would have across the target population, and involves a clinical judgement. Factors that need to be taken into account when estimating clinical impact include: the relevance of the evidence to the clinical question; the statistical precision and size  of the effect reported in the evidence-base; the relevance of the effect to patients, compared to other management options; the duration of therapy required to achieve the effect; and the balance of risks and benefits to the patient group, including potential harm. A hypothetical example of incorporating both clinical importance and potential harm may be for the use of statins in the control of dyslipidaemia where there is a very large body of evidence with low risk of bias indicating Clinical impact is arguably the most subjective of the five evidence components rated in the evidence statement. However, we have found in assisting many guideline development groups to produce clinical practice guidelines using the FORM process that it is often clearer for clinicians than it is for methodological experts. Clinicians seem to grasp the net benefit concept quite easily, although often robust discussions occur before a consensus is reached regarding the rating of this component. A strength of FORM is that these discussions contribute to formulating appropriate recommendations, and the final conclusion can be documented so that users of the guideline can see how the developers arrived at the recommendation.The assessment of generalisability involves determining how precisely the available body of evidence answers the clinical question that was asked. Issues to be considered include: how well the participants and settings of the included studies match the patient population being targeted by the guideline; the clinical setting where the recommendation will be implemented; and other factors such as the stage of the disease (e.g. early versus advanced), the duration of illness and (for diagnostic accuracy questions) the prevalence of the disease in the study population as compared to the target population for the guideline.This component addresses whether the evidence base is relevant to the Australian health care system generally, or to more local settings for specific recommendations . Factors that may reduce the direct application of study findings to the Australian or more local settings include organisational factors  and cultural factors .The FORM matrix forms part of the overall process which is detailed in Additional file Evidence statements may be developed by outcome measures for each intervention and then the multiple evidence statements for a single question can be collapsed into a single recommendation. Guideline developers can produce a combined recommendation taking into account the balance of benefits and harms or separate recommendations for benefits and harms, if this is more appropriate. The FORM process allows considerable flexibility in developing the recommendation.Good Practice Points (GPP) to be included when developers feel it is important to provide non-evidence-based guidance.The overall grades for recommendations should indicate the strength of the body of evidence underpinning the recommendation. This assists users of the clinical practice guidelines to make appropriate and informed clinical judgments. Grade A or B recommendations are generally based on a body of evidence that can be trusted to guide clinical practice, whereas Grade C or D recommendations must be applied carefully to individual clinical and organisational circumstances and should be interpreted with caution . These show that the evidence base, consistency and impact were high for dermoscopy, but not so high for total body photography :• Training and utilisation of dermoscopy is recommended for clinicians routinely examining pigmented skin lesions: Grade A;• Consider the use of baseline total body photography as a tool for the early detection of melanoma in patients who are at high risk for developing primary melanoma: Grade C (p xxii ).Developers are also asked to consider how the guideline will be implemented at the time that the guideline recommendations are being formulated. The Evidence Statement Form requests developers to consider whether: the recommendation will result in changes in usual care; there are any resource implications associated with implementing the recommendation; the implementation of the recommendation will require changes in the way care is currently organised; and the guideline development group are aware of any barriers to the implementation of the recommendation. This information is used to inform the implementation plan for the Guideline.Over the trial and consultancy period for the FORM grading process, we obtained feedback from invited experts (see acknowledgements), from current guideline developers and from the public. These issues and suggestions were carefully considered at the face-to-face meeting of the GAR consultants in 2007 (see methods). Where appropriate, we amended the FORM methodology and/or supporting documents to incorporate the suggestions or address problems. This iterative process ensured that the development of FORM was responsive to the needs of its core user group - guideline developers - and was as clear and comprehensible as possible, even for developers with limited methodological expertise. It also allowed the FORM development process to keep abreast of the sometimes rapidly changing methodology underpinning guideline development internationally and incorporate changes into FORM as appropriate. As developers of FORM and also methodological experts assisting guideline developers we (the authors) have been able to field-test the FORM process and gain first-hand feedback and direct experience about problems and issues encountered. This has been invaluable in modifying FORM to be more effective and useful.The following issues were identified in the first consultation and addressed in the second iteration of FORM where appropriate:• deciding between grades - but this has become easier with time and familiarity• determining and extracting relevant information from synthesised sources (such as existing systematic reviews) which are incompletely reported• insufficient funding, human resources and/or time for the rigorous systematic literature reviews needed to underpin the evidence statements• need to accommodate subjectivity in the interpretation of the components and the final recommendation/sIn response to specific suggestions made in the first consultation period, we made the following modifications to the FORM supporting documentation:• revision of the notes, matrix and form to be more user friendly• the addition of 'explanatory notes' sections for developers to document reasons for particular decisions within the matrix• the addition of a 'dissenting opinions' and 'unresolved issues' sections to the Evidence Statement Form to keep decision making transparent and informed• a flowchart to assist in navigationFeedback from the second stage of consultation showed that the modifications were a major improvement and that guideline developers agreed that the FORM system of grading was an improvement on the previous system where recommendations were 'graded' according to the level of evidence from the NHMRC evidence hierarchy ,6. They As methodological experts assisting guideline developers, we have found the framework provides additional flexibility, especially when handling evidence with more than one outcome measure . Variable results/evidence statements for multiple outcomes can be captured by a single recommendation. Furthermore, the framework also allows a recommendation to be developed that balances the benefits and harms of an intervention (i.e. safety and effectiveness), but with enough flexibility to keep them separate if it is felt to be important. More than 20 NHMRC guidelines have now been completed using FORM.The formulation and inclusion of recommendations is one of the defining differences between clinical practice guidelines and other evidence syntheses such as systematic reviews. A recent review of the adequacy of guideline recommendations has highlighted that over half of the recommendations (52.7%) give no indication of the strength of that recommendation .The FORM process for formulation and grading of recommendations in clinical practice guidelines is logical, simple to use and intuitive. Its concurrent development with Australian levels of evidence  means thIn areas like public health where there may never be high-level evidence supporting the use of different interventions, practice recommendations developed using other grading systems would consistently rate a lower grade than is felt appropriate by experts in those fields. Examples of such areas include large-scale dietary questions, passive smoking or exposure to environmental chemicals. This does not occur using the FORM methodology. Using the NHMRC levels of evidence for aetiology questions as an alternative to the levels for intervention questions  allows tThe extensive pilot of FORM and subsequent uptake by both new and experienced guideline developers has shown that the framework is feasible and accepted. The component approach allows transparency in how recommendations are formulated, with users of the guidelines able to explicitly see the various contributions of factors such as quality of the evidence and clinical impact. A further strength is that implementation and resourcing issues are considered separately, which means that effective but potentially costly interventions are not penalised with a downgraded recommendation as the developers of this system felt that users' willingness to pay will vary according to the context of use. Arguably the greater ability to differentiate strength of recommendation (four levels) in FORM offers more precision for developers.The UK National Institute for Health and Clinical Excellence (NICE) has decided to discontinue summary grades for recommendations, on the grounds that their previous grading system was being misinterpreted. They have stated that they are not sure that the GRADE system's approach to summary labels overcomes this . We are Recommendation formulation and grading can be particularly challenging when the evidence is scant and/or poor, or conflicting. NICE has outlined some strategies to address these challenges, including using consensus when no evidence is found for a particular clinical question and highlighting gaps in the evidence where evidence is scant or poor. NICE remCurrent evidence frameworks are grappling with how to integrate other forms of evidence needed to answer qualitative questions such as optimal quality of life, and we anticipate that FORM will need to be periodically reassessed in the light of international debate about levels of evidence and grading recommendations.The purpose of clinical practice guidelines is to change or guide health professionals' behaviour and to improve quality of care. Therefore, the ultimate test of guidelines and the processes used to develop and implement guidelines will be improved health outcomes and improved systems. One way of facilitating this is by developing recommendations that are transparently produced through a process that is user-friendly, weighs up multiple concepts when formulating a course of action , and provides clear advice on the confidence or uncertainty associated with the recommended course of action.FORM provides a contemporary and internationally relevant structure within which clinical guideline developers can consider current literature related to specific clinical questions. It has been developed through a unique partnership of government, academic, private consultancy and clinical personnel with considerable experience in evidence-based practice and development of clinical practice guidelines. Our work with over 20 guideline developers during the piloting of the FORM process has demonstrated it to be a logical, simple to use and intuitive system for formulating and grading recommendations in clinical practice guidelines.The authors declare that they have no competing interest. Meeting attendance fees were paid to the authors (or their institutions) by the NHMRC - a not-for-profit research organisation funded by the Australian Government.All authors were involved in the process of drafting and revising the grading process as described and all authors were involved in the preparation and final approval of the current manuscript.History of NHMRC Guidelines Assessment Register (GAR) and members of the Levels and Grades Working Partypost hoc in guideline development could be pre-empted.In 2002, the NHMRC convened a register of methodological experts (Guidelines Assessment Register [GAR]) to assist external guideline developers in Australia through the process of identifying and synthesising evidence for guidelines in a way that complied with NHMRC specified requirements and would assist them in gaining NHMRC endorsement for their work. The main role of the GAR consultants was to oversee the methodological processes in external development of guidelines, particularly reviewing and classifying the quality of the evidence, and how these classifications correlated to the resultant recommendations. The expected outcome of the involvement of the GAR consultants was that consistently high quality guidelines would be submitted to HAC for approval, and that problems identified Kristina Coleman, Sarah Norris, Adele Weston Karen Grimmer-Somers, Susan Hillier Tracy Merlin Philippa Middleton, Rebecca Tooher Janet Salisbury (Biotext Pty Ltd)The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/11/23/prepub(landscape Document)NHMRC Evidence Statement . This file includes the FORM documents discussed - the matrix and trigger questions for guideline developers to complete the Evidence Statement.Click here for file


Jill Luoto and colleagues apply different frameworks to the same body of evidence for three advocated global health interventions and compare the ratings and policy recommendations resulting from each.Please see later in the article for the Editors' Summary Evidence-based decision-making is critical to informing policy in global health interventions and programs.Existing frameworks for evaluating evidence that were developed or recommended for community or public health decision-making vary in their criteria and application.We compared how different community or public health evidence frameworks assessed the same body of evidence for three advocated global health interventions and find there can be substantial differences in the rating of evidence, which could contribute to differences in policy recommendations.All current frameworks emphasize effectiveness, and have shortcomings on other important factors into policy decision-making such as costs, implementation issues, context, and sustainability.As global health policymakers move towards evidence-based approaches, we find a gap between what is currently available and the needs for an evidence framework appropriate for application to a global health setting in a low- and middle-income country context. More work is needed to either adapt one or more existing frameworks, or to develop an entirely new framework to meet the needs of policymakers and others responsible for implementing global health interventions.A major movement in global health and development in the past 10 years has been the enthusiastic adoption by many of randomized controlled trials (RCTs) from the field of medicine to represent the most rigorous method to evaluate a program's causal impact We define a global health evidence framework as one which uses multiple domains to arrive at a summary judgment of the evidence for community or population health interventions or programs, which could be applied to the kinds of interventions or programs that are commonly being considered in low- and middle-income countries. This includes frameworks explicitly developed for global health interventions, frameworks that were presented with a global health intervention as an illustrative example of its application, and general community or population health frameworks that could be applied to global health interventions. Details of our search methodology are summarized in We next identified a diverse set of global health interventions as potential candidates with which to apply these existing frameworks by considering the major causes of morbidity and mortality in developing countries or the major diseases of focus among international global health financing bodies. We developed a draft set of key dimensions for classifying global health interventions in order to map out these potential exemplars to select a diverse set of interventions along these dimensions . We were advised on this project by a multidisciplinary panel of experts (listed in Acknowledgments) composed of global health experts in academia, donor agencies, policymakers, and practitioners who provided input on the dimensions and on their preferred exemplars. From this exercise, we selected three interventions as exemplars for assessing the frameworks that represented a diversity of interventions: household water chlorination, prevention of mother-to-child transmission of HIV (PMTCT), and lay or community health workers to reduce childhood morbidity and mortality. For each of the three chosen global health exemplar interventions we located published systematic reviews of their effectiveness by conducting a Medline search. For each of these reviews, we retrieved the original research studies cited and used both the original studies and the systematic reviews as sources of evidence when applying the frameworks. As is customary and recommended in most evidence-based medicine processes, we used two members of the research team to independently apply the six frameworks to this evidence base for each of the three exemplar interventions. Disagreements were settled by a group consensus process. The results of the applications were compared both quantitatively  and qualitatively. For studies of household water chlorination, we consider the primary clinical outcome of (self-reported) diarrheal incidence over measured water quality due to its clinical importance. The evidence frameworks generally conclude that the evidence for diarrheal outcomes is weak or moderate. Only the U.S. Community Preventive Services Task Force (USCPSTF) framework assigns household water chlorination its highest grade (“strong”). All of the remaining frameworks assign the evidence grades that are lower than their highest possible rating, with the evidence classifications ranging from the highest categorization of “strong” by the USCPSTF framework, to the next-to-lowest grade of “C – satisfactory” within the Australian NHMRC framework.For PMTCT studies, all of the frameworks assign their highest possible grade to the body of evidence with the exception of the framework by Tang and colleagues, which assigns a “Grade 2B, Level 1 Possible.” However, this grade is the result of our strict interpretation of the rule that only interventions with a relative risk (RR) of greater than two qualify as “strong.” If there is some flexibility with this strict cutoff, the rating would change to the highest grade of “Grade 1 level 1 strong.”For interventions involving community or lay health workers, we chose the outcome “reduce morbidity in children under 5 years old compared to usual care” as it seemed both to be an outcome very important to communities and to have enough studies to make a meta-analysis meaningful. With this intervention the various frameworks again generally rate the evidence as being of low or moderate quality with the exception of USCPSTF, which assigns the highest grade of “strong.” HASTE, on the other hand, would rate this same body of evidence as grade three “insufficient,” and GRADE also assigns it a “low quality of evidence.”Overall, We find that assessing the same body of evidence using existing public health frameworks yields somewhat to markedly different conclusions depending on the framework applied. Thus, in practice, if the current push towards evidence-based global health policy making includes adoption of an evidence framework (one key method for ensuring an “evidence-based” approach), the choice of framework for evaluating the evidence could potentially lead to different policy decisions, a potentially unintended consequence of the choice of framework. For example, had policymakers used the USCPSTF framework, they would have reached the conclusion that all three interventions were equally strong and supported. Conversely, had policymakers used the GRADE or HASTE framework, they would have concluded that the three interventions varied from “insufficient” or “low quality” to “strong” and “high quality.” Had six different policymakers been considering the same evidence on household water chlorination to reduce diarrheal outcomes and each used a different framework, they could have reached differing conclusions about the strength of support that ranged from grade “C” to grade “B” to “possible” to “moderate quality” to “strong.” Actual policy decisions will include other factors, such as feasibility, financial resources, and health systems capacity, but the current push for “evidence-based” decision-making makes the adoption of an evidence framework likely, and, therefore, the rating of evidence would likely be one important factor in decision-making.Why should these frameworks differ in their conclusions? One possible reason is that they differ in whether and to what degree they deal with the following domains: (1) how strict or explicit the rules are for classifying the strength of evidence; (2) the magnitude of potential benefits versus harms; (3) what role, if any, context is taken into consideration in evaluating the evidence; (4) how much is reported about the details of implementation; (5) whether the ease of implementing the intervention or program is taken into consideration; (6) total costs for the program or intervention; and (7) sustainability of the program or intervention, both cost-wise and programmatically. The USCPSTF, Australian NHMRC, the UK National Health Service (NHS) Health Development Agency, and GRADE have stricter rules for classifying the strength of evidence than the HASTE framework and the framework from Tang and colleagues, which allow for more individual interpretation. The Tang and colleagues framework, GRADE, the USCPSTF, and Australian NHMRC all make explicit a consideration of the magnitude of the benefits, while HASTE and the NHS Health Development Agency do not. Only the Australian NHMRC framework explicitly considers context, and only the HASTE framework includes a detailed assessment of implementation data, although context could be considered part of “widely demonstrated” in the Tang and colleagues framework and could be considered in the “corroboration” criterion in the NHS Health Development Agency Framework. The USCPSTF considers barriers to implementation in their evidence review but not as part of the overall assessment of the body of evidence. Costs and sustainability are not included routinely in any of the frameworks, although GRADE does have guidance on including cost as an outcome and on incorporating cost into the strength of the evidence An additional cause for variability in the conclusions among different frameworks when assessing the same global health evidence may be variability in applying the individual frameworks themselves. When individual team members initially applied the frameworks to the evidence, they sometimes reached different conclusions, largely due to the need for individual interpretation of the criteria used in the frameworks. These differences were resolved in a consensus process, as is standard practice in most evidence-based medicine processes. Nevertheless, this situation raises the possibility of potentially poor inter-rater reliability within frameworks, which has also been observed with frameworks used to assess the risk of bias or strength of evidence for conventional medical therapies Although a similar exercise could have been undertaken with more than three exemplars, our initial choice of three proved sufficient to identify variability both within and across frameworks in how evidence is assessed. Moreover, additional exemplars will not change the identification of context, costs, and implementation data as important missing domains of these frameworks. We also recognize that our results may be sensitive to the composition of participants on our technical expert panel who provided input at each stage of this process, and further evaluation of these results with a wider group of stakeholders is warranted. However, these stakeholders' identification of a need for more data about implementation is consistent with the increasing recognition of the importance of implementation reporting in other health-related fields As global health policymakers move towards evidence-based approaches, our study reveals a gap between what is currently available and the needs for an evidence framework appropriate for application to a global health setting in a developing country context. More work is needed to either adapt one or more existing frameworks, or to develop an entirely new framework to meet stakeholders' needs. For example, Lewin and colleagues on the Task Force on Developing Health Systems Guidance of the World Health Organization recently described the beginnings of an adaptation of the GRADE framework Box S1Search methodology.(DOC)Click here for additional data file.Box S2Full list of potential exemplars.(DOC)Click here for additional data file.Table S1Diversity of exemplar interventions across key dimensions.(DOC)Click here for additional data file.Table S2Evidence base for three global health exemplars.(DOC)Click here for additional data file.


A realist synthesis attempts to provide policy makers with a transferable theory that suggests a certain program is more or less likely to work in certain respects, for particular subjects, in specific kinds of situations. Yet realist reviews can require considerable and sustained investment over time, which does not always suit the time-sensitive demands of many policy decisions. ‘Rapid Realist Review’ methodology (RRR) has been developed as a tool for applying a realist approach to a knowledge synthesis process in order to produce a product that is useful to policy makers in responding to time-sensitive and/or emerging issues, while preserving the core elements of realist methodology.Using examples from completed RRRs, we describe key features of the RRR methodology, the resources required, and the strengths and limitations of the process. All aspects of an RRR are guided by both a local reference group, and a group of content experts. Involvement of knowledge users and external experts ensures both the usability of the review products, as well as their links to current practice.i.e., the amount of literature available that identifies context, mechanisms, and outcomes for a given topic); understanding tensions/paradoxes in the evidence base; and, reinforcing or refuting beliefs and decisions taken.RRRs have proven useful in providing evidence for and making explicit what is known on a given topic, as well as articulating where knowledge gaps may exist. From the RRRs completed to date, findings broadly adhere to four (often overlapping) classifications: guiding rules for policy-making; knowledge quantification  published/research and practice-based evidence available, RRR provides a practical, outcomes-focused knowledge synthesis method. Efforts to improve the connections between research evidence and policy decisions have resulted in a number of sophisticated synthesis tools . Pawson Pawson notes that the underlying principles of realist approaches are the links between interventions (I), contexts (C), mechanisms (M), and outcomes (O) . In contIn developing these program theories, realist reviews assume a broad inclusion of evidence, both quantitative and qualitative, which is used to develop an understanding of ‘what works, for whom, in what contexts, to what extent, and most importantly how and why ,3?’ Theret al. close enough to observed data to be incorporated in proportions that permit empirical testing’ [The RRR method also differs from a more traditional realist review in the role of theory, and specifically the presentation of theory in the final synthesis product. A ‘traditional’ realist review explicitly grounds the review process in theory; it produces a narrative summary of the interactions between context, mechanisms, and outcome configurations . The revtesting’ ) to accoReviewing the similarities and differences between the RRR and more traditional realist review methods highlights several strengths of the RRR approach. Similar to realist reviews more generally, the RRR method is time efficient, and may in fact be more amenable to time limitations than other types of realist reviews (due in part to the increased involvement of the expert panel). RRR is grounded in local context, with explicit, extensive, iterative engagement with a local reference group comprising representatives of potential ‘knowledge users’. The reference group provides feedback on the project at critical times, including when developing project scope and research questions, identifying key literature, and interpreting results (‘Do these findings and recommendations make sense for our context? Will we be able to use these results?’). As such, the RRR method is responsive to local policy needs and is therefore adherent to the goals of a realist approach to knowledge synthesis . As a reSimilar to traditional realist reviews, the RRR methodology also works well in situations where there is a small but emerging body of evidence on which to base policy decisions. The use of the expert panel allows for incorporation of knowledge-in-the-making into policy recommendations, as well as the practice knowledge of those who have been engaged in application of research findings to a policy context. For example, in the review of methods to increase organizational capacity to address health literacy issues within healthcare organizations, the actual published literature specifically relating to the review topic was limited. As a result, the expert panel helped frame the findings significantly to be consistent with what they had experienced in practice. This enhanced the value of the final report for the knowledge users.Finally, the RRR methodology allows for the ability to scale the project based on the time and resources available. For example, the evidence informed public health policy and practice review consisted primarily of a literature review and synthesis with no reference group. On the other end of the spectrum of review complexity, the large system transformation review (KAST) included a much broader and more extensive consultation with content experts. While this was not critical to the success or usefulness of the final report, it added to the nuanced contextualization provided in the final report. Key to the KAST project, the report commissioners explicitly requested recommendations for government action that could enhance the likelihood of success for large system transformation projects. Having such a request made it easier to elicit useful feedback from the content experts that were surveyed for the project.While the RRR method has several strengths and advantages, as noted above, there are also several limitations to the RRR approach.The RRR method explicitly uses an expedited search process. This is an advantage with respect to the faster turnaround time for the finished review, but it may also result in certain resources/references being missed, potentially introducing a source of bias. However, the effects of this potential limitation are buffered by the engagement of the expert panel. Experts validate the findings to ensure that critical elements are not missed, and that nuances from emerging practice are included.The short time frame in which RRRs are typically conducted can make it difficult to fully theorize the mechanisms that are identified, as well as the interactions between context, mechanisms and outcomes, as is at the core of a full realist review. This may consequently limit the generalizability and potency of findings. Therefore, policy makers interested in sharing results more broadly may need to consider the inclusion of other examples of when the same (or similar) mechanism is in operation, though this might be an intervention that has not been labelled as being the ‘same’ as the one under consideration. However, this is unlikely to be an impediment to the actual utilization of the review findings in practice. Working with the local reference group throughout the project ensures that the project scope, research questions, and format of project outcomes will meet the needs of those who will use the information.Given the time-intensive nature of the RRR review process, as well as the critical role of the expert panel, it is unlikely that government agencies or departments will have the in-house capacity to complete these reviews. Adding the RRR method to a policy maker’s toolkit will routinely require external input from expert teams. However, this may actually be an advantage; it allows expert methodologists to be engaged at specific times in order to answer specific questions, thus targeting resources strategically. One critical advantage is the experiential learning that results, deepening the systems thinking skills required for effective change. The relative strengths are brought to bear in ways that maximize the allocation of resources, time and talents, making such a process efficient as well as effective.Forming the membership of the reference group and ensuring sustained involvement are challenges for all reviews, particularly in rapidly changing political environments where membership of the reference group may change during the course of a given project. However, the role of the reference group in the RRR methodology allows key agency or government staff to be engaged in the process without requiring excessive time commitments. Similarly, the engagement of expert panels does rely on panellist availability and willingness to contribute their time and ideas. Providing honoraria for expert panellists, and/or providing opportunities for them to be involved in dissemination of the work through conference presentations, papers, or other avenues, can make it easier to engage the right experts at the right time.e.g., a province) rather than adding to the knowledge-base of middle-range theories. While they need to understand C-M-O interactions to be able to identify potentially effective interventions, their focus tends to be more in intervention (I) – outcome (O) links. We see a growing interest in the how and why interventions work or don’t work for various audiences, but understanding the program theories that underlie success or failure of interventions tends to be lower on the list of priorities than producing results. As a result, our reports have had to highlight I-O links, and use the program theories we developed to help explain why those interventions might work well in a given context for specific populations.A final potential weakness can result from the requirement that all members of the review team be well versed in realist review methodology, while at the same time commissioners of realist reviews (such as policy makers) do not share the same deep understanding of the methodology and its value for policy. In our experience, policy makers tend to be more concerned about what they are able to accomplish within one specific context  published/research and practice-based evidence available, RRR provides a practical, outcomes focused knowledge synthesis method.The Rapid Realist Review methodology described in this paper is designed to bridge multisectoral barriers to knowledge use. Although the authors have academic interest in this methodology, they also engage in the methodology as consultants on a range of projects. Some of these projects are funded by grants managed by universities, some through contracts to the InSource Research Group. InSource was created to provide a vehicle for responding to policy maker needs for knowledge synthesis in a more timely way than is possible through the normal grant funding process.In the interests of transparency and to address conflict of interest concerns, it is very possible that publication of this paper could enhance InSource’s reputation and could result in future contracts for the company. It must also be noted that two of the four authors (JB and AB) are InSource directors, and the other two (JS and CW) have worked, or in the future may work, under contract to InSource.In sum, the authors may receive financial gain in the future from the publication of this manuscript. In our view, the conflict between academic and business interest in this area is unavoidable, and bridging the gap between these interests is vital to supporting the research to policy and practice process. The evidence is clear that knowledge uptake is poor without effective structures to support the process.JS made substantial contributions to: the acquisition, analysis and synthesis of data presented in Table


Anterior cruciate ligament (ACL) surgical reconstruction is performed with the use of an autogenic, allogenic or synthetic graft. The document issued by the Italian National Guidelines System  at the National Institute of Health aims to guide orthopaedic surgeons in selecting the optimal graft for ACL reconstruction using an evidence-based approach.A monodisciplinary panel was formed to define a restricted number of clinical questions, develop specific search strategies and critically appraise the literature using the grading of recommendations assessment, development, and evaluation (GRADE) method. The final draft was shared by the panel and then sent to four external referees to assess its readability and clarity, its clinical relevance and the feasibility of recommendations.Autograft shows moderate superiority compared with allograft, in relation to the relevant outcomes and the quality of selected evidence, after an appropriate risk–benefit assessment. Allograft shows higher failure rate and higher risk of infection. The panel recommends use of autografts; patellar tendon should be the first choice, due to its higher stability, while use of hamstring is indicated for subjects for whom knee pain can represent a particular problem .Autograft shows better performance compared with allograft and no significant heterogeneity in relation to relevant outcomes. The GRADE method allowed collation of all the information needed to draw up the recommendations, and to highlight the core points for discussion. The anterior cruciate ligament (ACL) plays a crucial role in knee stability, as it contrasts the combined movement of the tibia against the femur, anterior translation and internal rotation. ACL injuries can affect one or both strands  and, on the basis of individual characteristics, can affect ligament function and knee stability, raising the need for surgical reconstruction.Defining the prevalence of this condition is not easy, as lesions are often asymptomatic; a study carried out on a large sample of students from a US college showed that the possibility of ACL injury may be over 3% in 4 years of physical activity, with a higher risk in female population .Surgical reconstruction for primary isolated ACL lesions is performed using autograft (mostly patellar or hamstring tendons) or allograft , while use of synthetic ligaments has recently attracted interest after being abandoned in the past due to a high failure rate.The choice of technique is based on clinical and biomechanical factors, or on tradition and surgeon experience, or for reasons of context, as shown by various investigations carried out among surgeons from different countries , 3.The heterogeneity in the surgical management of this condition raised a need for clarity on the effectiveness of the different types of grafts, through a systematic, critical appraisal of the literature. The National Guidelines System  of the Ministry of Health at the National Institute of Health  therefore engaged in the elaboration of a document aimed at guiding orthopaedic surgeons in the choice of best practice for primary anterior cruciate ligament reconstruction. The document does not evaluate the different systems for graft fixation, nor the different techniques for preoperative preparation or postoperative rehabilitation.The quick review document is an instrument designed to handle very specific clinical issues through a faster process than the one used to draw up guidelines. The panel created to carry out the activities needed to elaborate the quick review is monodisciplinary, in contrast to the one created for guidelines, and aims to answer a small number of clinical questions defined as crucial by the specialists, and to reduce all heterogeneous and sometimes inappropriate clinical practices.The panel of experts who collaborated to draw up this document included 14 orthopaedic surgeons, 2 physiatrists, 1 physiotherapist and 2 epidemiologists familiar with evidence-based medicine and the methodology for guidelines development. The working group included representatives from all the main national scientific societies of reference , supported by a balanced group of independent experts.All participants signed a declaration of absence of conflict of interests and of acceptance of the methodology as explained during the first meeting.http://www.snlg-iss.it.The panel met twice (4 July 2008 and 6 February 2009), and all materials produced during the process for the elaboration of the document are available at: The objectives of the document, the clinical questions on the effectiveness and safety of the types of graft to be used for anterior cruciate ligament reconstruction, the inclusion and exclusion criteria for studies and the timeframe to be considered in the bibliographic search were defined by the panel during the first meeting at the Italian National Institute of Health (ISS).Specific search strategies were defined in accordance with each established clinical question.The following databases were searched to gather evidence: PubMed, Embase and Cochrane Library, including randomized clinical trials (RCTs) and systematic reviews (SRs) dated 2000–2008.Observational studies dated 2000–2008 from the PubMed database were included for questions concerning safety. Figure Qualitative assessment of systematic reviews, RCTs and observational studies was carried out using a structured method , 5.The selection of studies, their methodological evaluation and the extraction of data were carried out by specifically trained personnel. The evidence gathered from each study was summarized in tables, each specific to a single question and type of study. The summary tables adopted in this document are those defined by the National Institute for Clinical Excellence (NICE), updated in 2007.The recommendations were drawn up for each clinical question without adopting any specific grading system, that is, without using any structured system to grade the strength of recommendations. The intensity and certainty supporting all recommendations are reported in narrative form, without any symbol, graded score or hierarchy. Each recommendation is introduced by a description of the discussion that led to its definition, to make clear the level of agreement of the working group.The panel adopted the GRADE system to carry out the critical appraisal of literature and to draw up the recommendations –11.The critical appraisal of the literature was carried out for each outcome considered relevant by the panel, following the principles of this method. The quality of evidence, finally, was related to the assessment of all risks connected to adopting that specific procedure, thus reaching the definition of the recommendation.http://www.snlg-iss.it.The final draft was shared by the panel in the second and last meeting, and then sent to four external referees, asking them to assess its readability and clarity, its clinical relevance and the feasibility of recommendations. The referee group included renowned orthopaedic surgeons with an interest in knee surgery and with active scientific production in the field. The full text of the document , including all suggestions from the referees, is available on the SNLG website at: The panel agreed on two clinical questions, one related to the effectiveness and the other to the safety of arthroscopic ACL reconstruction carried out using autograft Table , allograTable All selected reviews included mostly randomized or quasi-randomized prospective studies based on follow-ups of 2 or more years, and aimed at assessing the effectiveness of autograft using objectively measured or subjectively assessed mechanical or functional outcomes .The assessment of laxity and stability defined with various measures , the frequency with which patients return to pre-injury activity and the loss of flexibility support the hypothesis that PT in several cases performs better than HS –19, whilThe effect rates reported are often close to statistical significance (even if unable to prove superiority of one specific technique), confirmed by the results of the included RCTs. These –25 are nPromising experiences using four-strand hamstring tendon have also been carried out , or usinBoth seem to substantially improve HS graft performance in terms of stability, but require further investigation.Evidence in relation to safety of autograft comes instead from uncontrolled observational studies and refers to infections, and in one case  to mechaClusters of joint infections are reported among subjects who underwent ACL reconstruction (1.6–2.6%), with slight predominance with HS use (5.7%) and an increase of risk, probably due to former ACL reconstruction [relative risk (RR) = 5.1] or knee surgery (RR = 1.90) and to the use of some fixation systems for femur (RR = 4.5 for Endobutton) or tibial  fractures , 30. TheAlmazàn et al., finally , show thPT graft appears therefore to be fairly superior to HS graft, in terms of stability, return to pre-injury activity and flexural strength, while use of HS can be reasonably restricted to specific situations, due to its effectiveness in reducing pain and loss of extension. Evidence on safety is scarce and fragmentary, and no inferences can be made apart from a few suggestions on infective complications.The panel therefore decided to recommend use of PT due to its proven higher stability and to identify at the same time a possible subgroup of subjects for whom knee pain can represent a particular problem , or for whom reducing length and intensity of pain as much as possible could be important, and define for this subgroup a specific indication for use of HS.Table The study of Prodromos et al.  analyzesP < 0.001).The global stability rate indicates higher efficacy of autograft, with 72% normal stability  and 5.3% abnormal stability . The differences observed between the two types of grafts were statistically significant in both cases (P < 0.001) has been observed, and of non-patellar tissue (64%) versus patellar tissue .Moreover, higher efficacy of non-irradiated tissues (63%) versus irradiated tissues  = 5.03, 95% confidence interval (CI) 1.38–18.33] and of hop test results <90% versus healthy side .The panel, in accordance with the GRADE methodology, was invited to vote on the relevance of the outcomes considered in the selected studies. Table Graft rupture, re-operation rate, return to pre-injury activity and IKDC score were considered critical outcomes, and graft rupture in particular benefited from evidence much higher in quality than that gathered for other outcomes, supporting higher efficacy of autograft versus allograft. The other outcomes defined by the panel seemingly showed higher efficacy of autograft, even if the values did not reach statistical significance.Evidence in relation to safety, on the other hand, relies on two studies and essentially concerns infective complications.Centeno et al.’s study  was not Moderate superiority of autograft as ACL reconstruction technique is to be pointed out, on the basis of allograft performance (versus autograft), in relation to the outcomes considered relevant and the quality of selected evidence, subsequent to the risk–benefit assessment. Use of allograft, in fact, shows higher failure rate and higher risk of infective complications with aseptic tissue.Table The full text of a systematic review identified by the literature search  resultedTwo more studies (RCTs) gathered by the literature search compared patellar tendon autograft and synthetic graft. The full texts of both of these articles underwent critical appraisal , 40, andMuren et al.’s study investigated use of a polypropylene device, the Ligament Augmentation Device (LAD), stitched to the autograft, while Nau et al. investigated the use of the Ligament Advanced Reinforcement System (LARS), a device produced in France, fixed with titanium screws. This trial is included in Pichon Riviere’s systematic review.Muren et al. follow the 40 randomized patients for 7 years (3 years for arthrometric assessment with KT 1000), reporting no significant differences between the two groups. The results of this study refer to patients with acute ACL injuries, with time of injury less than 3 weeks prior to enrolment being an inclusion criterion.P < 0.05) and IKDC assessment. Patients expressed a slight preference [assessed using the Knee injury and Osteoarthritis Outcome Score (KOOS) score] for LARS treatment at 6 and 12 months, but not at 24 months.Nau et al., on the other hand, included patients who suffered ACL injury no less than 6 months before enrolment, and showed a substantial equivalence after arthrometric tests . The main feature of this type of document, apart from some methodological issues, is the specificity of topics, being mainly monospecialistic.The document on graft choice in primary ACL surgery is the first SNLG experience of a quick review . The analysis of sensitivity, based on the inclusion/exclusion of Gorschewsky et al.’s study, did not substantially modify the results, causing only a loss of power that did non allow the results to reach statistical relevance. Results, however, showed a certain superiority of autograft.The panel agreed that the sterilization procedure used in Gorschewsky et al.’s study is to be considered responsible for the lower efficacy of the allograft, but that it is not currently adopted. Therefore, the recommendation defined by the working group did not take into consideration the effects of this procedure on the effectiveness of the intervention.Global agreement was reached for all questions and recommendations, irrespective of divergences arising in interpretation and assessment of some studies. Evidence, in fact, showed overall homogeneity, and the clinical opinions from each member converged without affecting the richness of information. The results of the review are therefore coherent with current trends in clinical practice, although they do supply robust scientific data to support the choice of graft in ACL primary surgery.Available evidence allows recommendation of use of autograft over allograft in arthroscopic ACL reconstruction and to recognize, for autograft, better performance of PT over HS. It is therefore appropriate to select one of these two main choices (PT and HS), assessing the indication on a case-by-case basis. It is also appropriate to consider allograft and artificial ligaments only in very selected cases, discouraging widespread use, given the potential risks and paucity of well-performed, well-designed clinical studies. The indications for further research are also clear. Consolidation of the experience in use of two- and four-strand HS and in using specific techniques to contain laxity is suggested. Further investigations are also strongly suggested on use of synthetic grafts in studies comparing their effectiveness versus autograft. It is valuable to recall that stepwise introduction of new orthopaedic technologies should include preclinical testing, randomized clinical trials, multicenter studies and post-market surveillance, to provide surgeons with adequate information to make informed decisions regarding use of new technologies in their practice, including ACL reconstruction with synthetic ligaments .Finally, this experience confirms the feasibility of practice guidelines to drive an evidence-based approach in orthopaedic surgery. In this particular case, representatives from the scientific societies with an interest in knee surgery  participated in collecting, analyzing and discussing the available data to develop evidence-based guidelines using a standardized and reliable methodology. The practice of evidence-based medicine can be conceptualized as the integration of the best available research evidence, clinical circumstances and patients’ values and preferences. Evidence-based practice guidelines allow practitioners to develop treatments for a specific patient, on the bases of not only his/her experience and personal knowledge, but also the most up-to-date scientific evidence, reviewed and evaluated using a structured, detailed and explicit approach. Through the process of guideline development, clinical and methodological experts evaluate and condense the universe of information available on a clinical issue into a useful set of parameters that the physician can complete with his/her own experience and knowledge in managing a patient. Guidelines are not a substitute for continuing study, rather they represent a tool for the practitioner to provide the best care for his/her patients .


Consensus guidelines for radiological diagnosis of hepatocellular carcinoma (HCC) have been drafted by several large international working groups. This article reviews the similarities and differences between the most recent guidelines proposed by the American Association for Study of Liver Diseases and the Asian Pacific Association for the Study of the Liver. Current evidence for the various imaging modalities for diagnosis of HCC and their relevance to the consensus guidelines are reviewed. Consensus guidelines have been drafted by several large international working groups on different occasions in an attempt to standardise the surveillance, diagnosis, and management of HCC. Of the major working groups, the European Association for the Study of the Liver was the first to establish consensus guidelines on the clinical management of HCC following the Barcelona European Association for the Study of the Liver (EASL) Conference in 2000 . The AmeThe rationale for a set of guidelines on management of the growing problem of HCC is several fold. Firstly, it aims to maximise healthcare resources when targeting large populations at risk, based on current evidence-based practice. Secondly, it allows for a standardised method of diagnosis in the era of computed tomography (CT) and magnetic resonance imaging (MRI). Lastly, it provides clinicians with a guide to the treatment of HCC.Establishing universal guidelines for imaging diagnosis of HCC can be challenging, particularly in the lesions that do not display classical imaging features. Nevertheless, imaging diagnosis of HCC is important because it is noninvasive, given that the incidence of needle tract tumour seeding following biopsy of HCC is small but not negligible  , while tThe purpose of this paper is to review the similarities and differences between the more recent guidelines on radiological diagnosis of HCC as proposed by the APASL and the AASLD.The use of imaging in HCC diagnosis can be best divided into two main categories. The first is in the surveillance of patients at high-risk for developing HCC. The second is in the diagnosis of HCC based on an abnormal screening test.Prospective screening of patients at high-risk of developing HCC increases the proportion diagnosed with potentially curable disease. A screening strategy should focus on those patients with chronic HBV or HCV virus infection that has progressed to cirrhosis since more than 40% of these patients will develop HCC .As for the time interval between surveillance tests, both the AASLD and APASL recommend measurement of serum alpha-fetoprotein (AFP) levels combined with grey-scale ultrasound (US) of the liver for surveillance of HCC , 4 at 6-As with all diagnostic tests, the sensitivity profile of AFP is reduced when a higher threshold is applied in order to improve specificity. On its own, AFP is not sufficient as a screening test for HCC . Taking US screening is superior to alpha-fetoprotein assay for detection of HCC . CombineA feature common to the APASL and AASLD guidelines is that the recommendations for imaging diagnosis of HCC are to be interpreted in the context of patients at high-risk for HCC , 4. ThisThere is little disagreement between the consensus guidelines of the APASL and the AASLD on the definition of imaging features of classical HCC. The presence of arterial hypervascularity and washout are generally considered to be highly specific for the diagnosis of HCC, and shall henceforth be referred to as “classical imaging features” . In partArterial hypervascularity is defined as increased enhancement of the lesion in the hepatic arterial phase of imaging relative to the background liver. This is based on the fact that HCC receives predominant vascular supply via the hepatic artery. A precontrast and a dynamic postcontrast scan of the liver is necessary to demonstrate this on imaging. “Washout” of the lesion is based on the fact that HCC contains predominantly arterial blood and so, by the time portal venous and delayed images are acquired, the lesion is observed to be hypoattenuating on CT  to the surrounding liver at the portal venous or equilibrium phase. Washout can be explained in terms of tracer kinetic modeling of a lesion with high proportion of intravascular space  For demoThe presence of elevated AFP greater than 200 ng/mL is no longer required under the revised AASLD guidelines, as it is recognised that there are inherent false-positives (in cirrhotic patients) and false negatives , 25. DetDespite the abundant use of multidetector row technology, CT may underestimate the extent of disease in around 50% of cases . AlthougMost of the differences between the AASLD and APASL guidelines for the radiological diagnosis of HCC lie in the approach to lesions that do not demonstrate the classical imaging features of HCC. The AASLD essentially does not recognise use of nonvascular imaging criteria, and in the absence of the classical arterial hypervascularity and venous washout pattern of HCC, further evaluation is necessary. While this makes the AASLD guidelines more applicable to transplant guidelines (Milan and UCSF criteria), where diagnoses were based on vascular enhancement pattern of HCCs , 31, it Often, the lesions that do not conform to the classical imaging features are better differentiated and smaller than 2 cm in size. These “early” HCCs have been shown to contain not only fewer portal tracts but also fewer arterioles . This isConversely, for the larger lesions, even in the absence of the classical imaging features, size alone is a risk factor . In the In the revised AASLD guidelines, lesion size continues to predominate, though less so compared to the earlier edition. In the earlier AASLD guidelines, any lesion greater than 2 cm in size and demonstrates classical imaging features can be treated without biopsy. For lesions that were between 1 to 2 cm in size, two imaging modalities, rather than one, with classical features were needed to confirm the presence of HCC and avoid biopsy. This has been recently revised such that any lesion larger than 1 cm that demonstrate the classical pattern of HCC can be deemed as such and treated accordingly without biopsy. This is because as with the larger lesions, the approach of using a single imaging technique for lesions that are between 1 to 2 cm yields acceptable results –39. In the presence of atypical findings from a single imaging test (CT or MRI), the AASLD recommends a different imaging modality (CT or MRI) for further assessment. This has been validated by Khalili et al. in which single imaging scans were found to have similar specificity (91–99%) to two coincidental positive scans (91–100%) with much less resource utilization and higher sensitivity (74–89% versus 53–62%) . HoweverEven though the majority of cirrhotic nodules smaller than 1 cm are benign , Kim et  Instead of aggressively chasing the diagnosis through biopsy for lesions smaller than 1 cm , close interval followup in 3 months using the modality that best depicts the lesion is recommended. Here, the guidelines may be debated. It has been suggested that for among hypervascular nodules smaller than 1 cm, those smaller than 5 mm, are subcapsular in location, wedge shaped, or ill defined (more likely to represent vascular shunts) a 6-month followup is sufficient, but when the nodule is round, oval, intraparenchymal, or in a dominant mass (more suspcious for HCC), closer imaging followup at 3-monthly intervals should be performed . This maThe APASL guidelines approach the atypical lesions in different manners. Essentially, these focus on Kupffer cell density as a marker of benignity. It has been shown that Kupffer cell density decreases with dedifferentiation of the cirrhotic nodule , 46 and The APASL guidelines basically divides the atypical lesions into those that are hypervascular (and do not demonstrate washout) and those that are hypovascular . For hypervascular lesions that do not demonstrate washout, early HCCs can be reliably differentiated from focal nodular hyperplasia and arterioportal shunts based on differential uptake of Kupffer-specific contrast agents. On the parenchymal phase of imaging, HCCs should appear as unenhanced areas on CEUS and as T2*-hyperintense lesions on SPIO-enhanced MRI. However, a foreseeable limitation is in the characterisation of other hypervascular malignancies, such as neuroendocrine carcinoma metastases. The approach to the hypovascular lesion is a little more complex, while at the same time, the differential list for this includes a larger group of hepatic malignancies, including intrahepatic cholangiocarcinoma and metastases. Basically, if the lesion is initially shown to be hypovascular on CT and MRI, CEUS may be attempted to demonstrate enhancement in the hepatic arterial phase. If this is shown to be true, the lesion may be deemed HCC. Alternatively, if Kupffer-specific imaging demonstrates a relative lack of uptake, the lesion can be regarded as HCC. Again, the limitation of such an approach is that the other concomitant hypovascular lesions such as adenocarcinoma metastases are not definitely excluded.Although CT arterial portography and CT hepatic arteriography (CTPA and CTHA) are considered to be significantly more sensitive for demonstrating the early vascular changes in small HCCs , these aThe ensuing sections will briefly review various imaging modalities used in diagnosis and assessment of HCC; some of these are included in the current APASL guidelines, the rest are meant to inform the reader of recent advances in imaging of HCC that may potentially be integrated into future diagnostic imaging algorithms.Given that the APASL recommends the use of Kupffer-specific agents (Sonazoid and SPIO agents) for lesion characterisation, a more detailed discussion on the utility of these contrast agents needs to be made. However, in part because neither Sonazoid nor currently commercially available SPIO agents are approved by the United States Food and Drug Administration (FDA) for clinical use, these are not included under the diagnostic algorithm by the AASLD. CEUS on its own is an accepted imaging modality for HCC diagnosis under the APASL guidelines and this has been validated even for lesions smaller than 2 cm . Jang etInherently, the enhancement patterns of lesions on CEUS reflect tumour microvascular morphology, making it a valuable method for predicting the histological grade  while prAdding Kupffer-specific phase imaging to CEUS protocols may yield additional information that can be used to further assess histologic grades of tumour and enable better characterisation among dysplastic nodules, moderately-differentiated and poorly differentiated HCCs . As withSimilarly, use of SPIO has been shown in multiple studies to improve accuracy of MRI for detection of HCCs. However, detailed discussion of the SPIO agents will be avoided since these are currently out of production, except to say that experience with SPIO agents thus far had been promising and that it potentially improves imaging detection of HCCs , 53, 54.Although important for staging and treatment decision making, assessment of portal vein thrombosis for tumour involvement is currently not considered in both the APASL or AASLD guideline recommendations. Image guided percutaneous biopsy of suspected portal vein tumour thrombosis is feasible but invasive . It may Various noninvasive techniques have been investigated, and among them, CEUS appears to show fairly good success , superioFunctional imaging of HCC is fast becoming a reality and a brief mention of some of these techniques shall be made. Hepatocyte-specific gadolinium chelate agents are relatively new and are not currently included in the guideline recommendations. Gadoxetic acid  and gadopentetate dimeglumine  are two such contrast agents that have been shown to improve diagnosis of HCC, showing diagnostic performance similar to or better than SPIO , 68 and Hepatocyte-specific gadolinium chelate agents allow for multiphasic dynamic contrast-enhanced MR imaging to be combined with the hepatocyte-specific phase. These require delayed scanning of approximately 20 minutes in the case of Gd-EOB-DTPA and 60–120 mins in the case of Gd-BOPTA to provide maximal lesion to liver contrast . SpecifiGd-BOPTA-enhanced MRI with hepatocyte-specific phase imaging improves diagnosis over routine multiphasic CT or MRI , with quCombining Gd-EOB-DTPA-enhanced MRI and Sonazoid CEUS detected 73% of the nodules not detectable by multiphasic CT . It may DW MRI studies the random motion of water molecules and shows promise for detection and characterisation as well as posttreatment assessment of tumours . It impr18Fluorodeoxyglucose (FDG) PET is generally accepted to have low sensitivity (50–68%) for intrahepatic HCC [11C-acetate improves sensitivity for intrahepatic disease from 37–49% for 18-FDG and 11-C alone to 90% when combined [18F-choline [18F-thymidine [atic HCC –95 and iatic HCC . Dual trcombined . The rolcombined , with secombined . Newer t-choline  and18F-thymidine  have shoOur understanding of the pathophysiology of HCC has improved tremendously over the past decade. This has been paralleled by advancements in US, CT and MRI technology, along with development of various Kupffer- and hepatocyte-specific imaging contrast agents. As the treatment of HCC becomes more sophisticated, a greater need for highly accurate diagnosis is necessary. The consensus recommendations by the AASLD and APASL on the radiological diagnosis of HCC underscore the push for noninvasive diagnosis of HCC in avoidance of biopsy. While there is general consensus with regards to the surveillance for HCCs in high-risk patients, pertinent differences in the recommendations for imaging diagnosis of HCC exist. These reflect the differences in the availability of diagnostic imaging resources in different regions. For example, Sonazoid is not available for use outside of Japan and is therefore unique to the APASL guidelines. In a way, they also point to differences in practice patterns and the controversies in our understanding of “early” HCC. The AASLD guidelines demand that the classical enhancement features of HCC are demonstrated, accepting that this may limit sensitivity; biopsy is regarded as a means to restore sensitivity. On the other hand, the APASL guidelines emphasizes the use of Kuppfer specific imaging techniques to improve diagnostic performance.With rapid and continual improvement in diagnostic imaging modalities and validation of these guidelines, further refinements to the diagnostic algorithm can be expected in the near future. At present few of the established techniques have fallen out of favour; SPIO agents are on the decline due to decreased clinical usage, while double contrast MRI, CTHA and CTAP are cumbersome to perform and not compatible with routine clinical practice. Hepatocyte-specific MRI contrast agents are increasingly used in the United States, Europe and parts of Asia, as well as DW imaging, which is now already widely applied in routine clinical practice, demonstrate great promise to improve current methods of imaging diagnosis. However, before these can be incorporated into the imaging algorithms, validation of their utility is necessary. Similarly, the utility of imaging for other important aspects of HCC management, such as for noninvasive diagnosis of portal vein tumour thrombosis, may also need to be addressed in time to come.The authors have no conflict of interests to declare.