Wider adoption of evidence-based, health promotion practices depends on developing and testing effective dissemination approaches. To assist in developing these approaches, we created a practical framework drawn from the literature on dissemination and our experiences disseminating evidence-based practices. The main elements of our framework are 1) a close partnership between researchers and a disseminating organization that takes ownership of the dissemination process and 2) use of social marketing principles to work closely with potential user organizations. We present 2 examples illustrating the framework: EnhanceFitness, for physical activity among older adults, and American Cancer Society Workplace Solutions, for chronic disease prevention among workers. We also discuss 7 practical roles that researchers play in dissemination and related research: sorting through the evidence, conducting formative research, assessing readiness of user organizations, balancing fidelity and reinvention, monitoring and evaluating, influencing the outer context, and testing dissemination approaches. Although the public health community has developed many evidence-based practices to promote healthy behaviors, adoption of these practices has been haphazard ,2. In reSeveral conceptual frameworks have been developed to organize the extensive literature on diffusion and dissemination of evidence-based practices. Particularly relevant for the dissemination of evidence-based health promotion practices are those of Greenhalgh et al  and WandThese frameworks are useful for generating hypotheses for future research, but no practical framework exists for developing and testing dissemination approaches. Such a framework would serve as a guide to dissemination for community-based organizations and help researchers develop and test approaches to dissemination of evidence-based practices.We describe a practical framework for dissemination developed at the University of Washington Health Promotion Research Center (HPRC), a Prevention Research Center funded by CDC to conduct research on community-based prevention and control of chronic diseases. To illustrate the framework, we use 2 dissemination approaches we have developed and tested and discuss practical roles researchers play in dissemination and dissemination research.The evidence-based practices that can be disseminated using our proposed framework  include The HPRC framework acknowledges that practices can spread either passively or actively. The "diffusion" arrow illustrates that some evidence-based practices spread passively and are adopted without additional support or instigation from outside the user organization . The larThe HPRC framework has 3 main actors: researchers and disseminating organizations , left anResearchers and disseminators build a mutually beneficial collaborative partnership. The partnership can be initiated by either party and, in our experience, is most effective when the research team and the disseminator are trying to reach the same user organizations and are willing to learn about each other's resources and goals. Building on this learning, both the researchers and the disseminator are better equipped to design and test dissemination approaches that fit the disseminator's goals and capacity. Researchers work closely with disseminators in refining and testing the dissemination approach to make it more suitable for user organizations. Although researchers can serve as disseminators, disseminators have at least 2 advantages in disseminating best practices: 1) they can focus on dissemination, rather than on research objectives and funding, and thus focus on the support systems needed to reach the scale necessary to make a difference at a population level; and 2) they may be closer in culture and values to user organizations than are researchers and thus better able to promote best practices and adapt them to local needs.Successful dissemination of evidence-based practices in a user organization involves a cascade of steps . Steps iBetween the boxes showing the researchers and the disseminators on the left and the user organizations on the right is the bi-directional "linkages and learnings" arrow. This arrow highlights the need for understanding user organizations and all potential steps in the implementation process, from readiness factors through how to motivate adoption and facilitate implementation and maintenance. We believe these linkages and learnings should be informed by principles of social marketing, which focus on the needs and capabilities of user organizations. Applying these principles, the disseminator begins with a market analysis . This anconsumers of the practice(s).Consumers: the needs of the organizational and individual competition for the practice(s).Competition: the salient alternatives that serve as company  to support dissemination of the new practice(s).Company: the capacity of the collaborators — potential support networks or other partners — that can offer input on the constraints of real-world settings and facilitate dissemination in much the same way that a retailer helps distribute products to consumers.Collaborators: the strengths of context, described as the modifiable and unmodifiable "outer context" in our framework. Modifiable context — such as organizational networks, funding, and policy support  — is within the sphere of influence of the disseminators and researchers. Unmodifiable context includes the overall economic and political climate.Context: the sociopolitical segmentation can begin either at the organizational level (identifying unique groups of user organizations that are particularly well-equipped to adopt the practice) or at the individual level  that the practice promotes). In targeting, the disseminator identifies the specific segment(s) for which it wishes to develop a dissemination approach. Once the target is identified, the dissemination approach should be positioned to communicate the benefits of the promoted practices clearly and concisely, stressing the particular benefits that are expected to resonate with the target segment. At the level of the user organization, for example, positioning asks, What specific organizational needs will be met with this practice, more so than with other practices, and why?By conducting a full market analysis, the disseminator will be prepared to decide which segments of potential user organizations to target and how to best position the evidence-based practice(s) for these targeted segments . The proIn summary, the HPRC framework proposes that to disseminate evidence-based practices effectively, researchers must 1) collaborate with a disseminator and 2) work with user organizations to refine the practice and approach to dissemination, guided by the principles of social marketing.In this section, we present 2 examples to illustrate the dissemination framework . Enhanceevidence-based practice, is a low-cost, highly adaptable, group exercise program for older adults. EF was originally designed and tested by a research partnership that included the Northshore Senior Center in Bothell, Washington; Group Health Cooperative, a health maintenance organization; and HPRC. In its original trial, EF increased physical activity among participants and helped them to maintain a higher level of functioning than that of the comparison population   . After aThe HPRC framework for dissemination and dissemination research builds on the literature and has been shaped by our successful experience in developing and disseminating chronic disease prevention practices. As the 2 examples show, we have worked with private organizations as national disseminators, using theory-based dissemination approaches, and have achieved nationwide reach of evidence-based practices. The Workplace Solutions package of evidence-based practices includes environmental changes, policies, programs, and systems changes, so we believe this framework is applicable to all of these areas and to the broad field of translational research ,18.The HPRC framework incorporates elements from the Greenhalgh framework and parallels key elements of the Wandersman (ISF) framework; each of these frameworks is based on a review of a large body of literature. The key dissemination resources in our framework closely parallel those in Greenhalgh's resource system, namely the key roles for a change agency ("disseminating organization" in our terminology) and "knowledge purveyors" (researchers). Also from Greenhalgh, we adopt the idea that the process of implementation in an organization is organization-specific and involves a complex series of steps. Researchers, therefore, need to work closely with both disseminators and user organizations in designing dissemination approaches .Systems in the Wandersman ISF framework also parallel key elements of the HPRC framework. The ISF Prevention Support System, which assists in the process of dissemination, corresponds to our "dissemination resources" . And ourKreuter and Bernhardt  highlighIn the HPRC framework, researchers and disseminators form a partnership to enhance their ability to spread evidence-based practices. The contribution of researchers to the partnership includes 7 practical roles : 1) sortTo maximize real-world applicability of dissemination approaches, researchers should test them in collaboration with disseminators. When testing dissemination approaches, researchers and disseminators should be thoughtful about which dissemination approaches are being compared. Greenhalgh  suggestsWe have proposed a practical framework for designing and testing dissemination approaches for evidence-based practices. The framework outlines complementary roles for researchers, disseminators, and user organizations in disseminating evidence-based practices broadly. We have used the framework to disseminate evidence-based promotions for older adult physical activity and workplace health programs nationally. Other researchers and practitioners may find the framework useful in increasing the adoption of evidence-based practices.


Viewpoint our Perspective article published in the New England Journal of Medicine in which we argue for the urgent need for a clinical trial on when to initiate antiretroviral therapy (ART) in HIV-infected patients in sub-Saharan Africa [In this issue of the Journal, Delva et al. discuss in a n Africa . The autViewpoint authors’ arguments are important to address.As readers can review our Perspective article, we will not repeat arguments concerning the lack of definitive evidence to guide ART initiation, the limited evidence from observational studies nor the limitations of ongoing studies assessing timing for initiation of ART for African settings. Nonetheless, a few points that directly relate to the With regard to the observational studies the authors cite as evidence in support of early initiation of ART, two of the articles do not provide relevant information to the question of early versus deferred ART , 3 and tViewpoint's authors included a limited number of individuals who initiated ART in the higher CD4+ cell count strata, mostly derived from observational studies [3[Another argument that the authors present is that the use of early ART is associated with a protective effect against tuberculosis. However, it is important to note that the majority of available evidence thus far is based on the use of ART in patients with advanced HIV disease . In addi studies . HPTN 05udies [3. We beliThe authors argue that diversity in guidelines and practice internationally mainly reflects differences in financial resources available across regions, rather than the absence of definitive evidence. We disagree with this position; the attitudes and practices in the United Kingdom and other parts of Europe, for example, where therapy upon early diagnosis is not recommended, differ from those in the United States, but the reasons relate to different interpretations of the evidence and different approaches to care, rather than primarily financial considerations .The authors state that clinical trials are most appropriate for investigating efficacy and safety of new regimens, and are not of relevance to strategy questions that can inform guidelines. This overlooks the many clinical trials that compared various strategies rather than new drug regimens and that have had profound impact on guidelines and clinical practice. Relevant examples include HPTN 052, the SMART study, the SAPIT tuberculosis trial and the CIPRA 001 Haiti trial , 11–13. 3, the guideline document itself states that “further research is required to determine more fully the clinical benefits and disadvantages of earlier ART initiation” [There are compelling reasons to believe that early use of ART may best prevent adverse consequences of HIV infection. However, guidelines should be based as much as possible on firm evidence rather than expert opinion, especially when applied to millions of people and necessitating billions of dollars of funding. While the World Health Organization has released new guidelines that recommend initiation of ART in adults at CD4+ cell count of <500 cell/mmtiation” . Thus, eViewpoint pits patient autonomy versus clinical trials as reflected in the title. In contrast, we believe that clinical trials are necessary to provide the best quality evidence that would inform clinical guidelines and enable clinicians and patients to make informed decisions. We are struck that the individual approach advocated by the authors, with each patient/clinician making an independent decision based on incomplete evidence, is the antithesis of the public health approach promoted by the World Health Organization which has been the foundation of ART scale-up and perhaps the key factor allowing for successful access to ART to more than 9 million people globally, the majority in Africa [The n Africa .Health is both a personal right and responsibility, and no one should take medicines against his or her will. However, consistent advice based on the strongest evidence is an essential requirement for program implementation and assurance of accountability, and is fundamental to informed and empowered patients.


The German Society for Rheumatology recently published guidelines for the sequential therapy of rheumatoid arthritis (RA). These recommendations were developed as a transition from the 2010 EULAR (EUropean League Against Rheumatism) recommendations to the national clinical practice and are based on an updated systematic literature research and expert discussion. While most EULAR recommendations have remained unchanged, some were modified based on new evidence from randomized, controlled trials, current clinical practice, or national drug approval status. The guidelines also include a treatment algorithm for sequential therapy of RA with disease-modifying agents including biologics. On the verge of this year’s German Society of Rheumatology congress in Bochum, the new German guidelines for the medical treatment of rheumatoid arthritis have been published . Based oAlthough the predominant part of the single EULAR recommendations remains unchanged, recent data from the current systematic literature research resulted in distinct rephrasing of the original EULAR recommendations , Tab. 2.The national experts agreed to leave the overarching principles and the first two EULAR recommendations without modification. They refer to the quickest-possible diagnosis and treatment of rheumatoid arthritis at its best in accordance with the patient and the rheumatologist, targeting early remission and with respect to direct and indirect costs. Furthermore, it was beyond any doubt that according to recommendation 3, methotrexate should be part of the first treatment strategy.In case of methotrexate contraindication or intolerance, leflunomide and sulfasalazine are considered part of the first treatment strategy in both recommendations. However, injectable gold is not recommended in the current German guidelines. The main reason for not following the EULAR recommendation in this aspect and despite the available high-level evidence for the efficacy of injectable gold was the decreasing experience of the rheumatologists and the considerable side effects in long-term use.In contrast to previous decisions, EULAR recommendation 5 took for the first time a firm stand towards a DMARD monotherapy (disease-modifying antirheumatic drug) rather than a combination therapy of synthetic DMARDs. In fact, no clinical study has demonstrated a significant advantage of combination therapy over monotherapy in the absence of glucocorticoids. Nevertheless, the EULAR recommendation also states that while in DMARD naïve patients the balance of efficacy and toxicity favors monotherapy, the respective evidence is inconclusive in DMARD inadequate responders. Here, the German guidelines only state the lacking evidence for the advantage of combination therapy and recommend a monotherapy explicitly for DMARD naïve patients. In DMARD inadequate responders, a clear recommendation remains to be determined by reason of missing evidence.EULAR and German guidelines conform that adding glucocorticoids to DMARD monotherapy or combination therapy is beneficial for the patient. As evidence concerning doses and duration of glucocorticoid-bridging therapy is not available, both recommendations remain general and lack a more specific advice.The EULAR recommendation 7 emphasizes the importance of prognostic factors for the further treatment decisions. A biological DMARD can be added to a synthetic DMARD if poor prognostic factors are present in DMARD inadequate responders. Otherwise, a switch to another synthetic DMARD strategy should be considered. At this point, the German guidelines strongly recommend a combination treatment of several DMARDs in DMARD inadequate responders but also allow a biological DMARD as part of the combination therapy if poor prognostic factors are present. The conclusion of both guidelines is similar, even if the order of the German statements accentuates the possibility to primarily utilize the full potential of synthetic DMARD combination.According to the available evidence and the approval as first biological agents, the initiation of a biological therapy was mainly equalized with the initiation of a TNF (tumor necrosis factor) inhibitor in EULAR recommendation 8. This priority status of TNF inhibitors has been withdrawn in the German guidelines as the more recent biologic DMARDs abatacept and tocilizumab provided equivalent evidence for their efficacy and safety and are also approved as first-line biological agents. As indirect treatment comparison show similar efficacy for all biologic agents except for anakinra, no specific agents are recommended at this point for preferable first line therapy.The switch to a second biologic treatment after an inadequate response to the first biological therapy remains identical in the EULAR and German guidelines—as second TNF inhibitor, abatacept, rituximab or tocilizumab are possible agents without a specific ranking. However, full efficacy and safety data for a change to a defined second biologic agent following abatacept or tocilizumab are still lacking.In the case of rheumatoid arthritis refractory to several DMARDs and biologic agents, azathioprine, cyclosporin A and cyclophosphamide are specifically recommended by EULAR due to existing evidence on their efficacy, of course with respect to their individual toxicity. However, German recommendation 10 has been rephrased to a more general statement in order to also enable the application of other treatment options with the necessary reduction of evidence level to expert opinion.The final recommendations conform to the EULAR statements, even if recommendations 12 and 13 are combined in the German guidelines—specific suggestions concerning the procedure in case of a sustained remission are not provided due to the lack of evidence. Intensive treatment strategies and treatment adjustment considering structural progression, comorbidities and safety concerns are self-evident. EULAR recommendation 14 refers to DMARD naïve patients again and creates the possibility to begin a biologic agent in combination with MTX as a first treatment strategy in individual patients with poor prognostic factors. By reason of order, this exceptional case was included and discussed in recommendation 3 of the German guidelines.The aligned treatment algorithm summarizes the recommendations and represents the current practice subjected to the different strategy steps in the course of the disease .Taken together, the new 2012 German recommendations provide an update of the current evidence for the medical treatment on rheumatoid arthritis on the basis of the 2010 EULAR recommendations, providing an evidence-based real-life set of recommendations for the use in the daily practice of every rheumatologist in (and outside) of Germany.


Decision-making in child and youth mental health (CYMH) care requires recommendations that are developed through an efficient and effective method and are based on credible knowledge. Credible knowledge is informed by two sources: scientific evidence, and practice-based evidence, that reflects the "real world" experience of service providers. Current approaches to developing these recommendations in relation to CYMH will typically include evidence from one source or the other but do not have an objective method to combine the two. To this end, a modified version of the Grading Recommendations Assessment, Development and Evaluation (GRADE) approach was pilot-tested, a novel method for the CYMH field.GRADE has an explicit methodology that relies on input from scientific evidence as well as a panel of experts. The panel established the quality of evidence and derived detailed recommendations regarding the organization and delivery of mental health care for children and youth or their caregivers. In this study a modified GRADE method was used to provide precise recommendations based on a specific CYMH question .Overall, it appeared that early, parent-implemented interventions for autism result in positive effects that outweigh any undesirable effects. However, as opposed to overall recommendations, the heterogeneity of the evidence required that recommendations be specific to particular interventions, based on the questions of whether the benefits of a particular intervention outweighs its harms.This pilot project provided evidence that a modified GRADE method may be an effective and practical approach to making recommendations in CYMH, based on credible knowledge. Key strengths of the process included separating the assessments of the quality of the evidence and the strength of recommendations, transparency in decision-making, and the objectivity of the methods. Most importantly, this method combined the evidence and clinical experience in a more timely, explicit and simple process as compared to previous approaches. The strengths, limitations and modifications of the approach as they pertain to CYMH, are discussed. An essential ingredient in effective clinical practice is that clinicians benefit, in a timely way, from current research knowledge in their fields. Those responsible for the organization and delivery of child and youth mental health (CYMH) care require such knowledge, which is rapidly changing and ever increasing. Moreover, clinicians benefit from knowledge that is credible. This is especially critical in the development of clinical recommendations, which are intended to achieve the linked objectives of improvements in patient or client outcome and the efficient use of resources. To be credible, these recommendations must consider input from two sources: scientific evidence and the practice-based evidence, reflecting the "real world" expertise and experience of service providers.Clearly, it is less than practical to expect that individual clinicians can continuously survey and evaluate available knowledge and adapt clinical decisions accordingly. Despite this recognition, however, there are still obstacles that need to be overcome in improving the access to and practical application of credible knowledge in CYMH. Primarily, there is a lack of agreement about the best methods for organizations to identify, assess, and synthesize relevant studies -7. WithoNumerous professional bodies and scientific organizations have taken the approach of developing clinical practice guidelines. These guidelines include recommendations to aid patient and practitioner decision-making in specific clinical situations. A strength of this approach is that a panel of experts typically develops these recommendations. Panel members have access to the evidence, an understanding of the clinical problems and research methods, and sufficient time for deliberation. However, the recommendations are often formulated without systematically identifying and summarizing the evidence . A propeSystematic reviews take a different approach by using explicit search strategies and inclusion criteria to identify and judge evidence of the effectiveness of clinical interventions . The relInput from practicing health professionals can make recommendations more practical and clinically- relevant. This can be accomplished through an approach that integrates scientific evidence with the knowledge of patients' values and preferences as well as the implications for treatment planning and decision-making in clinical practice.The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach  incorporGRADE employs a sequential assessment of 1) the quality of the evidence, followed by 2) an evaluation of the balance between benefits and harms and 3) a subsequent judgment about the strength of recommendations . The sepGRADE was developed based on a systematic review that identified the fundamental elements of an effective evidence grading system. These elements include: the rating of the quality of individual studies; the quantity in terms of magnitude of effect, number of studies and sample size; and the consistency of the findings between similar and different study designs .The current approaches to synthesizing evidence-based practice research in the CYMH field should be more appropriately described as clinical treatment efficacy research . In the The process begins with the selection of a focused question that can lead to action and the establishment of an expert panel with knowledge related to the question. Ideally, and for the sake of efficiency, relevant published and unpublished scientific evidence relating to the question is gathered from an existing systematic review. A list of potentially important outcomes is then created which panel members must independently rate according to importance ranging from unimportant to critical. The improvement of outcomes with high patient importance  should lead to stronger recommendations .Following the rating of the importance of outcomes, the quality of the evidence is then rated. A number of factors can increase or decrease the quality rating and this should reflect the confidence in the studies' estimates of benefits and harms.Finally, the evidence, its quality rating and the outcome information is synthesized succinctly and transparently in evidence profiles. These profiles guide the panel in deciding on the strength of recommendation according to the tradeoffs between benefits and harms of the various alternatives.Based on conceptual arguments and past experience with the process, many developers and users of GRADE believe that this process can be applied across all forms of intervention . Others Against this backdrop, the Provincial Centre of Excellence for Child and Youth Mental Health at the Children's Hospital of Eastern Ontario completed a pilot project to investigate the use of a streamlined version of the GRADE approach. Autism intervention research was identified as a suitable and interesting area in which to pilot test this methodology. Specifically, this streamlined method was used to determine current credible knowledge concerning the effects of parent-implemented, early intervention for children with autism. Implementing a methodology such as GRADE has considerable implications for improving efficiency, communication, and clinical decision-making. GRADE may facilitate the progression of the CYMH field toward greater use of evidence-based practice by service providers. This pilot study is the first step in evaluating the feasibility of this methodology as applied to CYMH research and determining whether any modifications are necessary in its application to this field. The findings related to autism as well as the strengths and limitations of the approach will be discussed.The history of controversy surrounding autism makes this area suitable for this pilot project and is certainly illustrative of the importance of an established methodology for determining credible knowledge in the field of CYMH. Autism research has, for example, seen the persistence of a dispute surrounding evidence concerning causal linkages associated with the measles-mumps-rubella (MMR) vaccine or the vaccine preservative thimerosal. Notwithstanding, there has been no evidence to support an association between an MMR vaccine and autism, nor is there a plausible biological mechanism [as reviewed by 29]. Valuable time and resources were used to conduct as many as twenty studies to definitively respond to a study , whose aThere has also been intense debate surrounding the evidence base and suitability of the variety of educational strategies and interventions that have been developed for individuals with autism -37. For Narrative reviews of interventions in this area have sought to address this problem by identifying evidence-based treatments. Unfortunately, many of these reviews have not taken a systematic approach, limiting their comprehensiveness and validity ,35,40-44The National Research Council  and New Parent-implemented interventions have been a particular focus in recent reviews, as increased parental skills allow for continual opportunities for children's learning across a range of situations ,49. EvenThe process used for this pilot project was adapted from that described by the GRADE working group and is more fully described elsewhere ,24,50-53What is the current credible knowledge concerning the effects of parent-implemented, early intervention for their children with autism?Adjustments to the process were made throughout the course of the pilot project to adapt the method in answering a CYMH question. Any changes to the process are noted. The question selected for this pilot project was: A detailed literature search was performed for existing systematic reviews that addressed this question. Unfortunately, systematic reviews by Diggle, McConachie & Randle  and McCo(1) Families had to have at least one child who received a formal diagnosis of autism.(2) Parent-implemented: At least one parent was formally trained to deliver an intervention for their child with autism, which was different in some way from any intervention(s) delivered by service providers.(3) Early intervention: At the time of study initiation, within each study group at least one recipient of a parent-implemented intervention had been under eight years of age.(4) Research designs: prospective and controlled.(5) All publication dates considered.(6) Effects: any outcomes , which includes benefits and harms.Based on these criteria, an independent information specialist searched the following databases: ERIC, The Cochrane Library, MEDLINE, EMBASE, PsycInfo, CINAHL, and Dissertation Abstracts International. Other sources of information were examined including the bibliographies of systematic and narrative reviews and reference lists of key articles identified through the search strategy. Screening by three independent screeners of the results yielded 11 primary studies to be included -65. 	ThiA panel, which included three clinical and methodological experts in autism research, was then formed. In this pilot project, one role of the panel was to serve as a proxy for end users in determining the values and preferences of those end users in rating the importance of outcomes. The inclusion of additional members was beyond the resources of this work.In addition to using the panel's experience, an extensive literature search was performed to identify preference estimates from population-based studies in order to gain additional, evidence-based, perspectives in rating the importance of outcomes. These studies are intended to provide information that reflects the values, beliefs and preferences of those affected by autism. The panel could use these studies in rating the importance of outcomes. The panel then created a list of potentially important outcomes to rate, independent of the primary studies. Outcomes are rated on a scale from 1-9: a rating of 7-9 indicating that the outcome is critical for a decision or recommendation; a rating of 4-6 indicating it is important; and, a rating of 1-3 indicating it is not important.Quality of the evidence was then assessed using the GRADE four-category system . The quality of the evidence reflects the extent to which one can be confident that the estimates of the benefits and harms of a researched intervention are accurate. Evidence based on randomized controlled trials begins with a high quality rating, observational studies with a low rating and any other evidence with a very low rating. However, studies may then be upgraded or downgraded. Five limitations--related to study quality, consistency, directness, precision, and reporting bias--may lead to downgrading. Upgrading the quality of the evidence may result from large effects, dose-response gradient and if all plausible confounding variables or biases would decrease the demonstrated effect, such as only sicker patients receiving an experimental intervention, yet still faring better. It is important to note that evidence that is downgraded for any reason cannot be upgraded. To ensure transparency and objectivity, the panel provided explicit reasons for downgrading and upgrading by including footnotes in the evidence profiles; this is a requirement of the GRADE system.Due to the heterogeneity of the evidence base for the present application, namely that each study evaluated a different intervention using a different set of outcomes along with variations in the specific populations studied and those implementing the interventions, study results could not be combined nor directly compared. This diverged from the typical GRADE method where studies evaluating the same outcomes are combined and the quality of each outcome is then evaluated separately. In the present modified version of GRADE, studies were individually appraised for quality on a number of dimensions , according to the extent of difference between benefits and harms. The strength of a recommendation is also influenced by the values and preferences of end users, in that critical outcomes are the primary factors for establishing a recommendation. This allows the GRADE system to emphasize that weak recommendations in the face of high quality evidence and vice versa, can occur. This is because other factors, beyond the quality of the evidence, including the values and preferences of the relevant population and the importance of the outcomes, impact the strength of a recommendation . Given tRecommendations included details about the population and intervention. This should aid in the interpretation of the strength of the recommendations and help clinical decision-makers deal with individual patients. Given that some of the judgments leading to the recommendations are subjective, the decisions made are explicit and transparent in the remarks that go along with the recommendations. Specific suggestions for new studies to be undertaken were also noted . However, as opposed to an overall recommendation for these interventions as a whole, specific recommendations had to be made based on the question of whether a particular intervention's benefits outweighed its harms, a necessary departure from the original GRADE method.The first limitation may also be considered one of the strengths of this study. The GRADE approach was being implemented in a new field of research. Yet, as a result, there were no previous detailed applications or standards to draw from in the literature and modifications to the process were required. Some of the lessons learned from the current application of GRADE should inform future implementations of this approach in CYMH. This may also suggest that another panel may not reproduce the same results. In fact, others may disagree with aspects of our application to GRADE in this area of research and the subsequent grading of the quality of evidence and strength of recommendations. However, the transparency maintained by this process should allow others to understand the decisions made by this panel. For example, studies may be downgraded due to a lack of precision when there are very few patients or events, but there are no defined standards for these terms. Consequently, this panel followed the GRADE principle that a study may require greater precision in support of a recommendation when there is a close balance between the advantages and disadvantages of an intervention . Given tThe lack of variability in the composition of the panel may also be considered a limitation in the present application of GRADE. This panel was composed of clinical and methodological experts in the area of autism research. This may have reduced the variability of opinion in establishing the values and judgments and created a conflict of interest in reviewing and appraising evidence in their own field. This became a more prominent factor when the panel was required to appraise individual studies and provide individual recommendations for each intervention as part of the modified GRADE approach. However, these experts also had unique insight into the values and preferences of end-users, for which they were used as a proxy, as well as previous experience in this area of research that facilitated the timely completion of this pilot project. Moreover, these modifications to the process were necessary due to the evidence base and, again, all decisions were made transparent.Examining the cause of the somewhat unsatisfactory finding, it is seemingly a product of the evidence and not due to any shortcomings in the GRADE method itself. The results of the studies could not be combined or compared due to the high degree of heterogeneity in the evidence base. Two interventions were rarely comparable in terms of the content of the intervention, outcomes measured, specific population studied or those implementing the interventions. Therefore, it was not always possible to construct a consistent account of the effects of an intervention. This is a common finding in mental health, and specifically autism research ,49,67,68Research in this area has been identified as containing numerous gaps and being too heterogeneous to allow for direct comparisons of interventions ,68. A nuThe current approaches to synthesizing and evaluating this evidence cannot overcome the limitations in the research. An assumption of these approaches is that the development of the evidence-base has considered the fit between the intervention and its use in clinical practice. In reality, this fit has rarely been taken into account. Moreover, many make the assumption that all randomized control trials are of a high quality since they are generally considered the gold standard in research. The difficulty with this assumption is that there are situations where randomized control trials are impractical or unethical ,74. ThisIn effect, the evidence-base in CYMH is primarily comprised of efficacy studies for which development is based on theory, methods and models to evaluate treatments that do not correspond to the demands of clinic and community-based care . As a reThe GRADE approach explicitly addresses the fit between the research and its use in practice through the functions of the panel, a clear strength. While research should still attend to the factors noted above, GRADE can focus on the clinical relevance of the considerable body of evidence already in existence. Another strength of this approach is that it does not reduce the evidence underlying an intervention to a simple Yes/No conclusion, but it includes information about key variables from the studies in the recommendations. This should aid in the interpretation of the strength of the recommendations and may affect clinical decision-making due to diverse values and preferences of individuals implementing the interventions. The short amount of time required to prepare the recommendations was also a strength of this process. It took four months to complete the process, whereas guideline development and systematic reviews can take up to two years or more ,9,10.This approach also benefits from an explicit methodology, transparency, consistency in judgments about the quality of evidence, and linking evidence to recommendations. Though evidence is linked to recommendations, the judgments about the quality of the evidence and the balance of benefits and harms leading to the recommendations are made separately. Not all grading systems separate decisions regarding the quality of the evidence from the strength of recommendations . Those tIt should also be noted that GRADE has now been adopted by a number of organizations including the Cochrane Collaboration and the World Health Organization ,53. The These strengths contribute to our experience that GRADE is a feasible approach in CYMH. Its methodology promotes reconciliation between clinicians', methodologists', administrators' and patients' perspectives along with the scientific evidence. Eliminating this traditional separation, seen as a significant limitation in clinical practice guidelines and systematic reviews, will help the evolution of the CYMH field toward the use of evidence-based practice by service providers.Future implementations of the GRADE approach should include a broader representation of stakeholders in the panel than was used in this pilot project. Other members of the panel could include those of varying levels of service administration such as policy makers, parents, and other practitioners. They would provide more varied perspectives and reduce the gap between the evidence and practice. For example, considering practical barriers to implementation, such as funding restrictions or gaps in clinical expertise could be especially helpful in making the most useful recommendations. In addition, including stakeholders of varying levels would help to drive the knowledge exchange of the findings. Knowledge exchange is an area in and of itself that should be explored to provide guidance on the best procedure for translating results in order to produce meaningful change when using the GRADE method . Other cIn conclusion, it was determined that it is feasible to develop more timely recommendations in CYMH, specifically early parent-implemented autism interventions, using a modified version of the systematic and transparent GRADE approach. A definitive answer as to which is the best intervention was not produced. However, the GRADE approach appeared capable of providing a specific answer had the evidence been less heterogeneous and directly comparable. Due to the heterogeneity of the evidence a modified GRADE approach was presently evaluated. As research in this area increases and fills in the gaps identified above, it is important to have a method to evaluate the benefits and risks of new interventions, in an efficient manner, as they arise. The GRADE process meets this need and establishes credible knowledge that leads directly to recommendations for effective organization and delivery of CYMH care. Most importantly, GRADE and the recommendations developed from it, will help CYMH service providers and users to make treatment decisions that are informed by the scientific and practice-based evidence.The authors declare that they have no competing interests.JMVA conceived of the study, and participated in its design and coordination and drafted the manuscript. JDG, AP, and JAMR were expert panel members and as such contributed significantly to the findings and also contributed to the study design, and edited the manuscript. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/11/60/prepubAppendix - Sample Recommendation. Example of recommendation from the pilot-projectClick here for file


Health-system policy makers need timely access to synthesised research evidence to inform the policy-making process. No efforts to address this need have been evaluated using an experimental quantitative design. We developed an evidence service that draws inputs from Health Systems Evidence, which is a database of policy-relevant systematic reviews. The reviews have been (a) categorised by topic and type of review; (b) coded by the last year searches for studies were conducted and by the countries in which included studies were conducted; (c) rated for quality; and (d) linked to available user-friendly summaries, scientific abstracts, and full-text reports. Our goal is to evaluate whether a "full-serve" evidence service increases the use of synthesized research evidence by policy analysts and advisors in the Ontario Ministry of Health and Long-Term Care (MOHLTC) as compared to a "self-serve" evidence service.We will conduct a two-arm randomized controlled trial (RCT), along with a follow-up qualitative process study in order to explore the findings in greater depth. For the RCT, all policy analysts and policy advisors (n = 168) in a single division of the MOHLTC will be invited to participate. Using a stratified randomized design, participants will be randomized to receive either the "full-serve" evidence service  or the "self-serve" evidence service (database access only). The trial duration will be ten months . The primary outcome will be the mean number of site visits/month/user between baseline and the end of the intervention period. The secondary outcome will be participants' intention to use research evidence. For the qualitative study, 15 participants from each trial arm (n = 30) will be purposively sampled. One-on-one semi-structured interviews will be conducted by telephone on their views about and their experiences with the evidence service they received, how helpful it was in their work, why it was helpful (or not helpful), what aspects were most and least helpful and why, and recommendations for next steps.To our knowledge, this will be the first RCT to evaluate the effects of an evidence service specifically designed to support health-system policy makers in finding and using research evidence.NCT01307228ClinicalTrials.gov:  Systematic reviews are increasingly seen as a key source of information to inform these decisions [any question constitutes a more efficient use of time for busy policy makers because the research literature has already been identified, selected, appraised, and synthesised in a systematic and transparent way. Additionally, a systematic review makes possible more constructive policy debates because stakeholders can focus on the synthesis and its local applicability rather than on which single study has greater credibility [Health-system policy makers make important decisions every day about the governance, financial, and delivery arrangements within which programs, services, and drugs are provided and about implementation strategies . The natecisions . Reducedecisions . Drawingdibility .In order to make informed decisions, health-system policy makers need timely access to systematic reviews that can be easily retrieved using terminology that is understandable to them and that are presented in ways that facilitate rapid scanning for relevance, recency of searches for potentially relevant studies, the settings of studies included in the review, and quality of the review ,4. A syshttp://kuuc.chair.ulaval.ca/english/index.php) and CHAIN Canada (http://www.epoc.uottawa.ca/CHAINCanada/), do not focus on systematic reviews. Existing evidence services that focus on high-quality studies (not just systematic reviews), such as Evidence Updates (http://plus.mcmaster.ca/EvidenceUpdates/), do not target health-system policy makers[One response to the similar types of issues faced by clinical decision makers has been the development of evidence services that provide regular email alerts about newly identified research products and a searchable database of these products. Howevercy makers.i.e., by health-system arrangement or implementation strategy), type of review , and type of question addressed ; (b) coded by the last year in which searches for studies were conducted and by the countries in which included studies were conducted; (c) rated for quality using the AMSTAR (A MeaSurement Tool for the 'Assessment of multiple systematic Reviews') instrument [To address this gap, we developed a full-serve evidence service for health-system policy makers. First, we developed Health Systems Evidence, which contains over 1,400 syntheses about governance, financial, and delivery arrangements within health systems and about implementation strategies relevant to health systems. By syntheses we mean both systematic reviews and two types of review-derived products, namely, policy briefs and overviews of systematic reviews . A policstrument ,9; and  and developed a mechanism to reimburse publishers for full-text downloads of these reviews.Third, we developed the format for monthly email alerts, which (in tabular format) identifies new additions to Health Systems Evidence and describes the type of review, type of question addressed, health-system arrangement or implementation strategy addressed, and title of the review. A hypertext link for each review enables policy makers to view the availability of (and links to) user-friendly summaries, scientific abstracts, and the full-text review. A hypertext link to the online Health Systems Evidence webpage enables policy makers to view additional information about these same recent database additions, including the last year searched, quality rating, the countries in which included studies were conducted, and the complete citation. Our goal is to evaluate whether (and how and why) a full-serve evidence service increases the use of synthesised research evidence by policy analysts and advisors in the MOHLTC as compared to a 'self-serve' evidence service. The full-serve evidence service comprises database access (an effort to facilitate policy makers' efforts to 'pull' in research when they need it), monthly email alerts about new additions to the database (a 'push' effort), and full-text article availability . A systematic review found that simply providing information  can change clinical behaviour, which leWe will conduct this trial using a sequential explanatory mixed-methods design,  beginninTo recruit participants who deal with health-systems issues on a regular basis, we will invite all policy analysts and policy advisors from one purposively selected division of the MOHLTC to participate in the RCT. All division staff members similarly face a relatively new expectation about obtaining training in finding and using research evidence , as well as a new mandate for using the Ministry's 'Research Evidence Tool' for submissions that support decision making at the Ministry Management Committee and cabinet levels. Moreover, a trial endorsement letter will be signed by the Assistant Deputy Ministry responsible for this division. These contextual developments precede the launch of the trial and help to create a favourable climate for the use of research evidence among all potential trial participants.Based on estimates provided to us in June 2009 by the MOHLTC, there are approximately 49 policy analysts  and 99 senior policy analysts in the division (n = 148). We do not yet have an accurate estimate of the number of policy advisors in the division; however, this group is likely to include roughly 20 people and all of them are likely to be senior policy advisors. By including all three levels of policy analysts and (if applicable) both levels of policy advisors, we will gather evidence from a diverse group that plays different roles in the policy-making process. For example, a policy analyst might conduct the initial, extensive 'workup' of an issue, whereas a senior policy advisor might write a short briefing note for the Minister.i.e., 14 to 22 months before the trial will begin). We delivered five additional one-day workshops, one half-day workshop, and one half-day webinar for policy analysts and policy advisors, as well as one 1.5-hour workshop for more senior MOHLTC executives who set expectations for these staff, between January and March 2010 . Given the division's expectation about training, we can assume that most RCT participants will have received the training. However, newly hired policy analysts and advisors may not have received the training, and others may not have been able to participate due to scheduling conflicts; those that have received the training will differ in the recency of the training. Second, these RCT participants will differ in their experience, which is somewhat related to their position . To address each of these applicability/generalisability issues, we will stratify the randomisation based on past training and current position (see below).Selecting this sample of policy analysts and advisors raises two applicability/generalisabilty issues. First, these RCT participants will differ in whether and when they received training on finding and using research evidence. Two of us (JNL and MGW) delivered a series of five one-day workshops for policy analysts and advisors at the MOHLTC between July 2008 and March 2009  access (facilitating pull)• monthly email alerts (push)• full-text article availability (facilitating pull)http://www.healthsystemsevidence.org.Participants allocated to the self-serve evidence service will receive only database access, which is already publicly available at Participants will be randomised using a stratified design. After completing the baseline questionnaire during the two-month baseline period, participants will be allocated to strata based on past workshop attendance (yes or no) and their position . This two-layer stratification will produce six strata. Participants will be randomised after all those who consent to participate in the trial have completed the baseline questionnaire. We will assign a unique participant ID number to each participant and then provide the list of IDs to a biostatistician external to the research team who will conduct the randomisation and keep a log to provide a clear audit trail. The biostatistician will then communicate directly with a knowledge broker external to the research team who will be generating the email alerts and with the website server administrator at McMaster University who will be establishing which participants get access to which evidence service. The participants and investigators will be blinded to group assignment.e.g., healthcare professionals, teams, and institutions), which may in turn influence whether cost-effective programs, services, and drugs get to the patients who need them and have their desired impacts, and which in turn may translate into improved health [Measuring the impact of knowledge transfer and exchange (KTE) interventions, such as the evidence service proposed here, poses significant challenges . The ultd health . Moreoved health -18. Simid health -21.e.g., user-friendly summaries, scientific abstracts, and full-text reports) are accessed; the mean number of minutes per month that participants use the database (with a 'time out' set at 60 minutes); and the number of times the monthly email alerts are forwarded.Given these challenges, our primary and secondary outcomes for the trial are proxy measures for the use of research evidence in policy making. The primary outcome will be a measure of utilisation that is similar to the one used in a trial of the McMaster Premium Literature Updating Service (PLUS) . SpecifiHealth Systems Evidence will be hosted on a secure server at McMaster University and will require a user login that will be used to accurately track their usage of the database. A user login is necessary because individuals from the MOHLTC do not have a consistent IP address when accessing external websites, which would preclude the collection of utilisation data if the site were hosted without requiring users to login. In addition, requiring user login will partially protect against contamination of the control group. However, we cannot rule out the possibility that individuals in the intervention arm of the study will forward monthly email alerts and full-text systematic reviews that are available only by subscription to individuals in the control arm; however, we will collect data about alert forwarding.i.e., beliefs and judgments), subjective norms , and perceived behavioural control --that shape the behaviour intentions of people, which is in turn a strong predictor of future behaviour [For the secondary outcome, we will use a survey based upon the theory of planned behaviour to measure participants' intention to use research evidence. The theory of planned behaviour is a model of how human action is guided ,23, and ehaviour . In FiguThe theory of planned behaviour has been extensively used and tested in the fields of psychology and healthcare. Systematic reviews conducted in the psychology field have demonstrated that the theory explains about 39% of the variance in intention and about 27% of the variance in behaviour ,25. A nuet al. (2010) found moderate test-retest reliability of the instrument using generalisability theory (G = 0.50) [G = 0.9) when scores were generalised across the average of two administrations of the tool [et al. (2010), the first administration of the tool immediately followed a McMaster Health Forum stakeholder dialogue, which may have promoted enthusiasm for using research evidence among participants. This likely produced higher measures of intention on the first administration of the tool as compared to the second, resulting in the lower G score. Given that we won't be administering the tool in a similar atmosphere of enthusiasm for using research evidence, we are confident in the level of reliability of the tool without two administrations at both baseline and follow-up. We modified the instrument by adding a question to measure the perceived usefulness of the intervention, as well as questions about participant characteristics.Using a manual to support health researchers who want to construct measures based on the theory , we deve = 0.50)  when scothe tool . In the We will administer the instrument during the baseline period, as well as at the end of the six-month intervention period, through a brief online survey that takes approximately 10 minutes to complete. We will use unique identifiers for each participant to ensure their responses to the previous survey are linked for calculations of before-and-after changes in their intention to use research evidence. We will follow up with participants who do not complete the survey once per week for three weeks to minimise the number of participants lost to follow-up.Data will be entered into SPSS 16.0  after all data collection has been completed. Analyses will be conducted by two members of the research team (SH and MGW), and during the analysis, neither they nor other study investigators will have access to the key linking the participants to their unique identifiers.e.g., calculating the mean over two months) if there are insufficient data for analysis. Moreover, as part of a secondary analysis, we will assess whether there is an interaction between each of these variables (entered as a fixed factor) and the outcome measures. We will also qualitatively compare the number of participants in the intervention and control groups that do not complete the follow-up survey and assess whether their baseline characteristics can help to explain their loss to follow-up.We will treat both outcome measures as continuous variables and analyse the change in these measures over time using a two-way mixed-effects linear repeated-measures analysis of variance (ANOVA), with the interaction of intervention by time as the main feature of interest. In addition, we will control for four variables--past workshop attendance, position , branch within the division (of which there are six), and number of years working at the MOHLTC--using analysis of covariance. Given the likelihood that the distribution of the outcomes will be skewed, we will transform the data where necessary and possible, which may include adjusting the time period for which we calculate the mean number of site visits/participant  will be considered significant. For the primary outcome measure (mean number of site visits/month/participant), missing data are irrelevant because it is a naturalistic measure. For the secondary outcome measure (obtained through the survey), missing data can be taken into account through the use of a mixed-effects model.For all analyses, we will use the intention-to-treat principle and report 95% confidence intervals; Given a fixed sample size of at least 148 policy analysts and advisors in the division, a sample-size calculation is not relevant. Instead, we have calculated the level of statistical precision that we can expect given our fixed sample size. We had no mechanism to estimate the intraclass correlation coefficient (ICC) for measurements of the primary outcome for individuals over time. Therefore, we calculated estimates of statistical precision for ICCs of .2, .3, .5, .7, and .8 based on a six-month trial period with 80% power; an estimated standard deviation of 1.0; significance of .05; and 74 participants per study group . Assuming the primary outcome data will be collected from all 148 participants at baseline and at six follow-up points (one per month), the time-averaged detectable difference (in standard deviation units) between the two groups is at best 0.27 (ICC = .2), which increases with successively greater ICCs to 0.30 (ICC = .3), 0.35 (ICC = .5), 0.40 (ICC = .7), and 0.42 (ICC = .8).Given that this is the first RCT evaluating a KTE intervention for health-system policy makers (at least to our knowledge) and given the inherent limitations associated with measuring research use as an outcome, we will conduct a qualitative process study after the completion of the trial to explore the RCT findings in greater depth. The qualitative study will explore how and why the evidence service worked (or didn't work), including the role of past workshop attendance and position and the degree of contamination between the intervention and control groups.i.e., full-serve or self-serve evidence service), outcomes, past workshop attendance, position, branch within the division, and number of years working at the MOHLTC. We have assumed a 70% response rate , which means that we should sample approximately 40 policy analysts and advisors in order to achieve a sample size of 30.We will use a mixed-method sequential nested sampling procedure, whereby a larger sample is analysed in one study (RCT) and a subset of the larger sample is selected for further inquiry in the second study . SpecifiOne-on-one semistructured interviews will be conducted either by telephone or in person (where possible) on participants' views about and experiences with the evidence service, including whether and how they used it  and why, whether and how it was helpful in their work and why, what aspects were most and least helpful and why, and recommendations for next steps. Potential explanatory factors (for which we will probe) include past workshop attendance, position, branch within the division, and number of years working at the MOHLTC.i.e., we will send a brief, structured summary of what we learned from the interviews and invite comment on it).We will tape and transcribe all interviews, use NVivo 8  for data management, and use a constant comparative method for analysis -37. SpecTo our knowledge, this will be the first RCT to evaluate the effects of an evidence service specifically designed to support health-system policy makers in finding and using research evidence. While there have been a number of strategies developed to both support the production of policy-relevant research evidence and the identification and use of research evidence by health-system policy makers ,38, rigoThe main potential limitation of the RCT is that it will be conducted within one division of the MOHLTC, and hence, there is the potential for contamination of study groups despite the use of a user-specific login. Given that many of the policy analysts and advisors work collaboratively, resources from the full-serve evidence service may be shared with those who had been allocated to the self-serve arm. Unfortunately, there is no mechanism to protect against this fully. However, we will adjust for variables  that may be correlated with degree of collaboration, and hence likelihood of contamination; we will measure the number of times that monthly email alerts are forwarded; and we will ask about contamination in the qualitative process study. Furthermore, if we find a significant amount of contamination through the qualitative study, it suggests that the full-serve evidence service is perceived as highly useful by those not allocated to receive it.Three of the authors  were involved in the development, and remain involved in the continuous updating, of Health Systems Evidence, which is the intervention being tested in the trial.JNL conceived of the study, participated in its design, led its planning, and helped to draft the protocol. MGW participated in the design and planning of the study and drafted the protocol. JMG and RBH participated in the design of the study and provided feedback on drafts of the protocol. SH participated in the design of the study, supported the sample-size calculations, and provided feedback on drafts of the protocol. PR, RG, and MO provided feedback on drafts of the protocol. All authors read and approved the final manuscript.


Clinical practice guidelines are one of the foundations of efforts to improve health care. In 1999, we authored a paper about methods to develop guidelines. Since it was published, the methods of guideline development have progressed both in terms of methods and necessary procedures and the context for guideline development has changed with the emergence of guideline clearing houses and large scale guideline production organisations . It therefore seems timely to, in a series of three articles, update and extend our earlier paper. In this first paper we discuss: the target audience(s) for guidelines and their use of guidelines; identifying topics for guidelines; guideline group composition (including consumer involvement) and the processes by which guideline groups function and the important procedural issue of managing conflicts of interest in guideline development. Clinical practice guidelines (hereafter referred to as guidelines) are one of the foundations of efforts to improve health care. The modern age of guidelines began with a 1992. Institute of Medicine report, which defined guidelines as “systematically developed statements to assist practitioner and patient decisions about appropriate health care forspecific clinical circumstances”.[http://www.guideline.gov)) that identify and systematically characterize guidelines on a number of domains and the development of robust guideline appraisal instruments such as the AGREE tool [http://www.g-i-n.net/). Against this background it seems timely to, in a series of three articles, update and extend our earlier paper on the methods of developing clinical practice guidelines. This series is based on a background paper [To help users identify and choose guidelines there has been the emergence of guideline clearing houses  for guidelines and their use of guidelines; identifying topics for guidelines; guideline group composition (including consumer involvement) and the processes by which guideline groups function and the important procedural issue of managing conflicts of interest in guideline development. Inthe second paper we will move on to discuss issues of identifying and synthesizing evidence: deciding what type of evidence and outcomes to include in guidelines; integrating values into a guideline; incorporating economic considerations; synthesis, grading, presentation of evidence; and moving from evidence to recommendations.Finally, in the third paper we will discuss the issues of: reviewing, reporting, and publishing guidelines; updating guidelines; and the two emerging issues of enhancing guideline implementability and how guideline developers should approach dealing with the issue of patients who will be the subject of guidelines having co-morbid conditions. By updating our previous work we hope to offer to a general reader a useful current overview of the main methodological and procedural issues in guideline development.primary and secondary audiences. The primary audience is the category of clinicians (and patients) for whom the guideline is nominally intended and who are most likely to use the guideline in patient care settings. However, guideline developers often recognize a secondary audience that takes considerable interest in the recommendations. For example, a family medicine or paediatrics society may develop guidelines for its practitioners, knowing that other primary care professionals could refer to the guidelines in managing the same condition. Guideline developers are also aware that their recommendations may be used in policy processes such as informing coverage decisions by healthcare providers, insurers and employers’ benefits packages.An important first step in developing a guideline is to clarify the target audience. A clear sense of the target audience informs subsequent decisions about the guideline’s scope, objectives, and format and style of wording. Typically, guidelines have both Guidelines often inadvertently focus on physicians as the target audience and use the term “physician” or “doctor” in lieu of “clinician,” but the topics they address may be equally relevant to a wider range of clinicians. For example, a guideline on a primary care chronic disease management topic (such as diabetes mellitus) will be relevant not only to primary care physicians but also to nurse practitioners, physician assistants, and other clinicians or social workers who work in primary care settings. Even when the target audience is clearly physicians, it is useful to clarify the type of physician(s) for whom the clinical practice guideline is primarily intended. Guidelines intended for primary care physicians may include content of less interest to specialists, and vice versa. Guidelines on a highly specialized procedure, performed only by sub-specialists, are unlikely to be used by primary care physicians and therefore need not review basic background on the health condition, can focus on narrow evidence questions, and can use specialized terminology without extensive elaboration.There are many factors that influence the choice of topics for the development of guidelines including scientific criteria set alongside the needs of health care policymakers, specialty society professional identification, and in response to popular media coverage of certain high profile cases. Guidelines can be developed around conditions  or around procedures  The choice has to do with the target audience and the need. Most guidelines are developed for conditions. Among the scientific factors are the criteria in Table Thus, situations for the development of guidelines are often considered to be those conditions which are common, costly, have large effects on premature mortality or morbidity, for which there is good evidence that appropriate health care can make a difference in outcomes, and for which we know that there are wide variations in current care  or prima facie knowledge that current care does not meet some well-accepted standards. Clearly, judgment is needed in applying these criteria. There are some conditions that are rare, but for which receiving appropriate medical care is the difference between a normal health outcome and catastrophe (e.g. phenylketonuria). Such conditions should not be dismissed as a low priority for guidelines based solely on prevalence. Similarly, there are some conditions which have only modest effects on morbidity, but which are so common that the cumulative effects of appropriate management at a population level may be large (e.g. acute pharyngitis). Also, the evidence that differences in clinical management lead to differences in health care outcomes does not necessarily have to be positive - for some conditions, too much health care might produce negative individual health or population outcomes (e.g. treating acute upper respiratory infections with antibiotics may cause patients to have antibiotic associated diarrhea or may increase community antibiotic resistance rates).Most often the decisions about the relative importance of prevalence, cost, effect of health care, etc., are made implicitly; there is little published about how such decisions are made. The US Preventive Services Task Force (USPSTF) has a Topic Prioritization Workgroup that regularly assesses nominations for guidelines that come from professional societies and other groups. The USPSGuideline development involves both a technical process (systematic reviews of relevant evidence) and a social process (interpretation of the results of the systematic review and development of recommendations). The validity of guideline recommendations can be adversely influenced if either process is biased. While there has been considerable methodological focus on ensuring the validity of the technical process, less attention has been paid to optimizing the social process.-14 The qmore appropriate indications for that procedure. They also identified five studies that compared the recommendations made by uni-disciplinary and multi-disciplinary groups; in all studies, recommendations by multidisciplinary groups were more conservative. While there is no gold standard to compare with, these studies are usually interpreted to support the use of multidisciplinary recommendation groups. Lomas identified three reasons in support of this: firstly, the limited information available for guideline development needs to be supplemented by the interpretations of these stakeholders; secondly, legitimate conflicts over values need to be resolved; and thirdly, the successful introduction of a guideline requires that all key disciplines contribute to its development to ensure "ownership" and support.[There is consistent empirical evidence that the composition of recommendation making groups influence the resulting recommendations. Hutchings and colleagues completed a systematic review of factors affecting judgments produced by formal consensus development methods. They idesupport. Given thThe number of group members and the balance of disciplines should be influenced by the focus of the guidelines. When deciding on the composition of the group, all potential stakeholders should be identified including: health care professionals who are directly involved in the clinical management of patients in different health care settings , policy makers who may need to make decisions about resource utilization and patients. There then has to be made a decision about which categories of participant to involve within the guideline group. Guideline developers often have to weigh the desire for wide representation against the need for a cohesive working group. Small groups may not have sufficient experience within their members; larger groups may lack cohesiveness and be difficult to lead. In general, the optimum size for a small group is thought to be between eight and ten people, although groups of larger size have operated effectively.There is an increasing (though still limited) literature on consumer involvement in health decision making in general. A systematic review of patient and public involvement programs (PPIPs) in developing clinical practice guidelines  looked aThese issues are reflected in the empirical work of Boivin et al.  who, froFrom the review it appears that there are still few analytical empirical accounts of attempts to involve consumers in guideline development, and there are no robust evaluations of the effectiveness of different methods. A systematic review of the effectiveness of consumer involvement in health decision making  only ideWilliamson  has prop• incorporating individual patients in guideline development groups;• a ‘one off’ meeting with patients;• a series of workshops with patients;• incorporating a consumer advocate in guideline development groupsIndividual patients who participated in a guideline development group contributed infrequently and had problems with the use of technical language. Although they contributed most in discussions of patient education, their contributions were not subsequently acted on. Within a ‘one off’ meeting with a group of patients, participants again encountered problems with medical terminology and were most interested in sections on patient education and self management. Their understanding of the use of scientific evidence to derive more cost-effective care practices was unclear. The workshop format was relative resource intensive but made it possible to explain the technical elements of guideline development, enabling patients to engage in the process and make relevant suggestions. A patient advocate who served on a group felt confident to speak and was accustomed to discussions with health professionals and to medical terminology.However, the involvement of lay people in practice guideline development can also be problematic. The North of England experiences and Legare’s review  demonstrSeveral strategies exist for maximizing the benefits of consumer engagement while avoiding the difficulties. For example, as should occur with selecting any member of a practice guideline group, selection criteria can be applied in choosing a consumer representative with the ability to consider the evidence objectively and make recommendations that depart from preconceived views or self interests. The National Institute for Health and Clinical Excellence in the United Kingdom advocates the involvement of at least two consumers on guideline development groups and provides induction sessions that brief consumers on what to expect as participants.Another option is to not seat a consumer representative on the guideline group itself, limiting the latter to professionals, but eliciting the perspective of patients and the public as part of a larger stakeholder input exercise. For example, a group may not include a consumer representative but may invite patients or other laypersons to review draft documents or attend a group meeting to share their perspective. Guidelines groups can host an open forum, in which various stakeholder groups, such as patients, payers, manufacturers, and professional associations, are afforded the opportunity to express their perspectives and criticisms, present scientific evidence of relevance to the guideline, or raise concerns about the impact or implementation of proposed recommendations . The advrientation (defining the problem), evaluation , and control .[A range of psychosocial factors can influence the progress and content of group meetings. Guidelinrevail). Ideal corevail). Dysfunctrevail). MultidisThe risk of such psychosocial biases can be reduced with careful planning and attention to small group processes. For example, it may be appropriate to ensure that guideline development groups receive support to ensure that the group both functions effectively (the group process) and achieves its aims (the group task). It is poAn aspect that has gained increasing prominence with the “industrialisation” of guideline development is the need to consider conflicts of interest. Financial, intellectual and other investments in aspects of health care can result in biased judgement about a topic of interest; a concept that is generally called “conflict of interest”. A number of organizations have defined “conflicts of interest” (COI) Table . Grilli That COI can misinform healthcare decision makers is widely recognized ,31. OmisIntellectual COI is another type of conflict  that is The biases resulting from COI may be conscious or unconscious  and may The most knowledgeable individuals can be conflicted because of their expertise in the area of interest. These “experts” may possess unique insight into appropriate healthcare questions to ask in guidelines. But because of their involvement in research they may have both financial and intellectual COIs whilst also possessing unique insight into aspects of the existing evidence such as study design and decisions made during conduct of studies. Such individuals may be difficult to replace because of the unique insights they provide. The aim  should be to manage the potential COI appropriately ranging from informal consultation and exclusion from a group to participation without influencing recommendations. Approaches to doing this have been described in a recent policy of the American Thoracic Society; an examParticipants in guideline development should ideally submit written disclosures of all potential interests that may cause a conflict of interest and disclosures should be made prior to being involved in an official capacity. The individual can declare in writing: all known current and past interests relevant to the subject and scope of the matter for a number years prior to the date of declaration (many organizations use a period of three years); and any conflicts of interest relevant to the subject and scope of the matter that are expected to occur in the near future. Disclosure can be made through standard forms or a uniform online process that could span across organizations. Updating as individualcircumstances warrant, and attest to its accuracy and currency is important. All participants (committee/group members and/or presenters) should be asked and reminded to consider their own conflicts and conflicts of others during discussions and decision making. Participants should abstain from discussion and voting if they or a sizable proportion of the other participants identify a COI. The chair of the group should be free of COI.Those choosing participants for guideline development should review disclosures before deciding on participants, and consider excluding participants if there is a non-resolvable conflict of interest. The procedures (including step by step review and management) is best clearly described and part of a policy. This policy should involve independent review of COI and decision making by committees. Ideally guideline developers will provide group members with examples of COI management from anonymous cases that describe appropriate methods of COI management.Once the members of a guideline group have been assembled, COI of members should be identified and discussed before beginning deliberations. Individual participants (including project chairs and group members) should label where COI bear on specific recommendations.If groups involve members with (limited) COI, Chairs and group members on a guideline group should ensure that committees are reminded of the specific COI before discussion of individual conclusions or recommendations on which those COI bear. This will allow recusal from recommendations of those with important COI. Group chairs can play an active role and excuse group members from discussions or decision-making on particular recommendations.An organization should develop and oversee the procedures and instruments used to disclose, review and resolve COI, and should advise and assist chairs and organizers where necessary. In instances where determination of COI and actions taken to resolve COI are disputed, procedures should be in place to address the matter.In this first paper we have described target audiences for guidelines, the criteria for choosing topics for guideline development (including acknowledging the current debate about co-morbidities), group composition and group processes and the important procedural issue of managing conflicts of interest. In the next paper we move on to discuss issues of identifying and synthesizing evidence: deciding what type of evidence and outcomes to include in guidelines; integrating values into a guideline; incorporating economic considerations; synthesis, grading, presentation of evidence; and moving from evidence to recommendations.MPE is Editor in Chief of Implementation Science; Jeremy Grimshaw is an Editorial Board member. All decisions on this paper were made by another editor. The authors have all been involved in guideline development for a range of different organizations.All authors contributed to the writing of this article and approved the final draft.


Over 30% of individuals use natural health products (NHPs) for osteoarthritis-related pain. The Deficit Model for the Public Understanding of Science suggests that if individuals are given more information  they will make better health-related decisions. In contrast, the Contextual Model argues that scientific evidence is one of many factors that explain how consumers make health-related decisions. The primary objective was to investigate how the level of scientific evidence supporting the efficacy of NHPs impacts consumer decision-making in the self-selection of NHPs by individuals with osteoarthritis.The means-end chain approach to product evaluation was used to compare laddering interviews with two groups of community-dwelling Canadian seniors who had used NHPs to treat their osteoarthritis. Group 1 (n=13) had used only NHPs (glucosamine and/or chondroitin) with “high” scientific evidence of efficacy. Group 2 (n=12) had used NHPs (methylsulfonylmethane (MSM) and/or bromelain) with little or no scientific evidence supporting efficacy. Content analysis and generation of hierarchical value maps facilitated the identification of similarities and differences between the two groups.The dominant decision-making chains for participants in the two scientific evidence categories were similar. Scientific evidence was an important decision-making factor but not as important as the advice from health care providers, friends and family. Most participants learned about scientific evidence via indirect sources from health care providers and the media.The Contextual Model of the public understanding of science helps to explain why our participants believed scientific evidence is not the most important factor in their decision to use NHPs to help manage their osteoarthritis. I just took my friend’s word for it..If my son says take it, I’ll trust him. .Endorsement of a product by a health care provider was also an important source of information. The types of health care providers consulted  did not significantly differ between the two groups of participants. Trust also appeared to be a key element in participants’ interactions with their health care providers:I trust what he [my family doctor] says to me. I don’t think I’ve ever questioned anything he ever says to me. .Participants who discussed their physicians quoting scientific evidence usually recalled the information from their physicians as being negative or neutral about the efficacy of a NHP:And [the doctor] said, well it’s not a product that really works. The doctor, he told me I could take a placebo pill, it would be the same way […] he didn’t say it was bad [or] don’t take it, it’s going to do negative things. He just said, it has no value. .The advice of health care providers appeared to be playing slightly different roles in the decision making process for participants in the high and low scientific evidence groups. Participants in the high scientific evidence group related the endorsement of a health care provider to both the product’s efficacy (i.e. it decreased their pain) and how it worked (its mechanism of action). In contrast, participants in the low evidence category linked the knowledge that a product was endorsed by a health care provider with the perceived lack of (or fewer) side effects but did not identify a strong link to mechanism of action.Participants in both scientific evidence categories talked about the NHP’s pain reduction ability as being key in allowing participants to perform their activities of daily living, feeling healthier physically, and emotionally, and being able to participate in social activities. Two key linkages mentioned by participants in the high scientific evidence category, but not mentioned by participants in the low scientific evidence category, were the importance of social activity and being in control of one’s health, both consequences which the participants linked with an improvement in quality of life.Overall, two values  appear to be underpinning the decision to use NHPs for participants in both scientific evidence groups. Most participants in both groups described that experiencing less pain ultimately led to improved quality of life. “Quality of life” was defined by the participants as attaining more happiness, meaning, or purpose to live. Similarly, enabling improved activities of daily living led to greater levels of independence in both scientific evidence groups. In addition, more participants in the low scientific evidence group tended to associate the natural source of NHPs with being able to adopt a “natural way of living.” Only one participant in the low scientific evidence group made such a link (which means it is not present in the HVM because it was below the cut-off of three).not what we found. Participants in both groups appeared to have a very general understanding of the scientific method and identified scientific evidence as a factor in their decision-making. Upon detailed questioning in the semi-structured interviews, scientific evidence was not of primary importance to participants in either group. Further, participants who cited scientific evidence as a product attribute frequently associated the scientific evidence supporting an NHP with a reduction in side effects instead of acknowledging the greater probability of benefit of NHPs with higher levels of scientific evidence according to the EBM hierarchies of evidence[According to the Deficit Model participants who were in the high scientific group  should have reported more exposure to, and place more value on, scientific evidence than those purchasing products without scientific evidence of efficacy. This was evidence. This suevidence,13,29.We believe that the Contextual model is helpful at explaining our findings since it recognizes that it is not simply an understanding of scientific facts that it is important when people make decisions, but rather how that information fits into their everyday experiences, concerns and values. The ConOur participants identified scientific evidence as a product attribute impacting their selection of NHPs, which is consistent with previous qualitative study findings where patients with arthritis, breast Similar to other CAM decision-making studies in patients diagnosed with cancer, most of our participants did not report going to the scientific research to read it first-hand, instead, they considered indirect sources such as internet searches, newspaper-articles, or health care providers’ endorsements sources of scientific evidence,15,32,33The MEC approach is based on an assumption that product selection is ultimately driven by attempts to fulfill an end or core value. Three core values emerged in our findings – desiring a natural way of living, quality of life, and independence. The most distinct value which appears to characterize participants who selected products without scientific evidence support was the desire for a “natural way of living”. This is consistent with previous research which suggests that individuals who use CAM appear to value a holistic way of living and the connection between the mind and body,13,15, nA previous study by our team found that in consumers selecting sleep aids, individuals’ product selection process included making different tradeoffs between product attributes  and perceived consequences . These tradeoffs were done in an effort to maximize values such as overall quality of life. Future There are some limitations to this study. For example, we had a relatively small sample size (n =25). Nonetheless our sample  individuals with over half earning more than $50,000) has similar demographics to the target group -- individuals with arthritis who choose to use NHPs or CAM therapies in general,38. ThisIn conclusion, this study found that scientific evidence does play a role in consumer decision-making about NHPs; however, it is less important than the advice provided by health care providers, friends and family. Very few participants used direct scientific evidence sources in the decision-making process; most acquired knowledge of scientific evidence through indirect sources such as health care providers or the media, highlighting the responsibility of these groups to critically assess the quality and validity of scientific evidence since consumers/patients rely on them for accurate information. The Contextual Model of the Public Understanding of Science helps to explain why scientific evidence is not the most important decision-making factor because it helps us understand how perceived personal relevance is used to filter all information relevant to these types of decisions. Future research should focus on the content of the interactions between patients and heath care providers regarding scientific evidence, the perceived importance of different indirect sources of scientific evidence, and knowledge translation strategies to more effectively convey scientific evidence to the lay audience in ways they find personally meaningful.1. Products and therapies generally defined as not being part of the dominant health care system.2. Evidence level A or B only. Level A evidence is strong scientific evidence, where level B is good scientific evidence, both according to the Natural Standard Database.3. Evidence level C or traditional/insufficient evidence. Level C evidence is unclear or conflicting scientific evidence according to the Natural Standard Database.CAM: Complementary and alternative medicine; EBM: Evidence-based medicine; HVM: Hierarchical value map; MEC: Means-end chain; MSM: Methylsulfonylmethane; NHP: Natural health product; OA: Osteoarthritis.The authors have no competing interests.TT and HB conceived of the study and its design. TT and NK conducted the interviews. HB, TT, and NK participated in the analysis of the means end chain data and semi-structured interviews. AB and MK participated in refining the study design, in the means-end chain data-analysis and interpretation of the data. TT drafted the manuscript. All authors reviewed and revised the manuscript and approved the final version.TT, MSc NDResearch Network Coordinator, Canadian Interdisciplinary Network for CAM Research (IN-CAM), Leslie Dan Faculty of Pharmacy, University of Toronto, Ontario, Canada.Clinic Supervisor and Academic Instructor, Canadian College of Naturopathic Medicine, Toronto, Ontario, Canada.HB, BScPhm, PhDProfessor and Associate Dean, Graduate Education, Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, Ontario, Canada.AB, PhDAssociate Professor and Undergraduate Coordinator, Department of Food, Agricultural & Resource Economics, FARE, University of Guelph, Guelph, Ontario, Canada.NK, MEdLeslie Dan Faculty of Pharmacy, University of Toronto, Ontario, CanadaMK, MD MSc, FRCPCDirector of THETA, the F. Norman Hughes Chair in Pharmacoeconomics, Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, Ontario, Canada.Professor in the Faculty of Medicine and Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, Ontario, Canada.Scientist Toronto General Research Institute, Toronto, Ontario, Canada.Adjunct Scientist, Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6882/12/198/prepub


Systematic reviews (SRs) using individual participant data (IPD) are often referred to as the gold standard due to improved precision and reliability. Furthermore, they may identify treatment by participant subgroup interactions, enabling better targeting of treatments. IPD SRs have enormous potential to influence clinical guidelines, however their uptake by guideline developers, as Level 1 evidence, is unclear.We aimed to assess the impact of IPD SRs of interventions on clinical guidelines across a variety of healthcare areas.30 eligible IPD SRs published from 2008 to 2010 spanning clinical areas including cancer, cardiovascular disease, and epilepsy were identified from records of the Cochrane IPD Meta-analysis Methods Group. Clinical guidelines across corresponding healthcare areas, published or revised since 2010, were searched for references to these IPD SRs. Details of recommendations based on the results of these reviews and citations of other relevant SRs were collected.Searches identified 227 potentially relevant clinical guidelines. Preliminary results are based on four IPD SRs in cancer and 34 relevant guidelines. Each of the four IPD SRs was cited in 2 to 5 guidelines, however citations were only identified in 11/34 guidelines (32%). Three of the 14 guidelines also cited aggregate data SRs, with little distinction between results of IPD or aggregate data SRs.Further results for all 30 IPD SRs and 227 guidelines will be presented.Results of this study will help develop guidance regarding how best to use IPD SRs to inform guidelines and thus impact on clinical practice.