The Indian Psychiatric Society (IPS) constituted a task force on clinical practice guidelines (CPGs) in 2004 to formulate guidelines for management of various psychiatric disorders in the Indian setting. Over the next 4 years (2005-2008), the task force published 4 volumes of guidelines covering most of the psychiatric disorders and issues in special populations. However, till now, nothing is known about the usefulness, awareness and their implementation. This was a preliminary survey to know about the usefulness and awareness of the CPGs of the IPS.An email survey was sent to 1100 psychiatrists, of which 107 responded.Only half of the responders were aware about all the 4 volumes of the guidelines and only 12.7% of the responders had read all the four volumes. About two-thirds of the responders had referred to these guidelines in their clinical practice, either occasionally (46.1%), often (16.7%) or always (2%). Similarly, more than two-thirds of the responders considered these guidelines to be helpful in making day-to-day clinical decisions in their practice, either occasionally (48%), often (19.6%) or always (3.9%). In the open-ended questions, many of the responders discussed their dissatisfaction with these guidelines and gave suggestions as to how these guidelines could be improved.There is need for better dissemination of the guidelines and making recommendations that can be applied in an Indian setting. Over the last 50 years, there has been a tremendous advancement in the understanding of efficacy and effectiveness of various treatments in psychiatry. Hence, it is important to incorporate this new knowledge into the daily practice of every psychiatrist. One such approach is development of clinical practice guidelines (CPGs) by various national and international organizations. CPGs are “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.” ClinicalIndian Journal of Psychiatry for their comments. All the comments received were securitized by the Chairman and Convenor of the Task Force and, wherever it was felt necessary, guidelines were modified in consultation with the authors.The Indian Psychiatric Society (IPS) constituted a Task Force on CPGs in 2004 to formulate guidelines for management of various psychiatric disorders in an Indian setting. The Task Force identified psychiatrists from different parts of the country to formulate guidelines on specific conditions. For each guideline, a lead author was identified and was asked to formulate the draft of the guideline on the assigned topic along with the help of other psychiatrist(s) of his choice. Once the drafts were prepared, these were discussed in the Task Force meetings at Jaipur every year. The draft was initially read by a group of 5-9 psychiatrists, who gave their inputs. Later, the revised draft was presented to a group of 50-60 psychiatrists, who gave their comments, which were incorporated. A modified draft was supposed to be put up on the website for 4-6 weeks for everyone to comment; however, this did not happen consistently. The draft of the first volume was sentThe members who participated in various Task Force meetings included psychiatrists from various teaching institutes (both faculty members and trainee psychiatrists), psychiatrists working with non-government organizations (NGOs) and those in private practice. From the years 2005 to 2008, the Task Force published 4 volumes of guidelines, covering most of the psychiatric disorders.–6 These Although these guidelines have been published for the last 4 years, little is known about their usefulness. Hence, we carried out this survey to know the usefulness and awareness about the CPGs of IPS.For this study, a questionnaire consisting of 16 questions was designed to collect information from psychiatrists practicing in India. The questionnaire included some basic questions about the psychiatrist, viz., their age, place of work, position at work, number of years of standing in psychiatry (including the training period) and whether they were involved in the development process of CPGs of IPS in any way . The specific questions about the guidelines enquired their awareness about the existence of guidelines of IPS for various disorders, whether they have received the hard copies of the same, how many volumes they have read, do they refer to the CPGs of IPS in their clinical practice, do they consider these CPGs to be helpful in making day-to-day clinical decisions in their practice, do they use these guidelines as an educational tool and how they rate the content (information/evidence base) of the CPGs of IPS. In addition, 2 questions were asked regarding the other practice guidelines  and how frequently the psychiatrist follows them. Toward the end, in 2 open-ended questions, the psychiatrists were asked to give their views about the CPGs of IPS and give suggestions as to how these can be improved.This questionnaire was sent to 1100 psychiatrists whose email addresses were available, by one of the authors (SG) during the months of October and November 2008. The psychiatrists were informed that it is a survey and it was presumed that the psychiatrists who would respond would indirectly consent for the same. The questionnaire was sent 1-6 times varying on the response and, in total, 107 responses were received, of which 5 responses were of psychiatrists of Indian origin practicing outside India for varying duration and were not at all aware of these guidelines. Hence, their data were not used in the final analysis.The response rate of our survey was poor (9.72%).The mean age of the psychiatrists who responded was 43.3 years. In terms of age group, about 45% of the responders were less than 40 years of age and another 45% were aged between 41 and 60 years. A majority of them were male (79.4%). More than half of the responders were working in academic institutes at various levels , and more than one-third of them (37.3) were into full-time private practice. In terms of position at work, more than 40% were consultants in a private hospital or an NGO and more than one-third were in a faculty position (37.3) in institutes. About 20% of the responders were working as Residents (Senior/Junior). The mean number of years of standing in psychiatry of the responders was 17.58 years. Only one-fourth of the responders were involved in formulation of guidelines, either as authors or as members of various task forces.Only half of the responders were aware about all the 4 volumes of the guidelines. One-fourth of them were aware of 2-3 volumes and 12.7% of the responders were not at all aware of the existence of CPGs of IPS. However, less than one-fourth of the responders had received (22.5%) all the four volumes of the CPGs and slightly more than one-fourth of them (27.5%) had not received any of the four volumes of CPGs.“some guidelines from one or more volumes” and 20% of them had not read any of these guidelines. However, about two-third of the responders had referred to these guidelines for their clinical practice, either occasionally (46.1%), often (16.7%) or always (2%). Similarly, more than two-thirds of the responders considered CPGs of IPS to be helpful in making day-to-day clinical decisions in their practice, either occasionally (48%), often (19.6%) or always (3.9%). Further, more than half of the responders used the CPGs of IPS as an educational tool, either occasionally (29.4 %), often (20.6%) or always (4.9%).Only 12.7% of the responders had read all the four volumes of CPGs. Nearly one-third of the responders had read The most common response was average (39.2%), followed by good (27.5%) and poor (12.7%).Almost all were aware about the existence of guidelines of other societies and these guidelines were often, occasionally and always used by 41.2%, 35.3% and 14.7% of the responders respectively.n = 24) and those who were not involved, it was seen that those who were involved in the formulation of guidelines were more aware of all the volumes  and had more frequently received copies of one or more volumes . There was no significant difference in response to other questions.When we compared the responses of those who were involved in formulation of CPGs . The guidelines are also too ideal to be applicable in the real world - they are applicable to only a miniscule proportion of patients, because of myriad practical problems. The guidelines should be made in hierarchical fashion, to be applicable for primary, secondary and tertiary care centers, but there is no evidence base for this.” Another responder commented “They have nothing new to say. One gets an impression that they have been written taking few things from various guidelines. Guidelines have to be locally relevant. They are like Indian adaptations of international guidelines. The real problem is that there is no good data from controlled studies involving large samples of Indian patients. Therefore, guidelines end up saying what other guidelines recommend. If you have nothing new to say, there is no point in wasting time to develop guidelines.” Still another responder wrote “The guidelines appear to be put together based on those from the West. Nothing local or Indian about them and not based on any Indian evidence. The guidelines appear to be put together by members who may not have had practical experience with the specialty area being written about. There is no legal or societal mandate to follow these guidelines.” A responder suggested that “there should be clinical trials in the different parts of the country in different disorders with strict methodology.”There were many themes with regard to dissatisfaction. First, many of the responders were dissatisfied with the guidelines because the guidelines did not include Indian data or did not address to the situations that are faced specifically in Indian situations. However, some of the responders also acknowledged that there is lack of Indian data on some of the issues. One of the responders expressed that “lengthy,” “overinclusive,” “should be more practical,” “not private practice based,” “need to be more concise, focusing more on local cultural issues and day to day problems in patient management,” “simplify and to make it point by point presentation such that the practitioners, postgraduates and clinicians can benefit by them in their day to day practice,” “more user friendly format - like the algorithm for treatment of depression” and “quality of printing and typographical error in the document are the weak points.” The third common theme that emerged was that these guidelines were not uniform. One of the responders wrote that “most IPS CPGs are not clinical practice guidelines in the true sense” and another commented that they “should be standardized on the lines of APA and NICE guidelines.” Further, one of the responder wrote that “It's just a compilation of the literature on related topics, sometime not even providing any guideline, leaving the reader in a chaos.” The fourth common cause of dissatisfaction was that the guidelines did not take all the available evidence into consideration while formulation. One of the responder wrote that “evidence base has to be improved.” Further, one of the responders expressed dissatisfaction on the way the guidelines were formulated and wrote that “guidelines preparation are very specialized bits of work and ours do not follow the international norms for guideline preparation.”A second common theme of dissatisfaction that emerged was that these guidelines were not user-friendly. Some of the responders wrote that these guidelines are Again, some of the common themes emerged as to how the CPGs of IPS can be improved. The responses covered various areas like how to generate data and go about the formulation of guidelines, who should be part of the formulation of guidelines, how the guidelines should be presented and how to improve their circulation and utility.“Guidelines should be made locally applicable and useable from local evidence. Task force  in each area should be created for framing guidelines. The guidelines so framed must be circulated for consensus and applicability among larger body of professionals working in the specific area. If possible the guidelines should be field tested to see the usefulness and applicability in routine clinical practice.” Another responder expressed that the IPS can “take a detailed survey with questionnaires with leading practioners and try to find what guidelines each has developed on his/her experience which has shown good result in their clinical practice, then try to match with the theoretical academic model to find the truth why they are more useful.” However, one of the psychiatrists observed that “rather than formulating guidelines for everything, IPS should focus on few areas - there is little India-centered research to inform guidelines, also there are few experts who work exclusively in specific disorders, so the IPS should reconsider its approach to development of CPGs. It would be helpful if so many CPGs are not developed at one time, rather CPGs should be developed in fields that are mature enough for their development in India. Most western associations do not have guidelines for all disorders. May be we should stop aping the West and the IPS should think in terms of broader groupings like severe mental illness and common mental disorders etc. (to be able to gather enough genuine experts) or start by forming taskforces that can formulate directions for relevant research before coming to the issue of CPGs.”Regarding how to generate data and go about the formulation of guidelines, some of the responders expressed that there is need to conduct national-level studies with sound methdology before formulating the guidelines, whereas others expressed that expert consensus guidelines can be formulated and, if required, can be field tested before they are adopted. One of the responder opined that “only professors with established reputation and who have done commendable work in that particular area should be asked to prepare guidelines.”With respect to “who should be involved in the formulation,” a common theme emerged that the Task Force should include more academicians, practitioners and advocacy groups and should formulate guidelines on the basis of expert opinion. However, some of the psychiatrists were of the view that “I think that more attention should be given to eliminate printing mistakes, a more visually pleasing formatting and a better quality of the hard copies should be made available. One has to look at Western guidelines to see the superior looks and quality they have.”In terms of presentation, as discussed in the earlier section, one of the common opinions was that the guidelines must be more uniform. Other common themes were the printing mistakes. One of the psychiatrists wrote Another important aspect was the circulation of guidelines. Many psychiatrists expressed that the IPS should mail the hard copies to all in time and the guidelines must be put on the website, from where they can be easily accessed and the readers can also express their views on the website. Some of the psychiatrists expressed that the IPS should discuss the guidelines in conferences and continuing medication education programmes.Last but not the least, many psychiatrists expressed that this good endeavor of the IPS should continue in formulating the guidelines and the guidelines must be updated regularly.This is the first survey that assessed the views of Indian psychiatrists about the guidelines published by the IPS. The survey was emailed by one of the authors (SG), who has been a coauthor in 3 of the IPS guidelines but did not hold any office of the IPS. The survey was deliberately not sent by the other author of this article (AA), Convenor of the Task Force for formulation of guidelines, as it was considered that if his identity was disclosed as a member of the survey team, it may influence the responses.et al,[Although the survey was not sent to all the IPS members, it covered about one-third membership of the society. However, the response rate was poor compared with one of the previous surveys conducted on the practice of electroconvulsive therapy in India. However,et al, which evThe relatively poor response rate in the current survey can be due to various reasons. First, the survey was sent by one of the psychiatrist who was not very well known among the senior psychiatrists. Another reason for the poor response could be that many of the psychiatrists are themselves not computer savvy, the way the survey was performed  and lack of understanding in general about the importance and purpose of such a survey.Although the response rate was poor, there was a fair representation of psychiatrists of different age groups, place and position of work and number of years in psychiatry. Only one-fourth of the responders were involved in the formulation of guidelines, either as authors or as members of various Task Forces. Further, there was no statistically significant difference between those involved and those not involved in formulation on all the multiple choice questions except for being aware about more number of volumes and receiving more number of volumes. This suggests that the responses were not influenced by the participation in the formulation of the guidelines.Only 51% of the responders were aware about all the 4 volumes and only 22.5% of the responders had received these. This suggests that although IPS has taken an initiative in the formulation of these guidelines, they were not being disseminated properly. Hence, the IPS should make efforts to increase the awareness about the presence of such guidelines and they should be circulated to the membership. As suggested by some of the responders, the guidelines can be put on the IPS website and the guidelines can be discussed in some of the focused CME programmes to improve the dissemination of the guidelines.With respect to the use of the IPS guidelines, as is evident from the responses, at least two-thirds of the responders used them at least occasionally in their clinical practice and considered these to be helpful at least occasionally. Similarly, more than half of the responders used these guidelines at least occasionally as an educational tool, while two-thirds of the psychiatrists rated these guidelines as average or good. Although these responses may not appear great, we think that it reflects that at least some of the psychiatrists are interested in having Indian CPGs and are willing to follow them. In the survey of psychiatrists regarding the extent to which guidelines are used in the treatment of bipolar disorder in the West, 64.1% of the responders reported making routine use of treatment guidelines while taking clinical decisions. On compaStudies carried out on guidelines for various physical and psychiatric disorders in the West have shown that guidelines that are easy to understand, can easily be tried out and do not require specific resources have a greater chance of being used.–14 BasedHowever, numerous studies have demonstrated that despite the publication and dissemination of a large number of guidelines, they always do not lead to a meaningful change in day-to-day clinical practice.–17 This The views to the open-ended questions of this survey can be a good starting point from where we can reflect and plan ahead. In future, the IPS can take up some of the suggestions while formulating the revision of guidelines. According to our view, at least the following suggestions should be incorporated:Formulate guidelines on only 1 topic at a time.A uniform pattern must be followed in writing and compiling the evidence for all the guidelines.The expert groups should be bigger.Besides the academicians and practitioners, the expert group should comprise of subjects from advocacy groups and representatives of patients.The initial draft of guidelines can be presented in the zonal- and state-level conferences by members of the expert group to seek the opinion of various members.Carry out field trials at least in 1-2 centers before recommending a particular prescription. If not feasible, an expert consensus should be made, which can be arrived at by sending questionnaires to randomly selected members.The recommendations based on data available from other countries can be graded based on the level of evidence.The recommendations about the non-pharmacological management can be graded as to what minimum can be done when specific therapies cannot be carried out.Issues specific to the Indian context, on the basis of expert consensus can be incorporated.Usefulness of traditional methods like yoga and pranayam in the management of various psychiatric disorders can be arrived at by carrying out surveys and the findings can be incorporated in the guidelines.Proper dissemination of guidelines for soliciting suggestions before accepting the document.Indian Journal of Psychiatry.Proper dissemination of guidelines after publishing the same. It can be published as a supplement to the Guidelines should be updated periodically.The guidelines should be evaluated on an AGREE instrument.It is important to consider the limitations of this survey. It included the opinion of only 102 psychiatrists. Considering that there are about 3500 psychiatrists in India, this survey does not reflect the overall opinion of all the psychiatrists in India. In future, a larger survey should be conducted covering a greater number of members of the society.


Governments increasingly require policy documents to be evidence-based. This paper analyses the use of scientific evidence in such documents by reviewing reports from government-appointed committees in Norway to assess the committees' handling of questions of effect.This study uses the 'Index of Scientific Quality' (ISQ) to analyse all Norwegian official reports (NOUs) that were: (1) published by the Norwegian Ministry of Health and Care Services during 1994-1998 (N = 20); and (2) concerned with questions of effect either because these were included in the mandate or as a result of the committee's interpretation of the mandate. The ISQ is based on scientific criteria common in all research concerning questions of effect. The primary outcome measure is an ISQ score on a five-point scale.Three reports were excluded because their mandates, or the committees' interpretations of them, did not address questions of effect. For the remaining 17 NOUs in our study, overall ISQ scores were low for systematic literature search and for explicit validation of research. Two reports had an average score of three or higher, while scores for five other reports were not far behind. How committees assessed the relevant factors was often unclear.The reports' evaluations of health evidence in relation to questions of effect lacked transparency and, overall, showed little use of systematic processes. A systematic, explicit and transparent approach, following the standards laid down in the ISQ, may help generate the evidence-based decision-making that Norway, the UK, the EU and the WHO desire and seek. However, policy-makers may find the ISQ criteria for assessing the scientific quality of a report too narrow to adequately inform policy-making. Nearly 30 years ago, Lindblom and Cohen stated: '... in public policy making, many suppliers and users of social research are dissatisfied, the former because they are not listened to, the latter because they do not hear much they want to listen to' . Twenty A call for evidence-based policy-making is also a call for the use of scientific methods in data collection and in the validation of information. Whenever policies are discussed, the question is often simply, "What will work?" Accordingly, when making priorities and formulating healthcare policies, a key factor will be evidence of their effects. A transparent and explicit approach is essential for reliable, valid and rational decision-making.In interview studies, policy-makers state that three factors will facilitate their use of scientific evidence in policy decisions: high quality research; the inclusion of data on effectiveness; and summaries with clear recommendations ,15-18. IPrevious studies of health policy documents commissioned in the UK and elsewhere and by the WHO have investigated the consistency of suggested policies with evidence gathered in the field ,19-23. SIn 2002, the Norwegian government stated that it wished to adopt the evidence-based approach to policy-making. In NorwAll NOUs published by the Norwegian Department of Health and Care Services during 1994-1998 were collected and those that reported evidence of health effects were included in this study. Of the 20 NOUs published during the study period, three were excluded either because the mandates did not address questions of effect or because the committees did not interpret the mandates as doing so, resulting in a total of 17 analysed NOUs. A full list of titles appears in Table Each NOU included within the study was circulated between four researchers (see the acknowledgements at the end of the paper) until general agreement was reached regarding the relevant questions of effect in the mandate or the committee's interpretation of the mandate. The same researchers then worked in pairs to score all the reports using a predefined set of evaluation questions. Group discussions were then held to resolve opposing views. We also analysed the composition of each committee, i.e., the relative importance of the roles of researchers, policy-makers and representatives from interest organisations.The predefined questionnaire for evaluating the NOUs consisted of four main components table .The first component, concerning the mandate's description of the task, was included to clarify whether the committee was supposed to evaluate evidence of effect. The purpose of the second component, concerning the strategy used by the committee to gather research information, was to ascertain whether the intentions of the committee were research-based, how the committee proceeded to gather information, and the sources of its data.The third component concerned the use of evidence in the report's recommendations and summary. Research suggests that policy-makers often rely heavily on these aspects of reports, making an assessment of their scientific quality particularly important . The speThe fourth component concerned the quality of the scientific evidence. To assess the use of scientific evidence in the reports, we applied the Index of Scientific Quality (ISQ) ,26. The The index is based on a five-point scale, with a score of 5 representing the highest level of scientific quality. According to the ISQ criteria for documentation (question B), clear references to the evidence will score 4 or 5, while partly unclear or definitely unclear references to evidence will score 2 or 3, depending upon the degree of uncertainty. When references to evidence are potentially misleading, the ISQ score will be 1 or 2. Table Previously the ISQ has mainly been used by studies evaluating health reporting in newspapers. We know of no other studies that have used the ISQ to evaluate governmental reports. A few reservations are relevant regarding the use of the index in this context. Firstly, public policy formation is clearly much more complicated than the discussion here might suggest. In addition to scientific evidence, policy-makers need to weigh up different factors such as the values of stakeholders, budgets, cost-effectiveness, broader perspectives, political decisions and aims, other, perhaps more relevant, evidence than the existing evidence of effect, prudent practices, and commonsense understandings that are obvious, but not supported by research evidence. As a consequence, evidence of effect will only form one aspect of the decision-making process when evidence-based policies are made ,27. DiscSecondly, questions may be raised regarding the sensibility of the ISQ. A comprehensive sensibility analysis of the ISQ ,26 foundThirdly, a more contextual evaluation of the variations in scores, or treating each NOU as a separate case, could give important insights into the nature of the variations. An example of this problem arises in connection with one of the NOUs about the Norwegian cancer programme, where narrow questions of effect can only make a minor contribution to a broad mandate.Finally, the analysed reports are not the most recent ones, so they may not be representative for recent NOUs. And a move to promote evidence-based policy making in Norway appears to have been made in January 2004, with the formation of an institution called The Norwegian Knowledge Centre for the Health Services (NKCHS). The Centre is organised and funded by the Norwegian Directorate of Health. NKCHS is meant to be scientifically and professionally independent. The Centre has two overarching tasks: The first is to promote evidence-based policy making, by producing health technology assessment reports, systematic reviews, overviews and reports that provide policy makers with early warnings. The second task is to support health services at all levels to incorporate evidence into their practices.The NOUs published during 1994-1998 seem to be representative of the 19 NOUs published from 1999 to the present. There have been no significant changes in the formal framework for preparing these reports, although some minor revisions were implemented in 2005. Only a shallow investigation of the 19 NOUs published since 1999 is necessary to confirm that when committees interpret their mandates and design their reports, they do not utilize a systematic approach that is similar to the ISQ or familiar evidence-based approaches. Generally speaking it is still the government-appointed committee that determines the methods and procedures used to produce the report. Only one of the NOUs published after 1998 reported that the committee preparing the report asked the NKCHS for a review of the relevant research. The NKCHS responded by stating that such a report would take the Centre from 6 to 24 months to produce. The committee deemed this to be beyond the framework provided to it to complete its work.The themes of the reports varied from broad policy issues, such as the setting of healthcare priorities, to narrower issues such as the health consequences of particular stimuli . Some reports spanned a number of topics, such as those covering health and social services among the Sami population or public health services in local communities.Some mandates specified objectives relating to the organisation of care, e.g., 'How should care be organised to give the best medical and social outcomes for patients with multiple sclerosis?' Other mandates posed more concrete questions: 'Will dramatic experiences during war cause long-term psychiatric problems in soldiers?' 'What is the effect of directly observed treatment for patients suffering from tuberculosis?' The mandates were short, generally a half to one page in length. Most mandates required the committee to consider the following issues: need (15); the effectiveness of relevant interventions (16); and economic implications (15). None of the mandates specified the type or quality of data required. Six of the 17 mandates analysed required a discussion of relevant values, such as the preferences of patients, public considerations, or juridical or ethical consequences.Some committees consisted mainly of experts in the relevant field, while other committees consulted external experts and included their recommendations as appendices to their reports. Expert participation on the committees did not appear to influence the strategy used for information gathering. While all the reports but one referred to reliance upon research, none of the reports described the procedures or literature searches used to gather research information on the harm or efficacy of relevant interventions. Although one report (on electromagnetic fields) stated that it included all the relevant literature, it provided no information about how the literature search had been conducted. Another report (on complementary medicine) stated that various organisations had been asked to provide relevant publications, but did not state the type of literature requested.Component 3 in table The 17 NOUs were evaluated through the application of seven criteria to assess the quality of scientific evidence. This produced 119 evaluations distributed on a five-point scale. The scores for all 119 evaluations appear in Figure In general, evidence of harm and evidence of effectiveness were presented in combination with information about preferences and problems concerning access to, and availability of, health services. Fifty of the 119 evaluations (42 per cent) resulted in scores ranging between 3 and 5. Only 15 evaluations (13 per cent) resulted in scores exceeding 4.0. Sixty-nine of the 119 evaluations (58 per cent) only scored 1 to 2. In short, these results suggest that more than half of the evaluations identified insufficient descriptions and assessments of the quality of evidence of effect.Figure the specific population for whom the report was relevant. Question A produced an average score of 3.1, the only average score to exceed 3.0. While most reports were partly unclear about the population for whom they were relevant, many also provided important and relevant information. Question B asked whether the presented evidence relied on research and whether the reports referred properly to the research. The average score here was 2.3. Only two reports specified references for the most important information.Question A asked whether the report made clear the assessment of study quality. Only two reports provided details of study design and only one described how the committee had assessed the quality of the evidence. The average score here was 1.5. Question D concerned the reporting of the size of the effect and produced an average score of 2.1. Question E asked about the precision with which the effect had been measured and produced an average score of 1.6. Many reports described interventions as 'effective', but none reported the magnitude of the most significant effects. Questions C and E produced the lowest scores, both with an average slightly exceeding 1.5.Question C concerned consistency between studies). The average score for question F was 2.2. We identified discussions concerning more than one study in five reports. These were awarded scores of 3. The other 11 reports scored only 1 or 2 as they provided either potentially misleading information or no information at all about consistency between studies. As a general comment, committees that, for example, only identify a single research study relevant to their report should at least discuss whether this is the result of a low-quality literature search. In cases where there is only one study dealing with a given question of effect, an evidence-based evaluation should include an assessment of the robustness of the evidence.Only one report attained a score of 5 for its consideration of how the evidence presented related to other findings in the field  were above or just under 3.0. These reports generally provided important scientific information regarding questions of effect. Only one report had an average score exceeding 4.0, while one other report scored above 3.0. There was a wide range of average scores, since while one report failed to describe the effectiveness of the services suggested and therefore scored 1 on all questions, another received high scores all round.Systematic reviews of the outcomes of randomised controlled trials have probably been the key element in the development of evidence-based medicine. Systematic reviews assess search strategies and carry out critical appraisals. The ISQ is based upon the same principles as those underlying systematic reviews. Ministries, the government, and parliament rely on NOUs to provide the most comprehensive advice on subjects of national importance. Our analysis suggests that more than half of NOUs dealing with healthcare contain inadequate descriptions and assessments of the quality of the evidence of effect. We are not suggesting that NOUs provide the public with low-quality reports, merely that their scientific quality regarding questions of effect is low.Our study suggests that committees prioritise policy relevance, as the NOUs included in the study primarily focused on policy relevance and policy implications. In both respects we found the reports' use of scientific evidence to be generally mediocre. We identified a lack of clarity regarding both the reports' primary intended audience and the identification and assessment of the main consequences of the study. There was insufficient treatment of validity and precision in relation to hard science questions. This may arise from a belief among the authors of reports that policy-makers are uninterested in the science underlying committee recommendations. Committees may also not consider their time and resources best spent teaching policy-makers about scientific standards.It is impossible to be sure why committees give lower priority to hard science questions, since we have no data from interviews with committee members. We do know, however, from a systematic review of interview studies concerning health policy-makers' perceptions of their use of evidence, that policy-makers want summaries that make clear recommendations . The comThe fact that seven reports obtained an average ISQ score of below 2.0 suggests that committees give insufficient attention to the assessment of scientific evidence. In other words, policy-makers must look elsewhere for transparent descriptions of search strategies for literature, references to evidence gathered in the field, and assessments of the evidence of effect. Our raw data indicates that questions of effect escaped the committees' attention in three of the seven reports, making their ISQ scores of 1 or 2 hardly surprising. We have no information about why some of the committees largely avoided questions of effect.The mandates typically posed broad questions, rather than clear-cut questions of effect. Although NOUs were only included in our study if questions of effect were relevant to their mandates, these questions of effect were sometimes rather implicit and were always connected to broader issues and to a broader mandate. Since NOUs are intended to deal with questions of national importance, their mandates will naturally be broad. It is easy to imagine the broad mandate for the NOU report 'Care and knowledge: the Norwegian Cancer Programme', which deals with a variety of topics, such as: definitions of cancer; cancer as a national problem; the social consequences of cancer; care structures and processes for cancer sufferers; and the administrative and organisational consequences. Within the time span allotted for producing a report of this nature, it is very difficult for a committee to carry out a comprehensive review of questions of effect.When reviewing cancer treatments, it is also of course difficult to decide where to focus. Even if there is a decision to focus on breast cancer , it will still be necessary to select from the many different questions of effect in the treatment of breast cancer. Even a narrow focus on 10 different questions of effect with a protocol in the Cochrane library will still exclude more than 50 other protocol questions in the same database . Let us Politicians, scientists, bureaucrats, experts and lay people often have (quite legitimately) differing interests. If public governmental reports refer to evidence in an explicit and transparent way, readers will largely be able to make reliable judgments by comparing the pros and cons apparent from the evidence. Explicit and transparent references to supporting evidence will assist in the identification of bias rooted in special interests in the recommendations of governmental reports.This study shows that NOUs, in relation to their evaluation of health effects, lack transparency and, overall, show few signs of the use of systematic processes as far as scientific quality is concerned. A systematic, explicit and transparent approach - in fact, an approach that would generate a high ISQ score - may encourage the development of the evidence-based decision-making that Norway and other nations and international institutions desire and seek.Some people might object that applying the ISQ criteria would overly narrow the focus of reports and cause the omission of other kinds of information more relevant for policy-makers: instead of promoting democratic debate, application of the ISQ criteria might lead to meritocracy. However, such objections would be convincing only if reports scoring highly on the ISQ criteria were inherently difficult for politicians, bureaucrats, experts and lay people to understand. Instead of rejecting out of hand the possibility of writing understandable reports of high scientific quality, committees should focus on writing reports that are accessible to academics and non-academics alike. Committees that include experts as well as non-experts should make particular efforts in this regard. One possible solution would be to prepare separate reports for different audiences. The process of doing so might help clarify the role of scientists in such committees and protect researchers from having their results abused ,43.Research is a cumulative enterprise. Whether the ISQ is useful for policy-making, and whether a narrow focus on questions of effect helps policy-makers make wise decisions remains to be seen. The ISQ cannot and should not exclude politics from evidence-based policy-making. But if calls by governments for evidence-based policy-making are to be taken seriously, we need to develop and test evidence-based approaches such as those endorsed by the ISQ so that evidence-based policy may progress from aspiration to reality.EU: European Union; ISQ: Index of Scientific Quality; NKCHS: The Norwegian Knowledge Centre for the Health Services; NOU: Norwegian Official Reports (Norges Offentlige Utredninger); WHO: World Health Organisation.The author declares that he has no competing interests.The pre-publication history for this paper can be accessed here:


Healthcare decisionmaking is a complex process relying on disparate types of evidence and value judgments. Our objectives for this study were to develop a practical framework to facilitate decisionmaking in terms of supporting the deliberative process, providing access to evidence, and enhancing the communication of decisions.Extensive analyses of the literature and of documented decisionmaking processes around the globe were performed to explore what steps are currently used to make decisions with respect to context (from evidence generation to communication of decision) and thought process . Needs and methodologies available to support decisionmaking were identified to lay the groundwork for the EVIDEM framework.A framework was developed consisting of seven modules that can evolve over the life cycle of a healthcare intervention. Components of decision that could be quantified, i.e., intrinsic value of a healthcare intervention and quality of evidence available, were organized into matrices. A multicriteria decision analysis (MCDA) Value Matrix (VM) was developed to include the 15 quantifiable components that are currently considered in decisionmaking. A methodology to synthesize the evidence needed for each component of the VM was developed including electronic access to full text source documents. A Quality Matrix was designed to quantify three criteria of quality for the 12 types of evidence usually required by decisionmakers. An integrated system was developed to optimize data analysis, synthesis and validation by experts, compatible with a collaborative structure.The EVIDEM framework promotes transparent and efficient healthcare decisionmaking through systematic assessment and dissemination of the evidence and values on which decisions are based. It provides a collaborative framework that could connect all stakeholders and serve the healthcare community at local, national and international levels by allowing sharing of data, resources and values. Validation and further development is needed to explore the full potential of this approach. The objective of any healthcare intervention is to improve health; preventive measures, non-pharmacological and pharmacological treatments, and medical procedures are among the numerous available options. Decisionmaking in healthcare is a complex process taking place along a continuum that moves from evidence generation to deliberation on each particular intervention and communication of the resultant decision.Evidence-based medicine and evidence-informed health policymaking rely on evidence generated by developers of healthcare interventions, at least in the initial stages of the life cycle of an intervention. Evidence quantity, quality, usability and accessibility have been identified as hindrances to informed policymaking, highlighFrequent controversy surrounding drug coverage variation across jurisdictions with similar levels of economic development, values and political systems -7 highliSuch processes should fulfill two main functions. Firstly, they should support the complex deliberative process that requires simultaneous consideration of multiple factors such as clinical benefit, level ofSecondly, such processes should help legitimize the decision by ensuring that conditions for 'accountability for reasonableness' (A4R) are met by structuring the deliberative process to make rationale and principles on which decisions are based explicit and ultimately publicly available. Within the A4R framework, availability to public scrutiny is a necessary prerequisite to legitimizing decisions. As suggeSeveral approaches have been published for making healthcare coverage decisionmaking more consistent, rational and transparent. -19 For eIn this context, we hypothesized that healthcare decisionmaking could be facilitated by structuring access, consideration and communication of the evidence and the value judgments on which it is based. The objective of this study was to develop a practical framework to facilitate decisionmaking by supporting the deliberative process, permitting access to relevant evidence, and enhancing effective communication of decisions.Extensive analyses of the literature and of current decisionmaking processes were performed to identify steps leading to decisions, as well as the components of the thought processes underlying decisions. Needs and methodologies available to support such processes were identified to lay the groundwork on which to build the EVIDEM framework.Review of decisionmaking processes in jurisdictions worldwide ,24-44 wa∘ systematic and explicit consideration of all key elements of decision during the deliberative process;,18∘ each committee member's perspective needs to be captured and values shared in the committee;,45∘ relevant evidence in a digested, unbiased and systematic format;,17∘ data on quality of evidence in a structured system for all types of evidence considered; and∘ transparent, understandable, and acceptable communication of decision.,41These needs were all considered in developing the framework. Analysis of the literature revealed that decisionmaking can broadly be subdivided into scientific judgment and value judgment.,3Scientific judgment relies on globally accepted standards defining the quality of evidence. Such technical judgment can be applied using a system in which the elements of quality are explicitly identified and quantified (scored). Scientific technical judgments are not highly dependent of the evaluator  and can be standardized. A number of quality standards, country specific guidelines, checklists and instruments are available to assess the quality of various types of evidence . MCDA structures the deliberative process by breaking down a problem into the components expected to impact the value of an option, and by quantifying them using a scale with defined anchors.,68 MCDA Review of decisionmaking processes revealed that not all value components usually considered in decisionmaking are readily quantifiable.-44,61-67To facilitate value judgments related to a healthcare intervention, following needs were identified:∘ disentangle intrinsic and extrinsic value components;,76∘ develop a simple and rigorous system that applies MCDA from a pragmatic standpoint based on actual thought processes;∘ provide practical access to the evidence on which value judgments are based; and∘ provide a practical method for decisionmakers to provide feedback to data producers and all other stakeholders.Thus identified, these needs were used to develop the EVIDEM framework, processes and tools.A practical framework was developed structuring and making more shareable what is currently being done around the world. It was based on three main principles:∘ Support deliberative process by disentangling and quantifying when possible scientific judgment  and value judgment ;∘ Facilitate access to relevant evidence over the life cycle of a healthcare intervention using a collaborative structure; and∘ Enhance communication of decisions using transparent tools.The framework structures the context of decisions for a healthcare intervention in a given setting into seven modules Figure . The cenA MCDA Value Matrix (VM) was developed to include the value components usually considered in policy decisionmaking. MCDA was selected as a methodological model for the VM for its versatility, transparency and ease of application by a wide range of stakeholders. Value components that can not be readily incorporated into a matrix were not included and were listed as extrinsic components for consideration at the jurisdictional level or on a case-by-case basis  (module 6).The VM (module 5) was designed to address the key question: What is the value of a healthcare intervention with respect to its intrinsic characteristics? In other words, what does it bring to the health of society (for the jurisdiction being considered)? Such a question involves probing the value system of decisionmakers (weights) and assessing the healthcare intervention based on evidence available using defined scales (scores). The value estimate of an intervention is the combination of weights and scores.Components of decisionmaking identified in the analysis of current decisionmaking processes were specifically defined and structured to fulfill MCDA methodological requirements. These ar• Completeness: all currently-understood components defining the intrinsic value of an intervention are included;• Non-redundancy: all components are necessary, important and there are no duplicates;• Mutual independence: scoring of each component is independent from scoring of all other components ; and• Operationality: each component is defined unambiguously; data on which to base the evaluation is available; the numerical scale follows a shared sense of direction.Fifteen components were thus defined and grouped into four clusters; scoring directions were defined from a societal perspective Figure 68]. The. The68].The disease impact cluster was broken down into two components: disease severity (D1) and size of affected population (D2). It was assumed that an intervention for a very severe disease has more value than an intervention for a mild disease (D1) and that an intervention that benefits a large number of patients has more value than one that benefits a small number of patients (D2).The intervention cluster was broken down into seven components. The first (I1) explores the impact of clinical guidelines. Clinical guidelines serve multiple functions for numerous groups and have become ubiquitous. They canThe concept of improvement of medical service, used by the Commission de la Transparence in France, was usedThe economics cluster was broken down into three components to explore the impact of covering a new intervention on health plan budgets (E1), on other spending (E3), and its cost-effectiveness (E2). To ensure non-redundancy and to be in line with standard budget impact modeling practices, components E1 and E3 respectively were defined to cover financial impact of intervention only  and all other economic impacts . The latter is usually explored in economic evaluations, which are made more useful to decisionmakers by reporting disaggregated cost-consequence information. The econThe VM was then designed to be self-contained, with an emphasis on practicality Figure . It contweighting scale (1 to 5) to capture the value system of each evaluator independent of the healthcare intervention under scrutiny; standard deviation of weights for each VM component (Wx) among a group of evaluators can be used to support discussion among evaluators;• A Synthesized evidence for the healthcare intervention under scrutiny prepared using a standardized methodology to minimize bias (see below);• scoring scale (0 to 3) with defined anchors and scoring guidelines; it includes four scoring options to stimulate thought processes and avoid loss of information with a middle score, and zero to allow for exclusion of a component that does not bring any value ; standard deviation of scores for each VM component (Sx) can further stimulate deliberative process among evaluators;• A comments section for decisionmakers to provide feedback to the producers of evidence; includes a prompt to indicate whether low score is due to data limitation, providing a way to capture and communicate data needs;• A simple MCDA linear model to capture a value estimate (V) of the intervention for each evaluator:• A Wherex is the weight for each VM componentWx is the score for each VM componentSn is the sum of all weights ∑Wx is the value contribution for each VM componentVAn example of how a VM component is assessed is shown in Figure The value estimate of a healthcare intervention obtained from an individual or from a group of evaluators is reported in the VM Comparative Scale as a percentage of maximum score allowing for comparison across healthcare interventions Figure . InterprAccess to high level synthesized evidence is necessary to focus the thought process on key elements of decision but should be complemented by easy access to full text sources for those who want to access more details.To ensure minimally biased evidence is available to stakeholders, a methodology was developed to synthesize this evidence for each component of the VM (module 3). The principal objective was to provide the information necessary and sufficient to score each component with access as needed to full text sources. A template with instructions was developed for each component of the VM indicating where and how to find evidence , what to report and how . For full traceability, electronic links to full text sources were integrated into module 2.For example, to assess "disease severity", data to be identified and reported included disease acuteness, morbidity  and mortality, as well as disease stages or subtypes that differentiate therapies and target populations. Besides extracting study results, key elements used to define their validity are also reported, such as, number of patients included in pivotal trials, follow-up duration for safety data, key model features for economic evaluations and sources used for budget impact projections. For the quality of evidence cluster, quality scores for each type of evidence are provided by criterion of quality assessment , providing decisionmakers with structured access to results and rationale of quality assessment for each type of evidence.The QM was designed to quantify the quality of evidence generated for a healthcare intervention; it is grounded in current evidentiary requirements of healthcare decisionmaking bodies and derives from numerous existing tools and instruments to assess quality of evidence. The QM streamlines quality assessment of all types of evidence, disentangles criteria of quality, and provides access to a rationale for each score attributed via a deliberative process. The QM was designed with an emphasis on practicality and includes Figure :• Three criteria of quality assessment (columns);• 12 types of evidence currently required (rows);• For each cell of the QM:∘ Questions or instruments based on global standards∘ Prompt for evaluator to provide rationale for score∘ Scoring scalesFive elements defining quality were identified and clustered in three criteria Figure :Q1 Adherence to the requirements established by the decisionmaking body to which evidence is submitted;• Q2: Completeness of reporting, as prescribed by reporting guidelines, and consistency with cited sources and throughout the document; this criterion can be applied to individual studies or to a high-level document  that includes several studies;• Q3: Relevance of evidence to the decisionmaking body and validity of evidence, with respect to scientific standards and methodological guidelines in applicable fields of research.• Selection of the types of evidence  for inclusion in the QM was based on an analysis of current evidentiary requirements of over 20 decisionmaking bodies worldwide ,42-44,76Evidence concerning the disease and its management was broken down into three types: disease description (#1), current treatment patterns including practices and guidelines (#2), and impact of new intervention on therapy (#3). Epidemiology data included standard metrics and risk factors (#4). Information on the new intervention was broken down into four types: characteristics of intervention (#5), efficacy and safety data obtained from clinical trials (#6), patient reported outcomes (PRO) data (#7) and effectiveness data from trials and registries (#8) For the last type of evidence, identification of effectiveness used the criteria defined by the US Agency for Healthcare Research and Quality (AHRQ). Data on For each type of evidence contained in the QM, instructions, questions, and for the most complex types of evidence, specific instruments were developed. They were derived from current tools  to streamline scoring processes across types of evidence, distinguishing criteria of quality  while keeping the whole system practical. For example, for type of evidence "Economic evaluation", two 11-dimension instruments were developed: 1) an instrument to assess the completeness and consistency of reporting of the study; and 2) an instrument to assess the relevance and validity of study design and results Figure .A scoring scale with defined anchors was developed and full transparency requires that each score be justified by the investigator. Rationale and scores are reviewed by another investigator and validated by experts through deliberative process until consensus is reached. Comments, rationale and scores are all integrated into the QM for full traceability. Aggregated quality scores are estimated as a percentage of maximum score by criterion, by type of evidence or for the whole QM.The EVIDEM framework was tailored to reflect the thought process underlying decisionmaking and to fit the continuum from data generation to decision to communication of decision. It supports decisionmaking and deliberative processes by structuring, segregating and providing transparent access to evidence , while facilitating communication about value judgments and data needs among stakeholders.The instruments developed to operationalize the EVIDEM framework are rooted in existing processes and instruments; however, they integrate the essential components of decisionmaking into a comprehensive and cohesive structure. The VM draws on the flexibility and comprehensiveness of MCDA while disentangling extrinsic from intrinsic value components, and providing structured access to the evidence on which those value judgments are based. Unlike some earlier applications of MCDA,,74 the VThe EVIDEM framework needs to be tested in context, validated and further developed through iterative collaborative processes. In a proof of concept approach, the system was pilot-tested using historical cases in the Canadian context with the objective of assessing feasibility, practicality and value to end users. The Canadian Value Panel convened for the pilot study indicated that the VM with embedded synthesized data would be highly useful as a support for healthcare decisionmaking, to guide discussion and share values among decisionmakers, at both the policy and clinical levels, by systematically assessing strengths and weaknesses of healthcare interventions in a comprehensive and structured fashion.Practical use of this approach faces significant challenges. Among these are uptake by decisionmaking bodies; this will only happen if the new process is perceived as facilitating and simplifying their task, rather than adding complexity. The EVIDEM framework was designed to create a simple and practicable series of freely accessible tools that could be easily integrated into existing processes, while providing a common ground. In addition, integration of EVIDEM records into a web-based collaborative database is intended to provide a platform to all stakeholders for easy access to high level data on evidence available for healthcare interventions, as well as to value estimates.Another major challenge will be the bringing together of data producers and those who make decisions. There are issues of trust and bias that need to be surmounted to provide the collaborative environment that this process would need. This would permit the 360 degree transparency as envisioned by Dhalla & Laupacis.The framework was designed to be of use to a variety of healthcare decisionmakers. Several applications are envisioned Figure . RetrospAnother aspect of healthcare decisionmaking, which requires further development, is extrinsic or system-related value judgments (Figure Several features are integrated in the EVIDEM framework to facilitate communication between those who generate data and those who need data to make decisions. Through iterative processes, the framework can help define evidentiary needs of decisionmakers and be used as a planning tool for researchers and developers of new interventions, to ensure that the data that is generated addresses the needs explicitly defined (Figure Healthcare decisions have to be made in the context of a plethora of information, without easy access to all the necessary information and without an explicit decisionmaking framework. This often results in poor transparency and controversial decisions. The EVIDEM framework provides a comprehensive transparent structure grounded in global standards and local needs. The proposed framework is a step to organizing evidence and streamlining processes on a collaborative approach. This framework should not be viewed as a formula but rather as an aid to ensuring that all important data is considered and that rationales and values underlying a decision may be shared. It supports deliberative processes ,90 allowAHRQ: US Agency for Healthcare Research and Quality; CHEC: Consensus on Health Economic Criteria; CONSORT: Consolidated Standards of Reporting Trials; EVIDEM: Evidence and Value: Impact on DEcisionMaking; KTE: Knowledge transfer and exchange; MCDA: MultiCriteria Decision Analysis; MOOSE: Meta-analysis Of Observational Studies in Epidemiology; PRO: Patient reported outcomes; QoL: Quality of life; QUOROM: Quality of Reporting of Meta-analyses; STROBE: STrengthening the Reporting of OBservational studies in Epidemiology.The authors declare that they have no competing interests.MMG, DR & MW conceived the framework, developed the instruments and drafted the manuscript. HK, RL and LJE participated in the design of the methodology and drafting of the manuscript. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:


To make this process more transparent and potentially effective, we need to understand these influences. Psychological theories of decision-making and social influence provide a framework for this understanding.Health care practice based on research evidence requires that evidence is synthesised, and that recommendations based on this evidence are implemented. It also requires an intermediate step: translating synthesised evidence into practice recommendations. There is considerable literature on evidence synthesis and implementation, but little on how guideline development groups (GDGs) produce recommendations. This is a complex process, with many influences on communication and decision-making, This study aims to investigate the processes by which GDGs formulate recommendations, drawing on psychological theories of decision-making and social influence. The findings will potentially inform the further evolution of GDG methods, such as choice of members and procedures for presenting evidence, conducting discussion and formulating recommendations.e.g., dilemmas, conflicts, and uncertainty. Data will be analysed thematically and by content analysis, drawing on psychological theories of decision-making and social influence.Longitudinal observation of the meetings of three National Institute of Health and Clinical Excellence (NICE) GDGs, one from each of acute, mental health and public health, will be tape recorded and transcribed. Interviews with a sample of GDG members at the beginning, middle, and end of the GDG's work will be recorded and transcribed. Site documents including relevant e-mail interchanges, GDG meeting minutes, and stakeholders' responses to the drafts of the recommendations will be collected. Data will be selected for analysis if they refer to either evidence or recommendations; the focus is on "hot spots",  Widespread claims of evidence-based clinical practice are premised on development of recommendations based on research evidence. Using research evidence to inform practice requires three stages: synthesising evidence, translating evidence into recommendations, and implementing recommendations in practice. There is considerable literature on the processes of synthesising evidence, and of implementing recommendations, but little on how evidence is translated into recommendations. The model of guideline development adopted by the National Institute for Health and Clinical Excellence (NICE) in the United Kingdom involves a Guideline Development Group (GDG) comprising academic, professional, and lay representatives from relevant disciplines and practices. Groups meet between three and fifteen times to consider evidence and recommendations. Their face-to-face and e-mail discussions are informed by: Verbal and written presentation of research evidence by systematic reviewers and health economists as well as contextual evidence from co-opted experts; comments on the evidence and draft recommendations by stakeholders; and comments and advice by NICE internal review teams.et al., personal communication).Despite the stated evidence base of guideline recommendations, there is little information about the relationship between research evidence and recommendations. A study of 15 clinical guidelines on Type 2 diabetes from 13 countries found that research evidence is not necessarily the most powerful influence on the content of recommendations [e.g., the quality of evidence, the methods of presentation of the evidence, evaluation of different types of evidence, weight given to economic evidence, practical/resource constraints, opportunity costs, values , degree of change required by the recommendations, professional and scientific interests, potential direct or indirect financial interests [e.g. status, conformity and compliance) may influence guideline development [There is often no explicit framework for the conduct of the discussion or decision-making within GDGs. There are various possible influences on group discussion and decisions, nterests ,3, and telopment . Severalelopment .There are calls for greater transparency about the processes of synthesising research and other evidence and of translating these into recommendations, but how best to do this is unclear. There are also calls for current practice to be changed to briefer, more structured deliberations . HoweverCommunication with experts in the field and subsequent literature searches identified four studies that have analysed GDG processes. A pilot study has examined normative influences on guideline construction, although it was unclear what were "normative influences" . PagliarThe current study aims to build on this work by drawing on psychological theories of decision-making and social influence in describing and analysing how issues in these areas are resolved, with a goal of identifying good practice. It will do this across three health care settings .To investigate the processes by which GDGs formulate recommendations on the basis of the research evidence they receive, using psychological theory to guide data analysis.The main questions to be addressed concern:1. Who? Who has most influence, demonstrates particular approaches2. How? What strategies are used in considering evidence and in formulating recommendations3. Why? What beliefs about the purpose and nature of evidence and recommendations may explain these strategiese.g., clarity, comprehensibility.4. With what result? GDG members views about the quality of GDG process and outcome; objective quality indicators of the recommendations, A longitudinal observation study of the development of national guideline practice recommendations, from the first to last GDG meeting.The first NICE GDG to start from November 2006 in each of three areas of health care, acute health, mental health and public health, will be selected. The GDGs participating in the study are not specified in order to protect anonymity of the members.Data will be collected from several sources:Meetings of GDGs and their topic group will be tape recorded and transcribed. Using an ethnographic approach, a researcher will attend meetings, take field-notes, and make observations. Members will be asked to introduce themselves by name and role at the beginning of each meeting to assist in their identification. The researcher will position herself in a place that is as unobtrusive as possible, but that will allow identification of speakers and a view of members' facial expressions. Not all meeting recordings will be transcribed and analysed. A pre-validated process will be used to identify the key exchanges and interactions relating to recommendations and evidence. Data from the pilot study is being used to develop criteria for this process.Site documents include relevant e-mail interchanges, GDG meeting minutes, and stakeholders' responses to draft recommendations. Given the large number of e-mails that each GDG may generate, the research group will use the pilot study to develop a "key word" search approach to filter the site documents on the basis of their relevance to evidence and recommendations.i.e., service providers, academics, practitioners, service users, systematic reviewers, NICE project team and chair). A selection of group members, representing different levels of involvement, will be interviewed at the beginning, middle, and end of the GDG process. Higher order interview questions have been developed to investigate GDG members' perceptions of the factors influencing their thinking and decisions about recommendations . A study information sheet (additional file SM, JBS, PD and RR declare that they have no competing interests. GF has been paid to chair two NICE guideline development groups and one guideline development group by Weleda. FC, PA and SE advise on the methodology used for developing clinical guidelines at NICE.SM conceived and developed the study, drafted the study protocol, and leads its implementation. SM helped to draft the manuscript. JBS coordinates the ongoing study, collected and analysed pilot data, and helped to draft the manuscript. SP, GF, PD, RR, FC, PA, and SE are members of the Study Steering Group, and have contributed to the development of the protocol. All authors read and approved the final manuscript.Draft Interview Questions. These are the core questions that were adapted for interviews with Guideline Development Group members.Click here for fileStudy Information Sheet. This is the information sheet about the study given to eligible participants.Click here for fileStudy Consent Form. This is the consent form that participants signed.Click here for file


Despite the growing availability of clinical practice guidelines since the early 1990's, little is known about how guideline development and dissemination may have changed over time in Canada. This study compares Canadian guideline development, dissemination, and evaluation in two six year periods from 1994–1999 and 2000–2005.Survey of guideline developers who submitted their clinical practice guidelines to the Canadian Medical Association Infobase (a Canadian guideline repository) between 1994 and 2005. Survey items included information about the developers, aspects of guideline development, and dissemination and evaluation activities.Surveys were sent to the developers of 2341 guidelines in the CMA Infobase over the 12 year period, 1664 surveys were returned (response rate 71%). Of these, 730 unique guidelines were released from 1994–1999, and 630 were released from 2000–2005. Compared to the earlier period, more recent guidelines were being produced in English only. There has been little change in the type of organizations developing guidelines with most developed by provincial and national organizations. In the recent period, developers were more likely to report using computerized search strategies (94% versus 88%), publishing the search strategy (42% versus 34%), reaching consensus using open discussion (95% versus 78%), and evaluating effectiveness of the dissemination strategies (12% versus 6%) and the impact of the CPGs on health outcomes (24% versus 5%). Recent guidelines were less likely to be based on literature reviews (94% versus 99.6%) and were disseminated using fewer strategies (mean 4.78 versus 4.12).Given that guideline development processes have improved in some areas over the past 12 years yet not in others, ongoing monitoring of guideline quality is required. Guidelines produced more recently in Canada are less likely to be based on a review of the evidence and only about half discuss levels of evidence underlying recommendations. Guideline dissemination and implementation activities have actually decreased. Unfortunately, the potential positive impact on patient health outcomes will not be realized until the recommendations are adopted and acted upon. Their potential benefits, however, are contingent on both rigorous guideline development processes that incorporate the best available evidence and successful implementation of guidelines into practice [In the 21practice .Canadian developers of 1446 guidelines released between 1994 and 1999 that were included in the Canadian Medical Association (CMA) InfoBase were surveyed to determine how guidelines were developed, disseminated, and evaluated; the analysis of these data were reported in an earlier publication. At thatReviews of systematic reviews about the effectiveness of various strategies to increase health care professionals use of research and practice guidelines published in the late 1990s indicated that dissemination of educational materials and didactic educational sessions had little or no effect on changing professional practice,7. In thCurious about whether there had been any changes in guideline development, dissemination, and evaluation activities in Canada over the past decade, the CMA continued to survey the developers of guidelines released during the period 2000–2005 and included in the CMA Infobase. This paper reports on how guidelines were developed, disseminated and evaluated in Canada between 1994 and 2005. We examined changes in these activities between the 6 year periods of 1994–1999 (earlier period) and 2000–2005 (recent period)., contains guidelines that are endorsed or developed by Canadian organizations within 5 years of the current date and are of interest to the CMA membership. At present, there are no quality criteria applied to screen the guidelines that are admitted to the repository. The CMA assigns English and French versions of bilingual guidelines a separate identifying number when each version is entered into the database and therefore both are included independently in the total number of guidelines in the database. The CMA automatically withdraws guidelines 5 years after their release. If a guideline is updated at any time, the original version is withdrawn and the update entered as a new guideline and the developers surveyed about the updated version. The database can be searched electronically by keyword, medical subject heading (MeSH), guideline developer, and recent additions to the database.The Canadian Medical Association surveyed Canadian developers of guidelines whose guidelines were included in the CMA Infobase between 2000 and 2005. Results were compared with the results from our previous survey of developers of guidelines released between 1994 and 1999[Developer organizations were classified as being one of seven types, in the same way as for the earlier analysis. This clThe survey was the same one as used in the previous study  and 200Of 2341 guidelines included in the CMA Infobase between 1994 and 2005, completed surveys were received for 1664 guidelines (71%). There were 1446 guidelines listed in the Infobase for the earlier period 1994–1999 and 895 guidelines listed for the recent period 2000–2005. As with the previous study[Over the 12 year period, the 1360 unique guidelines were developed by a total of 96 different guideline developers: 75 different guideline developers for the 1994 to 1999 period and 56 for the 2000 to 2005 period. Over 80% of the guidelines in both periods were produced by national professional, para-government, government, or licensing bodies . Frequent developers, those who developed 4 or more guidelines, comprised 41 of the 96 guideline developers 42%) and were responsible for developing 1275 (93.8%) guidelines  and graded the quality of evidence used in the guideline (51.6% compared to 38.1%)Table .More recently, guideline developers engaged in significantly fewer dissemination and implementation activities per guideline than in the earlier period Table . On averAn examination of the actual strategies used per guideline Table  reveals Examining KT strategies by frequent and infrequent developers, we note that guidelines produced by frequent developers employed active, passive and educational strategies to promote their guideline at similar rates to the infrequent developers Table . FrequenThere was an increase in guideline developers who reported either evaluating the effectiveness of their dissemination strategies or the impact of the guideline on health outcomes in the recent period Table . DevelopIn order to more fully appreciate the decline in the use of dissemination and implementation strategies, we looked at their use by different guideline developers. There was a decline in the use of passive strategies by national professional organizations (71.4% down from 98.4%) and national health organizations (50% down from 95.8%). While licensing bodies were using less educational (34.4% down from 84.5%) and less active implementation strategies (18.8% down from 80.2%), government organizations were using more of both educational (56.9% up from 12.4%) and active implementation strategies (45.4% up from 2.7%). Provincial organizations also increased their use of educational strategies (14.3% up from 0%).Although guidelines produced by developers with 4 or more guidelines in the CMA Infobase were not more likely to evaluate the effectiveness of the dissemination and implementation strategies, they were more likely to plan to do so in future (44.4% compared to 29%). The greater reliance on computer searching for the evidence and greater transparency about the search strategy in the more recent period is positive but it is not known whether this translated in higher quality guidelines.Of concern is the fact that less than half the guidelines in the recent period graded the quality of the evidence and 6% did not even review the scientific literature and both at lower rates than that of guidelines produced in the earlier period. While we did not assess the quality of the guidelines in the CMA Infobase in this study, previous work has revealed that the quality of drug guidelines in this database was less than optimal and giveGiven the international efforts such as the AGREE Collaboration to improIn terms of guideline developer knowledge translation activities over the two time periods, there has been a small but significant decrease in the total number of dissemination and implementation strategies employed per guideline. This was largely due to the use of fewer passive dissemination strategies in the more recent period. One hypothesis for the decline might be growing awareness of early evidence that passive dissemination was ineffective at changing professional behaviour,7. More If the benefits of the guidelines produced are to be achieved, guideline developers and their stakeholders should reconsider their dissemination and implementation activities and how to work together to better encourage the adoption of their guidelines into routine practice. Based on the findings of Grimshaw and colleagues, it is reasonable to continue using passive dissemination but also important to use more targeted implementation strategies aimed at overcoming contextual barriers to implementation and embedding guidelines within organizational structures such as documentation and ordering systems. As thesIt is encouraging that in the more recent period, the effectiveness of dissemination and implementation activities is being evaluated in twice as many guidelines (12.2% vs. 6.1%). Since there remains considerable room for greater research on KT strategies, it is unfortunate that fewer developers reported intending to undertake such evaluations in the future. The proportion of guidelines whose developers report formally evaluating the impact of their guideline on health incomes has increased substantially over the 12 year period from 5% to 24% of guidelines. Data on the positive health outcomes of guidelines may be useful for encouraging others to incorporate guidelines into their healthcare decision making. The opportunity to evaluate the impact of a guideline on health outcomes may also provide a safe forum for potential adopters to try the guideline, to contextualize the recommendations of the guideline for their clinical setting and to support implementation under temporary research conditions. Evaluation research can be considered an active implementation strategy, especially where the changes in clinical practice recommended in the guideline are sustained.Our comparison of developers who submitted three or fewer, or 4 or more, guidelines to the CMA Infobase over the 12 year period revealed some interesting findings that will need to be confirmed by future research. Guidelines produced by more experienced guideline developers were more likely to have done a computerized search of the literature, graded the quality of the evidence, planned to formally evaluate the dissemination/implementation strategies they use, formally evaluated the impact of the guideline on health outcomes, and had a companion document for consumers. There were no differences in terms of the dissemination and implementation activities undertaken by the two groups. One interpretation of these findings is that the volume of guidelines produced by a developer may be important and related to higher guideline quality and evaluation but not KT activities. Graham and colleagues previously found lower quality in guidelines developed by government, para-government or professional organizations compared to those developed by other types of guideline developers. ConsequThese findings should be considered within the limitations of the study. First, the survey data were self-reported by the guideline developers and were not objectively verified. However, the information provided in the survey (information about the guideline development process) is available on the CMA Infobase with the guideline and therefore makes verification of survey responses possible. Another consideration was that the survey was also sent only after the guideline had been accepted for inclusion in the CMA Infobase rather than as part of the process of accepting the guideline. Both of these factors may have encouraged guideline developers to accurately report their responses. Furthermore, since more quality indicators remain unmet in the recent time period, the change in response between the two study periods is likely an accurate reflection of their activities.Another limitation relates to the questions used to assess the quality of guideline development. In the years since the survey was developed, the AGREE Collaboration has deve). However, CMA believes that the database represents the majority of CPGs published in English Canada since they have built a comprehensive searching and screening strategy that adds to what developers' submit: they search various databases (notably Medline) and websites regularly and hand search major medical journals for guidelines of interest to physicians. The CMA also believes that the guidelines that are not identified in this process are most likely published by some very small specialty groups. Finally, it is likely that developers will deposit their guidelines here since this is the primary repository for guidelines targeting Canadian physicians, who can access the CMA Infobase at no cost.A third limitation is that conclusions can only be drawn about the guidelines and their developers that were deposited into the CMA Infobase during the study period. We have no way of knowing what proportion of Canadian guidelines is deposited in the Infobase. It is possible that guideline developers producing guidelines in French may not be submitting them to the CMA Infobase since only about one-third of Quebec physicians are members of the CMA. There are also other repositories of guidelines for non-physician health care providers in Canada (for example, RNAO's Best Practice Guidelines at Guideline development processes in Canada have improved in some areas over the past 12 years yet not in others. Guideline dissemination and implementation activities have actually decreased. Therefore, guideline developers might benefit from having resources on best practices for guideline development. CMA have started this process with the updating of Handbook on Practice Guidelines . EncouraThe authors declare that they have no competing interests. Nan Bai is an information specialist at CMA in charge of Infobase.JK participated in the design of the study, completed the statistical analysis, and drafted the manuscript. IDG conceived of the study, provided input into the design of the survey, guided the analysis and helped draft the manuscript. DS contributed to the interpretation of the results and drafting of the manuscript. NB provided the data and contributed to interpreting the results. All authors read and approved the final manuscript.Survey instrument. Survey developed by the CMA in the early 1990s and used to report on guidelines in the CMA Infobase from 1994–2005.Click here for file