In social insurance, the evaluation of work disability is becoming stricter as priority is given to the resumption of work, which calls for a guarantee of quality for these evaluations. Evidence-based guidelines have become a major instrument in the quality control of health care, and the quality of these guidelines' development can be assessed using the AGREE instrument. In social insurance medicine, such guidelines are relatively new. We were interested to know what guidelines have been developed to support the medical evaluation of work disability and the quality of these guidelines.Five European countries that were reported to use guidelines were approached, using a recent inventory of evaluations of work disability in Europe. We focused on guidelines that are disease-oriented and formally prescribed in social insurance medicine. Using the AGREE instrument, these guidelines were appraised by two researchers. We asked two experts involved in guideline development to indicate if they agreed with our results and to provide explanations for insufficient scores.We found six German and sixteen Dutch sets of disease-oriented guidelines in official use. The AGREE instrument was applicable, requiring minor adaptations. The appraisers reached consensus on all items. Each guideline scored well on 'scope and purpose' and 'clarity and presentation'. The guidelines scored moderately on 'stakeholder involvement' in the Netherlands, but insufficiently in Germany, due mainly to the limited involvement of patients' representatives in this country. All guidelines had low scores on 'rigour of development', which was due partly to a lack of documentation and of existing evidence. 'Editorial independence' and 'applicability' had low scores in both countries as a result of how the production was organised.Disease-oriented guidelines in social insurance medicine for the evaluation of work disability are a recent phenomenon, so far restricted to Germany and the Netherlands. The AGREE instrument is suitably applicable to assess the quality of guideline development in social insurance medicine, but some of the scoring rules need to be adapted to the context of social insurance. Existing guidelines do not meet the AGREE criteria to a sufficient level. The way patients' representatives can be involved needs further discussion. The guidelines would profit from more specific recommendations and, for providing evidence, more research is needed on the functional capacity of people with disabilities. In the western world, work disability is a problem at the individual, company, and societal levels. Western countries spend about 1.2% of GDP on work disability benefits or 2% if sickness benefits are included, which, for most countries, is an increase over the past 15 years. The probability of returning to work after being granted a long-term disability benefit is below 2% annually on average. Work disability is the end of their working life for the vast majority of recipients . In the When resources are tight, it becomes even more important to determine in a valid and scientifically sound way who is and is not entitled to disability benefit. Internationally, the medical evaluations of work disability turn out to be relatively comparable while being part of social insurance systems that vary strongly -8. The qOne way of ensuring the quality of medical work is to use evidence-based guidelines , which iHaving guidelines for medical work does not necessarily mean that the quality of the work is supported. Guidelines need to be adequate for the process they are to support and they need to be used in practice. The Appraisal of Guidelines Research & Evaluation (AGREE) collaboration developed the AGREE instrument to assess the quality of clinical practice guidelines  and to eMedical practice in social insurance evaluations is different from clinical medical practice in several ways ,30. In cIt is difficult to diagnose the functional consequences of diseases in general and even more so for non-specific diseases such as lower back pain, chronic fatigue, and stress-related disorders. The association between a medical diagnosis and the functional limitations that may lead to work disability is weak and influenced by environmental and personal characteristics, as described in the International Classification of Functioning and Health (ICF) model . From a 1. What is evaluated in your countries' work disability evaluation?2. What instruments are used for these evaluations?We were interested to determine what guidelines exist in different countries and their quality by focusing on the following research questions:disease-oriented guidelines have been developed to support the medical evaluation of work disability?1. What 2. What is the quality of these guidelines in social insurance medicine?We used the EUMASS table to determine the countries in which guidelines were reported to be in use. The Netherlands, the Czech Republic, Germany, the United Kingdom, and Switzerland were visited based on their reported use of the guidelines; no other countries had reported using guidelines for medical evaluations. The status of guidelines was assessed during the visits by determining if they were officially prescribed. Copies of the guidelines with explanation were collected. For this article we focused on the guidelines for evaluating work disability by SIPs that were prescribed by law or as an instruction by the ISI. We distinguished between disease-oriented guidelines  and process-oriented guidelines , a distinction that is evident from the relative guideline's title. We selected disease-oriented guidelines. To compare guidelines, we selected those that addressed the same diseases.The selected guidelines were scored using the AGREE instrument, which uses 4-point scales for each item: scope and purpose (3 items), stakeholder involvement (4 items), rigour of development (7 items), clarity and presentation of the recommendations (4 items), applicability of the guideline (3 items), and editorial independence (2 items). To correct for the different number of items in each domain, The AGREE instruments suggests calculating domain scores by relating the obtained scores (OS) to the maximum possible score (MaPS) and the minimum possible score (MiPS) using the formulaAs a test, one (Dutch) guideline (burnout) was scored by two researchers (WdB and DB) using the AGREE instrument and its user guide to establish if additional rules for scoring would be required. The test showed the need for additional scoring rules. We specified the clinical question and the target population and we adapted user guide item 11  and 16 (options for management of the condition) [see Additional File This study was not submitted for ethical approval. The study included physicians who were not asked to perform specific professional actions for this study, but only to complete a questionnaire. All studied documents are in the public domain.In Germany seven guidelines for SIPs turned out to be officially in use. In the Netherlands twenty-four were found and one in Switzerland. These guidelines are partly process-oriented and partly disease-oriented. Process guidelines were used in Germany (1), the Netherlands (8), and Switzerland (1). The German and Swiss guidelines each contain many recommendations that in the Netherlands are distributed over eight smaller guidelines. The recommendations refer, for example, to the relevance of the diagnosis for the evaluation and to the boundaries of the concept of disease. Another topic of these guidelines is the claimant's obligation to attempt to recover and find gainful employment. Yet another aspect is the relevance of distinguishing between the opinions of the claimant and the SIP. These recommendations represent the consensus of legal and medical experts on the principles of evaluation, but not on scientific evidence. These process-oriented guidelines were excluded.Disease-oriented guidelines were in use in Germany (6) and the Netherlands 16), shown in Table 6, shown The Dutch guidelines, all implemented by law, were first developed by the Health Council of the Netherlands and later by the scientific association of SIPs (NVVG). The German guidelines were developed and prescribed by the German Institution of Social Insurance (DRV). The German guidelines were developed earlier than the Dutch and most have been updated since their inception.Of the guidelines, four diseases were common to both countries: breast cancer, chronic obstructive lung disease, lumbar intervertebral disc herniation, and myocardial infarction.The initial agreement between researchers was high for the Dutch guidelines (Kappa range 0.814-0.939), but low for the German counterparts (Kappa range 0.449-0.624). After discussing the different opinions of the researchers, agreement was reached on all items and scoring by the third researcher was unnecessary. The results are presented in Table The scope and purpose of the guideline were well described in all eight guidelines; the score in both countries was 100%. All guidelines were designed to support the medical evaluation of work disability by indicating what functional incapacities were to be expected in cases with a specific diagnosis.Stakeholder involvement was 52% for the Dutch and 33% for the German guidelines. Potential users were well defined , but the involvement of professional groups was found to be incomplete in seven of the eight guidelines. The patients' views were not sought in the German guidelines and only at the final stage in the Dutch. No guidelines were piloted among end-users before their publication.Rigour of development was 16% with the Dutch and 23% with the German guidelines. How evidence was gathered and the scientific grounding of recommendations were not explicit in any guideline.Clarity and presentation of the guidelines was 63% for the Dutch guidelines and 71% for the German. Although the recommendations were unambiguous and easily identifiable in almost all cases, they were not overly specific. Different options for assessing the condition of the guidelines were often mentioned, and the German guidelines provided tools for the evaluations.Applicability scored 6% in the Netherlands and 8% in Germany. Practical barriers and costs were not addressed in any guideline. The German guidelines contained indications of when to update them.Editorial independence was limited in both countries. The Dutch guidelines reached 50% on average as they were developed independently of the funding body, but with only a general procedure about conflicting interests. The German guidelines (0%) were developed entirely within the ISI and conflicting interests were not addressed.The Dutch expert was involved in developing 11 of 16 then-published guidelines in the Netherlands and 3 of the 4 protocols that we scored on the AGREE instrument. He agreed to all our scoring after we discussed our scoring rules with him. He attributed low scores to the newness of creating guidelines for social insurance medicine in the Netherlands and that the short time allotted to create them was a factor. Stakeholder involvement was also reduced because patients' involvement was controversial in the beginning as there was concern about patients being biased with regard to the recommendations. The low figure on rigour of development was because the methods of development had not been recorded and because the field had no scientific tradition. The lack of specificity of the recommendations was due mainly to a lack of existing scientific research. Applicability scored low in the Netherlands as the guidelines were developed by the Health Council, for whom this was not a regular activity. The aspects of applicability were considered by the ISI after publication of the guidelines.The German expert was involved in developing five of six guidelines published at the time in Germany and in all the guidelines we scored on the AGREE instrument. He agreed to nineteen of the twenty-three scores after we discussed our scoring rules with him. Differences were due partly to how the German guidelines were described  and to differences in the interpretation of items 13, 14, and 15. He commented that the development of guidelines was new in Germany and started from a need of the SIPs within the ISI, which explained the limited involvement of stakeholders. The involvement of patients' representatives was considered unhelpful because of expected bias. Testing among users was done implicitly as the guidelines were developed at the institution where the SIPs work. The selection of evidence and formulation of recommendations were carried out according to what the German experts considered the most important. No need had existed to document any more than they did for internal use, which accounted for the low score on the rigour of the guidelines' development. This internal development also accounted for the low score on applicability; this was included implicitly within the development process of internal guidelines. Editorial independence was not considered important, as the interests of the SIPs and the ISI were not supposed to conflict.In this study we looked for the existence of evidence-based guidelines for the medical evaluation of long-term work disability and the quality of development of these guidelines.Using the EUMASS comparison, we found guidelines for the medical evaluation of work disability, both disease- and process-oriented, in official use in four of seventeen European countries. In two of these countries we found twenty-two disease-oriented guidelines in official use in these evaluations. The AGREE instrument was applicable for scoring the selected Dutch and German guidelines, although minor adaptations to the AGREE instrument were necessary. Scoring German guidelines gave a smaller initial agreement than the Dutch, due to language problems and understanding of the German social insurance; however, the consensus procedure compensated for these issues. The guidelines scored well on 'scope and purpose' and 'clarity and presentation', and moderately on 'stakeholder involvement' in the Netherlands, but low in Germany; all guidelines scored low on 'rigour of development'. 'Editorial independence' and 'applicability' were low as a result of how production was organised.To our knowledge, this is the first study to identify and qualify medical guidelines in social insurance medicine at an international level. As we were looking for official guidelines, we do not believe that we missed any in the countries we included; however, focusing on official guidelines may have resulted in finding fewer guidelines than are in practical use. For example, in Germany and Switzerland, guidelines are published by specialists in scientific journals. These are not in official use, but they may support physicians in their evaluations.We used the AGREE instrument to determine the quality of the guidelines, which recommends using four appraisers for a good reliability . Using aOur study corresponds with other research; the distinction between legal and medical guidelines fits with the results of Boer et al.  about thWe found disease-oriented guidelines in only two participating countries, and there they are recent. Work disability is being evaluated on similar aspects in many countries, despite large differences in organisation of social insurance ; thus, wOur comparison of development quality is based on four Dutch and four German guidelines, on four different pathologies. The German and Dutch social insurance systems differ in many aspects, but both require a medical statement about functional capacity in cases of claims for work disability benefit. From this perspective the guidelines are comparable in and between countries. As the guidelines in these countries have been created in a similar fashion, we expect our results to be relevant to future disease-oriented guideline development in these countries.We used the AGREE instrument as a tool for evaluating the quality of guideline development in social insurance medicine, a procedure that, to our knowledge, is new. It is unclear if using the AGREE instrument in a different domain is without problems; however, neither we, nor the experts we consulted, noticed any clear incongruence. The AGREE instrument is now being utilised in both Germany and the Netherlands.With the AGREE instrument the quality of the development of guidelines can be scored, which is not the same as the quality of the recommendations. It is possible that the guidelines contain adequate recommendations that have been developed in a suboptimal way or whose development has been accounted for in a suboptimal way. Good practice, however, is best supported by guidelines that have been developed in a proven, optimal way. Several aspects need further consideration. The involvement of patients' representatives is now accepted in the Netherlands, after much discussion about the nature of their input; in Germany, however, this is not the case. This difference illustrates the ambiguity of the claimant's position in social insurance medicine: he is both passive object of the evaluation and participating subject in work disability. AGREE criteria are clear, however: participation of patients' representatives is mandatory. The development of the guidelines in the Netherlands has now been placed under the authority of the scientific association of SIPs, as this is viewed as the best way to retain independence from both the funding and implementing bodies. In Germany, financing, developing, and implementing within the ISI is considered effective, which illustrates the ambiguity of the profession of social insurance medicine as a discipline that needs to stress its independence and quality and a group of doctors working for administrative organisms with more interests than medical quality ,33. AGREThe inclusion of disease-oriented research into the practice of disability evaluation will help coordinate clinical, occupational, and social insurance medicine, in using the same concepts and findings, although in different spheres. The lack of scientific evidence may be compensated for, in part, by research on the aspects that influence disability with chronic conditions in general ,43. ParaWe expect that the diffusion of our results may aid further development of guidelines in social insurance medicine and, notably, help these become increasingly more evidence-based, which would assist in establishing a new and important mechanism for quality control in social insurance medicine. Paraphrasing Lohr , evidencEvidence-based guidelines form an important instrument for enhancing the quality of medical practice. Guidelines can provide a framework on which a clinician can ground diagnosis, therapy, and prognosis. Guidelines in social insurance medicine for the evaluation of work disability are a recent phenomenon, so far restricted to Germany and the Netherlands. We expect that disease-oriented guidelines can be useful in other countries as well, and can help the SIP ground his evaluation of capacity for work. For the practice of evaluating work disability, this would mean an important instrument to control quality. The AGREE instrument is suitably applicable for assessing the quality of guideline development in social insurance; nevertheless, some of the scoring rules need to be adapted to the context of social insurance. Existing guidelines do not meet AGREE criteria sufficiently. Notably, how patients' representatives can be involved and the editorial independence of the guideline developers need further discussion. The guidelines would profit from more specific recommendations and, for this, more research is needed on the functional capacity of people with disabilities. To date, research has focused primarily on the recovery from complaints, while mainly ignoring the resumption of work. The latter depends on much more than a health condition, but still, the challenge of health care should not only be to give relief for pain and suffering, but also to allow participation in society and to legitimise a disability benefit if needed for medical reasons.The authors declare that they have no competing interests.WB designed the study, carried out the field work, and prepared the manuscript. DB participated in the scoring and drafting of the article. AR participated in the field work. PD supervised the field work and participated in drafting the article. HA supervised and participated in the drafting of the article. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:AGREE from clinic to social insurance. this document describes the way in which AGREE criteria were used in the study of guidelines in social insurance.Click here for file


Over recent decades, the number of available clinical practice guidelines has grown enormously. These guidelines are increasingly used in health-care systems throughout the world to improve the quality of patient care, and this also applies to cancer care. Evidence-based guidelines are seen by professionals, authorities, managers and policy makers as powerful tools for achieving effective and efficient care . HoweverVarious problems with guidelines and their development that can impede their maximal use and profit have been reported:Lack of quality: There are currently too many low-quality guidelines. There seems to be a ‘guideline industry’ emerging in many Western countries with a considerable variation in guidelines from different sources . sources . Many ofLack of evidence: A second problem is that despite a rigorous search and analysis of the scientific literature, clear evidence is available for only part of the practical actions and decisions recommended in the guidelines  patients they should meet specific quality criteria , the collaboration's aim is to establish comparable frameworks for the assessment and monitoring of the quality of clinical practice guidelines, including the process of development and the reporting of the process. The AGREE Instrument was developed through a multistage process of item generation, selection and scaling, field testing and refinement procedures. A small working group first compiled a comprehensive checklist of 82 items from existing appraisal instruments and relevant literature that covered recognised components of guideline quality. The term ‘quality’ was defined as the confidence that the biases linked to the rigour of development, presentation and applicability of a guideline had been minimised during the development process. Most of the items were derived from existing lists or instruments e.g. . Generalhttp://www.agreecollaboration.org). Each domain is intended to measure a separate dimension of guideline quality.The final AGREE instrument consists of 23 key items (Table 1Scope and purpose (items 1–3): These items are concerned with the overall aim of the guideline, the specific clinical questions and the target patient population.Stakeholder involvement (items 4–7): These items focus on the extent to which the guideline represents the views of its intended users. Guideline development needs to be carried out by a multidisciplinary group involving all stakeholders whose clinical activities are likely to be covered in the proposed guideline. This also includes patient groups.Rigour of development (items 8–14): These items relate to the process used to gather and synthesise the evidence, and the methods used to formulate the recommendations and to update them. The recommendations should be explicitly linked to the supporting evidence. However, because most current guidelines use a mixture of ‘expert’ judgement and literature review, disclosure of disagreement or uncertainties encountered during the development may help to clarify the process. Guidelines should be reviewed externally before publication, and the process used clearly described. They should also always include a date of publication, and because guidelines need to reflect current research, they should contain a clear statement about the updating procedures.Clarity and presentation (items 15–18): These items deal with the language and format of the guidelines. Since the main role of guidelines is to help clinicians and patients make better decisions, busy clinicians need simple, patient-specific, user-friendly guidelines that are easy to understand. Good guidelines present clear information about the management options available and the likely consequences of each. This information can be presented in a variety of formats to suit the needs of the user.Applicability (items 19–21): These items cover the likely organisational, behavioural and cost implications of applying the guidelines. Guidelines should be feasible to use in the current organisation of care and must fit into routine practice and the time constraints present. In addition, review criteria should be developed that link the guideline use to audits and other quality improvement initiatives.Editorial independence (items 22–23): These items assess the independence of the recommendations and acknowledgement of possible conflict of interests for the members of the guideline development group. An increasing number of guidelines are funded directly, or indirectly, by external funding. There should be an explicit statement that the views and/or interests of the funding body have not influenced the final recommendations.To help users understand the items, the instrument contains a users' guide with explanatory notes. Each item is scored on a reduced four-point Likert scale, and there is an overall rating as to whether the guideline should be recommended or not for use in practice.The AGREE instrument was developed through a detailed and lengthy process that took many years to complete. Despite this, most of the AGREE quality criteria are still based on theoretical assumptions rather than on empirical evidence. They were developed through discussions between researchers from several countries who have extensive experience and knowledge of clinical guidelines. It remains to be shown that these criteria are actually linked to ‘better’ quality guidelines leading to improved patient care and outcomes. Another issue is that the AGREE instrument relies heavily on the quality of the background documentation on which the guidelines are based. Although defining quality by the rigour of reporting rather than the rigour of content may not provide information on the intrinsic quality of the guidelines, it is clear that without some information about the development process it is impossible to assess the quality of guidelines (http://www.sign.ac.uk). Uniform reporting gives a certain guarantee of quality. Moreover, it simplifies the comparison of guidelines for the same clinical conditions. For the development (or revision) of guidelines, the use of existing high-quality guidelines  can save a lot of time and effort (http://www.guideline.gov/index.asp). For example, the literature search and review could be used when similar questions are being examined. Above all, it is useful to see how other guideline development groups have collected and interpreted the evidence and how they have translated the evidence into recommendations. After publication, guidelines must be disseminated and implemented effectively ’, is currently being established and will be operational soon. Another collaboration, specifically for cancer guidelines, is also being prepared and will apply for funding under the European Union's 6th Framework Programme. Such networks will provide a platform for international information exchange and collaborative research. These efforts will apply to existing guidelines or guidelines under development, guidelines reviews, methodological information  and tools for application and evaluation of guidelines. We expect that many guideline organisations throughout the world will join the proposed networks.Clinical practice guidelines should meet specific quality criteria if they are to be valuable tools in the care for cancer patients. These criteria have been defined and validated by the AGREE Collaboration see . Better


David Moher and colleagues from the EQUATOR network offer guidance and recommended steps for developing health research reporting guidelines. Evidence-Based Medicine, a journal that is aimed at physicians working in primary care and general medicine. Treatment descriptions were inadequate in 41 of the original published articles, which made their use in clinical practice difficult if not impossible to replicate. This is just one of numerous examples of a large and disturbing literature indicating the general failure in the quality of reporting health research Publishing health research is a thriving, and increasing, enterprise. On any given month about 63,000 new articles are indexed in PubMed, the United States National Library of Medicine's public access portal for health-related publications. However, the quality of reporting in most health care journals remains inadequate. Glasziou and colleagues Inadequate reporting is problematic for several reasons. If authors do not provide sufficient details concerning the conduct of their study, readers are left with an incomplete picture of what was done. As such, they are not able to judge the reliability of the results and interpret them. There are also ethical and moral reasons for reporting research adequately http://www.equator-network.org) provides resources and training relating to the reporting of health research and assists in the development, dissemination, and implementation of reporting guidelines. As part of its initial resource development, the Network's Web site contains a comprehensive and up-to-date database of reporting guidelines relevant to heath research. A recent systematic review of 81 reporting guidelines found their development was often inadequate The EQUATOR  Network is a new international initiative seeking to improve the quality of scientific publications by promoting transparent and accurate reporting a checklist, flow diagram, or explicit text to guide authors in reporting a specific type of research, developed using explicit methodology. Some reporting guidelines recommend a flow diagram so that authors can clearly report information about sequential stages of their research project. A consensus process Reporting guidelines need to be differentiated from other efforts that produce a checklist or other guidance not specific to reporting research. We propose here a working definition of a reporting guideline: The main motivation for the development of reporting guidelines is to help researchers improve the completeness and transparency of their research reports and limit the number of poorly reported studies. However, reporting guidelines can be also used by peer reviewers and editors to strengthen manuscript review. And research funders can benefit from introducing reporting guidelines into the research application system In this paper we update and expand upon an earlier effort to outline a strategy for developing reporting guidelines that was published only in Spanish This strategy assumes the involvement of an executive group to facilitate the guideline development and the expectation of having a face-to-face meeting as part of the reporting guideline development. We propose 18 steps to occur in five phases, which are outlined in Developing a reporting guideline is complex and time consuming, so a compelling rationale is needed. Most reporting guidelines have been developed because researchers are convinced of the need to improve the quality of reporting of a certain type of health research. For some study aspects there may be direct evidence that inadequate reporting is associated with biased reports or harmful consequences. At this early stage, the executive group needs to set out clearly and explicitly their objectives and consider the scope of recommendations. For example, an early decision of the STROBE  group was to restrict consideration to three main epidemiological study designs Reporting guideline developers also need to consider whether a need exists to develop a new reporting guideline or to extend or implement an existing guideline.Developing a new reporting guideline assumes that there is no existing guideline on the topic under consideration. The development of STARD (STAndards for Reporting of Diagnostic accuracy), a reporting guideline for diagnostic accuracy studies, was undertaken because no guidance in this area existed previously and evidence suggested the need for one Sometimes there may be the view that an existing broad guideline can usefully be augmented by additional guidance for a specific set of studies. An example of such an extension is the recently published CONSORT  for nonpharmacological treatments (CONSORT for NPT) Most guidelines are defined by the study aims and methodology, leaving plenty of scope to illustrate their implementation in particular medical specialties. For example, recently a group implemented the CONSORT Statement for behavioral medicine http://www.equator-network.org). We recommend that reporting guideline developers interested in extending or implementing an existing guideline contact the authors and discuss their plans.At this early stage, prospective guideline developers should search for any existing reporting guideline covering all or part of the area being considered. We developed a search strategy to identify and characterize reporting guidelines as part of an ongoing project. In addition to electronic searching, developers are encouraged to search the EQUATOR Network database of reporting guidelines  meeting, among other activities we conducted a systematic review of studies that had reviewed the quality of reporting of systematic reviews. To help inform updating of the CONSORT Statement we search the literature every month for new studies of the reporting of randomized trials.For any guideline, both new and revised, there will be key pieces of information that must be included. For example, in a 1993 meeting that ultimately led to the development of the CONSORT Statement, we identified emerging empirical data on the importance of the methods and reporting of allocation concealment when describing the randomization process To date, no published data inform the costs required to develop a reporting guideline. Even if funding is available, most developers limit their fiscal requests to cover only the main reporting guideline meeting. In our experience, considerable resources are required for pre-meeting activities, such as literature searching and conducting a Delphi exercise, and post-meeting activities, including the need for a small writing group to be able to meet in person, perhaps on several occasions, during the drafting of the guidance and development of an explanation and elaboration (E&E) document.Depending on the extent of the pre-meeting activities we recommend allocating at least Can$10,000 for them. In our experience the main meeting costs are approximately Can$75,000. This covers the travel and accommodation costs of bringing together 20 to 30 participants. For new guidance in the form of a checklist, with a more detailed and extensive E&E, we recommend at least Can$35,000 for a small writing group to meet several times.There is no obvious choice as to where reporting guideline developers can seek funding; we are unaware of any granting agency or other group that has a specific remit to provide funding to those interested in improving ways to report health research. Of the 30 respondents to our survey Most reporting guidelines have been developed by an international multidisciplinary group involving 22 (median) people Well ahead of the proposed meeting date, a list of participants to invite should be developed. The CONSORT executive has typically invited participants in a two-stage process. First, we identify a small group of invitees whose participation we consider essential to hold a meeting. We invite this small group immediately. Once they have accepted our invitation and a date has been settled we invite the remaining participants. We also keep an additional smaller list of people to consider inviting should some second-wave invitees not be able to participate in the meeting. We recommend sending out the meeting invitations well in advance, ideally six months before the meeting date.Not all potential meeting participants can be invited or will be able to come to the main guideline development meeting. These people can still be engaged in the guideline development process. The CONSORT group has started using a Delphi consensus method n = 109) who were known to have an interest in the reporting of randomized trials, the structure of abstracts, or both were invited by e-mail to participate in a Web-based survey and rate the importance of 27 suggested checklist items selected from previous research. During three rounds of the survey, participants were asked about their views on the relative importance of the possible checklist items To help develop CONSORT for RCT abstracts, we used a three-stage Delphi process. Journal editors, health care professionals, methodologists, clinical trialists, and others with expertise in the reporting of RCTs . For example, the Delphi process for the SPIRIT  meeting resulted in 63 nominated items to consider for inclusion in the checklist discussed during the face-to-face consensus meeting. The executive may reduce the initial list of items to a more manageable number for discussion during the face-to-face meeting (see item 8).Regardless of the available funding we recommend keeping the size of the meeting to no more than about 30 participants. Larger meetings can lose their spontaneity and limit interaction among the participants, both important attributes of a successful meeting. The CONSORT for herbal medicines meeting was small, involving 16 participants. In contrast, the CONSORT for NPT meeting was large, at 32 participants. Here we were interested in many types of interventions, such as surgery and psychotherapy, requiring a broader number of clinical content experts. It is valuable to have several people who have participated in previous similar meetings.Reporting guideline meetings that we have been involved in have lasted from one to three days. We recommend a minimum of one day for developing a reporting guideline regardless of whether the guidance is new, an implementation, or an extension.The successful planning and implementation of any reporting guideline meeting requires thorough organization. If funding permits a coordinator should be hired to organize the meeting, including venue selection, meal plans, participant travel and accommodation, and finances.The executive must develop the form and structure of the meeting to ensure adequate time to discuss all of the agenda items. The most important outcome of the face-to-face meeting is an early version of the guidance checklist. We recommend that the following points be considered as part of developing the meeting agenda.7.3.1. Consider presentations on relevant background topics, including summary of evidence.A useful way to set the stage and maximize participant dialogue is to devote at least a few hours to presentations on topics underpinning the reporting guideline development. Presentations might also be usefully devoted to specific items that are being proposed for the checklist and any relevant empirical evidence.Ideally, by the conclusion of this session all participants should have up-to-date knowledge about the quality of reporting of the literature at which the guidance is aimed and the evidence relevant to considering the merits of including specific checklist items. Some of this material can be circulated in advance, but it is unlikely that people have time to read extensive materials. A major advantage of the Delphi exercise is that participants can think about some key issues before the meeting.7.3.2. Plan to share results of Delphi exercise, if done. We have found that a Delphi exercise provides important information when developing a reporting guideline, regardless of whether the guidance is new, an extension, or implementation. The results of a Delphi exercise were presented to participants during the first day of the following three meetings: CONSORT for abstracts, CONSORT for NPT, and SPIRIT.7.3.3. Invite session chairs.The main meeting will most likely be divided into sessions. While some of the chairs of sessions should be members of the executive group, other participants should also be invited to chair sessions, particularly if they have previous experience developing reporting guidelines and/or chairing meetings. It is essential that chairs have known ability to handle sessions effectively, to ensure that decisions are made.We recommend sending some materials to participants before the meeting, even though the same materials, and additional ones, should be readily available for each participant during the meeting. Sending the meeting agenda, participant list, one or two papers that might best highlight the quality of reporting of the content area, and the results of any Delphi exercise, if done, is a useful minimum and should be sent to the participants at least a week ahead of the meeting.The options include audio  recording of the entire event , hiring someone to take comprehensive minutes, or a combination of both. Depending on the meeting agenda, a more focused recording and/or minuting of certain parts of the meeting is another option. As a minimum we recommend comprehensive minuting of all discussions specifically related to the checklist development. Such minuting provides all participants with a record of events and decisions taken during the meeting.The meeting should follow closely the pre-meeting plans, although timings should be flexible. It is unlikely that all the participants will know each other, so it is helpful for everyone to introduce themselves and indicate the relevance of their particular experience. One of the first tasks of the meeting is to review the objectives and the way the meeting will run, and to clarify any outstanding issues among the participants.The substantive meeting begins with any formal presentations of background topics, empirical evidence from the literature, and results of any Delphi process (see item 5 above).The most detailed and structured discussions at the meeting revolve around which checklist items to include in the guideline; these discussions should focus on information content and not get distracted by seeking agreed wording at this stage. We have always considered the items included in a checklist to be a minimum essential set of items that should be reported, and some discussion of this perspective is valuable before the detailed discussions begin.The inclusion of each item is ideally supported by empirical evidence, when available. For example, the inclusion of allocation concealment as a checklist item of the original CONSORT Statement was based on an empirical study The views expressed in a Delphi exercise, and perhaps also how similar issues were handled in other reporting guidelines, are also important. Ultimately the views of the meeting participants will usually converge, although it may occasionally be necessary to vote on some issues. We recommend considering a classification scheme for selecting items for inclusion in the checklist; we provide an example of this approach that we've used for including items in the CONSORT checklist see .The majority of reporting guidelines have been limited to developing a checklist The most important deliverable is the final reporting guideline. Commonly this will be in the form of a journal article, written after the meeting that introduces the checklist (and flow diagram) and summarises the processes used to develop it. Some groups such as CONSORT refer to this document as a “Statement” to distinguish it from other types of publication. A detailed explanatory paper, an E&E, may also need consideration. Sufficient time needs to be included in the meeting agenda to discuss these activities. Developing an E&E is very time consuming, and will require further input from the meeting participants (see item 10). To accomplish this task we have often asked meeting participants, all of whom would be considered for authorship, to volunteer to help draft particular sections. Time needs to be set aside to discuss these issues during the meeting and the authorship model needs to be agreed upon.One of the last major sessions of the meeting should be devoted to issues pertaining to disseminating the reporting guideline. There are several points to consider here, especially a publication strategy. Since the simultaneous publication in three journals of the 2001 CONSORT Statement, several reporting guidelines have been published in multiple journals. Ultimately reporting guideline developers want to positively influence the reporting of health research and will need to consider how best to ask journals and editorial groups , for help achieving these goals. To address these issues adequately the PRISMA  Group formed a three-member dissemination committee. How to maximize journal endorsement and adherence to the reporting guideline can also be discussed during the meeting. Additional points for discussion can include: whether and how the guidance will be formally evaluated; how any criticism will be handled; and whether to create a Web site and what it might contain.Together with drafting and finalizing guideline documents the reporting guideline developers need to consider their implementation strategy, including publication issues , website development, and seeking or monitoring endorsement of and adherence to the reporting guideline.We recommend that initial efforts be focused on drafting the checklist for the proposed reporting guideline. Efforts aimed at extending or implementing existing reporting guidelines should make very clear which parts of their checklist remain the same as the original and which items have been modified or added. The process of developing the checklist is very likely to require several iterations. Initially, this entails taking the discussions from the face-to-face meeting (item 8.1 above) and crafting each item into a crisply and unambiguously worded checklist item. The most appropriate order for the checklist items also needs consideration.For a new reporting guideline we recommend a short document of about 2,000 words reporting on the rationale for developing the guidance and the development process, including a brief description of the meeting and participants involved. The article will include the checklist and flow diagram, if developed. In our experience drafting the checklist and article is best done by a small writing group made up primarily of members of the executive team and perhaps one or two others. This group will usually constitute the guideline authors on behalf of the reporting guideline group, although group authorship may be preferred. For extending an existing reporting guideline a similar approach is recommended. Whether a new guideline is being developed or an existing one is being extended, the full guideline group will need to sign off on the document's content prior to journal submission.Pilot testing the checklist and flow diagram is worth considering. During the development of the QUOROM Statement, two members of the executive group independently asked two separate sets of students taking a systematic review course for their feedback on the checklist that was used to help report their systematic review. Their comments were incorporated into the checklist revisions.To date most reporting guidelines have not been accompanied by a detailed justification and explanation of the recommendations . However, we believe that it is vitally important to provide an explanation of the rationale and evidence for a guideline item's inclusion and an elaboration on the details of an item, hence the title Explanation and Elaboration, or E&E, adopted by CONSORT and several other groups.While there will undoubtedly be an urge to complete all of the guideline revisions rapidly and submit the paper for publication consideration, we recommend holding off doing so until an E&E has been prepared (see Box 1). The E&E will be considerably longer than the guideline statement . As with the short guideline statement (see item 13) the full guideline group will need to sign off on the E&E's content before journal submission.A reporting guideline gives a set of recommendations regarding the information that should be included in the report of a research study. It may also include guidance on how some information may best be presented .Guideline developers aim to persuade editors and authors of the importance of adhering to their recommendations. To that end, simply providing a list of requirements, often as a checklist, is unlikely to be sufficient. Such a terse presentation may seem to make unsupported, even dogmatic statements without clear support, even when supported by relevant references to other publications. That concern underpinned the decision of the CONSORT Group to accompany the revised CONSORT Statement The E&E was intended to provide detailed rationales for all of the items in the CONSORT checklist and flow diagram. For each item, the paper included (a) an example of good reporting from a published paper, (b) the scientific background and rationale for including that information in a published article, (c) empirical evidence of bias associated with the way that aspect of a study is conducted or reported, and (d) any evidence relating to the extent of inadequate reporting of that information. In addition, some fundamental concepts that underpinned several of the items were discussed in boxes.The novel format of the CONSORT E&E was emulated for several later reporting guidelines—STARD, STROBE, and PRISMA We feel that simultaneous publication of the statement along with an E&E is the most important way to disseminate the work of the reporting guideline group. For limited extensions a single article may best cover both roles, as was done, for example, for CONSORT for herbal interventions We recommend that guideline developers negotiate a copyright agreement to retain the rights of the contents of the statement and related documents, such as the E&E. We have not found journals to be resistant to this idea. Similarly, we recommend developers to negotiate being able to put the reporting guideline on a dedicated website.Starting with the revised CONSORT Statement in 2001, several developers have published their guidance in multiple journals simultaneously to enhance the uptake and dissemination of their reporting guideline . For example, the STARD Statement for reporting diagnostic accuracy studies is relevant to a broad spectrum of potential users, from clinical chemists to radiologists, all of whom might not be aware of each other's literature. Publishing in a radiology journal Our experience is that the journal submission and peer review processes for multiple simultaneous publications can be exhausting. Some steps can reduce the burden. For example, the guideline author group might want to nominate an author as corresponding author to facilitate the coordination process with the various journals. Before submission it might be useful for the reporting guideline authors to discuss with prospective journals their interest in publishing the guidance and the need for all journals to agree a common text. When submitting the final manuscript the journals can be asked about their willingness to have a common set of peer reviewers. Such a move enables the reporting guideline developers to respond to a single set of peer reviewers rather than separately to peer reviewers from each journal. Similarly, to ensure consistency across journals we believe that copy editing should be limited to spelling, layout, and punctuation, ensuring identical text across journals. This might be best achieved by having one lead journal take care of these issues and circulating the copy edited version across the other journals publishing the article. The PRISMA Statement, recently published in five journals It is important to seek feedback and criticism from all stakeholders regarding the reporting guideline, and we have outlined several stages by which this can be achieved in the development process. We recommend that guideline developers also encourage feedback at any time after publication, either directly or via the related Web site. Constructive criticism can help improve the reporting guideline if an update is prepared . After the CONSORT Statement was originally published, Meinert Some journals, those whose editors initially were involved in the guideline development, will be eager to support the use of the reporting guideline. This support can be most readily achieved when a journal endorses the reporting guideline. However, some journals have used inconsistent language to describe their endorsement http://www.biomedcentral.com/bmcdermatol/ifora/).When journals want to endorse a reporting guideline we recommend that they use strong, clear language of their expectations of authors, and include this information in their Instructions to Authors. For example, BioMed Central, the publisher of 199 open access journals, states, “We recommend authors refer to the EQUATOR network website for further information on the available reporting guidelines for health research, and the MIBBI Portal for prescriptive checklists for reporting biological and biomedical research where applicable. Authors are requested to make use of these when drafting their manuscript and peer reviewers will also be asked to refer to these checklists when evaluating these studies” . We recommend that journals consider ways to maximise adherence to reporting guidelines, such as by asking authors to submit completed checklists and by asking peer reviewers to use them as part of their review.We recommend assessing the impact of any reporting guideline. Unfortunately, few guidelines appear to have been evaluated to date Another implementation strategy is to create a Web site dedicated to the reporting guideline. It makes the most sense to develop the Web site before publication so that the address can be included in the published articles. As a minimum we recommend including on the Web site the reporting guideline checklist, statement, ancillary documents such as an E&E, and any translated versions of the guideline. Also, the list of participants and funders should be included. The checklist and diagram should be available as both PDF and DOC files. Related unpublished documents can be published here too. The Web site is also a useful venue for alerting readers about emerging issues related to the guidance and perhaps inviting comments and discussion. We also recommend including a news section where new information about the guidance can be posted. Unfortunately, the lack of funding may impede such valuable additions.We recommend including an endorser section for the names of journals and editorial groups endorsing the reporting guideline. The journals' Instructions to Authors can be linked to their endorsement.http://www.equator-network.org).The Web site can also provide policy documents regarding the use of the guidance. Issues pertaining to copyright, use of documents and, if relevant, the permitted use of the guidance logo if there is one, can be included in this section. Finally, we encourage guideline developers to consider linking their guideline with the EQUATOR Network  guideline developers recognize the need to do so We recommend regular consideration of whether or not to update a reporting guideline and that reporting guideline groups maintain an executive group that can help make such decisions. Although it is possible to make frequent  small updates, we believe this is not wise and it may even be counterproductive. Making occasional major updates, with clear version numbers, seems to be a better approach. We convened a one-day meeting to update the CONSORT Statement and two-day meeting to update QUOROM (subsequently published as the PRISMA Statement).We assume that updating an existing reporting guideline will be completed by essentially the same team that initially developed the guidance, although some of the membership may change over time. That has been our experience from the CONSORT Statement updates. We also assume that several components of the initial guideline development process are still in place. For example, for updating the guideline, the developers would need to update their existing literature searches.When considering an update to an existing reporting guideline, developers should give serious consideration to the seven essential items (denoted with an †) in We also encourage reporting guideline developers to seriously consider how best to handle numbering and dates of updates. For example, the original version of CONSORT did not number the checklist items. We introduced numbering into the 2001 version, which meant that we have had to be careful in numbering new checklist items in the latest (2010) update. We have modified the numbering slightly. Updating a reporting guideline will influence checklist items of any published extensions and/or implementations, and due consideration is required as to how best to handle these issues.We hope this guidance on how to develop a reporting guideline, including an 18-step checklist, will fill a gap in the literature and be of help to potential and practicing developers. While some of the items are optional, there is a core set of steps we believe necessary to ensure adequate development of a reporting guideline. In the future, the EQUATOR Network might consider providing an appraisal grade for the reporting guidelines included on their database, reflecting the robustness of the guideline development process. We also encourage prospective guideline developers to contact the EQUATOR Network team to inform them about their work; this might prevent duplication of effort.We will make the 18-step checklist available on the EQUATOR Network Web site. We will continue to monitor the literature to help maintain the guidance, particularly the checklist. Furthermore, we will annually review the need to update the checklist.While there is increasing evidence that use of reporting guidelines is associated with improvements in the quality of reporting health research http://www.rdinfo.org.uk/) to guide researchers in how to start developing a research project. The flowchart includes reference to reporting guidelines and encourages researchers to “consult a relevant guideline in the early stages of research planning.”Reporting guidelines are currently focused at the end of the knowledge generation cycle. However, we believe that investigators would benefit from the knowledge of key principles of health research reporting and relevant guidelines at the beginning of their research, having the end in mind. Indeed, guidelines are possibly equally useful earlier on in this process, and some granting agencies have acted upon this concept. The SPIRIT initiative is aimed at providing guidance for writing protocols of RCTs. Similarly, the UK National Institute of Health Research developed a research process flowchart (Given the broad range of health research now covered by reporting guidelines, funders might start to require prospective applicants to use some parts of an “approved” reporting guideline when developing their research application and to include a completed checklist as part of this process. This might increase the overall quality of the applications to funders while increasing the potential return on their investment by emphasizing the importance of reporting much earlier in the knowledge generation cycle.Reporting health research in a complete, accurate, transparent, and timely manner is a shared responsibility of all stakeholders involved in research funding, conduct, and publication. High-quality research reports contribute to more efficient translations of new research findings into clinical practice and help advance scientific knowledge and patient care. We will all benefit from these collective efforts.


Organizations have been established in many countries and internationally to support the use of research evidence by producing clinical practice guidelines, undertaking health technology assessments, and/or directly supporting the use of research evidence in developing health policy on an international, national, and state or provincial level. Learning from these organizations can reduce the need to 'reinvent the wheel' and inform decisions about how best to organize support for such organizations, particularly in low- and middle-income countries (LMICs).We undertook a multi-method study in three phases – a survey, interviews, and case descriptions that drew on site visits – and in each of the second and third phases we focused on a purposive sample of those involved in the previous phase. We used the seven main recommendations that emerged from the advice offered in the interviews to organize much of the synthesis of findings across phases and methods. We used a constant comparative method to identify themes from across phases and methods.Seven recommendations emerged for those involved in establishing or leading organizations that support the use of research evidence in developing health policy: 1) collaborate with other organizations; 2) establish strong links with policymakers and involve stakeholders in the work; 3) be independent and manage conflicts of interest among those involved in the work; 4) build capacity among those working in the organization; 5) use good methods and be transparent in the work; 6) start small, have a clear audience and scope, and address important questions; and 7) be attentive to implementation considerations, even if implementation is not a remit. Four recommendations emerged for the World Health Organization (WHO) and other international organizations and networks: 1) support collaborations among organizations; 2) support local adaptation efforts; 3) mobilize support; and 4) create global public goods.This synthesis of findings from a multi-method study, along with the more detailed findings from each of the three phases of the study (which are reported in the three following articles in the series), provide a strong basis on which researchers, policymakers, international organizations (and networks) like WHO can respond to the growing chorus of voices calling for efforts to support the use of research evidence in developing health policy. Over the past three years, there has been a great deal of international discussion about how to harness health research evidence more effectively in order to achieve the United Nations' millennium development goals as well as national health goals, particularly in low- and middle-income countries (LMICs). One important focus in this discussion has been the call to develop mechanisms to support the use of research evidence in developing clinical practice guidelines, health technology assessments, and health policy. The chapter on linking research to action in the World Report on Knowledge for Better Health that was released by the World Health Organization (WHO) in early November 2004 provided a framework for appreciating the diversity and complementarities of many of these mechanisms . The heaAt the World Health Assembly that was held in Geneva in May 2005, these debates culminated in the passage of a two-part resolution that established specific accountabilities for developing mechanisms to support the use of research evidence in developing health policy . The firOrganizations have already been established in many countries and internationally to support the use of research evidence in developing health policy. These include organizations that produce clinical practice guidelines (CPG), undertake health technology assessment (HTA), and directly support the use of research evidence in developing health policy on an international, national, and state or provincial level . While there may be important differences among these organizations, there may also be many commonalities and opportunities for existing and new organizations to learn from this collective experience. A review of this experience can reduce the need to 'reinvent the wheel' and inform decisions about how best to organize support for evidence-informed health policy development processes.An important dimension of the context for any such review is the growing demand for more rigorous processes to ensure that decision-making is well informed by the best available research evidence. These processes, in contrast with traditional approaches that rely heavily on the opinions of experts, demand systematic and transparent approaches to access, synthesise, and interpret research evidence and to integrate that evidence with other information, values, and judgements in order to formulate recommendations or make decisions. The need for more rigorous processes to support clinical decision-making is underscored by evidence of inconsistencies between the available research evidence and expert recommendations ,5, insufOur objective was to identify organizations around the world, and especially in LMICs, that are in some way successful or innovative in supporting the use of research evidence in the development of CPGs, HTAs, and health policy, and to describe their experiences. We pursued this objective in a three-phase, multi-method study . In thisIn order to support our primary focus on LMICs, we convened a project reference group that drew on two or three individuals who were from, or who are very knowledgeable about, Africa, Asia, or Latin America, as well as individuals from North America, Europe (including a representative from the project funder), and WHO . Collectively, the reference group provided many perspectives on the three types of organizations under study and on potential country- or region-level differences in the opportunities and challenges confronting these organizations. The reference group provided feedback on our draft protocol, study population, questionnaire, interview guide, and case study data collection plan. We also engaged one individual from each of Africa, Asia, Europe, and Latin America to provide a very detailed review of the draft final report on which this series of articles is based.We undertook the project in three phases – a survey, interviews, and case descriptions that drew on site visits – and in each of the second and third phases we focused on a purposive sample of those involved in the previous phase. We drew on many people and organizations around the world to generate a list of organizations to survey. We modified a questionnaire that had been developed originally by the Appraisal of Guidelines, Research and Evaluation in Europe (AGREE) collaboration, adapted one version of the questionnaire for organizations producing CPGs and HTAs and another for GSUs, piloted both versions of the questionnaire, and made a small number of final modifications to both versions of the questionnaire . We sentWe used the seven main recommendations that emerged from the advice offered in the interviews to organize much of the synthesis of findings across phases and methods. We chose this organizing framework for three reasons: 1) our interest is in the views and experiences of particularly successful or innovative groups (particularly those based in LMICs), and the interviews allowed us to balance breadth and depth in soliciting these views and experiences; 2) the thematic analysis of the interview data yielded clear recommendations for other organizations; and 3) the analysis of the quantitative survey data, the written survey responses , and the case descriptions reinforced the broad applicability of the organizing framework. We used a constant comparative method to identify from across phases and methods themes relevant to WHO and other international organizations and networks.The principal investigator for the overall project (AO), who is based in Norway, confirmed that, in accordance with the country's act on ethics and integrity in research, this study did not require ethics approval from one of the country's four Regional Committees for Medical and Health Research Ethics.Seven recommendations emerged from the multi-method study for those involved in establishing or leading organizations that support the use of research evidence in developing health policy:This advice was reinforced by: 1) the (quantitative) survey finding that more than half of the organizations (and particularly HTA agencies) reported that examples from other countries were helpful in establishing their organization; 2) the  survey finding that many organizations producing CPGs or HTAs conducted a focused review of one particular organization that they then emulated or a broad review of a variety of organizational models; 3) the  survey finding that the advice that was most commonly offered by organizations producing CPGs, HTAs, or both was to seek support from similar existing organizations or networks, whether through informal interactions, study tours, mentoring relationships, twinning, partnerships or network memberships; 4) the  survey finding that working within national networks and, more generally, collaborating rather than competing with other bodies, was a commonly cited strength in how GSUs are organized; and 5) the case descriptions finding that one of the two types of advice offered to other organizations was to learn from other organizations.This advice was reinforced by: 1) the (quantitative) survey finding that a high proportion (88%) of GSUs involved target users in the selection of topics or the services undertaken; 2) the interview finding that, while informal relationships with policymakers were identified more frequently as important by GSUs than by organizations producing CPGs, HTAs, or both, nearly all of the organizations reported using personal communications with decision-makers, and particularly with policymakers; 3) the interview finding that organizations both within and outside government viewed their close links with policymakers as a strength; and 4) the case descriptions finding that the existence of a strong relationship between researchers and policymakers was repeatedly cited as one of two key organizational strengths .This advice was reinforced by: 1) the  survey finding that independence is by far the most commonly cited strength in how organizations producing CPGs and HTAs are organized; and 2) the case descriptions finding that the presence of conflicts of interest was repeatedly cited as one of two key organizational weaknesses.This advice was reinforced by: 1) the (quantitative) survey finding that most organizations have a small number of full-time equivalent (FTE) staff; 2) and the case descriptions finding that developing capacity among and retaining skilled staff and collaborators was one of their two frequently offered types of advice.This advice was reinforced by: 1) the (quantitative) survey finding that between 84% and 100% of organizations reported providing panels with or using systematic reviews; 2) the  survey finding that an evidence-based approach is the most commonly cited strength of the methods used by organizations that produce CPGs and HTAs; 3) the interview finding that using rigorous methods that are systematic and transparent (sometimes shortened to 'being evidence-based') was the most commonly cited strength among all organizations; and 4) the case descriptions finding that the use of an evidence-based approach was one of two organizational strengths that were repeatedly cited. However, all but one of the organizations producing CPGs, HTAs, or both used informal methods for setting priorities. Relatively few organizations producing CPGs and HTAs convened groups to develop CPGs or HTAs, took equity considerations into account, or had established a process for addressing conflicts of interest. GSUs were less likely to have a manual that described the methods they use and to conduct or use systematic reviews and more likely to report using non-systematic methods to review the literature. In addition, using systematic and transparent methods brought with it a related challenge, namely the time-consuming nature of an evidence-based approach.This finding was reinforced by: 1) the  survey finding that the most commonly cited weakness in how these organizations are organized is a lack of resources, both financial and human; 2) the  survey finding that the most commonly cited weakness of the methods used by organizations that produce CPGs and HTAs was their time-consuming and labour-intensive nature; 3) the  survey finding that GSUs advised others establishing a similar organization to attend to the need for secure funding; 4) the interview finding that the weakness noted by most of the CPG- and HTA-producing organizations was inadequate resources, more specifically insufficient numbers of skilled staff and time, together with using labour- and time-intensive processes that limit the number and quality of CPGs and HTAs that can be produced and updated; and 5) the case descriptions finding that a lack of resources was repeatedly cited as one of two organizational weaknesses.This advice was reinforced by: 1) the (quantitative) survey finding that less than half of all organizations provided a summary of take-home messages in their products; 2) the (quantitative) survey finding that between one-half and two-thirds of organizations do not collect data systematically about uptake; 3) the  survey finding that the most commonly cited weaknesses of CPG- and HTA-producing organizations' outputs are the lack of dissemination and implementation strategies for the outputs, and the lack of monitoring and evaluation of impact; 4) the interview finding that most organizations argued that it is the clients who requested a CPG or HTA – typically, the minister of health or more generally the department of health – who is responsible for implementing recommendations or policy decisions; 5) the interview finding that all types of organizations tended to focus largely on weaknesses in implementation when asked about both strengths and weaknesses, with few exceptions; and 6) the interview finding that most of the examples of success among organizations producing CPGs, HTAs, or both were occasions where there was a perception that clinicians adhered to the organization's recommendations or policymakers based their decisions (at least in part) on the work of the organization.Four recommendations emerged from the multi-method study for WHO, most of which were equally relevant for other international organizations and for networks such as the Guidelines International Network and the International Network of Agencies for Health Technology Assessment. The first recommendation was to support collaborations among organizations. This advice is supported by: 1) the  survey finding that many CPG- and HTA-producing organizations argued that WHO should play a facilitating role in coordination efforts, primarily to avoid duplication; and 2) the interview finding that when comments about WHO's potential role were offered they almost always pertained to the need to foster collaborations across organizations. The second recommendation was to support local adaptation efforts. This advice is supported by: 1) the  survey finding that some CPG- and HTA-producing organizations argued that WHO should play a facilitating role in local adaptation efforts in order to enhance local applicability; and 2) the  survey finding that some GSUs argued that WHO should play a role in helping to adapt global evidence to local contexts or at least in supporting such processes. The third recommendation was to mobilize support. This advice is supported by the case descriptions finding that one of only two suggestions that were offered with any frequency was that WHO should play a role in mobilizing one or more of government support, financial resources, and the participation of both policymakers and researchers. And the fourth recommendation was to create global public goods, which was supported by the case description finding that the second of only two suggestions that were offered with any frequency was that WHO should play a role in creating knowledge-related global public goods.By drawing on three phases of data collection and multiple methods, we identified seven recommendations for those involved in establishing or leading organizations that support the use of research evidence in developing health policy, particularly in LMICs: 1) collaborate with other organizations; 2) establish strong links with policymakers and involve stakeholders in the work; 3) be independent and manage conflicts of interest among those involved in the work; 4) build capacity among those working in the organization; 5) use good methods and be transparent in the work; 6) start small, have a clear audience and scope, and address important questions; and 7) be attentive to implementation considerations even if implementation is not a remit. We also identified four recommendations for WHO and other international organizations and networks: 1) support collaborations among organizations; 2) support local adaptation efforts; 3) mobilize support; and 4) create global public goods. We provide additional details about both methods and findings in the following three articles in this series -22.i.e., we surveyed GSUs as well as CPG- and HTA-producing organizations, we interviewed roughly equal numbers of CPG- and HTA-producing organizations and GSUs, and the majority of case descriptions were GSUs); 2) we achieved both breadth (through a survey) and depth  in our examination of their views and experiences; 3) we drew on a regionally diverse project reference group to ensure that our draft protocol, study population, questionnaire, interview guide, and case description data collection protocol were fit for purpose; 4) we adapted a widely used questionnaire and achieved a high response rate with our survey (86%); 5) we used explicit sampling criteria to identify particularly successful or innovative groups for more in-depth study through interviews and case descriptions, no organization declined to participate in the interviews, and only one individual declined to participate in the interviews conducted as part of the site visits; and 6) we employed a variety of independent checks on the credibility of our thematic analyses of the written questionnaire responses and the interview and case descriptions data. The study has two main weaknesses: 1) despite significant efforts to identify organizations in LMICs, just over half (54%) of the organizations we surveyed, and just under half (48%) of the organizations we interviewed, were drawn from high-income countries; and 2) despite efforts to ask questions in neutral ways, many organizations may have been motivated by a desire to tell us what they thought we wanted to hear .The multi-method study has six main strengths: 1) we examined the views and experiences of those familiar with three types of organizations that support evidence-informed policymaking, not just one of the two types of organizations previously studied .A similar assessment should be repeated in a few years, by which time many new and promising organizational forms for supporting the use of research evidence in the development of health policy, such as the WHO-sponsored Evidence-Informed Policy Networks, will have been established . These nThe authors declare that they have no financial competing interests. The study is part of a broader suite of projects undertaken to support the work of WHO Advisory Committee on Health Research (ACHR). Both JL and AO are members of the ACHR. JL is also President of the ACHR for the Pan American Health Organization . The Chair of the WHO ACHR, a member of the PAHO ACHR, and several WHO staff members were members of the project reference group and, as such, played an advisory role in study design. Two of these individuals provided feedback on the penultimate draft of the report on which the article is based. The authors had complete independence, however, in all final decisions about study design, in data collection, analysis and interpretation, in writing and revising the article, and in the decision to submit the manuscript for publication.JL participated in the design of the three-phase study, participated in analyzing the qualitative data and deciding how to present the quantitative data, and drafted the article and the report in which it is based. AO conceived of the study, led its design and coordination, participated in analyzing the qualitative data, and contributed to drafting the article. RM participated in the design of the study, led the data collection for the second and third phases of the study, led the analysis of the qualitative data, and contributed to drafting the article. EP led the data collection for the first phase of the study, contributed to data collection for the other two phases, and led the analysis of the quantitative data. All authors read and approved the final manuscript.