Dengue is a growing problem both in its geographical spread and in its intensity, and yet current global distribution remains highly uncertain. Challenges in diagnosis and diagnostic methods as well as highly variable national health systems mean no single data source can reliably estimate the distribution of this disease. As such, there is a lack of agreement on national dengue status among international health organisations. Here we bring together all available information on dengue occurrence using a novel approach to produce an evidence consensus map of the disease range that highlights nations with an uncertain dengue status.A baseline methodology was used to assess a range of evidence for each country. In regions where dengue status was uncertain, additional evidence types were included to either clarify dengue status or confirm that it is unknown at this time. An algorithm was developed that assesses evidence quality and consistency, giving each country an evidence consensus score. Using this approach, we were able to generate a contemporary global map of national-level dengue status that assigns a relative measure of certainty and identifies gaps in the available evidence.The map produced here provides a list of 128 countries for which there is good evidence of dengue occurrence, including 36 countries that have previously been classified as dengue-free by the World Health Organization and/or the US Centers for Disease Control. It also identifies disease surveillance needs, which we list in full. The disease extents and limits determined here using evidence consensus, marks the beginning of a five-year study to advance the mapping of dengue virus transmission and disease risk. Completion of this first step has allowed us to produce a preliminary estimate of population at risk with an upper bound of 3.97 billion people. This figure will be refined in future work. Previous attempts to map the current global distribution of dengue virus transmission have produced variable results, particularly in Africa, reflecting the lack of accuracy in both diagnostic and locational information of reported dengue cases. In this study, instead of excluding these less informed points we included them with appropriate uncertainty alongside other diverse evidence forms. After assembling a comprehensive database of different evidence types, a weighted scoring system calculated “evidence consensus” for each country a continuous measure of the certainty of dengue presence or absence when considering the full aggregate of evidence. The resulting map and analysis helped highlight important evidence gaps that underlie uncertainties in the current distribution of dengue. We also show the importance of local knowledge through incorporating questionnairebased responses that can help add clarity in uncertain regions. This analysis showed that presence/absence maps do not sufficiently highlight the uncertainties in the evidence base used to construct them. Mapping by evidence consensus not only encourages greater data inclusion, but it also better illustrates the current global distribution of dengue. Consensus mapping is thus ideal for a range of neglected tropical diseases where the evidence base is incomplete or less diagnostically reliable. Despite increased interest in dengue in recent years, the global distribution of dengue remains highly uncertain. Estimates for the population at risk range from 30% There is an urgent need to compile more extensive occurrence records of dengue virus transmission and assess them for contemporariness and accuracy. Evidence on dengue transmission comes in a wide variety of forms, with varying levels of spatial coverage and reliability. A global audit of dengue distribution therefore requires a transparent methodology to compile these disparate data types and synthesise an output map summarising the current consensus for each country. Such a methodology for compiling and assessing evidence must be robust, repeatable, able to evaluate a large variety of evidence types and incorporate expert opinion. An ideal output metric is a summary statistic (hereafter referred to as evidence consensus) that quantifies certainty on dengue virus transmission presence or absence given the accuracy and contemporariness of the evidence available. An evidence-based map of the current distribution of dengue virus transmission will have direct implications for design and implementation of dengue surveillance and, by showing gaps in contemporary knowledge, provide an advocacy platform for improved data.Plasmodium falciparumP. vivaxet al., who reviewed published literature to contrast historic, current and future limits of dengue Existing approaches to mapping the global limits of vector-borne diseases have used estimates of biological suitability of local environments, which have proved informative in the cases of some pathogens, such as This study is the initial part of a five year project to collect, analyse and publicise global dengue virus transmission data. While the map presented here is the most extensive display of current dengue evidence available, we hope that continual data acquisition will result in more evidence from uncertain areas, increasing the resolution at which we can map evidence consensus in future advances.Evidence for indigenous dengue virus transmission was obtained from four evidence categories: health organisations, peer-reviewed evidence, case data and supplementary evidence . The firCountry dengue status as defined by health organisations was determined by consulting the WHO http://www.ajol.info/) and China National Knowledge Infrastructure (http://en.cnki.com.cn/) were also searched. Extra publications were found by searching using the location term in Genbank nucleotide records for dengue viruses isolated from human hosts. The search of peer-reviewed sources of evidence resulted in a total of 285 articles being selected for 123 countries where positive dengue occurrence records were identified. This included evidence from returning travellers who were diagnosed upon return to their often non-endemic home countries as opposed to the transmission setting. For these cases, evidence was attributed to the place to which they had travelled. The added value of returning traveller reports is that the travellers are often more immunologically naïve to dengue infections, and also that diagnosis is often pursued more rigorously. Therefore, the sensitivity of detecting an infection is increased. The results of our search were then cross-referenced against a dengue occurrence-point database compiled internally, in a separate exercise. Unlike our country-specific searches, this database of 2836 articles results from searches simply for “dengue”, which were then geo-referenced using the article text. Full details are available in The peer-reviewed evidence category contained evidence of dengue occurrence as determined by peer-reviewed sources where details of diagnostic techniques were given. Peer-reviewed journal  and disease surveillance network  searches were conducted with search terms “country” or “Admin1/2” and “dengue”. Sources were included for the period 1960–2012 and only if cases were confirmed as resulting from indigenous (i.e. not imported) transmission. The specialist regional journal collections African Journals Online (The case data category contained evidence of dengue outbreaks (minimum 50 infections) where evidence contained less diagnostic detail, but was more informative about the magnitude of dengue transmission occurring. Case data from the most recent outbreak were obtained from the Program for Monitoring Emerging Diseases (ProMED) archive search, WHO DengueNet data query Aedes aegypti, Ae. albopictus or Ae. polynesiensis) as documented by peer-reviewed literature, confirmed presence of multiple other rarely diagnosed arboviral diseases as documented by peer-reviewed literature, news reports of dengue epidemics found using GoogleNews archives (http://news.google.co.uk/archivesearch) and travel advisories from the National Travel Health Network and Centre (http://www.nathnac.org/ds/map_world.aspx) issued at a country-level. We included evidence of multiple other rarely diagnosed arboviral diseases, as these are informative about the ability of a country to detect any possible dengue infection. If other arboviral diseases are poorly reported, but documented by peer-reviewed literature as present, then it is possible that dengue is also underreported. In addition to this, we cross-referenced our dataset with the HealthMap database . This website-based application automatically geo-positions cases from websites with news reports and outbreak alerts related to dengue and contains data from a wide variety of sources dating back to 2007 In many resource-poor countries, both surveillance and researcher-generated reports are rare. Therefore, in countries where other evidence categories were sparse, we looked for supplemental evidence that suggested possible dengue virus presence. Supplemental evidence types included: presence of an established mosquito vector population of public health significance  comprised discrete presence or absence answers. A consensus (+++ or −−−) scored 6 or −6 respectively, while a lack of consensus (++− or −−+) scored 3 or −3 respectively . This gaThese forms of evidence were each scored independently for contemporariness and accuracy. The date of occurrence was used for scoring as follows: between 2012–2005 = 3, 2004–1997 = 2 and pre-1997 = 1 . This coThis category was scored by contemporariness in eight-year intervals. The most frequent year in which an outbreak  occurred was again scored in average inter-epidemic period intervals: 0–7 years since the last outbreak scored 9, 7–14 years = 6, 14–21 = 3, 21–28 = −3, 28–35 = −6, 35+ = −9 . Where cThis formed part of the evidence base if there was some suggestion of dengue presence, but the above three categories were insufficient to provide certainty on dengue status. If only two evidence types were available (see above), a score of 2 was given, three types = 4, four types = 6 . SupplemWhere a national score showed some uncertainty and an additional factor existed that was not captured by the default scoring system, an adjustment of up to ±3 was applied. For example, if multiple evidence categories suggested dengue presence in a country with high HE, but there was no case data, then the case data score was adjusted so as not to hold a disproportionate weight in deciding overall dengue status. This is termed the “ad hoc adjustment” .To derive an overall country evidence consensus score, the scores for all evidence categories were summed, and then divided by the maximum possible score and multiplied by 100. Evidence consensus was then mapped according to nine equal interval categories from 100% to −100% that differentiated evidence consensus worldwide, with evidence consensus being defined as complete (±79% to ±100%), good (±57% to ±78%), moderate (±34% to ±56%), poor (±12% to 33%) or indeterminate (−11% to 11%). An odd number of intervals was chosen so as to highlight places where consensus is very low (indeterminate) and where improved surveillance is particularly needed. As such, the resulting classification of consensus scores should not be strictly interpreted, but rather taken as a general indication of the quality of dengue evidence in a given location. A full breakdown of the exact evidence included, individual scores and overall consensus percentages are given for each country in In countries where evidence consensus was at best moderate, we attempted to increase consensus through targeted questionnaires. The questionnaire asked about endogenous surveillance and data collection. If available, diagnostic method(s) and summary results were requested. Any returned data or reports were then entered into their relevant evidence categories and scored in combination with existing evidence. Questionnaires were distributed to healthcare officials in the country of interest as well as selected offices of the Institut Pasteur. Questionnaire responses and expert comments are part of an on-going process that will lead to future modifications of this map.To map public awareness of dengue worldwide, we searched the ministry of health websites of each of the 128 countries identified as dengue-present (evidence consensus positive but not indeterminate). A country was indicated as publicly displaying dengue data if national dengue case numbers were displayed annually or during epidemic years at a minimum.beta version provides gridded population count estimates at a 1×1 km spatial resolution for the year 2000 To calculate the maximum possible population at risk for dengue virus transmission we obtained total population counts from the Global Rural Urban Mapping Project (GRUMP) for the 128 countries identified as dengue-present. The GRUMP The global distribution of dengue virus transmission as defined by evidence consensus is shown in Even after performing the sensitivity analysis described earlier, the number of countries defined by our methodology as dengue-present but defined by WHO/CDC as absent never dropped below 36 . We therThe majority of these newly identified dengue-present countries were in Africa and the evidence type that allowed greatest identification was returning traveller reports. These sporadic reports established preliminary evidence, which we improved with supplementary evidence and questionnaire retrieval to clarify dengue status if possible . OutsideA total of 3.97 billion people live in these 128 countries outside the indeterminate consensus class. Of these, 824 million live in urban and 763 million in peri-urban areas. These numbers therefore constitute plausible preliminary estimates for the maximum possible population at any risk of dengue transmission. We expect more comprehensive population at risk calculations to refine this figure and quantify levels of risk in our future work, allowing us to give a more accurate estimate.Public display of dengue data varied by continent . In totaDengue presence is well documented in the Americas with a continuous set of good- or complete- consensus countries from southern Brazil to the Mexico-U.S.A. border . HoweverOther regions of uncertainty reflect dynamic dengue status at the limits of the disease distribution. Lower consensus estimates in areas of Florida and Argentina result from reliance on smaller amounts of evidence from recent epidemics. Although the disease extent is better described in Florida (both in terms of resolution and consensus) due to greater data availability, uncertainty is still present due to the unknown persistence of recent events. A similar pattern of uncertainty exists in Texas but for different reasons, being that the occurrence evidence is older and six of seven counties have no record of occurrence since the late 1980s.A total of 58% of Africa+ countries had a good consensus or better but Africa still showed the highest levels of uncertainty in countries with poor consensus. Concentrations of higher consensus were identified in East and West Africa . MultiplOutside of these higher-consensus regions, evidence consensus is low and a series of countries with moderate or worse consensus can be identified from Chad to Mozambique with only the Democratic Republic of Congo exhibiting good evidence consensus. For many of these countries, there are sporadic reports of dengue occurrence combined with poor disease surveillance and a general lack of data. Dated seroprevalence surveys in areas where many other arboviruses are circulating did little to increase certainty. These factors result in a positive evidence consensus that is nevertheless highly uncertain in large portions of Africa. Even where evidence was available from contemporary epidemics, such as in the case of the western Indian Ocean islands, it was often devalued because there was a lack of clinical differentiation between dengue and chikungunya despite epidemics coinciding. The lack of clear clinical distinction between the two diseases Despite the widespread uncertainty in dengue status in many African countries, we were able to differentiate multiple levels of uncertainty. Angola and Mozambique both show lower consensus due to dated evidence forms, yet they are still distinguishable from countries with no evidence or just sporadic occurrences such as Zambia or Congo.A wide variety of contemporary evidence allowed us to display a near continuous distribution of good or complete evidence consensus countries from Indonesia to as far north as Pakistan and Zhejiang, China . Within Evidence consensus in Asia is lowest in central Asia where contemporary dengue occurrence records combined with low surveillance capacity results in an unclear boundary to the disease. While evidence for dengue presence in the lowland urban centres of Pakistan is accurate and contemporary, reports from the more remote north-west provinces are contemporary, but not accurate Ae. albopictus along the Mediterranean coast Ae. albopictus invasion to maintain this status. This, combined with the lower levels of healthcare expenditure, has led to an observed greater uncertainty in some eastern European states.Although no countries in Europe were defined as dengue-present, sporadic indigenous transmission events have lowered consensus in some countries . Since tIn general, consensus on dengue presence and absence was well defined across Australia and the Pacific islands, with 85% of countries showing good or complete evidence consensus . Where lLower evidence consensus was observed for Papua New Guinea due to a lack of reported case data since the 1980's, yet multiple literature sources suggest that dengue is still widespread Here we present the distribution of dengue virus transmission as assessed by evidence-based consensus. By emphasising the need for accurate, contemporary evidence through a weighted scoring system, we were able to identify areas where dengue status was more uncertain, particularly in Africa and Central Asia, and identify evidence gaps where surveillance might be better targeted to more accurately assess dengue status. By including a wide variety of evidence we were able to cast doubt on dengue status in countries previously described by health organisations as dengue-absent.et al.While many studies have focussed on the future threat of dengue as a result of range expansion or climate change, this is the first to assess the entirety of knowledge regarding the extent of current virus transmission. We have found that evidence of dengue virus transmission is temporally dynamic and that a contemporary map must emphasise evidence by weighting it appropriately. By increasing temporal resolution to one inter-epidemic period, we have extended the approach of Van Kleef The suggestion that dengue is an under-recognised problem in Africa is not a new one http://globalhealthtrials.tghn.org/) that will allow evidence contribution from members and will provide a key platform for display of dengue data and consensus maps. Although the current approach was used to map the distribution of dengue, minor modifications to the scoring system would allow it to be utilised for a variety of diseases for which the quality of presence evidence is spatially variable.Evidence consensus maps provide a more informative alternative to existing country-level maps, such as those provided by the WHO In this work, our aim was to produce a standardised methodology that used the largest variety of evidence to assess country dengue status, whilst still being applicable in diverse healthcare settings and suitable at multiple spatial scales. We considered the stark contrast in evidence available in Africa as compared to the rest of the world. Our results show that the inclusion of supplementary evidence (used in 44% of African countries but only 11% of the rest), healthcare expenditure information (for case data absences) and questionnaires increased evidence consensus in these countries without impacting the methodology applied to the rest of the world. Similarly, we are aware that increasing resolution to Admin1 or Admin2 level may well reduce the evidence available for calculating evidence consensus in each area compared to country-level calculations. As a result, we carefully chose which countries should have increased spatial resolution based on whether sufficient evidence was available in smaller administrative units. We also limited the selection of these countries to those at the limits of the disease's distribution, as data deficiencies in these regions more accurately represent the uncertainty on dengue status given the dynamic nature of global dengue spread. Here we present the most flexible methodology available, to date, for overcoming these problems. We have demonstrated that a systemic approach with relevant optional categories has allowed us to utilise the maximum variety of evidence available for assessing dengue status in the widest variety of situations.We also openly provide a full list of evidence for each country by category . We inteMapping by evidence consensus is a useful approach to quantifying contemporary disease evidence and can be further integrated with geo-spatial modelling to produce worldwide continuous surfaces of dengue risk Figure S1Geographic locations of occurrence data globally. Country colouring is based on evidence based consensus (see main manuscript) with green representing a complete consensus on dengue absence and red a complete consensus on dengue presence.(TIF)Click here for additional data file.Figure S2Geographic locations of occurrence data in Africa+. Country colouring is based on evidence based consensus (see main manuscript) with green representing a complete consensus on dengue absence and red a complete consensus on dengue presence.(TIF)Click here for additional data file.Figure S3Geographic locations of occurrence data in Asia. Country colouring is based on evidence based consensus (see main manuscript) with green representing a complete consensus on dengue absence and red a complete consensus on dengue presence.(TIF)Click here for additional data file.Figure S4Geographic locations of occurrence data in the Americas. Country colouring is based on evidence based consensus (see main manuscript) with green representing a complete consensus on dengue absence and red a complete consensus on dengue presence.(TIF)Click here for additional data file.Figure S5Geographic locations of occurrence data in Australia. Country colouring is based on evidence based consensus (see main manuscript) with green representing a complete consensus on dengue absence and red a complete consensus on dengue presence.(TIF)Click here for additional data file.Figure S6Number of occurrence samples per year globally (a) and for Africa+ (b), Asia, (c) the Americas and Australia (d).(TIF)Click here for additional data file.Figure S7Map of evidence types used for each national and subnational area.(TIF)Click here for additional data file.Protocol S1An outline of the dengue occurrence point database construction and content. Data sources, searches and exclusion criteria are outlined and the method of geo-positioning explained. The regional bias of available occurrence points is also given in the accompanying figures. (DOC)Click here for additional data file.Table S1The collection of evidence used to assess evidence consensus for each country and Admin1 and Admin2 areas. Details of the scoring system can be found in the (DOC)Click here for additional data file.


Health-policy decision making is a complex and dynamic process, for which strong evidentiary support is required. This includes scientifically produced research, as well as information that relates to the context in which the decision takes place. Unlike scientific evidence, this “contextual evidence” is highly variable and often includes information that is not scientifically produced, drawn from sources such as political judgement, program management experience and knowledge, or public values. As the policy decision-making process is variable and difficult to evaluate, it is often unclear how this heterogeneous evidence is identified and incorporated into “evidence-based policy” decisions. Population-based colorectal cancer screening poses an ideal context in which to examine these issues. In Canada, colorectal cancer screening programs have been established in several provinces over the past five years, based on the fecal occult blood test (FOBT) or the fecal immunochemical test. However, as these programs develop, new scientific evidence for screening continues to emerge. Recently published randomized controlled trials suggest that the use of flexible sigmoidoscopy for population-based screening may pose a greater reduction in mortality than the FOBT. This raises the important question of how policy makers will address this evidence, given that screening programs are being established or are already in place. This study will examine these issues prospectively and will focus on how policy makers monitor emerging scientific evidence and how both scientific and contextual evidence are identified and applied for decisions about health system improvement.This study will employ a prospective multiple case study design, involving participants from Ontario, Alberta, Manitoba, Nova Scotia, and Quebec. In each province, data will be collected via document analysis and key informant interviews. Documents will include policy briefs, reports, meeting minutes, media releases, and correspondence. Interviews will be conducted in person with senior administrative leaders, government officials, screening experts, and high-level cancer system stakeholders.The proposed study comprises the third and final phase of an Emerging Team grant to address the challenges of health-policy decision making and colorectal cancer screening decisions in Canada. This study will contribute a unique prospective look at how policy makers address new, emerging scientific evidence in several different policy environments and at different stages of program planning and implementation. Findings will provide important insight into the various approaches that are or should be used to monitor emerging evidence, the relative importance of scientific versus contextual evidence for decision making, and the tools and processes that may be important to support challenging health-policy decisions. Health-policy decision making poses many challenges. There is an unprecedented proliferation of scientific information available to policy makers, and with it, increasing demand to ensure that decisions are based on the highest-quality up-to-date evidence . Similaret al., the heterogeneity of evidence used for health-policy decision making was characterized as “scientific context-free,” “scientific context-sensitive,” and “colloquial” [i.e., stakeholder opinion and values) [Unlike clinical decisions, policy decisions are inherently context-specific ,2 and reloquial” . In this values) . To addr values) ,6. HowevFinally, unlike clinical decisions, multifactorial policy decisions are not outlined in clear linear stages ,2. Insteet al.[Colorectal cancer (CRC) screening presents an ideal context to examine the challenges of health-policy decision making. The second most common cause of cancer deaths in Canada , CRC haset al., the useet al.. This tret al., NORCCAPet al., and PLCet al. trials, et al. retrospectively examined how four expert groups in Ontario made screening policy recommendations for four types of cancer screening  [In examining this subject, the proposed study builds on previous formative research on the use of evidence to inform complex cancer-screening policy decisions. From 2002–2006, Dobrow This study comprises the final phase of this work and will examine prospectively how emerging scientific evidence on FS is addressed by policy makers in five provinces. This study reflects an important progression in our work, revisiting the same five provinces and providing a new and innovative perspective in an area in which relatively little is known—the impact of emerging scientific evidence when policy decisions have already been made. This issue is timely and relevant in Canada, as screening programs continue to be developed and implemented, and the Canadian Partnership Against Cancer has convened an anticipatory science panel to examine new evidence on FS . This isIn developing the methodology and rationale for this study, we incorporate some key assumptions. Primarily, we assume policy makers use a variety of different approaches to monitor and identify emerging scientific evidence and that these approaches will vary based on jurisdiction and stakeholder group. We also make the assumption that there are limited formal tools and guidance to support policy makers in integrating emerging scientific evidence with relevant contextual information.We intend to approach policy makers with the question of whether to incorporate FS into population-based programs and hypothetically how this decision should be made. Consequently, we anticipate that policy makers will not provide a definitive yes/no response but will discuss gaps in contextual information that must be filled, particularly pertaining to program implementation, and context-specific factors that would determine how this issue is addressed.Based on the above, this study will address the following research questions:1.) How do policy makers monitor and become aware of emerging scientific evidence?2.) How does new scientific evidence impact established screening programs/policies?3.) How do policy makers identify/interpret contextual evidence?4.) How does contextual evidence impact established screening programs/policies?5.) How do policy makers integrate scientific with contextual evidence to address decisions about health system improvement?This study will employ a prospective, multiple case study design involving participants from Ontario, Manitoba, Alberta, Nova Scotia, and Quebec. Primary methods for data collection will include document analysis and semi-structured interviews in each province.Documents relevant to each province’s consideration of evidence for CRC-screening decisions will be collected. A systematic search strategy will be used to collect these data, beginning with documents available in the public domain, followed by requests to key informants and organizations for further material. Documents will include policy recommendations/reports/briefs, planning documents, meeting minutes, media releases, and correspondence.Interviews will be conducted with five to eight policy makers and cancer system stakeholders in each province, including administrative leaders, government officials, screening experts, and other high-level stakeholders from ministries of health and cancer agencies. Purposive sampling will be used to identify these informants, with snowball sampling used subsequently to further identify individuals involved in CRC-screening decisions for the province. All potential interviewees will be sent an introductory package explaining the project and describing procedures for confidentiality and informed consent. The research team will then contact the individual by phone and arrange a convenient interview time. Interviews will be conducted in person and will last approximately 60 minutes each. Any additional interviews will be conducted by telephone. All participants will be asked to sign consent forms prior to the interview. Interviews will be audio-recorded and transcribed verbatim where permission is granted. Data will be analyzed using NVivo  qualitative analysis software. For this project, ethical approval has been sought and obtained from the University of Toronto’s Research Ethics Board.This study will be completed in accordance with the timeline of our Canadian Institutes of Health Services Research Emerging Team Grant. It is anticipated that the data collection phase of this work, including document collection and key informant interviews, will be completed by September 2012, with transcription and coding of the interview data to be completed by the end of the calendar year. Following this, the analysis and write up of findings will take place and will be completed by March 31, 2013.As scientific evidence continues to emerge, posing new opportunities for health system improvement, it is important to examine how policy makers will respond. Primarily, the findings of this study will provide insight into how emerging scientific evidence is monitored, and how this varies across stakeholder groups and jurisdictions. If key informants are already aware of emerging evidence on FS and engaging in steps to address this evidence, study findings will allow for comparison of what methods and processes are currently being used. Conversely, if key informants are unaware or unable to discuss emerging scientific evidence, study findings will provide valuable insight into what methods/processes should be in place for this (if any), and how to better support policy makers in this important area.Furthermore, this study will provide important insight into the relative importance of scientific versus contextual evidence for decisions about health system change. Although scientific evidence is disseminated on a continual basis, evidence appears to be less clear for program and implementation-related questions that pose significant challenges for policy makers. For this, the multiple case study design is ideal and will allow for comparison of ways in which contextual evidence is identified and applied in five unique jurisdictions at various stages of program planning and implementation.Finally, in evaluating health-policy decision making, research is often reactive. Instead of examining retrospectively how decisions have been made, this study will provide unique insight into policy decision making from a hypothetical and prospective sense. We hope to contribute positively to the EBHP literature and to help support and advance the use of evidence in making challenging policy decisions that profoundly impact our healthcare system.The authors declare that they have no competing interests.All authors contributed to the original concept and methodology of this study. The initial draft of this manuscript was completed by HG. All authors reviewed the manuscript, provided critical comments and suggestions for revision, and ultimately approved the final completed work. All authors read and approved the final manuscript.


People making decisions about interventions, actions and strategies for natural disasters, humanitarian crises and other major healthcare emergencies need access to reliable evidence to help ensure that the choices they make are likely to do more good than harm. However, there are many gaps in this evidence base in a wide range of areas. This paper reports a priority setting exercise that was coordinated by Evidence Aid to identify thirty priorities for up-to-date systematic reviews of the effects of interventions, actions and strategies on health outcomes, which would be particularly relevant to those involved in disaster risk reduction, planning response and recovery at an international level. It builds from an ongoing needs assessment that had identified a couple of hundred relevant research questions, which were grouped under 43 themes. Ten of these themes were prioritized by an online survey and the questions attached to these themes were then discussed at a face-to-face meeting, leading to the generation of a list of 30 top priority questions which is presented in this paper. We recognize that a different group of people might have come to different priorities but regard this as an important starting point, and the extensive efforts that were made to be inclusive in gathering opinions should help ensure their wide relevance. However, there are many gaps in this evidence base in a wide range of areas.This paper summarizes a priority setting exercise that was coordinated by Evidence Aid to identify 30 priorities for up-to-date systematic reviews of the effects of interventions, actions and strategies on health outcomes that would be particularly relevant to those involved in disaster risk reduction, planning, response and recovery at an international level. It includes the list of these priorities. We used a combination of online surveys and a face-to-face meeting to solicit and rank these priorities, with involvement from low-, middle- and high-income countries, all continents and a range of agencies (see Methods).Evidence Aid (www.evidenceaid.org) is an international initiative that was established following the tsunami in the Indian Ocean in December 2004.The objective of this exercise was to identify approximately 30 high priority research questions under ten themes that could be addressed by systematic reviews in the area of planning for or response to natural disasters, humanitarian crises or other major healthcare emergencies. There was a particular focus on topics of particular relevance to low- and middle-income countries where the health impact of disasters may be greater than in high-income settings.,The methods used and the topics considered are outlined below and will be reported more fully in a subsequent report. In summary, this Evidence Aid review priority setting method started with an open, online survey asking humanitarian aid workers and others to provide up to three research questions or areas of uncertainty for which they need research evidence. These suggestions were supplemented at two Evidence Aid conferences and from published literature, before being arranged into themes. A second open, online survey was then conducted to produce the top ten themes, and the associated questions were then prioritized using a nominal group technique with experienced facilitators from the James Lind Alliance,Identification of questions and uncertaintiesThe Evidence Aid online needs assessment survey was opened in July 2011 and is available in Arabic, English, French, German and Spanish.In order of decreasing importance what would be your three (or more) priority topics for which you would like evidence in the context of natural disasters or other humanitarian crises?For funders, list any topics for which research evidence would help when making decisions about funding a projectFor funders , list any topics for which research evidence would help when analysing the reports you receive from recipient agenciesAt the time of this prioritization exercise, this first survey had been completed by 101 respondents of whom 83 provided information on the region of the world in which they based. Eleven participants were based in Sub-Saharan Africa (13% of participants who provided their location), 11 (13%) were in Asia, 5 (6%) in the Middle East, 2 (2%) in Eastern Europe and 2 (2%) in South and Central America. The largest proportion of respondents were in Western Europe  with 21 (25%) in North America and 8 (10%) in Australia or New Zealand. Among the 83 first survey respondents who provided information about the organisation they worked for at the time, 34 (41%) respondents worked in international non-governmental organizations (NGOs), 15 (18%) worked in local NGOs; 7 (8%) worked in UN Agencies; 16 (19%) were aid workers working for governments of high-income countries, 4 (5%) were aid workers working for governments in low- and middle-income countries, 6 (7%) worked as academics researchers and 1 (1%) worked for a global organisation.The Lancet series on Maternal and Child Health).A list of 216 research questions was compiled from the answers to the relevant sections in the survey, discussions with aid agency research centres and NGOs, ideas from participants and presenters during the Evidence Aid conferences in 2011 and 2012 and published literature  and grouped the questions into 43 main themes.Prioritization of themesAn online survey was conducted using SurveyMonkey to prioritize these 43 themes, given that it would be impractical to deal with all of them at the prioritization meeting. The snowballing technique was used for distribution of links to the survey. Information was also distributed using Twitter, Facebook, LinkedIn, the Evidence Aid website, the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) discussion forum, other relevant websites, and a variety of e-mail lists. Recipients were encouraged to complete the survey and to circulate the information about it within their networks. Respondents were asked to select their top ten themes and to rank these from first to tenth. These rankings were then aggregated to generate an overall top ten themes for discussion at the face-to-face prioritization meeting.Selection of questionsThe questions that had been attached to each of the top ten themes were refined, including restricting the topics to those relating to interventions, actions or strategies that might have an impact on health outcomes; and, with help from the Centers for Disease Control and Prevention, they were framed into questions about the effects of the interventions, actions or strategies. Further work reworded the questions so that they would be more suitable for answering through a systematic review.Prioritization meeting,A table of the questions linked to the ten themes was circulated to participants in advance of the meeting, asking them to prepare for it by ranking the questions within each theme. The meeting was held in London, England on 3-4 June 2013 and was facilitated by people experienced in the James Lind Alliance process  for reaching consensus in research prioritization.The online survey of 43 themes was opened by 280 people and 276 of these started to complete the survey, with 233 people submitting their selection of the top ten themes in ranked order. These 233 participants worked for a variety of organizations including international aid agencies, national aid agencies, United Nations agencies, and research centres, as well as people who work independently on issues related to disasters. They came from a varied geographic area: 117 were based in Europe, 45 in the USA, 21 in Asia, 19 in Africa, 12 in the Middle East, 8 in Canada, 7 in South America, and 4 in Australia and New Zealand. The top ten themes that arose from this online survey are shown in Table 1.Twenty-eight people from diverse backgrounds relating to disaster management attended the prioritization meeting (Appendix). They were from aid agencies, funders, NGOs, and included independent consultants who had worked in various capacities as well as the facilitators. The meeting led to agreement on a total of 30 prioritized questions for up-to-date systematic reviews (Table 2). It was recognized that further refinements to these questions are likely to take place as they are worked on for systematic reviews, and, as of September 2013, three are already under active discussion with the appropriate groups in The Cochrane Collaboration.Identifying priorities for research in any area is a demanding and complex task. This is particularly true in the disaster context where the range of evidence needed and the complexity of humanitarian response make it challenging to prioritize key questions that might provide the evidence for decision makers and others making choices about interventions, actions and strategies. However, consensus was reached at this meeting, showing that a group with a diverse knowledge base, responsibilities and experiences could reach agreement on a set of 30 questions to prioritize for up-to-date systematic reviews. We recognise that any process which seeks to draw on such a diverse community is open to potential biases arising from who was invited to take part and who chose to take part. However, the calls for input to both online surveys were wide ranging and inclusive, and the invitations for the workshop were targeted at those with experience or expertise in the ten themes that were to be discussed. We hope that this list of priorities will not only stimulate work to conduct the systematic reviews that will help to answer these questions, but will also encourage debate and action about priorities by others, such as those with a more nationally-orientated focus for disasters.,The next steps for Evidence Aid are for the priorities identified here to be adopted and used to prepare up-to-date systematic reviews of the evidence. This will require the refinement of some of the questions, to improve their suitability as the research question for a systematic review. As with reviews generally, this is likely to be an iterative process, involving key stakeholders in the review and will, ideally, include the prospective registration of the review in PROSPERO and the preparation of a protocol Furthermore, the reviewers are likely to need to overcome challenges arising from the difficulty of identifying, appraising and synthesizing research that might be spread across hundreds of journals, books, unpublished reports and websites and a lack of uniform reporting of data in the disaster setting. Therefore, we hope that initiatives to improve the accessibility and quality of the underlying research and data,


Best formats for summarising and presenting evidence for use in clinical guideline development remain less well defined. We aimed to assess the effectiveness of different evidence summary formats to address this gap.Healthcare professionals attending a one-week Kenyan, national guideline development workshop were randomly allocated to receive evidence packaged in three different formats: systematic reviews (SRs) alone, systematic reviews with summary-of-findings tables, and ‘graded-entry’ formats . The influence of format on the proportion of correct responses to key clinical questions, the primary outcome, was assessed using a written test. The secondary outcome was a composite endpoint, measured on a 5-point scale, of the clarity of presentation and ease of locating the quality of evidence for critical neonatal outcomes. Interviews conducted within two months following completion of trial data collection explored panel members’ views on the evidence summary formats and experiences with appraisal and use of research information.65 (93%) of 70 participants completed questions on the prespecified outcome measures. There were no differences between groups in the odds of correct responses to key clinical questions. ‘Graded-entry’ formats were associated with a higher mean composite score for clarity and accessibility of information about the quality of evidence for critical neonatal outcomes compared to systematic reviews alone . There was no difference in the mean composite score between SR with SoF tables and SR alone. Findings from interviews with 16 panelists indicated that short narrative evidence reports were preferred for the improved clarity of information presentation and ease of use.Our findings suggest that ‘graded-entry’ evidence summary formats may improve clarity and accessibility of research evidence in clinical guideline development.ISRCTN05154264Controlled-Trials.com  Policy decisions about which interventions to implement, modify or withdraw from health care, should be informed by the best available evidence. While systematic reviews are a key source of evidence Ethical approval for the conduct of this study was granted by the Kenya Medical Research Institute scientific committee and National Ethics Review Committee in Kenya (SSC Protocol No 1770). Individual written informed consent for participation and audio recording of discussions was obtained prior to the face-to-face interviews. Confidentiality of participant information was ensured by assigning anonymous codes to individual audio interviews and transcripts.This was a mixed-method study incorporating a randomised controlled trial (RCT) to assess the effectiveness of three different evidence summary formats with semi-structured follow-up interviews to explore panel members’ views of these formats, experience with appraisal, use of and engagement with the research evidence.The Kenya Medical Research Institute - Wellcome Trust Research Programme together with Kenya’s Ministry of Medical Services collaborated to host a one-week national guideline development workshop  Trial participants consisted of a multidisciplinary panel of healthcare professionals with varied roles in neonatal and paediatric policy and care in Kenya . All 77 We assembled evidence in three formats: systematic reviews (SR) alone (pack ‘A’), systematic reviews with summary-of-findings tables  and ‘graded-entry’ formats (pack ‘C’). Evidence pack ‘A’ represented the common standard practice of using systematic reviews and lengthy technical reports to inform healthcare policy and guideline development. Evidence pack ‘B’ represented the recently enriched format for preparing full Cochrane reviews Evidence pack C was designed to allow stepwise access (i.e. ‘graded-entry’) to the evidence. It started with a ‘front-end’ short interpretation of the main findings and conclusions, drawn from evidence synthesis . These fThe ‘front-ends’ in pack ‘C’ were adapted from existing review-derived formats designed to allow rapid scanning for key messages . FurtherThe GRADE  system Evidence summaries in pack A, B and C formats were prepared for three ‘tracer-interventions’ relevant to neonatal care where new guidelines were being considered and for which systematic reviews had been recently published: (1) feeding regimens in sick newborns The primary outcome was the proportion of correct responses to key clinical questions relevant to the specific tracer topics. This tested the understanding of the effects of tracer-interventions on critical neonatal outcomes . The secondary outcome measure was a composite score representing participants’ self-reports of the clarity and accessibility of the evidence; participants rated their responses on a 3 or 5-point scale. These domains have successfully been tested and used in previous trials Participants completed questionnaires on the first day  of the guideline development workshop before the panel discussions about guidance recommendations. The questionnaires were adapted from existing studies Individual face-to-face interviews were conducted (between July and August 2010) by NO following completion of the questionnaire but before analysis. Interview questions were derived from a review of literature on best practices in clinical guideline development The sample size calculation was based on the pre-specified primary outcome assuming a two-way comparison of the three-component pack (C) versus the minimum standard pack (A) and a 1∶1 allocation ratio. The trial was designed to have 90% power at an alpha (significance level) of 0.05 to detect a 40% absolute difference in the proportion of correct responses with 35 observations on each evidence pack . Assuming a non-response rate of 10% we targeted at least 40 participants for randomisation. All analyses treated the respondent as a unit of clustering.The crude odds (likelihood) of correct responses for pack C compared to the odds for pack A (assumed baseline pack) were estimated using logistic regression. The models were also adjusted for the effect of strata and the tracer intervention. Pack by strata interaction terms were included to assess whether the effect of the different evidence packs differed by strata. The secondary outcome measure was the clarity and accessibility score. The mean clarity and accessibility scores of pack C compared to the mean scores of pack A were estimated using linear regression and the lack of normality of the scores  was accoAudio interviews were transcribed verbatim by NO. Emerging themes and concepts were extracted by at least two co-investigators  working independently Although quantitative and qualitative components of this study were pre-specified in the proposal integration occurred after separate analysis of each component at the point of interpretation of the results.The trial profile is summarised in Sixty-five (93%) of 70 participants completed questions on primary outcome measures. A descriptive summary of the proportion of correct and incorrect responses for each evidence pack is shown in ‘Graded-entry’ formats (pack C), compared to systematic review formats (pack A), were associated with a significantly higher mean ‘value and accessibility’ score  found systematic reviews to be more difficult to read compared to narrative reports, but some (17%) responded that systematic reviews were easy to read. About half of the participants (51%) found systematic reviews to be easier to read compared to summary-of-findings tables (26%). A higher proportion of participants preferred evidence summarised in narrative report formats to the full version of the systematic reviews (53% versus 25%). See Participants’ self-rated experiences with research literature and familiarity with evidence based medicine terminology are presented in The majority of participants interviewed found research information summarised in the form of narrative reports to be clearer, easy to read, easy to understand and containing ‘just the right amount of information’. Conversely, participants expressed considerable variability in views for systematic reviews and GRADE summary-of-findings tables: while some found the comprehensive and structured nature of information presentation in systematic reviews to be useful, a number expressed difficulties with extracting pertinent information. Some participants found GRADE summary tables to be good for ‘rapid consultation’; however, a number of participants found them difficult to understand as stand-alone summaries. For example, one participant expressed difficulties with distinguishing between the GRADE categories of quality of evidence. Of note, many participants reported lack of time and the volume of evidence as factors contributing to more complete engagement with the evidence.The majority of participants responded that they were not conversant with assessing the quality of scientific literature or evidence-based medicine terminology, such as PICO . A number suggested that a short course on evidence-based medicine would be beneficial to support evidence-based guideline development.The current study assessed the effectiveness and panelists’ views and experiences with different evidence summary formats. The quantitative evaluation showed that ‘graded-entry’ evidence summary formats which included narrative reviews highlighting local factors relevant for implementation, compared to full systematic reviews, were associated with higher composite scores for clarity and accessibility of information on critical outcomes. However, the proportion of correct answers to questions testing the ability to identify, access and reproduce key messages from the evidence summaries was not improved. This failure may reflect a true lack of effect of format on this outcome. Alternatively, it may be due to inadequate power to detect a smaller but useful effect or inadequate participant preparation in those with limited prior exposure to research evidence . Althoug[006]: ‘The mini-review [narrative report] I think was like one of the most useful things I came across, because I found that they had just the right amount of information…and also the Cochrane reviews I found were very good’[010]: ‘…this particular one [table-of-key findings] was not very easy for me to understand…sometimes making a clear cut difference between especially the quality grades was not very easy…alone on itself I found it difficult…I needed to go back to the summaries, read a bit and come and compare…one needed to have some prior knowledge especially on how to interpret it…’[012]: ‘The systematic reviews give more information on what was done in the research but you will need a lot of time to go through it. So if I were to be asked which one I preferred, I would pick the mini-review [narrative report] because it was very brief and easy to understand. It’s easy to understand and you don’t go and go until you get tired along the way’[014]: ‘The Cochrane is good because it’s comprehensive and a very good summary of the work that’s available as well as what has been done. Its main disadvantage is that it takes more time to read through and digest. The mini-reviews [narrative reports] were the most useful because they were a condensed version, so they are easy to use even when you have limited time they will prove useful. The table-of-key findings is good for a rapid consultation but doesn’t give you a feel of how those conclusions were arrived at’[016]: ‘The problem with the reviews is that most of them had limited data…But I was feeling that a lot more other data probably from observation studies could be available and would have been of more value. But at the end of the day you base most of your decisions on randomised controlled trials, but you should know a little more of what was done in the other studies’Pre-workshop materials[006]: ‘…like the people I was sitting with, most of them had not gone through all of the material, a few had gone through about half and a few had not looked at it at all’[008]: ‘No, the reading of the materials appeared not to have been exhausting except you backed it up by making presentations…’[010]: ‘Generally I think very few of the participants went through the materials before coming to the workshop. Personally I would say I didn’t spend adequate time that would have enabled me to understand the material better, for better participation during the workshop…’[012]: ‘…and the volume of the document was another challenge. With the limited time people had and the volume of the document, people had very little time to go through them’[004]: ‘No, I can’t say I was comfortable, even when the PICO  format was introduced I personally had problems understanding it…but now I understand it very properly and even the GRADE system, for me it really sunk well after the evidence week’[011]: ‘Not at all, even being able to synthesize a research paper…it’s still a process that I would like to learn. For many of the participants I think a short course in that area would be useful and it would help a bit also in the voting’[013]: ‘Most of us take it [research information] as it’s given to us; we don’t sieve it and see whether it’s of high or low quality…the general practitioners don’t really know the difference of [between] these studies’[015]: ‘…I didn’t know there is low quality, moderate [quality] or even when I find in the books that it is low quality, I did not know actually what they based it on’Despite an absence of effect on our main outcome, ‘graded-entry’ summary formats and their components were reported to: improve clarity of information presentation, improve accessibility to key information, be more reader-friendly (both narrative reports and summary-of-findings tables) and were preferred by participants over systematic reviews. These formats are also likely to be more suited to the different backgrounds, interests and levels of expertise in multidisciplinary guideline panels. For example, those with limited training and skills in research synthesis (such as point of care clinicians) and busy policymakers may only need to scan the ‘front-end’ summaries for key results. On the other hand, those with expertise in research synthesis may prefer the detail available in narrative reports and systematic reviews to assess the reliability of the results before considering their applicability. These findings are further supported by the qualitative findings . A prefeTo compare current results with related studies we searched The Cochrane Library and PubMed (both from database inceptions up to March 2012) for articles examining ways of presenting evidence for use in healthcare decision-making (full search strategy available from the authors on request). The search identified six related studies: three systematic reviews In the first randomised controlled trial The qualitative interview (N = 13) study Overall, observed differences in results across systematic reviews and primary studies examining impacts of evidence presentation formats, including our own, could be explained by differences in: study designs , presentation formats , types of participants ; study contexts (evidence based medicine training versus policymaking environments); definitions of outcome measures  factors and values that might influence intervention effectiveness, to be liked by busy decision makers. Such narrative reports could be made even more policy-relevant by inclusion of information on wider health systems perspectives such as: potential structural changes needed for effective implementation, equity, feasibility of scaling up and monitoring and evaluation issues. However, current findings could also inform future guideline development work in high-income countries.Combining a quantitative RCT design with qualitative interviews and using both in our interpretation of data improved our understanding of, and confidence in the study results. Our decision to study multiple tracer-topics rather than focus on a single guideline development topic may be considered a weakness or a strength. Perhaps if participants had only to read the literature provided for one topic, ‘graded-entry’ formats may have proved superior even on the primary outcome. However, in a low-income country setting like ours, it seems unlikely that countries will have the resources in the near future to engage specific panels for meetings for each guideline topic. Thus, we deliberately wished to examine a potentially efficient approach in which the panel considered multiple topics. Finally, although our randomization strategy (stratified by job-type/seniority) may have reduced possible variation in format quality and expertise (within the population enrolled) on the main findings, we cannot rule out the possible influence of these factors on the intended outcomes.Although our findings suggest possible benefits of ‘graded-entry’ formats, evidence about the best formats for improving actual understanding and use of research findings in decision making processes (a key aim in knowledge translation) appears to be lacking. Thus, further studies are needed to learn more about the relative effectiveness of the various evidence summary formats that are available and how these may be combined with alternative approaches.‘Graded-entry’ formats were found to improve clarity and accessibility of research evidence, although this was not reflected in the number of correct responses to key clinical questions. Providing a ‘front-end’ summary of key information linked to locally relevant factors that support implementation, and the full systematic review, may help those developing guidelines access and contextualise research evidence.Box S1Research synthesis products.(DOC)Click here for additional data file.Webappendix S1Front-end evidence summary.(PDF)Click here for additional data file.Webappendix S2Randomisation.(DOCX)Click here for additional data file.Webappendix S3Distribution of value and accessibility scores.(DOCX)Click here for additional data file.Webappendix S4Data handling and analysis.(DOCX)Click here for additional data file.


Guideline developers use different consensus methods to develop evidence-based clinical practice guidelines. Previous research suggests that existing guideline development techniques are subject to methodological problems and are logistically demanding. Guideline developers welcome new methods that facilitate a methodologically sound decision-making process. Systems that aggregate knowledge while participants play a game are one class of human computation applications. Researchers have already proven that these games with a purpose are effective in building common sense knowledge databases.We aimed to evaluate the feasibility of a new consensus method based on human computation techniques compared to an informal face-to-face consensus method.We set up a randomized design to study 2 different methods for guideline development within a group of advanced students completing a master of nursing and obstetrics. Students who participated in the trial were enrolled in an evidence-based health care course. We compared the Web-based method of human-based computation (HC) with an informal face-to-face consensus method (IC). We used 4 clinical scenarios of lower back pain as the subject of the consensus process. These scenarios concerned the following topics: (1) medical imaging, (2) therapeutic options, (3) drugs use, and (4) sick leave. Outcomes were expressed as the amount of group (dis)agreement and the concordance of answers with clinical evidence. We estimated within-group and between-group effect sizes by calculating Cohen’s d. We calculated within-group effect sizes as the absolute difference between the outcome value at round 3 and the baseline outcome value, divided by the pooled standard deviation. We calculated between-group effect sizes as the absolute difference between the mean change in outcome value across rounds in HC and the mean change in outcome value across rounds in IC, divided by the pooled standard deviation. We analyzed statistical significance of within-group changes between round 1 and round 3 using the Wilcoxon signed rank test. We assessed the differences between the HC and IC groups using Mann-Whitney U tests. We used a Bonferroni adjusted alpha level of .025 in all statistical tests. We performed a thematic analysis to explore participants’ arguments during group discussion. Participants completed a satisfaction survey at the end of the consensus process.P=.30) for the medical imaging scenario, 0.07 (P=.97) for the therapeutic options scenario, and 0.89 (P=.11) for the drug use scenario. We found no significant differences in improvement in the degree of agreement between HC and IC groups. Between-group comparisons revealed that the HC groups showed greater improvement in degree of agreement for the medical imaging scenario  and the drug use scenario . Very few evidence arguments (6%) were quoted during informal group discussions.Of the 135 students completing a master of nursing and obstetrics, 120 participated in the experiment. We formed 8 HC groups (n=64) and 7 IC groups (n=56). The between-group comparison demonstrated that the human computation groups obtained a greater improvement in evidence scores compared to the IC groups, although the difference was not statistically significant. The between-group effect size was 0.56 (Overall, the use of the IC method was appropriate as long as the evidence supported participants’ beliefs or usual practice, or when the availability of the evidence was sparse. However, when some controversy about the evidence existed, the HC method outperformed the IC method. The findings of our study illustrate the importance of the choice of the consensus method in guideline development. Human computation could be an acceptable methodology for guideline development specifically for scenarios in which the evidence shows no resonance with participants’ beliefs. Future research is needed to confirm the results of this study and to establish practical significance in a controlled setting of multidisciplinary guideline panels during real-life guideline development. Evidence-based clinical practice guidelines can be defined as “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical conditions” . They arThe development of clinical practice guidelines requires a systematic and transparent process, in which the recommendations of the clinical practice guideline (CPG) are explicitly linked to the clinical evidence. However, guidelines cannot be deduced from evidence alone, and expert opinions are needed to contextualize the evidence to the target population . FurtherThe high cost in time, resources, and efforts needed for the Delphi method, and the intensive commitment required for the NGT, pose important practical and logistic problems . VariousConducting the consensus development process entirely online is an approach that can address these concerns -10. An oHuman-based computation is a technique in computer science in which the problems that a computer cannot yet solve are outsourced to humans. One class of human-based computation applications are the systems in which the tasks outsourced to humans are packaged as a game. These applications are called games with a purpose (GWAP). The idea behind these systems is to take advantage of people’s desire to be entertained while performing useful tasks as a side effect. This approach is effective in building large knowledge databases, but to date it has no known practical applications in medicine -13.Based on the principles of successful GWAP, we developed the CPGame  application as a new method for guideline development. We built a prototype based on human-based computation techniques and the goals of the experiment. Our objectives were threefold: (1) to investigate the similarities or differences in degree of agreement and evidence with an informal consensus method to explore whether the human-based computation method is a valuable alternative, (2) to investigate arguments in decision making during group discussion, and (3) to explore perceptions and opinions about the consensus method. The objectives of the study were hypothesis-generating in the first place.We performed a randomized controlled trial (RCT) to compare the feasibility of the human-based computation method with an informal consensus method using a face-to-face meeting. Different consensus groups participated in the trial. Each of these groups consisted of 8 participants ,15.We developed 4 multiple choice scenarios involving lower back pain. An example scenario is included as Students completing a master of nursing and obstetrics at the University of Leuven participated in the experiment while taking a course in evidence-based health care. All of the participants already had a bachelor of nursing and practical experience. They all had baseline knowledge about lower back pain that was sufficient to judge the clinical scenarios. They also had sufficient knowledge in evidence-based health care to understand the evidence terms and the scientific meaning of the levels of evidence. There were no exclusion criteria, and all students had Internet experience.We developed the CPGame application based on human-based computation techniques. The human-based computation method is comparable to an online Delphi method packaged as a game. CPGame is a real-time collaborative application written in PHP, JavaScript, and Ajax. We used a MySQL database as the data repository. We pilot tested the technical robustness of the application with a group of trainees in family medicine.Each participant in the HC groups was anonymously paired with another participant. The 2 participants played in a team against the other teams of 2 participants (giving 4 teams of 2 students in each group). Participants were given a user ID and password to log on to the CPGame application. The 2 participants on each team were given the same multiple choice questionnaire about a clinical scenario .Independently of each other, both participants on a team gave their opinion about the case by choosing their preferred answer from the multiple choice list. When they had given their answers, the application displayed a message stating whether or not they reached consensus. If they reached consensus, they were given the next clinical scenario. If they did not reach a consensus, the application displayed the evidence for each answer and the answer of the teammate .Each participant was given one chance to change his or her answer to try to reach consensus on a second attempt. After all the teams responded to all 4 scenarios, the CPGame application displayed the answers of all participants and the level of agreement between participants . Each paThe game behind the human computation application consisted of a point system, a high-score list, and time pressure. These elements are described in the literature as being salient features that make GWAP fun . The appAll participants completed the experiment in the same room at the same time. This one-room setting was possible because students participated in the trail during their course in evidence-based health care and had to come to the building for their class. A moderator was in the room in case there were technical problems during the experiment. The CPGame application was originally designed to be a self-directed process in which users independently participate online at their home.The online approach to human-based computation (HC) the informal consensus method was comparable with a traditional face-to-face meeting. We took several measures to make both HC and IC methods comparable; the only difference was the mode (face-to-face versus Web-based packaged as a game).The content of the scenarios and the evidence were similarly presented in both HC and IC groups. Before the start of the informal consensus process, each participant individually indicated his or her preference between several options for each scenario in the first round. After this first round, participants were randomly grouped into teams of 2, they were given the evidence for each treatment option, and each team discussed the cases. The discussion within the teams of 2 students was added as an additional step in the consensus process to ensure the points of measurement were equal between the 2 methods being reviewed. Each participant individually re-rated each scenario in the second round. After the second round, all 8 participants met in one group. A moderator, with experience in coordinating small groups, managed the discussion based on a written protocol designed to standardize the meetings. At the meeting, participants were told the judgments of the other group members and the research evidence. As a group, participants discussed each scenario and explored reasons for differences in opinions. After the group meeting, each participant individually re-rated each scenario in the third round.Participants in neither the IC groups nor the HC groups were given incentives to participate in the experiment. The possible incentive of winning the game in the HC groups was neutralized by the introduction of a last consensus round outside the game.All students were invited to participate in the trial a week before the experiment by one of the researchers. Participants were randomly assigned to 1 of the 2 groups (HC or IC) following simple randomization procedures. One of the researchers performed the randomization with an electronic random-list generator, initially in 1:1 ratio. A second step in the randomization procedure consisted of assigning individuals an additional consensus group number.When they entered the computer room, participants of the human-based computation group blindly chose an envelope at random with a user ID and password to log on to the CPGame application. The envelope contained a number from 1 to 8. Numbers 1 and 2 played the game in a team, numbers 3 and 4 were a team, etc. Participants did not know each other’s numbers and did not know who would be on each team. We used the same randomization procedure in the informal consensus group after first round ratings were completed. Participants blindly chose an envelope at random with a number from 1 to 8. Predetermined pairs of numbers were used to form the teams.Group assignments were given just before the start of the experiment. Although participants knew they were participating in a guideline development project about lower back pain, they did not know the outcomes and the goals of the project before participation. Researchers were not blinded to allocation, but outcomes were objective measures.We conducted descriptive statistics and graphical displays to describe the sample population. Baseline data about the participants’ gender and age were compared using chi-square and Mann-Whitney U tests, as appropriate.There were 3 points of measurement: (1) before the consensus process (round 1), (2) during the consensus process (round 2), and (3) at the end of the consensus process (round 3). Primary outcomes focused on the change of opinions towards consensus and towards evidence between round 1 and round 3. An analysis of the group’s level was warranted because of our interest in group decision making. As the group’s outcomes were treated as individual observations, we had not taken the clustering of individuals within a discussion group into account. It was appropriate to analyze the 4 scenarios separately because they were totally different in content. As such, degree of (dis)agreement and degree of evidence were calculated for each of the clinical scenarios. We used a Bonferroni adjusted alpha level of .025  for all statistical tests. Predictive Analytics SoftWare Statistics 18 was used for statistical analyses.We used a kappa statistic to express the degree of (dis)agreement within a group at the different rounds. We estimated the within-group change between round 1 and round 3 by Cohen’s d . We analyzed the statistical significance of within-group differences between round 1 and round 3 using Wilcoxon signed rank test.We calculated between-group effect sizes for the differences in the change in agreement between the HC and IC groups to get an idea of the magnitude of the intervention effect on the amount of (dis)agreement. We calculated between-group effect sizes or Cohen’s d as the absolute difference between the mean change in agreement across rounds in HC and the mean change in agreement across rounds in IC, divided by the pooled standard deviation. We assessed differences between the HC and IC groups in the change in degree of agreement using Mann-Whitney U tests. Final kappa scores were not adjusted for their baseline values because the subjects of comparison were the differences in the change in agreement across rounds, not the differences in final agreement.We calculated a group’s evidence score to have an idea of the degree of evidence in the answers of each group. We assigned different points to the different levels of evidence. An answer for which a high level of evidence existed got 4 points, a moderate level of evidence got 3 points, a low level of evidence got 2 points, and a very low level of evidence got 1 point. Answers for which there was evidence against got the same points with the opposite sign. Evidence points were multiplied by the number of participants who chose an answer with that level of evidence. The total sum was divided by the highest possible group’s evidence score for the specific clinical question. An evidence score of 1 meant that all group members chose the answer with the highest level of evidence.We estimated the within-group change in evidence score between round 1 and round 3 by Cohen’s d . We analyzed the statistical significance of within-group differences between round 1 and round 3 using Wilcoxon signed rank test.We calculated the between-group effect sizes for the differences in the change in evidence score between the HC and IC groups. We calculated the between-group effect sizes or Cohen’s d as the absolute difference between the mean change in evidence scores across rounds in HC and the mean change in evidence scores across rounds in IC, divided by the pooled standard deviation. We assessed differences between the HC and IC groups in the change in evidence score using Mann-Whitney U tests.We used a hidden camera to record meetings of the informal consensus groups to explore arguments in each group’s decision-making process. Hiding the camera was necessary to avoid social-psychological influences that arise with the awareness of recording. Two analysts anonymously transcribed and independently coded the recorded meetings of the face-to-face groups. Each communicative function within an utterance was defined as a dialogue act. Each dialogue act was coded and classified under a theme. We developed a preliminary list of themes based on the published list of themes created by Gardner et al . We applWe used the length of discussion time as a process measure of the group discussion. Discussion time was defined as the elapsed time between the start and the end of a group’s decision-making activities.We gave the students a paper questionnaire after the consensus process to explore perceptions about the consensus method. After the experiment, participants in the informal consensus group were notified about the hidden camera. Offline, we obtained informed consent to use the results for analysis from all participants. If one of the participants did not agree to allow us to use the hidden camera footage, we did not use the recordings and results of that group.We obtained approval from the University Hospitals Leuven Medical Ethics Committee for this study in December 2009. The full protocol and the approval form can be obtained from the corresponding author.A total of 120 out of 135 students completing a master of nursing and obstetrics participated in the experiment. The participants formed 8 HC and 7 IC groups. Fewer students than expected attended the experiment, so only 7 instead of 8 informal consensus groups could be constituted. A total of 3 students were not assigned to groups. They participated as observers of the group’s process and were not included in analysis. All participants who were randomly assigned to a group were analyzed in their original assigned groups .The 2 groups were similar in terms of age and gender. There were no statistically significant differences in baseline evidence score and baseline agreement at round 1 .P=.02) and for the drug use scenario (d=2.53 with P=.01). Within-group effect sizes in the IC groups varied between 0.39 and 2.33. IC groups showed a significant improvement in the degree of agreement for the therapeutic options scenario (d=2.33 with P=.02). No significant differences in improvement of degree of agreement were found between HC and IC groups. Between-group comparisons revealed that the HC groups showed greater improvement in degree of agreement for the medical imaging scenario (d=0.46 with P=.37) and the drug use scenario (d=0.31 with P=.59). The opposite was true for the therapeutic options scenario (d=-0.9 with P=.10) and the sick leave scenario (d=-1.25 with P=.05). The change in degree of agreement across the 3 rounds, within-group effect sizes, and between-group effect sizes are displayed in Within-group effect sizes (Cohen’s d) varied between 0.26 and 2.53 in the HC groups and were statistically significant for the therapeutic options scenario (d=1.44 with P=.01) and for the therapeutic options scenario in the IC groups (d=2.11 with P=.02). The between-group comparison demonstrated that the human-based computation groups obtained a greater improvement in evidence scores compared to the IC groups, although the difference was not statistically significant. Between-group effect size was 0.56 (P=.30) for the medical imaging scenario, 0.07 (P=.97) for the therapeutic options scenario and 0.89 (P=.11) for the drug use scenario. After 3 rounds, the mean evidence score increased for all clinical scenarios in both groups. Within-group changes showed a significant improvement in evidence score for the drug use scenario in the HC groups (d=3.67 with P=.001). Participant satisfaction scores are shown in Mean discussion time for the 4 clinical scenarios was 32.9 minutes (± 6.5 minutes) in the IC groups and 14.6 minutes (± 2.2 minutes) in the HC groups. Analysis of the time intervals revealed a statistically significant shorter discussion time in the HC groups compared to the IC groups  groups demonstrated closer group agreement compared to the human computation (HC) groups. Opinions were more likely to shift when groups met face-to-face, as suggested by the study of Hutchings et al . The choSupplying the evidence at round 2 had an influence on group judgment (shown by the positive within-group Cohen’s d for the evidence score) as well in the HC groups as in the IC groups. However, thematic analysis in the IC groups revealed that choices were more likely to be based on clinical judgment or conviction, rather than on clinical evidence (as supported by Raine ). The feHutching et al  demonstrThere were some methodological and practical limitations to our study. The limited number of clinical scenarios, especially for the type of case without evidence, may reduce the generalization of the results. It was a proof-of-concept hypothesis-generating study, so we did not power the study before the start. The obtained power was not robust enough for a reliable detection of a between-group effect, which increased our chance of false-negative conclusions concerning statistical significance.The high degree of variability between the individual groups may be seen as realistic reflections of variations in clinical perspective. However, it also confirms the importance of the composition of the guideline panel and the choice of the moderator -25. WhatTime intervals did not represent real discussion times in multidisciplinary guideline panels because of the aforementioned differences between our discussion groups and these expert panels. However, time analysis demonstrated the potential of the human-based computation method to be an efficient consensus method. The thematic analyses may also differ from discussions in expert groups or other guideline development groups. Participants were not specialized in one medical domain and did not have the intention to over-state the effectiveness of their specialist intervention, which may have reduced the amount of contentious issues. The thematic analyses also reflected the attitude towards clinical practice of a new generation of professionals recently educated in evidence-based health care.We gave equal weighting to the different levels of evidence in our calculation of the evidence score. We are aware that not all people give equal weights to a difference between evidence low and evidence moderate, or evidence moderate and evidence high, etc. However, the literature did not provide a scientific basis to assign unequal weights to the different evidence levels. As a consequence, equal weighting of the different evidence levels was considered as appropriate as unequal weighting.Another limitation of the study is the lack of a third online Delphi group without a game component to allow us to separately study the effect of the online approach and the effect of the game itself. While eliminating social-psychological influences inherent in face-to-face groups, we introduced a new psychological element of competition in the HC groups due to the game component. This could have influenced the results at round 2, but did not affect our final results. Participants got the chance to reconsider their answers in a third individual round after finishing the game.The students participated in the experiment during the hours of their class in evidence-based health care. Because of this, we could easily conduct the experiment with all the participants in one room. No additional logistic facilities were required; students had to come to the building for their class. This is in contrast to the suggested advantages of an online process, where experts participate at home. However, the method was originally designed to be a self-directed process, participants worked individually on a computer during the experiment, and no verbal communication with the moderator took place. Therefore, we believe the results could be easily generalized to a real-world online process.Relatively few studies compared different consensus methods for guideline development. Many of them differed in the consensus methods they compared or in the way they operationalized the method. Washington et al  and KadaOur newly developed method of guideline development by human-based computation proved very useful in the introduction of clinical evidence arguments, while neutralizing for social-psychological influences by authoritarian opinions.The findings of our study illustrate once more the importance of the choice of the consensus method in guideline development. Giving the same evidence summary and using a consensus process, HC and IC groups could come to different group views. The influence of the consensus method seemed to depend on the type of clinical question. Overall, the use of the informal consensus method may be appropriate as long as the evidence supports participants’ beliefs or usual practice, or when the availability of the evidence is sparse. However, when some controversy about the evidence exists, one could doubt the appropriateness of the informal consensus method. Because guideline programs are intended to reduce inappropriate variations in health care, guidelines are more important for clinical questions where the evidence shows no resonance with participants’ beliefs. Human computation outperformed the informal consensus method for this type of clinical questions.Participants perceived the human-based computation method as pleasant and enjoyable. Satisfaction was similar in the HC and IC groups. Only 3 out of 56 participants (5%) were dissatisfied with the group answer in the IC groups, while 5 out of 63 participants (8%) were dissatisfied in the HC groups. This is in contrast to the literature on computer-mediated communication versus face-to-face groups ,29, wherHowever, the major strengths of the method  was at the same time a reason for lower satisfaction. Participants perceived lack of group discussion and interaction in the HC groups as a negative aspect of the method. Participants seemed to need the opportunity to find out reasons for other members’ decisions . AlthougThe current format of the CPGame application was built for the purposes of the experiment. This prototype was essential to test the feasibility and the acceptability of the specific method for guideline development. However, if it is to be useful in practice, a more complex application will be needed.We believe it is important to draw on the advantages of both methods (human-based computation and face-to-face meetings) in view of future system improvements. The exploration of group views should be incorporated, while maintaining the existing advantages of human-based computation. A hybrid method could be considered, including an extra button to ask for the arguments of other players to complement the human computation method. An extension to an asynchronous mode would also allow large-scale advantages and let people choose when they participate in the process. We chose the current format of multiple choice questions to test the feasibility of the method because of its plainness. Extensions to other question formats are also feasible and probably more adapted to guideline development.The findings of our study illustrate the importance of the choice of the consensus method in guideline development. Giving the same evidence summary and using a different consensus process, two groups can come to different group views, which implies a considerable risk towards conflicting guideline recommendations on the same topic.Human computation could be a time efficient and acceptable methodology for guideline development specifically for scenarios in which the evidence shows no resonance with participants’ beliefs. Changes in evidence scores and agreement after 3 rounds were higher in HC groups compared to IC groups for this type of scenario. Controlled feedback is given while eliminating the social-psychological components of a group process. Level of evidence and level of agreement are separated, which could increase transparency of the guideline-development process.Future research is needed to confirm the results and to establish practical significance in a controlled setting of multidisciplinary guideline panels during real-life guideline development.


Maarten Taal discusses the potential benefits, risks, and evidence base relating to screening policies for chronic kidney disease. PLOS Medicine:This Perspective discusses the following new study published in doi:10.1371/journal.pmed.1001344Echouffo-Tcheugui JB, Kengne AP (2012) Risk Models to Predict Chronic Kidney Disease and Its Progression: A Systematic Review. PLoS Med 9(11): e1001344. A systematic review of risk prediction models conducted by Justin Echouffo-Tcheugui and Andre Kengne examines the evidence base for prediction of chronic kidney disease risk and its progression, and suitability of such models for clinical use.PLOS Medicine by Justin Echouffo-Tcheugui and Andre Kengne, which examined the evidence base for prediction of chronic kidney disease (CKD) risk and its progression, offers a chance to consider these questions for the clinical management of CKD Advances in understanding the pathogenesis and natural history of diseases as well as developments in medical technology have made it possible to diagnose a large number of diseases at early stages, often in asymptomatic individuals. It is intuitive to believe that earlier diagnosis is beneficial because it creates the opportunity to intervene and prevent progression. This simple paradigm may hold true for some conditions, e.g., hypertension, but, it is increasingly clear that for other diseases early diagnosis may not necessarily be beneficial—e.g., in prostate cancer, since not all cases identified will progress to cause symptoms or premature death. Thus there are calls for health professionals to critically appraise the evidence relating to screening policies, to prevent overdiagnosis  CKD initially seems to be a disease for which screening to facilitate early diagnosis would be beneficial. CKD is relatively common Diagnosis of CKD requires evidence of kidney damage and/or reduced glomerular filtration rate (GFR) that is sustained over at least 3 months Early diagnosis of CKD creates the opportunity for intervention to improve prognosis. Whereas there is clear evidence that even minor reductions in GFR and mild albuminuria are independent risk factors for adverse outcomes, evidence that intervention alters the prognosis in people with mild forms of CKD is sparse. Treatment with inhibitors of the renin-angiotensin-aldosterone system (RAASi) has been shown to slow progression of CKD in patients with diabetes Potential harms resulting from screening for CKD in the general population include the psychological effects of receiving a diagnosis of CKD as well as the burden of potentially having to undergo additional investigation or referral to secondary care. In addition, a CKD diagnosis may harm a person's potential for employment and obtaining life insurance. For health care systems, the risks of screening for CKD include the costs of increased patient visits and tests as well as opportunity costs due to the fact that resources are not available for other services. As far as I am aware, there are no published randomised trials of screening for CKD and the potential harms have not been studied.Despite expectations that screening the general population without diabetes or hypertension for CKD would afford net benefit, there is insufficient evidence to date to inform a recommendation. The United States Preventative Services Task Force (USPSTF) has recently confirmed this view after a comprehensive review of the evidence