Background: Calls for evidence-informed public health policy, with implicit promises of greater program effectiveness, have intensified recently. The methods to produce such policies are not self-evident, requiring a conciliation of values and norms between policy-makers and evidence producers. In particular, the translation of uncertainty from empirical research findings, particularly issues of statistical variability and generalizability, is a persistent challenge because of the incremental nature of research and the iterative cycle of advancing knowledge and implementation. This paper aims to assess how the concept of uncertainty is considered and acknowledged in World Health Organization (WHO) policy recommendations and guidelines.Methods: We selected four WHO policy statements published between 2008-2013 regarding maternal and child nutrient supplementation, infant feeding, heat action plans, and malaria control to represent topics with a spectrum of available evidence bases. Each of these four statements was analyzed using a novel framework to assess the treatment of statistical variability and generalizability.Results: WHO currently provides substantial guidance on addressing statistical variability through GRADE  ratings for precision and consistency in their guideline documents. Accordingly, our analysis showed that policy-informing questions were addressed by systematic reviews and representations of statistical variability . In contrast, the presentation of contextual or "background" evidence regarding etiology or disease burden showed little consideration for this variability. Moreover, generalizability or "indirectness" was uniformly neglected, with little explicit consideration of study settings or subgroups.Conclusion: In this paper, we found that non-uniform treatment of statistical variability and generalizability factors that may contribute to uncertainty regarding recommendations were neglected, including the state of evidence informing background questions  and little assessment ofgeneralizability, alternate interventions, and additional outcomes not captured by systematic review. These other factors often form a basis for providing policy recommendations, particularly in the absence of a strong evidence base for intervention effects. Consequently, they should also be subject to stringent and systematic evaluation criteria. We suggest that more effort is needed to systematically acknowledge (1) when evidence is missing, conflicting, or equivocal, (2) what normative considerations were also employed, and (3) how additional evidence may be accrued. The World Health Organization’s (WHO’s) role as the health authority in the United Nations (UN) system presents a particularly important institutional context for investigating this issue because its functions include establishing standards and articulating evidence-based policies for implementation in a range of settings and countries around the world. Specifically, its formal mandate includes normative powers for the production of policy recommendations.8 This paper aims to assess how the concept of uncertainty is considered and acknowledged in WHO policy statements.The role of research findings and evidence-based public health policy has become of increasing interest to researchers and policy-makers in recent decades.9 Oxman et al found a general lack of systematic and transparent methods for developing evidence-informed WHO guidelines, and that processes for making recommendations relied on content experts rather than methodologists or guideline users.10 Knowledge on the barriers and facilitators of evidence use by decision-makers for health policy appears to support the two communities hypothesis that researchers and policy-makers are two separate groups with distinctly different cultures.13 However, in the decision-making context of WHO as the international agency responsible for health, the division between these groups is not clear. Commentators have debated about the potential consequences of an overlap in WHO’s technical and political functions in executing its mandate, and the need for transparent processes to ensure scientific credibility and legitimacy with an increasingly crowded landscape of actors involved in global health governance.16 The use of evidence by WHO has been previously criticized on several fronts.10 For instance, the diversity of evidence sources used and the transparency of the development process for guidelines has been questioned.17 Since the publication of the WHO’s Handbook for Guideline Development in 2014, WHO recommendations have been further criticized to be inconsistent with GRADE  guidance.18 GRADE is a WHO working group that has developed an approach to grading quality (or certainty) of evidence and strength of recommendations. In general, these critiques focus on the quality and nature of evidence for (or against) the effects of specific interventions. However, as recognized by the 2014 Guidelines, additional sources of evidence are drawn upon to motivate policy recommendations, including “background” evidence. Moreover, the usage of any forms of evidence evaluation , etc nonetheless remain distinct from the final policy recommendations. We propose to evaluate the totality of scientific evidence usage across the policy-making process, paying particular attention to how uncertainty in the scientific evidence base is represented throughout.As the traditionally recognized international institution that sets norms for public health, there is an expectation for WHO to produce policy recommendations that are based on scientific evidence.3 The Canadian Institutes of Health Research (CIHR) define knowledge translation as a “dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge” (http://www.cihr-irsc.gc.ca/e/29418.html). Knowledge translation research is a domain of scientific inquiry by itself,22 producing tools and methods to support implementation of public health knowledge translation activities.24 However, little attention has been paid to the empirical realities of how evidence “survives,” or may be reshaped, by this translation process. There is a nascent body of knowledge about public health policy-making that demonstrates how evidence is but one kind of argument for influencing policy change in a complex network of policy actors25 and how different kinds of research-informed ideas are used by policy-makers to interact within the policy process.26 These studies underline the importance of the realities of policy-makers, and policy theory can inform strategies for evidence-based public health policymaking which better account for their decision-making processes, including how they deal with uncertainty.27 Given the above, it is problematic that few studies on the use of evidence in policy clearly define the terms evidence, policy or policymaker.13For evidence to inform policy recommendations, research findings must still be “translated” for decision-makers regarding their applicability for a given setting or population. Knowledge translation is one strategy to improve the use of evidence in policy.28 Following on from evidence-based medicine,29 evidence for public health policy not only refers to general bodies of information, such as demographic or survey data, but also specific products of empirical social and biomedical research, here termed “scientific evidence,” such as results of disease etiology or intervention studies.2 It is important to note that there is an ongoing scholarly debate on the definition of evidence for public health policy.31In its broadest sense, evidence denotes any “body of facts or information indicating whether a belief or proposition is true or valid.”32 propose three categories of scientific evidence relevant for informing public health policy: first, evidence for causal associations between a disease state and a potential risk factor that can be addressed through a public health policy ; second, evidence of potential impact or effectiveness of a specific intervention ; third, and least commonly, evidence for “external validity” – what features of and contexts for implementing interventions contributed to the efficiency of the policy. We will refer to these types as Causality, Effectiveness, and External Validity, respectively. Empirical strategies for producing this evidence are diverse and lead to a variety of potential products that may be used to validate truth claims in policy. Among these products, norms are used for classifying and ranking quality, or “strength of evidence” that are widely shared by empirical researchers and embodied by preeminent research institutions.Brownson et al35 Many guidelines exist to help those who conduct systematic reviews in the service of policy-making. Institutions such as the Cochrane or Campbell Collaborations36 provide various tools for conducting systematic reviews and integrating their results in translation processes for policy, especially for evidence of first or second type. In these guidelines, sources of information are generally ranked similarly, with scientific studies subjected to peer-review holding the greatest value, with various criteria about study design and analytic methods creating further subdivisions.37 For the purposes of “strength of evidence,” other sources of information are generally considered to be of lesser quality. This includes grey literature or studies not subjected to journal-based peer-review, expert consultation and community-based knowledge.37For example, researchers highly value systematic reviews, exhaustive summaries and syntheses of empirical studies which address a single question or family of questions.38 It is a way for scientists to systematically demonstrate that they do not know a quantity exactly, for example the proportion of individuals under the age of 45, and the extent to which they are unsure of their estimate. This may be due to several reasons, such as not being able to measure everyone in a target population, and can also be applied to estimates of effect, for example the proportion of individuals that would be cured by a given treatment. In practice, the concept is linked directly to confidence intervals circling a potential ‘true’ parameter. For example, a 95% confidence interval means that if the same population is sampled on many occasions and interval estimates are made at each time, the resulting intervals would circle the ‘true’ population parameter in approximately 95% of the cases. Accordingly, very uncertain estimates, as indicated by wide ranges for confidence intervals, can qualitatively provide an indication that the scientific evidence supporting a certain truth claim  is weak. Consequently, statistical variability is an important attribute for the first two types of scientific evidence described above – Causality and Effectiveness. In contrast, “generalizability” is related to the extent to which the results of one study can be applied to other contexts and populations and thus directly corresponds to the third type of scientific evidence – External Validity. This notion of external validity has been largely discussed in academic literature, but remains quite a complex topic that defies simple quantification.39 Faithfully representing uncertainty that arises from empirical research are fundamental to the process of justifying public health policy recommendations that both (i) embrace the variability inherent in empirical evidence-making and (ii) consider the issues of applying conclusions from one population/place to another. We note again, that while evaluations of uncertainty  of the body of scientific evidence and how it should be used, which is accomplished by many expert bodies including the Cochrane collaboration, the in vivo usage of these evaluations and the source findings themselves in policy statements is a distinct process. To our knowledge, there have been no other evaluations of how these collection of tools ensure, or not, faithful translation of evidence (or evidence synthesis) to actual policy recommendations.Uncertainty in the setting of empirical evidence production and translation is understood to arise from various domains. For example, biases in study design and measurement are important source of uncertainty in evidence production. When translating and interpreting empirical evidence, these uncertainties may be represented by specific modalities. Based on the three categories of scientific evidence defined above , we have chosen to focus on two modalities: statistical variability and generalizability. Statistical variability can be defined as “the estimated range of values which is likely to include an unknown population parameter, the estimated range being calculated from a given set of sample data.”In order to address how these two uncertainty components are translated and incorporated in policy recommendations, we chose to focus on four case studies of WHO policy statements, in which the process of evidence-based policy-making has been given significant attention, and where uncertainty, particularly in generalizing recommendations is an essential consideration.Handbook for Guideline Development.37 These aspects are presented in We first reviewed WHO’s standards and rules regarding “evidence-informed” guideline creation to capture the norms and values the organization establishes for this practice. To contextualize our specific policy analyses, we described key features of the WHO guideline development process and the values that motivate them found in the WHO To our knowledge, this is the first study to-date to investigate the translation elements of uncertainty into public health policy recommendations. Consequently, we propose a novel, practical, and adaptable framework as a first attempt to formalize the evaluation and rating of uncertainty translation . The fraA. Use of uncertainty information from empirical studies,B. Critique or discussion of such information,C. Translation into policy recommendation,D. Discussion of new uncertainty introduced by the recommendation.For each item, we propose several criteria or subcategories as shown in 40 for the pilot since it was a policy document where the fields of expertise of both reviewers overlapped. Based on this pilot, modifications to the worksheet including eliminating elements for which there was substantial overlap in content and altering commonly described elements, such as confidence interval, from qualitative descriptions to counts. We did not include any ranking of importance for the domains of the analytic framework.Two reviewers (TB and JYH) completed this worksheet for a common documentation in order to test its content. We chose “Closing the Gap in a Generation,” a report of the WHO Commission on Social Determinants of Health,Each reviewer then selected two WHO guidelines from those produced between 2008 and 2013 based on their expertise in two fields, namely: Maternal and Child Health and Environmental Health. The four cases of WHO policy recommendations/statements selected were: maternal vitamin D supplementation (JYH); infant feeding (JYH); the heat health action plan (TB); and Malaria Policy Advisory Committee (MPAC) recommendations (TB). While a convenience sample not meant to be representative or systematic, these cases included policy statements from a range of settings in which the evidence base was limited (vitamin D) or extensive  or where existing policy was numerous (infant feeding) or rare (Heat).41 by combining the four domains and subcategories with additional emergent themes in a deductive way. Each section of the different documents was thus analyzed using this process. Finally, the data was synthesized using identified themes.Each reviewer first read the two WHO policy statements corresponding to their field of expertise and described each by summarizing the process of its production, the declared aims of the policy, and its general structure. Then, each reviewer completed a worksheet for each statement. Next, the reviewers summarized general themes and results from their respective worksheets. Finally, the reviewers met to discuss common and emergent themes from the reviews. While each reviewer read the notes and worksheets of the other and discussed interpretations that were unclear, no attempts to cross-verify or replicate the results were made. Again, these reviews were intended to elicit themes regarding the translation of empirical findings and not a definitive evaluation. The reviews were conducted independently between the 2 reviewers. Documents were analyzed using thematic analysisWHO Handbook for Guideline Development . This interest was informed by beliefs that pregnant women were widely deficient in vitamin D and findings of associations between deficiency and a number of adverse pregnancy and birth related outcomes (Framework Domain B). The guideline specified that Handbook procedures were followed and that evidence was evaluated by a designated existing Guideline Development Group (GDG), the Nutrition Guidance Advisory Group (Framework Domain B) (pp 1). In summarizing the evidence, the GDG identified one Cochrane systematic review on vitamin D supplementation on maternal and neonatal outcomes and made a strong recommendation against supplementation to prevent maternal pre-eclampsia and a conditional recommendation against routine supplementation (Framework Domain D).Generally, representation of empirical work was qualitative (reporting trends and associations without numbers) when addressing background questions on mechanisms and more precision with numerical estimates and confidence intervals when addressing key (foreground) questions (Framework Domain A). The citation of empirical evidence was fairly evenly split between the two (Framework Domain B). The Cochrane review used to answer key questions was conducted by a WHO epidemiologist of the GDG (Framework Domain C). The summary of evidence, including GRADE criteria, were drawn directly from review and perhaps as a consequence, no further consideration of the strength of the review or any new issues of uncertainty raised was apparent (Framework Domain D). For example, the risk of side-effects from supplementation had a wide confidence interval (risk ratio = 0.17 [95% CI: 0.01-4.06]) and was cited as evidence for no effect, despite substantial uncertainty regarding precision (Framework Domain A) (pp 4). Additionally, questions about the generalizability of the limited randomized controlled trials (RCTs) conducted mainly in France and the United Kingdom or the potential benefits for subgroups, such as vitamin D deficiency, were not clearly addressed (Framework Domain D). Nonetheless, statements regarding key questions were generally measured, confidence intervals were presented to indicate uncertainty, a current lack of high quality studies was acknowledged, and deliberate plans for re-review of evidence in 2016 was stated (Framework Domain D). Moreover, the guidelines were clear to disclose what other factors were considered by the GDG to make recommendations see . On the These guidelines were adapted from existing guidelines and incorporated into consolidated guidance on “Essential Nutrition Actions: Improving Maternal, Newborn, Infant and Young Child Health and Nutrition.” The guidelines for young infant (0–5 months) feeding evaluated for this paper consisted of a set of four policy recommendations: early initiation of breastfeeding at birth, exclusive breastfeeding for 6 months; mother or donor milk feeding for low birth weight infants; and exclusive breastfeeding by HIV-infected mothers. While there was a stated commitment that most recommendations were updated to comply with the evidence-informed standards in the 2012 Handbook (Preface), it is not clear from the Handbook or the guideline statements how such updating would occur, for example whether new GDGs would be convened to re-asses the evidence (Framework Domain C). Moreover, several of the policy statements were developed in conjunction with other groups such as the United Nations International Children’s Fund (UNICEF), further complicating any adherence to Handbook standards (Framework Domain B).44 Moreover, systematic evidence for certain recommended actions, such as maternal support in the workplace was absent, and other means of justification were unclear (pp 14) (Framework Domain C). On the other hand, the lack of evidence on policies for HIV-infected mothers was clearly and transparently stated and the alternative justifications by principles and context were clearly outlined (pp 17-18) (Framework Domain B). Additionally, the guideline was able to use the identification of sub-optimal study types  to justify quality of evidence for an equivocal effect of early breastfeeding promotion practices. Finally, the availability of evidence to address the Member State implementation question and special considerations for subpopulations (in the two policies not related to sub-populations) were not considered despite a strong recommendation for implementation in all countries (Framework Domain D).Overall, statistical variability and confidence intervals were infrequently cited or considered in the presentation of empirical evidence. Similar to the vitamin D guidelines, attempted to use evidence to address background questions as to the importance and burden of child malnutrition through breastfeeding were hampered by uncited facts, and associations cited as causes (pp 2-4) (Framework Domain A). Most notably, the suggestion in the background that there was causal evidence for early breastfeeding in preventing infection-related neonatal mortality cited a single observational study suggesting association (pp 11) (Framework Domain B). Unlike the vitamin D supplementation guideline, however, the lack of consideration for uncertainty was also apparent in the addressing of key questions related directly to the recommended interventions (Framework Domain C). For example, a Cochrane review of community-based interventions to improve maternal and neonatal outcomes was cited as providing a “statistically significant impact on the intuition of breastfeeding within 1 hour of birth” in reference to an average relative risk of 1.94, without any confidence intervals. Additionally, a large confidence interval (7% to 70%) for the effects of early breastfeeding interventions on increasing sustained breastfeeding was considered evidence for being “effective” (pp 14). Relatedly, the generalizability or “indirectness” of summarized evidence was poorly considered, leaving questions of the relevance of systematic reviews to answering key questions as well as the identity of the key questions themselves  (Framework Domain C). Other evidence cited as relevant systematic reviews include references that were not per se, for example summary of findings from expert consultation  (Framework Domain B). Like the supplementation policy, the new uncertainty issues raised by the systematic review were not appropriately considered. Most importantly, while a prominent 2001 systematic review on the impact of exclusive breastfeeding was considered along with a 2009 update, the most recent update in 2012, was not (pp 12) (Framework Domain B). Consequently, subtle issues such as the need to manage infants individually based on growth, or the potential harms of exclusive breastfeeding to iron-deficient infants was overlooked.46 The document’s foreword states that “Recommendations in this publication are based on results of the two-year project on improving public health responses to extreme weather/heat-waves (EuroHEAT).” The stated objective of this report was to “describe the general principles and core elements of national or regional heat–health action plans, gives options and models for interventions and practical examples and tools from various European countries.”This document was motivated by both the extraordinary heat wave that occurred in 2003 in Europe that caused serious health and social issues, and the contemporaneous findings by the Intergovernmental Panel on Climate Change (IPCC) projecting the effects of climate change on human health.The authors of the guidelines did not explicitly present the methodology on which they based their recommendations, but specified that the content was based on research results, experience, and lessons learned (Framework Domain B). They also highlighted that the policy process was mostly based on the EuroHEAT project, which was conducted by an expert panel. This document was also complementary to a technical document “Preparedness and response to heat-waves in Europe, from evidence to action,” which contained their review of the scientific literature. The latter was therefore included in our assessment.infants and children are sensitive to the effects of high temperatures” contrast with other epidemiologic findings including a recent systematic review.47Empirical results  were mostly reported qualitatively (Framework Domain A). Yet in some sections, results and figures, estimates are largely presented with confidence intervals (Framework Domain A). For example, the section about short-term relationships between temperatures and health outcomes highlighted statistical variability, while the section about vulnerable populations did not. Regarding vulnerable populations, it should be further noted that the guideline did not consider some empirical evidence showing opposite results. For example, the statement that “48 was used to describe the phenomenon. Furthermore, neither statistical variability nor generalizability were duly considered when a single study was used to justify the statement (Framework Domain C): “In general, the impact of hot weather on hospital admissions appears to be lower than the impact on mortality.” Moreover, the authors did not include an explicit section acknowledging the limits of available data (Framework Domain B). There is only this brief disclaimer in the foreword: “These suggestions for countries need to be scrutinized for their respective feasibility and applicability on a national or regional basis and may need to be adapted accordingly.” This aspect is not discussed further in the document, nor any motivating evidence provided for guidance.Indeed, similar to the previous policies, the empirical evidence used to motivate background questions, including the etiology of vulnerability to heat, was generally weak (Framework Domain B): A discussion about the determination of the sufficient level of evidence to characterize vulnerability to heat was absent and a single narrative reviewHowever, the authors dedicated a large part of the document asking for further study (Framework Domain D), especially policy evaluation studies and providing further etiologic evidence about heat effects and vulnerable subgroups. They also specifically acknowledged that their recommendations only reflect the current, limited quantity of evidence (Framework Domain D). Additionally, authors contributed a dedicated section, including methods and indicators, to support monitoring and evaluation following policy implementation (Framework Domain D).http://www.who.int/malaria/mpac/en/) which provides a detailed, transparent account of their policy recommendations processes (Framework Domain B). For example, they publish all their expert meeting reports. This committee is supported by evidence review groups and technical expert groups. The former assesses the evidence  in the existing scientific literature following the WHO guidelines while the other make the policy recommendations (see description above). Recommendations are made for very specific objectives in malaria control  (Framework Domain C).We assessed a set of related recommendations made by the MPAC. This committee was created in 2011 to provide independent strategic advice to WHO on developing policy recommendations on malaria control and elimination. They have a dedicated web page . Critically however, limitations of empirical evidence are generally discussed only in the context of studies that present a challenge to their conclusions or recommendations (Framework Domain B). For example, in the evidence review document on Malaria Diagnosis in Low Transmission Settings, the critique of observational studies is mainly present for negative results about the efficiency of a program.On their website (see above), they state how they classify the original data they use (Framework Domain B). There, they largely discuss the limited generalizability of evidence from the original data and their recommendations (Framework Domain D). However, a section devoted to discussing the limitations of their own recommendations is uncommon. Additionally, calls for monitoring and evaluation of recommended programs and updates to the evidence are frequently missing (Framework Domain D).37 the use was non-systematic and in many cases, no numerical or confidence interval information was provided, no conflicting evidence presented, and strength of evidence was generally overstated . In addition, such poor consideration of uncertainty in the presentation of evidence in WHO guidelines can have important implications. An important implication could be for example that it may not adequately acknowledge the importance of context nor encourage the implementation of local experiments for which implementation and impact evaluation would be conducted. Indeed, if the need for translating and adapting interventions for different contexts is not appropriately acknowledged in a given WHO guideline, it may lead to the direct implementation of an intervention. Yet, by doing so, less evidence will be produced across different contexts that may useful to produce knowledge about generalizability resulting in an unproductive feedback loop.First, we must acknowledge that the systematic collection and review of empirical evidence, while the primary fodder for translation evaluated herein, in fact, is only one of many inputs into the WHO policy recommendations development process. Moreover, the practice of adapting existing guidelines, including those developed in conjunction with external partners to adhere to newer standards for evidence-informed policy is not clear or straightforward. Nonetheless, we found the quality of translating uncertainty due to statistical variability and generalizability to be quite predictable with respect to the guidance provided by the Handbook. Specifically, uncertainty was poorly considered in the presentation of evidence used to address background questions of prevalence, mechanisms, and burden or distributions of health problems. Despite guidance to justify any evidence cited,In contrast, the systematic review of evidence, strongly laid out in the Handbook and supported by numerous known tools and frameworks  was more deliberate in the representation of uncertainty including frequent use of confidence intervals (see details in a) commitments to other standards or recommendations were acknowledged and (b) when guideline making deviated from the established standards.We found the use of evidence, even reviews, to be particularly opaque and deterministic, even when issues of variability are reasonably translated, highlighting the role of evidence as a rhetorical rather than an instrumental informant of policy. Even when evidence was systematically reviewed , recommendations often avoided subtle issues raised and may even be unrelated to the evidence produced. These discrepancies were clearest when , better care should be taken with regards to the interpretation of observational evidence including the consideration of contrary findings or alternative interpretations.We believe greater care should be taken in the translation of uncertainty, particularly since it is motivation for policy-making in the first place. Moreover, we emphasize that any participants in WHO policy-making processes should make clear when other factors, such as previous knowledge and commitments to action circumscribe or influence the current use of knowledge. In the case of vitamin D supplementation for example, strong recommendations were made on the basis of limited evidence, presumably based on principles of minimal harms. However, such justifications should be more openly discussed. Moreover, newer fields of empirical data analysis have suggested avenues for dealing with statistical uncertainty including estimating bounds for causal effectsWe also found some practices to be exemplary, including the MPAC, and further policy recommendations working groups should replicate their efforts at transparency. Yet, one may argue that in many public health fields, the presence of strong empirical evidence such as experimental studies is not as readily available as they were for malaria control. It is often the case that policy is recommended in the absence of strong scientific evidence, and that a recognized need from constituents can be a strong and legitimate sufficient motivation. In these cases, it is especially important to be clear when recommendations do not strictly come from strong scientific evidence. Another good example of this was the feeding recommendation for HIV-infected mothers.We note again that this study aimed to apply a novel framework to elicit challenges in the translation of uncertainty from empirical studies to policy recommendation. As such, it was not intended, at this stage, to be as a definitive tool for evaluating WHO or other policy statements. Several limitations must be addressed in order for this tool to become more applicable. First, the domains selected for evaluating evidence transition must continue to be refined for both content and reproducibility. Notably, Domain A which is highly quantitative is likely to be highly reproducible while Domain C, which is qualitative and unstructured, unlikely to be so. Additionally, if the framework is itself to be turned in to a generalizable instrument its replicability needs to be tested by independent reviewers (on the same policy statement) and in other policy contexts. Nonetheless, we found this framework to have good-face validity in capturing many of the concerns we had in the use of evidence in the reviewed statements.Based on our admittedly preliminary analysis, the notion of uncertainty does not appear to be fully and homogeneously addressed in the four cases of WHO policy recommendations we reviewed. Acknowledging that each public health topic has its own challenges towards the two uncertainty components we addressed in this paper and WHO translation through the policy-making process, we fully recognize the limitations of the framework to completely solve such challenges. However, we encourage policy actors with a strong commitment to evidence-based decision making to consider some of the issues we have explored here, with a particular focus on transparency and systematically including sections of guidelines relating to uncertainty. To that end, we propose the following list of sections to be included in WHO policy recommendations and guidelines:Description of how the empirical scientific evidence was assembled and assessed .Description of how a hierarchy in the empirical data was considered and what approach was used .Discussion of the limitations that are presented in the original data. These discussions can include timing of evidence, validity limitations described in the original material, limitations about external validity (Framework Domain B).Discussion of the new uncertainty introduced by the policy recommendations. This section can include discussion about generalizability, changing over time, and local policy contexts (Framework Domain D).Recommendations and explicit guidelines about monitoring and evaluation of the recommendations introduced (Framework Domain D).Additionally, we noted that the role of evidence in informing generalizability was largely avoided in the WHO policy guidelines we reviewed. While this may be a function of WHO’s position as needing to inform broad recommendations for diverse member states, it is our opinion that some guidance should be provided to direct how evidence may be used to inform context-specific recommendations and may be needed even in overarching recommendations. One idea may be to employ novel innovations in causal modeling such as generating effect bounds or agent based modelling to operationalize member-state specific knowledge about compliance or local average effects. The use of qualitative methods to better understand barriers and facilitators for implementing WHO policy recommendations and guidelines in specific contexts are complementary to the methods recommended presently. Approaches to guide where generalizability is or is not informed by scientific evidence may be helpful to better understand potential policy efficacy within specific contexts.51 argue that transparency of decisions about identification and evaluation of appropriate evidence and its use to inform policy is a significant criterion for the good governance of evidence. We suggest that the framework used in this paper for the purpose of analyzing how the concept of uncertainty is interpreted and acknowledged in WHO policy recommendations and guidelines may also be, with more testing, a useful contribution to future revisions to the WHO Handbook for Guideline Development.Policy recommendations made by WHO, the foremost public health organization in the world, carry substantial weight with a variety of decision-makers and practitioners. Our assessment of four WHO policy statements and recommendations found that the presentation of the evidence in these cases lacked a clear and systematic consideration of uncertainty, a problem that is exacerbated by absence of explicit references to the evidence selected and the recommendations made in some instances and the inadequate discussion of new uncertainties generated by such policy recommendation. These findings underscore that even in the institutional context of an intergovernmental organization such as WHO, which has a substantial commitment to evidence-based health policy, the interpretation and consideration of uncertainty is an important problem for the instrumental uses of scientific evidence in policy-making. It is unlikely that WHO policy recommendations in the areas we selected are outliers in their treatment of evidence, and we may expect similar results in WHO recommendations on other issues, in particular those with a dearth of experimental studies. This is why we argue that more work is needed to consider how best to translate uncertainty from empirical studies to policy recommendations, with particular attention given to WHO recommendations which are founded more on normative values or principles rather than empirical bases. Our recommended list of additional sections for inclusion in WHO policy statements, recommendations, or guidelines all relate to increasing the transparency of how evidence is used in these documents. Hawkins and ParkhurstThis work is supported by the Montreal Health Equity Research Consortium  which finances Tarik Benmarhnia and Jonathan Y. Huang postdoctoral awards, although no direct funds were received or allocated for the writing of this paper. Catherine M. Jones is supported by a Vanier Canada Graduate Scholarship from the CIHR, Ottawa, ON, Canada Grant No. CGV127503.Not Applicable.Authors declare that they have no competing interests.TB and JYH contributed equally to this work by conceiving the design of the research. They took the lead in drafting the manuscript. CMJ contributed to the interpretation of the findings, the manuscript writing and provided critical revisions to the manuscript.1Institute for Health and Social Policy, McGill University, Montreal, QC, Canada. 2Department of Family Medicine and Public Health & Scripps Institution of Oceanography, University of California, San Diego, CA, USA. 3Chaire approches communautaires et inégalités de santé, Institut de recherche en santé publique, École de santé publique, Université de Montréal, Montreal, QC, Canada.Taking World Health Organization (WHO) policy statements as a well-studied context for evaluating the translation of scientific evidence, wesuggest that more effort is needed to acknowledge when evidence is missing, conflicting, or equivocal beyond that addressed by systematicreview of intervention effects, including in background knowledge. Consequently, it should be made explicit what normative considerationswere employed in lieu of such knowledge, and where relevant knowledge gaps exist.We propose some recommendations for the consideration of uncertainty in future WHO policy recommendations documents. Theserecommendations may help provide to policy-makers a more accurate picture of the state of evidence.The translation of uncertainty from empirical research findings into policy recommendations is a persistent challenge because of the incrementalnature of research and the iterative cycle of advancing knowledge and implementation. In this paper, we highlight that even when quality of evidenceis systematically evaluated with respect to effects of specific interventions, uncertainty with respect to the state of supporting evidence, includingissues of generalizability, may be overlooked. This paper reinforces the notion that evidence used to motivate policy is multifaceted and systematicevaluation is quite challenging.


To explore the association between health literacy and the risk of cardiovascular disease (CVD), and to assess the differential effects by health literacy level of a nurse-coordinated secondary prevention program (NCPP) in patients with coronary artery disease (CAD).Data were collected in two medical centres participating in the RESPONSE trial . CVD risk profiles were assessed at baseline and 12-month follow-up using the Systematic Coronary Risk Evaluation (SCORE). Health literacy was assessed by the short Rapid Estimate of Adult Literacy in Medicine (REALM-D) and the Newest Vital Sign (NVS-D); self-reported health literacy was evaluated by the Set of Brief Screening Questions (SBSQ-D).p < 0.01]. Nurse-coordinated care seemed to reduce CVD risk irrespective of health literacy levels without significant differences.Among 201 CAD patients, 18% exhibited reading difficulties, 52% had difficulty understanding and applying written information, and 5% scored low on self-reported health literacy. Patients with low NVS-D scores had a higher CVD risk [mean SCORE 5.2 (SD 4.8) versus 3.3 (SD 4.1), Inadequate health literacy is prevalent in CAD patients in the Netherlands, and is associated with less favourable CVD risk profiles. Where many other forms of CVD prevention fail, nurse-coordinated care seems to be effective among patients with inadequate health literacy. Patients with manifest coronary artery disease (CAD) are at high risk of recurrent coronary events and death. Secondary prevention, consisting of a healthy lifestyle and optimal drug therapy, can reduce this risk . HoweverIt is estimated that almost 27% of the Dutch population has limited health literacy, but the prevalence of low health literacy in CAD patients in the Netherlands is unknown . StudiesThe general aim of this study was to gain insight into the prevalence of health literacy among patients with established CAD in the Netherlands, and to investigate the effectiveness of nurse-coordinated secondary prevention on CVD risk in patients with low and adequate health literacy. For this purpose we used SCORE, an assessment tool for CVD risk that is based on age, gender, smoking status, systolic blood pressure, and cholesterol levels.What is the prevalence of inadequate health literacy among patients with established CAD in the Netherlands?What is the association between inadequate health literacy and cardiovascular risk profiles (as assessed using SCORE)?Is there a difference in effectiveness of nurse-coordinated care in patients with inadequate and adequate health literacy?Our research questions were:We performed a cross-sectional survey embedded in the RESPONSE trial, a multicentre randomised, clinical trial in the Netherlands that investigated the effect of a nurse-coordinated prevention program (NCPP) on top of usual care (controls) , 22. TheReferral to the NCPP included up to four visits during the first six months after inclusion. At each visit, patients were seen by a trained nurse specialist. The NCPP focused on  promotinPatients aged 18–80 years were eligible for participation in RESPONSE if they had been hospitalised for an acute coronary syndrome , non-ST-segment elevation myocardial infarction (non-STEMI), or unstable angina). Exclusion criteria were: visits to the prevention program not feasible; not available for follow-up; insufficient mastery of Dutch; surgery, percutaneous coronary intervention or other interventions expected within 8 weeks after index event; limited life expectancy; previously enrolled in an NCPP; NYHA class 3 or 4 congestive heart failure.Participants for our survey were recruited between February and June 2010 in two participating centres of the RESPONSE trial . All RESPONSE participants received an introductory letter describing the objective of the current study and data collection on health literacy, and were subsequently invited for the survey by telephone.During the RESPONSE trial, data were collected at baseline and at 12-month follow-up by patient files and patient interviews, as appropriate. Detailed information on data collection in RESPONSE has been reported by Jørstad et al. , 22.Background characteristics consisted of: gender; educational level ; ethnic background (patients’ and his/her parents’ country of birth); weight; height; cardiovascular history; index event (acute coronary syndrome) and any revascularisation; smoking status prior to index event.Cardiovascular risk was assessed using SCORE, which estimates the absolute 10 years cardiovascular mortality risk based on age, gender, total cholesterol, systolic blood pressure and smoking status [g status . Blood pHealth literacy was assessed in separate personal interviews at or after 12 months of follow-up in RESPONSE. Interviews were performed at the Academic Medical Centre Amsterdam or by telephone. We used the following instruments to measure health literacy: The Rapid Estimate of Adult Literacy in Medicine (REALM) is a word recognition test consisting of 66 health-related words divided into three lists of increasing complexity. Examples are cancer (list 1), hormones (list 2), hypertension (list 3) [(list 3) . Respond(list 3) .The Newest Vital Sign (NVS) includes objective assessments of numeracy and the ability to understand and apply written information. It consists of six questions about the information on a food label , resulting in a total score range of 0–6. A score between 0 and 1 suggests a likelihood of ≥50% of limited literacy, 2–3 indicates the possibility of limited literacy, and 4–6 almost always indicates adequate literacy .The Set of Brief Screening Questions (SBSQ) measures perceived health literacy , 27. It p ≤ 0.01) differentiate between low and high educated patients on the basis of statistically significant differences in mean scores. The correlation of REALM-D scores was strongest with the SBSQ-D scores . The correlation between REALM-D and NVS-D was moderate . For the REALM-D and NVS-D, patients received printed forms by regular mail, in a sealed envelope, labelled with instructions not to open the envelope until the start of the interview. This was done to ensure that the patient did not study the health literacy tests in advance.We previously translated these measures into Dutch , assessed the psychometric properties and evaluated the cross-cultural applicability of the measures –30. The p-values of ≤0.05 were considered significant. We used SPSS statistics V.23 for all statistical analyses.Health literacy scores were dichotomised into adequate and inadequate health literacy per measure, following predefined cut-off points , 29, 30.n = 16), fatigue (n = 4), or family circumstances (n = 2), and 46 patients could not be reached. In total, 201 patients were included in our study (75%). Patient characteristics are shown in Table Fig. Table p < 0.01). This difference remained significant after correction for educational level and ethnic background. After correction for educational level, ethnic background and age, the difference in SCORE was no longer significant. The difference in SCORE for the SBSQ-D was significant when the model was corrected for educational level and ethnic background (p = 0.04) but not in the other models. The difference in SCORE for REALM-D was not significant in any regression model.Table p = 0.22) or between patients with inadequate and adequate health literacy (p = 0.23). Patients with inadequate health literacy in the intervention group showed a greater improvement in SCORE than patients with adequate health literacy. For example, patients who had low health literacy according to the REALM-D had a change in SCORE of −0.96, this change was −0.28 in those that had adequate health literacy. While patients with adequate health literacy in the control group improved in SCORE after 12 months of follow-up, those with inadequate health literacy did not improve. However the observed differences between health literacy groups were statistically not significant.Table Our study shows that inadequate health literacy is highly prevalent in patients with CAD, ranging from 18% who have inadequate reading skills to 52% who have difficulty understanding and applying written information. Patients with low health literacy had significantly worse CVD risk profiles. However, the NCPP led to similar reductions in CVD risk both in individuals with inadequate and adequate health literacy and was thus equally effective for all.Health literacy scores found in our study are comparable with health literacy levels in the general population in the Netherlands, the UK and Ireland , 33, 34.To our knowledge, our study is the first to investigate the impact of health literacy on the effects of secondary prevention by nurse coordinated care as prescribed in the current European guidelines . We obsep = 0.003). This difference is probably related to the fact that our sample was much smaller (201 compared with 754) than the sample in the RESPONSE trial, and that we only recruited in two medical centres that participated in the RESPONSE trial. A larger sample is needed to confirm the significance of the differences in the effectiveness that we found between health literacy groups.Several factors need to be taken into account when interpreting our results. First, we did not find any statistically significant results regarding the effectiveness of the NCPP. This is in contrast to the findings in the RESPONSE trial where good risk factor control was achieved in 35% of patients in the intervention group compared with 25% in the control group at 12 months  were lacking on deceased patients and patients who withdrew consent. Third, the absolute estimates of the SCORE function are inaccurate in secondary prevention. We were unable to use the SMART score for secondary prevention , since CInadequate health literacy is highly prevalent in patients with documented CAD, and is associated with adverse risk profiles. It seems that an NCPP leads to the improvement of risk profiles and that this does not differ between patients with inadequate and adequate health literacy. A larger sample is needed to confirm the significance of the differences in the effectiveness that we found between health literacy groups.Patients with inadequate health literacy are generally less likely to receive and/or follow preventive treatment. However, because of their less favourable CVD risk profile, their need for effective secondary prevention is greater. We found that an NCPP is equally effective across health literacy levels. Where many other forms of prevention fail, an NCPP seems effective among patients with inadequate health literacy and therefore offers a promising concept of secondary prevention of CVD.The protocol for our survey was approved by the institutional committees on human research of both participating hospitals.


The GRADE-CERQual  approach has been developed by the GRADE  Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision making, including guideline development and policy formulation.CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research : (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on making an overall assessment of confidence in a review finding and creating a CERQual Evidence Profile and a CERQual Summary of Qualitative Findings table.We developed this guidance by examining the methods used by other GRADE approaches, gathering feedback from relevant research communities and developing consensus through project group meetings. We then piloted the guidance on several qualitative evidence syntheses before agreeing on the approach.Confidence in the evidence is an assessment of the extent to which a review finding is a reasonable representation of the phenomenon of interest. Creating a summary of each review finding and deciding whether or not CERQual should be used are important steps prior to assessing confidence. Confidence should be assessed for each review finding individually, based on the judgements made for each of the four CERQual components. Four levels are used to describe the overall assessment of confidence: high, moderate, low or very low. The overall CERQual assessment for each review finding should be explained in a CERQual Evidence Profile and Summary of Qualitative Findings table.Structuring and summarising review findings, assessing confidence in those findings using CERQual and creating a CERQual Evidence Profile and Summary of Qualitative Findings table should be essential components of undertaking qualitative evidence syntheses. This paper describes the end point of a CERQual assessment and should be read in conjunction with the other papers in the series that provide information on assessing individual CERQual components.The online version of this article (10.1186/s13012-017-0689-2) contains supplementary material, which is available to authorized users. GRADE-CERQual  is an approach for assessing how much confidence to place in review findings from systematic reviews of qualitative research or qualitative evidence syntheses (QES). The approach is being developed by the GRADE-CERQual Project Group, a subgroup of the GRADE  Working Group. The importance of assessing confidence in qualitative evidence is discussed in the first paper in this series .The GRADE-CERQual approach  is applied to individual review findings from a qualitative evidence synthesis . We defiGRADE-CERQual includes four components for assessing how much confidence to place in review findings: (1) the methodological limitations of the individual qualitative studies contributing to a review finding, (2) the coherence of the review finding, (3) the adequacy of data supporting a review finding and (4) the relevance of the data from the primary studies supporting a review finding to the context  specified in the review question. Making an overall assessment of confidence in a review finding involves moving from the judgements made for each CERQual component to a final assessment. The overall assessment of confidence in a review finding takes into account the concerns identified in relation to each of the four components and, in our experience, is an iterative process that benefits from discussion among the review team or those making the CERQual assessment (when this is being done for an existing synthesis).The aim of this paper, the second in this series Fig. , is to dIn this paper, we discuss the following processes: firstly, moving from a review finding to a summary of a review finding; secondly, determining the review findings to which CERQual should be applied; thirdly, making an overall CERQual assessment of confidence in each individual review finding; and, finally, creating a CERQual Evidence Profile and a Summary of Qualitative Findings (SoQF) table. Papers 3, 4, 5 and 6 in the series describe how to assess each CERQual component  table, which present each review finding along with its CERQual assessment. Fourth, the summaries of review findings in the CERQual Evidence Profiles and SoQF may be used in guideline and other decision making processes to clearly and concisely present the evidence, and confidence in this evidence, to decision makers and other end users. A summary of a review finding therefore needs to be written with the end users and key stakeholders for the review in mind—the review team should consider what these potential users would want to know. Review teams also need to strike a balance between splitting issues emerging from the synthesis into many review findings, resulting in findings that lose their usefulness to end users and their ability to reasonably represent the phenomenon of interest, and creating a smaller number of broad findings that may oversimplify and fail to adequately capture variations across different contexts. Table Our intention is that CERQual can be applied to the full range of types and levels of review findings. For example, a qualitative evidence synthesis may include review findings ranging from more descriptive or aggregative to more interpretive or configuring , as wellThere are important reasons why it may be useful to apply CERQual to as many of the review findings as possible. Firstly, it is easier to undertake a CERQual assessment as part of the process of developing review findings, when the necessary data are easily accessible. Secondly, it is not always possible to predict which findings may be useful for a decision. Applying CERQual to all review findings means that individual review findings can be more quickly integrated into decision making processes. Where the review team decides not to apply CERQual to all review findings, a justification for this should be provided.For a particular synthesis, the review team also needs to decide at which level of interpretation to apply CERQual. For instance, users may want to apply CERQual to a broad theory emerging from a review which, in turn, may draw on several individual review findings. In other cases, users may want to apply CERQual to a set of descriptive review findings . To date, the experience in applying CERQual to more explanatory or interpretive review findings is limited—Additional file No or very minor concerns regarding [methodological limitations/coherence/adequacy/relevance] that are unlikely to reduce confidence in the review findingMinor concerns regarding [methodological limitations/coherence/adequacy/relevance] that may reduce confidence in the review finding [concerns to be described]Moderate concerns regarding [methodological limitations/coherence/adequacy/relevance] that will probably reduce confidence in the review finding [concerns to be described]Serious concerns regarding [methodological limitations/coherence/adequacy/relevance] that are very likely to reduce confidence in the review finding [concerns to be described]Within the CERQual approach, confidence in the evidence is an assessment of the extent to which a review finding is a reasonable representation of the phenomenon of interest. Confidence should be assessed for each review finding individually, and not for the synthesis as a whole. An assessment of our confidence in a review finding is based on the judgements made for each of the four CERQual components –8 Fig. . Each CEimportant concerns that may impact on confidence in the evidence—very minor concerns need not be listed. Tables Few review findings are likely to have no concerns regarding methodological limitations, coherence, adequacy or relevance. However, in making judgements in relation to each of the CERQual components, the review team should be looking for We do not recommend attempting to numerically score assessments for each component as this may give a false sense of precision regarding these assessments. CERQual assessments are judgements and our aim is to make these judgements as explicit and transparent as possible through providing explanations of the reasons for the assessments made. Any concerns with regard to each component should be described in the CERQual Evidence Profile in sufficient detail to allow users of the review findings to understand the reasons for the assessments made. Tables Four levels are used to describe the overall assessment of confidence in a review finding—high, moderate, low or very low—and these are defined in Table CERQual should ideally be applied by the review team to their own review findings, given that prior familiarity with the evidence is needed in order to make reasonable judgments concerning each CERQual component. This has additional benefits for the qualitative evidence synthesis as it provides opportunities for reflection on the way in which the synthesis has been conducted and the review findings formulated.We recommend that CERQual assessments for each review finding should be made through discussion between at least two members of the review team. This is preferable to use by a single review author as it offers an opportunity to discuss judgments within the review team, may assist members of the review team in clearly describing the rationale behind each assessment and can be a helpful part of the iterative and reflexive process of formulating review findings. Because CERQual assessments are judgements, there is likely to be variation across assessors. In such cases, a strength of CERQual is that it provides a system that guides assessors through the key components of this assessment and an approach to reporting that promotes transparency and an explicit record of the judgements involved . We therWhile the focus of this paper is on the application of CERQual by the review team, a CERQual assessment can, in principle, be applied to review findings from well-conducted qualitative evidence syntheses undertaken by others. In this case, sufficient time should be devoted to gaining a full understanding of how the synthesis has been conducted and presented before applying CERQual. Initial guidance for applying CERQual to a synthesis done by another team is provided in Additional file This paper provides broad guidance on how to approach a CERQual assessment. Detailed guidance on factors that may lead the review team to have concerns about a particular component is provided in the papers discussing each component –8.An overall assessment of confidence is based on the assessments for each CERQual component. Currently, CERQual gives equal weight to each component because there is no evidence to suggest that the components should not be given equal importance. Further research is needed on whether giving different weights to different components would be more appropriate.The review team should look iteratively across the four CERQual components in order to make an overall CERQual assessment. Review authors should also note potential interactions and overlaps between the components and avoid downgrading a review finding twice for the same concern across components . For exaCurrently, there is no established order in which to assess CERQual components. Thus far, review teams have typically begun with methodological limitations, but experience suggests that an assessment is an iterative process involving moving between the assessment for each component and the overall judgement of confidence in the evidence. Further research and experience is needed to ascertain if there is an optimal order of applying the components.The overall CERQual assessment for each review finding should be explained in a transparent manner, preferably in a Summary of Qualitative Findings table that includes a narrative explanation of the CERQual assessment (see below). Where a review finding hypothesises a connection or pathway between factors, the accompanying assessment of confidence should take into account the strength of the evidence for this proposed pathway.The CERQual Evidence Profile and SoQF table both intend to provide a structured summary of the review findings and of the information contributing to the CERQual assessment for each finding, with the Evidence Profile providing more detail than the SoQF table. Developing these outputs encourages the review team to carefully consider what constitutes a finding in the context of the synthesis and how to express each review finding clearly. It also helps to ensure that judgements underlying the CERQual assessments are transparent.full review findings reported in the ‘Findings’ or ‘Results’ section of the synthesis. This is the most detailed presentation of each review finding and may include verbatim data extracts from the studies contributing to the review finding as well as the CERQual assessments for each review finding. Secondly, in the Evidence Profile, which includes summaries of the review findings, information on the judgments for each CERQual component underlying the overall CERQual assessment as well as the overall assessment and its explanation. The Evidence Profile is where members of the review team can make explicit and transparent their judgments underlying each CERQual component assessment as well as the judgments underlying the overall assessment. Some users of review findings may find this level of detail helpful. Thirdly, in the SoQF table, which is a shorter version of the Evidence Profile and includes summaries of the review findings, the overall CERQual assessments, an explanation of each CERQual assessment and references to the studies contributing to each review finding. The SoQF table is most useful to users of the review findings .We suggest that review findings and CERQual assessments be presented in three ‘layers’ in a synthesis: firstly, as In the sections below, we describe in more detail the content of the Evidence Profile and SoQF table.A CERQual Evidence Profile is used to provide information on all the component assessments that contribute to the overall CERQual assessment for each review finding. The Evidence Profile includes five elements Tables  and 4:  a summary of each review finding; (2) an overall CERQual assessment for each review finding; (3) reference to the studies contributing data to the review finding, including clarification of the contexts in which those studies were conducted. A SoQF table may also include an explanation of the overall CERQual assessment. As far as possible, a SoQF table should include only findings emerging from the analysis and should not include related background information, for instance on the methods used in the synthesis. Tables Review teams need to decide how best to organise review findings, together with their CERQual assessments, in the SoQF table. In doing so, they should be guided by what would make most sense to those likely to use these findings. Options include organising review findings by broad themes or by settings or population groups; from higher to lower priority in relation to the review question; and by level of confidence, starting with the highest confidence review findings.CERQual is intended to be applied to all review findings reported in a synthesis. It is preferable for all review findings to be reported in the SoQF table regardless of their level of confidence. However, as determined by the requirements of particular audiences, review teams may decide to include only the highest priority review findings in a SoQF table in the main text of the review and to include the remaining findings in a supplementary SoQF table. It may also be useful to include the Evidence Profile as a supplementary table. The following types of review findings should generally be included in the main SoQF table: review findings that relate most closely to the review question; review findings relevant to the decision making process for which the synthesis was commissioned; review findings that are more novel or are unexpected given the current level of knowledge; and review findings that are more likely to affect practice or policy.Review findings that are assessed as very low confidence should also be included in a SoQF table, and review finding 2 in Tables In both SoQF tables and Evidence Profiles, the explanation of each overall CERQual assessment is important to the end user as this shows how the final assessment was reached and increases the transparency of the judgement process. Further, where end users seek evidence for a question that differs slightly from the original synthesis question , they are able to see clearly how the assessment has been made and adjust their own confidence in the review finding accordingly.In other papers in the series, we discuss the implications for future primary research associated with each CERQual component assessment –8. WhereThis paper has discussed the process for making an overall CERQual assessment of confidence in a review finding and outlined how to create an Evidence Profile and a Summary of Qualitative Findings table. The Evidence Profile and SoQF Table can, in turn, feed into decision making processes on health, social or other interventions.To inform a decision on whether to recommend or implement an intervention or policy, findings from a qualitative evidence synthesis regarding, for example, the acceptability and feasibility of an intervention, typically need to be presented alongside findings on other factors, such as the benefits and harms of an intervention and its resource use. Evidence-to-decision frameworks set out in a structured and transparent way what is known about each factor , and can be used to record the judgments of those making the decision regarding each factor as well as how each factor contributed to the decision . SoQF taAs with the GRADE approach for assessing the certainty of evidence of effectiveness , GRADE-CPeer review reports for this article are available in Additional file Additional file 1:Key definitions relevant to CERQual. (PDF 617 kb)Additional file 2:Deciding whether to apply CERQual to a review finding—interpretive or explanatory level findings. (PDF 637 kb)Additional file 3:Guidance on applying CERQual to a qualitative evidence synthesis conducted by another review team. (PDF 650 kb)Additional file 4:Recommended standard phrases for describing the assessment for each CERQual component and the overall assessment. (PDF 598 kb)Additional file 5:Open Peer Review reports. (PDF 154 kb)


Schizophrenia (SCZ) is a complex mental disorder that may result in some combination of hallucinations, delusions and disorganized thinking. Here SCZ patients and healthy controls (CTLs) report their level of confidence on a forced-choice task that manipulated the strength of sensory evidence and prior information. Neither group's responses can be explained by simple Bayesian inference. Rather, individual responses are best captured by a model with different degrees of circular inference. Circular inference refers to a corruption of sensory data by prior information and vice versa, leading us to ‘see what we expect' (through descending loops), to ‘expect what we see' (through ascending loops) or both. Ascending loops are stronger for SCZ than CTLs and correlate with the severity of positive symptoms. Descending loops correlate with the severity of negative symptoms. Both loops correlate with disorganized symptoms. The findings suggest that circular inference might mediate the clinical manifestations of SCZ. Schizophrenia is a mental disorder characterized by hallucinations and delusions. Here the authors report a novel probabilistic inference task in which compared to healthy subjects, schizophrenia patients show greater degree of circular inference that matches the severity of their clinical symptoms. Schizophrenia (SCZ) is a serious and disruptive mental disorder that has a massive effect on public health236813On a normative side, the impairments associated with SCZ described above might be understood within the framework of Bayesian inference, predictive coding or both. In this framework, feed-forward and feedback neural processing are interpreted as a propagation of bottom-up sensory evidence and top-down prior knowledge, which are combined using Bayes' theorem. Through this mechanism, the brain is able to form a deep hierarchical representation of the environment, from low-level sensory features to high-level scene interpretations or behavioural choices15171819Although these two conceptualizations of SCZ  can account for many phenomena, they have not yet been coherently related. We recently showed how E/I imbalances in hierarchical neural processing can result in a pathological phenomenon called ‘circular inference'The present study attempts to validate the circular inference hypothesis by quantifying how patients with SCZ and matched healthy controls (CTLs) derive confidence from sensory evidence and prior information as well as how this process is correlated with symptom severity. We used the Fisher task, a variant of the beads task probabilistic paradigm232526In our variant of the bead task, participants were asked to report the ‘chance'  that a red fish caught by a fisher came from one of two lakes (left versus right lake)c=0 (absolute certainty for the left lake) to c=1 (absolute certainty for the right lake). The average c was close to 0.5  and did not differ between the groups, excluding a bias for one of the two lakes , and a sensory evidence value of 1 corresponded a case in which all of the red fishes were found in only one of the lakes (completely unambiguous). Finally, the prior congruency was defined as the basket size for the lake with the largest proportion of red fishes minus the basket size for the opposite lake . This difference was scaled to vary from −1 (in which the prior contradicted the sensory evidence most strongly) to 1 (in which the prior confirmed the sensory evidence most strongly).We first used a model-free analysis to confirm that the participants account for the different types of information  to derive and report a graded level of confidence about the origin of the fish. The participant's absolute confidence . Moreover, the absolute confidence was substantially lower when the prior was incongruent with sensory evidence , although the effect of prior congruency was strongly modulated by the type of sensory evidence . In trials with unambiguous sensory evidence, the effect of prior congruency was totally abolished (F<1), confirming that the CTLs disregarded prior evidence when the sensory evidence was unambiguous. In trials with ambiguous sensory evidence, the effect of sensory evidence was larger in trials with congruent priors than in trials with incongruent priors . These results confirm that the CTLs accounted for prior evidence when the sensory evidence was ambiguous and revealed that the effect of sensory evidence was steeper when this piece of information was congruent with prior evidence.Among the CTLs, the absolute confidence monotonically increased with sensory evidence and increased notably when the sensory evidence was unambiguous . The conventional indices for jumping to conclusions are provided in F1,520=319, P<0.001) and prior congruency . As F1,797=100, P<0.001) captured the differential effects of prior congruency between the two groups based on the type of sensory evidence. In trials with unambiguous sensory evidence, the effect of prior congruency remained significant in patients with SCZ , in contrast with CTLs . However, in trials with ambiguous sensory evidence, the combination of prior and sensory evidence was similar between the two groups . Similar to the CTLs, the effect of sensory evidence on patients with SCZ was larger in trials with congruent priors than in those with incongruent priors . Together, these results reveal that patients with SCZ integrated prior and sensory evidence differently from the CTLs, and they were incapable of disregarding prior evidence when sensory evidence was unambiguous.In accordance with previous findings29F1,1,127=305, P<0.001) and prior congruency . The patients with SCZ responded more slowly than the CTLs . No interaction was significant (P>0.075), although the patterns of variation for RT closely resemble those for the confidence rate, as was evident after comparing the two columns of Excluding the potential confounding effect of speed/accuracy trade-offs, similar reaction time (RT) patterns were observed for both groups. Both groups responded faster in cases of sensory evidence . Note that logits can vary between negative infinity (when certain left lake) and positive infinity (when certain right lake). Only trials with ambiguous sensory evidence  were used to fit the model parameters. The model prediction for unambiguous trials is reported in c was re-scaled to range between 0.01 and 0.99 (these boundaries were chosen to not significantly affect the results reported here). To avoid confusion, note that the term ‘prior' is not used to describe the probability of the sample given for each lake as it has been in previous studies. We interpreted the (sample+fish) proportions as sensory evidence because all of the corresponding information was visually available at the time of the response. In contrast, the basket size information was provided before the response display, and the response required a prediction based on memorized information. Also note that the target fish colour  did not vary across trials. Thus, the prior was not predictive of the visual display.To compare the participants' behaviour with predictions from exact or circular Bayesian inference, we constructed different models, the predictions of which are displayed in ws, and the prior weight, wp (wp) would still exist stating that this prediction was incorrect . The weighted Bayes model is the binary-choice equivalent of the weighted combination of sensory and prior estimates used in many other studies for continuous variables.The simple Bayes model  assumed ight, wp . For exaαs and αp,which represented the strength of ascending loops (sensory over-counting) and descending loops (prior over-counting), respectively, the circular inference model enabled the sensory evidence, the prior or both to be ‘reverberated' and counted multiple times  for the CTLs and patients with SCZ as a function of likelihood logit (Ls) and prior logit (Lp) are provided in The chance logits (Ls is larger when Lp approximates 0 (and vice versa) indicates that prior and likelihood are not treated as two independent sources of information. Rather, the likelihood information is corrupted by the prior information, or vice-versa, before they are combined. Both phenomena can be considered as a signature of circular inference .The weighted bias and circular inference models were individually fitted on each participant's responses (see ‘Methods' section). To estimate the models' performances while accounting for the difference in their number of free parameters (four per participants for the circular inference versus two for the weighted Bayes versus none for the simple Bayes), we measured each model's Bayesian Information Criteria (BIC) score . SmallerSubsequently, all of the reported parameter values  refer to those fit with the circular inference model. In ws) and prior weight (wp) of the CTLs were 0.64 and 0.56, respectively. This result is not surprising given the qualitative nature of the basket size information (see ‘Methods' section), while the smaller sensory weights might reflect uncertainty about the exact fish proportion in the two lakes. The amount of ascending loops (αs) and descending loops (αp) was moderate in the CTLs, although the contribution of these loops to behaviour remained significant (see the BIC scores). Overall, the behaviour of the CTLs was relatively close to Bayesian, which accounts for the acceptable fit between their responses and the simple Bayes model . The interaction between parameter type and group was also significant . In particular, the patients with SCZ severely over-counted the sensory evidence, as demonstrated by the amount of ascending loops, αs, which was significantly larger than that for CTLs . The mean sensory weight, ws, was also slightly larger in patients with SCZ than CTLs . In contrast, the mean prior weight, wp, was substantially smaller in the patients with SCZ than in the CTLs .The SCZ group differed from the CTLs as demonstrated by the significant effect of parameter type . In unambiguous trials, the participants were instructed to click on the extremity of the scale, which the CTLs did . Intereswp must be treated with some caution because the prior is somewhat qualitative, in contrast with the sensory evidence. For example, part of the logit non-linearity might be because of the distortion in perceived sizes. However, because we have no particular reason to believe that this effect would differ between the CTL and SCZ groups, we continued to interpret differences in wp as representing differences in prior weighting. Because this information must be memorized, the between-group differences in wp might solely be a manifestation of the types of cognitive dysfunction that are highly prevalent in patients with SCZ32P=3.1e–06, wp over the whole sample . However, WM did not differentially affect the other parameters values within the SCZ and CTL groups .The parameter αs) were positively correlated with non-clinical beliefs, which were measured across the whole sample using the 21-item Peter's Delusional Inventory (PDI-21) scale .The amount of ascending loops  and the total number of loops, αs + αp , with a significant interaction (P=0.04). Interestingly, αs was particularly strongly associated with delusions , hallucinations , and unusual thought content .Additional analyses were restricted to the SCZ group. The associations between the model parameters and psychopathology were tested using the Positive and Negative Syndrome Scale (PANSS) scores measured in the SCZ sample. Global severity was associated with the number of ascending loops (or sensory over-counting), αp, did not significantly differ between the SCZ and CTL groups, it varied greatly from patient to patient. Among patients with SCZ, αp (but not αs) was correlated with the severity of negative symptoms , with a significant interaction (P=0.03). The αp parameter was particularly correlated with emotional withdrawal , poor rapport , and lack of spontaneity . Thus, the severity of negative SCZ symptoms might be associated with an over-counting of prior beliefs.Although the severity of descending loops, αs  and αp  as well as with αs+αp  was particularly strongly associated with conceptual disorganization , difficulty in abstraction  and poor attention .In addition, disorganized symptoms (PANSSdis) were correlated with In contrast, affective symptoms were not significantly correlated with the parameters evaluated using the PANSS excitation and depressive factors. This result suggests that different mechanisms are involved in these dimensions, which are common to both SCZ and bipolar disorder.Using the Fisher task paradigm, we compared the degree of confidence that the SCZ and CTL groups derived from combining prior information and sensory evidence. By fitting a model inspired by a hierarchical neural network with impaired E/I balance, we interpreted the between-group differences in terms of specific impairment in Bayesian inference mechanisms. The model successfully captured not only the group-averaged behaviour but also the large inter-participant variability, within both the SCZ and CTL groups.ws) to the likelihood information, most of the difference between the patients with SCZ and the CTLs came from the large number of ascending loops, αs, in the patients with SCZ. In particular, when the fish proportion only slightly differed between lakes, patients with SCZ showed strong differences in confidence . This result cannot be explained by Bayes' theorem alone, regardless of how reliable the sensory information was assumed to be. Even when maximal trust was placed in the sensory evidence (ws=1), a change in the likelihood logit caused an equal change in the chance logit.The most salient results of this study are the large over-counting of sensory evidence and the under-weighting of priors in SCZ. Importantly, the over-counting of sensory evidence cannot be simply explained by an overly large amount of trust placed in the sensory evidence compared with the prior. Although the patients assigned slightly more trust . One explanation for the tendency of patients with SCZ to mistrust the basket information is that they could not always remember it well. In fact, WM in patients with SCZ was correlated with prior weight, wp, but not with the parameters directly associated with the jumping-to-conclusions bias . This result suggests that WM impairments are not a major factor in the over-counting of sensory information and its association with psychosis, which is consistent with the data showing the independence of cognitive deficits and psychotic symptoms in people with SCZ48The manner in which cognitive impairments might have interfered with our results was of particular concern because altered WM might affect the jumping-to-conclusions biasTo date, the central challenge of SCZ research has been the lack of an integrative model that accounts for the heterogeneity of the disorder. By fitting a circular inference model, we were able to capture the three main clinical dimensions of SCZ: positive symptoms , negative symptoms and disorganization. These different clinical dimensions rely on partially overlapping impairments, which correspond to the cortical circuit correlates of the components of the circular inference modelIn addition to the association between ascending loops and psychosis, we found a correlation between negative symptoms and descending loops, which suggests that the tendency of prior beliefs to overcome the sensory evidence is one of the factors that lead to negative symptoms. We also found a correlation between all of the loops and the dissociative features of SCZ. This observation is in line with our theoretical predictions because the presence of both types of circular inference results in dissociations between low-level sensory representations and their high-level interpretationsFinally, one intriguing result of this study was that circular inference was necessary to account for the responses of both patients with SCZ and CTLs. Only a minority of CTLs could be considered as ideal Bayesian observers . Although we did not find a correlation between the PDI score and ascending loops in our CTLs, such a relationship might emerge in a larger, more heterogeneous population , indicating that moderate circular inference is not necessarily pathological  criteria for SCZA statistically valid sample size was defined at 21 participants/group, based on an The participants viewed stimuli on a LCD monitor placed 50 cm away. After successfully performing three separate training blocks with the prior only, likelihood only and the combination of the prior and likelihood information, all 50 participants completed the Fisher task, which consisted of 8 blocks of 30 trials each for a total of 240 trials/participant. The participants were allowed to take breaks between each block. After a fixation cross was presented for 800 ms, the prior, which consisted of two fish baskets (left and right), was presented for 1,000 ms. The participants were told to interpret the size of these baskets as the degree of preference that the fisher had for the left or right lake. After a 50-ms delay during which the fixation cross was again presented, the likelihoods were presented at the same locations where the baskets were presented. The likelihood consisted of the proportion of black and red fishes found within the left and right lakes. A fixed number of fishes was included in each lake, and the values for the prior and the sensory evidence associated with the two lakes were symmetrical .The participants were instructed to report their confidence that a red fish originated from one of the two lakes. A red fish was used for all trials. The participants replied using a semi-circular scale that ranged from 100% confidence that the fish had been caught in the left lake to 100% confidence that the fish had been caught in the right lake. To avoid misinterpretation, we instructed participants to answer at the extremity of the scale . RTs were recorded. After the response was collected, the fixation cross was presented alone during a 200-ms delay before the onset of the next trial. The prior and likelihood information were manipulated for each trial and could take the following values: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8 and 0.9 for the prior and 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 and 1 for the likelihood. The trial sequence for each block was generated using a pseudo-random algorithm.Three distinct variables were manipulated in the Fisher task. First, the size of the basket reflected the prior probability for each lake. For example, a basket size ratio of 8:2 corresponded to a prior probability of 0.8 for the right lake, whereas a 4:6 ratio represented a prior of 0.4. Note that the prior was intentionally qualitative in nature. Thus, although we were unable to compare different participants, each participant could have a different interpretation of the mapping between basket size and prior information (see circular inference model).PR, was 0.80 for the right lake.Second, the proportion of fishes in each lake reflected the sensory evidence in the form of the likelihood that the fish originated from the right or left lake. In contrast to the prior, the likelihoods could be precisely manipulated by changing the proportion of black and red fishes in each lake. For example, if 40 black fishes and 10 red fishes were present in the left lake, and 10 black fishes and 40 red fishes were present in the right lake, then the likelihood of a red fish, c, was the normalized position on the confidence scale . We called this hypothesis the ‘simple Bayes' model, which can be expressed in terms of log odds ratios, xp) imperfectly predicts the origin of the fish . In turn, fish origin imperfectly predicts the sensory evidence provided by the fish colour and the proportions . For the sake of simplicity, we assumed that all three variables could take only one of the two possible states: right lake or left lake  and prior information  100% reliable, we interpreted the hierarchical structure of the Fisher task using a small normative causal model composed of three binary variables , then this model would be mathematically equivalent to the simple Bayes model.In particular, if the participant trusted both the sensory information and the prior completely  resulting in a redundant contribution, αs=αp=0. If the weights are also equal to 1, then it is equivalent to the naive Bayes model. In other words, the two previous models are special cases of the circular inference model.Note that this model is equivalent to the weighted Bayes model for c=0.5 depends on the variance of the responses, not the mean. Because the noise we added is extremely small, it makes no difference except for patients with SCZ featuring extreme levels of ascending loops. In trials with zero sensory evidence, these patients respond randomly at the extremes of the scale, not in the middle. According to the model, the slope of the logit at approximately 0 is so large that any small noise pushes the response to the right or left of the scale, reproducing this behaviour. The predictions of this ‘noiseless' model are shown in To predict the absolute confidence in ws and wp). These parameters were fit on a participant-by-participant basis by minimizing the mean-squared distance between the predicted and reported chance levels, c , on all of the trials with ambiguous sensory evidence. The mean-squared error appeared appropriate because although the participants were estimating probabilities, their responses were also corrupted by their own motor errors, even when they chose the extremes of the scale. However, we also minimized the KL divergence between c and its prediction. This yielded similar results (not reported here), including those of all statistical tests that reached significance.The circular inference model contained four free parameters per participant P<0.05. Continuous variables were compared using paired-samples t-tests when normally distributed or using the Wilcoxon rank sum test . Categorical variables were compared using Pearson's χ2-test.Statistical analyses were conducted using R 3.3. Significance was considered as We analysed the absolute confidence of each group using a general linear mixed model that modelled the parametric effects of sensory evidence, prior congruency and their interaction as fixed effects with repeated measures across participants, whereas the participants were treated as a random factor. Absolute confidence was computed as |(confidence – 0.5) × 2|. The parametric effect of sensory evidence corresponded to the absolute value of the log likelihood ratio. The parametric effect of prior congruency corresponded to the difference between the log prior for the option with larger log likelihood relative to that for the other option. In this sense, the effect of the relative log prior ratio modelled the strength of the prior belief congruent with the sensory evidence. The group-comparison analyses also modelled the fixed effects of group without repeated measures across participants as well as the two- and three-way interactions with the effects of sensory evidence and prior congruency. We modelled the variance-covariance matrix of the fixed effects with a simple variance component that estimated the contribution of the random effect to the variance of the fixed effects without additional assumptions regarding the contribution to the covariation across the fixed effects. To describe the significant interactions between the parametric effects, we performed breakdown analyses restricted to trials with unambiguous sensory evidence  and trials with ambiguous sensory evidence . The same set of analyses was conducted on the RTs to reject the hypothesis concerning the confounding effects of a speed-accuracy trade-off between the two groups.2-test. These analyses are presented in Complementarily, and to facilitate comparisons with previous findings in the literature, the jumping-to-conclusions bias , was explored by examining the degree of confidence in the replies. Three different tests were conducted: (1) a group comparison of the mean confidence level ± the s.d. (Wilcoxon rank sum test); (2) a group comparison of the number of extreme ratings (Wilcoxon rank sum test); and (3) a group comparison of choice quality , k is the number of free parameters , and σ2 is the mean-squared difference between the chance estimate c on each trial and the model prediction, averaged over all participants.where αs, αp, ws and wp). We also investigated the contribution of WM deficits using a multilevel model with αs, αp, ws, wp and WM as within-participant factors.To assess the different factors that might affect group differences in the model parameters, we computed a multilevel model for mixed designs, with parameter value defined as the dependent variable and parameter type as the within-participants factor  and the participant values of The Matlab codes implementing the circular inference model are available in the The data that support the findings of this study are available from the corresponding author upon reasonable request.How to cite this article: Jardri, R. et al. Experimental evidence for circular inference in schizophrenia. Nat. Commun.8, 14218 doi: 10.1038/ncomms14218 (2017).Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Supplementary Figures, Supplementary Methods, Supplementary Discussion and Supplementary References