Strengthening public health systems has been a concern in Canada in the wake of public health emergencies. In one Canadian province, British Columbia, a high priority has been placed on the role of evidence to guide decision making; however, there are numerous challenges to using evidence in practice. The National Collaborating Centre for Methods and Tools therefore developed the Evidence Informed Public Health Framework (EIPH), a seven step guide to assist public health practitioners to use evidence in practice. We used this framework to examine the evidence literacy of public health practitioners in BC.We conducted a secondary analysis of two separate qualitative studies on the public health renewal process in which the use and understanding of evidence were key interview questions. Using constant comparative analysis, we analyzed the evidence-related data, mapping it to the categories of the EIPH framework.Participants require both data and evidence for multiple purposes in their daily work; data may be more important to them than research evidence. They are keen to provide evidence-based programs in which research evidence is balanced with community knowledge and local data. Practitioners recognise appraisal as an important step in using evidence, but the type of evidence most often used in daily practice does not easily lend itself to established methods for appraising research evidence. In the synthesis stage of the EIPH process, synthesized evidence in the form of systematic reviews and practice guidelines is emphasized. Participants, however, need to synthesize across the multiple forms of evidence they use and see the need for more skill and resources to help them develop skill in this type of synthesis.Public health practitioners demonstrated a good level of evidence literacy, particularly at the collective level in the organization. The EIPH framework provides helpful guidance in how to use research evidence in practice, but it lacks support on appraising and synthesizing across the various types of evidence that practitioners consider essential in their practice. We can better support practitioners by appreciating the range of evidence they use and value and by creating tools that help them to do this. Framework for Core Functions in Public Health . Because you can’t take what works, statistically, with this group and transform it to this group of citizens living in a low income housing unit, for example. Right? Like, you have to have some kind of needs assessment evidence from the community and then try to find some sort of hybrid as to what the best practice says” (Study Participant).Some participants, particularly medical health officers, tended to value epidemiological data and research-derived evidence more highly. On the other hand, those working in food security preferred to draw on community-based experiential evidence over the scientific evidence, seeing it as more valid.But evidence for me is community-based evidence that I see every day. So, the food security work that I do is so grass roots that the evidence just comes from the truth of what I’m being told by the people in my community.Although many participants view evidence as important in guiding and informing their practice, they also believe that evidence is not the only thing to take into consideration. As one person described, it is not just about evidence, but “it’s the process and sometimes it can come back and bite us,” suggesting that lived experience and community development processes are important considerations in public health decision-making. Although evidence, in whatever form, is important to consider as a resource, it is not necessarily the main driver of the work.“When we start seeing a need come out, often times it’s, you know, we’re approached by clients or a group of clients that have come together and said, “You know, we’re seeing a problem with x and such and such. Can you please help us out with it?””“Who is my target group? What is the issue we are dealing with? And what specifically are we trying to change?” [The first step in the EIPH framework is defining the question. This step answers the questions: change?” . ParticiEach participant spoke about the specific health issues on which they were working, such as food safety, food security, various injuries or the causes of injuries such as falls or traffic crashes. However, they talked less about needing evidence on a specific health issue and more about needing evidence to inform processes or practices in the work they were doing. Many participants wanted evidence to demonstrate that an issue was indeed a problem that needed to be addressed more than they wanted evidence about the effectiveness of a program they might be considering. Similarly, others were interested in evidence to justify programs or policies already implemented rather than evidence of effective programs, as reflected in the following quotation:“So there isn’t a lot of traction in going down the road with this, so anything I can get that has some weight to it, including some research backing, whether it is specific to brain injury or not, but something that says this is an area of practice that matters to the health regions gives me justification for staying involved” (Study Participant).Several participants also expressed a need for data or evidence to document the work they were doing, what that meant, and whether it was effective. This could include locally relevant process data or evaluation evidence on the effectiveness of their work. Demonstrating that their work was having an impact was important because, with all the organizational changes and budget cuts going on in the health authorities, participants wanted to demonstrate the value of their work in hopes that additional resources could be obtained. As one participant stated: “We only have two dietitians for the whole island so it would be great to be able to show that what we’re doing is effective and we need more of it.”Each participant identified specific changes they wanted to document through evidence such as: improving population health, reducing the burden of injury, improving indigenous health, developing safety standards, and shifting the health authority mindset from treatment to prevention. For these things, they believe they need locally generated epidemiological and evaluation data as well as research evidence of effectiveness.“It’s so much easier to Google these things now: has anywhere else in the world done this?”“Where should I look to find the best available research evidence to address the issue?” [This second step in the EIPH framework asks the question  issue?” . The queWith respect to helpful sources for locating evidence, participants identified provincial and national sources, professional groups, disease-based foundations, Health Authorities, or local groups and networks as important. They were resourceful and creative in finding relevant and available sources of data. Some relied less on academic journals and more on grey literature, in part because grey literature is more accessible and understandable to them. Again, we note that participants equate data with evidence, even though it may not come from research.formal research, participants said it was important to have access to a university library to track down references and not all of them had that access. They might search for systematic reviews from the Cochrane Database, or other research sites. Randomized controlled trials were important to some, but seen as not always feasible in PH. Expert organizations  were important sources because they are trusted groups that provide good evidence summaries. When it comes to searching for research, however, practitioners often relied on others to do this work:Participants identified three sources of data for obtaining the best available evidence: formal research, big data, and practice. In “Personally, I usually rely on others. I look to my colleagues in Population Health and Wellness. They are usually the ones who have pulled together the lit searches. They have some program experts, so I go to them assuming that I am going to get the latest and greatest information” (Study Participant).Most participants relied on electronic sources such as webcasts, list-serves, and e-mail lists to get updates on research. Distribution lists delivering monthly updates make it easy to scan for a match to something they might be working on. Connecting with peers and colleagues to ask for information is a common practice; finding out what others are doing is easier now with Google.big data sets like the Canadian Community Health Survey or to medical service plan and hospital data. But, not everyone has the skills or confidence to access these data sources; having others available who can do this is important. At the same time that access to data was important, one participant cautioned about hiding behind the need for data before moving to action:For some participants, searching for evidence meant looking to “I think we often hide behind getting the data. We always tell ourselves, we need more data, we need more data. It needs to be more applicable. It needs to be more localized. Well we can move forward still. You don’t have to be stymied by waiting for data. I think sometimes we’re always looking for evidence” (Study Participant).big data might be better than no data. For a local perspective, however, practice can be a valuable source of information.This participant recognizes that there is often a tension between the need for data and the need for action. If local data are not available, waiting for it may delay action unnecessarily, hence For many participants, searching for evidence largely occurred within practice. This involved looking in-house to clinical records, to the strategic information department, vital statistics, or various medical health officers’ reports that identify local priorities and information. They also commented on evidence being embedded in policies, guidelines and regulations found within the context of their work. As discussed previously, many participants displayed a strong preference for obtaining evidence from their communities. Having hands-on experience and getting community input was important evidence for them. They found evidence through windshield surveys or needs assessments. Thus, the community voice is an important part of searching for evidence, and seen by many as the ‘truth’ or the most valid form of evidence.In some cases, locating evidence was informal through sharing at conferences, or talking with local experts, colleagues, and networks, or through partnerships with Masters student projects. In a few instances, evidence-sharing involved formal relationships with researchers. Despite the variety of sources accessed by participants, some identified particular challenges when searching for evidence. For example, access to data might be restricted by organizational agreements or privacy regulations. Practitioners often see too much formalization related to data access.“Well the qualifier then is, what is the quality of that evidence?”Were the methods used in this study good enough that I can be confident in the findings?” Participants clearly recognized the importance of critical appraisal skills for their work:The third step in the EIPH framework is critical appraisal. Here, the quality of study methods is assessed to ascertain whether its findings are “trustworthy, meaningful and relevant” for uptake . The que“But I certainly think that we need to be cognizant of the quality of evidence that we’ve got and what are the shortcomings and strengths of the type of evidence that we have. And where evidence is weak, i.e., it’s not coming from a gold standard type of methodology” (Study Participant).Although some articulated a preference for a traditional hierarchy of research designs, this was tempered by the realization that such studies were not always feasible or ethical to conduct in PH and that other types of research could be useful.“So I guess while, you know, random controlled studies are seen as the best evidence, the ultimate, I’m not sure that they’re the only sources of evidence. Certainly, we do have to look at those other sources as well” (Study Participant).defining the question and searching for evidence sections, several participants preferred evidence that came from the community, and their confidence in the evidence relates to how well the evidence is connected to the local context.Some interviewees spoke about “relenting” to the idea of accepting “promising practices” as evidence in PH when higher quality evidence was not available. At the same time, they recognized the need to be critical about the evidence: “So hopefully being a little bit more critical about what we take in and look at and use. Not just taking it and going ‘well, this sounds good.’ Plop. Let’s use it.” On the other hand, as discussed above in the Reports of local projects or programs may not provide strong evidence of success but may be “good enough” to adapt in the local context and then evaluate, thus contributing to the evidence base. But what constitutes “good enough”? Practitioners have little guidance on making these decisions and the EIPH appraisal step focusses only on appraising research evidence. There may be a risk in taking up an unstudied intervention, but it may be all that is available. Coupled with evaluation, however, participants believe that it can provide useful information.Although not related to the specific appraisal question in the EIPH framework, our participants also spelled out the characteristics of evidence that are important for PH practitioners. They appraise the usefulness of evidence on the extent to which it reflects these characteristics. In general, they require evidence (or data) that is timely, relevant to their context and purpose, current and regularly updated, synthesized and translated into manageable bite sized pieces, trustworthy, and of different types at different levels. Examples are presented in Table “And you know, one needs to wonder… to what extent then after all those layers of translation, does the program that’s put on the ground or the quality improvement agenda resemble the type of evidence in the literature, and to what extent would that even be suited to the context.”“What does the research evidence tell me about the issue?” [In the fourth step of the EIPH framework, users are asked to interpret the evidence they have gathered in synthesized form to produce ‘actionable messages’. They are advised to produce recommendations from the “highest quality and most synthesized research evidence available” . The que issue?” . The synAs intended in the EIPH framework, some participants report drawing on findings from systematic reviews and RCTs and interpreting this evidence to make decisions about practice:“So if we’re trying to recommend, for example, the use of child restraints for motor vehicle safety, I mean we wouldn’t put out that recommendation as is. We would justify it and back it up with a lot of studies that have, you know, shown the use of this, that have been evaluated. We look at systematic reviews, we look at if at all there have been any recognized control trials about data showing the actual use of it” (Study Participant).In addition to systematic reviews, some participants draw heavily on synthesis work as reflected in policy and position papers conducted by trusted agencies and organizations. Other participants discussed using clinical practice guidelines as exemplifying high quality evidence and embodying the synthesis process. This is very much in keeping with the evidence hierarchy discussed on the EIPH website.“So these are the ‘treating tobacco use and dependence’ clinical practice guidelines, the 2008 update. So we use these as kind of like our bible for tobacco control. And that is probably what I would look at as the best evidence right now” (Study Participant).What is very telling, however, is that many participants make reference to a synthesis and interpretive process that involves pulling together evidence from multiple sources such as community needs, local context, and practice experience, in addition to research. One participated stated:“I think a lot of where we find our need is from the community partnerships that we have built with vulnerable populations and the different agencies that work with them. And then we kind of take that information and combine it with evidence that we have, and try to move forward with some kind of an initiative. So there is a real grassroots component to our evidence-finding, as well as taking academic and clinical knowledge and all of that, to make it work.”Here, it is not just high quality research evidence that is being synthesized to make decisions about action but other types of knowledge are integrated with the research evidence. Consistent with the stated purpose of the EIPH that different types of evidence need to be considered in decision making, many participants are putting various types of information together, interpreting that data, and drawing implications for practice. The EIPH synthesis step, however, focuses primarily on using synthesized research evidence.The lack of skills and capacity in the organization to do the synthesis work was a common theme in the data. In some cases, however, practitioners could rely on someone else, either a trusted organization or an in-house staff member who had the skills or for whom it was part of their job, as it was for the participant who made this comment: “It’s part of my job to keep up with the evidence that is there and then to incorporate that into all of the work that I do.” There is very little in the research literature, however, about the process of synthesizing other forms of evidence with research evidence. Participants expressed the need for support in doing this work.“It’s always a matter of adapting. It’s never straightforward – something that we can take from somewhere else and plunk it down here.”“Can I use this research with my client, community, or population?” [The fifth step of the EIPH framework is adapting the evidence to the local context. It addresses the question lation?” . AdaptinSeveral participants who talked about adapting the evidence were well grounded in their communities and expressed a strong sense of the importance of working with community members, not only in adapting evidence to apply to the community, but also in recognizing that initiatives coming from within the community should be made available as evidence for others. As one participant noted:“I think we always adapt the guidance or evidence to our own community. And, it’s kind of always a partnership with other community groups or organizations that we’re working with and how does implementation make sense for our own community? Also, a lot of great initiatives have come from our own community that have gone or may be able to go and move elsewhere as well.”The idea that “knowledge is in the communities” is important to consider when there is a strong focus on evidence-informed practice because participants believe that the local context influences program implementation and success. That is, local knowledge and community engagement can help to modify evidence-informed programs to make them fit the context. At the same time, as the participant quoted above stated “It’s never straightforward”.“I think, for the most part, if there is evidence, I think that would inform most of the planning, implementation, and evaluation of my work. Otherwise, you’re just kind of flying by the seat of your pants.”How will I use the research evidence in my practice?” [Implementing, or applying evidence, is the sixth step of the EIPH framework. It addresses the question “actice?” . At this“As we were doing program planning and implementation, we were basing it on sort of the best knowledge that we could get at that point in time. So, we initiated a number of new programs or new directives and we did try to collect the evidence before we did that to help us decide. So, for example, one was looking at postpartum depression. We then created a program plan for implementation” (Study Participant).Some participants also talked about using evidence as part of communications: “As our Senior Medical Health Officer said in a conversation with me, like we’ve got to use the data to tell compelling stories.” Some expressed concern, however, that evidence might be cherry picked to reaffirm existing practices rather than to inform necessary policy or program changes:“Because they would say that they look at the evidence and they probably do, but I think it’s used more in a symbolic use rather than an instrumental use. …. But symbolic use is more like, I’ve already made up my decision. I’m going to go and find some research to support my decision and I think that’s the strongest use of research in the Health Authority when it comes to decision-making” (Study Participant).Thus, many participants report using evidence to inform program planning, implementation, funding and evaluation decisions and they say they try to use the best available evidence. At the same time, they also indicate that evidence may be used inappropriately in some circumstances to justify what is already being done rather than to make necessary changes.“So, we would use that form of evidence or research to check back on is this working kind of thing – benchmarks.”Did we do what we planned to do?” “Did we achieve what we expected?” [The seventh and final step in the EIPH framework is to evaluate the use of the evidence to inform PH programs. The questions addressed in this step are: “pected?” . The fir“I’ve thought of indicators for success, and that’s where evaluation comes in, of the programs that we are doing. And for the education piece, we did a formal evaluation of that through the research centre here, and we had time to do that”.Additionally, participants recognized the challenges inherent in the process. Evaluation takes time, preparation, and knowledge. As one person noted:“I think that sometimes I feel that I don’t necessarily have the time to do, sort of like a logic model: ‘this is what I’m going to do, these are the indicators, and that …’. You are just going so quickly, and trying to keep up with everything. We have minimal staff time, where you have a lot of things going on.”Participants acknowledge that they may need help doing an evaluation, perhaps from a research centre in the organization. But time for evaluation is always limited. It seems that consistent evaluation, while recognized as valuable, is not factored into the regular workload and is not a high priority in the organization. A lack of funding may contribute to a tension between a recognized need for evaluation evidence to guide practice and the competition for scarce resources.Another issue that influences whether evaluation is done is the organizational capacity for evaluation. Although some participants are comfortable with the evidence-informed process and see evaluation as a natural part of this, others are not as comfortable. Some have taken courses in evaluation and can provide some of that capacity, but overall, evaluation capacity is limited, and is not well supported in many organizations.Some participants reported doing evaluation in a more informal and less rigorous way by asking simple questions like: “Have you learned anything from this program?” For many, a more informal process is all that is feasible for them with such limited resources available for evaluation.“I think it’s hard to think of doing that in a formal process because we are so stretched thin that I don’t have time to come back and report that or make up a document saying this garden was successful because we had five people there digging today. You know, so it’s hard” (Study Participant).Overall, evaluating the use of evidence or the success of evidence-informed programs is not at top of mind for participants, but they did identify the value of evaluating programs, recognizing the need for assistance and organizational capacity. They appear to rely more on informal evaluations to provide feedback for improvements.How do PH practitioners understand and use evidence? How ‘literate’ are PH practitioners in using evidence to inform decision making? What can we do to improve evidence literacy? Do we need any adaptations/modifications to the EIPH framework or does it provide a helpful guide for evidence informed decision-making? In addressing these questions, we draw on related literature and highlight the implications or recommendations for strengthening evidence literacy in PH with a focus on the EIPH framework.Based on our findings, we will answer four questions in the discussion that follows: A secondary analysis of two data sets, focused on the implementation of different PH programs, provided an opportunity to examine the meaning and use of evidence in PH practice using an evidence-informed public health framework. Participants in all three core PH programs across different health authorities described their broad understandings of the term “evidence” as well as their own use of what they considered the best available evidence . They reThe experience of PH practitioners’ use and understanding of evidence has been explored in other health care systems including Australia , the UniAs in our study, Li et al. (2015) found that PH practitioners used evidence for both strategic and practical purposes. Overall, practitioners valued a combination of both “scientistic” , and humanistic evidence that was generated by practitioners or communities. Humanistic evidence was valued for its practical implications and was perceived as more useful for guiding practice. Scientistic evidence was valued for strategic purposes, but was valued less highly. This is very consistent with the findings of our study. These understanding need to be better reflected in existing evidence-informed decision making frameworks such as the EIPH.the ability to: define the PH problem at hand and the need for evidence; search for and locate relevant evidence; appraise, interpret, adapt, and implement the appropriate evidence; and evaluate the use of evidence to inform public health practice.”In recognition that the steps in the EIPH framework mirror the steps in health and information literacy frameworks . They use a range of evidence to help them define important PH issues.Participants were resourceful in their searches for evidence drawing on diverse sources, congruent with their broad definitions of evidence. They knew who to approach for assistance in their own organizations, which organizations could provide reliable and trustworthy evidence in a digestible form, and how to access information and evidence online. They were committed to using evidence, of various types, to inform their practice.Public health practitioners recognize the importance of appraising the research evidence for quality but observed that the type of evidence most often used in PH practice does not easily lend itself to methods for appraising research evidence. They have a more pragmatic set of criteria for appraising evidence than is outlined in the EIPH. Their criteria are based on: a) a view that multiple forms of evidence are needed and should be valued at least as highly as research evidence; b) how practical it is for use in planning and implementing programs; and c) how well it relates to the local context. They do not shy away from making judgements about evidence quality, but struggle with how to appraise the various forms of evidence to make decisions. The EIPH framework focusses only on appraisal of research evidence.Our participants use already synthesized evidence such as systematic reviews and clinical practice guidelines, but these are often not available. When necessary, they do engage in a synthesis process to integrate different types of evidence, but it is not a synthesis of research evidence and does not follow the process laid out in the EIPH framework. They experience challenges in putting all types of evidence together and would like guidance on how to do this better.The participants in our study state that they “always” adapt the evidence to the community context. The need for adaptation is absolutely central to their views of evidence, no matter the source, but they also recognize that the adaptation process must be done in consultation with the community. This is one area where they did not perceive significant challenges.What and how they implement depends on the issue for which they need evidence, the type of evidence available, and how they interpret it. Interpretation is often dependent on the local context and the purpose for which the evidence will be used. Participants report regularly using evidence to support planning and implementation of PH programs and services. Our data on implementation are a bit thin but it is clear that there is strong support of the need for evidence, including research evidence, to implement relevant effective PH programs.Although some participants have evaluation skills or they are available in the organization, most practitioners tend to use informal versus formal evaluation processes. In this there is reliance on “if the community is happy then so are we”. There is limited support and resources in the organization to allow for evaluation.Overall, we conclude that, in general, the individual practitioners we interviewed had a fairly good level of evidence literacy, with most people expressing some limitations in one or more aspects. At the same time, across each of the three HAs they represented, there was a high level of collective evidence literacy; high enough to provide a good foundation for building further organizational capacity for evidence literacy in public health. The areas in which there were challenges include the steps of appraisal, synthesis, and evaluation. With respect to appraisal and synthesis, there was limited if any guidance available in the EIPH framework for the kinds of appraisal and synthesis that PH practitioners saw as most important in their work. This had to do with their strong belief in the importance and value of multiple types of evidence, and the need to bring together, appraise, and synthesize these multiple forms of evidence. The challenges with respect to evaluation were most strongly related to limited organizational capacity to do evaluation.There is an assumption that practitioners should be evidence literate, or that health authorities should be supporting the use of evidence in practice. We have been teaching research utilization and evidence-based practice for decades in Canadian health science universities, and the valuing of evidence is reflected by study participants. There can be problems with relying too much on needing evidence that programs or innovations cannot move forward without the supporting evidence. To combat this, some authors advocate for practice-based evidence, where researchers and practitioners can design and test programs that work in real-world settings . This idRelying on qualitative data, and with no formal test of “evidence-literacy performance”, it is possible that participants over represented their evidence-literacy. Therefore, although we believe that PH practitioners in our study are generally evidence literate, there are always opportunities for improving PH through the use of evidence. There are important questions about evidence literacy that existing frameworks do not address, such as how to appraise, synthesize and integrate locally relevant knowledge, as well as how to synthesize across multiple and different sources of evidence. Given that the EIPH framework was developed for use of research evidence, it is not always clear when and how this integration might occur within the process of evidence-informed public health.evidence collection spreadsheet to synthesize evidence.With respect to the gap in the practice of synthesizing community-based evidence, local surveillance or big data, with research evidence, there may be existing tools that could be integrated into the EIPH framework. Kothari et al.  describeCommunity generated knowledge can also be facilitated by the use of community-based or participatory approaches to research but such processes require additional time, resources, expertise and partnerships , 40. In Although not directly stated by participants, it is clear that there are important values related to, and the privileging of, certain kinds of evidence that impact the importance given to community relevant knowledge in the process of evidence informed decision making. Interestingly, in our data there are examples in which practitioners sought to challenge hierarchies of evidence that privileged traditional sources of empirical knowledge that are often seen as most valuable by researchers. Newton  argues, Overall, the EIPH framework is a helpful guide to developing and improving evidence informed public health practice and decision making. For the most part, however, it provides guidance in relation to the use of research-derived evidence to support practice and decision making but we believe that some modifications would be very helpful to PH practitioners. The framework privileges empirically derived research evidence, and does not provide guidance on appraising or synthesizing all of the types of evidence that practitioners actually use. This was surprising to us given the fact that the EIPH framework explicitly states that evidence-informed decision making draws on four other types of evidence in addition to research evidence. The synthesis step in the EIPH framework emphasizes the retrieval of already synthesized evidence in the form of systematic reviews and yet practitioners report needing to synthesize diverse evidence types to make decisions in practice. There is a heavy emphasis on appraising research from a bio-medical perspective, rather than appreciating the value of community-based evidence that practitioners actually rely on and trust. Where and how are practitioners developing the expertise to appraise and integrate various types of evidence?We agree with the importance of research-based practice in PH and laud the work of the NCCMT in developing and disseminating tools and frameworks to guide evidence-informed public health practice and decision making. But, practitioners use and value multiple forms of evidence and this seems unlikely to change anytime soon. They need practical guidance in how to appraise and synthesize a range of evidence types. We strongly encourage the NCCMT to consider making adaptations to their EIPH framework to address the gaps in evidence literacy we found in this study. Given the experience PH practitioners have developed on their own and in the context of grappling with their practical evidence needs, they could be important allies and informants in the process of adapting the EIPH framework to accommodate their learning and practice needs.There are a number of limitations to this study. Because we did not set out with this evidence framework as an a priori interview guide, participants did not speak directly to each of the questions posed in the evidence framework (see Table Strengthening evidence literacy in PH is an essential aspect of public health improvement processes. Greater gains can be made by embracing the roots of public health by valuing community-based evidence, while supporting and learning from practitioners in the work they do.This study provided an in-depth look at public health practitioners understanding and use of evidence. We found that practitioners interviewed in these two studies demonstrated a good level of evidence literacy. This was particularly evident at the collective level within the organization. While the EIPH framework provides helpful guidance in how to use research evidence, the framework lacks in providing needed support on how practitioners can appraise and synthesize across various types of evidence they consider essential in their practice. Public health practitioners use and value a range of evidence and need tools that take this range into consideration to apply all evidence to practice.


While sugar sweetened beverage (SSB) consumption has declined in the last 15 years, consumption of SSBs is still high among children and adolescents. This research synthesis updates a prior review on this topic and examines the evidence regarding the various health impacts of SSBs on children’s health . We searched PubMed, CAB Abstracts and PAIS International to identify cross-sectional, longitudinal and intervention studies examining the health impacts of SSBs in children published after January 1, 2007. We also searched reference lists of relevant articles. Overall, most studies found consistent evidence for the negative impact of SSBs on children’s health, with the strongest support for overweight/obesity risk and dental caries, and emerging evidence for insulin resistance and caffeine-related effects. The majority of evidence was cross-sectional highlighting the need for more longitudinal and intervention studies to address this research question. There is substantial evidence that SSBs increase the risk of overweight/obesity and dental caries and developing evidence for the negative impact of SSBs on insulin resistance and caffeine-related effects. The vast majority of literature supports the idea that a reduction in SSB consumption would improve children’s health.The online version of this article (10.1186/s40608-017-0178-9) contains supplementary material, which is available to authorized users. Sugar sweetened beverages (SSB) – which include drinks with added sugar such as soda, fruit drinks and energy drinks – are frequently consumed by children and adolescents in the United States (U.S.) . There iIn light of the frequent consumption of SSBs among children and adolescents in the U.S., there has been an interest in critically examining associated health consequences. As a result, there has been a substantial rise in the number of studies investigating the health effects of SSBs over the past decade. Evidence has emerged linking SSB consumption to a number of health consequences among adults including weight gain , 5, cardThis review aims to synthesize the existing evidence regarding the impact of SSB consumption on children’s health. Unlike previous reviews which have been limited in scope  , 15, thiFor each of the health impacts , separate searches were conducted of PubMed, Web of Science and PAIS International. For all searches, a search hedge was created in three parts: 1) terms relevant to SSBs including “beverage” and “sodas”, 2) terms restricting to children and adolescents including “pediatric” and “teens” and 3) terms specific to the outcome being examined such as “body mass index” and “body weight” for the search on overweight and obesity risk  among children and adolescents, with almost all evidence pointing towards a strong positive association Table . While tThe vast majority of cross-sectional studies found evidence for a positive association between SSB consumption and dental caries , 69–82. N = 51) [One cross-sectional study reported null results, finding no association between self-reported SSB consumption and dental caries among Alaska Natives – a result which may have been related to the small sample size (N = 51) .All longitudinal studies included in this review found a positive or mixed association between SSB consumption and dental caries in at least part of the study population –88. One é et al. conducted community-wide interventions to reduce SSB consumption, improve breastfeeding practices, and promote consumption of water for thirst among American Indian toddlers. While the intervention communities demonstrated improvements in the number of dental caries, it is not possible to attribute this specifically to reduction in SSB consumption as the intervention was a multi-pronged approach.Only one intervention study has been conducted to assess SSB consumption and dental caries . MaupoméA growing number of studies reported on the caffeine-related effects associated with SSB consumption with studies almost exclusively cross-sectional Table .Table 4SA number of cross-sectional studies examined the effects of energy drink consumption among children and adolescents –97, withSome of the cross-sectional studies examined caffeine-related effects of cola drinks , 96, 97.One longitudinal study was conducted and it found evidence that increased energy drink consumption was associated with attention deficit/hyperactivity disorder inattention and hyperactivity at 16-month follow-up, but did not find evidence for associations with depression, panic and anxiety .Since the most recent relevant review was published on this topic in 2009 , there hThe majority of this research on SSBs over the past decade has centered on the relationship with weight gain. The findings of this review confirm that there is clear and consistent evidence that the consumption of SSBs heightens obesity risk among children and adolescents. Although a formal quality assessment or strength of evidence evaluation was not conducted, the vast majority of cross-sectional, longitudinal and intervention studies find strong evidence for a positive relationship in all or part of their study population. The exact mechanism through which SSBs impact childhood obesity is not entirely understood. Generally, the research points to the low satiety of SSBs and incomplete compensation , 99. In The association between SSB consumption and weight gain is paramount, given that childhood obesity affects roughly one in six (13 million) children in the U.S., disproportionately impacting children who are low-income and racial and ethnic minorities . From 19This review also finds strong and consistent evidence that consumption of SSBs is associated with dental caries among children and adolescents. The mechanism for the association between SSB consumption and dental caries is well understood: dental caries are caused by acids produced by bacteria metabolizing sugar in the mouth. Increased sugar from SSBs intensifies the acid production and causes further decay of teeth . The majWhile evidence has shown a positive relationship between SSB consumption and type 2 diabetes among adults , 12, 109The findings of this review also point to an association between caffeinated SSBs and a wide range of health issues including poor quality or reduced sleep, headaches, risk-seeking behavior and depressive symptoms. The presence of caffeine in energy drinks and other caffeinated SSBs , in conjunction with the large volumes consumed, can lead to neurological and psychological effects associated with high caffeine consumption. The majority of studies examining the caffeine-related effects of SSBs focus on energy drinks, with very few analyzing the effects of other caffeinated SSBs such as colas. One reason for this may be the considerably higher level of caffeine content in energy drinks: a 250 mL energy drink has an average of 80 mg of caffeine (range: 27-87 mg), compared to 40 g of caffeine (range: 30-60 mg) in a 330 mL cola drink . AdditioWhile there is a large and growing body of research examining the impact of SSBs on children’s health, important gaps remain. First, researchers should utilize more rigorous study designs  and move away from a reliance on cross-sectional studies. This will strengthen the evidence base and allow firmer conclusions to be made regarding the causal relationships between SSB consumption and negative health consequences. Second, more consistency is needed in the definition of SSBs  and measurement strategy . Similarly, more uniformity is needed in assessing outcomes, particularly in the risk of overweight/obesity where studies vary considerably in the outcomes measured . Third, researchers should more rigorously examine differences in health risks by subpopulations  to determine if the intake of SSBs in particularly harmful in certain population subsets. While it is established that low-income and racial and ethnic minorities consume more SSBs, it is unclear the extent to which health consequences are magnified among these groups. This is important particularly for targeting interventions and policy approaches to reduce children’s SSB consumption. Better insights in these areas have the potential to inform real-world policies and recommendations that may greatly benefit children’s health. Finally, additional research is needed about caffeinated SSBs and their impact on children’s health. Energy and sport drink consumption is rising rapidly in the U.S.  and so sThis review has several limitations. First, it only focuses on four main health effects associated with SSB consumption and does not address other potential consequences which have been documented among consumers of SSBs . Second, our conclusions for a particular health consequence did not include a quality assessment and was limited to an informal evaluation of consistency and lack of conflicting studies. Third, article screening was not done in duplicate, although all included articles were confirmed by a second reviewer.This review provides clear and consistent evidence that consumption of SSBs increases obesity risk and dental caries among children and adolescents, with emerging evidence supporting an association with insulin resistance and caffeine-related effects. In general, the strength of evidence for all four health consequences could be improved through the implementation of more longitudinal and intervention studies. Additionally, more consistency is needed from studies in the measurement of exposures  and outcomes  to create a stronger evidence base. Future research should compare low-income and racial/ethnic minority subgroups in order to determine if differences in health risks associated with SSBs exist. Although SSB consumption has declined in the last 15 years, consumption still remains high (61% of children consume at least one SSB per day). The vast majority of the available literature suggests that reducing SSB consumption would improve children’s health.Additional file 1: Appendix.Search Strategies (Contains the full list of search terms and PRISMA diagrams). (DOCX 128 kb)


The decrease in peripheral blood lymphocytes induced by radiation lessens the antitumour effect of the immune response, which might cause immunosuppression. We aimed to investigate the correlation between the decrease in peripheral blood lymphocytes during radiotherapy (RT) and the spleen irradiation dose in patients with hepatocellular carcinoma (HCC).n) and the mean spleen dose (MSD), were calculated using Eclipse treatment planning. Potential associations between dosimetric variables and the Min ALC were assessed by multiple linear regression analysis.The subjects were 59 patients with HCC who had received RT from 2005 to 2014. The Min ALC  was collected from the routine workup for each patient prior to RT and weekly during RT. Spleen dose-volume variables, including the percentage of the organ volume receiving ≥ n Gy (VP < 0.001). The Min ALC correlated with the MSD (P = 0.005), spleen V5 (P = 0.001), spleen V25 (P = 0.026) and spleen V30 (P = 0.018). Controlling for the Karnofsky performance status (KPS), sex, age, Child-Pugh grade, total dose and tumour stage, a multiple linear regression model with bootstrap analysis of 1000 replicates showed that only the spleen V5 was correlated with the decrease in the Min ALC (P < 0.05). According to the receiver-operating characteristic (ROC) curve analysis, the predictive cutoff values of the MSD, V5, V25 and V30 of the spleen for the Min ALC were 227.72 cGy, 17.84, 0.98 and 0.42%, respectively . Furthermore, an MSD ≥ 227.72 cGy  and V5  of the spleen significantly predicted the Min ALC.Peripheral lymphocytes decreased during RT (5 should be limited in clinical practice. Maximum sparing for spleen irradiation during RT is recommended to preserve peripheral blood lymphocytes, which may decrease immunosuppression.Higher spleen irradiation doses were significantly correlated with lower Min ALC during RT for HCC. V The immune system plays a crucial role in cancer suppression and determines cancer prognosis –3. IonizLymphocytes, essential effector cells in antitumour immunity, could specifically recognize and kill tumour cells or release a series of cytokines to activate the host immune system. Accumulating studies have demonstrated that peripheral lymphocytes decrease during the course of radiation therapy (RT) because of their high radiation sensitivity . The minTherefore, we hypothesized that a radiation dose to the spleen would reduce the counts of peripheral blood lymphocytes. To test these hypotheses, we investigated the correlation between the decrease in peripheral blood lymphocytes and spleen irradiation dose in patients with HCC during RT.The subjects for this study were 59 patients with inoperable stage II-IV  HCC who received RT at Shandong Cancer Hospital Affiliated to Shandong University between January 2009 and December 2015. All patients satisfied the diagnosis criteria according to radiological or histological findings as recommended by the American Association of the Study of Liver guidelines , 15. EliAll patients received conventional RT with curative intent up to 50–60 Gy and did not receive other anticancer treatments, except for transarterial chemoembolization, after RT. The study was approved by the institutional review board of Shandong Cancer Hospital Affiliated to Shandong University, and all participants gave informed consent to participate.Detailed clinical data obtained prior to the initiation of RT were collected from enrolled patients and included patient age, sex, Karnofsky performance status (KPS), blood test results, abdominal enhanced computed tomography (CT) or magnetic resonance imaging (MRI), and American Joint Committee on Cancer  stage. Blood samples were obtained by venous puncture and were collected 0–3 days before the start of RT to quantify the lymphocytes; blood samples were collected again once a week during and after RT. Changes in the lymphocyte counts during RT were evaluated. The Min ALCs during RT treatment were measured. The decrease in lymphocytes was quantified by the difference between the baseline values of the absolute peripheral lymphocyte before RT and the Min ALC during RT. Additionally, we analysed the Min ALC and the days on which the Min ALC were measured.x), were extracted from the treatment plan.The spleen was contoured for each patient, and its corresponding dosimetry was calculated and approved using the Pinnacle treatment plan by a dosimetrist. Spleen dosimetric variables, including the mean spleen dose (MSD) and a set of volumetric proportions of the spleen receiving ≥ x Gy  throughout the period of antitumour treatment, and no hepatitis virus activation was observed. After RT completion, contrast-enhanced CT scans or MRI of the abdomen were obtained within 3 months and then every 6 months. Overall survival (OS) was calculated from the completion of RT to the date of death from any cause or the last follow-up.t-test. Bivariate correlations were employed to investigate the correlations between the Min ALC value and spleen dosimetric parameters. Spearman correlation coefficients were applied on the association among different dosimetric covariates, then stepwise backward elimination with a selection criterion of p < 0.1 was applied to find the best subset of variables. The association between spleen dosimetric characteristics and Min ALC was analysed using a multiple linear regression model. The bootstrapping method was applied for internal validation. External validation could not perform in this study. All statistical analyses were performed using SPSS software , and a P value < 0.05 was considered statistically significant from two-sided tests. The procedures were based on the TRIPOD statement [Patient characteristics were evaluated with descriptive statistics. Receiver operating characteristic (ROC) curve analysis was performed to select the most appropriate Min ALC cutoff values for identifying the one-year survival rate and predictive cutoff value of spleen dosimetric variables for the Min ALC. We divided the patients into two groups according to the threshold value of the Min ALC. The lymphocyte cells during RT were compared using Student’s tatement .The characteristics of the 59 evaluable patients are summarized in Table P < 0.001 and area under curve (AUC) = 0.885]. The baseline characteristics of patients with respect to the cutoff Min ALC of 300 cells/μL are shown in Table P < 0.05).At the end of the follow-up period, 19 (33.93%) patients were alive. The 1-year OS rate was 22.03%. ROC curve analysis identified an optimal cutoff point of 300 cells/μL for the Min ALC to predict the 1-year OS [odds ratio (OR) = 28.8; 95%  CI = 27.23–30.37; Fig. P < 0.001). Spearman correlation analysis revealed that spleen dosimetric variables, including the MSD, V5, V25 and V30, were significant factors correlated with the decrease in lymphocytes . Among them, V5 was selected as the best fitting model by stepwise selection. In multiple linear regression, when controlling for patients’ KPS, sex, age, Child-Pugh grades, total dose and tumour stage, only V5 was demonstrated to be an independent predictor for a decrease in lymphocytes .The peripheral blood lymphocytes decreased during RT  for the Min ALC. According to the ROC curve analysis, the predictive cutoff values of the MSD, V5, V25 and V30 of the spleen for the Min ALC were 227.72 cGy, 17.84, 0.98 and 0.42%, respectively  of the spleen significantly predicted the Min ALC  predicts better 1-year overall survival based on 59 HCC patients. We further found that spleen irradiation dose was significantly correlated with a lower Min ALC during RT. Therefore, the maximum sparing for spleen irradiation during RT is recommended to preserve peripheral blood lymphocytes, which may decrease immunosuppression. Furthermore, we identified the predictive cutoff value of spleen dosimetric variables for the Min ALC to identify the spleen dosimetric constraints for clinical practice.The occurrence and progression of a tumour result from the interplay between the tumour-promoting environment and host immune system . LymphocThe spleen is the largest peripheral immune organ, and it plays important roles in regulating immune responses. The specialized structure of the venous system of the red pulp gives it a unique capacity to filter blood. Different cell components in the blood, including a large number of lymphocytes, pass through the spleen on a daily basis , 25. Our5. Therefore, it appears that the low-dose area plays a major role in decreasing lymphocyte counts. Reducing the spleen low-dose area may be critical to reducing the risk of a decrease in the peripheral lymphocyte counts. To the best of our knowledge, this study is the first to explore the correlation between spleen irradiation and peripheral lymphocytes during RT and to further identify the spleen dosimetric constraints used for clinical practice. Our results suggest that we should try to decrease spleen irradiation, which may prevent radiation-induced lymphopenia and improve the overall survival for HCC patients.The spleen is not routinely treated as an at-risk organ in treatment planning, and the tolerance of the spleen to radiation has yet to be fully established. In the current study, the only significant independent parameter associated with the Min ALC, as determined by multivariate analysis, was the spleen VThis study has several limitations. First, it is a retrospective analysis that is subject to all the limitations of post hoc analyses. Second, our sample size is relatively small. Third, our work is the lack of any external validation of these data. This fact, with the low number of cases, could increase the value of the overfitting and returning higher significance level than expected with higher case numbers. Prospective studies are warranted to validate our findings.In summary, our study is the first to demonstrate a dose-response correlation between a higher spleen irradiation dose and lower Min ALC. This finding indicates that the spleen should be treated as an at-risk organ and supports the need to identify possible dose constraints for HCC patients undergoing RT treatment. Prospective studies assessing changes in the peripheral lymphocytes in patients who have undergone spleen sparing during RT may help better define constraints for this organ in the future.


Quantifying evidence is an inherent aim of empirical science, yet the customary statistical methods in psychology do not communicate the degree to which the collected data serve as evidence for the tested hypothesis. In order to estimate the distribution of the strength of evidence that individual significant results offer in psychology, we calculated Bayes factors (BF) for 287,424 findings of 35,515 articles published in 293 psychological journals between 1985 and 2016. Overall, 55% of all analyzed results were found to provide BF > 10 (often labeled as strong evidence) for the alternative hypothesis, while more than half of the remaining results do not pass the level of BF = 3 . The results estimate that at least 82% of all published psychological articles contain one or more significant results that do not provide BF > 10 for the hypothesis. We conclude that due to the threshold of acceptance having been set too low for psychological findings, a substantial proportion of the published results have weak evidential support. While most often this low replicability of statistically significant findings is attributed to publication bias [3], questionable research practices [4] and flawed statistical analyses [5], replicability is also conditional on the degree to which significant results in themselves constitute evidence for an experimental hypothesis. To explore this question, we provide a general overview of the evidential value of significant results in psychological journals.The reliability of evidence published in psychological science recently became a major concern as a high rate of published experiments failed to generate significant results upon replication [th century [6], social sciences have been relying on this logic when providing evidence for their theoretical hypotheses.In general, empirical scientists form hypotheses about the world and collect data to evaluate these hypotheses. In order to quantify their evidence, researchers in psychology and other social sciences use their data to distinguish true effects from random chance. In practice, they overwhelmingly use null-hypothesis significance testing (NHST), which estimates the probability of receiving an observation (or more extreme ones) if the null-hypothesis of no effect is true. If this probability, the p-value, is equal to or less than a preset threshold (customarily 5%) then researchers reject the null-hypothesis and claim support for the effect. Since the development of this procedure in the early 207]. For example, p = .05 is often misinterpreted as 5% probability that the null-hypothesis is true or that chance alone produced the effect [8]. Unfortunately, however, the NHST approach cannot be used to directly evaluate the hypothesis since it estimates the probability of statistical significance under the assumptions of the null-hypothesis, but ignores the probability of statistical significance under the alternative hypothesis. In fact, the long-term rate of false discovery depends not just on the p ≤ 0.05 rule, but also on the prevalence of the effects and the power of the tests. Taking these into account, the false discovery rate of NHST tests is estimated to be much higher than 5% [9]. In sum, although p < .05 results have been interpreted as convincing evidence in support of psychological effects, statistical logic does not admit such use of the method.Nevertheless, it is a common misunderstanding that the NHST method can quantify the evidence for a hypothesis [10–12] allow researchers to effectively employ Bayes factor analysis techniques to quantify the strength of the evidence against, as well as for, the null-hypothesis. A major difference from the traditional statistics is that this Bayes factor computation requires specifying what the hypotheses predict, known as the priors. While the choice of prior can be debated and gives the impression of subjectivity, there are two main reasons why this feature of the method should not be a major concern. First, it is important to see that the need to specify the predictions of the theory is not a limitation, but a basic requirement of science [14]. Second, applications of the method, such as this work, show that the results are often robust to the choice of reasonable priors [15].One way to quantify evidence is to assess how much the data support one or a competing hypothesis. The Bayes factor is a tool that can be used to calculate the ratio of the likelihood of the hypotheses in light of the data, thus expressing how much the observation should change our beliefs about the hypotheses. Recent developments in the method [16](Table 1). Assuming the reader has no prior expectations regarding the outcome of the research, by Bayes factor of 1 we can infer that the evidence is equally in favor of the null and alternative hypotheses, a clearly insensitive case. The range of Bayes factors between 1 and 3 has been suggested as a region for anecdotal evidence for the alternative hypothesis, while Bayes factors between 1 and 1/3 indicate anecdotal evidence towards the null [17]. Bayes Factor 3 indicates that the odds are 3:1 for the alternative hypothesis, allowing 75% confidence in the results. The next level is the moderate evidence, which is used for Bayes factors between 3 and 10, and Bayes factors between 1/3 and 1/10. To be able to claim strong support, Bayes factor > 10 and < 1/10 are considered to be the threshold for H1 and H0, respectively [17].While the Bayes factor is a continuous scale between 0 and positive infinite, customary labels are often used to categorize the strength of evidence [16]. Others suggested that the conventional threshold is too liberal and Bayes factor 6 is more efficient in correctly capturing the effect [18] and 10 provides more compelling evidence [19]. Nevertheless, it is important to emphasize that it would be a misuse to blindly apply Bayes factors as hard cut-off levels. Instead, these indices should inform the reader about the degree to which the observations support one compared to another hypothesis. Although researchers often set expected levels of evidence for their decisions, the relative nature of the support should never be disregarded.Different categories have been suggested as a requirement for a result to be taken as evidential. Originally, Jeffrey proposed that Bayes factor 3 mostly correspond with the level of evidence that is usually obtained with conventional significance testing [20], the question remains as to how much evidence is provided by individual significant results in psychology. A previous investigation compared all p-values and the computed Bayes factors of 855 t-tests published in two experimental journals in 2007 [21]. Those results demonstrated that if p-values are misinterpreted as evidence then they mostly agree with the Bayes factor results about the direction of their support, yet they systematically overestimate the strength of the evidence against the null-hypothesis. It is a question, however, whether this sample of the two journals can be generalized to all domains of psychology. To extend this analysis and assess the overall picture about the strength of evidence in psychological science, we computed Bayes factors with additional robustness analyses for 287,424 significant t- and F-tests using a text-mined dataset of 35,515 psychological articles published between 1985 and 2016 [22]. This dataset contains all reported t- and F-tests from 293 psychological journals from this period of time, representing a substantial corpus of all published research of the field. Here, we use this sample to estimate the strength of evidence that individual significant results provide for the alternative hypothesis.While there is a functional correspondence between p-values and Bayes factors [22]. This dataset contains the metadata and statistical parameters of 688,112 null-hypothesis test results text-mined from 50,845 articles published in 321 psychology journals between 1985 and 2016.The analysis plan of this study was preregistered on osf.io/5rf2x prior the conduct of all the reported analyses. For the analyses, we used a dataset collected by Hartgerink  and we assumed an independent sample design, the most lenient condition for supporting the alternative hypothesis. First, we selected 403,929 test statistics . Using the recalculated p-values we excluded the non-significant test statistics , as the question of our interest was to calculate the degree of evidence behind the significant results. To exclude ambiguous cases from our analyses, we filtered out those results which did not provide the exact values of the test statistics (374 results), as well as the one-sided tests (4878 results), having no information about the direction of the hypotheses of those tests. Furthermore, we removed those test statistics where the recalculated p-values changed whether or not the results were statistically significant using the 0.05 decision threshold (705 results). Next, we rounded up all of the non-integer degrees of freedoms of the t-tests as we assumed that these scores were reduced by a correctional process . We calculated the square root of the F-values and analyzed them as t statistics thereafter. Afterward, we calculated the sample sizes of the results from the reported degrees of freedom assuming independent-sample design. Finally, we took out the results where the overall sample size did not exceed 5 to exclude the extreme cases . After these selections, 114,272 t-tests and 173,152 F-tests remained in our dataset.For our main analysis, we limited our database to the ttest.tstat function of the BayesFactor R package [23]. The calculation of the Bayes factors requires specification of the predictions (prior distribution) of the null- and the alternative models . In cases where the effect size of the alternative model can be high, it is suggested to employ a two-tailed Cauchy distribution centered on zero as a prior distribution of the alternative hypothesis [15]. The scaling of this prior distribution can influence the calculated value of the Bayes factors, and so the level of evidence favoring the hypotheses. Accordingly, for smaller effects a high-scaled prior distribution can underestimate the degree of positive evidence compared to a low-scaled prior. To determine the scaling factor of the Cauchy distribution, we applied the median effect size (d = 0.93) of a collection of 100,000 significant statistical tests, extracted from 10,000 psychology articles, as a rough estimate [25]. As this calculated value could be influenced by a small number of extremely high effect sizes, we decided to employ the so called default JZS prior (√2/2) for our main analyses, which is more lenient towards smaller effect sizes [26]. This value is the default scaling factor of the ttest.tstat function and is considered a ‘medium’ scaling factor [23]. To explore the robustness of our results, we repeated the analyses with different sizes and shapes of the prior distribution  for the alternative hypothesis, while half of the remaining results do not pass the level of even anecdotal evidence (B = 3) . As illu0 for theests see . Repeatiests see . Conside27]. These results demonstrate that without the computation of Bayes factors the interpretation of psychological results remains fairly limited.As Figs 4]. Ad hoc exclusion of outliers [28], selective reporting of p-values [29], and motivated stopping rules for data collection [4] have all been diagnosed in the field as additional factors that increase the rate of false discovery. The preference in publication culture for significant results over nonsignificant results [3] also increases the prevalence of false positive reports and incentivizes researchers to produce significant results. Although the exact impact of these biases is unknown, recent analyses revealed a high level of distortion in published statistical results .It is to be noted, however, that our analyses cannot provide precise estimates to all psychological results, as they cover only the most frequently used statistical methods  and, furthermore, they are subject to the errors in the original reports and to the limitations of text-mining. In particular, it is important to highlight that the analyzed results certainly contain a proportion of manipulation check, therefore, the present results are not exclusive descriptors of focal results. An additional distortion of the sample can originate from the fact that several questionable research practices affect the likelihood of finding evidence for false hypotheses [34]. Relying more on reported confidence intervals [35] or using lower decision thresholds for p-values  [36] have been recommended to improve the use of NHST statistics. Despite the many reform attempts, NHST is still misinterpreted and misused, at an increasing rate in certain areas [37]. While, certainly, the use of Bayes factors can only be part of the solution, their proponents emphasize that besides being able to quantify evidence against, and even for, the null-hypothesis, its use is also less prone to potential biases of multiple comparisons or optional stopping rules [38]. In fact, within this approach, for any study with an insufficient level of evidence, the researchers may continue to collect data until the results allow decisions to be made about the hypotheses [39]. Nevertheless, even without Bayesian calculations great improvement in existing practices can be achieved, for example by simply reporting the largest possible ‘post-experimental rejection ratio’ calculated from the p-value of any standard statistical test [40]. As the new statistical methods become easily applicable [41], researchers and journal editors may become convinced to extend the statistical toolbox of psychological research when evaluating the relationship between theories and their empirical evidence.Changes to statistical practices have been repeatedly proposed in recent decades [19, This study aimed to highlight that disregarding all the intentional or unintentional biases in research practice, a sizable proportion of published results labeled as “statistically significant” provide, in themselves, very little or no convincing evidence for the presence of the effect. From this aspect, it is not that surprising that previous ‘evidential’ findings fail to replicate, rather it is a necessary consequence of the threshold of acceptance having been set too low for the purported effects 19, . As failS1 Supplementary methods and results(DOC)Click here for additional data file.S1 Analysis main text(RMD)Click here for additional data file.S1 Analysis supporting information(RMD)Click here for additional data file.


The aim of this paper is to describe a conceptual framework for how to consider health equity in the Grading Recommendations Assessment and Development Evidence (GRADE) guideline development process.Consensus-based guidance developed by the GRADE working group members and other methodologists.We developed consensus-based guidance to help address health equity when rating the certainty of synthesized evidence . When health inequity is determined to be a concern by stakeholders, we propose five methods for explicitly assessing health equity: (1) include health equity as an outcome; (2) consider patient-important outcomes relevant to health equity; (3) assess differences in the relative effect size of the treatment; (4) assess differences in baseline risk and the differing impacts on absolute effects; and (5) assess indirectness of evidence to disadvantaged populations and/or settings.The most important priority for research on health inequity and guidelines is to identify and document examples where health equity has been considered explicitly in guidelines. Although there is a weak scientific evidence base for assessing health equity, this should not discourage the explicit consideration of how guidelines and recommendations affect the most vulnerable members of society. Using explicit prompts may be helpful in this process This paper provides guidance to address health equity when rating the certainty in synthesized evidence using the Grading Recommendations Assessment and Development Evidence (GRADE) approach. This paper is the third paper in a four-part series on health equity and GRADE, with the introduction [Welch et al.], overall process [Akl et al.], and evidence to decision methods [Pottie et al.].2As discussed in the earlier two papers in this series, several authors have assessed how guidelines consider health inequity concerns . None of these papers focus on rating the certainty of synthesized evidence  using the GRADE approach.3The GRADE approach of presenting the evidence by outcome and the associated certainty  involves the production of summary tables. These tables include evidence profiles (with details on the rating of certainty for each outcome) and summary of finding (SoF) tables that are intended for the public, patients, purchasers, payers, practitioners, product makers , and policy makers a)Include health equity as an outcomeb)Consider patient-important outcomes relevant to health equityc)Assess differences in the magnitude of effect in relative terms between disadvantaged and more advantaged individuals or populationsd)Assess differences in baseline risk and hence the differing impacts on absolute effects for disadvantaged individuals or populationse)Assess indirectness of evidence to disadvantaged populations and/or settings.Five methods can be used to assess health equity with the GRADE approach:3.1If health inequity is considered an important concern by relevant stakeholders, then health equity could be included as an outcome in the Population, Intervention, Comparison, Outcome questions, analytic framework, and SoF table. In doing so, guideline developers must recognize that health equity is primarily assessed with a subgroup analysis. The developers should also note that this may risk excluding other patient-important outcomes, if SoF tables are limited to only seven outcomes as recommended by GRADE. For example, the NICE guideline on maternal and child nutrition identified impact on health inequalities as one of its key priorities and framed its key question as: “What nutritional interventions are effective in improving the health of preconceptual, pregnant, and postpartum mothers and children (up to 5 years) and reducing nutrition-related health inequalities” The direction and size of the effect on health equity is influenced by decisions such as the reference comparator group, use of relative or absolute measures, and whether the outcome is a desirable or undesirable event A lack of evidence about a critical health equity outcome should not be a reason to omit this from the SOF table. Indeed, this should be explicitly identified as an empty row, highlighting the need for further research to answer questions about health equity.3.1.1The Community Guide Water fluoridation guideline 3.1.2“Equity impact” was the primary outcome of a systematic review on interventions to reduce smoking in adults 3.2As described in the previous paper in our series [Akl et al. in this series], the evidence synthesis process should consider the relative importance of different outcomes, determined with input from stakeholders representing disadvantaged groups. The evidence base for these outcomes should then be assessed. Examples of patient importance and health equity were provided in the previous paper in this series such as the importance of inconvenience of a subcutaneous chelation pump for people with sickle cell disease [described in Akl et al. in this series].3.3Average effects obscure differences between subpopulations—that is, subgroup effects may exist. Examining whether effects differ across socioeconomic status or other variables relating to health inequity requires investigating heterogeneity in the treatment effect—for example, using statistical approaches such as meta-regression or subgroup analysis. However, such results may not be available in the literature. There is evidence that systematic reviews underreport subgroup analyses from primary studies P-value associated with a statistical test for interaction, and providing results from within-study comparisons. Sun et al. also showed that subgroup analyses reported in the literature rarely meet these criteria. Evidence synthesis that involves subgroup analyses should therefore consider the full set of credibility issues, using an appropriate checklist, and avoid making conclusions based on chance findings  3.4A higher baseline risk of adverse events in any population may lead to greater absolute harm from an intervention and conversely a higher baseline prevalence of the outcome of interest may lead to greater absolute benefit 3.4.1In 2011, WHO recommended vitamin A supplementation for children aged 6 months to 5 years in countries where vitamin A deficiency is a public health problem (strong recommendation) 3.4.2In Australia, a guideline panel sought to determine the optimal age at which to begin a series of preventive interventions in the Australian Aboriginal and Torres Strait Islander population. The panel recommended preventive interventions at an earlier age than the general population on the basis of higher prevalence of preventable diseases in Aboriginal and Torres Strait Islander populations. For example, type II diabetes is 3–4 times more common than in the general Australian population at all ages, leading to a recommendation for screening starting from age 18, instead of age 40 years for the general population 3.5GRADE quality (or certainty) “reflects our confidence that the estimates of the effect are correct. In the context of recommendations, quality reflects our confidence that the effect estimates are adequate to support a particular recommendation. ‘Quality’ as used in GRADE means more than risk of bias and so may also be compromised by imprecision, inconsistency, indirectness of study results, and publication bias” Indirectness refers to the comparability between the population, the intervention, or the outcomes measured in research studies and those under consideration in a guideline or systematic review As a rule, certainty of the evidence should not be rated down for indirectness for population differences unless there are compelling reasons to anticipate differences in effect due to biology/physiology, sociocultural influences, or setting-specific resource issues that impact the effectiveness or harms of the intervention. In other words, one anticipates a different subgroup effect in either relative or absolute impact of treatment, though evidence is not available to make a formal assessment.  more than the control groups after 1 year. This evidence was rated down for indirectness by the guideline panel because none of the trials included persons with hepatitis C virus (HCV) infection 4In developing this guidance, we identified a number of methodologic challenges. First, assessing effects on health equity is not a linear process. There may be a need to revisit the focus of the guideline during the evidence review process, including the consideration of important disadvantaged groups. NICE does this explicitly by revisiting their key questions regarding health equity throughout the process.Second, there are often limitations in the underlying evidence base including poor reporting of sociodemographic characteristics Third, epidemiologic evidence addressing baseline risk for specific disadvantaged groups may be difficult to obtain for the population or geographic region for which the recommendations are being developed. Health systems at local, regional, and national levels do not have consistent or reliable methods for reporting health status across all sociodemographic indicators of interest. Guideline panels should transparently report how they determined baseline risk estimates.Fourth, assessing directness of evidence depends on the clinical and methodological expertise and judgment of SoF developers. The GRADE Guideline Development Tool includes an explicit checklist when producing SoFs to ask whether the evidence is direct across population, intervention, comparison, and outcome and document the decision for rating down, if performed.5The most important research priority in the field of health equity and guidelines is to systematically identify further examples of how guideline panels have assessed health equity considerations and incorporated these assessments into recommendations using transparent methods. For example, all WHO guidelines make their evidence to recommendation tables and SoFs publicly available for research such as this. These assessments could provide examples of whether, and how, the five issues (a–e) above have been considered for different situations, such as assessing the credibility of subgroup analyses and judging indirectness for disadvantaged populations.In conclusion, the GRADE process provides a structured approach to assess effects on health equity. Health equity considerations warrant increased use of these methods in systematic reviews and guidelines. The findings of assessing health equity using these five steps in guideline development provides a basis for judging “impact on equity” which is part of the DECIDE framework, and details about this process are covered in the fourth paper of this series [Pottie et al.].