There is a growing recognition of the ways in which culture and sport can contribute to wellbeing. A strong evidence base is needed to support innovative service development and a 3-year research programme is being undertaken to capture best evidence of wellbeing impacts and outcomes of cultural and sporting activities in order to inform UK policy and practice. This article provides an overview of methods and findings from an initial coproduction process with key stakeholders that sought to explore and agree principles and parameters of the evidence review for culture, sport and wellbeing (CSW).A two-stage DELPHI process was conducted with a purposeful sample of 57 stakeholders between August and December 2015. Participants were drawn from a range of culture and sport organisations and included commissioners and managers, policy makers, representatives of service delivery organisations (SDOs) and scholars. The DELPHI 1 questionnaire was developed from extensive consultation in July and August 2015. It explored definitions of wellbeing, the role of evidence, quality assessment, and the culture and sport populations, settings and interventions that are most likely to deliver wellbeing outcomes. Following further consultation, the results, presented as a series of ranked statements, were sent back to participants (DELPHI 2), which allowed them to reflect on and, if they wished, express agreement or disagreement with the emerging consensus.A total of 40 stakeholders (70.02%) responded to the DELPHI questionnaires. DELPHI 1 mapped areas of agreement and disagreement, confirmed in DELPHI 2. The exercise drew together the key priorities for the CSW evidence review.The DELPHI process, in combination with face-to-face deliberation, enabled stakeholders to engage in complex discussion and express nuanced priorities while also allowing the group to come to an overall consensus and agree outcomes. The results will inform the CSW evidence review programme until its completion in March 2018. The UK What Works Wellbeing Centre has commissioned a 3-year programme of research synthesis and secondary data analysis across three areas: work and learning, community wellbeing, and culture and sport. The evidence reviews are funded by the Economic and Social Research Council from June 2015 to May 2018. This article reports the initial consultation process for the Culture, Sport and Wellbeing (CSW) review. The review is a collaboration between four UK universities and seeks to determine wellbeing impacts in culture and sport practices in diverse communities and contexts, and to establish how evidence can be used effectively to inform policy and practice decisions. This article briefly introduces the policy context relating to culture and sport as well as concepts of wellbeing relevant to the sectors. We then present an overview of the methods and findings from a two-stage DELPHI process with key stakeholders that sought to explore and agree principles and parameters of the CSW evidence review programme.There is a growing recognition of the ways in which culture and sport can contribute to wellbeing. Developments in policy including local commissioning of needs-based health and social care have created new opportunities for culture and sport activities to be integrated into service delivery.4 Recent developments have stimulated policy-related recognition of the importance of measuring and understanding SWB.5 Since 2011, the Office of National Statistics (ONS) has assessed ‘personal well-being’ or ‘subjective wellbeing’ using four dimensions: satisfaction with life, worthwhileness, happiness and anxiety (the ONS4). Recognition of the significance of leisure practices for individual and collective wellbeing was formalised through the inclusion of measures of cultural and sporting engagement in SWB data collection by the ONS from 2013.The importance of measuring and understanding subjective wellbeing (SWB) is becoming established in policy and decision-making in the United Kingdom.7 In the 1970s, leisure providers saw cultural, sports and community interventions as a means of enhancing quality of life.9Scholars have long-recognised the connections between culture and wellbeing, though in varied terminology. The 1930s historians and economists saw spheres of arts and leisure/amusement as antidotes to Marxist notions of alienated labour or routes towards a well-lived life.10 with positive effects on wellbeing relationships reported for taking part in sports, arts, heritage, museums, libraries, and archive activities.11 Significant associations between wellbeing and engagement in cultural and sport practices have been calculated and a financial value placed on participation in the arts , libraries  and sports .12 Physical activity is reported to be associated with increased life satisfaction, one of the ONS4 dimensions of wellbeing, for both men and women.13Worldwide evidence on the drivers, impact and value of engagement in culture and sport is available,15 The relevance of arts to public health concerns has been recognised.16 As well as examining the impact of arts on individuals, research has linked cultural interventions with personal and collective wellbeing benefits for particular social groups, including improvements in public health,17 reduced inequalities,18 cultural value and social capital.19Within the culture sector, a growing body of research has examined the impacts of arts programmes on health and wellbeing in both clinical and community contexts.23 Most recently, policy makers and academics have focused on conceptualising and measuring SWB; the feelings, experiences and sentiments arising from what people do and how they think.20 To date, SWB has been most often assessed by asking people to provide global and retrospective evaluations of their life and experiences . There is a distinction between evaluative and experiential measures of SWB, the former capturing how people think they feel overall, the latter identifying how people feel over time based on what they are doing in a particular setting. Evaluative approaches to SWB are useful to policy making but are likely to provide only partial representations of people’s wellbeing experiences over time and in different contexts. It has been found that cultural and physical activity–related activities  are strongly associated with improved experienced happiness and reduced anxiety.24While the evidence in support of positive wellbeing impacts of culture and sport is growing, there is still a lack of consensus in policy, academic and practice circles about how to conceptualise wellbeing and best methods for measuring wellbeing outcomes.25 Frameworks need to recognise the broad range of evidence types currently used to inform policy and practice decision-making.27 There is some reported ambivalence from commissioners, managers, deliverers and researchers working in sport and culture towards the imposition of evaluation models that privilege methodologies that may not be appropriate for these contexts.1In terms of evidence review work, there are ongoing questions about what criteria to use in judging the evidence for CSW, as well as concern about the influence of hierarchies of evidence drawn from medicine and health disciplines.Supplementary Material). The DELPHI approach offered a way to build consensus through an iterative, structured process that gives all participants an equal voice.The project seeks to work with stakeholders to develop a shared understanding of what works, and what does not work, in culture and sport in order to inform public policy and funding priorities as well as professional practice. The project commenced with a 6-month collaborative development phase, which included a two-stage DELPHI activity involving 57 participants from the UK culture and sport sectors, designed to establish agreed principles and parameters for the 3-year CSW evidence review programme  was devised following consultation at a 2-day event at Brunel University London in July 2015 with a purposeful sample of 34 delegates drawn from national and regional policy, commissioning, research and service delivery organisations (SDOs) in culture and sport. The initial sample was identified by the research team, which includes culture and sport academics from four UK universities, who have been working and collaborating with a wide range of sector stakeholders for several years. Snowball sampling identified further respondents suggested by participants and through consultation with the national What Works delivery partners.31 The results were discussed with a sub-sample of professionals, practitioners and service users from the participating organisations at a series of dissemination events in December 2015, including sessions where researchers observed activities delivered by culture and sport SDOs and discussed evaluation issues with participants. Written informed consent was taken for all data capture activities. The project received ethics approval from the Brunel University London Ethics committee (reference CHLS-RE41-14).Telephone interviews were subsequently conducted with 20 stakeholders who could not attend the workshops. Participants discussed definitions of SWB, research challenges and priorities for the CSW evidence review. Data were audio recorded and transcribed and thematic analysis was undertaken using principles outlined by Braun and Clarke.31 of project data revealed four broad themes including definitions and dimensions of wellbeing, the purpose of evaluation, hierarchies of evidence, and culture, health and wellbeing populations, settings and interventions. These data will be reported on in full in a different paper. Here, we describe the next stage of the consultation process, which sought to establish the key principles and map the parameters of the review using the DELPHI method. The data were drawn together into 10 questions addressed at stakeholders (Supplementary Table 1).Thematic analysiseholders . These wagree strongly) through 3 (neither agree nor disagree) to 1 (disagree strongly). The statements drew directly on the interpretive process of the qualitative data and adopted the same language and phrasing used by participants during discussions and conversations in order to ensure a grounded approach to the consensus development process. Hence, the questionnaire design process was inductive: the questionnaire, while it covers many issues that one would expect to see in a discussion of evidence review for culture and sport, is specific to the project and not intended as a generally applicable tool.For each question, respondents were asked to rate how much they agreed or disagreed with up to eight pre-set responding statements by rating each of them from 5  test compared the means of groupings with each other to identify similarities and differences in group preferences.The DELPHI 1 questionnaire was sent to 54 participants identified from the workshops including commissioners and managers (14), policy makers (14), scholars and fellows (13), and representatives of SDOs (13). Questionnaire data were entered into a spreadsheet and imported into statistical analysis software . As well as examining overall rankings, we compared the rankings for different sub-groups with the overall means in order to explore possible disagreements and nuanced standpoints of the different stakeholders. In view of the small numbers in the groups, we calculated The results were shared with stakeholders in collaborative dissemination events before a second questionnaire was sent out to 57 people . In the DELPHI 2 questionnaire, the top five statements from DELPHI 1 were presented in rank order, with a rank of 1 indicating the most important statement, a rank of 2 indicating the second most important statement and so on. Out of a total of 69 statements, 9 were excluded at DELPHI 2; hence, this second questionnaire was not substantially different in scope from DELPHI 1. The purpose of DELPHI 2 was to offer participants the opportunity to mark any disagreements with the DELPHI 1 results. However, given that the DELPHI 1 results had been arrived at via a considerable consultation process we did not expect substantial disagreements at this stage.Participants who disagreed with the order in which the statements were presented were able to assign new ranks; otherwise, they did not assign new ranks to the statements. It was made clear that answers that were left blank would be treated as agreeing with the initial rankings.Supplementary Table 1). In one case, three people from the same organisation completed a joint response, which was counted as a single response. The results (Supplementary Table 1) show the range of priorities and illustrate key issues that are pertinent to the review process, including definitions and dimensions of wellbeing, the role of evidence, approaches to quality assessment, and the populations and settings and interventions most likely to deliver wellbeing in culture and sport. The statements and rankings reflect the range of responses discussed during workshops and interviews.The DELPHI 1 questionnaire was returned by 38 participants: commissioners and managers (9), policy makers (11), scholars and fellows (10), and representatives of SDOs (8) .As well as determining the overall rankings of statements, we examined variations in ratings between the groups using unpaired p = .0369). SDO rankings also differed from the overall rankings in their responses to the statement, ‘To what extent should a hierarchy of evidence inform the culture and sport evidence review?’ The statement ‘The review should include anecdotal evidence and testimonials’ was ranked first by SDOs  compared with fifth overall . This difference was very significant (p = .0047). There were non-significant (p = .5041) differences in rankings of the statement, ‘Evaluation should be embedded in practice and should not disrupt service delivery’. This was ranked second by SDOs  compared with fourth overall .When asked ‘What is meant by wellbeing in the context of the culture and sport evidence review’, the SDO group gave second ranking to the statement ‘we need to adopt an inductive approach that assesses wellbeing in each specific context’  = 0.926). This statement was ranked fifth overall . The difference was significant . This statement was ranked fifth overall .When asked ‘what is the purpose of evidence in culture and sport?’, scholars gave second ranking to the statement ‘Evaluation should be as independent as possible’ , compared with fifth overall . Policy makers ranked this statement third . An ANOVA test revealed that overall, the groups voiced similar preferences with few significant differences. Hence, the scholars and policy makers did not have a significant difference of opinion from each other, or all the respondents taken together.The analysis shows that there were just two statements in relation to which significant differences between groups were found, both of which related to SDOs. The statistics show that with these two exceptions all the groups are in agreement, giving very similar responses to the questions in DELPHI 1. This would suggest that the resulting consensus statement can enable a common strategy to be drawn up to assess the effect of policies and initiatives on wellbeing with which all the groups can agree.Supplementary Table 2), bringing the total of responses to 40 (70.02%). After two follow-up communications, eight respondents returned the questionnaire with the original rankings confirmed, or sent an email or spoke to a researcher on the phone confirming their agreement. A further six did not reply and, as they had been advised, their responses were logged as signalling agreement. A total of 26 respondents returned the questionnaire with minor changes, but these did not result in any changes to the overall rankings (Supplementary Tables 5–10). The final results show a strong pattern of agreement, which is perhaps unsurprising given that the DELPHI 2 questionnaire came at the end of a 6-month in-depth face-to-face and online consultation. To a degree, all the statements are important to stakeholders and have the potential to inform the evidence review. While it is not necessary for the purposes of this exercise to impose a rigid threshold of consensus, here we make a perhaps arbitrary distinction between statements that made it into the top three and those ranked fourth or fifth in order to highlight key priorities.Two commissioners/managers who had not responded to DELPHI 1 completed DELPHI 2 (Supplementary Table 5), the highest ranked statement  was ‘we need a common definition of wellbeing on which to base evaluation and research’ followed by ‘We need to include a range of wellbeing dimensions into evidence reviews in addition to the ONS4’  and ‘The ONS four dimensions of wellbeing  are important to the sector’ . Lower priority was given to ‘We need to avoid reductionist, over-simplistic approaches in order to capture complex dimensions of wellbeing’  and ‘We need to adopt an inductive approach that assesses wellbeing in each specific context’ .In response to the question ‘What is meant by wellbeing in the context of the culture and sport evidence review?’ (Supplementary Table 6). The top three statements were the following: ‘Evidence is needed to secure and maintain funding for culture and sport’ , ‘Evidence is needed to inform programme planning and best practice in culture and sport’  and ‘Evidence should help to understand the experiences of those who take part in culture and sport’ . Lower ranked were ‘Evidence should identify potentially negative wellbeing impacts of culture and sport’ and ‘Evaluation should be as independent as possible’ .Participants were asked to respond to the question, ‘what is the purpose of evidence in culture and sport?’ (Supplementary Table 7) were the following: ‘Evaluation should identify what doesn’t work as well as what works’ , ‘There is a need for longitudinal evidence in culture and sport’  and ‘We need evidence of how things work not just outcomes measurement’ . Lower ranked were ‘Evaluation should be embedded in practice and should not disrupt service delivery’  and ‘Case studies and stories are needed in order to explain the wellbeing benefits of participation in culture and sport to participants and funders’ .Respondents were asked, ‘what forms of evidence and evaluation are most needed?’ The higher ranking statements (Supplementary Table 8). The highest ranked statements were ‘The review should encompass evidence produced by a diverse group of stakeholders including professionals, practitioners and service delivery organisations’ , ‘We should prioritise helping to develop clear quality criteria that can be used in future evaluation in culture and sport’  and ‘The review should help to develop theories of change based approaches to wellbeing evaluation’ . Lower ranked were as follows: ‘The review needs to recognise that the current evidence base for wellbeing in culture and sport is weak’  and ‘We need to prioritise collating and evaluating published literature in culture and sport’ .Respondents were asked, ‘How should the review respond to the state of the current evidence base in culture and sport?’ (Supplementary Table 9). Highest ranked statements were the following: ‘The review should include rigorous qualitative evidence in culture and sport’ , ‘The review should include examples or approaches that are relevant to small organisations, e.g. case studies’  and ‘The review should include unpublished (grey) literature’ . Lower ranked were ‘The review should prioritise finding robust quantitative research, including RCTs, in culture and sport’  and ‘The review should include anecdotal evidence and testimonials’ .Participants were asked, ‘To what extent should a hierarchy of evidence inform the culture and sport evidence review?’ (Supplementary Table 10). Highest ranked statements were the following: ‘Quality assessment in evidence should include the extent to which evaluation has been informed by the views of those who take part in culture and sport’ , ‘The review should adopt a clear step wise progression from lower quality evidence to gold standard’  and ‘Applying health based hierarchies of evidence will overlook good evidence in culture and sport’ . Lower ranked were ‘Quality assessment in evidence should include the extent to which the evaluation has been informed by stakeholder views’  and ‘The review should only include evidence that meets accepted quality criteria in quantitative and qualitative research’ .Participants were asked, ‘How should the review approach quality assessment?’ (Supplementary Table 11). The results regarding personal dimensions of wellbeing reveal priorities of ‘confidence and self-esteem’, ‘happiness’ and ‘meaning and purpose’ . ‘Reduced anxiety’ and ‘optimism’ were ranked lower . Regarding sector-specific dimensions, priorities are ‘coping and resilience’ , ‘capability and achievement’  and ‘personal identity’ . ‘Sporting or creative skills and expression’ and ‘life skills such as employability’ were ranked lower . Regarding social dimensions of wellbeing, highest ranked were ‘belonging and social identity’ , ‘sociability and new connections’  and ‘bonding and social capital’ . After this came ‘reducing social inequalities’  and ‘reciprocity and giving to others’ .Participants elaborated on dimensions of wellbeing relevant to culture and sport. These were divided into personal dimensions, sector-specific dimensions and social dimensions (Supplementary Table 12). Higher priority are ‘General population: open access community-based culture and sport’ , ‘Specific, targeted populations ’  and ‘People who have been identified as having a specific health condition’ . Lower ranked were ‘People in targeted geographical areas’ and ‘People who are members of cultural and sporting interest groups’ .Participants were asked, ‘What populations are effective wellbeing interventions likely to be working with?’ (Supplementary Table 13). Priorities were ‘Community-based culture, sport and leisure’ , ‘School based arts and culture and sport’  and ‘NHS/social care, statutory & third sector’ . Lower priority were ‘School based sport’ and ‘Culture and sport in commercial organisations’ .Participants were asked, ‘What settings are most likely to be delivering effective wellbeing interventions?’ (Supplementary Table 14). Highest priorities were ‘Group based interventions led by a volunteer or peer’ , ‘Individual activity that can be done alone’  and ‘Group based interventions led by a professional’ . Lower ranked were ‘Non-active sport and culture (e.g. TV)’ and ‘Taking part in “elite” culture and sport’ .Finally, participants were asked, ‘What culture and sport interventions are likely to have the strongest impact on wellbeing?’ /social care, statutory and third sector as well as commercial organisations. Finally, in terms of delivery models, the approaches include group-based interventions led by volunteers, peers and professionals, individual activity that can be done alone, non-active sport and culture and taking part in ‘elite’ culture and sport.The outcomes of a DELPHI process can be influenced by the placement of the threshold of consensus. A high threshold will produce fewer consensus statements, while a low threshold may mask important disagreements. We have not sought to generate definitive answers using the DELPHI in this study or impose a rigid threshold of consensus. To a degree, all the statements are important to stakeholders as they are drawn from an extensive consultation process in which a range of views were expressed. Instead, we have sought to illustrate broad and sometimes nuanced priorities that will inform the scope and parameters of a 3-year evidence review programme examining the impact of culture and sport on wellbeing.The interpretation of the quantitative results is informed by the qualitative analysis which revealed four broad themes including definitions and dimensions of wellbeing, the purpose of evaluation, hierarchies of evidence, and culture, health and wellbeing populations, settings and interventions.With regard to definitions and dimensions of wellbeing, the results show that there is broad support for the adoption of a common definition of wellbeing on which to base evaluation and research. The value of the ONS4 is recognised, but other dimensions are relevant to the culture and sport sectors, such as personal dimensions of wellbeing; for example, confidence and self-esteem, happiness, and meaning and purpose, sector-specific dimensions such as coping and resilience, capability and achievement and personal identity, and social dimensions, including belonging and social identity, sociability and new connections, and bonding and social capital. These dimensions may not represent a complete fit with the ONS4, but they reflect practice in the culture and sport sectors and therefore need to be taken into account in evidence review.With regard to the purpose of evaluation, there is strong acceptance of the principle that evidence is needed to secure and maintain funding for culture and sport and that its evaluation is needed to inform programme planning and best practice in culture and sport. There is also general support for the principle that evaluation should identify what does not work as well as what works.25 Sector particularities may have also influenced participants’ statements in relation to quality assessment, where there is an emphasis on consultation and participant engagement as well as recognition of the need for progression from lower quality evidence to gold standard. Thereseems to be some disagreement about what constitutes low quality evidence as well as scepticism towards the application of health-based hierarchies and fear that this will overlook evidence that is valued by stakeholders in culture and sport.It is with regard to hierarchies of evidence that some interesting results have emerged. Discussions about hierarchies of evidence in workshops, and the resulting statements and priorities, reflect the particularities of the sector and may appear to be at odds with conventional wisdom in public health. During workshops, participants devoted considerable energy to discussing the current state of evidence in culture and sport. They gave strong support to the idea of a collaborative programme of evidence generation that engages diverse groups of stakeholders. They also recognised the need for the development of clear quality criteria that are relevant to assessment of evidence in culture and sport. However, participants ranked rigorous qualitative evidence, case studies and grey literature more highly than they did experimental methods and randomised controlled trials (RCTs). This reflects the state of the evidence in culture and sport, where most projects are evaluated using mixed methods and many evaluations are unpublished reports. Methodologies such as RCTs are only beginning to be applied and have often been viewed as challenging and inappropriate in complex and sensitive contexts.Regarding populations, settings and interventions, a number of priorities emerged. Priority populations are community-based culture and sport populations as well as specific populations targeted because of demographic and socioeconomic characteristics or existing health conditions. Priority-settings are communities, schools and health care rather than commercial organisations. Priority interventions are group based and individual, but not necessarily passive forms of engagement or taking part in ‘elite’ culture and sport. The results also indicate the populations, settings and intervention models which might reveal best evidence.Regardless of whether consensus is reached, the DELPHI process records all views and therefore allows minority viewpoints to be clearly identified. These may be just as important to policy makers, stakeholders and researchers as majority views. Our study shows that there is broad agreement across the three groups of stakeholders across these key principles and parameters. There were some differences in the priorities of three groupings included in the study: SDOs seem to favour inductive, qualitative approaches to evaluation that do not disrupt service delivery, and are also more ready than others to recognise the value of anecdotal evidence and testimonials. Apart from this, there were no significant differences between the groups. These sub-group differences at DELPHI 1 did not prevent the group from reaching an overall consensus at DELPHI 2.There are few areas of strong disagreement in these DELPHI results; however, the nuanced views, particularly in relation to the conduct of evaluation as well as hierarchies of evidence and quality assessment, reveal that SDOs may have slightly different agendas when it comes to evaluation and evidence than other groups. This may reflect the relatively early stage of development of the evidence base in this field and indicates a need for ongoing consultation to ensure that nobody is left behind when it comes to development and application of evaluation methodologies.A strength of the study is the response rate of 70.02% from a purposeful sample of participants selected on the basis of their organisational role and involvement across a range of CSW contexts. However, this was not a comprehensive survey of a representative sample of organisations and stakeholders involved in CSW. Time and resources prohibited this, but the Delphi technique has proved useful for including a focused sample of participants from diverse backgrounds in deliberation and consensus development. It has the advantage of being democratic in that it allows all participants an equal chance of influencing the results.It could be said that the presentation of statements in rank order at DELPHI 2 was leading. However, DELPHI 2 was the culmination of 6 months of collaborative development in face-to-face workshops, interviews, observations and online communication with stakeholders, many of whom the research team have known and been collaborating with for several years. We would not have expected substantial reworking of priorities at this stage. In order to distinguish higher and lower priorities, we have applied a basic threshold of consensus, identifying the top three statements as higher priority. However, we recognise that this is somewhat arbitrary and that all the statements were of value to stakeholders.The DELPHI process, in combination with extensive face-to-face deliberation, has enabled stakeholders to engage in complex discussion and express nuanced priorities while also allowing a diverse group to come to an overall consensus and agree priorities. We have demonstrated the importance of consultation to ensure that professionals, practitioners and service users can contribute to evaluation and evidence review. The degree of consensus reached suggests that the resulting statement can inform the development of a broad strategy to assess the effect of policies and initiatives on wellbeing with which all the groups can agree.The exercise revealed strong support across stakeholders for a common definition of wellbeing on which to base research and evidence, provided that this can encompass dimensions that are important in culture and sport, such as confidence and self-esteem, as well as social dimensions of wellbeing. The exercise also revealed strong support for evidence generation, to show wellbeing outcomes of culture and sport, to help to secure funding and therefore contribute to sustainability, to inform programme planning, and to understand the experiences of those who take part in culture and sport. Furthermore, the data showed clear support across stakeholder groups for the use of quality criteria to assess evaluation. Stakeholders’ aspirations to go beyond the ONS4 and focus on the importance of context may be revealing a preference for experiential measures of SWB in culture and sport. Our project will explore how people feel while they are engaged in experiences of culture and sport as well as identifying how culture and sport impact on SWB overall.While there is support for quality standards in evidence review, the ambivalence that was revealed surrounding hierarchies of evidence, together with support for forms of evidence typically seen as lower quality, is perhaps a reflection of the relatively early stage of development of evidence generation for wellbeing outcomes in culture and sport.32 and income and/or socioeconomic status. Hence, while we have not sought to generate definitive answers from this DELPHI, the process has helped to clarify the aims and scope of the CSW evidence review programme. This is of value because the systematic study of wellbeing in culture and sport has been prioritised by programmes such as the What Works Programme in order to strengthen policy making, commissioning and funding of projects and programmes.The process has highlighted the dimensions of wellbeing that are important to the sector, as well as the populations, settings and types of intervention that it is important to focus on. Together with the analysis of the qualitative data, it has also helped to establish the topics and the parameters for the evidence reviews. Over the next 3 years, the research team will undertake reviews and secondary analysis of wellbeing outcomes of music and singing of adults (18+ years); sport and dance for adolescents and young people (14–25 years); visual arts for adults (18+ years) with a mental health condition; and sport and recreation across the family lifecourse. A cross-evidence review programme will examine wellbeing outcomes of co-produced culture and sport, community, and work and learning interventions. Each review will examine the processes by which wellbeing outcomes are achieved and will include analysis of all protected characteristics


In most developing countries including Nigeria, one of the most challenging issues associated with evidence-to-policy link is the capacity constraints of policymakers to access, synthesize, adapt and utilize available research evidence. The purpose of this review is to assess the efforts and various initiatives that have been undertaken to deliberately engage policymakers and other stakeholders in the health sector in Nigeria for the promotion of evidence informed policymaking. A MEDLINE Entrez Pubmed search was performed and studies that investigated policy making process, evidence to policy link, research to policy mechanism, and researchers/policymakers interaction in Nigeria in relation to health policy were sought. Of the 132 publications found, 14(10.6%) fulfilled the study inclusion criteria and were selected and included in the review. Of the fourteen scientific publications identified, 11 of the studies targeted both researchers and policymakers and the principal tool of intervention was training workshops which focused on various aspects of evidence informed policymaking. All the studies indicated positive outcomes and impacts in relation to quantifiable improvement in policymakers' knowledge and competence in evidence to policy process. Capacity strengthening engagement mechanism is needed for both researchers to generate better evidence and for policymakers and health-care professionals to better use available evidence. The process of utilizing evidence from research to make health policy which is known as evidence-informed policy-making is characterized by the systematic and transparent access to, and appraisal of, evidence as an input into policy-making , 2. In eAccording to Lavis and colleagues , there iA MEDLINE Entrez Pubmed search was performed in August 2015 and studies published in English that investigated policy making process, evidence to policy link, research to policy mechanism, and researchers/policymakers interaction in Nigeria in relation to health policy were sought. The keywords used for the search included: Nigeria, evidence, health policy; these yielded 132 entries. These 132 publications were subjected to the study inclusion criteria which included the following: (i) must have been conducted in Nigeria; (ii) must be a primary scientific investigation and not a review article; (iii) must target policymakers and researchers or only policymakers; (iv) must address health issue of policy relevance to Nigeria; (v) must produce evidence that is policy relevant; (vi) may of may not have an intervention component. Of the 132 publications found, a total of 14(10.6%) fulfilled these study inclusion criteria and were selected and included in this review –34 Tabl, Table 4Knowledge transfer/exchange process involving Nigeria policymakers is recent: All the studies which fulfilled the study inclusion criteria were published within the eight years (2008-2015). The outcome of this review clearly suggests that research on evidence-informed policymaking and knowledge transfer/exchange processes involving policymakers is new or still at infancy stage in Nigeria. Out of the 14 scientific publications that fulfilled the study inclusion criteria, 9(64.3%) of them were published between 2012 and 2015 (and 2015 , Table 3Of the fourteen scientific publications identified, 11 of the studies targeted both researchers and policymakers –31,34, aQuantifiable improvement in policymakers' knowledge and competence in evidence to policy process: Most of the studies reporting policymakers' capacity enhancement process for evidence informed policymaking were mostly recent ranging from 2012-2015 (012-2015 , Table 3012-2015 . AlthougIn Nigeria, the grossly deficient capacity among policy-makers to use of evidence for policy-making remains a major challenge associated with evidence-to-policy link ,39. The There is currently a shift away from opinion-based policies to a more rigorous approach that uses high-quality research evidence to inform health policy-making;One of the most challenging issues with evidence-to-policy link is the capacity constraints of policymakers to access, synthesize, adapt and utilize available research evidence;Policy-makers need the capacity to be able to access and assess research findings and incorporate them in decision making.Research on evidence-informed policymaking and knowledge transfer/exchange processes involving policymakers is new or still at infancy stage in Nigeria;Bringing researchers and policymakers together to participate in interventional training workshops can enhance their capacity for evidence informed policymaking;Capacity strengthening is needed for both researchers to generate better evidence and for policymakers and health-care professionals to better use available evidence.The authors declare no competing interests.


A major problem is that we tend to take small p-values at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. Also significance (p ≤ 0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third of the cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. However, applying significance thresholds makes cumulative knowledge unreliable. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’. Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. We review how confusion about interpretation of larger p-values can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment.The widespread use of ‘statistical significance’ as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process . We review why degrading  Related to that, (6) publication bias occurs when papers are more likely to be published if they report significant results  as a license for making a claim of a scientific finding (or implied truth) leads to considerable distortion of the scientific process”  published a statement on  experts . While tprocess” . And a sractice” .p-values not as part of a binary decision rule but as a continuous measure of evidence against the null hypothesis has had many advocates, among them the late Ronald Fisher ’. We continue by summarizing opposing recommendations by Ronald Fisher versus Jerzy Neyman and Egon Pearson that led to the unclear status of nonsignificant results, contributing to publication bias. Finally, we outline how to use graded evidence and discuss potential arguments against removing significance thresholds. We conclude that we side with a neoFisherian paradigm of treatingp-values as graded evidence against the null hypothesis. We think that little would need to change, but much could be gained by respectfully discharging significance, and by cautiously interpreting p-values as continuous measures of evidence.We therefore do not focus on general misconceptions about BOX 1Significance thresholds and two sorts of reproducibilityInferential reproducibility might be the most important dimension of reproducibility and “refers to the drawing of qualitatively similar conclusions” from an independent replication of a study  as evidence in favor of a null effect, or as no evidence at all. Researchers will increase inferential reproducibility if they refrain from turning their conclusion upside down once a significance threshold is crossed, but instead take the p-value as providing graded evidence against the null hypothesis. a study . Some peResults reproducibility, or replicability, “refers to obtaining the same results from the conduct of an independent study”  had p < 0.05 in the replication , and of >0.44 with 10% probability. For p = 0.5 in a first study, p-values will have sizes between 0.003 and 0.997 .Unfortunately, ion of p . Cummingp-values’ . Second, most studies end up with a power much smaller than 80%. For example, average power to detect medium sized effects was about 40–47% in 10 journals on animal behavior , and medIf we repeat the exercise by p-value. It is the fault of us defining replication success as the event of crossing a significance threshold. As This is of course not the fault of the In fact, it would be highly dubious if replication success in terms of statistical significance were larger than just described. This would indicate that researchers suppress nonsignificant replications and selectively report significant outcomes, and that there is publication bias against nonsignificant studies . HoweverHowever, selective reporting was encouraged since the early days of significance testing. As No wonder that a bias towards publishing significant results was observed for a long time and in many research areas . Today, The problem with selective reporting becomes even worse because significant results are not a random sample from all possible results—significant results are biased results. If a number of studies are done on a population with a fixed effect size, studies that due to sampling variation find a larger effect are more likely to be significant than those that happen to find smaller effects . Using sThe inflation of effect sizes in significant results declines as statistical power increases (for example because sample sizes are large), and inflation becomes negligible as power approaches 100% . One wayn > 100. However, as In the ecological and neurological studies summarized in Selective reporting of inflated significant effects while ignoring smaller and nonsignificant effects will lead to wrong conclusions in meta-analyses synthesizing effect sizes from a larger number of studies . This isp-values, but then select what to interpret and to discuss based on significance thresholds, the effects from which the authors draw their conclusions will be biased upwards.Of course, selective reporting of significant results leads to inflated effects not only in meta-analyses but in every single study. Even in cases in which authors report all conducted tests regardless of their The problem arises not only by consciously discarding nonsignificant findings. Also largely automated selection procedures may produce inflated effects, for example if genome-wide association studies select findings based on significance thresholds . In statp-value can be seen as a measure of surprise ”. What if this test happens to be nonsignificant? In 38%–63% of articles sampled from five journals of psychology, neuropsychology and conservation biology, nonsignificant results were interpreted as ‘there is no effect’, which means that a null hypothesis was accepted or ‘proven’ (‘proven’ .p > 0.05).’ And this is not far from reality. Similar cases abound in the published literature, such as “lamb kill was not correlated to trapper hours ”  ”. The conclusion was that “the results demonstrated no difference between rofecoxib and naproxen”. Later, the unjustified proof of the null caused more suffering, and the manufacturer Merck took Vioxx off the market and faced more than 4,200 lawsuits by August 20, 2005  indicates the strength of the evidence against the hypothesis” (p. 80) and that “tests of significance are used as an aid to judgment, and should not be confused with automatic acceptance tests, or ‘decision functions’ ” (p. 128).Throughout his life, Fisher used the nificant , p. 53. nificant , p. 42. nificant , he thendeductive inference, from the population to the sample; A main point of controversy was ‘inductive inference’ that was central to Fisher’s thinking , p. 90. In contrast, Neyman and Pearson thought that inductive inference is not possible in statistical analyses on single studies: “As far as a particular hypothesis is concerned, no test based upon a theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis” , p. 291.Applying Neyman–Pearson decision rules means accepting the null hypothesis while rejecting the alternative hypothesis, or vice versa. And Neyman and Pearson did indeed use the word ‘accept’ e.g., . Today, Thus, accepting null hypotheses was encouraged by half of the founding schools of modern statistics. No wonder that “in approximately half of all cases, authors interpret their nonsignificant results as evidence that the null hypothesis is true” , p. 43.p-values as graded measures of the strength of evidence against the null hypothesis ’. Instead, we should listen to the ASA-statisticians who say that if the “p-value is less than 1, some association must be present in the data, and one must look at the point estimate to determine the effect size most compatible with the data under the assumed model” ” ” . As we sly large , 1D; andperiment , p. 16.For our next scientific enterprise using frequentist statistics, we suggest that we(a) do our study and our analysis as planned(b) report our point estimate, interpret our effect size(c) report and interpret an interval estimate, e.g., a 95% confidence intervalp-value(d) report the exact p-value is relatively large(e) do not use the word ‘significant’ and do not deny our observed effect if the (f) discuss how strong we judge the evidence, and how practically important the effect is.Do we need a scale for interpreting the strength of evidence against the null hypothesis? Graded evidence means there are no thresholds to switch from ‘strong’ to ‘moderate’ to ‘weak’ evidence . It meanp-value, we must be aware of the large variability of p-values, keep in mind that our evidence against the null hypothesis might not be as strong as it seems, and acknowledge that our point estimate is probably biased upwards. If we have obtained a large p-value, we must be even more aware that many hypotheses are compatible with our data, including the null hypothesis. Looking at the values covered by the confidence interval will help identifying those competing hypotheses. Very likely there are hypotheses compatible with the data that would cause even greater concern than a zero effect, e.g., if the effect would be in the opposite direction  of a conclusion, the plausibility of alternative hypotheses, or the robustness of the test.When discussing our results, we should “bring many contextual factors into play to derive scientific inferences, including the design of a study, the quality of the measurements, the external evidence for the phenomenon under study, and the validity of assumptions that underlie the data analysis” . These include ‘flirting with conventional levels of significance (p > 0.1)’, ‘on the very fringes of significance (p = 0.099)’ and ‘not absolutely significant but very probably so (p > 0.05)’.”People seem prepared to interpret response . Nuzzo  in favor of a null effect, or as no evidence at all. This anarchical state of affairs undermines ‘inferential reproducibility’, which might be the most important dimension of reproducibility and “refers to the drawing of qualitatively similar conclusions from either an independent replication of a study or a reanalysis of the original study” . However, knowing that the data are unlikely under the null hypothesis is of little use unless we consider whether or not they are also unlikely under the alternative hypothesis  effect, this will create a downwards bias (Whatever method of statistical inference we use, biased effect sizes will be a problem with criteria that are used to select results for publication or interpretation. If effect sizes are reported because they are large, this will of course create an upwards bias, similarly to selecting rds bias . If resurds bias .Inflated effects will occur when a discovery is claimed because a Bayes factor is better than a given value or a false discovery rate is below a given value . Selectip-values, simply because of the dominant role of null hypothesis significance testing. If we learn to judge the strength of evidence based not on the event of passing a threshold, but on graded summaries of data like the p-value, we will become more aware of uncertainty. And statistical methods are not simply applied to a discipline but change the discipline itself (However, the greatest source of bias probably comes from selectively reporting small e itself . A greate itself  could she itself .p-values, to conclusions that are based on inflated effects, and to publication bias against larger p-values. But larger p-values are to be expected also if there is a true effect, and they must be published because otherwise smaller p-values are uninterpretable. Thus, smaller p-values need to lose reputation, and larger p-values need to gain reputation. This is best accomplished by removing fixed significance thresholds, by cautiously interpreting p-values as graded evidence against the null hypothesis, and by putting more emphasis on interpreting effect sizes and interval estimates, using non-automated informed judgment. We give the last word to Part of an apparent crisis of unreplicable research is caused by the way we use statistics for analyzing, interpreting and communicating our data. Applying significance thresholds leads to overconfidence in small but highly variable


The subsequent SNP-based replication analysis revealed a strong association of onset latency with the SNP rs901561 on the neuregulin gene (NRG1) in an additive model , indicating that subjects with the AA and AG genotypes had slower mean latency than subjects with GG genotype. In conclusion, startle latency, a highly heritable measure that is slowed in schizophrenia, may be a useful biological probe for genetic contributions to psychotic disorders. Our analyses in two independent populations point to a significant prediction of startle latency by genetic variation in NRG1.Latency of the acoustic startle response is the time required from the presentation of startling auditory stimulus until the startle response is elicited and provides an index of neural processing speed. Latency is prolonged in subjects with schizophrenia compared to controls in some but not all studies and is 68–90% heritable in baseline startle trials. In order to determine the genetic association with latency as a potential inroad into genetically based vulnerability to psychosis, we conducted a gene-based study of latency followed by an independent replication study of significant gene findings with a single-nucleotide polymorphism (SNP)-based analysis of schizophrenia and control subjects. 313 subjects from an urban population of low socioeconomic status with mixed psychiatric diagnoses were included in the gene-based study. Startle testing was conducted using a Biopac M150 system according to our published methods. Genotyping was performed with the Omni-Quad 1M or the Omni Express BeadChip. The replication study was conducted on 154 schizophrenia subjects and 123 psychiatric controls. Genetic analyses were conducted with Illumina Human Omni1-Quad and OmniExpress BeadChips. Twenty-nine SNPs were selected from four genes that were significant in the gene-based analysis and also associated with startle and/or schizophrenia in the literature. Linear regressions on latency were conducted, controlling for age, race, and diagnosis as a dichotomous variable. In the gene-based study, 2,870 genes demonstrated the evidence of association after correction for multiple comparisons . Pathway analysis of these genes revealed enrichment for relevant biological processes including neural transmission ( Schizophrenia is a psychiatric disorder associated with significant morbidity, mortality, and global burden of disease with a prevalence estimated at 1% of the general population . AlthougOne emerging strategy to approach the genetics of schizophrenia is the use of endophenotypes, measurable traits discoverable by a biologic test. To qualify as an endophenotype, the trait must be associated with the disease and be present whether or not the disease is active. The trait must also be heritable, co-segregate with the disease within families, and be found in non-affected family members at a higher rate than the general population . The advOne endophenotype that is well represented in schizophrenia research is prepulse inhibition of acoustic startle (PPI). The startle reflex, easily measured in animals and humans, is mediated by a simple subcortical circuit . PPI is Another startle measure that has shown promise in recent findings is latency of the acoustic startle response. Startle latency is the time that it takes for the startling stimulus to travel through a three-synapse subcortical circuit that mediates the startle response, thus providing a putative index of neuronal processing speed. The finding of slower acoustic startle latency in schizophrenia than in controls has been reported in a number of studies –15, althImportantly for the feasibility of genetic studies, the slowing of latency in subjects with schizophrenia can be detected in those who are treated with antipsychotic medications, as several studies report prolonged latency in schizophrenia cohorts all or most of whom were on antipsychotics , 22, 28.RELN gene  , or the Subjects were tested in an acoustic startle session designed to assess startle magnitude and fear potentiation of startle. Subjects were seated in a sound attenuating booth during the test session. The eye blink component of the acoustic startle response was measured by electromyography (EMG) of the right orbicularis oculi muscle recorded with the Biopac MP150 system for Windows  using methods previously published by our group . The EMGThe acoustic startle probes were 40 ms 108 dB [A] SPL white noise bursts with near instantaneous rise time presented binaurally through Maico headphones (model TDH-39-P). The session began with four trials of noise alone startle stimuli to assess baseline startle magnitude, from which data on acoustic startle latency were extracted. Startle data obtained later in the session to assess fear potentiation were not used for the current analyses.Startle magnitude for each trial was the maximum EMG signal occurring between 20 and 120 ms after the startling stimuli. Latency was the time of maximal EMG magnitude after the presentation of the startling stimuli (peak latency). Latency data were considered valid if the startle recorded did not begin before 20 ms after the startle stimulus and the startle magnitude was at least 15 µV.The subjects provided a saliva or blood sample. DNA was extracted from saliva in Oragene collection vials  using the DNAdvance kit , while DNA from blood was extracted using either the E.Z.N.A. Mag-Bind Blood DNA Kit  or the ArchivePure DNA Blood Kit .Genotyping was performed using the Omni-Quad 1M . Genotypes were called using the Illumina’s GenomeStudio software. We used PLINK to perform quality-control (QC) analyses on the genetic data. In brief, initial QC involved removing samples with very low call rates and those outside acceptable levels of heterozygosity (−0.25 < Fhet > 0.25); the remaining samples were recalled in GenomeStudio. We then removed SNPs with call rates less than or equal to 98%, a frequency of less than 0.01, and individuals with greater than 2% missing data. We further identified and removed related individuals by using PLINK to estimate the proportion of identity by descent (IBD) for each pair of individuals. Among pairs of individuals with an IBD proportion >0.12 (indicating cousins or a closer relation), we removed the individual in each pair with the higher rate of missing genotype data. Using data (autosomes only) pruned in PLINK, we performed principal-component analysis to infer axes of ancestry. Subjects that fell within 3 SDs of the medians of the first and second PCs in our sample were considered outliers and removed.p-value was calculated by using the permuted p-value of the best individual SNP association within the given gene. minSNP computes single SNP F-statistics for each SNP within a gene and uses the best F-statistic within that gene as its test statistic. The p-value was calculated using 1,000,000 permutations to correct for gene size , and the false discovery rate (FDR) was controlled at 5% to account for multiple testing. Pathway analysis was performed with DAVID , including sex, and the top 10 PCs of genome-wide data as covariates , 48. Genene size . The preth DAVID , and theWe next focused on these genes in a SNP-based replication study on the Atlanta Veterans Affairs Medical Center (VA) cohort, which consisted of 324 subjects, of whom 185 had schizophrenia and 139 were psychiatric controls. This study was carried out in accordance with the recommendations of the Emory University Institutional Review Board with written informed consent from all subjects. All subjects gave their informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Emory University Institutional Review Board and the Atlanta Veterans Affairs Medical Center Human Subjects Committee.We examined regressions on both onset latency (the time from presentation of the startling stimulus until the onset of the startle blink) and peak latency (the time from presentation of the startling stimulus until the time of peak magnitude of the startle blink). The subjects in this independent replication sample were collected through a VA funded Merit Review Project . The savia EMG recording of the right orbicularis oculi muscle. Two Ag/Ag-Cl electrodes were positioned 1 cm below and right pupil and 1 cm below the right lateral canthus, with a ground electrode behind the right ear. The resistances for all subjects were less than 6 kΩ as measured with a Checktrode impedance meter . EMG activity was amplified and digitized using a computerized EMG startle response monitoring system . The EMG signal was filtered with low- and high-frequency cutoffs at 30 and 1,000 Hz, respectively. The system was set to record 250 1-ms readings starting at the onset of the startle  stimulus. Digital signals were full-wave rectified and smoothed by an averaging routine that calculates a rolling average of 10 data points by means of SR-LAB analysis software.Hearing acuity was assessed by means of audiometer testing with a Grason-Stadler audiometer Model GS1710). To be included, subjects had to detect tones in each ear at a threshold of 40 dB [A] at 0.25, 0.50, 1, 2, 4, and 8 kHz. Subjects completed a startle paradigm designed to assess magnitude, latency, and prepulse inhibition of startle using methods developed by Braff and colleagues 10. To be, 55. BriThe startle session began with a 60-s acclimation period consisting of 70 dB white noise that continued as background noise throughout the session. The pulse-alone stimuli were 116 dB, 40-ms bursts of white noise. The first block of the session consisted of six pulse alone stimuli. From this block, data on startle magnitude and latency were extracted for the current analyses. The next portion of the session, designed to assess prepulse inhibition of acoustic startle, was not used in the analyses presented herein.DISC1), v-erb-b2 avian erythroblastic leukemia viral oncogene homolog (ERBB4), nitric oxide synthase 1  adapter protein (NOS1AP), and neuregulin (NRG1).To select genes for the focused replication analysis, we filtered the 2,870 significant genes from the gene-based analysis above based on whether they are expressed in the pons, a region of the brain that mediates startle response, and whether they have a prior association in the literature with startle (either latency or magnitude) or with schizophrenia or schizophrenia-related endophenotypes. This resulted in the following four genes that met these criteria: disrupted in schizophrenia 1 (p ≤ 0.01), resulting in 29 SNPs that were analyzed for their relation with latency. Six SNPs had minor allele frequencies below 5%, and MAF was considered in interpretation of the results.We proceeded with a SNP-based analysis of these four genes. Genotyping was performed in 384-well format using iPlex Gold kits and the Sequenom MassARRAY system. Amplification and extension primers were designed by SpectroDESIGNER software. The MassARRAY™ Typer software was used to assign the genotype calls. Each 384-well genotyping plate contained positive and negative controls. No samples were excluded based on missing data rates >5%. In total, 38 SNPs were genotyped of which 9 were eliminated because of call rates <95% and deviations from Hardy–Weinberg proportions  and peak latency (the time from presentation of the startling stimulus until the time of peak magnitude of the startle blink). For each of these potential models, we examined association using linear regressions that incorporated covariates of age, race, and diagnosis (SCZ vs. CON). All t-tests, as is noted in Table n = 48). Latency did not differ between those who were positive vs. negative . Additionally, those subjects admitting to current drug use on structured interview or having positive urine toxicology at the time of testing did not differ significantly in latency from those without current drug use or positive urines .Demographic information on the Grady sample is listed in Table p = 0.0029), synaptic transmission (p = 0.0032), and neuron development (p = 0.024) among others , in the SCZ subjects , and in the CON subjects .Table DISC1, ERBB4, NOS1AP, and NRG1) that predicted slowing of latency (p < 0.05) in either an additive or dominant models. After Bonferroni correction of p = 0.0017 for multiple comparisons, only the finding for the NRG1 gene was significantly associated with startle latency. Our analysis revealed a strong association of onset latency with SNP rs901561 on NRG1, as can be seen in additive and dominant models shown in Figure NRG1) had slower mean latency than subjects with the GG genotype .Table S2 in Supplementary Material provides details for the 29 SNPs that were tested on these four genes. Table DISC1 predicted slowing of latency, although the significance level did not survive our correction for multiple comparisons. Subjects with the AA genotype at rs2082552 had slower predicted latency than subjects with GG, with AG subjects intermediate . The DISC1 SNP, rs3738402, significantly predicted peak latency in a dominant model , with subjects with the CT or TT genotypes demonstrating slower latency than those with the CC genotype. For this SNP, there was only one TT homozygote, so an additive model could not be computed.Two SNPs in ERBB4 also predicted latency, although the significance level did not survive our correction for multiple comparisons. Subjects with the GG genotype at rs10932374 had slower predicted onset latency than AA subjects, with AG subjects having intermediate latency . Similarly, rs12467225 also predicted onset latency in an additive model , with subjects that had the CC genotype exhibiting slower latency than TT subjects and CT subjects exhibiting intermediate latency.Two SNPs on NRG1 gene, with two loci on the DISC1 gene, and with two loci on the ERBB4 gene, although these latter four loci did not survive Bonferroni correction.In this study, we used a gene-based analysis to examine the association of startle latency with multiple genes in a hypothesis neutral manner. This revealed associations with 2,870 genes involved in neural transmission, synaptic transmission, and neuronal development. We then conducted a SNP-based follow-up study on an independent sample targeting genes expressed in the pons, a region that mediates startle, and known from prior work to be associated with schizophrenia, schizophrenia endophenotypes, or acoustic startle. This SNP-based analysis revealed a significant association of startle latency with a SNP on the The gene-based analysis revealed a significant association of >2,000 genes with startle latency. The preponderance of these genes is known to be involved in processes that could be expected to be relevant to startle latency, such as transmission of nerve impulses, synaptic transmission, and neuronal development. We have hypothesized that slowing of startle latency may reflect a more generalized slowing of neural transmission . AlthougNRG1 gene in our gene-based analysis, and in our SNP-based replication, rs901561 was significant in both additive and dominant models. This gene has been reported as being associated with SCZ in large GWAS studies . In the Finger Tapping Task, a test of motor speed, we found that a higher score  corresponded with faster onset latency in both dominant  and non-dominant  hands.Our analysis revealed a strong association of latency with the  studies  and conf studies , 59, but studies . The genophrenia , 56, 62.ophrenia , 63, NRGelations . HoweverERBB4 SNPs with latency, rs10932374 and rs12467225, although the significance for these SNPs did not survive correction for multiple comparisons. ERBB4 has been extensively investigated as a vulnerability gene for schizophrenia  acoustic startle latency as indirect evidence of slowing of neural processing supports the underlying hypothesis that genetically determined alterations of the neuregulin-ERBB4 and glutamate signaling pathways play a role in the pathophysiology of schizophrenia. These results need replication in larger datasets, and future studies should be conducted on other genes that were significant in our discovery sample. However, our latency findings may provide a foundation from which we can characterize a distinct subgroup of schizophrenia patients and discover the neurobiology underlying the association of slowed latency with schizophrenia risk.In summary, the results of this study need replication in a larger data set, but the preliminary results support an association of startle latency with the multiple genes involved in neuronal signaling and development, including the This study was carried out in accordance with the recommendations of the Emory University Institutional Review Board with written informed consent from all subjects. All subjects gave their informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Emory University Institutional Review Board.AS, TJ, KR, and ED designed the experiments. AS, TJ, SN, BC, BB, KR, and ED performed the experiments. AS, TJ, VK, AL, LG, SL, NM, BB, KR, and ED analyzed the data. AS, LG, SL, and ED wrote the manuscript. All authors edited and approved the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Lutzomyia neivai was usually treated as a synonym of the morphologically similar Lutzomyia intermedia, which has long been considered a vector of Leishmania braziliensis, the causative agent of much cutaneous leishmaniasis in South America. This report investigates the likely range changes of both sand fly species in response to a stabilisation climate change scenario (RCP4.5) and a high greenhouse gas emissions one (RCP8.5).Before 1996 the phlebotomine sand fly L. intermedia (sensu lato) produced using the GARP algorithm in 2003, the current investigation modelled the two species separately, making use of verified presence records and additional records after 2001. Also, the new ensemble approach employed ecological niche modelling algorithms  that have been widely adopted since 2003 and perform better than GARP, as well as using a more recent climate change model (HadGEM2) considered to have better performance at higher resolution than the earlier one (HadCM2).Ecological niche modelling was used to identify areas of South America with climates currently suitable for each species, and then the future distributions of these climates were predicted based on climate change scenarios. Compared with the previous ecological niche model of Lutzomyia intermedia was shown to be the more tropical of the two species, with its climatic niche defined by higher annual mean temperatures and lower temperature seasonality, in contrast to the more subtropical L. neivai. These different latitudinal ranges explain the two species' predicted responses to climate change by 2050, with L. intermedia mostly contracting its range (except perhaps in northeast Brazil) and L. neivai mostly shifting its range southwards in Brazil and Argentina. This contradicts the findings of the 2003 report, which predicted more range expansion. The different findings can be explained by the improved data sets and modelling methods.Our findings indicate that climate change will not always lead to range expansion of disease vectors such as sand flies. Ecological niche models should be species specific, carefully selected and combined in an ensemble approach.The online version of this article (doi:10.1186/s13071-017-2093-9) contains supplementary material, which is available to authorized users. Leishmania species  that cause human leishmaniasis ).Binary outputs were restricted to areas historically accessible to both species via dispersal . ThThe performance of each model was evaluated by the true skill statistic (TSS), a derivative of Cohen’s kappa. While kappa alone is a popular measure of model performance, recent studies suggest that it is overly dependent on presence data, and equal proportions of presences and absences only contribute to the kappa score when sensitivity and specificity are uniform, which biases estimates of predictive accuracy , 53. TSSL. intermedia and L. neivai. Outputs with the highest TSS scores from each algorithm were overlaid and areas of agreement extracted per the majority ensemble rule [L. intermedia and L. neivai were assessed from these maps.Only model outputs with TSS scores above or equal to 0.7 were retained for mapping the climatic suitability areas of ble rule , to prodThe approximate area of climatic suitability was extracted from the final binary ensemble maps of each species under each scenario , to give an objective numerical overview of potential climatic habitat contraction and expansion. Binary model predictions from each scenario were overlapped in ArcGIS v. 10.00  to map eAs explained in the Methods, a subset of six less correlated variables was selected for ecological niche modelling, to reduce collinearity in the initial dataset of 19 climatic variables. These six variables include mean annual temperature and precipitation as well as measures of diurnal (temperature) and seasonal (temperature and precipitation) climatic changes ; while most records in the top right corner of the scatter plot  are for L. intermedia (7 out of 7).Annual mean temperature was higher for ant Fig. , Table 1L. intermedia (17 out of 23); while most records in the bottom right quadrant of the scatter plot  are for L. neivai (83 out of 91)  are for L. neivai than for L. intermedia. Most records in the bottom left quadrant of the scatter plot  are for L. intermedia (80 out of 112); while most records in the top right quadrant of the scatter plot  are for L. neivai (43 out of 43)   for all five ecological niche modelling algorithms, with only BIOCLIM and GLM occasionally having mean TSS scores < 0.8 Fig. . Model oL. neivai, the model agreement for current climatic conditions was higher in south-east/south Brazil, east Paraguay, northeast/north-west Argentina and a small area in southern Bolivia  predicted similar geographical distributions for the combinations of climatic variables identified by the ecological niche modelling algorithms as being suitable for dia Fig.  and L. nL. intermedia in the Andean mountains, southern Colombia, southern Venezuela and southern Amazonian Brazil  and, in the current analysis, it provided similar predictions to our ensemble models for both climate change scenarios when a selection of pre-2002 presence records was combined for both species , but not for each of the two species owing to the paucity of confirmed presence records at the time. Based on a GARP model, Peterson & Shaw [L. intermedia (s.l.) extended from Rio Grande do Norte and Ceará states in northeast Brazil, south along Brazil’s eastern coast into Uruguay and Argentina, and west into Paraguay, Bolivia and Peru, with small disjunct pockets in Colombia and Guyana. Using two climate change scenarios, HHGSDX50 (conservative) and HHGGAX50 (extreme), they predicted L. intermedia (s.l.) will experience slight climatic improvements in its current habitats, and spread further along the eastern slopes of the Andes [Peterson & Shaw  previousn & Shaw  found thhe Andes .There are several potential explanations for the differences in our findings and those of Peterson & Shaw . First, L. intermedia and L. neivai.In a comparative study of the performance of five modelling techniques, Elith & Graham  found GAAdditionally, the resolution of current and past climate data utilised by Peterson & Shaw  was coarL. intermedia, as they appear to be for L. neivai [Again considering resolution, Peterson & Shaw  utilised. neivai , then thL. intermedia and L. neivai as incriminated vectors of CL, with natural infections of Leishmania detected in São Paulo (L. intermedia (s.l.) [L. intermedia, [L. intermedia [L. intermedia (s.l.) [L. neivai [L. neivai [L. neivai [There is sufficient evidence to treat both a (s.l.) , 59), Riermedia, , 60), Estermedia ), Paranáa (s.l.) ), Santa . neivai ); Rio Gr. neivai ) in ArgeL. intermedia and L. neivai had such high frequencies that their presence in peridomestic areas suggested a high risk of CL transmission [L. intermedia and L. neivai was considerably higher in anthropic areas [L. intermedia predominated in peridomestic areas [L. intermedia and L. neivai might influence local changes in the distribution of both species and in the transmission of CL.The intersection map Fig.  shows thsmission . In Corismission . The rolsmission , 9. Thisic areas , 65. Simic areas . Climateic areas . The futL. intermedia will become more tropical, while L. neivai will shift southwards and become more subtropical. This contradicts the suggestion that L. whitmani might replace L. intermedia (s.l.) as the more important vector of L. braziliensis in southern Brazil and nearby Argentina [L. whitmani is likely to share transmission of CL with L. intermedia only in the south-east region of Brazil, such as in Espírito Santo and Minas Gerais [L. whitmani has spread into the far south of Brazil, where L. neivai is the only reported vector [L. neivai produced by Maxent for north-west Argentina [L. neivai is the most abundant sand fly species in CL transmission areas [Our models indicate that rgentina . Currents Gerais . In factd vector , 63. Ourrgentina . In Argeon areas , and modL. intermedia mainly in central Piauí state. This is a region of transition between the Cerrado and Caatinga biomes, which are substantially drier and warmer than the Atlantic Forest where L. intermedia currently occurs [L. longipalpis and L. whitmani as potential vector species [L. intermedia in Piauí State, in the capital Teresina [L. intermedia and other potential vectors.In northeast Brazil, the models predict a future increase in climatic suitability for y occurs . This re species . There iTeresina , 28. FutL. intermedia and L. neivai in response to climate change will affect regional investigations of transmission cycles, including those using overlaid ecological niche models of Leishmania and its vectors in south-east Brazil [The shifting distributions of t Brazil . Becauset Brazil , 70, preLutzomyia intermedia and L. neivai will have smaller areas of climatic suitability available to them in the future, but they might disperse into new areas, such as southwards into Brazil and Argentina (L. neivai) and within northeast Brazil (L. intermedia). Ecological niche models should be species specific, carefully selected and combined in an ensemble approach.Our findings indicate that climate change will not always lead to the expansion of the geographical distribution of disease vectors such as sand flies.


Priority setting of health interventions is generally considered as a valuable approach to support low- and middle-income countries (LMICs) in their strive for universal health coverage (UHC). However, present initiatives on priority setting are mainly geared towards the development of more cost-effectiveness information, and this evidence does not sufficiently support countries to make optimal choices. The reason is that priority setting is in reality a value-laden political process in which multiple criteria beyond cost-effectiveness are important, and stakeholders often justifiably disagree about the relative importance of these criteria. Here, we propose the use of ‘evidence-informed deliberative processes’ as an approach that does explicitly recognise priority setting as a political process and an intrinsically complex task. In these processes, deliberation between stakeholders is crucial to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. Such processes then result in the use of a broader range of explicit criteria that can be seen as the product of both international learning  and learning among local stakeholders . We believe that, with these evidence-informed deliberative processes in place, priority setting can provide a more meaningful contribution to achieving UHC. This rationale also underpins the development of many international disease control guidelines. For example, the most recent WHO guidelines on when to start antiretroviral therapy for HIV are largely based on the expected epidemiological impact and cost-effectiveness.5In January 2016, the Prince Mahidol Award Conference (PMAC) in Thailand brought together more than 900 delegates from 60 different countries, to discuss priority setting of health interventions to achieve universal health coverage (UHC).In this editorial, we argue that the mere provision of cost-effectiveness information does not adequately support countries to make optimal choices. The reason is that priority setting is in reality a value-laden political process, in which multiple criteria beyond cost-effectiveness are important and stakeholders often justifiably disagree about their relative importance. Instead, we propose the use of ‘evidence-informed deliberative processes’ as an approach that does explicitly recognise priority setting as a political process and an intrinsically complex task. In these processes, deliberation between stakeholder is crucial to identify, reflect and learn about the importance of relevant values, informed by evidence on these values.We first outline the need for evidence-informed deliberative processes, illustrate this with examples from Indonesia, Thailand, and the Netherlands, discuss how to preserve the use of social core values in these processes, and then elaborate on their use to achieve UHC.10 It involves “pluralistic bargaining between different lobbies, modified by shifting political judgements made in the light of changing pressures”11 and is described as “a complex interaction of various decisions at diverse levels in the organization. There is no self-obvious set of ethical principles or scientific tools to determine what decisions we should take at various levels.”12It has been recognised since long that priority setting is in reality a value-based political process which takes place in an environment of social values and diverging interests.18 We argue that the whole of these values should be considered when setting priorities. Second, stakeholders often disagree about the importance of these values, and may have good reasons to do so when it comes to setting priorities. In the light of this, Daniels and Sabin have proposed the use of fair processes as an alternative approach to priority setting. In their seminal work on ‘accountability for reasonableness’ (A4R), they argue that it is more likely that stakeholders will agree on a fair process to set priorities than on the use of specific social values – and if they do so, stakeholders are then also more likely to confer legitimacy to the decisions that are made through this process.10If we agree that priority setting is a value-based political process, this then requires a paradigm shift in how research should approach the challenge of priority setting, addressing two important issues. Firstly, society – including relevant stakeholders such as patients, providers, insurers, and citizens – has a wide range of social values to judge decisions. These go beyond only health maximization as reflected in the criterion ‘cost-effectiveness,’ for example caring for the worse off in society or responsibility for own health.10 The aim of such processes is to develop a mutual basis for decisions among stakeholders, through the identification, interpretation and deliberation on a range of values that they find important, and informed by evidence where possible.Following this logic, and in acceptance of these two issues, the central question in priority setting becomes: ‘How can priority setting processes be organised such that stakeholders confer legitimacy to the decisions that will eventually be taken’? Or in other words: ‘so that they accept these decisions as reasonable?’ Daniels and Sabin propose conditions for transparency, relevance, appeal, and enforcement that processes should meet to achieve legitimacy.19 We see ‘evidence-informed’ and ‘deliberation’ as the two essential elements to achieve legitimacy in priority setting. To date, the work on fair processes (or how we name it: evidence-informed deliberative processes), has mainly been theoretical, and we see large potential for their practical application to support countries in their strive for UHC.Where Daniels and Sabin speak of ‘fair processes,’ we preferably name these ‘evidence-informed deliberative processes’ to better position the approach in the present public health debate. On the one hand, these processes are based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of relevant social values. On the other hand, they are based on rational decision-making – through evidence-informed evaluation of the identified values where possible. We speak of evidence-informed rather than evidence-based evaluation as the former concept leaves ample room for clinical experience as well as the constructive judgements of stakeholders such as practitioners and patients who are in constant interaction and dialogue with one another.20 The dotted lines reflect that the process is iterative. The way these steps can be applied in a decision-making context to foster the legitimacy of the eventual decisions, depend on the already existing priority setting process. To illustrate this, we provide two examples: (i) a case study to support HIV control in Indonesia, where priority setting has historically been implicit and entire processes need to be established; (ii) reimbursement decisions in the Netherlands and Thailand where HTA agencies are firmly established, and processes are relatively well-developed. It is obvious that in the former example, there is a relatively large need and large potential to improve the legitimacy of decisions, as compared to the latter example.We distinguish six steps in the use of evidence-informed deliberative processes , and the23 The process included a situational analysis on the current response to the HIV epidemic (step 1); the formation of a multi-stakeholder consultation panel (step 2); the identification of stakeholder values for the most important goals in HIV control, resulting in a set of criteria for priority setting that were considered reasonable by all panel members (step 3); a listing of HIV/AIDS intervention options by the consultation panel, including the collection of evidence to assess their performance (step 4); a deliberative discussion among the consultation panel members on this evidence, in view of their values and interests, to reach agreement on the final rank order of interventions (step 5); and a listing of institutions that would be suitable and/or fund high priority interventions (step 6). The overall aim of the process was to organise priority setting as an interactive learning process, in which the consultation panel refined the participatory steps of identifying, elaborating and deciding on the inclusion of further relevant stakeholders, criteria and evidence. A recent evaluation indicated that panel members were overall positive about the process, as it had improved the quality of decision-making̶especially in terms of use of multiple criteria and concrete evidence, active participation of stakeholders, and transparency of decision-making.23We recently supported West Java provincial authorities in Indonesia in the development of their strategic plan on HIV/AIDS.25 We see our work as a stepping stone towards such institutionalized processes, by spelling out important principles, documenting the initial experiences, and thereby creating awareness about its potential and limitations. In this context, we applaud the pioneering work of the International Decision Support Initiative (iDSI), which provides policy-makers at sub-national, national, regional, and international levels with technical support in coordinating priority setting as a means towards achieving UHC.25Yet, we also recognize that, for logistic and budgetary reasons, it is not feasible to develop such processes from scratch in all decision-making contexts where no priority setting process is in place. Ideally, countries should work towards the development of more generic centrally-led institutionalized processes, which would then be used at decentralised level as guidance for priority setting.26 In Thailand, the National Health Security Office employs a consultation panel which works with a large group of stakeholders to select interventions for assessment. The panel appraises the interventions on several criteria and deliberates to reach consensus on which interventions should be adopted in the benefit package.27 While these agencies employ various elements of evidence-based deliberative processes, they can still improve on other elements eg, stakeholder involvement in the Netherlands. This in order to further foster the legitimacy of their decisions.A number of national health technology assessment (HTA) agencies have already important components of evidence-based deliberative processes in place. For example, the Netherlands Health Care Institute (ZINL) employs an appraisal committee that represents the Dutch society and advises the Minister of Health. The committee deliberates on the social value of health technologies on the basis of the available evidence on four generic criteria , along with other contextual criteria that are considered relevant to the interventions under scrutiny.10 multi-criteria decision analysis (MCDA)28 and programme budgeting and marginal analysis (PBMA).30 The frameworks have been applied in various settings.36 The added value of our framework is that it combines these elements in an integrated approach.From a more methodologically point of view: various approaches to priority setting contain important elements of evidence-informed deliberative processes, including A4R,An important issue in the use of evidence-informed deliberative processes is how important social values can be preserved in the process. Stakeholder consultation, especially in the presence of vested interests, does not necessarily capture such public interests.18 We consider these as ‘core’ criteria, representing social values for which there is broad consensus on their importance and which are of generic relevance across countries, disease areas and health interventions. Their identification can be seen as the product of international learning, particularly in academic circles, on priority setting.37 One way to preserve these ‘core’ values in evidence-informed deliberative processes is to consider them as mandatory criteria for healthcare priority setting. Another, less stringent option is to use them as ‘opt-out’ criteria, for which a decision-maker should provide compelling arguments when declining them. We argue that the use of evidence-informed deliberative processes would then be instrumental to consider additional criteria for priority setting for which there is no broad consensus or which are only relevant for a particular decision – we call these ‘contextual criteria.’ These can include many criteria, eg, ‘responsibility for own health’ or ‘size of the population affected.’37In its report ‘Making fair choices on the path to UHC,’ the WHO recently proposed the use of ‘cost-effectiveness,’ ‘priority to the worse off,’ and ‘financial protection’ as the three most essential criteria for countries to consider when setting priorities.38 We applaud this effort as it provides valuable evidence on what we see as one of the core criteria in priority setting to achieve UHC. Yet, we recommend that this DCP work should be coupled with the development of evidence-informed deliberative processes at the country level, to also identify other contextual criteria.22Above we criticize WHO and the DCP project for its use of cost-effectiveness analysis as the sole criterion. Yet, how we see it, ‘cost-effectiveness’ can very well be a ‘core’ criterion in priority setting but should be interpreted in the context of other stakeholder values. In other words, our critique concerns the dominant use of cost-effectiveness analysis, not cost-effectiveness analysis as such. In fact, the DCP project is now conducting extended cost-effectiveness analysis (ECEA), to also capture the financial protection which interventions offer to target populations.39 We are:As a research group, we carry out various activities under the heading of the REVISE  project to further develop evidence-informed deliberative processes.40developing best practices on the various elements of these processes, eg, on whose values to consider, how to best guide the identification of criteria, and how to deal with vested interests. Our first findings are published in a companion paper;42collaborating with other disciplines such as public administration and political sciences, to learn from similar processes in other fields;experimenting with the implementation of processes in various contexts and evaluate these; and39stimulating the debate between researchers, policy-makers, and society on the need to set legitimate priorities in healthcare.It is time to focus on the development of evidence-informed deliberative processes to set legitimate priorities, and we call for more research in this area. We believe that with such processes in place, priority setting can provide an important contribution to achieving UHC.Not applicable.Authors declare that they have no competing interests.All authors have contributed to the conceptualisation and the writing of this paper.1Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands. 2Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands. 3Harvard T. H. Chan School of Public Health, Harvard University, Boston, MA, USA. 4Africa Centre for Population Health, Mtubatuba, South Africa.