In the UK, responsibility for many public health functions was transferred in 2013 from the National Health Service (NHS) to local government; a very different political context and one without the NHS history of policy and practice being informed by evidence-based guidelines. A problem this move presented was whether evidence-based guidelines would be seen as relevant, useful and implementable within local government. This study investigates three aspects of implementing national evidence-based recommendations for public health within a local government context: influences on implementation, how useful guidelines are perceived to be and whether the linear evidence-guidelines-practice model is considered relevant.Thirty-one councillors, public health directors and deputy directors and officers and other local government employees were interviewed about their experiences implementing evidence-based guidelines. Interviews were informed and analysed using a theoretical model of behaviour .Contextual issues such as budget, capacity and political influence were important influences on implementation. Guidelines were perceived to be of limited use, with concerns expressed about recommendations being presented in the abstract, lacking specificity and not addressing the complexity of situations or local variations. Local evidence was seen as the best starting point, rather than evidence-based guidance produced by the traditional linear ‘evidence–guidelines–practice’ model. Local evidence was used to not only provide context for recommendations but also replace recommendations when they conflicted with local evidence.Local government users do not necessarily consider national guidelines to be fit for purpose at local level, with the consequence that local evidence tends to trump evidence-based guidelines. There is thus a tension between the traditional model of guideline development and the needs of public health decision-makers and practitioners working in local government. This tension needs to be addressed to facilitate implementation. One way this might be achieved, and participants supported this approach, would be to reverse or re-engineer the traditional pipeline of guideline development by starting with local need and examples of effective local practice rather than starting with evidence of effectiveness synthesised from the international scientific literature. Alternatively, and perhaps in addition, training about the relevance of research evidence should become a routine for local government staff and councillors.The online version of this article (doi:10.1186/s13012-017-0589-5) contains supplementary material, which is available to authorized users. Evidence-based guidelines are produced worldwide to inform clinical and public health practice; for example, in the UK by the National Institute for Health and Care Excellence (NICE); in the US by the National Guideline Clearinghouse and U.S. Preventive Services Task Force; in Canada by the Canadian Task Force on Preventive Health Care and in Australia by the National Health and Medical Research Council. Despite the considerable financial and human investment into the development and dissemination of guidelines, suboptimal implementation of recommendations within guidelines is documented in a range of arenas including public health –3. BarriIn England, in 2013, responsibility for the delivery and commissioning of many public health services transferred from the National Health Service (NHS) to local government, the largest reorganization of public health since the creation of the NHS in 1948 . MandatoWhat are the influences on implementation of guidelines?How useful are guidelines perceived to be?To what extent is the linear evidence-guidelines-practice model fit for purpose?NICE has been developing public health guidelines since 2005. Although broad in scope and reach, the principal audiences for the guidelines were originally the public health teams working in the NHS who were responsible for implementing guidelines in the context of local variation. The NHS was by 2005 already well used to receiving and implementing NICE Technology Appraisals and Clinical Guidelines. The way that public health guidelines were produced by NICE drew heavily on the methods of evidence-based medicine, an approach which was widely understood if not always supported, in the NHS , 7. ThisA unitary authority in the north of EnglandAn outer London boroughA two-tier authority with rural areasAn inner London boroughThirty-one interviews were conducted in four local government with councillors, local government officers  and with members of Clinical Commissioning Groups. Potential case study sites were identified through contacts already known to research team members, through a short article about the research project circulated in a newsletter produced by the Association of Directors of Public Health, and through approaches made to attendees at a NICE local government event. The final four case study sites were selected to reflect a range of local government characteristics:Table An interview schedule was developed covering topics relevant to the research questions and using a theoretical framework that focussed on the behaviour change required to increase implementation of evidence. The theoretical framework has three broad elements required for behaviour change: capability, opportunity and motivation (COM-B model) and has Interviews were conducted face to face and audio-recorded, with the recordings subsequently anonymised and transcribed by a UCL-approved professional transcribing service. Interview subjects were sent the transcription of their interview and given an opportunity to correct errors or to clarify anything they felt was unclear. Local government officers were asked about their job role prior to 2013, to identify whether they came from an NHS or a local government background. Table Analysis of the 31 interview transcripts was conducted in three waves, with 11 transcripts analysed in the first wave and 10 analysed in each of the second and third waves. Conducting the analysis in three waves allowed the researchers to develop, refine and apply an analytical framework, including identifying themes that emerged from the data during analysis. Examining the transcripts in three waves allowed a process of triangulation within the data, confirming which findings could be generalised across all four case study sites. Examination of the third wave of transcripts confirmed that a level of saturation had been achieved, with no novel findings emerging at this stage.To investigate the influence of participants’ capabilities, opportunities and motivations on guideline implementation using the COM-B model, a deductive approach was considered to be the most appropriate for initial analysis . QuotatiThe data were further analysed inductively to identify themes within COM-B sub-components . LA, CL Data are presented by theme rather than theoretical sub-component for conceptual accessibility.Role of context in implementation—budget, capacity and political influence were important influences on implementation.Limitations of research evidence—the concerns expressed about guidelines included that recommendations are presented in the abstract, lack specificity and do not address complexity or local variation.Using local evidence—local evidence was seen as very important, being used to provide context for recommendations and sometimes being used instead of recommendations when they conflicted with local evidence.Theme saturation was achieved, and three key themes related to the three research questions were identified:I don’t think it’s as straightforward as to say it was all implemented, and therefore it was all evidence-based. Elected members and public health team members considered specific aspects of context to be important influences on how guidelines were implemented. The limited budget of local government and accommodating political perspectives influenced how and which recommendations were implemented. Participants acknowledged that factors beyond evidence influenced implementation of recommendations.And I know putting it on the web’s brilliant and I do think the pathways do help, but the only problem with the pathways is you see the recommendation of it out of context and its bias of recommendations (don’t serve who they are supposed to) Context was identified as important in implementing recommendations but was seen to be missing from the way recommendations are framed.We tend to talk about evidence-based decision making…Which is part of this delusory narrative… there are other things that have a place…A big bit of which, equally important, is public preference… And there are other things, in that model, such as, what resources you’ve got…Because there’s no point in the guidance being separated from capacity…So, I think, one of the things that would help in the future, is, for a bit more work to be done, on that conceptual model. The challenge for NICE is that this is not an environment where you can be driven totally by evidence or by science The world’s more complex, than the NICE evidence base, really, can do it. Some participants challenged the traditional evidence-based model arguing that contextual factors including public preference, i.e. what the members of the public say they would like, and resources were as important as evidence in informing implementation.In relation to any of these major public health issues, it’s a range of different things and, kind of, no one knows the exact balance because that answer will be different with different people in different communities. But participants also acknowledged that the balance of different contextual influences will vary across local governments.What’s the difference between, this kind of platonic, perfect world, in which evidence-based decision making happens… And everything is rational. And, the real world, where, a whole lot of other factors, become part of the decision making process. And, what you’re trying to do is, balance, and make those as effective and rational as possible, within certain parameters…And that situational decision making, is something which we haven’t got a model for…The model is based, on here’s the evidence, everybody should do what the evidence says, because the evidence is right, and this is what it says. And, it’s kind of really, I just think, we should be a bit more sophisticated, in the traction model, or something. I can’t, it’s… I’m not quite describing it properly, but the way it gets taken up in the real world Participants conceptualised the evidence-based based model as a ‘platonic world’ where evidence drives implementation, contrasted with the ‘real world’ where other factors influence implementation.This is about the evidence we use, kind of, transferring with us into local authorities, where the evidence needs to speak the different language that we need to speak This was echoed by others acknowledging that the evidence needs to ‘speak the different language’ in local government.I met with the obesity team last week in the council and they said they have no time, they spend their whole time reviewing contracts and planning to get new contracts set up for 2016. So if I want to do anything in driving forward the obesity strategy, I must provide management resource from the CCG. Implementation of recommendations is hindered by a lack of capacity as staff time is taken up with administrative duties.For the last maybe 20, 30 years, funding has been quite stable, but then in the last five years, we’ve had to change rapidly and do things very differently, and sometimes, out of adversity, you actually come up with better solutions Recent budget cuts are viewed as an opportunity by some to deliver more effective services.I know councils don't like being told what to do. They don’t like dicta and whereas the NHS might be able to say through its hierarchy of command and control, NICE has said this is, you know, this has passed through the hurdles, it must be implemented within three months, they can do that. You can't tell a local authority to do that. They say, well, thank you for your opinion. We'll weigh it up carefully and we'll do what our voters tell us. Welcome to democracy. There was the perception that implementation was more difficult in local government compared to that in NHS because political preference would take priority.The NHS has a very strong narrative, that it tells itself, that it’s evidence-based, and scientific, and rigorous, and everything like that. And that local government decides thing on whim, and political priorities. My experience of coming over, in to local government, was actually, that the truth of that is much more complex But this view was not shared by all.Well, as you know, every politician works on an anecdote. We have to use evidence either to support or refute the anecdote and sometimes you get overruled. If you manage to … ensure the evidence base is followed 75% to 85% of the time probably in this environment, we’re doing pretty well. Some participants perceived evidence-based recommendations as being used to respond to political influence.Participants highlighted limitations in the evidence used to inform recommendations—it was out of date, and in the way the evidence was presented in recommendations, they lacked specificity and did not take account of complexity and scale.I would say, is that some people have a view, that NICE guidance is not realistic, to the local situation, sometimes. Some participants appeared to think that national recommendations were of limited relevance at the local level.There was a really broad recommendation and it was about doing the system change stuff, but because the evidence was based on a big system change, we know that worked, but actually what you want to understand is smaller scale stuff. What difference does this particular intervention do? And it’s quite hard to do that if, you know, it’s been implemented as part of the big package of things, and also the context is just really important. However, there appeared to be two views on the way this limited relevance was manifest. Some perceived recommendations to be too broad to capture complexity. For example, where evidence informing recommendations is drawn from a number of interventions delivered as part of a strategy, there was perceived to be little guidance on which specific aspects of the package were effective.National evidence is important in terms of big areas, volume areas. I think where it is less effective is being able to allow you to translate that to your local population, and, depending on your population mix, something which is a big issue nationally may not necessarily translate across to your local population. So, let’s take, for example, in xxxx, you know, around 26% ethnic minority communities; that isn’t the picture nationally. So, you see, the problem that you’ve got is to meet and understand and reflect your local needs. Or where recommendations were not relevant to a local population.It will give you very sophisticated analysis of, how to increase x target group, to three times 30 per week. But, it misses some of the big, systemic questions. It’s always down to a small project level, intervention level. It doesn’t look at mainstream world, changing, the way it is. For others, recommendations were viewed as missing the ‘big picture’.I try to get my Health Intelligence Team when they’re looking at it, to actually look at the research that was underpinning it as well, because I think sometimes if it’s based on these big studies and there are all these big, grand recommendations you think, actually that’s not always helpful locally when you’re doing some bits of it but you just want to know, will this extra thing work or not? To address the issues of complexity and scale, some public health teams went back to the original evidence informing the recommendations to determine the extent to which they were relevant to their local situation.The councillors knew exactly what was going on, how it would work, who, or how it was working, how it was evolving. It was in real time, happened in the last 12 months. The problem with NICE guidance is, it would never capture that sort of phenomenon. …NICE guidance, by definition, can only include evidence, that’s based on well-funded research, that appears in peer reviewed journals, that meets research threshold and criteria, that by definition, by the time it ends up in an effectiveness review, is three years old, at least. So, it’s always behind time, influenced by who’s funded the research, and the way they view and frame their world, and actually, their economic interest as well, in terms of their clinical guidance. So already, you’ve got a research time, and framing bias, in the NICE guidance. And, that’s very problematic, because, in, when you’re trying to solve complex problems, in real time, in a democratic setting, you need a different way of working, and a different, a different framing of solutions, and what evidence counts The time lag between research evidence being produced, synthesised and informing recommendations was contrasted with the perceived speed of local government decision making.I think sometimes NICE recommendations can seem a little bit vague at times, quite abstract I think sometimes the recommendations aren’t specific enough. It’s really not clear. …some of the recommendations sometimes come across as an outcome that you want to achieve, whereas I think practically quite often you want recommendations about what activity would give you that outcome. Guidelines were thought to lack specificity and were considered to be framed in terms of outcome rather than behaviours involved in implementation.I remember when I went reading guidance and thinking, god this is so turgid and just like thinking, I’m just not going to read it. Now I’ve got used to it, I don’t really have a problem … just think it feels a bit like an academic report. Some participants found the recommendations hard to read initially.I think if you can ground them… more sort of practical examples, case studies, things like that, but again it’s always got to be concise Using case studies to illustrate implementation was suggested as a way of making recommendations less abstract.I don’t necessarily think [the local authority understanding of what evidence is] is always a different standards thing, it's just a different interpretation of how significant evidence should play in decision making. People… will dismiss it, because they feel it was done under… research conditions, and doesn’t really apply to the real world. Local evidence was used in most cases to provide the context for recommendations. Local evidence took precedence where recommendations were viewed as not relevant to the local situation or in conflict with local evidence.We’re doing something similar looking at the obesity pathway, because we’re bringing in people from… because obviously this is a system wide thing that needs to happen, and we have used guidance to come up with recommendations, but we’ve invited them to feed in. So, you know, this is what the guidance says, what does the local evidence say? So it does help us to have conversations about things, I think, and they have different types of evidence sometimes as well. We will go to what happens in [this part of] London, who else is doing something, or what happens in, I think it’s xxxx or xxxx, which borough is similar demographically to [this borough], and what’s happening in these areas. So we will either follow the national Guidance or we will look at other areas and see what best practice we can then apply to xxxx We’ve got a very diverse borough. It’s quite different demographically to the rest of xxxx, the rest of the county. I think there’s something about taking the national evidence, taking the national guidance and then always asking the question what does this mean for us. Does this speak to our community, you know? Does this take into account ethnicity? Does this take into account our levels of deprivation, etc? Does this apply in this case? And often it does. Sometimes you might think, actually, there’s shifts we need to make. It’s about localising the national. I probably do start with the national. Participants described the process of combining recommendations and local evidence.It’s how you transfer that into a local area, I think that’s the art of public health, I guess, because there are – does it – is it – is – can you apply it to the local area? And I think you would always use that as a framework to go and then audit, check what we’re doing locally, is it in line with this, that and the other? So you check all that out. And where there is a difference it’s understanding, why is there a difference? Some viewed implementing public health recommendations as an ‘art’.So do acknowledge that any local intelligence from research or any local engagement activities or surveys or fieldwork; it’s less strong because it’s not been through that rigour of process, but valid as a piece of… for what it is in terms of, this is public opinion, service user feedback, it’s a sense of opinion or feeling around this topic of what needs to be done and what the priority areas might be to be addressed. The limitations of local evidence for example, service user feedback, was acknowledged, but was still viewed as an important aspect of implementation.I think, very practical evidence of, you know, showing me a, sort of, case study, if you like, which has been done on the scale somewhere else. And so they would be probably the most persuasive types of evidence, and I think the more it’s, sort of, national or the more it’s vague or it’s less specific, then the less persuasive that would be If there’s a national guideline for something, we should aim to hit the target but make sure that it’s not to the detriment of what we’re providing for a person. Sometimes we miss targets because we are doing things differently. So targets are all well and good as long as they are effective. It might be NICE guidance, but we might say, well that won’t work in xxxx, because we know of x, y, z issues. But that sense of, again, NICE guidance, well this is what is says but it doesn't fit our local area, we didn't see any fit for us. Without necessarily looking at the detail of it. Just, it's coming from outside therefore we probably shouldn't have anything to do with it. .Both elected council members and members of the public health team cited examples where local evidence was more persuasive than national recommendations.The context in which recommendations are implemented, resources and political influences were perceived to influence approaches to implementation. Guidelines were thought to be useful; however, participants identified several limitations. These limitations included the following: first, recommendations not addressing the complexities, local nuances and scale faced by local government when implementing recommendations; second, a view that the research evidence on which the recommendations were based was sometimes out of date due to the time taken to produce evidence-based guidelines; and third, that recommendations focused on desired outcome rather than the process public health practitioners should undertake in order to achieve that outcome.Limited support was found for the linear evidence-guidelines-practice model as local evidence was seen as the starting point to provide a context for implementing recommendations, or in some cases, recommendations were disregarded in favour of local evidence where they were not specific enough or where they contradicted local evidence.The finding that recommendations do not align with context is echoed in the global public health setting, suggesting that evidence used to inform public health policy does not take account of contextual factors such as implementation in low-income settings  or lack Participants in this study reported that the lack of specificity of recommendations was a barrier to communication. Similarly, a study of implementing evidence-based recommendations in the UK National Health Service reported using examples of local initiatives were helpful where there was a lack of clarity on how recommendations should be implemented . WritingThe emphasis on the use of local evidence over national recommendations chimes with the extant literature. An investigation into sources of evidence used by UK public health policymakers reported that experts in the area, principally directors of public health and council officers, were used as sources of evidence more often than NICE guidance . The samArguably a priority for those producing evidence-based guidance for public health is also to provide more information on how to implement recommendations for public health practitioners. These could include case studies of recommendation implementation illustrating how local context can be taken into account. Participants cited examples of learning from the best practice elsewhere to guide implementation and suggested case studies of implementing recommendations be included in guidance as a way of addressing the lack of specificity.It could be argued that public health decision-makers and practitioners in local government need to develop skills in understanding research and the practice of implementing evidence-based recommendations. However, recommendations based largely on academic research also need to be fit for purpose so that they are meaningful to decision-makers and practitioners  and takeA second priority is making recommendations as specific as possible to support implementation and to present these in an accessible style and format.Limited financial resources were identified by participants as a barrier to implementing recommendations. Some participants noted that where cost saving in guidance was mentioned, it was in relation to costs saved to the NHS. In order to make guidance more relevant to local government, a demonstration in guidance of costs saved to local government budgets may support implementation of recommendations.Where evidence is limited or non-existent public health practitioners can use tools and methods developed in behavioural science to inform the design of public health interventions . As noteAs for many studies with a qualitative interview design, the main limitations of this investigation are the self-selection of a modest sample size. However, because the sample includes a range of local government employees from project officers to directors of public health as well as elected members of councils, it is unlikely these limitations compromise the integrity of the study.The evidence-based medicine model is premised on a linear pathway from primary research evidence to the appraisal and synthesis of the evidence, to the development of guidelines, and on to practice. In clinical medicine, guidelines must however be used with an absolute regard for the needs of the individual patient in front of the physician; clinical judgements must always therefore be made. In the words of Sir Michael Rawlins, the first chair of NICE, ‘Guidelines are not tramlines!’.In public health, there is no individual patient, so clinical judgement about individuals is not really relevant. Instead the question is whether the guideline has local applicability to local populations. Public health guidelines are based on the best national and international research evidence and will have been assessed for their internal validity. While NICE’s public health committees work hard to assess transferability and external validity, the particular needs of local communities generated by local circumstances may well confound their efforts so long as the guideline production pipeline works in the conventional direction of evidence-based medicine. This knowledge transfer problem was thrown into stark relief with the move in England of much public health from NHS to local government, where the standard assumptions of evidence-based medicine are not universally shared and where the methods of traditional guideline production are not necessarily appropriate.Public health is characterised by much more diverse and less well-defined problems than clinical medicine and has complex pathways of action and intervention –26. Pre During the planning for the reorganisation of public health into local government before 2013, there was no public discussion about the knowledge transfer issues which were going to arise as a consequence of the reforms. NICE was not consulted by the Department of Health and the learning that NICE had already accumulated about implementation in local government was not used to inform the process or the structures of engagement between the centre and local government post-2013 . The res


A realist review of empirical studies of healthcare QI initiatives will be undertaken to examine the influence and impact of contextual factors on quality improvement in healthcare settings and explore whether QI initiatives can work in all contexts.Context shapes the effectiveness of knowledge implementation and influences health improvement. Successful healthcare quality improvement (QI) initiatives frequently fail to transfer to different settings, with local contextual factors often cited as the cause. Understanding and overcoming contextual barriers is therefore crucial to implementing effective improvement; yet context is still poorly understood. There is a paucity of information on the mechanisms underlying The review will explore which contextual factors are important, and how, why, when and for whom they are important, within varied settings. The dynamic nature of context and change over time will be explored by examining which aspects of context impact at key points in the improvement trajectory. The review will also consider the influence of context on improvement outcomes (provider- and patient-level), spread and sustainability. The review process will follow five iterative steps: (1) clarify scope, (2) search for evidence, (3) appraise primary studies and extract data, (4) synthesise evidence and draw conclusions and (5) disseminate findings. The reviewers will consult with experts and stakeholders in the early stages to focus the review and develop a programme theory consisting of explanatory ‘context–mechanism–outcome’ configurations. Searches for primary evidence will be conducted iteratively. Data will be extracted and tested against the programme theory. A review advisory group will oversee the review process. Review findings will follow RAMESES guidelines and will be disseminated via a report, presentations and peer-reviewed publications.The review will update and consolidate evidence on the contextual conditions for effective improvement and distil new knowledge to inform the design and development of context-sensitive QI initiatives. This review ties in with the study of improvement programmes as vehicles of change and the development of an evidence base around healthcare improvement by addressing whether QI initiatives can work in all contexts.CRD42017062135PROSPERO  Evidence-based healthcare is the translation of evidence derived from valid and reliable research and evaluation to improve healthcare practice and patient outcomes . QualityAt its most simplistic, context can be defined as ‘all factors that are not part of a quality improvement intervention itself’ , i.e. ‘tThe theories underpinning an intervention and its implementation also contribute to the context . AccordiTraditional QI approaches, generally focusing on efficacy, often paid little attention to context, with contextual factors ignored or poorly reported in implementation reporting , 22. HowThe increased interest in context in recent years has resulted in an emergent number of implementation frameworks and models, many including hierarchical checklists of factors to enable assessment of context influences at different system levels by clinicians and QI teams and to aid organisations with the design and implementation of their improvement efforts , 27–30 aAlongside these frameworks, there is a growing body of literature on context in QI. Existing reviews of improvement studies and reviews of reviews in this field have focused on identifying, describing and categorising contextual factors that influence QI implementation and effectiveness , 17, 31,how interventions work, unlike for example, theory-driven and realist evaluation, which seeks to understand the interactions between intervention, variations in context and underlying mechanisms of change [Despite the fact that new methods of identifying and addressing context are emerging, and understanding about the influence of context in QI is increasing, evidence suggests that the science of assessing or measuring contextual factors is still ‘immature’ . Contextf change . Althougf change , 34.how and why) is required to unpick the complex interactions between context, QI intervention and outcomes. This realist review aims to bridge these gaps by providing a context-sensitive evidence synthesis focusing on the influence of contextual factors on QI implementation and transferability.The variability across the literature on context and its associated disparate body of evidence suggests that further work is required in this area. Therefore, the question remains: in terms of improvement methodologies and implementation, which characteristics of context matter, and how, why, when and for whom do they matter? Furthermore, how can this knowledge interface with current and future health improvement work? Existing reviews focusing on effectiveness have not tended to address the underlying causal mechanisms by which context influences evidence-based practices and why QI initiatives do or do not work in given contexts , althougThis review ties in with the study of improvement programmes as vehicles of change and the development of an evidence base around healthcare improvement by addressing the question of whether improvement programmes can work in all contexts. Studying the role of context in impacting on the implementation of ‘improvement methodologies’ (as opposed to interventions) is a key part of this learning. This suggests the need to collate information on the settings and circumstances in which improvement methodologies are effective, capturing the multi-level and multi-factorial nature of contextual influences, and to identify the successful components that potentially are generalizable across different programmes. Dissemination of such knowledge would support a more context-sensitive approach to the design and implementation of improvement initiatives, incorporating the influence of local context into the design of new interventions and the evaluation of existing programmes. Not only will this be of importance to academic researchers but ultimately will be of practical value to QI practitioners, on an individual, team and organisational level, in real-world settings.By testing and refining existing theories of context, this realist review will address Bate’s concern that ‘we do not need a new model or framework to study the role of context in quality improvement; rather, we need to test and synthesise existing ones’ . In updaRealist reviews, building iteratively on theoretical and empirical literature, are able to accommodate and synthesise a diversity of evidence types ; indeed,Realist inquiry focuses on taking an explanatory approach with an emphasis on understanding causation. It is based on the assumption that all programmes and interventions have underlying theories—whether implicit or explicit—about what might cause change and that these theories may be modified as evidence emerges , 39. ThuDespite the relative newness of the method, realist inquiry has been utilised in recent reviews in the field of evidence-based healthcare improvement –51. The The overarching research question to be addressed is the following: how and why does context influence healthcare QI initiatives? The primary aims of this realist review are to (i) identify the contextual factors that influence the implementation, effectiveness, sustainability and transferability of quality improvement initiatives in healthcare; (ii) provide a theoretical explanation of how, why, when and for whom these contextual factors are important and (iii) transform the wealth of data in this area into a cohesive evidence base of ‘what works, for whom and in what circumstances’, in terms of implementation in given contexts. A further, secondary aim concerns the interface between research, policy and practice. After completion of the review, findings will be translated into evidence-based, practical knowledge and recommendations that can be shared with and applied by policymakers and practitioners.For the purposes of this review, quality improvement in healthcare is defined as ‘better patient experience and outcomes achieved through changing provider behaviour and organisation through using a systematic change method and strategies’ , broadlyAt the beginning of the review process, a multi-disciplinary review advisory group will be set up to support the review team, oversee progress, and help develop consensus by providing a forum for monitoring, discussion and the sharing of feedback at all stages. This group will initially consist of the review team and members of the Scottish Improvement Science Collaborating Centre (SISCC), a multi-sectoral network of researchers, educators and practitioners across academia and health and social care. The composition of the group may change or expand during the course of the review; other identified experts and stakeholders may be invited to collaborate with the research team as the review progresses. The group will contribute to all aspects of the review from theory development to interpretation of findings.The review will follow the fundamentals of realist inquiry, a theory-driven interpretive approach that seeks to explain the root causes of programme outcomes and diverse patterns of impacts and effects, by evaluating evidence from a range of sources . The aimC + M = O; a programme theory may consist of multiple CMO configurations that explain the context, mechanism and outcome relationships and the pattern of outcomes. By recognising the importance of external and environmental influences, realist inquiry can be used to unpack complex programmes and interventions and explore the reasons underlying how and why they do or do not work in given contexts [In realist thinking, programmes are theories . The hypcontexts .The review will be conducted following the five key stages of Pawson’s realist review template : (1) claAs the area of interest is potentially very wide-ranging, a priority at the first stage will be to refine and narrow the research question via a broad-based scoping exercise. It is at this point that key themes and concepts are defined as the focus of the review begins to take shape and the reviewers begin to develop a theoretical understanding of the research topic. Following Pawson’s methodology , relativAs the scoping exercise progresses and findings begin to emerge, the reviewers will develop an outcome-focused realist programme theory, based on the identified existing programme theories and structured around explanatory context–mechanism–outcome (CMO) configurations, consisting of propositions about the ways in which contextual factors are anticipated to influence improvement initiatives in practice.In keeping with realist methodology, key stakeholders and experts, including QI practitioners, will be consulted throughout this first stage in order to provide a range of additional perspectives and an ‘expert framing’ of the issues that can contribute to the developing programme theory. Consultation will take place in person or by email. Theory development will also be informed by meetings of the review advisory group. The resulting theoretical, explanatory model and its component contexts, mechanisms and outcomes of interest will be used as the framework for the succeeding stages of the review.Systematic searches for primary studies that are relevant to the programme theory will then be carried out. This second search phase will involve seeking additional relevant evidence  primarily to test and refine the programme theory. The search process and appraisal of evidence will be theoretically driven, purposive and conducted iteratively. Searching will be progressively extended and refined, based on emerging findings, as the review evolves, until saturation is reached. The aim of realist review is theoretical saturation as opposed to fully comprehensive coverage of the topic area. Selection of sources will be based on relevance to aspects of the programme theory.A combination of search strategies will be utilised. Internet search engine and electronic database searching will be carried using keywords based on the programme theories identified in the exploratory search. This will be supplemented by a ‘cited by’ article search and a search of citations included in the reference lists of included papers. Sources of grey literature, including unpublished reports, will also be investigated, as well the websites of relevant organisations, such as The Health Foundation and the Institute for Healthcare Improvement. The reviewers will make use of snowballing techniques and consultation with experts and stakeholders. Given that a wide range of documents may contain data that can contribute to a realist review, multiple types of evidence will be included. Based on the data retrieved and emerging findings, the direction of the review may shift or expand in scope.Search results from electronic databases and other sources will be imported into reference management software (Endnote) and duplicates removed.The realist appraisal and extraction process differs from a traditional systematic review, as inclusion and exclusion criteria are based on the programme theory and what the literature is able to contribute to it. In a realist review, the units of analysis are not the interventions themselves per se but the theories underpinning the interventions. Accordingly, evidence will be selected for inclusion based on its relevance to the programme theory; the screening process will be based on an assessment of ‘fit’ to the research question, with the aim of identifying evidence that can empirically test, revise or refine the programme theory (or elements of it). ‘Bespoke’ data extraction forms  will be Again, in contrast to the traditional review process, during this stage, the project team will revisit and if necessary revise the focus of the review based on emerging findings. This may require further refinement of the inclusion/exclusion criteria and further purposive searches in response to any revisions of the programme theory, followed by the integration of any additional evidence.The key analytic process in a realist review involves iterative testing and refinement of theoretically based explanations (i.e. the programme theory) using empirical findings in data sources . The goaFindings will be translated into evidence-based, practical knowledge and recommendations that can be shared with and applied by policymakers, and QI researchers and practitioners. They will be disseminated in the form of a final report, presentations to stakeholders and peer-reviewed publications.This study will use a realist review approach to synthesise the available evidence and to enable a greater understanding of ‘what works, for whom, in what circumstances, when, how and why’, in terms of the influence of contextual factors on quality improvement initiatives in healthcare settings, thus providing a new insight into the ‘problem of context’ in QI. The use of a realist approach will allow the review to describe and explain how and why QI initiatives work (or fail to work) in different contexts by exploring the underlying programme theories and the interactions between contextual factors, mechanisms of change and outcomes.Synthesising current knowledge on evidence of context  within quality improvement and making this knowledge available and accessible to stakeholders will facilitate the design and development of evidence-based, context-sensitive improvement activities that can be planned and delivered in a way that takes this latest evidence into account, mitigating for known contextual barriers and enhancing facilitators in advance wherever possible and incorporating local knowledge to enhance implementation and improvement strategies and improve transferability. This approach will ensure that current knowledge of contextual barriers can be taken into account in designing or planning for implementation and that ‘common’ or evidence-based contextual problems are not repeated from the outset but are mitigated for in planning, including tailoring of local plans.


Decisions in public health should be based on the best available evidence, reviewed and appraised using a rigorous and transparent methodology. The Project on a Framework for Rating Evidence in Public Health (PRECEPT) defined a methodology for evaluating and grading evidence in infectious disease epidemiology, prevention and control that takes different domains and question types into consideration. The methodology rates evidence in four domains: disease burden, risk factors, diagnostics and intervention. The framework guiding it has four steps going from overarching questions to an evidence statement. In step 1, approaches for identifying relevant key areas and developing specific questions to guide systematic evidence searches are described. In step 2, methodological guidance for conducting systematic reviews is provided; 15 study quality appraisal tools are proposed and an algorithm is given for matching a given study design with a tool. In step 3, a standardised evidence-grading scheme using the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE) methodology is provided, whereby findings are documented in evidence profiles. Step 4 consists of preparing a narrative evidence summary. Users of this framework should be able to evaluate and grade scientific evidence from the four domains in a transparent and reproducible way. The potential fallacies of relying solely on expert opinion to establish best practice in clinical decision-making and public health policies are well known globally . In guidJudging the effectiveness of infectious disease prevention and control interventions creates challenges related to the population-level effects and long-term aspects of the intervention. In addition, a variety of other elements need to be considered in decision-making, including disease burden, risk factors for infection or mode of transmission. In adopting the concept of ‘best available evidence’ , evaluatAlthough considerable progress has been made regarding the use of systematic reviews for public health decision-making, the approaches currently used for conducting and appraising systematic reviews in public health have their limits. They regularly report effect estimates and risk of bias, but often do not assess the certainty of the evidence for the entire body of such across outcomes. Moreover, traditional approaches mainly focus on intervention effectiveness and safety, but do not provide a generalised approach that addresses all factors relevant to decision-making in infectious disease prevention and control, such as the epidemiology, interventions as well as diagnostics and risk factors.New evidence appraisal and grading system approaches that incorporate information from studies with different designs have been developed. The most prominent system, developed by the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE) , has beeAn approach to infectious disease prevention and control that adheres to the principles of evidence-based public health, using a defined framework for the assessment of the certainty in the evidence, has a number of advantages over conventional approaches. In particular, such an approach:Helps to improve the quality of the resulting public health recommendations.Reduces anticipated or actual arbitrary decisions.Improves transparency.Builds trust and supports the acceptance of recommendations by professionals and the public.Helps to compare recommendations endorsed by different countries or institutions.The approach presented here applies the most advanced methodology for assessing certainty of the evidence, the GRADE methodology. Even though the individual elements  of the proposed approach are not new, they have never been integrated into one comprehensive framework that guides users from identification of the relevant research questions to preparation of a final evidence assessment summary for the area of infectious disease prevention and control. Moreover, in contrast to other approaches, the framework puts particular emphasis on question framing and the selection of appropriate risk of bias tools. The framework provides evidence assessment guidance in infectious disease epidemiology, prevention and control, but is not designed to conduct a rapid assessment for the purpose of answering urgent questions in public health crises or emergencies.PRECEPT intends to provide methodological guidance for public health agencies, scientists working in the field of evidence-based public health, and other institutions and individuals involved in appraising evidence and developing public health guidance with a focus on infectious diseases. Here we present an overview of the PRECEPT approach .The PRECEPT framework focuses on the following four domains:Disease burden (significance of the problem), which encompasses studies on the incidence, prevalence and severity of diseases and complications, as well as studies on the perception of diseases in target populations. For example, what is the incidence of hepatitis B in sex workers in eastern European countries?Infection and disease risk factors (causes of the problem), which encompasses studies on preventable and non-preventable risk factors for infection, disease and complications. For example, is sepsis acquired in the neonatal intensive care unit a risk factor for cerebral palsy?Diagnostics (detection of the problem), which encompasses studies on diagnostic accuracy (sensitivity and/or specificity of diagnostic tests/measures). For example, what is the sensitivity and specificity of tests for tuberculosis in children?Interventions (consequences of action against the problem), which encompasses studies examining efficacy, effectiveness and adverse effects. For example, what is the effectiveness of vaccination of infants against rotavirus for the prevention of hospitalisation?The general approach to this process of assessing evidence within these four domains consists of four steps .PRECEPT proposes applying the extended Patient/Population, Intervention, Comparator/Comparison/Control, and Outcome (PICO) format and integrating other questions to the extent possible.PICO is considered to be the most appropriate way of framing questions related to interventions (domain (iv)) ,9. HowevIn infectious disease prevention and control, researchers are regularly confronted with complex of questions that have to be addressed in combination. For example, the question, ‘Should vaccination against rotavirus be recommended for all infants?’ comprises multiple questions from different fields that need to be addressed e.g.:What is the incidence of rotavirus infection among children < 5 years of age?Is age a risk factor for rotavirus infection among children < 5 years of age?What is the effectiveness of vaccination against rotavirus?What is the risk of intussusception associated with the vaccine?In such situations, developing a logic model  tends to be helpful  to identEvidence should be identified and synthesised using a rigorous systematic review process. A systematic review usually includes six steps  of each study identified during the systematic review, the PRECEPT framework proposes using specific quality appraisal tools (QATs) according to study design . A selecIn the case of quantitative data, data synthesis can be conducted using meta-analytic techniques. If statistical pooling appears to be inappropriate, e.g. if interventions are too heterogeneous to be grouped in a meaningful way, if data are highly heterogeneous or if study designs differ considerably, a tabular, graphical or narrative synthesis might be more useful .It is estimated that a new full systematic review takes between six to 24 months, but using existing systematic reviews for the development of new evidence assessments can shorten this by one to two thirds . The usePRECEPT recommends a standard approach that uses the GRADE methodology to evidence-grading across all four types of domains.According to GRADE, the certainty in the evidence indicates the extent to which one can be confident that the estimate of effect is correct . The uniFor questions related to burden of disease (domain (i), outcomes can be measures of incidence or prevalence, as well as mortality or disability-adjusted life years. For questions regarding risk factors (domain (ii)), outcomes are those variables that are dependent on the risk factor. For diagnostics (domain (iii)), true positives, true negatives, false positives and false negatives are regularly used as surrogates for subsequent clinical outcomes. For questions regarding interventions (domain (iv)), outcomes are endpoints of clinical trials or observational studies. Taking the entire body of evidence, not an individual study, on one outcome into account, four levels of certainty in the evidence, i.e. confidence in the estimate of the effect, are applied to the results of the review:  very low, low, moderate and high. For interventions, RCTs are initially graded as high certainty, whereas all types of observational studies are classified as low certainty. Based on a defined set of criteria, decreasing (downgrading) or increasing (upgrading) by one or two levels is possible. Five criteria are applied for downgrading: (i) risk of bias, (ii) inconsistency, (iii) indirectness, (iv) imprecision and (v) publication bias. Three criteria are used to upgrade the certainty in the evidence: (i) large effect, (ii) evidence for a dose-response relationship and (iii) all plausible confounding would have reduced the effect. The lowest quality level among all critical outcomes defines the overall level of evidence across all outcomes. PRECEPT proposes the following unified approach using GRADE for all four domains , which iFor each body of evidence related to an outcome, an initial rating of the certainty in the evidence is performed. For some of the domains, this initial rating depends on study design.Risk of bias is assessed using the appropriate QAT for the individual studies (see Step 2). A judgment about the risk of bias is made for the body of evidence, and evidence certainty can be downgraded, if necessary.Thereafter, the other GRADE criteria for downgrading the certainty in the evidence  are applied.For the domains of ’intervention studies’ and ‘risk factor studies’, upgrading of the certainty in the evidence is possible, according to the criteria introduced by GRADE. Evidence certainty should usually not be up graded after having been downgraded. It is currently unclear whether and how upgrading criteria are applicable to bodies of evidence on prevalence and diagnostics.For rating the evidence certainty of qualitative studies, PRECEPT proposes to assess limitations in the individual studies using the appropriate QATs. In addition, users of PRECEPT might consider the GRADE–CERQual  approach for assessing the confidence of evidence from reviews of qualitative research .At the end of the evidence appraisal process, a narrative evidence summary for communication of the results should be prepared. The following points should be captured: (i) the overall question, which describes the key question of the review, (ii) detailed questions, which lists the questions addressed by the review, and (iii) the volume of evidence, which describes the studies identified during the review, and (iv) the evidence statement and grading, which summarises the evidence which was identified by the review and the results of the grading process.Over the past three years, the PRECEPT team has developed this framework for the assessment of evidence in the field of infectious disease epidemiology, prevention and control. Currently, the application of the framework is being tested in other systematic reviews and projects ,24. Furt


The Contextualized Health Research Synthesis Program (CHRSP), developed in 2007 by the Newfoundland and Labrador Centre for Applied Health Research, produces contextualized knowledge syntheses for health-system decision makers. The program provides timely, relevant, and easy-to-understand scientific evidence; optimizes evidence uptake; and, most importantly, attunes research questions and evidence to the specific context in which knowledge users must apply the findings.Involves intensive partnerships with senior healthcare decision makers who propose priority research topics and participate on research teams;Considers local context both in framing the research question and in reporting the findings;Makes economical use of resources by utilizing a limited number of staff;Uses a combination of external and local experts; andWorks quickly by synthesizing high-level systematic review evidence rather than primary studies.As an integrated knowledge translation (KT) method, CHRSP:Although it was developed in the Canadian province of Newfoundland and Labrador, the CHRSP methodology is adaptable to a variety of settings with distinctive features, such as those in rural, remote, and small-town locations.Clinical and cost-effectiveness: telehealth, rural renal dialysis, point-of-care testing;Community-based health services: helping seniors age in place, supporting seniors with dementia, residential treatment centers for at-risk youth;Healthcare organization/service delivery: reducing acute-care length of stay, promoting flu vaccination among health workers, safe patient handling, age-friendly acute care; andHealth promotion: diabetes prevention, promoting healthy dietary habits.CHRSP has published 25 syntheses on priority topics chosen by the provincial healthcare system, including:These studies have been used by decision makers to inform local policy and practice decisions.By asking the health system to identify its own priorities and to participate directly in the research process, CHRSP fully integrates KT among researchers and knowledge users in healthcare in Newfoundland and Labrador. This high level of decision-maker buy-in has resulted in a corresponding level of uptake. CHRSP studies have directly informed a number of policy and practice directions, including the design of youth residential treatment centers, a provincial policy on single-use medical devices, and most recently, the opening of the province’s first Acute Care for the Elderly hospital unit. The past two decades have seen increasing efforts by both health-system knowledge users and researchers to increase the use of research-based evidence in health-policy decisions. Researchers have been striving to improve the uptake of their work by decision makers in healthcare, while decision makers have become increasingly concerned about using research evidence more effectively in the development, implementation, and evaluation of health policies and programs . SystemaResearch from Canada and other developed countries points to a variety of barriers to the uptake of evidence in health policy –7. The cThe issue of timely access to evidence is also critical—research uptake must be coordinated within organizational deadlines for policy making , 12–15. Applied health researchers aiming to support health system leaders and to see their work utilized within the health system must, therefore, seek to produce knowledge syntheses that combine several features rarely found together: their reports must be scientifically robust, accessible, timely, of direct concern to decision makers, and sensitive to the specific challenges and capacities facing those decision makers and the health systems they manage –7, 21.Mini-HTAs, produced by individual hospitals or health regions, represent one approach to addressing these barriers. They tend to support managerial decision making and to focus on drugs or technologies. They are faster to execute than full HTAs and are contextualized for the setting from which they are produced . These cThe Contextualized Health Research Synthesis Program (CHRSP) of the Newfoundland and Labrador Centre for Applied Health Research (NLCAHR) has been specifically designed to incorporate these features to address the challenges noted above. CHRSP synthesizes high-level evidence , produces reports quickly, and, most importantly, optimizes the relevance of its products to the concerns and capacities of decision makers by building an ongoing partnership with high-level provincial health-system leaders, having them generate the research questions and, above all, tailoring the presentation of findings to a carefully developed understanding of the context(s) in which these decision makers must operate.CHRSP was established in 2007 with the goal of increasing the use of health evidence by decision makers in the Newfoundland and Labrador health system. To build strong institutional support for the program, CHRSP partnered with the leaders of the province’s health system: the deputy ministers of key provincial government departments  and the Chief Executive Officers of the four provincial Regional Health Authorities.A key objective of CHRSP is to maximize the use of limited, locally available resources and expertise to synthesize existing systematic reviews on topics chosen by local knowledge users and to interpret the findings in light of local contextual factors.Evidence in Context (EiC) report which takes from 9 to 12 months to complete and follows seven steps, as outlined in Fig. CHRSP now produces several types of reports. Our gold standard is the CHRSP consults directly with the leaders of the province’s six health-system organizations to identify upcoming decisions on practices, programs, or policies (“interventions”) that would benefit from evidence-based knowledge synthesis. From the beginning, the program has invested in relationship-building with these leaders to establish strong and ongoing collaboration. The commitment to, and active participation in, the program by these deputy ministers and CEOs are central to the program’s success.In 2009, with funding from the Canadian Institutes for Health Research, CHRSP hosted a Knowledge Exchange Forum in which local health-system leaders met with leading Canadian experts in knowledge translation and evidence-based practice. The forum critically reviewed CHRSP’s processes and discussed lessons learned by similar research initiatives in other jurisdictions. The results were used to revise CHRSP methods, including its first major innovation—the introduction of “CHRSP champions,” whose role would be to serve as dedicated links between the healthcare system’s leaders and CHRSP researchers. Since that time, each of the six health-system leaders involved in CHRSP has been appointing a senior member or members of its staff to serve as the organization’s CHRSP champion(s). These champions are senior health-system managers with ready access to their system’s leader and a solid understanding of their healthcare organization and its catchment population. In each organization, they act on behalf of the leader to canvass the organization for issues for potential synthesis by CHRSP and to help the leader decide on which issues to submit for consideration.Consulting with CHRSP champions to clarify and refine the research questions;Consolidating duplicate and overlapping topics;Assessing submitted topics in terms of the feasibility of the topic for study using the CHRSP methodology and the appropriateness of the requested timelines; andConducting preliminary literature searches to assess the scope and nature of available evidence.Each year, CHRSP works with the leaders of the provincial healthcare system and their CHRSP champions to develop a slate of topics for study. While each organization has developed its own way of generating topics, in general the CHRSP champions will solicit potential topics from front-line workers, managers, administrators, decision makers, and health-system leaders  and pass the results to their organization’s leadership for review. On average, each organization submits 10–12 topics for consideration, thus generating a “long list” of potential studies. CHRSP filters this long list of submissions by:Evidence in Context Report—a comprehensive, in-depth examination of a topic, consisting of a 30–50-page report, a four-page summary, a one-page briefing note, and an online companion document for reference, and taking 9 to 12 months to complete; ORAn Rapid Evidence Report—an expedited scoping review of the evidence on a topic, consisting of a 12–20-page report which requires 30 working days to complete.A Each organization’s submission to CHRSP includes an indication of its preference for timing, which will be considered when identifying which of our reporting formats is the most appropriate for each topic:Jurisdictional Snapshot Reports—which will provide decision makers with rapid jurisdictional scans of relevant policies, programs, or practices that have been implemented elsewhere and that might be considered for use in the province. This new product is described later in this article.At the time of writing, CHRSP is in the process of piloting a new product—Website materials and a champions’ handbook that highlight key aspects of CHRSP for knowledge users. These materials are publicly available to other members of the healthcare system, to local researchers, and to the public at large. They address how to identify and prioritize issues and how to set appropriate and realistic timelines. They outline how evidence is synthesized, how it should be interpreted, and how CHRSP builds its research teams .A standardized CHRSP topic submission form that guides respondents in conceptualizing, articulating, and operationalizing a potential CHRSP question. These include framing the issue as a research question using the PICOS framework,CHRSP has developed a series of support tools and guidelines that explain, in plain language, the criteria for a feasible CHRSP topic as well as the processes involved in knowledge synthesis and contextualization. These tools help our system partners generate their long list submissions more effectively while enhancing their capacity to think of policy and program issues as potential research topics and developing a sympathetic understanding of the challenges faced by their research partners. The support tools include:These resources have been developed iteratively with our health-system partners and have provided an excellent opportunity for learning on both sides of the partnership. The use of these tools has helped our knowledge-user partners to increase their ability to think of issues as potentially researchable topics and to judge the usefulness and limits of research-based evidence for their decision-making processes. At the same time, developing and revising these resources has helped CHRSP researchers better appreciate the perspectives and needs of our health-system partners.The consolidated long list of topics is returned to the CHRSP champions and health-system leadership for them to review and rank. Each organization informs CHRSP of its top five topics. This vote yields a “short list” which forms the ballot for a second vote to determine a final list of seven topics ranked in order of priority for the next CHRSP research cycle. In addition to launching a new round of CHRSP projects, this priority-setting process gives health-system executives an opportunity to compare their challenges and priorities with those faced by their colleagues across the province. The process also provides the CHRSP research team with a privileged view of the current preoccupations of the province’s health-system leadership and an understanding of how these concerns develop over time.Evidence in Context studies. For the shorter Rapid Evidence Review studies, a streamlined approach is used that will be described below.A project team is assembled for each CHRSP project. What follows describes the composition of the teams for full Evidence in Context study, the team typically includes:An established national  subject-matter expert who, through a short-term contract, agrees to provide guidance for setting the focus and scope of the project, interpreting the results of the synthesis, and structuring the report; the expert reviews all drafts of the report and participates in the main dissemination event;A health system leader  who informs the project team about the background of the CHRSP topic and who helps to identify/delegate local health-system managers for the team;A health economist (when required) who provides support identifying, appraising, synthesizing, and interpreting evidence about economic dimensions of the topic, including likely costs, savings, and, benefits;Local co-investigators from the province’s university and healthcare system, including front-line workers, managers, decision makers, and members of community-based health organizations whose mandates have relevance to the topic. Local co-investigators are chosen from across the province in order to have appropriate geographic representation;A CHRSP staff project coordinator who is responsible for leading the project, including recruiting the expert, coordinating project team meetings and communications, developing search strategies, screening and filtering the results, data extraction and evidence synthesis, contextualization of the findings, and drafting the report;A health sciences librarian; andOther CHRSP staff and NLCAHR support personnel, including the CHRSP program director, who oversees all projects.For each The project team works together throughout the CHRSP process—from the beginning to the end of each project. A CHRSP project officially starts with the first project team meeting at which time the focus and scope of the project, as well as any other parameters, are set by consensus. Team members are kept informed about research progress and preliminary results; they are consulted when challenges arise, and they are asked to review the report at three stages: evidence synthesis, contextualization, and final draft. The project team attends end-of-project dissemination events and members also assist within their organizations, wherever possible, in knowledge translation efforts to optimize the uptake and impact of the studies.Each CHRSP project also involves context advisers and an external reviewer. Context advisors help the project coordinator to identify and explore possible contextual factors that could have an impact on the implementation of the findings. Local project team members may also serve as context consultants to provide input and data for the contextualization exercise. The external reviewer is a national/international subject matter expert who has not authored any of the evidence included in the synthesis. The external reviewer is also invited to participate in the end-of-project dissemination event.Rapid Evidence Reports, the teams tend to be more compact, as will be explained below.For our Once a CHRSP project team has established the focus, scope, and eligibility criteria for studies , CHRSP researchers search for systematic reviews  in a limited number of the most relevant periodical indices and databases, including sources of gray literature.Evidence in Context reports in a timeframe that still qualifies them as “rapid reviews,” according to most definitions of the term [CHRSP’s focus on systematic reviews is an approach that allows us to assess a large body of evidence relatively quickly and to deliver results more rapidly than most knowledge synthesis programs, without compromising scientific rigor. This makes the approach primarily meta-synthetic, producing systematic reviews of reviews or “overviews” as the Cochrane Collaboration calls them . This scthe term –32.Evidence in Context studies, the methodological quality of each systematic review being considered is assessed using the AMSTAR tool (Assessment of Multiple Systematic Reviews) [In our Reviews) , 34. EacReviews) . CriticaFrom the start, the CHRSP research team has experimented with different methods for rating the strength of the overall body of evidence for each particular intervention, working with decision makers and methodology experts to develop a system that is reliable and that has both internal and external validity. In 2015, based on criteria derived from the literature, and drawing on consultations, experience, and feedback from knowledge-user partners, CHRSP developed an Evidence Rating System to provide a robust and consistent approach.The first step in the CHRSP ERS is to critically appraise eligible systematic reviews. AMSTAR scores are divided into three categories: high quality (70% or higher), moderate quality (between 40 and 69%), and low quality (less than 40%). Systematic reviews of low quality are excluded from the synthesis.The second step is data extraction: the contents of retained systematic reviews are coded using the PICOS framework: Population, Intervention, Control, Outcome, and Setting. Only findings with matching PICOS parameters can be synthesized across multiple systematic reviews . Individual PICOS-defined findings are also coded as being quantitative or qualitative, statistically significant or not, and as favoring the intervention being studied or the control group.The third step in the CHRSP ERS is an enumeration of the primary literature that is covered by the systematic reviews for each PICOS-defined finding; when different systematic reviews synthesize the findings of the same primary research studies, this can result in the same evidence being “counted” more than once. When considering the strength of the body of evidence for a particular finding, this repeat counting may result in overestimating the strength of a body of evidence. In order to mitigate this possibility, the CHRSP ERS indexes the primary studies included within a given systematic review for each PICOS-defined finding. This index is then used to determine how many unique primary studies have been combined in the various systematic reviews addressing each PICOS-defined finding.The methodological quality of the systematic reviews;The number of unique primary research studies that underpin the findings; andThe consistency of the review evidence The CHRSP ERS combines the results of the above steps to establish a measure for the strength of the body of evidence for each PICOS-defined finding. This measure considers:The resulting evaluation of the body of evidence is reported using a five-point scale: Very Strong, Strong, Moderate, Weak, and Very Weak. The ERS is conservative by design, and our approach discounts any findings that are Weak or Very Weak.The result is that a CHRSP meta-synthesis is highly specific , takes into account the methodological strengths and weaknesses of the systematic review evidence, assesses the true size of the evidence base , and emphasizes convergent research findings. Our knowledge users have indicated that this rating system makes intuitive sense to them and that its use in our reports provides them with a significant degree of confidence in the findings presented.What works?” and “What will work here?” This approach acknowledges the “need for a two-pronged analysis, focused at once on the effects of the policy being studied and on the issues surrounding its implementation” [From the outset, each CHRSP project addresses two fundamental questions: “ntation” .hat Works?” is answered by the synthesis of research-based evidence. The second question, “What will work here?” requires an assessment of local contextual factors and their implications. A given contextual factor may have an impact on:health equity of an intervention, that is, the differential effectiveness either positive or negative, of an intervention for different groups in a population.The feasibility of implementing an intervention, including costs, infrastructure, the recruitment and/or training of health human resources, and patient volumes.The acceptability of an intervention from the perspective of relevant stakeholders, including decision makers, health service providers, political leaders, patients, caregivers, and families.The The first question, “WHealth equity, feasibility, and acceptability are all critical considerations for decision makers . SignifiEvidence in Context studies, if appropriate evidence is available for any particular contextual variable, CHRSP assesses its likely impact for our decision-making partners. If not, our reports frame the contextual factor as a question for consideration by decision makers. For our Rapid Evidence Reports, we compile a list of potential contextual factors that may influence how the intervention or interventions being considered are likely to work in our specific context but we do not analyze them in any detail.A distinctive feature of the CHRSP approach is that we address the issue of context explicitly and consistently in our reports. CHRSP interprets synthesis findings in light of local characteristics, capacities, and conditions that will have an impact on the implementation of health policy by the health decision makers of Newfoundland and Labrador and its four regional health authorities. We interview key informants from across the province including front-line healthcare workers, administrative data holders, community organizations, union representatives, managers, senior decision makers, patients, informal caregivers, and other appropriate stakeholders. Project team members are usually interviewed first and they suggest additional interviewees. Our consultations yield a series of potential contextual factors. For our CHRSP uses a framework that groups contextual factors thematically, as indicated in Table Evidence in Context project is a 30 to 50-page report. It includes brief sections on background, methods, and search results. The main body of the report focuses on the synthesis, the contextualization, and an analysis of the implications of these for the province’s decision makers.relevant evidence synthesis findings and reflect the state of the available research. These key findings are then considered in terms of the contextualization results to come up with a list of the “implications for decision makers.”The product of a CHRSP For strategic reasons, CHRSP has opted to use the term “implications” rather than the more common “recommendations.” When we began our work, our Health System Partners were new to this type of decision support and we felt, correctly as it turned out, that they would be wary of “recommendations” that would appear to require action on their part. Using the term “implications” acknowledges that research-based evidence is only one of multiple types of input that health-system decision makers need to consider. CHRSP intended its reports to say: “here are things you should think about when considering this issue” rather than “this is the option you should choose when making a decision about this issue.”Key messages and implications for decision makers are reviewed by the project team and by an external reviewer. The external reviewer is an acknowledged expert on the subject matter being examined who has been screened for potential conflicts of interest.Evidence in Context reports are published in three plain-language formats: no specialized expertise is required to understand the studies. The formats include a report of 30–50 pages (10–15 pages for Rapid Evidence Reviews), a four-page summary, and a one-page briefing note for decision makers. Other online documents and presentations, press releases, and companion documents are also produced, as applicable.The final report, the executive summary, and the one-page briefing note are delivered to the six Health System Partners for an embargo period of ten business days in order to give them time to prepare for its public release. All CHRSP project materials are then posted on the NLCAHR website and disseminated by direct e-mail to health system, community, and research groups across the province.In addition, a dissemination event is organized to present the study to an invited audience. Each event is tailored to the individual project and ranges from a 2-h internally organized and supported meeting to day-long co-partnered and externally funded forums. The project’s subject-matter expert attends in person or via webinar and the external reviewer is invited (but not required) to attend. In addition to the project team, health-system administrators, managers, and front-line workers involved with the topic under study are invited to attend, as are academic researchers, students, and relevant patient/caregiver and community groups. Participants are encouraged to share invitations to open meetings to maximize impact beyond the immediate communication networks available to CHRSP.In some cases, a separate dissemination event is organized that is intended exclusively for a particular sub-group; in other cases, we have organized a small group in camera session involving key decision makers, the subject expert, and CHRSP researchers. The key objective of these various dissemination events is to stimulate the use of CHRSP products and methods in decision making.Evidence in Context and Rapid Evidence reports are published on the NLCAHR website and placed in the Memorial University Libraries Research Repository, in Canada’s National Library and Archives and in Memorial University’s Yaffle Research Repository. They are also posted on the internal listservs of all Regional Health Authorities as well as in the Newfoundland and Labrador Medical Association’s Nexus Newsletter and the newsletter of the Association of Registered Nurses of Newfoundland and Labrador. Once a report has been disseminated, the NLCAHR team will solicit feedback from stakeholders and will host further meetings and events, as required, to help facilitate optimal uptake of the research results.Once sufficient time has elapsed between publication and the opportunity to apply the results (this depends on the topic and the complexity of the interventions under consideration), CHRSP solicits direct feedback from stakeholders and decision makers to evaluate uptake by asking how the reports have been used and to identify areas for potential improvement.Now that the program has matured, the team is also beginning to update syntheses that are more than 5 years old in order to ensure that the original findings remain both current and relevant.Rapid Evidence Reports to provide an expedited research-based decision-support product for decision makers. This report takes only 30 working days to produce after a consensus has been achieved among health-system partners and CHRSP researchers on the scope and wording of the research question. This expedited approach is chosen for topics for which our health-system partners have requested quick turnaround of the evidence to support a pending decision.In 2012, CHRSP introduced Rapid Evidence Reports provide a brief overview of the evidence in the systematic review literature and in primary studies not captured in the review literature. Each report includes a description of the scope of the research-based evidence, the strengths and gaps in the literature, and the principal areas of consensus, disagreement, and uncertainty in the research on the topic in question. Rapid Evidence Reports are carried out under the supervision and guidance of a key informant from the provincial health system and an external subject matter expert. Rapid Evidence Reports are not intended to provide an exhaustive synthesis of all the available literature, or to develop a systematic assessment of the methodological quality of the available research, or to thoroughly contextualize the findings of our literature scan. Rather, they provide decision makers with a reliable indication of the available research-based evidence and the core findings on the topic. If our key knowledge users find the contents of a given Rapid Evidence Reports suggestive but insufficiently comprehensive or authoritative, they can request that we carry out a full Evidence in Context study on the topic.CHRSP also uses an integrated knowledge translation approach in the development of its methodology. From the beginning, the program has worked closely with knowledge users to identify opportunities for improvement. At the time of writing, CHRSP is developing two new innovations:Starting in 2016, CHRSP has been consulting and planning to expand the scope of its integrated KT approach by developing a method for engaging patients and caregivers in the process. Adding patient/caregiver engagement will enhance CHRSP’s contribution to evidence-informed decision support by bringing people directly affected by healthcare policies and practices into the process, potentially broadening the scope of topics chosen and enhancing the range and the quality of the contextualization work involved. Patient and caregiver participants will work with CHRSP to make the design and dissemination of reports more accessible and relevant to lay audiences with diverse backgrounds and education levels.Jurisdictional Snapshot Reports to provide decision makers with an overview of healthcare practices, programs, and policies from other jurisdictions. Jurisdictions are selected in consultation with decision makers and may include other provinces or regional health authorities in Canada, as well as other countries. These reports are intended to inform decision makers about the health policy landscape across jurisdictions with a focus not on research studies but on programs, tools, and other policy initiatives. Where possible, Jurisdictional Snapshot Reports will also clarify whether research-based evidence has been used to develop and/or evaluate the programs, tools, and policy initiatives involved. Jurisdictional Snapshot Reports may also help inform topic selection for subsequent CHRSP products, such as Evidence in Context or Rapid Evidence Reports.In 2016–2017, CHRSP introduced Evidence in Context reports, seven Rapid Evidence Reports, and one Jurisdictional Snapshot Report. These were based on a selection from approximately 125 topic submissions by the health system. At time of writing, our research team is at work on two Evidence in Context projects and one Rapid Evidence Report.As indicated in Table Health System Leaders, CHRSP champions, project team members, and context advisors;All senior managers and administrators who were involved in the decision(s) related to the CHRSP project topic (including implementation);Community members who were involved in the project; andEveryone who attended a dissemination event.CHRSP recognizes that it takes time for its products to be taken up by the health system. Accordingly, we wait approximately 3 years before seeking feedback about whether and how our reports have been used. At that point, we solicit input from the following:How was this report useful/relevant to your organization? Please tell us briefly how the report was considered or used in policy or practice decisions. Was it distributed within your unit, discussed at meetings, referenced in any briefs, incorporated into any decisions, added to your research library, etc.?If the report was not useful or relevant, please tell us why not and how it might have been improved.CHRSP asks these participants two broad questions:To date, CHRSP has received feedback on 12 of its published studies. Responses have been generally positive and constructive , our health-system partners have become increasingly eager for rapid turnaround in decision support. At the same time, their questions to CHRSP have become steadily more complex and multi-faceted. Further confounding the challenge of growing complexity, budgetary constraints have limited the ability of the NLCAHR to increase or even maintain its staff complement. As already noted, we have responded to time pressure by developing new, computer-aided processes for assessing the weight of evidence in our reports and have added two rapid response products, We have also sought, with varying success, to simplify complex questions by making a persistent effort to focus on the refinement of the research question during the initial stages of each study in an effort to keep the parameters of each study as well-delimited as possible. We have also eliminated one key CHRSP position, a full-time program manager, and transferred her responsibilities to the director of the NLCAHR who has taken on the role of program director for CHRSP.Given the multiple contexts of the province , identifying “contextualization factors” can be a complicated task. The process of contextualization is further limited by the fact that our researcher teams cannot possibly locate and interview every appropriate contextual advisor from within such diverse communities. The fact that we must rely on a representative sample of informants means that placing the evidence in context may, or may not, reflect every contextual reality, even in such a small jurisdiction as Newfoundland and Labrador.Newfoundland and Labrador differs in a variety of important ways from most other Canadian provinces and from the national and international jurisdictions where much of the available health research has been conducted. Decision makers in this province are keenly aware that the findings of much of this literature are, therefore, of limited or questionable applicability within the local context and are highly supportive of the local production of context-sensitive syntheses.The province’s health system is comparatively compact and involves a small number of key organizations whose leaders know one another and who are accustomed to working together. Securing their ongoing and active participation in CHRSP has thus proved easier than it might be in larger, more diverse jurisdictions.For a variety of reasons, this province’s health-system organizations have increasingly sought evidence to support their decisions and have striven to become learning organizations that are now skilled at generating “pull” for contextualized knowledge synthesis products.CHRSP has worked well in Newfoundland and Labrador for a variety of reasons:Even in places where these factors do not pertain or are of lesser import, the CHRSP approach can still be useful. It appears particularly appropriate for jurisdictions (or parts of jurisdictions) in which contextual considerations are clearly necessary, for example in rural, remote, and northern regions in most Canadian provinces. The CHRSP methodology could also be used to contextualize a single set of findings for more than one context at a time, thus making it of greater potential interest to decision makers –20. It iThe program is already expanding. Funding from the Manitoba Workers Compensation Board allowed the NLCAHR to collaborate with the research synthesis team at Toronto’s Institute for Work & Health on a project that examined the adaptation of the CHRSP methodology, in conjunction with the Institute’s own research synthesis program, for context-sensitive use in the field of occupational health and safety. Based on the report produced by that project, an Occupational Health and Safety model of the program is now being developed in Manitoba. In addition, working with a team of researchers and knowledge users in Northern Ontario and Northern British Columbia, CHRSP is now developing a proposal for a research program of integrated knowledge translation and contextualized evidence syntheses to support health decisions in various rural, northern, and remote regions of Canada.