Antimicrobial resistance (AMR) is a prominent threat to public health. Although many guidelines have been developed over the years to tackle this issue, their impact on health care practice varies. Guidelines are often based on evidence from clinical trials, but these have limitations, particularly in the breadth and generalisability of the evidence and evaluation of the guidelines’ uptake. The aim of this study was to investigate how national and local guidelines for managing common infections are developed and explore guideline committee members’ opinions about using real-world observational evidence in the guideline development process.Six semi-structured interviews were completed with participants who had contributed to the development or adjustment of national or local guidelines on antimicrobial prescribing over the past 5 years ). Interviews were audio recorded and transcribed verbatim. Data was analysed thematically. This also included review of policy documents including guidelines, reports and minutes of guideline development group meetings that were available to the public.Three key themes emerged through our analysis: perception versus actual guideline development process, using other types of evidence in the guideline development process, and guidelines are not enough to change antibiotic prescribing behaviour. In addition, our study was able to provide some insight between the documented and actual guideline development process within NICE, as well as how local guidelines are developed, including differences in types of evidence used.This case study indicates that there is the potential for a wider range of evidence to be included as part of the guideline development process at both the national and local levels. There was a general agreement that the inclusion of observational data would be appropriate in enhancing the guideline development process, as well providing a potential solution for monitoring guideline use in clinical practice, and improving the implementation of treatment guidelines in primary care. Evidence-based medicine (EBM) has been one of the defining features of health care policy and practice since the 1990s. The evidence base for clinical decision-making is built upon critical assessment of relevant literature and assessment of the strength of scientific evidence to inform clinical practice along with clinical reasoning . ClinicaSince 1999, National Institute for Health and Care Excellence (NICE) role is to improve outcomes for people using the National Health Service (NHS) and other public health and social care services. NICE do this through: producing evidence-based guidance and advice for health, public health and social care practitioners; developing quality standards and performance metrics for those providing and commissioning health, public health and social care services; and, providing a range of information services for commissioners, practitioners and managers across the spectrum of health and social care .Although many guidelines have been developed over the years, their impact on health care practice varies . The accThe usability and implementability of guidelines can be improved by broadening the evidence base, and by bringing the evidence that is used and clinical practice closer together . In idenUse of evidence that identifies a patient’s risk of developing a disease is highly relevant for identifying patient groups that need to be targeted with treatment. This type of data analysis, however, cannot be captured in clinical trial-derived evidence, as this needs incorporation of data about variability within a population. In guidelines on the prevention of cardiovascular events, evidence on identifying patients’ risk of a cardiovascular event has been incorporated , 15. HowIn primary care, the prescription of antibiotics leads to discomfort in decision-making , due to Prescribing behaviours appear to be based on symptoms rather than on an assessment of the risks to the patient, as the latter is often unclear to GPs . IncorpoThe aim of this study was to investigate how national and local guidelines for managing common infections are developed and explore guideline committee members’ opinions about using real-world observational evidence in the guideline development process.This case study is based on interviews with guideline developers involved in the creation of national and/or local guidelines for antimicrobial prescribing, as well as on a documentary analysis of existing national and local guideline information. Data were collected between September 2018 and March 2019.Research team members from the Health E-Research Centre at the University of Manchester conducted six semi-structured face-to-face interviews were conducted with participants who had contributed to the development or adjustment of national or local guidelines on antimicrobial prescribing over the past 5 years. The national guidelines were developed by the NICE in England. Interviews focused on the types of evidence used in the guideline development process and potential challenges in changing prescribing behaviour for treating common infections. An interview guide was developed to explore and capture guideline developers’ experiences and opinions regarding the how evidence was chosen and the challenges of incorporating other forms of evidence within the development process.The interview guide also included three vignettes that were developed by the research team: the first vignette presented two published research studies. Both studies gave information about two different types of scoring tools for diagnosing sore throat in clinical practice; guidelines recommended the clinical use of these tools , 26. TheInterviews lasted between 45 to 90 min. They were audio-recorded and transcribed verbatim, with respondents then sent a copy of the transcript to review and make any necessary amendments. In addition to the interviews, policy documents including guidelines, reports and minutes of guideline development group meetings, were collected from NICE and the Greater Manchester Clinical Standards Board. Where scientific evidence was cited in these guidelines, this was also collected. All information was drawn from publically available data.Thematic analysis was used to analyse interview and document data . ThematiParticipation in the study was based on informed consent and was voluntary. All participants received a participant information sheet and were asked to sign a consent form prior to starting the interview. This study received ethics approval from the University of Manchester Research Ethics committee, Manchester, U.K (Ref: 2018–4682-6712).Perception versus actual guideline development process, 2) Using other types of evidence in the guideline development process, and 3) Guidelines are not enough for changing antibiotic prescribing behaviour. Quotes from participants are used to exemplify and clarify themes. Due to under-recruitment to the study and to further ensure participant anonymity, no further information regarding participant characteristics will be given.Three key themes emerged through our study: 1) “They [NICE] are just concentrating on systematic reviews. And, especially with something like this, the antibiotic guidance, which is so broad, there’s a huge quantity of experience out there that they could ask about for each of the different guidance. I’ve said each time, for each one, they ought to get other people involved.”Whilst publicly available documents gave a general overview of the process involved for developing national guidelines, the interviews provided a more detailed description of the methods used and who was involved throughout the process. In particular, a difference was noted in some areas of the guideline development process, with the majority of the work undertaken by staff at NICE and limited input from committee members and other stakeholders. Several participants described how suggestions made to NICE, such as including expert opinions and a wider range of evidence were considered but not acted upon, preferring to focus efforts on the literature. Whilst all participants reflected that they understood why this was the process, several felt that there was a lot of value in speaking with experts with extensive clinical knowledge and practice.The guideline development process for developing guidelines at the local level was not well documented, with documentation published online for the year 2017 to 2018 only. As a result, interviews with participants who contributed to local guideline development provided useful insight in this area. Participants were able to describe a clear difference between national and local guideline development. Locally, an emphasis was placed on ensuring that local contexts and issues were built into the guidelines whilst cross-referencing national guidelines. Participants involved in the development of local guidelines discussed how local practitioners were encouraged to raise issues they had encountered when treating patients to ensure that these could be addressed in updated guidelines to ensure practitioners felt confident in their practice. As a result, guidelines are updated to reflect these issues and any other localised/national changes every quarter.All participants felt that a much wider range of evidence should be used in national guideline development, rather than relying on systematic literature reviews and RCTs that met strict criteria as set out by NICE. Several participants commented that some of the studies reviewed in the national guideline process were difficult to generalise to the U.K population because they were either a systematic literature review or the RCT was conducted in a very specific type of population or in a different country.“They have a very strict criteria about what guidance they use, and it’s on a hierarchical basis. So, they, basically, will use a systematic review, if that’s available of the area, and then if there aren’t any systematic reviews they will go to RCTs. But, they would not go to RCTs if there was a good systematic review. And then, they really very rarely go anywhere else for any other information. I think they should use qualitative evidence, as well, because I think that’s really useful and it gives you more valuable insight into how things work in real life, which you don’t get with a RCT or systematic review.”Whilst some participants were able to reflect as to the rational for why NICE restrict their studies to systematic literature reviews and RCTs, all participants noted that other studies that provided evidence for best antibiotic prescribing behaviour using other research methods were being excluded . Many felt that given the current urgency to address antibiotic resistance, there was the potential that other types of studies, such as a qualitative study, could provide useful insight into prescribing behaviours that could then be considered in guideline development.The interview included asking participants a series of questions based on three vignettes. Vignette one presented two published research studies. Both studies gave information about two different types of scoring tools for sore throat. Nearly all participants were familiar with the scoring tools presented which prompted discussions around how such scoring tools can be useful in clinical practice to support decision making. Despite the limitations of the first scoring tool, the majority of participants felt that having some form of tool to use in clinical practice was better than having none at all. Some participants who were familiar with the second study noted that whilst this was a more appropriate study to consider as it was conducted in the U.K compared to the first scoring tool that was developed in the U.S; this study did not meet the NICE requirements for inclusion in the guideline development process. Despite its limitations, some felt it was a missed opportunity for a recently U.K developed scoring tool to be included in the NICE guideline development process.Moving through the first vignette, participants tended to focus on the table of information which presented the predictive value of bacterial clinical scoring methods for complication, describing how useful it was to see the data presented for each study side by side. Participants indicated that presenting all results in this manner during the guideline development process would be more useful.“It’s really interesting to see the study compared with the population. Reading the study, I felt it was as representative as you could get in general practice, but seeing it here with CPRD data is very beneficial because RCTs tend to have a particular type of population which might not represent Wigan. It just makes you feel a bit more confident in the published stuff.”Vignette two presented compared UK primary care CPRD data on antibiotic prescribing for acute sore throat to a recently published study that compared different antibiotic prescribing strategies. Nearly all participants noted how interesting it was to see the research study data presented side by side with population data. In particular, being able to see the study data with the corresponding CPRD data made participants feel more confident in interpreting the results of published studies, as well as providing additional evidence that could assist in determining whether specific interventions could be generalised to the U.K population. All participants felt that having study results presented alongside the relevant population dataset would be beneficial in future guideline development.“Things like if you’re looking at something like antimicrobial prescribing, it would be useful to see when there were bad years for flu, like swine flu…when those happened and what the knock-on impact of that was, you know, other outbreak kind of related stuff. When there were…I don’t know whether…if there’s a really bad winter, does that make a bigger impact on antibiotic use in terms of more transmission of…you know, is there more transmission of upper respiratory tract infections, things like that. I know obviously the flu does have an impact, so that would be useful to see in the data.”In discussing the third vignette, most participants found the third figure within the vignette to be the most useful. The third figure shows average prescribing rates for each year with yellow lines indicating the changing/updating of guidelines and red lines indicated the three main changes in the rate of prescribing. Participants noted how the combination of information was complementary, with several suggesting the addition of key flu outbreaks (such as Swine Flu) labelled would make this figure even more valuable, particularly in monitoring antibiotic prescribing over time using local data.Opinions about the usefulness and effectiveness of national guidelines for antibiotic prescribing varied. All participants stated that they were aware of current governmental policies and guidelines aimed at reducing antibiotic prescribing. However, participants involved in some form of clinical practice admitted to rarely using national guidelines in practice, relying more on the expertise of other clinical professionals, their own clinical knowledge or a combination of locally produced guidelines and expert knowledge. Most participants stated there were other guidelines for antibiotic prescribing that were more useful to prescribing practitioners on a day-to-day basis. Participants commonly referred to the guidelines produced by Public Health England and locally produced guidelines when asked what other types of guidelines either they themselves used or their colleagues used in practice. In describing what they felt was more useful about these other guidelines; participants noted the usefulness of having information presented in a tabulated format on the different types of antibiotics for a range of health conditions.“They’re quite strange targets really because they’re arbitrary, and based on an England mean from 2013. It doesn’t account for anything that’s issued from walk-in centres, out-of-hours places that use different budgets and private prescriptions, which in affluent areas, GPs might offer both. So if you’re not going to make it, it’s not an incentive to reduce your prescribing behaviour. Plus there’s no penalty and the financial incentive isn’t worth it if you’re already struggling to meet that target. What we should do is have more localised targets that address the contexts we work in…that way if becomes more like comparing apples with apples, rather than apples with pears.”All participants noted that changes and updates to NICE antibiotic prescribing does not necessarily result in reductions in antibiotic prescribing. There was a mixed response as to whether GP’s were monitored on their prescribing behaviour. Regional variations were noted in how antibiotic prescribing was monitored. It was noted that some local Clinical Commissioning Groups  were more proactive than others, whilst other CCGs did not have the capacity to provide additional support to GPs in reducing prescribing behaviours. Financial incentives were deemed not to work as they were too negligible to make a difference to prescribing behaviours, particularly those who work in high prescribing areas. One suggestion was to have localised targets that were attainable, rather than a national target.Another issue that was raised by participants was that national guidelines do not take into account the localised contexts, such as areas with a higher rate of elderly patients, which often results in higher rates of prescribing. Many felt that as a result of this, many GPs and other forms of prescribers simply did not bother to look at the NICE guidelines, preferring to use more localised versions that provided recommendations specific to the populations that they worked with. In addition, several participants highlighted that prescribing data did not necessarily represent prescribing behaviour, particularly if GPs offer private prescriptions.Several participants reflected on the pressures that GPs face in seeing a patient within a 10-min appointment slot. It was suggested that providing GPs with additional support (such as mentoring), regular meetings to discuss difficult cases and learning from others in how they have able to tackle these types of cases, as well as a forum to discuss guidelines/new regulations, were key to reducing antibiotic prescribing in GPs. At the same time, participants who discussed these issues also noted the importance of sharing the responsibility with the public, rather than it being the sole responsibility of the GP or healthcare practitioner. Suggestions were made for the Government to provide more public health education campaigns to the general public, with targeted approaches to population groups who may insist on being given antibiotics when there is not a medical need.Although many guidelines have been developed over the years, particularly on managing common infections, their impact on health care practice varies. Treatment guidelines are often based on assessments of evidence from clinical trials or systematic literature reviews, but these have limitations, particularly in representing the whole population of patients in primary care. This suggests that there is a need for bringing more clinically relevant evidence into guidelines to improve their uptake in clinical practice.This case study provides an in-depth look at the guideline development process within treating common infections, the challenges of reducing antibiotic prescribing that go beyond the guideline process and how other forms of evidence could be utilised in the guideline development process. Three key themes emerged through our study: 1) Perception versus actual guideline development process, 2) Using other types of evidence in the guideline development process, and 3) Guidelines are not enough for changing antibiotic prescribing behaviour. In addition, our study was able to provide some insight between the documented and actual guideline development process within NICE, as well as how local guidelines are developed.Whilst discussing other types of evidence, such as the availability of population data, all participants stated that they would welcome additional evidence that provides them with relevant population data that could be presented alongside the results of published studies. In addition, interview participants were able to reflect on the constraints imposed onto NICE in the guideline development process by other stakeholders within government.NICE guidelines make evidence-based recommendations on a wide range of topics, from preventing and managing specific conditions, improving health and managing medicines in different settings, to providing social care to adults and children, and planning broader services and interventions to improve the health of communities . HoweverIt has been recommended that in order to be used, clinical guidelines need to be integrated as part of other healthcare quality improvement processes , 28. ThiOur results are based on a single case study of common infection guideline development with an emphasis on antibiotic prescribing. There might be other aspects that need to be considered in other contexts and other ways of achieving reduced levels of antibiotic prescribing in primary care clinical practice. On a more practical level, regular meetings between local and national guideline developers would provide insight into the different patient groups across the U.K and the difficulties of utilising national guidelines in a range of population settings. In addition, further consideration could be given to utilising complementary local data to monitor the effectiveness of local interventions, which could then be considered at the local guideline level.This case study has provided insight into the processes for antimicrobial guideline development at the national and local levels in the U.K, particularly in the types of evidence that are currently used. The study has indicated that guideline developers could benefit from including a wider range of published research studies, as well as the input from experts during the early stages of developing or updating antibiotic prescribing guidelines. In addition, the inclusion of population data was found to be a source of evidence that many guideline developers would like to see utilised as part of the wider range of evidence presented to them. However, there will be other aspects that need to be considered in using other forms of evidence within the process of developing guidelines, as well as in how these are used for monitoring purposes. Furthermore, Government and guideline developers will also need to consider focusing on addressing the wider issues that can impact antibiotic prescribing and ensure that the necessary resources and support are put in place for both GPs and the general public to achieve rational antibiotic prescribing.


Strategies to improve the effectiveness and quality of health and care have predominantly emphasised the implementation of new research and evidence into service organisation and delivery. A parallel, but less understood issue is how clinicians and service leaders stop existing practices and interventions that are no longer evidence based, where new evidence supersedes old evidence, or interventions are replaced with those that are more cost effective. The aim of this evidence synthesis is to produce meaningful programme theory and practical guidance for policy makers, managers and clinicians to understand how and why de-implementation processes and procedures can work.The synthesis will examine the attributes or characteristics that constitute the concept of de-implementation. The research team will then draw on the principles of realist inquiry to provide an explanatory account of how, in what context and for whom to explain the successful processes and impacts of de-implementation. The review will be conducted in four phases over 18 months. Phase 1: develop a framework to map the preliminary programme theories through an initial scoping of the literature and consultation with key stakeholders. Phase 2: systematic searches of the evidence to develop the theories identified in phase 1. Phase 3: validation and refinement of programme theories through stakeholder interviews. Phase 4: formulating actionable recommendations for managers, commissioners and service leaders about what works through different approaches to de-implementation.This evidence synthesis will address gaps in knowledge about de-implementation across health and care services and ensure that guidance about strategies and approaches accounts for contextual factors, which may be operating at different organisational and decision-making levels. Through the development of the programme theory, which explains what works, how and under which circumstances, findings from the evidence synthesis will support managers and service leaders to make measured decisions about de-implementation.CRD42017081030PROSPERO  In the current climate, health and care systems are under pressure to demonstrate that actions to ensure efficient use of resources do not compromise the quality of the care provided. However, practices that have limited evidence for their efficacy continue to be performed, for example, imaging for low back pain, antibiotics for sinusitis, and antipsychotics for dementia . In the De-implementation is one of many terms used in the literature to describe the removal or reduction of costly or potentially hazardous approaches to care. A scoping review  found 43There are some individual studies that have specifically paid attention to de-implementation. For example, a study that explored the literature and analysed data about implementation and de-implementation from two studies among Dutch orthopaedic surgeons concluded that further research is required to understand the required leadership and champion characteristics that can accelerate the process of de-implementation . The autIn the drive to increase health service efficiency, current health and care policy and guidance highlight the importance of using only clinical and other practices which are effective and valuable and reflect the high quality of care for patients. Consequently, interest is developing in evidence to inform health professionals about de-implementation –9. ThereRealist synthesis or review is a systematic, iterative, theory-driven approach that draws on a heterogeneous evidence base to establish what works, how, in what context and for whom . Unlike The principal aim of this realist synthesis is to generate evidence and theory to guide managers and service leaders to effectively de-implement practices across health and care services. The output generated will be in the form of a programme theory, populated with evidence that describes what works about de-implementation, how and the contextual factors that influence their impact on outcomes. We will collaborate with managers and health and care leaders to translate this programme theory into actionable recommendations for different organisations, populated with practical examples from the evidence.To identify and map the range of different de-implementation approaches and/or strategies currently being utilised across health and care and in the fields of behavioural psychology, behavioural economics, organisational sociology, crime and deviance, paying attention to ways in which they are assumed to workProduce a typology of de-implementation types, processes, contextsExamine and understand the range of impacts of these approaches and/or strategies across different settings, paying attention to contextual conditions which influence how they workProduce an evidence-based, realist programme theory that explains the successful processes and impacts of de-implementationExplore, through stakeholder engagement, the decision-making processes associated with de-implementationProvide recommendations about ways in which different approaches and/or strategies can help managers and service leaders plan and prioritise de-implementation in a systematic and efficient mannerStimulate a wider debate about avoiding and stopping services that are considered wasteful, of low value and non-efficient, for future provisionThe main aims are:Drawing on our previous experience of conducting realist syntheses (e.g. NIHR HS&DR 12/129/32: HS&DR 14/194/20), and publication guidance for realist synthesis [The protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on November 17, 2017 (registration number CRD42017081030). The protocol has been reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis Protocols (PRISMA-P) 2015 checklist [The first phase involves the development of the initial programme theory, a practical framework which represents what is thought to work, how and under which conditions, about successful de-implementation practices that are of low value and not evidence-based. This work will draw on an extensive scoping review of the existing literature and theories. We will incorporate consultation with stakeholders  and patient and public involvement (PPI) representatives through workshops. The purpose of the workshops is to develop an understanding of the complexities of the contexts in which de-implementation efforts are situated. Through interactive workshops, participants’ thoughts and ideas about de-implementation can be made visible, which serves as a basis for discussion to find out the contextual influences within systems that might impact on different de-implementation strategies.To build on the theory-building work, we will additionally conduct telephone interviews with clinicians, managers and organisational leaders. A semi-structured interview schedule will be developed using the emerging theories from the workshops and will focus on understanding the decision-making processes and influences on participants’ judgements during de-implementation processes. Using the principles of framework analysis , we willTogether, these activities will contribute to the development of an initial programme theory that provides an early explanation of the complexity of de-implementation and the contextual conditions that underpin the process. The initial theoretical framework will guide the literature mining and consultation work with stakeholders. To initiate the process, we will draw on different theories and approaches, some of which may traditionally be used in investigations of ‘positive’ implementation. Elements of these theories will provide the conceptual platform for commencing phase 1 of this synthesis. These may include habit formation, unlearning, decision-making theories, nudge and behaviour change, organisational sociology and diffusion of innovation. It is the initial programme theory that is tested in the following phases.In phase 2, we will draw on evidence to test and refine the programme theory. In the spirit of the realist approach, priorities are always focused on theory development rather the internal validity of the evidence, and a diversity of evidence provides more opportunities for mining the evidence and seeking an in-depth explanation. As some evidence about de-implementation is likely to be unpublished, we will use our stakeholder group and wider networks across different organisations to identify a range of possible evidence sources. Inclusion of grey literature, e.g. in the form of project reports, case studies, campaigns, policy evaluation, film clips and secondary data, will ensure that important evidence from a range of perspectives is included. The review process includes screening literature for evidence of relevance and extracting and charting key data around the programme theory.Our search will not be limited to health and care, to ensure classic studies on social group dynamics and behaviour change can be included, but specific literature and evidence published in the last 20 years will capture the emergence of the ‘what matters is what works’ philosophy  and inteWe will determine keywords starting with the 43 terms listed by Niven et al.  in a scoWe will also conduct Internet-based searches for grey literature, such as de-implementation reports relating to national and local initiatives; where possible, evaluative information about these initiatives held in the public domain will be requested. We will also use citation tracking and snowballing techniques and draw on the expertise of the project team, steering group, other key researchers and service leaders to ensure we have not missed evidence that might be relevant, but not visible through traditional and hand searching methods.https://www.evidence.nhs.uk/) and Open Grey (http://www.opengrey.eu/).We will draw on the experience and networks of our project team in the identification of interventions and programmes that describe de-implementation. We will scope Internet resources and identify repositories of research and relevant reports, to seek relevant guidance, for example, NHS evidence . Evidence will not be excluded, unless it does not relate to the programme theory areas. The programme theories being ‘tested’ through the review are made visible through bespoke data extraction forms . The tesOrganise the extracted information into data tables representing the different bodies of evidence that inform the programme theory areas, and the range of potential impacts from different de-implementation strategies and approaches .Undertake an abductive and retroductive approach  to look Link the demi-regularities to form a refined programme theory, consisting of CMOs which are explanatory statements reflecting the complexity of de-implementation strategies and processes. This explanatory account will provide an explanation of  what works, how and under which circumstances.The analytical task involves synthesising across all extracted information to search for the relationships between mechanisms, contexts and outcomes. Through our previous experience of realist synthesis (NIHR HS&DR projects 12/129/32:14/194/20)  and prinTo test and refine the programme theory, phase 3 work will provide a check of validity of the findings from the literature in relation to ‘what works,’ how and under which conditions in the current approaches to de-implementation of health and care practice and interventions. In 10 audio-taped telephone interviews with managers and organisational leaders, we will analyse data to test the synthesis findings and refine the programme theory, to specify its practical relevance/potential. As in phase 1, we will use both purposive and convenience sampling strategies to identify clinicians, managers and organisational leaders who can help us develop the realist syntheses programme theory. A semi-structured interview schedule will be developed that reflects the constituents and links between elements of the programme theory, and the interviews will focus on ‘sense-checking’ from the participants’ perspectives. We will use thematic analysis to analyse the data.A refined set of hypotheses with accompanying evidence-based narrative.In the final phase of the study, we will develop actionable recommendations for managers and service leaders. We will convene a second workshop with key stakeholders, which will allow us to test out and refine any element of the programme theory. We will also engage with the workshop participants and our broader network of stakeholders to (a) ensure different approaches to capture the right audience are used, (b) check that we are using language appropriate information for different audiences and (c) ensure that the key points from the synthesis are conveyed effectively for managers and service leaders, patients and the public.A full final report, using the evidence to provide examples of approaches and strategies to de-implementation, what features promote their use and success, what hinders them, and an understanding of the contextual influences which enable or disable their useAn executive summary of the final report, suitable for use as a separate reportA plain English summary of the report, suitable for use for briefing patients and the publicOpen-access publications of findings that set out an implementation plan for de-implementation.A range of blog posts to engage different audiences with the study and the findingsPresence and presentation at a nominated UK conference, which is focused on the strategies to improve efficiency and value in health and careCreative animation easily accessible on YouTubeA comprehensive evidence digest of our findings that provides an information selection source of the latest evidence about de-implementationWe will engage with teams of other commissioned studies to share learning and expertise around de-implementation.Using our synthesis findings, we will recommend a series of strategies that will help managers and clinicians de-implement low-value processes, and disseminate these recommendations through the following channels:This evidence synthesis is important for the public and patients, and the NHS as the current use of ineffective practices that lack an evidence-base or value translate into sub-optimal care and a waste of resources. Additionally, as an increasing number of new innovations are developed, there is a need to understand how these could replace or modify (rather than add to) existing practices and services. This evidence synthesis will address gaps in knowledge about de-implementation across health and care services and ensure that guidance about strategies and approaches takes into account contextual factors, which might be operating at different organisational and decision-making levels. To reflect the ethos of the realist approach, attention will be paid to the constituents of de-implementation strategies and approaches across different health and care organisations, and in articulating the explanation of how particular features of these approaches are more likely to promote efficiency and high-quality care for patients. Through the development of the programme theory, which explains what works, how and under which circumstances, findings from the evidence synthesis will support managers and service leaders to make the right decisions about de-implementation, whilst concurrently promoting the confidence of patients and the public and generating the potential for efficiencies.


Environmental impact assessments (EIAs) are used globally as the evidence-base for planning decisions, yet their efficacy is uncertain. Given that EIAs are extremely expensive and are enshrined in legislation, their place in evidence-based decision making deserves evaluation. The mean is the most commonly used summary statistic in ecological assessments, yet it is unlikely to be a good summary where the distribution of data is skewed; and its use without any indication of variability can be highly misleading. Here, using bats as an example, we show that EIAs frequently summarise these data using the mean or fail to define the term ‘average’. This can lead to the systematic misinterpretation of evidence which has serious implications for assessing risk. There is therefore a pressing need for guidance to specify data processing techniques so that planning decisions are made on a firm evidence-base. By ensuring that data processing is systematic and transparent it will result in mitigation decisions and conservation strategies that are cost-effective and proportionate to the predicted degree of risk. Environmental Impact Assessments (EIAs) are relied on as the evidence-base for planning decisions, yet their efficacy is uncertain  there is guidance for calculating ‘annual means’ of bat populations, whilst in South Africa the estimated collision risk to bats at wind farms is based on the mean number of bat passes per hour , detailed pre-construction risk assessments are frequently undertaken. The rapid technological development in acoustic recorders has made the use of static surveys, where automated bat detectors are left out for multiple nights, commonplace during these assessments in an attempt to capture ‘average’ or ‘peak’ bat activity, and so permit an inference of risk. The summary statistic most commonly used to report bat activity in professional ecological assessments is the mean. For example, in a sample of 30 randomly selected UK ecological assessments (EcIAs) considering bats, 17 (57%) cited the mean, whilst the remainder (43%) failed to define the term ‘average’. In European guidelines for the surveillance and monitoring of bats . A good environmental assessment should disclose all relevant information to allow the significance of the environmental effects of the proposed development to be determined (Elkin and Smith


To identify the core components of successful early warning systems for detecting and initiating action in response to clinical deterioration in paediatric inpatients.A hermeneutic systematic literature review informed by translational mobilisation theory and normalisation process theory was used to synthesise 82 studies of paediatric and adult early warning systems and interventions to support the detection of clinical deterioration and escalation of care. This method, which is designed to develop understanding, enabled the development of a propositional model of an optimal afferent component early warning system.Detecting deterioration and initiating action in response to clinical deterioration in paediatric inpatients involves several challenges, and the potential failure points in early warning systems are well documented. Track and trigger tools (TTT) are commonly used and have value in supporting key mechanisms of action but depend on certain preconditions for successful integration into practice. Several supplementary interventions have been proposed to improve the effectiveness of early warning systems but there is limited evidence to recommend their wider use, due to the weight and quality of the evidence; the extent to which systems are conditioned by the local clinical context; and the need to attend to system component relationships, which do not work in isolation. While it was not possible to make empirical recommendations for practice, the review methodology generated theoretical inferences about the core components of an optimal system for early warning systems. These are presented as a propositional model conceptualised as three subsystems: detection, planning and action.There is a growing consensus of the need to think beyond TTTs in improving action to detect and respond to clinical deterioration. Clinical teams wishing to improve early warning systems can use the model to consider systematically the constellation of factors necessary to support detection, planning and action and consider how these arrangements can be implemented in their local context.CRD42015015326. The literature in this field is heterogeneous and better at identifying system weakness than it is effective improvement interventions. By deploying social theories and a hermeneutic review methodology it was possible to develop a propositional model of the core components of an afferent component paediatric early warning system.The model is derived from logical inferences drawing on the overall evidence synthesis, social theories and clinical expertise, rather than strong empirical evidence of single intervention effectiveness.There is a growing consensus of the need to take a whole systems approach to improve the detection and response to deterioration in the inpatient paediatric population and this paper offers an evidence-based framework for this purpose.Failure to recognise and act on signs of clinical deterioration in the hospitalised child is an acknowledged safety concern.Despite the growing use of TTTs there is limited evidence of their effectiveness as a single intervention in reducing mortality or arrest rates in hospitalised children.What sociomaterial and contextual factors are associated with successful or unsuccessful Paediatric Early Warning Systems (with or without TTTs)?We performed a hermeneutic systematic review of the relevant literature. A hermeneutic systematic review is an iterative process, integrating analysis and interpretation of evidence with literature searching and is designed to develop a better understanding of the field.Data extraction and interpretation was informed by translational mobilisation theory (TMT)Box 1Object formation—how people draw on the interpretative resources available to them within a strategic action field to create the objects of their practice. Enrolment into an escalation trajectory requires multiple examples of object formation beginning with construction of an individual as at risk of deterioration and a regime of vital signs monitoring instigated, through recognition that the patient’s physiological status is a cause for concern, to the identification of the patient as requiring a specific intervention. How this is achieved is highly dependent on the features of the local strategic action field.Translation—the processes that enable practice objects to be shared and different understandings accommodated. It points to the actions necessary in order for a patient that is an object of concern for nursing staff to be translated into a clinical priority for the doctor and, if necessary, to be translated into the focus of intervention by the emergency response team.Articulation refers to the secondary work processes that align the actions, knowledge and resources necessary for the mobilisation of projects of collective action. It is the work that makes the work, work. Responding to deterioration is time critical and articulation work is necessary to ensure the availability of resources and materials to support clinical management. This is not a mundane observation; catastrophic failures in patient safety are often attributed to the lack of functioning equipmentReflexive monitoring refers to the processes through which people collectively or individually appraise and review activity. In a distributed field of action, reflexive monitoring is the means through which members accomplish situational awarenessSensemaking refers to the processes through which agents create order in conditions of complexity. It draws attention to how the material and discursive processes by which members organise their work, account for their actions and construct the objects of their practice also give meaning and substance to the institutional components of strategic action fields that shape activity and condition future activity.The literature in this field identifies four integrated components which work together to provide a safety system for at-risk patients: (1) the afferent component which detects deterioration and triggers timely and appropriate action; (2) the efferent component which consists of the people and resources providing a response; (3) a process improvement component, which includes system auditing and monitoring; and (4) an administrative component focusing on organisational leadership and education required to implement and sustain the system.Literature was identified through a recent scoping review,10.1136/bmjopen-2018-028796.supp1Supplementary dataWe undertook systematic searches of the paediatric and adult EWS literature . For the adult literature we used the same search strategies but added a qualitative filter to limit the scope to studies most likely to yield the level of sociomaterial and contextual detail of value to the review. Literature informing additional areas of interest was located through a combination of systematic and hand searches. Systematic searches (searches 2 and 3) were undertaken in areas where we anticipated locating evidence of the effectiveness of specific interventions to strengthen EWS. Theory-driven searches reflected the conceptual requirements of the model development.A systematic search was initially conducted across a range of databases from 1995 to September 2016 to identify relevant studies on the PEWS literature. This search was updated to cover literature from September 2016 to May 2018. An additional three systematic searches were conducted from 1995 to September 2016 to identify supplementary papers to aid in developing understanding on the PEWS literature:Adult EWS.Interventions to improve SA.Structured communication tools for handover and handoff.Detailed information on the search methodology can be found in Additional theory-driven searches were conducted in the following areas:Family involvement.Observations and monitoring.The impact of electronic systems.These were a combination of exploratory, computerised, snowball and hand searches. As the analysis progressed, we continued to review new literature on EWS as this was published.After removing duplicates 5284 references were identified for screening. A modified Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram is provided . Papers Data extraction template 2  was applA propositional model was developed specifying the core ingredients of a paediatric early warning system . It compThis review was conducted as part of a larger mixed methods study (ISRCTN 94228292), which used a formal, facilitated parental advisory group. The group comprised parents of children who had experienced an unexpected adverse event in a paediatric unit and provided input which helped shape the broader research questions and wider contextual factors to consider, specifically within the family involvement element of the system. The results of the review will be disseminated to parents through this group.Eighty-two papers were included in the review. Forty-six papers focused on TTT implementation and use in paediatric and adult contexts (24 from the paediatric search and the remaining 22 from the adult-focused search); the remaining 36 papers contributed supplementary data on factors related to the wider warning system. See detection of signs deterioration; the planning needed to ensure teams are ready to act when deterioration is detected; and the initiation of timely action. While we have focused on the afferent component, it is important to remember that all elements of the overall safety system  need to be working in concert in order to maintain an optimal paediatric early warning system. In the next section, we report on the literature in relation to each subsystem.In TMT the primary unit of analysis is the ‘project’, which defines the social and material actors  and their relationships involved in achieving a particular goal. The goals of the afferent paediatric warning system are: first, that the child is identified as at risk and a vital signs monitoring regime instigated; second, that evidence of deterioration is identified through monitoring and categorised as such; and third, that timely and appropriate action is initiated in response to deterioration. Our analysis of the literature suggests that three subsystems within the afferent component of EWS support these processes: the The goal of the detection subsystem is to recognise early signs of deterioration, so the child becomes the focus of further clinical attention. This requires, first, that the child is identified as at risk and a vital signs monitoring regime instigated and, second, that the child is identified as showing signs of deterioration.Despite widespread use, the evidence on TTT effectiveness in predicting adverse outcomes in hospitalised children is weak.TTT’s effectiveness in fulfilling these functions depends on certain preconditions. The review highlighted that TTT use was impacted by the availability of appropriate and functioning equipment,et alTTTs are also used differently depending on the experience of the user. For juniors, they provide a methodology and structure for monitoring clinical instability and identifying deterioration, whereas more experienced staff reportedly use TTTs as confirmatory technologies.There is growing interest in the literature in strategies that facilitate patient and relative involvement in the early detection of deterioration.While much of the literature reports on intermittent manual vital signs monitoring and paper-based recording systems, across the developed world there is a growing use of electronic technologies, which have important implications for the wider detection subsystem.Detecting and responding to deterioration involves the coordination of action in conditions of uncertainty and competing priorities. The goal of the ‘Planning’ subsystem is to ensure the clinical team are ready to act in the event of evidence of deterioration and is reflected in the growing interest in the literature on structures to facilitate team SA, group decisions and planning.et al describes the use of the ‘watcher’ category to designate a patient as at risk where staff have a ‘gut feeling’ deterioration is likely.TTTs have been found to support SA. Their use enabled clinicians to have a ‘bird’s-eye’ view over all admitted patients on a ward as well as encouraging staff to consider projected acuity levels of the ward.Handovers are integral to clinical communication and contribute to SA. The extensive literature on handover indicates that information sharing can be of variable qualityThere is also a literature on the use of common information spaces—such as whiteboards—in facilitating SA in the healthcare team.et al discuss the three main supports for SA—whiteboard, handover and coordinator role—and illustrate how these interacted in organisations with strong SA compared with those with reduced levels. Crucially, this ‘interplay’ between the different activities was highly context dependent; ‘the same supports used differently generate different outcomes’ (p 52).The literature indicates that SA can be facilitated in different ways in different contexts and it is the relationship between system elements that is important.The goal of the ‘Action’ subsystem is to initiate appropriate action in response to evidence of deterioration. The literature suggests that mobilising action across professional boundaries/hierarchies is challenging, with differences in language between doctors and nurses and power dynamics contributory factors.et al highlight that audit data suggest resistance to SBAR, with others cautioning that overextending SBAR use carries the risk of SBAR fatigue and attenuation of its effects.Several studies also report on the use of SBAR in this context. Like TTTs, SBAR translates information into a form that provides structure, consistency and predictability when presenting patient information. SBAR has been shown to help establish common language and expectations, minimising differences in training, experience and hierarchy and facilitating nurse–clinician communication. While several papers advocate combining SBAR with TTTs,Structured communication tools like TTTs and SBAR do not solve all the challenges of acting in response to evidence of deterioration. Barriers to action were widely reported in the literature where these tools were in place. These include: a general disinclination to seek help,No literature reported on successful interventions to facilitate RRT use, but several propose strategies to support escalation where there was no designated response team in place in the efferent component. These include informal peer support, where inexperienced staff team up with more experienced staffnot made.There is a small literature on family involvement in the Action subsystem. Several studies report on Condition-Help, a programme developed in the USA to support families to directly activate an RRT if they have concerns about their child’s condition. Families are also becoming increasingly recognised as playing a key role in the activation of RRTs in Australia.The literature in this field is heterogeneous and stronger on the sociomaterial barriers to successful afferent component paediatric early warning systems than it is on solutions. While a number of different single interventions have been proposed and some have been evaluated, there is limited evidence to recommend their use beyond the specific clinical contexts described in the papers. This reflects both the weight and quality of the evidence, the extent to which paediatric systems are conditioned by the local clinical context and also the need to attend to the relationship between system components and interventions which work in concert not in isolation. There is also a growing realisation in the quality improvement field that an intervention that has been successful in one context does not necessarily produce the same results elsewhere which cautions against a ‘one size fits all’ approach.While it is not possible to make empirical recommendations for practice, a hermeneutic review methodology enabled the generation of theoretical inferences about the core components of an optimal paediatric early warning system. These model components are logical inferences derived from an overall synthesis of the evidence, informed by our theoretical framework and clinical expertise. These are presented as a propositional model conceptualised as three subsystems: detection, planning and action see .This paper reports on one of three linked reviews undertaken as part of a wider UK study commissioned to develop and evaluate an evidence-based national paediatric early warning system.The literature in this field is heterogeneous and better at identifying system weakness than it is effective improvement interventions. It was only by deploying social theories and a hermeneutic review methodology did it prove possible to develop a propositional model of the core components of an afferent component paediatric early warning system. This model is derived from logical inferences drawing on the overall evidence synthesis, social theories and clinical expertise, rather than strong empirical evidence of single intervention effectiveness. Consequently, there is a growing consensus of the need to take a whole systems approach to improve the detection and response to deterioration in the inpatient paediatric population.Failure to recognise and act on signs of deterioration is an acknowledged safety concern


The paper carries out a situational analysis to examine the production, dissemination and utilisation of reproductive and child health-related evidence to inform policy formulation in Ghana’s health sector.The study used Wald’s model of knowledge production, transfer and utilisation as a conceptual model to collect relevant data via interviews and administration of questionnaire to a network of persons who either previously or currently hold policy-relevant positions in Ghana’s health sector. Additional data was also gathered through a scoping review of the knowledge transfer and research utilisation literature, existing reproductive and child health policies, protocols and guidelines and information available on the websites of relevant institutions in Ghana’s health sector.The findings of the study suggest that the health sector in Ghana has major strengths  and opportunities . What remains a challenge, however, is the absence of a robust institutional-wide mechanism for collating research needs and communicating these to researchers, communicating research findings in forms that are friendlier to policy-makers and the inability to incorporate funding for research into the budget of the health sector.The study concludes, admonishing the Ministry of Health and its agencies to leverage on the existing strengths and opportunities to address the identified challenges. Evidence-based practice or evidence-informed policy-making generally refers to systematic efforts to ensure that research evidence becomes an important input into policy-making . This haThe idea that virtually all forms of policy should be based on strong scientific evidence can be traced to the establishment of the evidence-based medicine (EBM) framework. At the core of EBM is the use of clinical evidence (resulting from scientific research) to guide clinical practice. The growth of EBM has transcended clinical practice and greatly influenced the call for non-clinicians  to abandon policy development approaches that rely heavily on common sense, popular support and political ideology in favour of approaches that are primarily based on scientific facts/evidence generated through appropriate and robust scientific research. It is not uncommon for discussions about evidence-informed policy to generate debate about what constitutes evidence. Generally, evidence can be operationalised to mean facts  gained through observation or experiences and used to support a conclusion . The NatNotwithstanding the importance of using scientific research evidence to guide the formulation and implementation of health policies, there is evidence in the research utilisation literature to suggest the existence of a major gap between available research evidence and actual practice  . The resAlthough Ghana’s investment in health has improved over the years and is regarded as one of the best in the sub-Saharan African region, it compares unfavourably to other developing countries. For example, health expenditure (HE) as a percentage of Gross Domestic Product (GDP) and HE per capita increased from 3% in 2000 to 5.4% in 2013 and USD 82.4 in 2000 to USD 214.2 in 2013, respectively. Ghana’s HE as a percentage of GDP and HE per capita in 2013 compares favourably to Kenya (4.5% and USD 44.5), Nigeria (3.5 and USD 115), The Gambia (6% and USD 28.9), Uganda (9.8 and USD 59.1) and Mali (7.1 and USD 53.3). Although Rwanda’s HE as a percentage of GDP is 2.5 percentage points higher than Ghana, their HE per capita of USD 83 is lower compared to Ghana  is to generate information through relevant research to strengthen decision making, set health priorities, efficient resource allocation, and inform health interventions planning and implementation in order to deliver better health services to improve health status of the Ghanaian population” [Information provided on the website of the GHS, in addition to discussions with key managers within the GHS, suggest that there are currently formal structures in place to ensure a coherent approach to evidence gathering and synthesis. The GHS has a Research and Development Division headed by a director. The division has three research centres , in addition to three other departments responsible for (1) documentation, dissemination and advocacy, (2) ethics and research management and (3) research. As captured on the GHS website, “ulation” .To confirm that the information above, as captured on the GHS website, is operational, a review of the websites of the three research centres was performed. The results from the review suggest extremely impressive activity in terms of completed projects, on-going projects and new projects funded by internationally reputable funding organisations such as WHO, International Labour Organisation (ILO), Bill and Melinda Gates Foundation, Rockefeller Foundation, Pfizer Corporation, Volkswagen Foundation, National Institute of Health, World Bank, West African Health Organization (WAHO), and other bilateral and multilateral donors. In addition to these projects, the three centres have impressive outputs in the form of peer-reviewed scientific publications from completed projects.For example, the information on the website of the DHRC showed 6 completed projects, 3 on-going projects and 29 peer-reviewed scientific publications . In DodoBesides the work of the research centres, the capacity of actors within GHS (staff and their collaborators) to produce good quality scientific evidence on MNCH was also examined. The results Table  suggest The output of the research centres is also augmented by a strong health information management system, which collects, processes and stores data from different clinical and non-clinical institutions within the GHS. Discussions with actors within and outside of the GHS indicate that the output of the health information management system constitutes strategic evidence (inputs) into different policies, including RCH. The health information management system is equally augmented by data from several demographic surveillance systems, in addition to data from nationally representative surveys such as the GDHS and the Ghana Living Standards Survey, conducted every 5 years. Mostly, data from the three sources mentioned above are processed and presented in a format that can easily be used by policy-makers.Clearly, the sources above point to the fact that the health sector in Ghana has enormous capacity both at the individual and organisational level to produce the much-needed evidence for policy formulation. This view is equally supported by results of responses to the questionnaire administered to participants of the Accra stakeholder meeting. The majority of respondents  believe that the GHS and its staff have adequate capacity to produce good quality evidence to aid policy-making.This sub-section examines issues related to (1) the relative advantage and complexity of the knowledge to be transferred, (2) the compatibility of the knowledge to be transferred with existing beliefs and organisational norms, and (3) how the knowledge to be transferred is aligned to the needs of policy-makers.Most of the research outputs found on the websites of the three research institutions are in the form of peer-reviewed research papers instead of alternative forms, such as policy briefs, that can be easily used by policy-makers. Although respondents to the questionnaire suggest that knowledge of policy briefs and how to use them is widespread in Ghana’s health sector, the evidence from the study suggests that RCH-related researchers in Ghana hardly use such means to disseminate their findings.Although some respondents during the group discussion argued that, generally, public institutions in Ghana do not have the culture of evidence-based policy-making, this was not seen to be the case in the health sector. The difference may be due to the fact that the health sector has relatively better qualified staff who are also more experienced in research compared to other sectors in Ghana. Thus, the use of research evidence in policy-making may not be foreign to such actors. This assertion is equally confirmed by the results from the questionnaire. However, what may constitute a challenge is the extent to which the available evidence is aligned to the needs of policy-makers. As earlier on indicated, and as confirmed by the results of the questionnaire, a larger proportion of research in the health sector in general, and on RCH in particular, are mostly funded by donors who often have their own objectives, which may be different from those of policy-makers. Thus, besides a few circumstances where users of evidence collaborate with donors to conduct research, evidence produced from on-going research may not be aligned to the evidence/knowledge needs of policy-makers. This therefore constraints on-going efforts to improve the use of evidence to inform policy.This section deals with identified barriers and incentives to knowledge transfer at the individual, organisational and environment level. As is evident in the research output of the three research centres and the large presence of GHS staff in the RCH literature, the capacity of individuals within the health sector to produce, synthesise and potentially use research evidence in the formulation of RCH policies cannot be in doubt. In addition, the multiplicity of professionals  within the health sector creates internal competition with respect to knowledge production and synthesis at the individual level. It is important though to caution that the capacity to produce and use evidence at the individual level may not in itself translate into the actual use of evidence in policy formulation.Besides the individual level capacity, the findings of the study also suggest that there are several institutional and environmental incentives that promote the use of evidence in policy formulation within Ghana’s health sector. These include the sector-wide performance assessment framework linked to a set of internationally acceptable indicators that are also evidence-based, the subscription of the government through the MOH to major health-related international policies and conventions that are based on research evidence. This is evidenced by the several guidelines and protocols developed to ensure that actions at lower levels are consistent with the policies, protocols and conventions subscribed to. In addition, the evidence-based nature of work in the health sector makes it an environment that generally has few barriers to the use of research evidence for policy-making. This notwithstanding, major constraints to the use of evidence in policy include (1) how to ensure that research in general, and for that matter health-related research, gets the right attention at the highest level of political and administrative leadership so as to attract the right level of funding, and (2) the lack of robust and comprehensive institutional structures that ensure that the numerous health sector-related research being performed  is aligned to the needs of policy-makers. The current disconnect between knowledge production activities and the needs of policy-makers eventually affects the extent to which policy-makers make use available knowledge to inform policy formulation. This position is confirmed by the results of the survey (questionnaire) where respondents suggest that institutional level incentives for (1) carrying out research, (2) use of research evidence implementation committees, (3) capacity to present research evidence in concise and accessible languages, and (4) capacity to synthesise different research evidence that addresses a common problem into a single document that could be attractive to policy-makers are inadequate, albeit that the required capacity and willingness to carry out good quality research exist at the individual level.This aspect of the knowledge transfer and utilisation framework looks at actual interventions put in place to facilitate/ensure the transfer of knowledge/evidence for the purposes of policy-making. In the context of Ghana’s health sector, the most common intervention is post research dissemination workshops/conferences. In few instances, targeted policy briefs from completed research are disseminated to key stakeholders. However, as already indicated based on evidence from the three research centres, the use of policy briefs as a tool for ensuring the uptake of research evidence into policy in general and RCH in particular is rather scarce. The evidence gathered also suggests that advocacy by researchers as well as dialogue between donors and policy-makers constitute some of the channels via which research evidence on some key policy issues are discussed and consensus is reached on the possibility of adopting such evidence to aid policy formulation.n = 11), neonatal and child health (n = 1), general infection prevention and control (n = 1) and malaria prevention (n = 2) were also reviewed. Out of the 15 protocols and practice guidelines, 5 were evidence informed. Surprisingly, however, the policy documents and practice guidelines that suggested the use of scientific evidence did not explain the evidence generation processes and how the evidence was used in the policy/practice guidelines. Nevertheless, it is clear that the 8 policy documents and 15 protocols and practice guidelines were all developed through a consultative and participatory approach mostly involving stakeholders and professionals knowledgeable in the subject area. It is also important to state that discussions with key managers within the RCH division of the GHS indicated that almost all the standard protocols and practice guidelines in use are adaptations from regulators or key global institutions , which are known to rely on scientific evidence in the production of such guidelines.This component of the knowledge transfer process looks at the actual use of available evidence in the formulation of policies. Following from the evidence gathered, we examined the level and extent to which the extensive evidence available, both within and outside of the GHS, is used in the formulation of policies and guidelines. Thus, 8 policy documents  , 47 was From the above, one can argue that, although a structured and robust organisational mechanism to ensure the use of scientific research evidence to inform policy formulation does not exist, actors within the health sector are relying on existing, albeit weak, institutional platforms to transfer knowledge gained from research to policy and practice.The information extracted in this study is herein used to answer the key question of whether Ghana’s health sector has institutional structures or mechanisms in place for the production and use of evidence to inform policy formulation in general and RCH policies in particular.The findings above suggest that there are currently organisational-wide structures in place for the production of RCH evidence. This is based on the fact that the GHS has a well-functioning and extremely active research and development division, as is evident in the number of projects and peer-reviewed scientific research publications completed. The fact that most of the research projects carried out by the research centres are funded by internationally reputable funding organisations such WHO, ILO, Bill and Melinda Gates Foundation, Rockefeller Foundation, Pfizer Corporation, Volkswagen Foundation, National Institute of Health, World Bank, WAHO, and other bilateral and multilateral donors, speaks to the quality of their research output. It is important to emphasise that quality, as used in this context, is mainly in reference to the rigorous nature of the evidence produced as opposed to coverage . The rigour argument is based on the fact that funding organisations, such those indicated above, will normally ensure that research funded by them follow very rigorous standards.Besides the output of the research centres, the findings indicate that staff from within the GHS are highly active contributors to the RCH literature in Ghana. Again, the existence of organisational and national level data repositories that can easily be accessed and used by policy-makers to aid policy formulation is ample testimony of the existence of high quality scientific research evidence that can be used to inform policy formulation. Additionally, the findings suggest that both GHS staff and other external collaborators perform scientific research within the GHS. Thus, the findings indicate to a certain systematic approach at the organisational level to enhance the production of evidence through scientific research.The existing institutional arrangements for the production of evidence may, in some form, create opportunities for the use of evidence produced to inform policy. For example, there is evidence in the existing knowledge transfer and utilisation literature  that sugOn the contrary, the results paint a mixed picture. One set of evidence indicates that systematic organisational structures as well as individual capacity that makes it relatively easy for policy-makers to use available evidence in the formulation of RCH policies exist. For example, project reports that capture the details  of how findings of executed projects have been used in the formulation of policies is instructive. The fact that almost all the standard protocols and practice guidelines are adaptations from benchmark documents issued by regulators or key institutions such as WHO, UNICEF, UNFPA and the World Bank, which rely highly on scientific evidence in the production of such guidelines, is also important. Thus, modelling RCH protocols and guidelines around documents of such institutions suggests that such protocols and practice guidelines are informed by evidence.Further, evidence on the scaling up of RCH interventions based on lessons learnt from the pilot phase of most of these projects is ample testimony that scientific research evidence gathered from project execution constitutes a relevant input to policy formulation and implementation. As already indicated, the scaling up of the Navrongo experiment (i.e. the CHPS concept) has had and will continue to have a profound impact on the nature and type of strategies adopted to improve RCH outcomes in Ghana. The possibility of scaling up capitation as a payment mechanism for Ghana’s social health insurance from one region/province to national coverage is also important in this direction. Additionally, the referencing of scientific research papers in the bibliography section of some of the policies, standard protocols and practice guidelines is suggestive that findings from such papers constituted a good piece of evidence used in the formulation of the respective policies. On account of this evidence, one can suggest that capacity exists within the GHS that makes it possible to use available scientific research evidence to aid the formulation and implementation of RCH policies.Nevertheless, there are findings from the study that equally suggest the existence of constraints that can compromise the ability of the GHS to use existing evidence to inform RCH policy. For instance, several of the policy documents and standard protocols and practice guidelines did not have any scientific research publication acknowledged in the list of references. For example, only 3 out of the 8 policy documents reviewed cited scientific research papers. In the case of the standard protocols and practice guidelines, only 5 out of the 15 documents reviewed cited the use of a scientific research paper. More importantly, those documents that cited scientific research papers did not explain the processes through which evidence emanating from these scientific papers was captured and used in either policy formulation or drafting of the standard protocols and practice guidelines. Equally important in this regard is the apparent lack of both physical and electronic evidence that shows that findings from the numerous completed projects as well as peer-reviewed publications have been converted to a form that can be easily used by policy-makers (e.g. policy briefs or policy summaries). Additionally, the divergence of donor priorities and those of policy-makers in terms of research may mean misalignment between evidence produced from research and information needed by policy-makers at a particular point in time. Thus, RCH evidence may be available (e.g. output of the research centres) and yet may not be useful for decision-making. It is important to caution that the policy-maker/donor divergence in research priorities may not necessarily mean a misalignment between research output of the research centres and population health needs, given that research undertaken by the research centres and individuals within the sector are all on issues of national relevance. Thus, the identified shortcomings should be viewed mainly as structural weaknesses within the MOH and its agencies, specifically the GHS, which constrain the use of evidence to inform policy formulation, and not necessarily as evidence to support the notion that policies and guidelines are not informed by existing scientific research evidence.However, what seems to be clear from the evidence gathered so far with respect to the management of the knowledge transfer processes in Ghana’s health sector and, for that matter, RCH, is that apart from the actual knowledge production component, the rest of the process seems to be rather informal. For example, the key components of problem identification and communication, implementation of interventions and actual use of evidence do not follow any structured institutional system. Additionally, funding for research is not centrally coordinated but mostly through an arrangement between individuals or clusters of researchers within the health system and donors. It is important to emphasise that these weaknesses have collective negative implications for establishing an appropriate monitoring and evaluation framework to evaluate the knowledge production process and how it is impacting policy formulation. It is therefore not surprising that key performance indicators used to evaluate sector-wide performance in Ghana do not capture knowledge production and transfer as an activity of interest to monitor , 24.A careful examination of the findings suggest that Ghana, compared to many sub-Saharan African countries, has made progress in producing evidence and using this to inform policy formulation. It is therefore important that the strengths of the existing system are highlighted and that appropriate measures are instituted to deal with inherent weaknesses and threats so as to take advantage of current and future opportunities to improve existing knowledge transfer processes in the health sector.The findings are suggestive that, within Ghana’s health sector and particularly in the RCH division of the GHS, policy formulation is evidenced based. There is also enough evidence to suggest that the GHS has organisational level structures in place (the three research centres) to aid the production of the required evidence to inform policy formulation. Also important is the fact that the GHS has a critical mass of researchers who are extremely active in contributing to the production of scientific knowledge in RCH. As earlier indicated, formal and informal structures that drive organisational learning and norms that value the importance of evidence in decision-making , 19 are The ability of researchers to summarise very complicated scientific language into simpler but easy and ready to use material by policy-makers is seen as a key facilitator of research uptake into policy . In thisA familiar argument within the knowledge transfer and research utilisation literature has been the need to bridge the cultural and institutional gap between researchers and policy-makers, given that it constitutes a major barrier to research uptake in policy formulation –53. ThusOutside of the internal structures there are clear opportunities that can be exploited by the GHS to improve on its uptake of scientific research evidence as inputs to the policy formulation process. For example, the extensive network of collaborations with researchers outside of the GHS as well as access to major financiers  of scientific research are all great opportunities that can be exploited to improve internal structures for producing and using good quality evidence to inform policy formulation and implementation. It is however important to emphasise that, to be able to take advantage of the opportunities enumerated, a major threat that ought to be managed well is the issue of funding. Issues around the nature and source of funding, as well as the coordination and management of such funding, are crucial. The fact that a larger proportion of funding for research in the health sector comes from outside the health sector budget is a threat to implementing any robust central coordinating mechanism for knowledge production and transfer.It is important that as Ghana’s strengths and opportunities for knowledge production and transfer are highlighted, effective and immediate steps are taken to address inherent weaknesses and threats to ensure that it improves on the current system of producing evidence and using it to inform the formulation of RCH policies. As clearly articulated by the participants of the Accra stakeholder meeting, improving the knowledge transfer process is a collective task that must involve all stakeholders  with the health sector taking the lead.Issues of RCH and MNCH are important areas in the strengthening health systems in developed and developing countries alike. In developing countries such as Ghana, the issue is even more critical given that health systems are generally weak and have low levels of investment and poor RCH indicators, especially during the 1990s. However, the implementation of pragmatic policies and commitment of health sector managers over almost two decades has ‘paid off’ in terms of tremendous improvement in the state of RCH.Nevertheless, Ghana still has challenges in RCH and further efforts to improve RCH policy-making processes are likely to yield even better results. It is therefore important that the MOH and its agencies, especially the GHS, take advantage of its current strengths and opportunities  to confront and address the structural and organisational weaknesses inherent in the process of translating research evidence to evidenced-informed policy. Specifically, there should be deliberate efforts to mainstream research in the health sector so that an appropriate budget can be allocated to fund research. The mainstreaming of research within the sector could also help resolve the issue of the establishment of research priorities and their coordination between researchers and policy-makers.


An EU-wide cooperation on HTA has been proposed recently by the European Commission, focusing on relative effectiveness assessment (REA) for pharmaceuticals and medical devices. This cooperation is operationalised through a proposal for a regulation. While a good step in the right direction, this HTA cooperation framework needs to be more explicit and pragmatic about clinical value definition, what constitutes quality of evidence, how real-world evidence is handled, whether the same assessment requirements will apply for medical devices as they do for pharmaceuticals, and how to safeguard consistency in REA interpretation. If demand-rather than supply-driven, this initiative can deliver wider benefits: Europe can improve its power in global drug design and development, while Member States will have at their disposal more resources to assess performance of interventions in their healthcare systems. On January 31st, 2018, the European Commission released a proposal for a new regulation on health technology assessment (HTA) cooperation. This aims, among others, to streamline disparate national HTA processes and generate a single, joint clinical assessment, with a focus on joint relative effectiveness assessment (REA) of pharmaceuticals and certain types of medical devices, the promotion of early dialogues and the identification of emerging technologies and other opportunities for voluntary cooperation [Although the draft regulation aims to provide a common foundational framework for HTA at EU level, it falls short of addressing a number of much needed changes in the assessment of medical technologies and adapting to the reality of post-clinical and post-launch data generation. Scientific challenges in novel drug discovery and development  and the Specifically, the HTA draft regulation needs to provide a clearer steer on what constitutes clinical value; clarify aspects relating to the quality of evidence; clarify what is the use of real-world evidence (RWE) in joint assessments and beyond; outline how should medical devices be assessed; and ensure consistency in REA interpretation.The credibility of a collaborative assessment at European level and its subsequent uptake by national competent authorities depend on the robustness of the process and the criteria that underpin it. In that context, it would be desirable to see a clearer definition and a more common understanding of what ‘clinical value’ is, infused with a degree of pragmatism. Beyond a technology’s clinical effectiveness and safety, dimensions such as burden of disease, severity and the level of unmet need for new treatments could be acknowledged or even accounted for. The first four of the nine domains of the European Network of Health Technology Assessment (EUnetHTA) HTA Core Model for Rapid Relative Effectiveness  could seA major area underpinned by significant challenges and where clear guidance is needed relates to what constitutes appropriate or acceptable quality of evidence and its assessment. A number of challenges are involved here. First, there is a divergence between the ‘political’ and the ‘patient-centric’ drive to faster approvals, increasingly through regulatory early access schemes or clinical evidence with immature data –8, but fIf, as its proponents suggest, the proposed regulation becomes mandatory, then there needs to be some clarity on how promising products with less mature or earlier phase data or data from non-randomised studies, will be treated from an assessment perspective. This is because the regulatory view, through EMA’s accelerated or conditional approval pathways, differs from the HTA view, which is much more conservative, considering uncertainties of evidence. If there is no clarity, then such products run the risk of a generalised ‘rejection’ on account of insufficient evidence. This, in turn, may conflict with the EC’s objective of improving the availability of innovative health technologies. There is a significant value arising from EUnetHTA in terms of facilitating a sustainable HTA collaboration in Europe and, more specifically, on methodology advances . EquallyThe use of RWE is key to the proposed EU regulation. In some diseases, the value resonates in having long cohort data over many years, whereas in others, which could be driven by specific genetic subtypes, companion diagnostics and testing become very important. Obtaining these data from existing databases is usually not possible, necessitating the wider availability and use of registry data. A key challenge from an HTA perspective remains the variable acceptance of RWE. Despite improvements in infrastructure, issues remain with access to data in several Member States, including privacy issues, the lack of incentives for data sharing, availability and use, and the ongoing debate about RWE distrust. It is, therefore, fundamental to have a discussion on what needs RWE is going to fulfil. For example, while studying uncertainty in long-term effectiveness remains key, it is also important to study how the effectiveness of a new therapy can re-shape clinical practice. Having this ex ante clarity will also drive methodological and empirical developments in RWE generation further. If we accept the argument that bringing RWE into decision making will improve the quality of decisions and that expertise is needed on both sides of the data continuum, then these issues need to be addressed urgently.HTA for medical devices poses significant challenges, particularly for small companies, in terms of fulfilling evidence requirements. Medical devices have a shorter life cycle than pharmaceuticals and the market is much more competitive as patents are easier to circumvent for medical devices, consequently, while long RCTs may be desirable from an evidence standpoint, they may not always be appropriate or feasible for medical devices. Unlike pharmaceuticals, medical device companies do not have data exclusivity, which means that companies who do not conduct clinical studies on their own device can often claim equivalence  to anothAdditionally, the number of medical devices far exceeds that of pharmaceuticals and performing HTA on all medical devices would be extremely resource intensive for all stakeholders involved even in the subset of devices envisaged by the proposed regulation (class IIb and III and subject to a scrutiny mechanism). Both assessments and re-assessments can be challenging in the medical device sector given the frequency of product modifications, the level of competition and the timing of competitor entry, which is much faster than in pharmaceuticals. Additional competitors will enter the market and have an impact on the price of an existing product, possibly placing first entrants at a disadvantage, creating an obvious barrier to entry for the first-in-class, which is absent for all followers in that class. Evidence requirements in HTA often tend to be strictest for first movers in a therapeutic area and this is likely to affect medical devices more given the market dynamics in this sector. Finally, there needs to be a common understanding about appropriate comparators for a treatment, as it is well known that countries vary substantially in defining these. The same holds for a number of therapy areas as well, such as cancer therapy, as current standard of care often differs between countries. In sum, EU HTA needs to be more circumspect about the need for REAs in medical devices and perhaps define the need, the associated evidence requirements and a methodological framework guiding these more tightly, beyond joint scientific consultations providing advice on study designs.A final issue relates to the consistency of REA interpretation and the ex ante definition of the rules of the game. There may be competing interests and different remits among the 80 plus EUnetHTA partner organisations from 30 countries, even among partners from the same country, which may lead to differences in assessments. Additionally, how factors such as indirect comparisons and RWE are able to provide a predictable interpretation of that evidence is important to clarify. EU-wide REAs would be beneficial so long as the reproducibility of any report and a predictable pathway to their interpretation by national competent authorities can be guaranteed. If this is not the case, then the unavoidable consequence will be further duplication of the overall HTA effort. To that end, independent assessments from a tertiary group of experts and based on common principles could minimise this.The extent to which a mandatory EU-wide HTA process will reduce duplication and achieve efficiency gains depends on whether such an assessment results in (a) clarity and transparency in evidence requirements across therapeutic areas, enhanced by systematic early dialogue; (b) consistency of methods and their acceptability, recognizing the peculiarities and challenges of modern drug development; (c) clarity on the timing of performing REAs and, therefore, predictability of outcomes; and (d) a greater dialogue between the regulatory side (EMA) and HTAs around alignment of evidence requirements. This by no means implies centralisation of HTA processes; rather, common methods, joint scientific consultations and joint clinical assessments are expected to be synergistic.There is a perception that because centralisation has been achieved for marketing authorisation, the same can be done for HTA. The two are completely different: centralisation of regulation results in a decision applicable to all, whereas HTA is not a decision, but a collection, synthesis and analysis of different information pieces that are subject to interpretation prior to arriving at a recommendation. The challenge in making this latter process mandatory is a challenge about making EU HTA demand-rather than supply-driven. If it is demand-driven and Member States use this information to shape national priorities, then there is value in generating it. But if it is supply-driven and the objective is to be churning out reports to inform the rest of Europe, then there is a danger of more duplication, unless there is buy-in from decision makers on the value of those reports. Assuming a demand-driven EU HTA, there needs to be consensus on a template for REAs; the need for such a template is already recognised by the proposed regulation.The EU HTA cooperation is unlikely to result in substantial savings, as HTA bodies and national competent authorities will still want to understand local epidemiology and need, conduct budget impact analysis and negotiate pricing. But it can result in benefits for Member States and for Europe. Member States can free up and re-allocate resources to measure performance of interventions in the wider health care system ; this is urgently needed to balance the over-emphasis on drug-related HTA. From a broader perspective, this initiative would enable Europe to improve its negotiating power in global drug design and development if views can be aligned on evidence requirements. But at the moment they cannot and so the FDA and the USA still dictate to a large extent what clinical studies measure and what endpoints are used.