Recently two important evidence-based recommendations about the use of probiotics in the prevention of allergy were published , 2 with We tried to retraces the path proposed by the Authors to analyse three important questions about the efficacy of probiotics in preventing allergic diseases if administrated to pregnant women (first question), to breastfeeding mothers (second question) and in healthy infants (third question).These questions have been investigated by using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach , to perfFrom a methodological point of view, it should be noted that the recommendations are directed to patients, clinicians and other health care professionals with different objectives, as clearly explained in the guidelines introduction. Each recommendation can have different levels of strength: strong recommendation indicates that patients would like to receive the intervention and that clinicians should actuate it. Conditional or weak recommendation indicates that the majority of patients would like to receive the intervention, but many other not, as well as they hope that clinicians would recognize that different choices may be appropriate for different patients, by taking advantage of decision-making tools to help patients to make consistent choices.In the results section of the guidelines we can read that “Currently available evidence does not indicate that probiotic supplementation reduces the risk of developing allergy in children. However, considering all critical outcomes in this context, the World Allergy Organization (WAO) guideline panel determined that there is a likely net benefit from using probiotics resulting primarily from prevention of eczema” .In the first sentence, the guideline panel confirmed the absence of evidence of effectiveness in the use of probiotics in the primary prevention of the common allergic diseases . The second sentence deserves a careful review, because it seems to contradict the meaning of the first sentence and the position of the other guidelines, including the recent European Academy of Allergy and Clinical Immunology (EAACI) guideline .In order to comment this phrase, we started from the description of the evidences (GRADE) in Appendix 2 - Question 1: “Should probiotics vs. no probiotics be used in pregnant women?” In this appendix we can found declared how many studies addressing this question have been selected for each of the clinical objectives: 15 randomized trials were considered for the prevention of eczema.The comment of this question can be also reliable if it is applied to the other two questions above, as approximately the same studies have been selected for critical review.In this regard, we carefully examined the individual trials to better analyze and understand the results of the question. The most important clinical features that supported our considerations are reported in Table A preliminary observation about the 15 trials above –18 is thAlmost all studies identified patients with family history of atopy as targets for treating, indicating that children at high risk of atopy should be the potential beneficiaries of a preventive intervention with probiotic supplementation.The total number of subjects included in the trials is relevant, but the type of probiotic or association of probiotics used is widely different among the studies. As the Authors underlined, we have to face a "heterogeneity of the interventions and limitations in reporting of original studies”, so it was not possible to analyze neither the effects in each group separately nor the effects of individual probiotic species.probiotic currently includes bacteria associated with beneficial effects; according to the World Health Organization definition [probiotics refers to “live micro-organisms which, when administered in adequate amounts, confer a health benefit on the host". Probiotics may act by different mechanisms, as well as the Authors have pointed out, that means it is important to consider also if the action is the same or if there are differences of action between the different probiotics used in the different trials. Since a wide literature is available on probiotics, we can take for example the data related to Lactobacillus rhamnosus, the most used probiotic species.This statement is clearly expressed in the guideline, but it deserves some particular considerations. The term finition , the terLactobacillus rhamnosus isolated from different food sources and habitats from the human body  [A recent trial examined 100 strains of  vagina)  by carryLactobacillus rhamnosus, showing a likely causal role of two strains: Lactobacillus rhamnosus LRHMDP2 and LRHMDP were isolated in the dental pulp, furthermore the analysis of their genes showed differences with the probiotic Lactobacillus rhamnosus GG (ATCC 53103) which does not seem implicated in the pathogenesis of dental caries [A recent study analyzed the relationship between dental caries and l caries .Lactobacillus rhamnosus GG (ATCC53103), Lactobacillus rhamnosus LRV, Lactobacillus rhamnosus LRI, Lactobacillus casei BL23 (ATCC393), Lactobacillus casei LCY, and Lactobacillus casei LCA showed different genomic profiles, as well as different functional capabilities in glucose metabolism and in the ability to adhere to intestinal mucus [Also a comparison between Lactobacillus strains isolated from dairy products market of al mucus . As the Lactobacillus rhamnosus GG (ATCC 55103) is the most widely studied probiotic, both alone and in combination, but the data only from studies that use this type of probiotic would be insufficient.In the selected studies, different strains have been used in various combinations: it is necessary to consider different mechanisms of action and possible interactions in the bowel, in addition to the different effects of different combinations of micro-organisms. The species The duration of treatment in the late pregnancy is fairly homogeneous across all studies, and it varies from 2 to 6 weeks before the expected date of birth.Among the 15 selected studies, we considered the main clinical features associated with the diagnosis of eczema useful to analyze the question 1 of the guideline and we reported them in Table Lactobacillus rhamnosus HN001 was statistically significant while that one treated with Bifidobacterium animalis lactis subspec HN019 was not significant.It is more difficult to evaluate the results on the occurrence of eczema with its two clinical phenotypes: the IgE-associated and non IgE-associated, which differ both in the clinical features and in the evolution toward other allergic diseases. IgE-associated forms are more prone to the evolution in allergic rhinitis and asthma, especially if there is also a clear sensitization to inhalant allergens. The data shown in Table In 7 of these trials, the analysis can be restricted to the appearance of IgE-associated eczema: 4 studies showed a statistical significance , 17, whiFinally, the analysis of the presence of allergic sensitization assessed by specific IgE and/or skin prick test provided significant results only in two cases, while in 13 others it was not statistically significant.Another important issue discussed in this guideline is the detection of adverse effects. The analysis took into consideration also an extensive review of a study carried out by the Health Assessment Technology , which dThis guideline indicates the lack of clear evidence of efficacy of probiotic supplementation in the primary prevention of allergies. In the recommendation 1, the WAO guideline panel suggests "using probiotics in pregnant women at high risk for allergy in their children, because considering all critical outcomes, there is a net benefit resulting primarily from prevention of eczema ”; in the case of eczema this recommendation is also provided regarding the other two clinical questions.The low number of reported adverse effects confirms a high safety profile. The "net benefit" highlighted in the prevention of eczema appears to be overly optimistic when considering the wide heterogeneity of interventions and the inconsistence of the results in the statistical analysis. However, the GRADE system takes into account aspects that are helpful to "support each person in reaching a management decision consistent with his/her values and preferences" . NeverthEczema and allergic diseases represent a complex puzzle in which many factors interact. It is often difficult to separate the various factors involved, also because some of their effects may develop after a long time and are difficult to demonstrate. The guideline provides an important scientific synthesis that allows us to highlight the limitations of certain studies, but at the same time to get good points for further research. The time to suggest that preventive approach in a generalized way has not yet come, but these findings may be the starting point to search for more solid evidence and for new trials.Cuello-Garcia CA, Brożek JL, Fiocchi A, Pawankar R, Yepes-Nuñez JJ, Terracciano L, Gandhi S, Agarwal A, Zhang Y, Schünemann HJthe discrepancy in the conclusions between the WAO  and the the possibility to generalize recommendations to use probiotics when children are not at high risk of developing allergy; anda possibly “overly optimistic” assessment of benefits in the face of heterogeneity of interventions and inconsistency of the results in the analysis.We would like to thank Drs. Ricci and Cipriani for their insightful comments on the World Allergy Organization (WAO) guideline for the prevention of allergies with probiotics. In their letter, Drs. Ricci and Cipriani raised three concerns calling for additional clarification:food allergy in infants. However, in order to provide advice whether or not to use probiotics for the prevention of allergy, the WAO guideline panel considered all outcomes important to patients and their caregivers in this context. Those included development of food allergy but also other outcomes deemed by the WAO panel to be important: development of wheezing and/or asthma, allergic rhinitis, eczema, urticaria and anaphylaxis, nutrition status, infection with probiotic bacteria and other adverse effects.The main focus of the EAACI guidelines was on prevention of food allergy, whereas the WAO guidelines focused on a broader question whether or not probiotics should be used in the context of prevention of any allergy in otherwise healthy infants. The WAO and EAACI experts’ conclusion about the paucity of evidence was the same – both found not enough published evidence to confirm or reject an effect of probiotic supplementation on development of WAO advice to use probiotics in pregnant women, breastfeeding mothers and infants at high risk of developing an allergy was explicitly based on the likely lower risk of developing eczema. The recommendation stated, “….there is a net benefit resulting primarily from prevention of eczema” and that “…there is lack of evidence that probiotics prevent any other allergy”. This was a conditional recommendation meaning that clinicians should recognize that different choices would be appropriate for different individuals, often depending on their values and preferences. The WAO guideline panel, again explicitly, informed that the evidence supporting this advice was of very low quality, meaning that the panel was very uncertain how probiotic supplementation would actually affect all outcomes of interest. This implies that any further research – if done – is very likely to change both the strength and even the direction of those recommendations.Drs. Ricci and Cipriani also expressed a concern about the possibility of broadening the recommendations to use probiotic supplementation to situations when infants would be at low risk of developing allergies. The WAO guideline panel was again very explicit suggesting using probiotics in infants at high risk. Usually, one can extrapolate from the evidence in high-risk populations to lower risk groups with additional uncertainty about the expected effects introduced by the lack of direct evidence. In the case of WAO recommendations, the guideline panel thought that this additional extrapolation from already indirect evidence would make any information too unreliable to support an informed advice for low risk situations.The last comment is in relation to a possibly “overly optimistic” assessment of net benefits from probiotic supplementation in the face of heterogeneity of probiotics used in the studies and inconsistency of the results in the analysis. Dr. Szajewska and colleagues have previously expressed similar concerns whether the effect of probiotics on preventing eczema is a class effect or if one probiotic might have better efficacy than others might . We sharWe agree with Drs. Ricci and Cipriani that more research evidence is needed. Until it becomes available, parents and clinicians will need to make decisions under uncertainty about the true balance of benefits and downsides of using probiotic supplementation in pregnant women, breastfeeding mothers and infants with the intention to prevent allergies.


Objectives: There is widespread commitment—at least in principle—to “evidence-informed” clinical practice and policy development in health care. The intention is that only “appropriate” care ought to be delivered at public expense. Although the rationale for an appropriateness agenda is widely endorsed, and methods have been proposed for addressing it, few published studies exist of contemporary policy initiatives which have actually led to successful disinvestment. Our objective was to explore whether the direct involvement of policy stakeholders could advance appropriateness and disinvestment.Methods: Several collaborative engagements with policy stakeholders were undertaken to adapt and combine conceptual and empirical material related to appropriateness and disinvestment from the literature to create tools and processes for use in Canada and the province of Ontario in particular.Results: By combining inputs from the literature with colloquial evidence from policy stakeholders, a definition of appropriateness was developed and, importantly, endorsed by all the provincial and territorial ministers of health in Canada. Second, a reassessment framework was successfully implemented for identifying priorities for selective disinvestment.Conclusions: When scientific evidence was combined with colloquial evidence from policy stakeholders, progress was made on the design and successful implementation of policies for appropriateness and disinvestment. Our objective was to explore the involvement of policy stakeholders in the design and implementation of tools and processes related to appropriateness and disinvestment in Canada, and the province of Ontario in particular, and to assess their effectiveness.There is unprecedented interest in decreasing the frequency of unnecessary medical tests and procedures. International examples include the Choosing Wisely campaign , the UniThe authors performed a literature review and engaged with health technology assessment (HTA) experts to identify literature related to appropriateness and disinvestment. The general method was then to engage with Canadian policy stakeholders, primarily government staff but also HTA experts and clinicians with policy  influence, regarding the best ways of adapting and combining ideas and evidence from the literature to create tools and processes that would be directly useful in defining appropriateness and enabling disinvestment. Government staff (rather than independent researchers) served as knowledge brokers with the role of summarizing and presenting research evidence to groups of policy stakeholders. In engaging with policy stakeholders, the knowledge brokers emphasized an intention to combine technical concepts and empirical findings as reported in the literature with the stakeholders’ understanding of local contexts, their judgments of feasibility of implementation and their policy values.Regarding “appropriateness,” the process began with a preliminary definition drafted by the authors and based on a World Health Organization report  and concThe Ontario “reassessment framework” was based on 16 criteria proposed by Elshaug et al.  The authIn the context of health care, appropriateness is the proper or correct use of health services, products, and resources. Inappropriate care, in contrast, can involve overuse, underuse, and/or misuse of health services, products, and resources.Appropriateness is primarily determined by analyses of the evidence of clinical effectiveness, safety, economic implications, and other health system impacts.The practical application of appropriateness is made when these analyses are qualified by (a) clinician judgment, particularly in atypical circumstances, and (b) societal and ethical principles and values, including patient preferences.The definition of “appropriateness” was agreed by Canadian provincial and territorial ministers of health in March 2013:The Ontario reassessment framework was established as a set of seven criteria (triggers) adapted from the sixteen criteria proposed by Elshaug et al.  Local poThe application of the Ontario reassessment framework did not require information about every trigger for each possible candidate. The emphasis was on gathering sufficient rather than complete information to determine whether further assessment was warranted. The framework documents clearly stated that absence of evidence related to a trigger was not to be interpreted as evidence that the trigger was not met.The Ontario reassessment framework identified candidates that have since proceeded to selective disinvestment, resulting in an estimated $59 million per year (CDN$) freed up for investment elsewhere. For example, partial disinvestment has been implemented in Ontario for routine vitamin B12 testing, routine ferritin testing, and daily use of diabetes test strips by people with diabetes who do not take insulin. Each of these practices was a priority for reassessment based on triggers 1 and 6 of the reassessment framework and, after further involvement of local stakeholders , selective disinvestment was implemented.The Ontario reassessment framework was also used by the ad hoc ministry committee for disinvestment to identify priority candidates from a list of potential low value practices in Australia . Of 156 Three of the seven practices related to diagnostic imaging in children. Based on their presence on the Australian list of low value or harmful practices (trigger 1), regional variation with no apparent justification (trigger 4) and possible safety issues related to unnecessary medical radiation (trigger 3), reassessment was recommended for these practices, and is now in progress.The remaining four practices related to cancer screening or treatment. In each case the literature suggested that a practice was appropriate only for specific risk groups. The internal committee identified unexplained regional variation (trigger 4) but did not have access to risk information for Ontario patients receiving the practice. The committee consequently referred the reassessment candidates to another technical group for additional research.Significant challenges affect the implementation of disinvestment. These include political and social barriers which require the involvement of policy stakeholders for resolution. Lomas et al. describe three different categories of evidence: medical effectiveness research (context-free scientific evidence); social science-oriented research (context-sensitive scientific evidence); and the expertise, views, values, and realities of stakeholders  . The thrIN THE CONTEXT OF HEALTH CARE, appropriateness is the proper or correct use of health services, products AND RESOURCES. Inappropriate care, in contrast, can involve overuse, underuse and/or misuse of health services, products AND RESOURCES.Appropriateness is primarily determined by analyses of the evidence of CLINICAL effectiveness, SAFETY, economic implications, and other health system impacts.The PRACTICAL APPLICATION of appropriateness is made when these analyses are qualified by (A) CLINICIAN JUDGMENT, PARTICULARLY IN ATYPICAL CIRCUMSTANCES AND (B) societal and ethical principles and values, INCLUDING PATIENT PREFERENCES.The revisions to the appropriateness definition that were made based on policy stakeholders’ input, as shown in capital letters below, clearly reflect colloquial evidence including values, pragmatics, resources, and professional experience and expertise.Notably, the finally agreed definition of appropriateness integrated each of the three categories of evidence: clinical effectiveness evidence was a form of context-free scientific evidence, economic implications represented context-sensitive scientific evidence, and clinician judgement and patient preferences were examples of colloquial evidence.The triggers of the Ontario reassessment framework, which were selected through deliberation with local policy stakeholders, also included each of the three categories of evidence. Trigger 1, “an evidence-based recommendation by an external body,” will often involve consideration of context-free scientific evidence from other jurisdictions. Trigger 6, “change would be cost saving,” prompts consideration of context-sensitive scientific evidence. Trigger 2, “nominated by a local clinical expert,” may be exclusively evaluated based on colloquial evidence, for example, reflecting clinicians’ professional experience/expertise.Local stakeholder involvement was essential for successful implementation of selective disinvestment. For example, the Ontario reassessment framework identified vitamin B12 and ferritin laboratory testing as reassessment candidates in November 2011. Implementing changes, however, involved months of work including validation by local HTA experts  and govePolicy stakeholder input also contributed to a decision not to disinvest. One priority for reassessment was identified based on published evidence that the practice was of low value; however, given the absence of evidence of harm in the literature, and local policy stakeholder advice that the practice was provided infrequently for a small number of people in Ontario, reassessment and selective disinvestment were not pursued.ad hoc committee, the provision of context-free scientific evidence related to it, opportunities for group discussion and a high degree of agreement between the internal ad hoc ministry committee members all strengthened committee members’ resolve and interest in specific disinvestment opportunities.The mechanisms and processes for obtaining policy stakeholder contributions were also important. For example, each of the first five priority candidates identified by the reassessment framework were already being considered for reassessment and/or disinvestment at some level within government departments, but had not been acted on. Collectively, the formation of the Although the ad hoc committee on disinvestment was initially formed to respond to a “push” of research evidence from the authors , it has Scientific evidence combined with and qualified by colloquial evidence from policy stakeholders was a recurring theme: (i) A definition of appropriateness was developed by using policy stakeholder input to modify and adapt text from the literature—the result was a definition which included explicit reference to including colloquial evidence in determining appropriateness. (ii) The Ontario reassessment framework was developed through work with local policy stakeholders to adapt a framework from the literature for Ontario—the result was a framework which included criteria based on colloquial evidence. (iii) Implementation of selective disinvestment integrated scientific evidence with colloquial evidence in the forms of practical advice and values inputs from local policy stakeholders.By using a method that combined and qualified scientific evidence with colloquial evidence from policy makers, progress was made on appropriateness in Canada and Ontario. Although the definition of appropriateness and the design of the reassessment framework were specifically tailored for the Ontario and Canadian context, the fact that these arrangements succeeded in integrating policy-makers' colloquial evidence and have been successfully implemented, may make them a good starting point for related work in other jurisdictions and settings.Dr. Elshaug reports grants from The Commonwealth Fund, during the conduct of the study; personal fees from NPS MedicineWise, grants from HCF Research Foundation, grants from National Health and Medical Research Foundation (NHMRC), other from Cancer Australia, personal fees and other from Capital Markets Cooperative Research Centre, personal fees from Australian Commission on Safety and Quality in Health Care, outside the submitted work.


The use of evidence in decision-making at the program management level is a priority in health care organizations. The objective of this study was to identify potential barriers and facilitators experienced by managers to the use of evidence in program management within health care organizations.The authors conducted a comprehensive search for published, peer-reviewed and grey literature that explores the use of evidence in program management. Two reviewers selected relevant studies from which data was extracted using a standard data abstraction form and tabulated for qualitative analysis. The results were summarized through narrative review. The quality of the included studies was assessed using published criteria for the critical appraisal of qualitative, quantitative and mixed methods research.Fourteen papers were included in the review. Barriers and facilitators were categorized into five main thematic areas: (1) Information, (2) Organization – Structure and Process, (3) Organization – Culture, (4) Individual, and (5) Interaction.This paper reviews the literature on barriers and facilitators to evidence-informed decision-making experienced by program management decision-makers within health care organizations. The multidimensional solutions required to promote evidence-informed program management can be developed through an understanding of the existing barriers and facilitators of evidence-use. Health care organizations are complex and comprised of highly trained individuals whose responsibilities include management of conflicting demands for services. Consequently, the types of decisions that must be made are often complicated and they are different, depending on the level within the organization. These could range from governance-related decisions by the Board of Directors or Trustees to decisions on treatment options for individual patients by a physician.Over the last two decades (and particularly in Canada), clinicians have begun to take on increased roles in management structures, mainly because they are often viewed as major “drivers” of health care expenditures. This has resulted in greater involvement in budget and operational decisions, as well as the development of the concept of program management. It is defined as “a type of management structure in which services are grouped into programs by medical specialty, specific diagnosis or populations groups” , pp. 186Many organizations have established evidence-informed processes for making acquisition decisions around new equipment, supplies, and pharmaceuticals. However, at the program management level, such processes are less well-developed ,4. In faThis may be, in part, attributed to perceived barriers to the use of evidence by program level decision-makers. Clinicians involved in determining the most appropriate patient care options face different barriers, such as patient conditions that may fall outside of clinical practice guidelines or rare conditions for which guidelines have not yet been developed. In contrast, those faced by program managers often relate to complex organizational issues involving multiple stakeholder communities with competing interests. Nonetheless, the need to find ways of overcoming these barriers has been underscored by a widely held view that evidence-informed program management decision-making may serve to improve the acceptability of decisions to such stakeholder communities .Evidence-informed program management decision-making requires two sets of skills: 1) those for identifying and critically assessing the evidence and 2) those for applying it to their local context in a way that reflects an awareness and understanding of factors potentially affecting uptake, implementation or sustainability of the evidence within a complex setting ” [making]” , p. 93.Internal resource constraints included human resource constraints -22,25, fOrganizational culture was identified as a barrier in program management within health care organizations, particularly the decision-making culture of organizations -18,21 anDecision-makers in health care organizations also experienced barriers to evidence use at the individual level. A deficit in the skills and experience of decision-makers in research literacy and research utilization, and a lack of formal management training were expressed as barriers to evidence-use in program management ,16,18-22“Capacity was lowest for the domains related to: acquiring research (subsection I); assessing the reliability, quality, relevance, and applicability of research evidence (subsections III and IV); and summarizing results in a user-friendly way”[.ndly way”, p. 3.One of the included studies also identified issues related to the interaction between researchers and decision-makers as barriers to the use of evidence in health care organizations. The gap between researchers and decision-makers, in terms of a lack of contact and mutual understanding was identified as a barrier to evidence use .The majority (10) of the included studies identified facilitators of evidence use for program management. The majority of facilitators of evidence-use experienced by managers were informational (10), for example,“Public health decision-makers value the use of systematic reviews to facilitate the decision-making process. They indicated that systematic reviews were particularly useful because they integrate the results of many studies into one, which allows them to bypass the stage of looking at individual studies. This saves them time and gives them more confidence knowing their decisions are based on the culmination of many studies instead of just a few”[st a few”, p. 159.Organizational structure and process or organizational culture were identified as facilitators of evidence use in eight of the studies. One study concluded that evidence use “in health service organizations was more complex and much more sensitive to organizational factors and processes than previous studies seemed to affirm” , p. 407.Within each theme, different specific types of facilitators of evidence use were detailed. Table The studies included in this review identified information related facilitators of evidence-use experienced by decision-makers in health care organizations. Access to information -21,24,28“research-producing organizations knowing not only who their target audience(s) are and what their needs are concerning research evidence, but also what questions require answers, and what kind of answers are optimal for different types of decisions”[.ecisions” p. 9.The advancement of research methods to meet the needs for evaluating complex interventions  was alsoOrganizational structure and processes also emerged as facilitators of evidence use for program management. Facilitators of evidence-use that relate to the structure and processes of health care organizations included administrative support ,25,28 anThe studies included in this review also reported that evidence-informed decision-making is influenced by an organization’s culture. An organizational culture that is supportive of evidence use, providing required supports and demonstrating through action that evidence-use is valued ,24,27,28Facilitators to evidence-informed decision-making were also identified at the individual level. Individual skill building for decision-makers in research literacy, research utilization and research application was identified as a facilitator of evidence use within health care organizations. The use of evidence for decision-making was also facilitated through the building of individual researcher’s skills to produce evidence that is useful to decision-makers and disseminate evidence to decision-makers more effectively ,24,25,27Interaction between researchers and decision-makers was identified as a facilitator of evidence use. Opportunities for direct contact and communication between researchers and decision-makers were found to facilitate evidence-informed decision-making ,26. SustThis review fills a gap in the literature by synthesizing recent evidence on barriers and facilitators of evidence use at the program management level. To date, the literature has focused largely on decision-making at the clinical  and poliFindings from the included studies suggest that decision-makers in health care organizations experience barriers to using evidence at both the organizational and individual level and that efficient ways of integrating evidence-informed decision-making into organizational processes is required. Managers need organizational leaders to not only support them in using evidence, but also to address human resource challenges that inhibit evidence-use. Evidence use could be further increased through the development and implementation of formal organizational processes for decision-making and organizational investment in systems to support evidence-use. In addition, improvements to internal communication mechanisms and processes within organizations and a demonstrated commitment to evidence development and sharing across the organization could be made.While addressing barriers to evidence-use associated with organizational culture requires executive leadership, those at the individual level require strategies directed at individual skill building. Opportunities for increased interaction between researchers and decision-makers would also serve to promote evidence use.A key strength of this review is its focus on program management decision-making. In addition, the inclusion of empirical studies without limiting by study design is another strength. A major limitation of this review is the broad search strategy that was employed. The lack of a standard definition of program management in the literature led the authors to include a broad range of search terms, which ultimately resulted in the identification of a large number of studies not on the topic of interest for this review. Future reviews would benefit from a narrower search strategy. Other limitations of this review include the fact that it was limited to the English language. Therefore, some relevant studies published in other languages may have been excluded. The date limit imposed may also have resulted in relevant studies being excluded. While the reason for limited country representation and Canadian dominance across included studies is not clear, it may be due to an emphasis on evidence-informed decision-making at the program management level by funding agencies in Canada over the last decade. In addition, there may have been differences in the terminology used to describe these activities in other countries. However, the search strategy included a broad range of terms, reducing the likelihood that this was the case.The findings from this review of 14 studies exploring the use of evidence in decision-making in program management suggest that barriers and facilitators to evidence use in program management decision-making within health care organizations can be categorized into four distinct groups: (1) Informational, 2) Organizational, (3) Individual, and (4) Interactional. Understanding the barriers and facilitators to evidence-use experienced by managers is an essential first step in developing strategies to promote such evidence-informed decision-making within health care organizations. Health care organizations seeking to improve evidence-informed decision-making by their program managers could use this comprehensive list of barriers and facilitators to identify and address their organization-specific challenges. The findings also confirm that evidence-informed management requires more than encouraging research utilization within organizations. To address informational barriers to evidence-use experienced by managers various sources of evidence need to be considered at different times throughout the decision-making process . Researc OrganizaThe authors declare that they have no competing interests.SH participated in the design of the study, selection of the articles for inclusion, data abstraction, data analysis and interpretation and drafted the manuscript. TS participated in the design of the study, data abstraction and interpretation and helped draft the manuscript. ZM participated in data interpretation and helped draft the manuscript. DM participated in the design of the study, selection of the articles for inclusion, data analysis and interpretation and helped draft the manuscript. All authors read and approved the final manuscript.Senior Author: Devidas Menon.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/14/171/prepubLiterature search strategy.Click here for file


There is currently limited evidence regarding the extent Real World Evidence (RWE) has directly impacted the health and social care systems. The aim of this review is to identify national guidelines or guidances published in England from 2000 onwards which have referenced studies using the governmental primary care data provider the Clinical Practice Research Datalink (CPRD).The methodology recommended by Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was followed. Four databases were searched and documents of interest were identified through a search algorithm containing keywords relevant to CPRD. A search diary was maintained with the inclusion/exclusion decisions which were performed by two independent reviewers.N =7) relevant to diseases of the Central Nervous system (CNS). The 43 studies provided evidence of disease epidemiology, incidence/prevalence, pharmacoepidemiology, pharmacovigilance and health utilisation.Twenty-five guidance documents were included in the final review (following screening and assessment for eligibility), referencing 43 different CPRD/GPRD studies, all published since 2007. The documents covered 12 disease areas, with the majority  was noticed. However, there seems to be an increasing trend in the use of healthcare system data to inform clinical practice, especially as the real world validity of clinical trials is being questioned. In order to accommodate this increasing demand and meet the paradigm shift expected, organisations need to work together to enable or improve data access, undertake translational and relevant research and establish sources of reliable evidence. It is generally agreed that the provision of healthcare should be based on evidence, principally so that a patient receives the best advice or treatment for their condition . As mediGuidelines and guidances are documents which incorporate current evidence from reviewed sources in order to develop clear and comprehensive recommendations on the prevention, treatment and care of patients with specific diseases and conditions . These dAlthough the process for developing these documents has moved from being primarily expert knowledge based to being primarily evidence based, there is still concern regarding the sources of evidence. There is heavy reliance on Randomised Controlled Trials (RCTs) for generating evidence for clinical guidelines as (according to many ‘hierarchies’ of evidence) they are thought of as the ‘gold standard’ . Howeverthey help to strengthen current understanding of healthcare delivery and the outcomes of patients ,they greatly increase the potential of generating new knowledge as researchers can work to answer important clinical questions (which may otherwise not have been possible)  andthey can support the development of evidence-based personalised medicine through the linking of EHRs to genomic datasets . EHRs maSince the transition of paper healthcare records to Electronic Health records (EHRs) it has been possible to create large datasets containing important information such as clinical events, laboratory results, treatment history, etc.  These arLastly (and in this case more importantly), RWD could help with the dissemination of key information by bridging the knowledge gap for clinicians and by improving the quantity and quality of evidence used in guidelines and guidances. Best evidence can only be generated when starting with the best data . An examCPRD  is one of the largest longitudinal databases in the world containing anonymised EHR data  for 11.3 million patients in the UK . CPRD haIn this context, this study aimed at systematically reviewing the literature to identify guidelines or guidances published from 2000 onwards in England which have referenced studies using RWD from the CPRD. The review has focused particularly on governmental organisations (in terms of data providers and guideline developers) as the UK healthcare system is one of the best integrated systems globally.Guidances and guidelines were categorised according to the definitions provided by the NICE.NICE Clinical Knowledge Summaries: “A readily accessible summary of the current evidence base and practical guidance on best practice in respect of over 330 common and/or significant primary care presentations” .Technology appraisals guidance: “Recommendations on the use of new and existing medicines and treatments within the NHS” .Clinical guidelines: “Recommend how healthcare professionals should care for people with specific conditions” .The categories were as follows:Results which did not fit these categories but focused on the delivery of medications were defined as ‘prescribing’ guidelines.This systematic review adopted the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines  and was “CPRD”, “Clinical Practice Research Datalink”, “GPRD”, and “General Practice Research Database”. Indicatively for Medline the following algorithm was used: ) AND “guideline”[Publication Type]) AND (“2000/01/01”[Date - Publication]: “3000”[Date - Publication]).Guidelines and guidances of interest were identified by a systematic search of four databases from 1st January 2000 to 21st March 2016 (last day of search update): NICE Evidence Search, Medline PubMed, Embase and the National Clinical Guideline Centre. All four databases were searched for guideline/guidance documents referencing studies using data from CPRD using combinations of the following keywords: A search diary recording the search results for each database and the meeting of the inclusion/ exclusion criteria for each document was maintained. The specific inclusion criteria were as follows: 1) a UK guideline/ guidance and 2) references research using data from CPRD or GPRD. Documents were excluded if they met one or more of the following criteria: 1) Irrelevant, 2) Not written in English, 3) Not a guidance or guideline (any other primary or secondary research paper), 4) Only mentions CPRD/GPRD , 5) Not available (as being updated) and 6) Draft documents or in consultation. Reference lists of all studies previously identified as having met the inclusion criteria were also manually reviewed for additional relevant documents.The search and assessment of eligibility for included studies were performed by two reviewers working independently. Any duplicate documents were consolidated. All decisions were reached by consensus, with the addition of a third reviewer where required. A relevant PRISMA flow chart was constructed to detail the number of papers retrieved and the steps undertaken.All data were extracted by two independent investigators and consensus was reached after the involvement of a third investigator where required. Full text was available for all documents. The following information was extracted for each identified document meeting the inclusion criteria: title, year of publication, disease area, the CPRD studies cited, the exact sentences referencing and the type of guideline/ guidance and the references. Each guidance document was categorised by disease area following the categories of the British National Formulary (BNF). The information was then summarised and tabulated in a standard form. In the assessment of the results, studies referenced in more than one guideline document, were treated separately as providing different evidence in each.A purely descriptive approach was adopted for data synthesis. Sums and means were derived where appropriate. No further statistical analysis was undertaken. As this review did not include any primary research, no form of quality assessment was necessary.The PRISMA search strategy yielded 297 documents in total from four bibliographic databases, of which 293 were not duplicates. Following screening, 178 documents were excluded as they were not a guidance or guideline. A further 90 records were excluded based on the other exclusion criteria. In total, 25 documents were included in the final review  were guidances. The guidelines/guidances were published between 2007 and 2015 with 2008 being the only year a guidance/guideline referencing CPRD had not been published. No relevant documents were identified before 2007. The majority of documents were published in 2012 and 2015 (N = 7 for each year). Approximately 16 of the documents were published in the last 3 years. Of the guidelines identified, ten were clinical and the remainder were prescribing. Of the guidances identified, seven were Clinical Knowledge Summaries, two were prescribing, three were Technology Appraisals and the remainder were clinical.Baseline characteristics and detailed information of the included studies, including how the evidence from these studies was used to inform the documents, are listed in Table N =7) were focused on diseases of the Central Nervous system (CNS). The guidances covered eight topics with the majority again focusing on diseases of the CNS. The Guidelines covered seven topics with the majority focusing on Infections, the Central nervous system and Ear, nose and oropharynx (N =4 for each topic). The top seven disease areas are listed in Table The guidelines and guidances covered 12 topics (grouped according to the BNF). The majority of the documents (N =15) was ‘Suspected Cancer: recognition and referral’ (2015). Only three guidances referenced more than one CPRD study ,’Osteoporosis: assessing the risk of fragility fracture’ (2012), ‘Rivaroxaban for the prevention of stroke and systemic embolism in people with atrial fibrillation’ (2012), and ‘Sore throat – acute’ (2012).Forty-three studies using CPRD data were referenced in the 25 documents with three studies receiving the most citations , 22, 24.N = 32) while the remainder were in relation to pharmacovigilance (N =12), pharmacoepidemiological evidence (N = 5), incidence or prevalence rates (N = 3) and health utilisation (N = 1).The evidence used from CPRD studies can be grouped into five categories. Almost three quarters provided information on disease epidemiology . This is not because there is a lack of studies on the subject. The possible reasons for this have already been discussed above and as a result, there needs to be a review of the kind of evidence used in guidelines and guidances, for if the data is not for the purpose it was created then it is a waste of a potentially health-changing resource.Studies using CPRD data were not referenced in guidelines in other disease areas at all, such as nutrition and blood, eye, and anaesthesia. For most of these disease areas, the reason is quite clear. CPRD currently has no or limited data on diseases and treatments that are mainly administered in secondary care (e.g. drugs administered through the eye) and there are no efficient centralised databases containing such information. However, the linking of datasets from varied health settings can provide a fuller picture of disease and health outcomes in the general population and therefore provide even more robust evidence . A good This shows that the ability to link datasets from a variety of sources provides immense opportunity to not only get a fuller picture of a patient’s medical history, but also investigate the interactions and associations between different treatments/diseases in different clinical settings and possibly developing predictors of health outcomes .Looking towards how clinical evidence can be improved, one would directly look into the organisations providing the data. As CPRD and other data providers continue to expand their linkages to other data sources and the benefits of linked data sources increase in recognition, funds should be invested in creating datasets in all sectors of health. This will enable healthcare professionals to make sound decisions based on RWD, regardless of their line of work. The Health and Social Care Information Centre (HSCIC) who manage and maintain the balance between the sharing of information for community benefit and respecting the confidentiality and wishes of patients have been key in enabling research through linked data in the UK .There are several strengths to this study. Firstly, it was conducted using the gold-standard method for conducting systematic reviews . The PRIHowever, limitations of the review need to be acknowledged. Firstly, this review focused only on guidelines and guidances which used evidence from studies using CPRD data. There are other longitudinal databases in the UK such as The Health Improvement Network (THIN) and QResearch. It also did not look into what type of data was used in each study (e.g. use of linked data). For future studies, it would be interesting to investigate whether evidence from other longitudinal databases have been used to inform guidelines/guidances. This study also focused on guidelines and guidances published for health and social care in England. Future studies could compare how RWD is used not only nationally but also internationally and identify the trends and differences that may exist. Lastly, guidelines and guidances which were currently ‘under review’ were not included in the review as they were liable to change once published. Therefore it would be worth conducting the review again at such a time when these guidelines/guidances become available to see how they affect the current results.In this systematic review, we confirmed that Real World Evidence from the Clinical Practice Research Datalink has been used inconsistently but increasingly in the last decade, to inform guidelines and guidances published in the United Kingdom. The increased uptake in recent years, noted in our results, shows that this area of healthcare is changing and this review captures a phase in this transition. To capitalise on the potential value of using Real World Evidence, researchers need to ensure they undertake research of translational value to the healthcare community. Organisations which develop guidelines should also work to identify Real World Evidence sources which will give a more realistic view of how an intervention works in actual healthcare settings. Finally, key points extrapolated from our review include increasing the quality of available Real World Evidence  and maintaining public trust (which will be key for wider uptake).ACC, American college of cardiology; AHA, American heart association; BNF, British national formulary; CALIBER, cardiovascular disease research using linked bespoke studies and electronic health records; CNS, central nervous system; CPRD, Clinical Practice Research Datalink; EHR, electronic health record; HES, Hospital Episodes Statistics; HSCIC, Health and Social Care Information Centre; MINAP, myocardial ischaemia national audit project; NHS, National Health Service; NICE, National Institute for health and Care Excellence; ONS, Office for National Statistics; PRISMA, preferred reporting items for systematic reviews and meta-analyses; RCT, randomised controlled trial; RWD, real world data; RWE, real world evidence; THIN, the health improvement network


Policymakers such as English healthcare commissioners are encouraged to adopt ‘evidence-based policy-making’, with ‘evidence’ defined by researchers as academic research. To learn how academic research can influence policy, researchers need to know more about commissioning, commissioners’ information seeking behaviour and the role of research in their decisions.In case studies of four commissioning organisations, we interviewed 52 people including clinical and managerial commissioners, observed 14 commissioning meetings and collected documentation e.g. meeting minutes and reports. Using constant comparison, data were coded, summarised and analysed to facilitate cross case comparison.The ‘art of commissioning’ entails juggling competing agendas, priorities, power relationships, demands and personal inclinations to build a persuasive, compelling case. Policymakers sought information to identify options, navigate ways through, justify decisions and convince others to approve and/or follow the suggested course. ‘Evidence-based policy-making’ usually meant pragmatic selection of ‘evidence’ such as best practice guidance, clinicians’ and users’ views of services and innovations from elsewhere. Inconclusive or negative research was unhelpful in developing policymaking plans and did not inform disinvestment decisions. Information was exchanged through conversations and stories, which were fast, flexible and suited the rapidly changing world of policymaking. Local data often trumped national or research-based evidence. Local evaluations were more useful than academic research.Commissioners are highly pragmatic and will only use information that helps them create a compelling case for action.Therefore, researchers need to start producing more useful information.To influence policymakers’ decisions, researchers need to 1) learn more about local policymakers’ priorities 2) develop relationships of mutual benefit 3) use verbal instead of writtencommunication 4) work with intermediaries such as public health consultants and 5) co-produce local evaluations.The online version of this article (doi:10.1186/s12913-015-1091-x) contains supplementary material, which is available to authorized users. Since the mid-1990s, policymakers have been encouraged to increase their use of evidence , 3. ThisThose charged with delivering ‘evidence-based policy-making’ within the English National Health Service include local healthcare commissioners that sit within organisations called Clinical Commissioning Groups. Currently there are over 200 local Clinical Commissioning Groups across England that are responsible for about £110 billion annually. Clinical Commissioning Groups plan and allocate funding for healthcare providers from hospital and community services. A major objective of the recent reorganisation of the National Health Service  was to increase the level of clinical input into commissioning . Within Few studies of local healthcare commissioners have taken place, and the majority are interview based . They haLearning more about how and where commissioners access information is important to increase understanding about the contribution that researchers and research evidence can make to policy-making. Using observational, interview and documentary data, this paper offers ‘real life’ findings from one of the first studies of clinical commissioning under these new arrangements in the English NHS. The aims of the study are to elucidate the reasons that prompted commissioners to seek information, to clarify which sources and types of knowledge commissioners commonly consulted and to describe the use of research evidence in decision-making. Throughout this paper, we use the terms ‘knowledge’, ‘information’ and ‘evidence’ interchangeably, as commissioners do, and we use the term ‘research’ to distinguish academic research. Although knowledge acquisition  and knowledge transformation  are different concepts, in practice we found the two happened concurrently, but for the sake of clarity we focus here on knowledge acquisition; findings on knowledge transformation are published in the full study report  and anotThe data presented here come from a larger study of knowledge exchange between commissioners and external providers such as commercial companies. We used a mixed qualitative methods case study approach . The stuCarnford - struggling financially and highly collaborative with its healthcare providersDeanshire – relatively confident as a commissioning organisation, focused on governance, carrying out some innovative projectsNorchester - financially challenged, piloting new ways of commissioning contracts with an emphasis on  evidence-based policy makingPenborough – creating an integrated network of health and social care provision with an emphasis on public involvement and innovationWe could only selected those commissioning organisations that had contracts with the commercial providers that participated in the larger study. In total, these commercial providers had less than a dozen contracts with over 12 different commissioning organisations that were geographically located across the country. We selected the commissioning organisations on the basis of geography , size , population  and length of time with contracted provider . Using pseudonyms, the four commissioning organisations selected were:We collected interview, observation and documentary data from February 2011 to May 2013. Data were collected by LW and EB, who are experienced qualitative researchers.The contracts between the commissioning organisation and external providers involved in the larger study of knowledge exchange were the focus of our sampling strategy. Through snowball sampling, we identified candidates for interview from within commissioning organisations, such as the lead NHS contact for the contract. As our enquiry broadened, we identified those with particular roles who could further illuminate the knowledge acquisition behaviours of commissioners .Candidates were sent information before interviews and written or recorded consent was obtained at interview. The topic guide, which was revised as new questions emerged, covered the types of knowledge wanted by commissioners, information sources and how information was accessed and influenced decisions. Face to face and telephone interviews lasted 20–60 min; they were recorded and transcribed by an external transcriber. In total, we interviewed 52 people Table .Table 1In = 7), clinical operations committees (n = 3), unscheduled-care boards (n = 3) and a project board (n = 1). Permission was obtained verbally before attending meetings and re-confirmed at the start of each meeting. Observation notes included details of participants, room layout, verbal exchanges and researcher reflections. Notes were usually typed up within a week. All interview and observation participants were given pseudonyms.We conducted 14 observations of commissioning meetings including director-level boards  cycles . The stuReasons that commissioners seek informationSources and types of information soughtWays that information was exchangedUse of academic research and local evaluationsOur results are structured along the following themes identified in the analysis:Commissioners purposefully sought out information either because they were told to take a particular course of action  or because they wanted to find out how best to proceed and no predetermined course existed. Both occurred frequently across the sites. In both scenarios, commissioners used information to try to build a compelling, cohesive and persuasive case to a) navigate a way through the system b) justify decisions and c) convince others to approve and/or follow the suggested course. Decision-making occurred through repeated cycles of finding information, persuading others, justifying decisions, finding more/different information, persuading others etc., as proposals moved through the commissioning decision-making process.clinicians and healthcare provider organisations  needing to make the necessary changesservice users and the public, for example to smooth the introduction of potentially unpalatable changes to servicesthe press to avoid negative media attention‘evidence purveyors’ such as public health that lobbied commissioning organisationsRegardless of the motivation for information seeking, because commissioning decisions were publically accountable as commissioning organisations are statuatory bodies, they had to “stand up to extremely close, possibly legal scrutiny and have to be owned by the organisation” . Decisions had to be resilient to challenges from many possible directions. These included challenges from:Again it was an example of centralisation of things coming down from above. They said that we had to put out two or three services to any qualified provider [open competition]. So [commissioning organisation X] decided what we should do. And one [option] was ultrasound, non-obstetric ultrasound, which we have an absolutely excellent service provided by the hospital, even routine ones are done within a week, brilliant service. So why do we do it? So there’s all this process and people – they ended up with a list of seven providers. But, you know, it was a complete waste of time and money. Substantial pressure also came from those with a policy or performance management role, especially senior policymakers such as the Department of Health and/or NHS England area teams. Local circumstances dictated the extent to which such top-down attention was welcomed. For example, several board documents from one commissioning organisation noted that local activities had substantial “interest from Number 10” [i.e. the Prime Minister’s office], which spurred momentum. But sometimes this was more problematic, as with one commissioning organisation’s response to the national mandate to tender a specific list of services to open competition, which proved troublesome as this organisation was happy with their services in these areas.GP commissioner: I just see how the world works and the pressure that organisations are put under if they don’t conform. You know, these words like, “You are at risk. Your organisation is at risk. You are at personal risk for this.” And that’s not a nice thing to be on your shoulders….Interviewer: Yeah, so you’re identified as risky?GP commissioner: Yeah and then you get lots of phone calls, and the chief exec gets phone calls, and he has to speak to you going, “Oh I’m getting a lot of flak about this. Can you not just smile sweetly and say you’ll engage?” This organisation routinely resisted national directives from the Department of Health and NHS England and consequently experienced pressure to comply.In addition to managing these external forces, commissioners also had to convince their internal colleagues. For example at one board meeting of senior commissioners, a junior commissioning manager putting forward a business case for a lymphoedema service costing £92,000 annually. This was up against several other competing proposals suggested by colleagues from the same organisation, each trying to persuade the board that their particular business case should be funded.In summary, commissioners had to influence and collaborate with external and internal interested parties to build a compelling case for taking a particular course of action. Not all challenges came into play, and there was also variability in the strength of each as a proposal traversed the different stages in the decision-making process. But invariably, at the centre of this web of pressurising forces, the commissioner juggled competing agendas, priorities, power relationships, demands and their own inclinations to make the best decision circumstances allowed. Just as there is an ‘art of medicine’, this was the ‘art of commissioning’ see Fig. .Fig. 1PrThe principal outcomes of the study indicated: a significant difference between the patient groups in the number of hospital re-admissions and a significant reduction in social care costs. To a large extent, commissioning was a matter of pragmatically pulling together the appropriate knowledge and information that would satisfy competing agendas and help manoeuvre the proposal. Unsurprisingly, as that knowledge was juggled and steered, it was re-considered, re-framed and re-shaped, usually becoming quite different as a result. For example, an evaluation of a rehabilitation project for those with long term conditions was carried out by public health showed a small reduction in hospital admissions for service users after the intervention, but offered no comparable results for the comparison group  for examples of similar pathways devised by other commissioning organisationsCommissioners sought knowledge and information from many sources to determine, clarify, substantiate and defend their case. For example, in re-designing new services for a patient pathway, commissioners might use:For commissioners, the word ‘evidence’ often meant any source of information other than personal experience or anecdotes. When asked, they commonly named NICE guidelines, best practice guidance, hospital and primary care data and Department of Health documentation as sources of ‘evidence’. An overview of the sources and types of knowledge cited by commissioners and observed in commissioning meetings is detailed in the following tables, classified into ‘people’ and ‘organisations’. These are not exhaustive, as other sources and types of knowledge undoubtedly exist and the named sources offer other types of knowledge not directly referenced by participants. As this is not a quantitative survey , we makeAmongst national sources, commissioners appeared to trust NICE, the King’s Fund and NHS Improving Quality (formerly NHS Institute for Innovation and Improvement), which were perceived as independent. Clinical commissioners were more sceptical of output from the Department of Health, sometimes because they felt that this ‘evidence’ did not stand up to scrutiny.Cindy: we’re not entirely sure but it seems to have some evidence that something happening…It’s soft measures such as empowerment. Lily: telemedicine helping people to feel they can take control but it’s hard to measure. Teresa: Marvellous! Cindy: so watch this space. Local information often trumped national evidence, academic research or information from other localities. For example, ‘telehealth’ was a national ‘must do’ with limited clinical support in one commissioning organisation. An unscheduled-care board was presented with a review of the academic research behind ‘telehealth’ presented by a public health consultant. This suggested weak evidence of clinical and cost effectiveness. A month later, the board became more enthusiastic, when provided with before and after data from a local evaluation of just the first eight local patients.Commissioners acquired information through conversations, stories and documentation including online sources. Verbal exchange was particularly well suited to the fast paced, rapidly changing environment in which commissioners worked. Conversations were an important way of getting information quickly through chance encounters, formal meetings and informal gatherings. The contingent nature of these exchanges however meant that if a different combination of individuals had happened to meet, different knowledge would have been acquired and perhaps a different set of decisions would have been made.And I have often thought in the past you need the story of the change. Because ideally from the time you’ve gone through health scrutiny committee, the CCG, other local groups, other stakeholder groups and especially if you get to a procurement exercise where actually you might draw services to a close and you know people may be…made redundant and there’s some heavy duty consequences for people, you need a compelling story. And often that is much more powerful than data that you want to throw at people, and so having the clinical stories is really important and the patient’s story is really, really key. Knowledge was also acquired through stories. A commissioning manager said that patient stories were important to maintain momentum through the lengthy, repeated decision-making cycles. Conversations and stories are different in that the first is a form of engagement with other people while the second is a way of packaging and communicating information and convincing other people. This raises an important distinction between stories as a dubious source of  evidence and stories as a preferred way to convey sound evidence. We found that often stories were privileged as much as other sources of information.Although substantial knowledge acquisition occurred through informal conversation, more formal verbal exchanges such as meetings were recorded to leave a paper trail documenting discussions for accountability. Commissioners had extensive access to other documentation, much of it unsolicited and sent electronically, including national and regional directives, meeting papers, business cases, reports, patient satisfaction surveys, guidelines, pathways, and performance, activity, financial and referral data. To capture commissioners’ attention, these documents often had a summary of no more than one side of A4 with clearly bulleted action points.British Medical Journal and the British Journal of General Practice, no other academic journal was named. This was not surprising as commissioners were often unable to access full length articles without journal subscriptions. Participants frequently mentioned the Health Service Journal as a source of research evidence, although this professional journal usually showcased best practice and service innovations with only the occasional synopsis of academic research (www.hsj.co.uk).Few participants mentioned accessing academic research directly, and those that did said that they searched Google and Google Scholar. Apart from the the challenges of finding applicable, relevant research for commissioning , given that most health related academic research was clinically focused“academically very robust” interventions not working in practice the challenge of drawing conclusions from literature reviews with substantial variations in interventionslack of commissioner time and skills to do comprehensive reviewsdifficulties in interpreting the “spin” within abstracts the length of time needed for locally commissioned research to produce meaningful outputschallenges in applying negative or inconclusive findings, as commissioners’ focus was on identifying initiatives that might workWhen asked about use of academic research, participants mainly discussed the difficulties. These included:Participants talked about academic research being implicit in the system. Clinicians and commissioners were “expected to keep on top” of their area  sometimes by having their “ear to the ground” with national and local networks . The expectation was that clinical commissioners would bring a research perspective. In an observation of one commissioning meeting, this higher level of understanding of research was visible when a GP commissioner explained ‘funnel plots’ to his colleagues.Academic research was regarded as being implicit in national guidance, especially from professional bodies and NICE. One GP commissioner, on his own initiative, provided a regular and enthusiastically received clinical briefing that summarised academic research alongside an update on local services . More generally, NICE guidance and guidelines were seen as “absolutely crucial” to commissioning decisions, although they were sometimes described as problematic - “we know we would go bust if we implemented all of them” . Another difficulty was the absence of local services required to deliver NICE recommendations. Commissioners tended to select the “best ones” (defined as most “do-able” of NICE recommendations) .Teresa comments that this presentation is very, very helpful. People are interested in more reviews of research evidence around unscheduled care, to help with things like education and self-management….. Anthony: That is the sort of input we want, and have not had in past. Public health departments were often cited as places to find academic research. For example, academic research evidence was formally presented at the unscheduled-care board meetings of one commissioning organisation which included representatives with many different viewpoints including patients. These evidence summaries were clear, simple documents consisting of a table with three columns: type of intervention, research evidence of impact, overall conclusion (i.e. effective/not effective/inconclusive). The committee welcomed to these reviews.So for instance the work [on unscheduled care] an awful lot of it was like, “Well there is no evidence that this works and there’s no evidence that that works.” And that, it’s really interesting, but then you think, well if we are trying to get people who have these really kind of strict targets, real pressure to reduce costs, and to just come in every month saying “We’ve looked at this and it doesn’t work,” then that’s quite a difficult position to be in. But despite the committee’s enthusiasm, they did not know how to apply the information. For example, when faced with an evidence review on unplanned hospital admissions, the committee scanned through the interventions and briefly discussed one intervention with inconclusive evidence, which a member subsequently stated was a “good idea”. The document was then put to one side. This committee did not appear to consider the disinvestment (i.e. decommissioning) opportunities highlighted by these reviews. The public health consultant was aware of the difficulties in applying the information, especially as much of the research evidence was about ineffective interventions.…[they become] frozen because you can’t make pragmatic decisions. Because it’s like America is out there but they will not set sail until they know it’s there….So with interventions like ‘telehealth’, “they’ll say ‘well we’re not doing that.’” One commissioning organisation was highly enthusiastic about ‘evidence-based policy-making’ largely because of the influence of a respected academic GP with substantial influence amongst the senior directors. However instead of the insistence on research evidence facilitating policy decisions, it was sometimes a barrier. For example, if limited academic evidence was available, some commissioners from this organisation would prevaricate and avoid moving forward, even for nationally mandated directives with which local commissioners had to comply.I’ve had conversations [with colleagues] about, “Well, you know, we shouldn’t be putting that down to say it will make savings because there’s no evidence that it will,” versus me saying, “But actually we’ve still got a statutory responsibility to deliver a balanced plan, and if I take those savings out they need to come from somewhere else.” Elsewhere commissioning organisations also struggled with their use of academic research. A GP commissioner stated that the demand for research evidence reduced innovation, because commissioners could not wait until an initiative was “piloted and proven” . Consequently, some commissioners stated that it was permissible to fund initiatives without research evidence as long as these were couched as pilots and not much money was invested. Moreover, sometimes initiatives with negative or no evidence were funded because commissioners still needed to deliver a viable financial plan. The imperative to come up with an initiative that looked like it could save money would push it through, regardless of research that suggested the contrary .Although use of academic research was infrequent, in our observations of commissioning meetings, commissioners repeatedly called for local evaluations of services and innovations “to say ‘yes’ and it is delivering what we expected – or it isn’t” . For example, one chair mentioned that they had eight different initiatives to address problems around delayed transfer of care, but no mechanism to find out if any were “contributing well or contributing badly to that final outcome” .Three service evaluations in particular were mentioned by participants from three different commissioning organisations. These included a controlled cohort study of the rehabilitation project  to provide ‘generalisable’ outputs without then offering commissioners the necessary ‘local’ interpretation required. Fourthly, we make little effort to understand the context commissioners work in and why our outputs are seldom used. Instead, we continue to produce more research in the same ways and disseminate it using methods consistently shown not to work.Dubrowa et al. make a useful distinction between the different positions of researchers and commissioners . More pohttps://www.bristolccg.nhs.uk/library). Thirdly, researchers need to learn to package their messages in commissioner-friendly ways. Swan et al. suggest that case studies offer context-rich information [1This study suggests several ways for researchers to address the ‘production’ problem’. First, academic researchers need to engage with commissioners using commissioners’ preferred methods of conversations and stories, to find out what is wanted and how best to deliver it. Researchers need to stop relying mainly on written communication to get their messages across. Starting locally would make this more manageable. Secondly, researchers need to learn more about local commissioning priorities to produce more relevant, useful research. Increasingly, finding out about local priorities is easier as commissioning organisations are on-line . Rather that commissioners tended to view academic research as not often making a useful contribution and so they sought more helpful information elsewhere. If researchers are to contribute to commissioners’ decision-making, then we need to produce more useful information and develop relationships of mutual benefit through what van de Ven calls “engaged scholarship” . This is