Evidence-based decision-making and practice are pivotal in public health. However, barriers do persist and they relate to evidence properties, organisations and contexts. To address these major knowledge transfer (KT) issues, we need to rethink how knowledge is produced and used, to enhance our understanding of decision-making processes, logics and mechanisms and to examine the ability of public health services to integrate research findings into their decisions and operations. This article presents a realist evaluation protocol to assess a KT scheme in prevention policy and practice at local level in France.This study is a comparative multiple case study, using a realist approach, to assess a KT scheme in regional health agencies (ARS) and regional non-profit organisations for health education and promotion (IREPS), by analysing the configurations contexts/mechanisms/outcomes of it. The KT scheme assessed is designed for the use of six reviews of systematic reviews concerning the following themes: nutrition, alcohol, tobacco smoking, physical activity, emotional and sexual life and psychosocial skills. It combines the following activities: supporting the access to and the adaptation of scientific and usable evidences; strengthening professionals’ skills to analyse, adopt and use the evidences in the course of their practices and their decision-making process; facilitating the use of evidence in the organisations and processes. RAMESE II reporting standards for realist evaluations was used.The aims of this study are to experiment and characterise the factors related to the scheme’s ability to enable public health stakeholders to address the challenges of KT and to integrate scientific knowledge into policy and practice. We will use the realist approach in order to document the parameters of successful KT strategies in the specific contexts of preventive health services in France, while seeking to determine the transferability of such strategies. Evidence-based decision-making and practice are major issues in public health. For researchers, this means looking ahead to the dissemination of findings and integrating different types of knowledge and decision-making challenges . It alsoDespite the general agreement about the interest of evidence informed practices and policy-making (EIDM), barriers do persist in both the production and use of evidence. These barriers relate to people, organisations, contexts and properties of evidences . To addrIn France, there is no formal and structural KT scheme. There are a few initiatives led by the National Public Health Agency (ANSP) and the National Cancer Institute (INCA), which for instance produce literature reviews. But policy-makers and prevention professionals do not use them. It confirms that a passive diffusion of knowledge is not effective, and the effectiveness of KT strategies depends on the context in which they are implemented –10. The “Awareness” (M1) is defined as building awareness for, and positive attitudes toward, evidence-informed decision-making (EIDM). This mechanism emphasises the importance of decision-makers’ valuing the concept of EIDM.“Agree” (M2) is defined as the building mutual understanding and agreement on policy-relevant questions and the kind of evidence needed to answer them. This mechanism emphasises the importance of building mutual understanding and agreement on policy questions and what constitutes fit-for-purpose evidence.“Communication and access” is (M3) defined as providing communication of, and access to, evidence. This mechanism emphasises the importance of decision-makers receiving effective communication of evidence and convenient access to it.“Interact” (M4) is defined as the interaction between decision-makers and researchers. This mechanism emphasises the importance of decision-makers interacting with researchers in order to build trusted relationships based on mutual trust, collaborate, and gain exposure to a different type of social influence.“Skills” (M5) is defined as supporting decision-makers to develop skills in finding and making sense of evidence. This mechanism emphasises the importance of decision-makers’ having the necessary skills to identify, appraise, synthesise evidence, and integrate it with other information and political needs.“Structure and process” (M6) is defined as influencing decision-making structures and processes. This mechanism emphasises the importance of decision-makers’ psychological, social and environmental structures and processes  in providing means and barriers to action.Various strategies are available to overcome barriers to the use of KT. A recent work conducted by Langer et al. identified six mechanisms involved in effective KT:The authors underline that these strategies are effective if combined and contextualized in their implementation setting, confirming previous work of Ridde et al.  and BarwIn this paper, we present the protocol of a realist evaluation study of knowledge transfer strategies implemented in the field of health prevention at a local level in France. We have used RAMESE II reporting standards for realist evaluations .The objective of the study is to identify the configurations contexts/mechanisms/outcomes of an effective KT scheme in local prevention sector. This study will be conducted in four French regions and within two types of organisation and their partners: regional health agencies (ARS), which are responsible for policy-making and prevention policies; and non-profit organisations (IREPS). IREPS develop health promotion and prevention programs and provide methodological supports to field professionals for the implementation of prevention interventions in different settings . ARS and IREPS work together to implement prevention and health policies in local contexts.We have reported this manuscript in line with the RAMESES II reporting standards for realist evaluation.This study is a comparative multiple case study of a KT scheme in the field of health prevention using a realist approach , 17. It The realist approach  is increTo conduct a realist evaluation, we alternate theoretical and empirical stages , which will be written on the basis of six reviews of systematic reviews ; an international scientific committee have carried out these reviews. They concern the following themes: nutrition, alcohol, tobacco smoking, physical activity, emotional and sexual life and psychosocial skills. These themes are primary in France. They present effective prevention practices.Supporting the access to and the adaptation of scientific and usable evidences especially the policy briefs.Strengthening professionals’ skills to analyse, adopt and use the policy briefs in the course of their practices and their decision-making process .Facilitating the use of evidence in the organisations and processes .Based on the report entitled “The science of using science: researching the use of research evidence in decision-making” , the schAccording to the initial middle range theory and the 4 theories of change, professionals will make an action plan to apply them in their local settings. Theses KT activities will be set up over a 12-month period.ARS public health professionals: five agents per region ;IREPS professionals: ten people per region .Members of specialised prevention commissions within the Regional Conferences on Health and Autonomy (CRSA) and members of the Public Policy Coordination Commission (CCPP) both devoted to prevention in French regions (five people) and partners of IREPS and ARS.The targeted population is composed of prevention and public health services operating in French regions, namely, the ARS and IREPS and their partners. The study will focus on three groups of stakeholders:We already have the agreement for the data collection given by the four ARS involved in the project since it began.Data will be collected to document the support scheme’s mechanisms and contexts parameters involved in effectiveness. They will be collected before the implementation of the KT scheme at the end and throughout the implementation. They will be collected on the 3 categories of people described before; 20 people per each region .Collected data will characterise the context, the mechanisms relating to the organisation and to the individuals involved, the PBs and the set-up for KT.A description of data collected and how and they will be collected are presented in Tables Semi-structured interviews conducted with the above-mentioned population (20 people per region)The observation of health promotion meetings and collective events resulting from the scheme’s implementation: project selection committees, selection processes, trainings, seminars, presentations etc. The aim of these observations is to study the types of interactions between the professionals who deal with KT strategies .A documentary analysis Data will be collected by means of:The observation and documentary analysis grids and the interview guideline will be designed based on the four intervention theories and tested on a sample of five stakeholders not involved in the process, but belonging to the IREPS network.The collection will last 12 months.Nvivo to assist us in conducting and integrating a thematic analysis of the interviews and an analysis of the observation reports. The qualitative analysis will lead to:Document the uptake of evidence and the practice changes triggered by the intervention. This will be carried out on a case-per-case basis in monographic format, in order to identify the mechanisms at play, the degree of intervention, the contextual contingencies and the changes arising in the three types of knowledge use .Identify the most regular CMO configurations by a cross-analysis of the different cases and a combination of the different data collected according to their linkage with the “context” meanings, “mechanism” meanings and “outcomes” meanings  which refers to a use of knowledge as a means to justify decisions or actions.Based on the observed elements, we will classify the outcomes in three categories of use as recorded in the literature , 28.InstWe will process these data to characterise the mechanisms identified in the intervention theory and induced by the intervention. This will enable us to answer the following questions about several mechanisms related to knowledge, people and organisations: (1) were they present in the contexts studied? (2) Did they positively or negatively influence the outcomes from a user’s perspective? (3) Which mechanism(s) was (were) actually active (which parameter influenced which other parameter and/or which outcome)? (4)Which outcome(s) was (were) produced?Then, a secondary middle range theory will then be developed, leading to future guidelines.Based on this analysis, we will compare the CMO configurations, which will be identify with the initial middle range theory, and we will figure out an adjusted middle range theory. This work will be conducted during a second interdisciplinary workshop, based on a discussion about analysis from data collected, gathering all ARS staff responsible for prevention and public health, IREPS directors and project officers, plus agents from ANSP and INCA.KT development guidelines in France will emerge from this meeting.Developing short and practical policy briefs about knowledge transfer to national policy-makers and practitionersDelivering presentations at local, national meetings in France and relevant international meetings for professionals and researchersInitiative en Santé Publique pour l’Interaction entre la Recherche, l’Intervention et la Décision (InSPIRe-ID), a knowledge transfer consortium, led by the French Ministry of Health.Regular project review meetings and continuous engagement with key decision-makers and practitioners, in particular as part of the Public Health Initiative for the Interaction between Research, Intervention and Decision-Delivering presentations at national and international conferences and publishing articles in peer-reviewed academic journals with emphasis on open accessDeveloping a project research report for the funder, with a publishable executive summaryDifferent types of actors will be involved throughout the study: prevention professionals, policy-makers and researchers. Thus, multiple methods will be used to communicate research results:This article describes a protocol using a realist design to understand how a KT scheme works, for whom and in what circumstances. In research, realist evaluation is valuable for evaluating interventions in their contexts; it addresses contextual factors in relation to the mechanisms and outcomes of these interventions. Thus, partial patterns can be revealed to explain how interventions may foster enhanced KT.However, even if there is weak support in France to develop KT at a local level, KT is highly recommended by health national authorities. Consequently, we need to address a potential social desirability bias, resulting both from the subject and the fact that the data are not self-reported . This biThe aim of this study is to experiment and characterise the success factors of a KT scheme in health promotion and disease prevention settings. By success, we mean the scheme’s ability to (1) enable public health stakeholders to address the challenges of KT and (2) bring about changes in public health policy and practice: integration of evidence-informed public health, collaborative practices etc. We will seek to explain the parameters and conditions of these strategies in order to determine their transferability into other contexts. This will provide a basis for the production of operational and contextualised guidelines in order to develop KT to inform regional policy-making on health promotion and disease prevention. Ultimately, this research aims at enhancing overall policy-making and quality of implementation in the sector. With this in mind, this project will be of great interest for public policy-makers who are currently moving towards evidence-informed health promotion and disease prevention in France.


When designing and analysing clinical trials, using previous relevant information, perhaps in the form of evidence syntheses, can reduce research waste. We conducted the INVEST  survey to summarise the current use of evidence synthesis in trial design and analysis, to capture opinions of trialists and methodologists on such use, and to understand any barriers.Our sampling frame was all delegates attending the International Clinical Trials Methodology Conference in November 2015. Respondents were asked to indicate (1) their views on the use of evidence synthesis in trial design and analysis, (2) their own use during the past 10 years and (3) the three greatest barriers to use in practice.Of approximately 638 attendees of the conference, 106 (17%) completed the survey, half of whom were statisticians. Support was generally high for using a description of previous evidence, a systematic review or a meta-analysis in trial design. Generally, respondents did not seem to be using evidence syntheses as often as they felt they should. For example, only 50% (42/84 relevant respondents) had used a meta-analysis to inform whether a trial is needed compared with 74% (62/84) indicating that this is desirable. Only 6% (5/81 relevant respondents) had used a value of information analysis to inform sample size calculations versus 22% (18/81) indicating support for this. Surprisingly large numbers of participants indicated support for, and previous use of, evidence syntheses in trial analysis. For example, 79% (79/100) of respondents indicated that external information about the treatment effect should be used to inform aspects of the analysis. The greatest perceived barrier to using evidence synthesis methods in trial design or analysis was time constraints, followed by a belief that the new trial was the first in the area.Evidence syntheses can be resource-intensive, but their use in informing the design, conduct and analysis of clinical trials is widely considered desirable. We advocate additional research, training and investment in resources dedicated to ways in which evidence syntheses can be undertaken more efficiently, offering the potential for cost savings in the long term.The online version of this article (doi:10.1186/s13063-017-1955-y) contains supplementary material, which is available to authorized users. When designing and analysing a clinical trial, it is important to look at previous evidence and use relevant information to inform aspects of the new trial, thereby reducing waste in research . Previouused to inform trial design. The recent review of Bhurke et al. [In a survey of 24 investigators whose trials were included in an update of a Cochrane review, only 8 33%) indicated that a previous review had influenced trial design and only 2 (8%) had used the previous Cochrane review 3% indica) referene et al.  found the et al. . To our Here, we report results from the INVEST  survey. The main objectives of the survey were to summarise the current use of evidence synthesis in trial design and analysis across clinical trials teams, to capture current opinions of trialists and methodologists on such use, and to understand any barriers to use in practice.The sampling frame consisted of all delegates at the 2-day International Clinical Trials Methodology Conference (ICTMC) on 16–17 November 2015. The conference was open to both those involved and those who have an interest in clinical trials methodology. Approximately 638 people registered to attend the conference across a range of disciplines including trialists, clinicians, statisticians, health economists, information specialists and qualitative researchers. Ninety-five percent of the registered delegates were from the UK and the Republic of Ireland, with the remaining 5% from Australia, Canada, Denmark, France, Germany, Holland and the United States. The main UK research centres represented were Aberdeen, Birmingham, Bristol, Cambridge, Cardiff, Coventry, Glasgow, Leeds, London, Liverpool, Manchester, Oxford and Southampton. Conference delegates were first invited to take part in the survey during the opening plenary session, then by researchers from the INVEST team during breaks. The survey could be completed either on paper or online, with a closing date of 18 December 2015. The survey in full is available in Additional file should be used versus current use in practice.Following details about their job role, job setting and the length of time that they had spent working in clinical trials, respondents who indicated that they had been involved in trial design  were further asked questions about whether, and how, they have used evidence synthesis in practice. All respondents were then asked about their views on the use of evidence synthesis in trial design and analysis. They were also asked to rank what they considered to be the three greatest barriers to such use. There were nine potential barriers listed including an ‘other’ category allowing free text. The subsets of respondents who indicated that they had been involved in trial design  were used to contrast views on whether evidence synthesis methods Respondents who indicated they had personally been involved in trial design were asked to consider any trials in which they had been involved over the last 10 years and to specify, if applicable, how evidence synthesis had been used in practice. A matrix style layout was chosen to allow multiple responses, with rows for each area of trial design and columns for types of evidence synthesis. In addition to (1) a description of previous evidence, (2) a systematic review and (3) a meta-analysis, we listed three evidence synthesis methods that extend meta-analysis: (4) network meta-analysis (NMA) which allows the simultaneous comparison of the effectiveness of multiple interventions through the use of direct and indirect evidence, (5) an economic decision model which can be used to evaluate intervention effects formally in the context of other factors, such as costs and potential harms, and make decisions on the use of interventions in practice, (6) a value of information (VoI) analysis which is sometimes used to assess whether there is value in conducting a new study, and to identify the optimal design for such a study within an analytical modelling framework . A finalshould be used to inform aspects of trial design. This question was formatted to match the earlier question about how those involved in trial design were using evidence synthesis methods, facilitating comparison between ideal and current practices.We also asked all respondents which of the listed evidence synthesis methods they thought Respondents who indicated they had personally been involved in trial analysis were asked which (if any) of three types of external evidence they had used in practice, during the last 10 years: (1) external information about the treatment effect , (2) evidence around the likely size of potential biases arising from trial conduct (e.g. blinding infeasible) and (3) other quantities involved in the analysis (e.g. correlations or baseline event rates).should be used to inform trial analysis. For each of these, the options were ‘yes’, ‘no’ and ‘don’t know’. An overall ‘don’t understand’ response was also included since we anticipated that some of these uses of evidence synthesis might be new concepts to some respondents.We asked all survey respondents whether each of these three types of external evidence Our analysis is descriptive, as sample sizes were not sufficient for a robust assessment of associations or subgroup comparisons. Missing responses were excluded from denominators and are indicated in footnotes in the tables that follow.For the subsets of respondents involved in trial design or analysis, we compared their responses for desirability versus actual use of evidence synthesis. For each of the five aspects of trial design, we categorised each respondent who indicated they had been involved in trial design into one of the following: ‘used and think desirable’, ‘used but don’t think desirable’, ‘not used and don’t think desirable’ and ‘not used but think desirable’. For each of the three aspects of trial analysis, we added three categories to these options: ‘used and don’t know whether desirable’, ‘not used and don’t know whether desirable’ and ‘don’t understand’.To summarise responses about the three greatest barriers to the use of evidence synthesis, we assigned three points to the first (greatest perceived) barrier, two to the second and one to the third for each respondent. If a respondent had ticked three barriers but not indicated a ranking, each was assigned two points. No points were allocated for respondents who did not answer the question. For each potential barrier, the scores were then summated across respondents, so that higher overall scores indicated greater perceived barriers.Although highly exploratory in nature because of small numbers, we examined answers to specific questions for two subgroups: the perceived barriers to the use of evidence synthesis in practice by statisticians specifically, statisticians’ use versus perceived desirability of using evidence synthesis in trial analysis, and the views of health economists on VoI analyses.There were 106 respondents, of whom 54 (51%) were statisticians, 8 (8%) were health economists and 18 (17%) worked in trial management. These are overlapping categories, i.e. respondents were asked to select all roles that applied to them. All respondents had spent some time working in the area of trials: 86 (81%) for at least 3 years and 32 (30%) for more than 10 years. Ninety-six (91%) respondents indicated that they had been involved in the design, setting up or running of trials (77 (80%) in a clinical trials unit and 9 (9%) in industry). Eighty-five (80%) indicated that they had been involved in trial design, 71 (67%) in trial conduct, 73 (69%) in statistical analysis and 52 (49%) had been involved in undertaking a systematic review of trials. Only three (3%) respondents indicated that they had not been involved in any of these. Full details are shown in Additional file Figure Of the 85 respondents who indicated involvement in trial design, Fig. Seventy-nine percent (79/100) of respondents indicated that external information about the treatment effect should be used to inform aspects of the analysis  were not sure. Similar patterns were seen for using external evidence on potential biases and other quantities. Full results are shown in Additional file less use of external evidence in each of the three areas  respondents were involved in trial analysis. Figure Figure Our INVEST survey indicates a high level of support for the use of evidence synthesis to inform aspects of trial design and analysis. Support was generally high for using a description of previous evidence, a systematic review or a meta-analysis when designing a trial. Fewer respondents indicated support for the use of NMA, decision models and VoI analyses. Only a few respondents (approximately 5%) felt that external evidence about particular parameters should not be used in the analysis of a trial; however, many (up to 20%) did not know if such evidence should be used in practice. Our results indicate some discrepancies between the evidence synthesis methods that people think should be used and what they are using in current practice. In particular, respondents did not appear to be using systematic reviews, meta-analyses, NMAs, decision models and VoI analyses as much as they wanted across all aspects of trial design. The greatest perceived barrier to using evidence synthesis methods in trial design or analysis was time constraints, followed by a belief that the new trial was the first in the area.The sampling frame was approximately 638 people, but only 106 completed the survey, providing a response rate of approximately 17% and a potential for selection bias. We were unable to obtain information on the characteristics of the nonrespondents which would have enabled us to explore the representativeness of our sample, but it is possible that respondents were more enthusiastic about evidence synthesis methods than nonrespondents. Some 95% of our sampling frame were from the UK and the Republic of Ireland, so the results may not be generalisable to the international clinical trials community. Further, our sampling frame consisted of conference delegates closely involved in trial design and analysis, who are likely to have a strong interest in promoting good practice. As such, we might expect our sample to answer some of the questions more favourably than the wider population of people involved in clinical trials. In particular, half of respondents were statisticians (51%), who may be expected to be more open to advanced statistical methods (such as using evidence syntheses to improve precision in estimates of some parameters) compared with other contributors to the design, conduct or delivery of trials. Statisticians are also influential members of the multidisciplinary teams that are involved in trial design and may be useful advocates for the increased use of available evidence in trial design. Although it would have been interesting to explore differences across research centres and countries, we chose not to collect such geographical data to protect anonymity and minimise the burden of survey completion. To summarise the barriers to the use of evidence synthesis, we assigned scores based on an arbitrary assumption of linearity, i.e. such that an individual’s highest ranked barrier is three times as important as their third barrier. These scores, although helpful for summarising data, might not reflect respondents’ true views. We intended all listed barriers to be interpreted as reasons why a trial team might not seek or carry out evidence synthesis. However, it is possible that some respondents who chose ‘Believed to be the first trial in the area’ could have been thinking of the situation where a literature search or systematic review reveals no previous trials. The extent of this barrier would then be overestimated.In trial design, for both whether a trial was needed and for choosing an intervention, more respondents said that a systematic review, rather than a less formal description of previous evidence, should be used. It therefore seems that respondents felt the need for a thorough, systematic approach in order to show convincingly whether there is a gap in the evidence base that merits a new trial. For the other aspects of trial design, there may not be sufficient available evidence to warrant a systematic review, so that a less formal description of previous evidence might be felt to be adequate.lower estimates of these proportions, although the small sample size precludes strong assertions. We feel that it is unlikely that these relatively advanced methods are being used so frequently in practice. As such, we suggest that many respondents may have interpreted these questions in ways other than intended. This explanation appears to be supported by the result that fewer statisticians than nonstatisticians claim to be using external evidence in this way: it is likely that confusion about these questions was higher among nonstatisticians although we have no direct evidence of this. In particular, respondents might have interpreted the incorporation of ‘external information about the treatment effect ’ in trial analysis as meaning including the new trial results in an updated meta-analysis. Our intention had instead been to elicit views on the use of informative prior distributions in a Bayesian statistical framework. In retrospect, we should have clarified these questions about relatively complex issues using examples, although we were keen to be as concise as possible. We propose that future qualitative research should be conducted to explore the use of informative priors with particular focus on evidence about the treatment effect, potential biases and other quantities in trial analysis [The large proportions of respondents who indicated that they had either used evidence synthesis to inform trial analysis or that they believed evidence synthesis should be used in this way were surprising. Even more surprisingly, a sensitivity analysis including only statisticians provided slightly analysis . This woFunders of clinical trials often highlight the importance of taking into account existing evidence in grant applications . HoweverThe INVEST survey provides generally higher estimates of the use of systematic reviews in trial design than the recent review of Bhurke et al. , with thWe found less support for the use of NMAs, decision models and VoI analyses in trial design which may be because they are more complex to conduct and require a greater investment of time and expertise. These methods could further help inform decisions but also require additional assumptions and ‘a priori’ parameter estimates, such as the cost-effectiveness threshold and parameters related to structural uncertainties in the case of VoI, which may not be available. A policy framework on when, and how, to perform such analyses and how they are used could be a useful next step . We alsoTrial teams responding to the INVEST survey generally reported that they are using evidence synthesis in trial design and analysis more than we might have expected, but less than they might like to. Time constraints was identified as the greatest barrier to more widespread use. Further research on ways to undertake evidence synthesis more efficiently, and training on how to incorporate results from these into existing procedures will help to ensure the best use of relevant external evidence in the design, conduct and analysis of clinical trials.Additional file 1:Shows the INVEST survey. (DOCX 130 kb)Additional file 2:Shows the characteristics of respondents. (DOCX 14 kb)Additional file 3:Shows tables summarising desirable and current use of evidence synthesis to inform trial design. (DOCX 20 kb)Additional file 4:Shows tables summarising desirable and current the use of evidence synthesis to inform trial analysis. (DOCX 3086 kb)


Integrating multiple sources of pharmacovigilance evidence has the potential to advance the science of safety signal detection and evaluation. In this regard, there is a need for more research on how to integrate multiple disparate evidence sources while making the evidence computable from a knowledge representation perspective . Existing frameworks suggest well-promising outcomes for such integration but employ a rather limited number of sources. In particular, none have been specifically designed to support both regulatory and clinical use cases, nor have any been designed to add new resources and use cases through an open architecture. This paper discusses the architecture and functionality of a system called Large-scale Adverse Effects Related to Treatment Evidence Standardization (LAERTES) that aims to address these shortcomings.LAERTES provides a standardized, open, and scalable architecture for linking evidence sources relevant to the association of drugs with health outcomes of interest (HOIs). Standard terminologies are used to represent different entities. For example, drugs and HOIs are represented in RxNorm and Systematized Nomenclature of Medicine -- Clinical Terms respectively. At the time of this writing, six evidence sources have been loaded into the LAERTES evidence base and are accessible through prototype evidence exploration user interface and a set of Web application programming interface services. This system operates within a larger software stack provided by the Observational Health Data Sciences and Informatics clinical research framework, including the relational Common Data Model for observational patient data created by the Observational Medical Outcomes Partnership. Elements of the Linked Data paradigm facilitate the systematic and scalable integration of relevant evidence sources.The prototype LAERTES system provides useful functionality while creating opportunities for further research. Future work will involve improving the method for normalizing drug and HOI concepts across the integrated sources, aggregated evidence at different levels of a hierarchy of HOI concepts, and developing more advanced user interface for drug-HOI investigations. A recent report from the United States Department of Health and Human Services noted that, while medications help millions of people live longer and healthier lives, they are also the cause of approximately 280,000 hospital admissions each year and an estimated one-third of all hospital adverse events . The fieThe use of formal  definitions for the meaning of entities represented in the database such as drugs and HOIs.Formally defined relationships between the entities represented in each integrated evidence source.Computational methods for inferring new knowledge from evidence, for example using rule-based or machine learning methods.Consistent with these results, there has been a recent call for more research on “combinatorial signal detection” that is defined as integrating multiple disparate evidence sources while making the evidence computable from a knowledge representation perspective  . ExampleExisting frameworks that have integrated various sources in a way that provide some of these features include ADEPedia , 14, MetWith the exception of Bio2RDF, none of the aforementioned systems have an open architecture that would enable the integration at large-scale, systematically, while facilitating the integration of new sources. Bio2rdf does have an open architecture. The code for loading a data source into Bio2RDF is open source enabling motivated scientists to create a local version of Bio2RDF and then integrate a new source by writing code that translates the sources data into a Resource Description Framework (RDF) graph according to Bio2RDF conventions . They caIn a previous paper , we presThis paper discusses the results of this workgroup thus far. Specifically, the paper discusses the architecture and functionality of a prototype system called Large-scale Adverse Effects Related to Treatment Evidence Standardization (LAERTES). LAERTES provides open and scalable architecture for linking evidence sources relevant to investigating the association of drugs with HOIs. The remaining sections of this paper will discuss the motivating user story, the system’s architecture, implementation details, and the current beta release.Safety physicians and risk management analysts investigate new adverse drug event reports and emerging drug safety signals. A typical way to express the requirements imposed on a software system by a specific user group is via the so-called “user stories”. The user story which drove the development of the LAERTES platform is defined as follows:“As a safety physician or risk management analyst monitoring the safety of a marketed drug, I want to do a comprehensive search across known or emerging drug-HOI evidence so I can thoroughly and expeditiously triage emerging potential safety signals and assess their potential impact.”quickly determining if a specific adverse event has been previously reported for a given drug;gauging if a potential safety concern is at the clinical drug , ingredient , or class  level;assessing the credibility of the sources reporting the association, anddeciding what priority an adverse event signal might warrant for further investigation.In order to do that, a number of tasks have to be carried out including:The LAERTES system architecture Fig.  includesRDF is a standard developed through the World Wide Web Consortium (W3C) that uses Uniform Resource Identifiers (URIs) and a graph-based data model to represent any kind of connected data . Since tIn LAERTES, a specific piece of evidence in favor or against an association between a drug and an HOI from any integrated data source is represented using the Open Annotation Data (OA) model. OA is a standard for representing human and computer annotations that is gaining broad adoption among many publishing communities. In LAERTES, every item of evidence about a drug-HOI pair is represented as an OA resource present in the RDF data store Fig. . The reaEach OA resource provides the data about the source of a specific evidence item (the “target”) and the semantic tags used to identify the record as a relevant evidence item (the “body” or “bodies”). The body of each OA has resources for drug, drug group, and HOI concepts that are represented using standard vocabularies , Medical Dictionary for Regulatory Activities (MedDRA), Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT), and RxNorm). To enable integration with other data sources in the OHDSI clinical research framework and facilitate the reuse of the LAERTES platform in the context of the OHDSI analytical tools, the source codes from the relevant terminology are replaced with the equivalent concept_id from the concept table which is part of the Observational Medical Outcomes Partnership (OMOP) Standard Vocabulary . Fig. 3 As the system diagram in Fig. The schema for the primary tables used in the relational data store is shown in Fig. The drug_hoi_evidence table provides aggregate summary statistics for every drug-HOI pair from a given source, noting, wherever possible, if the evidence supports or refutes an association. Aggregation across the sources is possible because all drug and HOI concepts are translated to RxNorm and SNOMED-CT, respectively. The aggregation is based on a number of different scores and coefficients. For example, a specific drug-HOI association might have evidence from spontaneous reporting in the form of adverse event counts as well as the results of disproportionality analyses over the reporting database . If so,# The URI to the source document from which the anonymous PubMed drug-HOI OA# resource represented by ?s is returned in the ?sourceDocument variablehttp://www.w3.org/ns/oa#>PREFIX oa: <http://purl.org/net/ohdsi#>PREFIX ohdsi: <SELECT ?sourceDocumentWHERE {?s a ohdsi:PubMedDrugHOIAnnotation;oa:hasTarget ?target.?target oa:hasSource ?sourceDocument.} LIMIT 10An important data element in the drug_hoi_evidence table is the evidence_linkout column. This holds a URL that functions as a foreign key to the adr_annotation table through a table (not shown) that maps the linkout to annotation identifiers. This enables analysts using the relational database to examine an OA represention of the source records used to create the summary data located in drug_hoi_evidence. Client programs can also use the linkout as a URL to retrieve JSON data from the RDF store that has a linked source open annotation data for the purpose of displaying this evidence. An example will help clarify the functionality. First, the following SPARQL script (executable on the public OHDSI RDF store ) shows h# The URI to the source document that provides evidence for an association between simvastatin# and Rhabdomyolysis is returned in the ?sourceDocument variablehttp://www.w3.org/ns/oa#>PREFIX oa: <http://purl.org/net/ohdsi#>PREFIX ohdsi: <SELECT ?sourceDocumentWHERE {?s a ohdsi:PubMedDrugHOIAnnotation;oa:hasTarget ?target;oa:hasBody ?body.# “simvastatin”?body ohdsi:ImedsDrug ohdsi: 1539403.# “Rhabdomyolysis”?body ohdsi:ImedsHoi ohdsi: 45619309.?target oa:hasSource ?sourceDocument.}This query can be used to retrieve the evidence item for the specific drug-HOI pair “Simvastatin 20 MG Oral Tablet” (identifier:1539411) and HOI “muscle weakness” (identifier: 36516876) from a specific source (in this case the a MEDLINE record). The important changes are shown in bold font:SPARQL queries like this one can be sent to an RDF endpoint, in order to facilitate the reuse of the annotations through produced other Linked Data applications. Returning our focus to the relational data model Fig. , each ofOne of the key features of the proposed platform is that safety evidence may be linked to the drug at different levels of granularity: at the clinical drug product level, comprising the active ingredients, strength, formulation, and brand name for the product, but may also be more coarsely defined simply as evidence for a particular active ingredient. For example, an adverse event might be mentioned in the drug product label for only one clinical drug containing a specific active ingredient. However, a published case report might discuss an adverse event that appears to be associated with all the drugs containing the active ingredient. LAERTES supports querying the evidence at four different “rollup” levels: (1) by RxNorm drug ingredient, (2) by RxNorm drug ingredient and SNOMED-CT HOI, (3) by RxNorm drug ingredient and RxNorm clinical drug, and 4) by full detail which was across the RxNorm drug ingredient, RxNorm clinical drug, and SNOMED_CT HOI. These “rollup” queries are supported by a table called laertes_summary Fig. . Data ar by full At the time of this writing, six evidence sources have been loaded into LAERTES representing three literature sources, two drug product label sources, and one spontaneous reporting source. Table Each evidence source was processed using a custom Extract, Translate, and Load (ETL) module developed in Python. All ETL modules follow a similar pattern involving 1) transforming the source data to an RDF OA graph and 2) loading the graph into an RDF endpoint, and 3) executing a query that generates statistics (e.g. count data) and linkouts. Each linkout is URL-encoded and then shortened using a custom implementation of the HarryJerry Linx URL shortener . Python Interested persons can currently access the data in LAERTES in a few different ways. The RDF database is hosted on a public-facing server  and the Progress has been made on integrating the cross-platform web services provided by the current version of LAERTES with other applications in the OHDSI clinical research framework. Specifically, the vocabulary browsing component of the OHDSI ATLAS web based tool  can use Quickly determining if a specific adverse event has been previously been reported for a given drug: LAERTES currently brings together three main types of information where drug-HOI associations are reported . The system’s open architecture makes it possible to add additional sources such as data from clinical trials. Because drugs and HOIs are normalized from the source terminologies to RxNorm and SNOMED-CT, a single query accomplishes the task of identifying existing evidence from any of the included sources. This is possible via SPARQL and SQL queries as well as through the Web API.Identifying if a potential safety concern is at the clinical drug, ingredient or class level: LAERTES allows searches specifically at the clinical drug or ingredient levels while also providing evidence “rollups” which aggregate evidence at the ingredient level  and the study type of the article from which the item was found . Similarly, drug-HOI evidence from product labeling is tagged with the specific method used to identify it and the section from which it was pulled. These tags are intended to be useful for filtering or prioritizing evidence based on a user’s perception of the relative credibility of the sources or evidence type. However, further research is necessary to test this assumption and identify the requirements for other ways to help users more rapidly assess evidence credibility.Deciding what priority an adverse event signal might warrant for further investigation: At present, LAERTES provides only an experimental graphical user interface Fig.  but the The current version of LAERTES is a prototype that can support some of the requirements of the safety physician and risk management analyst whose user story is mentioned at the beginning of this paper:-- retrieve the count of distinct patients exposed drug-HOI combination for which there is-- evidence in LAERTES from MEDLINE or European product labeling using RxNorm drug identifier,-- SNOMED-CT drug identifier, evidence type, evidence ‘linkout’, andselect rxnorm_drug, snomed_hoi, evidence_type, evidence_linkout,count(distinct person_id) pcountfrom sub1on sub1.drug_concept_id = drug_hoi_relationship.drug and sub1.condition_ concept_id = drug_hoi_relationship.hoiwhere evidence_type in) sub2group by rxnorm_drug, snomed_hoi, evidence_type, evidence_linkoutorder by pcount desc;An example of querying patient data on the OHDSI CDM using drug HOI pairs present in the LAERTES evidence base. The query counts the number of cases present in the clinical dataset where a patient condition is recorded within 30 days of the start of a drug (as indicated by data in the CDM drug_era table). The results shown are a subset of the results that were generated when the query was ran on a simulated population available to the OHDSI research community and the general public . The res-- RESULTSrxnorm_drug | snomed_hoi | evidence_type | evidence_linkout | pcounthttps://goo.gl/ikKucQ| 14Acetaminophen| Edema| MEDLINE_SemMedDB_CR|https://goo.gl/49EvSE| 14Lisinopril| Abdominal pain| MEDLINE_MeSH_CR|https://goo.gl/IVPdzx| 13Albuterol| Atrial fibrillation| MEDLINE_MeSH_CR|https://goo.gl/gMTfzk| 13Chlorthalidone| Hyperlipidemia| MEDLINE_MeSH_ClinTrial|https://goo.gl/kL3GIz| 13Metformin| Edema| MEDLINE_MeSH_ClinTrial|https://goo.gl/AB1Lue|13Enalapril| Anemia| MEDLINE_MeSH_CR|https://goo.gl/Cfhkzt| 13Captopril| Anemia| MEDLINE_MeSH_CR|The new system is entirely open source so that any interested researcher can download, run, modify, and extend the code to fit their purposes. For example, the system currently does not have a data source that provides adverse event data from social media sources. Such new evidence sources could be integrated in the platform by developing additional Python ETL modules . The sysDrugs on the market need to be monitored for public safety. A safety physician or risk management analyst has to review all the available information for drug safety issues, following what is currently a highly manual, time-intensive, and error-prone process. Improvement of the automation of bringing all the relevant evidence together in a consumable format will help such individuals better achieve their goals. LAERTES provides an open source framework that uses OHDSI technology to bring together evidence from multiple sources in a way that will enable the development of software to meet the user goals mentioned at the beginning of this paper.While the current version of LAERTES provides useful functionality, it also leaves opportunities for further research. In order to integrate the evidence sources, drug and HOI concepts have to be converted into RxNorm and SNOMED-CT concepts, respectively. Table Another opportunity for research is on how to appropriately aggregate evidence at different levels of a hierarchy of HOI concepts. For example, we have observed that some evidence sources map the concept myocardial infarction (concept identifier 4329847) directly to the SNOMED equivalent, whereas others map directly to a more specific concept like acute myocardial infarction (concept identifier 312327). The OMOP Vocabularies can be used to address this issue using the hierarchy provided. However, unlike drugs, it is not always clear the appropriate level to rollup HOI concepts. Future work will examine this issue in more detail and explore the use of alternate definitions of concept similarity .Evidence sources that do not use standardized terminologies have to be processed to map the source concept names to SNOMED-CT and RxNorm concept codes. Even for evidence sources that use controlled terminologies, there is often a conversion process required to integrate them with all of the included LAERTES sources. One limitation is that, because of the prototype nature of the current version of LAERTES, we currently do not have precise precision/recall metrics for each of the methods we used Table .Another limitation is that some terminologies are incompatible and result in imperfect mappings. In cases where mapping is incomplete, some parts of the source data will not be integrated. We mentioned above the example of the MEDLINE source using MeSH to code drugs at the ingredient level, while drugs in US product labels are coded at the clinical drug level. These cases can be addressed with the drug rollup queries described above. Table Post-marketing drug safety surveillance is an important, continuous, and demanding process. Accurate and timely identification and verification of safety signals remains a major challenge. Drug–HOI evidence exists in many sources, which are disjointed, and variable in their representation of drugs and HOIs. The current practice of reviewing drug-HOI evidence is a highly manual time intensive process that is wrought with opportunity for failure. Improvement of the automation of bringing this information together in a consumable format will greatly improve the pharmacovigilance field. LAERTES provides a framework that accepts data from multiple sources and leverages the OMOP Vocabulary to translate those sources to one terminology for drugs and one for conditions, while also enabling integration with clinical data stored in the OMOP CDM. In addition, LAERTES is open-source facilitating domain experts’ participation in its development. As the breadth of evidence available on drug-HOI associationsis too wide for any individual to be expert, an open-source model allows for niche domain experts to contribute their knowledge improving the usefulness of LAERTES for the entire drug safety community.This paper started with a motivating safety physician and risk management analyst user story to help guide LAERTES use cases. LAERTES has already collated several of the evidence sources individuals in this role would traditionally use for investigating drug-HOI signals. The LAERTES workgroup believes that the framework is in place to address the motivating example but realizes there is more work to be done. The workgroup fully expects and welcomes feedback from the community; for example, on new data sources, improvements to the Web API, and user interface design.


Implementing research findings into healthcare practice and policy is a complex process occurring in diverse contexts; it invariably depends on changing human behaviour in many parts of an intricate implementation system. Questions asked with the aim of improving implementation are multifarious variants of ‘What works, compared with what, how well, with what exposure, with what behaviours (for how long), for whom, in what setting and why?’. Relevant evidence is being published at a high rate, but its quantity, complexity and lack of shared terminologies present challenges. The achievement of efficient, effective and timely synthesis of evidence is facilitated by using ‘ontologies’ to systematically structure and organise the evidence about constructs and their relationships, using a controlled, well-defined vocabulary. Implementing research evidence into practice depends on changing human behaviour—at individual, organisational, community and population levels . The patA considerable body of evidence about behaviour change in relation to health has accumulated, as evidenced by systematic reviews published in the Cochrane Library (Cochrane Effective Practice and Organisation of Care Group (EPOC) ). This eTo make significant improvements, we want to be able to answer with confidence, variants of the questions posed by implementation and other researchers, policy makers and practitioners: ‘What works, compared with what, how well, with what exposure, with what behaviours (for how long), for whom, in what setting and why?’ and we want to be able to do this for questions that often lack large amounts of high-quality evidence. This will require very extensive searches to identify evidence and the ability to make inferences where evidence is lacking. Evidence of which behaviour change interventions are effective is required by a wide range of sciences  and by those involved in developing policy and implementing interventions in practice.Current efforts to systematically generate and apply the available evidence about behaviour change are limited by the quantity of evidence, its complexity and the lack of standardised terminology and structures for reporting interventions and their evaluations.There is no shortage of publications on behaviour change and interventions. For example, Google Scholar searches identified over 145,000 publications for ‘behaviour change intervention’ in 2016 which reflects the high publication rate for the last 20 years. At this pace, only a very efficient system could aggregate the evidence and we currently lack such a system. Our limited search and review methods may additionally contribute to the problem of reproducibility of results as cognitive biases may influence our search terms and interpretation of findings .The issue of quantity of evidence is exacerbated by its complexity. An effective aggregation of evidence should take account of the active content and delivery of the interventions, the theoretical basis for the intervention, the characteristics of populations and settings in addition to specifying the target behaviours and the methods of investigation. However, while evidence continues to be generated at a fast pace, there is no agreed framework for accumulating that evidence and as a result, evidence is fragmented and gaps in the evidence are difficult to identify.Evidence synthesis is further hampered by the wide variability and quality of research methods used to evaluate the effectiveness of interventions. Even for the same behaviour, outcomes may be assessed in different ways, e.g. objective versus self-report measures, with different follow-up periods. Even when a randomised controlled trial design is used, interpretation of findings can depend on the support or treatment given to the control groups.Complexity is exacerbated when reporting of interventions and studies is incomplete , a problIn sum, there is currently too much, incompletely reported evidence on behaviour change interventions for current methods to synthesise within an acceptable time frame. In addition, the systems available for aggregation of the evidence reflect a complexity requiring more human and funding resources than are available. Although these points have been made about behaviour change interventions, they apply also to implementation interventions.Recent advances in behavioural, computing and information sciences provide a step change in the possibilities for addressing complex questions efficiently, effectively and in a timely manner. Developments in information science include automating parts of the process such as the evaluation of risk of bias in trials. Methods of making synthesised evidence accessible are advancing but depend on success in generating useful search terms for accessing the accumulated evidence. In computer science, developments such as natural language processing, machine learning and artificial intelligence provide new opportunities for synthesising the burgeoning and fragmented scientific literature and making inferences where gaps exist , 11. ComHowever, a review of automated evidence extraction methods noted the incompleteness of methods, pointing to the importance of having a strong framework or ‘ontology’ for organising the information that might be extracted . An ontoDevelopments in behavioural science are providing shared terminology and structures for conceptualising behaviour change interventions providing a method for more co-ordinated extraction and synthesis of evidence. Ninety-three behaviour change techniques, the active ingredients of behaviour change interventions, had been defined by 2013 , 17 orgaStructures for conceptualising behaviour change evaluations and interventions have been developed such as Cochrane’s PICO ontology  , and TIDBehavioural science and implementation science could gain enormously by drawing on computational methods to optimise the value of current evidence. However, in order to make these gains, we need to make our evidence computer readable. This will require a consensus-based system of organising our knowledge that identifies the key ‘entities’ which define our evidence and which can be reliably recognised, on a limited scale, by humans. In sum, a behaviour change intervention ontology is required to allow accumulated evidence to be computer readable as a basis for more rapid and effective evidence synthesis.If implementation science continues to depend on human integration of evidence using heterogeneous terminologies, we will accumulate more systematic reviews but is unlikely to achieve the theoretical understanding about behaviour change that is needed by implementation researchers and practitioners, and the research, policy and practice communities more broadly.The Human Behaviour Change Project, funded by the Wellcome Trust from 2016 to 2020, aims to build on the scientific developments to provide a step change in our ability to synthesise evidence at scale, speed and granularity , 24. It The project will be collaborative throughout, generating data that can be used by the research community to address a wide range of research questions and involving a large international consortium of stakeholders and scientific advisors. The project will also build a user interface to allow researchers to access up-to-date syntheses and interpretations of evidence.What works, compared with what, how well, with what exposure, with what behaviours (for how long), for whom, in what settings and why? Ontologies offer a way forward by enabling us to structure our knowledge to make it recognisable to computers in order to gain the scale and completeness required.We believe that ‘business as usual’ is not an option if we are to make the step changes in evidence synthesis and in our capacity to answer the ‘big’ shared question for implementation and behavioural science: Are there important aspects of implementation interventions which we cannot currently report and which (a) would benefit from the development of an ontology or (b) where an ontology is likely to be impossible?How acceptable and feasible are taxonomies and ontologies to those conducting, reporting, interpreting, synthesising and implementing evidence?Are more easily understood?Are interpreted accurately, i.e. as in the original intervention? andLead to faster, more precise implementation of effective interventions?Do taxonomies and ontologies lead to evidence reports whichThis work is in its infancy and suggests a wide range of research questions, a few of which we set out here:


Complementary and alternative medicines (CAM) are sometimes used by individuals who desire to improve the outcomes of their fertility treatment and/or mental health during fertility treatment. However, there is little comprehensive information available that analyzes various CAM methods across treatment outcomes and includes information that is published in languages other than English.n = 270). After exclusion criteria were applied, 148 articles were analyzed in terms of their level of evidence and the potential for methodological and author bias.This scoping review examines the evidence for 12 different CAM methods used to improve female and male fertility outcomes as well as their association with improving mental health outcomes during fertility treatment. Using predefined key words, online medical databases were searched for articles  improves fertility or mental health outcomes for men or women. Acupuncture has the highest level of evidence for its use in improving male and female fertility outcomes although this evidence is inconclusive.Overall, the quality of the evidence across CAM methods was poor not only because of the use of research designs that do not yield conclusive results, but also because results were contradictory. There is a need for more research using strong methods such as randomized controlled trials to determine the effectiveness of CAM in relation to fertility treatment, and to help physicians and patients make evidence-based decisions about CAM use during fertility treatment.The online version of this article (10.1186/s12906-018-2224-7) contains supplementary material, which is available to authorized users. People facing infertility concerns are increasingly turning to the use of assisted reproductive technologies. However, successful treatment outcomes are far from certain because 68.5% of IVF cycles do not result in a live birth . These hCAM is defined by the Centers for Disease Control (CDC) as a “group of diverse medical and health care systems, practices, and products that are not presently considered to be part of conventional medicine” . The curThe stigma, costs, and uncertainty associated with biomedical fertility treatments often entice those who are having problems conceiving to use CAM as a first line of treatment before engaging in more medically invasive treatments , 9. ReceCAM is not only used to increase fertility, but also to decrease patient levels of stress and anxiety during the taxing process of treatment , 10. A lThe wide range of fertility patients reporting CAM use is partially due to differing definitions of CAM in the literature. The review of CAM by Rayner, Willis and Burgess  reports Rayner, Willis and Burgess  found thThis paper provides a scoping review of the potential benefits of various CAM methods, while describing the quality of that evidence. This paper advances the literature in this field by a) looking at a wider variety of CAM methods by including methods such as Ayurveda, herbal medicine, osteopathy, and hypnosis that have not been previously summarized in the literature; b) assessing CAM effectiveness in relation to female and male factor infertility as well as mental health outcomes; c) reviewing articles in multiple languages including Chinese, English, French and Korean; d) not limiting the evidence to randomized clinical trials or previously compiled meta-analyses.The specific research questions were: 1) What is the available evidence written in English, French, Chinese and/or Korean on the use of CAM in conjunction with male and female factor medical fertility treatment? 2) What evidence exists on using CAM for reducing psychological distress, including, stress, anxiety, and depression during men’s and women’s medical fertility treatment? 3) What is the quality of the evidence available on using CAM for improving fertility outcomes and fertility-related mental distress for both men and women experiencing infertility? Based on the review of the evidence, the conclusion contains recommendations about the potential benefits of particular CAM for both male and female fertility patients.A scoping review was conducted to compile and evaluate the evidence available for the use of CAM during fertility treatment. A scoping review is “a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, types of evidence and gaps in research related to a defined area or field by systematically searching, selecting, and synthesizing existing knowledge” . ScopingThe research questions (see above) were identified while collecting evidence surrounding the effectiveness of CAM. During this process, it became clear that there was no comprehensive review of the effectiveness of CAM in relation to specific fertility outcomes.On the basis of the National Center for Complementary and Alternative Medicine’s (NCCAM) review of popular CAM methods, the following methods were included: acupressure, acupuncture, Ayurveda, homeopathy, hypnosis, Chinese medicine, chiropractory, massage therapy, meditation, Mercier Therapy, mindfulness, naturopathy, relaxation, reflexology, reiki, touch therapy, yoga , and/or if the full-text articles could not be located using the Colombo Interlibrary Loan system , the accuracy of the abstract (i.e. if the abstract contained correct findings and/or interpretation of results), and any bias (i.e. a prejudice in favor of the authors’ hypotheses and/or conflicts of interest reported or omitted by the study authors)  were reviewed in both areas. The following results describe the rated evidence as per Australia’s National Health and Medical Research (NHRMC). The risk of bias was assessed for each article. Articles with lower levels of evidence had increased risk for author bias. Table A total of 148 out of 270 articles were determined to be relevant to review after applying the exclusion criteria  were rated as having level I evidence (with one study assessing multiple outcomes). Five level I studies showed an improvement in fertility outcomes: two showed an improvement in female fertility, two showed an improvement in male fertility, and two showed an improvement in mental health (see Table Eighteen articles were reviewed with evidence ranging from level II to level IV. Some articles examined multiple outcomes. All articles were written in English. Six of the eighteen studies (33.3%) were rated as having level II evidence. All five (100%) of the level II studies showed an improvement in fertility outcomes: three showed an improvement in female fertility, two showed an improvement in male fertility, and one showed an improvement in mental health  (see Table Seventy-two articles were reviewed with evidence ranging from level I to level IV. Some articles examined multiple outcomes. Forty-two articles were written in Chinese with thirty-six of those articles providing an English abstract. Twenty-nine articles were written in English. Fourteen of the seventy-two articles (19.4%) were level I studies with twelve (16.6%) evaluating female fertility outcomes and two (2.7%) rating male fertility outcomes. Nine of these level I studies showed an improvement in female fertility outcomes and two showed an improvement in male fertility outcomes (see Table Seventeen articles were reviewed with evidence ranging from level I to level IV. Some articles examined multiple outcomes. All articles were written in English. Three of the seventeen articles (17.6%) were level I studies with one evaluating female fertility outcomes and two evaluating male fertility health outcomes  examined the use of CAM in relation to female fertility, male fertility and/or mental health outcomes, the body of evidence is not evenly distributed amongst different types of CAM methods. Most studies focused on acupuncture and Chinese Herbal Medicine. Both acupuncture and Chinese Herbal Medicine are considered Alternative Medicine Systems (AMS) by the National Center for Complementary and Alternative Medicine (NCCAM). However, there was much less evidence for biological-based therapies (BBTs), manipulative and body based therapies (MBBTs), and mind-body therapies (MBT) (see Table While numerous studies (n = 44). There was a smaller body of evidence written in Korean (n = 3). Most of these articles had English abstracts (n = 47); in all but one case, the English abstract contained information that was consistent with the results presented in the body of the paper. This finding suggests that researchers could safely rely on abstracts written in English to capture the findings of studies written in languages that they do not read. Attention should be given to English abstracts where the article is also written in English as there was a certain number of English abstracts for English-language papers  which contained inaccurate information such as incorrect reporting of sample size or results that suggested a more positive outcome than what was presented in the discussion. If possible, researchers and clinicians assessing the evidence provided by a particular study should verify that the abstract and full-text are congruent.There was a large body of evidence found written in Chinese for both acupuncture and Chinese Herbal Medicine  there were few studies with high levels of evidence , [Overall, there was a lack of level I and level II studies in evaluating specific CAM therapies in relation to fertility treatment. This may be due to lack of funding provided to CAM researchers, and also due to the difficulty conducting randomized control trials for treatments that require holistic and individualistic approaches. For example, in acupuncture there is a debate about which type of placebo to use , and doses are often individualized . These iTRICTA), , ChineseTRICTA), , and herTRICTA), ; howeverThe mechanisms of actions of the variety of CAM techniques are largely unknown. It was difficult to determine how each specific CAM technique affected the specific outcome measured as within various CAM methods there are different types of interventions used. For example, in Ayurvedic medicine, oils were used to attempt to clear fallopian tubes while in other cases roots were given in order to improve pregnancy rates. Additionally, the studies analyzed did not necessarily investigate the mechanism that may have caused improvement, but were more interested in the outcome itself. Future studies of CAM should consider the mechanism of action of these methods to produce better quality of evidence.A scoping review of the literature was performed to evaluate twelve different CAM methods. The review was not confined to the English literature but rather included articles that were written in four different languages. Although the individual study results were not statistically combined, the collating of various outcomes allows for a determination of which methods may improve fertility outcomes and/or need more statistical evidence. The article also evaluates multiple CAM methods, some of which were not previously systematically reviewed (e.g. hypnosis).This scoping review was only performed on published studies. Therefore, we may have a publication bias towards positive results, suggesting that the results of our scoping review be interpreted with caution. There may also have been articles that our English search terms missed that were not evaluated. Additionally, we were not able to have multiple authors read the individual articles to determine the accuracy of the levels of evidence as we only had one author who could read and interpret the Chinese articles. However, author one did confirm that the type of study written by each author correctly matched the level of evidence. When the type of study was unclear, authors conferred about the type of study and the level of evidence was interpreted conservatively. Since there was a broad variety of fertility outcomes identified, sometimes with conflicting definitions, the review could not both assess the entire body of evidence and the individual fertility outcomes identified by every article.This scoping review has highlighted the lack of strong evidence for CAM methods to improve fertility outcomes. However, this lack of evidence should not mean that no future studies be performed on CAM and fertility outcomes. We suggest that future research should look towards the most promising areas of CAM , and subsequently, implement studies with more purposeful and valid research designs in order to form conclusions that are based on higher levels of evidence. In order to discourage publication bias, publishers should also be willing to publish evidence that shows negative or no effects as this is important evidence in determining the effectiveness of CAM. In other words, scientists and practitioners of CAM should more systematically research CAM therapies that have already been proven to be effective . Additionally, more standard definitions should be applied to articles assessing fertility outcomes so that results are statistically comparable across studies. For example, we suggest that measurements of improvement in female fertility outcomes should follow the BESST (Birth Emphasizing a Successful Singleton at Term) endpoints, which define assisted reproductive success based on live full-term singleton births .There is a lack of evidence for the use of CAM to improve fertility outcomes as most techniques are not evaluated by studies that produce high levels of evidence (i.e. level I and level II). Additionally, across fertility outcomes and CAM techniques, studies do not always show a significant improvement in fertility outcomes. Rather, the overall evidence points to CAM having no significant effect on the fertility outcome of interest. Although there has been an increase in the use of CAM use over time , despiteFertility physicians should inform their patients about the lack of standardized empirical evidence for all CAM methods with the realization that patients may choose to engage in CAM despite the lack of evidence. Since no methods were found to negatively impact fertility or mental health outcomes, there should be less concern with patients engaging in CAM that may hinder their fertility outcomes. However, physicians and patients should consider the evidence base for treatments that claim to significantly improve fertility and/or mental health in order to manage expectations regarding their efficacy. It should also be acknowledged that patients may choose to undergo a particular CAM procedure because it helps them feel more empowered in their treatment. The widespread use of CAM among fertility patients should provide an impetus for the design and implementation of high-quality studies of their effects.Additional file 1:Definitions of CAM Methods-Definition of CAM methods analyzed. (DOCX 14 kb)Additional file 2:CAM Review Sheet-Copy of the Google review sheet used to record the information from the articles. (PDF 90 kb)


To improve consistency and streamline development and publication of clinical guidelines (GL), there is a need for appropriate software support. We have found few specific tools for the actual authoring and maintaining of GLs, and correspondingly few analyses or reviews of GL development tool functionality. In order to assist GL developers in selecting and evaluating tools, this study tries to address the perceived gap by pursuing four goals: 1) identifying available tools, 2) reviewing a representative group of tools and their supported functionalities, 3) uncovering themes of features that the studied tools support, and 4) compare the selected tools with respect to the themes.We conducted a literature search using PubMed and Google Scholar in order to find GL development tools (GDT). We also explored tools and Content Management Systems (CMS) used in representative organisations and international communities that develop and maintain GLs. By reading a selected representative group of five GL tool manuals, exploring tools hands-on, we uncovered 8 themes of features. All found tools were compared according to these themes in order to identify the level of functionality they offer to support the GL development and publishing process. In order to limit the scope, tools for designing computer-interpretable/executable GL are excluded.After finding 1552 published papers, contacting 7 organizations and international communities, we identified a total of 19 unique tools, of which 5 tools were selected as representative in this paper. We uncovered a total of 8 themes of features according to the identified functionalities that each tool provides. Four features were common among tools: Collaborative authoring process support, user access control, GL repository management, electronic publishing. We found that the GRADE methodology was supported by three of the reviewed tools, while only two tools support annotating GL with MeSH terms. We also identified that monitoring progress, reference management, Managing versions (version control), and Change control (tracking) were often the missing features.The results can promote sector discussion and eventual agreement on important tool functionality. It may aid tool and GL developers towards more efficient, and effective, GL authoring.The online version of this article (10.1186/s12911-017-0530-5) contains supplementary material, which is available to authorized users. Clinical practice guidelines (GL) are developed to provide evidence-based advice on diagnosis and treatment to clinicians at the point of care . During Organizations responsible for developing and maintaining GLs adopt Content Management Systems (CMS) in order to manage concurrent authoring, revision, review and publishing. Our previous case study on information systems support for maintaining national GLs in Norway showed that the currently used information systems or CMS are neither fulfilling the authors’ needs in terms of functionality, nor facilitating communication between authors developing guidelines collaboratively . DiffereDevelopers of GDTs have their own assumptions and points of views, which results in software with different functionalities and features. In addition, the proposed GDTs have different foci in the guideline development process. To the best of our knowledge, there is no peer-reviewed paper analysing or comparing GDTs in detail. Therefore, characterising features supported by the GDTs, comparing them and presenting their area of focus were the main motivations behind this research. Consequently, this study has four main goals: 1) identifying available GDTs, 2) reviewing a representative group of GDTs and their supported functionalities, 3) uncovering themes of features that the studied GDTs support, and 4) compare the selected tools with respect to the themes.We note that the scope of this study is limited to the GDTs for authoring GLs; therefore tools for guideline adaption and implementation are omitted. Furthermore, tools for authoring formalized (computer-interpretable) guidelines are also omitted, since they represent a distinct later step in GL development, overlapping with general software design and development tools that are well described in the research literature.The rest of the article is organized as follows: The ‘We searched for peer-reviewed papers outlining relevant GDT or CMS in literature, contacted representative guideline development organisations and communities, and attended an international conference and a national meeting.The primary literature search was performed using PubMed and Google Scholar in March 2015 and the last search was conducted in May 2016. The selection and review process was conducted according to PRISMA guidelines  and is pWe reviewed the title and abstract of the retrieved papers. In case the title and abstract of the reviewed papers were not clear, we screened the full text. The two authors performed study selection in duplicate and discussed the inclusion or exclusion of papers until consensus was achieved.▪ Guideline International Network (G-I-N) .▪ NICE , United Kingdom▪ Agency for Healthcare Research and Quality (AHRQ), USA▪ National Health and Medical Research Council (NHMRC), Australia▪ Norwegian Health Library (Helsebiblioteket)▪ St. Olav’s University Hospital Trust (Norway)▪ Innlandet Hospital Trust (Norway)In order to learn more about actual GDT in use, we contacted a sample of organizations and national/international communities with known GDT-related activity.http://www.cebhc.co.za/decide-conference), in Edinburgh, 2014, conference theme: evidence-based guideline development.▪ DECIDE International Conference  identified GDTs and CMSs, 2) a selection of representative GDT, 3) a set of feature themes, and 4) a characterization of the selected tools according to the identified themes.The identified tools based on the searched sources are presented in the following sections.Our literature search phrases and corresponding results were presented in Table Adapt and implement existing guidelines: CAN-IMPLEMENT© [PLEMENT© Make and publish summary of finding tables (SoF): DECIDE [: DECIDE Perform systematic review: Doctor Evidence platforms [latforms , RevMan latforms , Distilllatforms , Rayyan latforms , JBI-SUMlatforms , SRDR [3latforms , EPPI-Relatforms , CREBP Slatforms  and Covilatforms Maintain systematic review repository: Epistemonikos [emonikos , and SRDemonikos Do semi-automated citation screening for systematic reviews: Abstrackr [bstrackr Maintain guideline repository: GIN guideline library [ library Detect duplicates in systematic reviews: CREBP Systematic Review Assistant [ssistant Maintain evidence profile repository: GDTs database of evidence profiles [profiles The G-I-N have published a list of suggested tools on their website . Again, By contacting to the other organizations, we found that NICE and NHMRC have their own in-house CMS; AHRQ does not use any tools or CMS. We found that the Norwegian Health Library, the St. Olav’s University Hospital Trust and the Innlandet Hospital Trust all used general-purpose CMS. The Norwegian Health Library piloted MAGICapp, and Innlandet Hospital Trust piloted purpose-built “Håndboka”.The Decide conference and the supplier conference uncovered, again, MAGICapp  and GRADAccording to the tool selection criteria presented in Section 2.2, the selected tools were: MAGICapp, GRADEpro, Internet Portal, BRIDGE-Wiz, and Håndboka. BRIDGE-Wiz  is desigBased on hands-on experience and the thematic analysis method, we identified a total of 8 themes representing GDTs features: 1) Team and contribution management, 2) Project management, 3) Evidence management, 4) Guideline development, 5) Document management, 6) Guideline content enhancement, 7) Import, export and publication, 8) User experience enhancement. A list of identified themes, their description and intension are presented in Table We reviewed the five representative tools  based on the uncovered themes and present the results in in Table As shown in Table Conflict of Interest and user accessWe recommend fine-grained access control and administrator roles on section/recommendation level with different restriction level.Although GRADEpro and Internet portal can manage conflict of interest (COI) and have role based user access, they lack fine-grained edit access for COIs on specific guideline sections. Workflow supportWe recommend that GDTs interface with workflow management systems that can support the entire GL development process for distributed teams. In order to control and track how a guideline under development moves from one team member to another based on their contribution and track the steps, proper workflow management system support is necessary. A workflow model to manage content from initial draft to the publishable form would be an improvement. Plans milestones, checklists, progress of voting, review/approval should be considered. Progress monitoring at individual, group and project level is critically important. In addition, a workflow model should provide details about development/update, content changes, versions, feedback, task assignment, and active reminders. A customizable workflow management system can be used to tailor development to local practice.The ‘development checklist’ and ‘setting milestones and deadline’ in MAGICapp are not integrated with workflow management. Similarly, GRADEpro does not track task completion based on the milestone manager. Collaborative literature search and evidence analysisWe recommend that GDTs supports annotation, annotation comments and storage of search results for each individual article.Evidence assessment is a collaborative process and authors may need to highlight or annotate the reviewed article. Our results show that only Internet Portal can import articles to a repository accessible to team members for further analysis, data extraction with the aim of making a systematic review. Automatic tracking of changes and version controlWe recommend that edit history, changes, forking, roll-backs and supporting user/role/group on the same granularity as COI-management and voting. Furthermore, we recommend that versioning may be automatic and milestone- or interval-based so as to simplify use of long-term, continuous development processes.Developing guidelines is a collaborative process and change tracking is intertwined with development process and conflict resolution. Customizable guideline templateWe recommend that the GDT support use and creation of guideline templates. This may significantly lower effort for developing new or local versions of a GL, and it will help less experienced developer teams. A customizable template can be used to tailor development to local practice and potentially increase tool adoptionUsability of guideline development tools. However, we recommend that user interface design, wizard-based and template-based authoring, interactive user guides, and workflow management support be thoroughly evaluated with respect to utility and usability.Usability is a core concern for any software tool. BRIDGE-Wiz has been evaluated , but we Electronic GL publication and usabilityIn our previous studies about usability evaluation of GLs on the Web –43 we foGuideline annotationEHR integration is very important, but relies on computational formalization, and domain modelling, which begins where we stopped. However, simple GL annotation may enhance search and retrieval related to specific patients. MAGICapp describes EHR content in the GL, enabling the GL recommendations to be instantiated with current patient data.In this section we provide an analysis of the results with suggestions for improvement.The analysed GDTs in this paper are developed by a joint collaboration of multidisciplinary and international teams , GRADEPro , Finland, and Denmark), BRIDGE-Wiz (USA), internet Portal (Germany), and Håndboka (Norway)), hence the presented results and findings will be applicable to other guideline development organisations in other countries.Much research and development have gone into formalizing GLs to make them computer interpretable. The accompanying software development tools, or integrated development environments  support In this study we searched literature and contacted guideline development organisations to identify tools and CMS for development of GLs. We reviewed five academically reported GDTs  and Håndboka, as a representative of GL-specific CMSs. The results in this paper are useful for organisations and authors assessing and selecting a GDT.By reading tool manuals, exploring the tools hands-on, comparing them against other, we uncovered, and proposed 8 themes that represent tool functionalities and features. Tools were selected according to their support of the guideline development process. Finally, the tools were characterized according to the 8 themes. Our purpose was not an overall GDT ranking, since GDTs are made for different purposes (i.e. only developing recommendations or a full guideline) and methodologies (i.e. developing recommendations based on GRADE). The identified themes can however be used for comparing tools, and in the future reconcile common features, and components. As guideline development manuals and methods are becoming more aligned, even standardized, this will drive alignment of GDTs. Furthermore, guideline representation and language standards encourage vendors to develop commercial applications for guideline implementations and EHR integration.The tools may have limitations, and features, that are only possible to uncover after realistic use. Guideline author and editor requirements are also best captured when actually using the GDTs. Further study is required to understand if, and how, standard terminologies and ontologies impact the guideline encoding process. So, in summary:▪ Guideline authors mainly use text editors or CMS for authoring and publishing clinical guidelines.▪ The need for software support of GL authoring, maintenance and dissemination.What was already known?▪ Identification of research prototypes and tools from literature, meetings, professional networks and health organizations.▪ Functionalities and features of representative GDTs.▪ Eight themes representing the GDT features.▪ Similarities and differences with regard to features among reviewed GDTs▪ The GDT application area and supported development methodology.What this study added to our knowledge: