Uganda changed its malaria treatment policy in response to evidence of resistance to commonly used antimalarials. The use of evidence in policy development—also referred to as knowledge translation (KT)—is crucial, especially in resource-limited settings. However, KT processes occur amidst a complex web of stakeholder interactions. Stakeholder involvement in evidence generation and in KT activities is essential. In the present study, we explored how stakeholders impacted the uptake of evidence in the malaria treatment policy change in Uganda.We employed a qualitative case study methodology involving interviews with key informants and review of documents. A timeline of events was developed, which guided the purposive sampling of respondents and identification of relevant documents. Data were analysed using inductive content analysis techniques.Stakeholders played multiple roles in evidence uptake in the malaria treatment policy change. Donors, the Ministry of Health (MoH), service providers, and researchers engaged in the role of evidence generation. The MoH, parliamentarians, and opinion leaders at the national and local levels engaged in dissemination of evidence. The donors, MoH, researchers, and service providers engaged in the uptake of evidence in policy development and implementation. Stakeholders exerted varying levels of support and influence for different reasons. It is noteworthy that all of the influential stakeholders were divided regarding the best antimalarial alternative to adopt.a priori will enhance uptake of evidence in policy development.Our results showed a diverse group of stakeholders who played multiple roles, with varying levels of support and influence on the uptake of evidence in the malaria treatment policy change. For a given KT processes, mapping the relevant stakeholders and devising mechanism for their engagement and for how to resolve conflicts of interest and disagreements The online version of this article (doi:10.1186/s13012-014-0150-8) contains supplementary material, which is available to authorized users. Since 2001, Uganda has changed its malaria treatment policy twice, after efficacy studies demonstrated significant resistance against first-line antimalarials . The useKT processes occur amidst a complex web of interactions between stakeholders, who are hereby defined as individuals who, or institutions which are affected by the policy change, directly influence it, or have an interest in the outcome even when not directly involved . In thisThe available literature highlights several potential roles that stakeholders may play in KT. For example, civil society organisations (CSOs)—which here are defined as formally organised non-profit groups concerned with public interests —reportedTomlinson et al. point out that the specific composition, roles, and impact of stakeholders in KT are influenced by the context within which KT processes take place and the nature of the policy . For exaThe present study is part of a larger study that seeks to enhance our understanding of how we can improve evidence uptake in health policy development. In our previous work, we explored the roles, relationships, and interactions of key stakeholders involved in KT in Uganda, without specific reference to a piece of evidence or a policy. Nabyonga Orem et al. showed that stakeholders in KT were perceived to play both positive and negative roles, as well as identified the challenges that they had to overcome to effectively play the positive roles Table  30].Tab.Tab30].TIn the present article, we explore how different stakeholders have shaped evidence uptake in relation to malaria treatment policy change in Uganda, with specific assessment of the roles they played and their level of influence and support. Furthermore, we investigated the extent to which the previously identified roles of stakeholders in KT in Uganda  differedThis case study employed qualitative methods to explore how the involved stakeholders impacted evidence uptake in the malaria treatment policy change process. The case study approach was chosen based on the need to understand complex contextual issues . Data weA timeline of key events was drawn based on the review of documents, in consultation with two persons from the WHO and two persons from the MoH who had each held malaria-focused positions for over 10 years. This timeline guided the identification of key milestones, the involved processes, the key documents to be reviewed, and the institutions involved, which subsequently informed the selection of respondents  in charge of medicine procurement and distribution and the National Drug Authority (NDA) in charge of medicine regulation Table .Table 2KIs were interviewed using an in-depth interview guide that comprised open-ended questions designed to elicit the respondents’ perceptions on whether evidence had been used and who the stakeholders were, the roles they played, and their level of interest in and support for evidence uptake in the policy change process. The interview guide was developed by the first author, was reviewed and refined by the research team, and was pretested with two volunteer colleagues in the WHO Uganda office, two technical officers in the MoH, and one researcher from the Makerere University School of Public Health. KIs were contacted by email or telephone and invited to participate in the study. All identified respondents agreed to participate and were interviewed. All interviews were conducted by the first author, in English and face-to-face. The interviews lasted an average of 45 min. During the interviews, the first author made additional notes to record initial findings and impressions that were used to augment the transcribed interviews.The timeline of key events guided the identification of relevant documents to be reviewed. We included a broad range of documents relevant to the case to ascertain the processes involved, the stakeholders, and their roles. All identified documents were retrieved and reviewed. Additional file Interviews were recorded, transcribed verbatim, and entered into MS Word software for editing as the first step towards a “formal” analysis. All interviews and reviewed documents were coded using QRS Nvivo Software Version 10. Content analysis techniques were used to construct emerging categories linked to the research issues . In the Converging issues were again reviewed by the rest of the research team. Where interpretation differed, consensus was achieved through revisiting the raw data and discussions. Where necessary, quotations that best represented emerging issues were slightly edited for flow, while preserving the meaning of the text. The findings from document analysis and from the analysis of KI interviews were integrated throughout the analysis.Informed consent was obtained from all respondents prior to the interviews. Study participants were informed about the purpose of the study and the scope of issues in the in-depth interview guide. Confidentiality was ensured in data management, and only aggregate information without subject identifiers is reported. All data were secured in a safe location accessible only to the study team. Ethical approval was obtained from the Institutional Review Board of the Institute of Tropical Medicine, Antwerp (Belgium) (IRB number IRB/AC/ac/197) and the Uganda National Council for Science and Technology (number SS 2920).the decision to change the malaria treatment policy was based mainly on evidence from efficacy studies, which showed high resistance of plasmodium falciparum to CQ”. Other types of evidence were also used. For example, one research remarked that “communities complained first `I have been taking this medicine for malaria with no improvement”. This was then picked up by the health workers and then by the scientists”. Additional different types of evidence were cited in all 18 documents that were reviewed.We integrated the results from the review of relevant documents and from the interviews with KIs. These findings are presented in three sections, namely, the roles played by the different stakeholders, the stakeholders’ level of support and influence, and other external influences on the uptake of evidence. The respondents reported that evidence informed the malaria treatment policy change, which was supported by the review of documents. For example, documentation of the malaria treatment policy change process in Uganda (MoH 2006) states that “having reached a consensus that the malaria treatment policy needs to change in light of high resistance to the first line anti-malarials, task forces including experts in malaria from the different institutions/agencies were commissioned to work out the policy change and implementation process”. The available documentation further indicated that the different task forces were expected to use available evidence in their deliberation. For example, the report of the supply chain management task force (MoH 2004) stated that “the task force calculated the amount of AL required based on the malaria prevalence, health seeking behaviour, and population growth rate”  in the NMCP who was very clear. Then there were people who were doubtful despite availability of evidence. If the technical programmes were very clear maybe even top management would have been convinced and rallied behind the decision. The technical programmes wavered in their positions and that could have had an influence on top management.” MoH respondent“some researchers did not think we needed to change to ACTs (AL). They were saying this was commercially driven and that other options like AQ and AS would have been more appropriate”. Some respondents felt that some researchers used evidence to exert pressure, as one journalist made the following remark: “there was a man called [Y] who wrote an article in the Lancet criticizing the MoH and WHO for sticking with CQ/SP as the first-line treatment. He was saying `look, you are having CQ/SP and both of them were having high resistance rates; so by combining the two you are not making matters any better’”.Researchers were reportedly very influential because they were highly respected even if they were divided. In terms of the research results, there was a consensus regarding the high levels of resistance to CQ/SP; however, there were arguments among researchers as to what alternative drug was most suitable, given the evidence showing effectiveness of both ACT specifically (AL) and, AQ and AS. A donor respondent remarked that “Others questioned the neutrality of the research community, noting that some of the research teams had been funded by donors who did not support choosing AL as an alternative first-line drug. There were suspicions that the researchers were perhaps being influenced, as shown in the following quote:These researchers were funded by donors, and those particular donors had different views on possible alternative anti-malarials. When these researchers presented their results, there was a lot of disbelief initially. People made statements like `Anyway, if you look for the resistance you will find it’. We were using CQ/SP properly and no one was complaining until efficacy studies started reviewing the data. Then we started saying we have resistance to CQ/SP.” Donor respondent“Donors were also divided. While they were reportedly very influential, they seemed to rally behind the evidence generated from the studies that they supported, as shown in the following quote:I remember some donors were in for AQ and AS, while others were in for AL. So the MoH, as the chair at that time, listened to both sides as they debated. We listened to the pros and cons of the evidence they were giving and at the end we decided to opt for AL, based on the evidence we had and other considerations. Although AQ/AS was more affordable and its efficacy was more than 90%; artemether had an efficacy of 99% and in most areas 100%. We did not want to go through another costly policy change process in the near future so we opted for AL.” MoH respondent“The WHO reportedly had a strong influence on the MoH, as shown in the following quotes:The WHO has the greatest influence on health policy in the world with relation to disease areas like HIV, malaria, etc. So Uganda as a country will often look at the WHO for the decisions that it makes. The WHO has some of the best doctors in the world and they are looking at all of the evidence that is published, and weighing the different evidence to try and make decisions on what makes sense. So countries rely on the decision making ability of the WHO and use that in their own decision making.” Donor respondent“I remember our guidance was that any research that was conducted and funded by the WHO was most credible, because we rely on the WHO’s advice and take the WHO recommendations seriously.” MoH respondent.“the WHO provided technical assistance and each of the five combinations treatments recommended by WHO for the African Region at that time was considered as a possible alternative”  saying children were dying because we are treating them with CQ and not with ACTs. And then of course, there was a lot of push, given that many countries had started to use ACTs.” Researcher respondent“we don’t have systems for managing conflicts of interest. That’s the difference between the MoH and some research institutions like the Uganda National Academy of Health Sciences where I am a member. The first meeting is to prove that no one has a conflict of interest. We always have to sign and agree on the procedure.”In light of these external influences, respondents decried the lack of systems to manage conflicts of interest. A MoH respondent remarked that “Our present results identified a diverse group of stakeholders who played multiple roles and had varying levels of influence and support, which poses a challenge to evidence uptake. Woelk et al. documented similar challenges to evidence uptake in a case regarding efficacy of bed nets, where a diverse group of stakeholders differed in their interpretation and use of evidence, and aligned with different positions based on ideology and commercial interests . The preOur findings showed that donors, CSOs, the MoH, service providers, and researchers engaged in the role of evidence generation. Lack of evidence is a known barrier to KT ,39], wh, wh39], Although the literature has highlighted that CSOs have limited capacity for evidence generation , we founDissemination of evidence was performed by the MoH, parliamentarians, and opinion leaders at the national and local levels. In contrast, the literature highlighted that the media, knowledge brokers, and structures either within or external to the MoH undertake such evidence dissemination ,9]. Thi. Thi9]. …were saying the Global Fund grants should only be used for buying ACTs…”, while the MoH was relying on the Global Fund round 4 to fund implementation of the new policy. Challenges of donor dependency have also been reported in other countries. For instance, Cameroon’s national efforts to coordinate research were undermined . In. In53]. ez-Block  documentThe top management cadre of the MoH that was ultimately responsible for decision making was noted to be a weak and passive supporter, which could be explained by their limited involvement in the research processes. Panisset et al. documented successful experiences in Bangladesh, in which the involvement of the Directorate General of Health Services in the research process enhanced the uptake of zinc for use in diarrhoea management among young children . AnotherThe participation of researchers in the policy development process improved evidence uptake . AlthougIn the majority of countries, implementation of evidence is performed by the MoH, public and private service providers, and CSOs . HoweverIt has been reported that actors play different roles depending on the nature of the policy and their level of interest in a given issue ,27],,, CSOs weIn our present study, some respondents decried the lack of systems to manage conflicts of interest. Boyd and Bero emphasised the importance of identifying and managing conflict of interest in order to improve evidence uptake . The proStakeholders in KT play different roles and exert different levels of influence and support based on the nature of the policy and the evidence under consideration. This raises the need to map relevant stakeholders and to work out mechanisms for their involvement and for how evidence can be utilised by all in an objective manner. Provision of regular updates along research processes and targeted dissemination of evidence beyond in-country stakeholders may enhance inclusiveness and consensus building. Additionally, given the multiple array of stakeholders, there is a need to utilise mechanisms to manage conflict of interest.Our study has several strengths. Firstly, we interviewed a wide array of stakeholders and used multiple data collection methods, which provided a rich data set to use in assessing the roles and influence of stakeholders in evidence uptake. Secondly, the multidisciplinary nature of the research team helped to ensure objectivity in the data analysis and interpretation. Thirdly, interviews were conducted by the first author who is experienced with policy development in Uganda.The study also had several weaknesses. Firstly, recall bias may have impacted the accuracy of the responses, given the timing of the policy process and data collection. However, we noted consistency among the responses, suggesting that perhaps recall bias was not as much of a problem as might be anticipated. Secondly, with the exception of researchers, we considered stakeholders to be organisations/institutions/units. It should be noted that individuals may not necessarily represent the views of the institution in which they work; however, this was not explored in our study. Thirdly, it is possible that the stakeholders’ levels of influence and support in the uptake of evidence in policy could change at the different stages of policy development, but this was also not assessed in our study.Stakeholders played multiple roles in the uptake of evidence in the malaria treatment policy change in Uganda. The donors, MoH, service providers, and researchers engaged in the role of evidence generation. The MoH, parliamentarians, and opinion leaders at the national and local levels engaged in dissemination of evidence. The donors, MoH, researchers, and service providers through representation engaged in the uptake of evidence in policy development and implementation. Stakeholders exerted varying levels of support and influence for different reasons. For example, donors were influential due to their significant financial support, researchers were influential because they commanded respect, and the MoH was influential because the uptake of evidence and subsequent policy implementation heavily depended on them. It is noteworthy that all of the influential actors were divided regarding the best alternative antimalarial to adopt.Our findings suggest that the roles and influence of stakeholders in KT will vary given the nature of the policy and the available evidence. For a given KT process, mapping the relevant stakeholders and devising mechanism for their engagement and for resolving conflicts of interest will enhance the uptake of evidence in policy development. Additionally, structures and systems must be put in place to encourage community participation in research processes and decision making.Additional file 1: Documents reviewed: policies, guidelines, meeting minutes, position papers, research report. Details of milestones and list of documents reviewed. (DOC 52 KB)Additional file 2: Examples of how manifest content and thematic content analyses were performed. (a) Example of how manifest content analysis was performed. Details of how data was analysed to come with categories. (b) Example of how thematic content analysis was performed. Details of how data was analysed to develop themes. (ZIP 16 KB)Authors’ original file for figure 1Authors’ original file for figure 2Below are the links to the authors’ original submitted files for images.


It has been proposed that the capacity to provide evidence-based practice is one of five core competencies that all healthcare professions should possess to meet the needs of the 21st century healthcare system. New nurses are faced with a challenging work environment which, combined with shortcomings in undergraduate education and their limited clinical experience, may affect their evidence-based practice. The aim of this study was to prospectively examine the extent of Swedish nurses’ evidence-based practice during the first five years of professional life.This was an observational longitudinal study, with yearly data collections over the course of five years. Data was collected in two national cohorts (named EX2004 and EX2006) of Swedish registered nurses. They had completed a three year academic nursing program and mainly worked in hospital settings. Participants were recruited while studying at any of the 26 universities in Sweden. A total of 2107 (EX2006) and 2331 (EX2004) nursing students were eligible. 1207 and 1227 nurses were included in the current longitudinal samples. The nurses had a mean age of 31.2/33.9 years old and a majority were female. The cohorts were representative of the general nursing population. Data was self-reported and collected through annual postal surveys. Evidence-based practice was conceptualized as a process and measured with an instrument including six items. Data was analyzed using latent growth curve modelling.Implementation of evidence-based practice was stable, between the two cohorts and over time. Individual differences existed and remained stable over time. However, the extent of practicing the different components of evidence-based practice on a monthly basis varied considerably, from 10% of the nurses (appraising research reports) to 80% (using information sources other than databases to search for knowledge).The extent of evidence-based practice remained unchanged during the first five years of professional life. It appears important to enhance both the contribution of undergraduate education and the contextual conditions in work life, in order to improve evidence-based practice among newly graduated nurses.


Adoption of personalized medicine in practice has been slow, in part due to the lack of evidence of clinical benefit provided by these technologies. Coverage by insurers is a critical step in achieving widespread adoption of personalized medicine. Insurers consider a variety of factors when formulating medical coverage policies for personalized medicine, including the overall strength of evidence for a test, availability of clinical guidelines and health technology assessments by independent organizations. In this study, we reviewed coverage policies of the largest U.S. insurers for genomic (disease-related) and pharmacogenetic (PGx) tests to determine the extent that these tests were covered and the evidence basis for the coverage decisions. We identified 41 coverage policies for 49 unique testing: 22 tests for disease diagnosis, prognosis and risk and 27 PGx tests. Fifty percent (or less) of the tests reviewed were covered by insurers. Lack of evidence of clinical utility appears to be a major factor in decisions of non-coverage. The inclusion of PGx information in drug package inserts appears to be a common theme of PGx tests that are covered. This analysis highlights the variability of coverage determinations and factors considered, suggesting that the adoption of personal medicine will affected by numerous factors, but will continue to be slowed due to lack of demonstrated clinical benefit. The field of personalized medicine is anticipated to significantly impact clinical practice as evidence continues to accumulate for new applications that will lead to greater adoption. The excitement surrounding personalized medicine has, in large part, been due to advances in genome-based technologies to assess variation at the DNA, RNA, protein and metabolite level. Already, a number of genomic applications are clinically available that inform medical decision-making and improve health outcomes. In particular, pharmacogenetic (PGx) testing, or tests to predict risk of adverse drug response or likelihood of response, is on the forefront of the personalized medicine movement. Despite the potential of personalized medicine and continuing development of new clinical tests, adoption of these tools in practice has been relatively slow. The slow uptake is likely due to a combination of factors including lack of physician knowledge, patient interest, insurance coverage and reimbursement and lack of evidence demonstrating clinical utility ,3,4,5,6.In this study, we evaluated medical coverage policies of the largest U.S. insurers for genomic disease-based tests and PGx tests to determine which tests were covered and the evidence basis for the coverage decisions. This snapshot provides a comprehensive overview of insurance coverage policies for these rapidly developing technologies. Overall, we find that insurers generally agree on their coverage policies for genomic tests, but few tests are currently covered. To identify coverage determination policies for genomic and PGx tests, we conducted an online search of the top dozen U.S. insurers (based on U.S. News & World Report listing ). We excCoverage policies for UnitedHealth Group, Aetna, Humana, Independence Blue Cross Group and Cigna are publicly accessible through the company’s website and search engines. We used the following terms to search for genomic and PGx policies: gene, DNA, mutation and genomic. For the Independence Blue Cross Group, all coverage determinations within the category ‘pathology and laboratory’ for genetic and genomic tests were reviewed (no coverage policies were found in the other categories). We defined genetic and genomic tests to include laboratory analysis of DNA, RNA, or protein. Our search was conducted in August 2012.If a drug is specifically indicated for patients with a particular genetic aberration, we excluded the tests required for identifying those patients from our analysis. For examples, testing for KRAS , ALK (anaplastic lymphoma kinase for the drug crizotimib), and Her2/Neu  were excluded from our dataset. For these drugs, coverage of testing is likely to be influenced by pharmacy benefits and use of drug without testing would constitute off-label use.We also reviewed health technology assessments of genomic and PGx tests conducted by two groups: the Centers for Disease Control and Prevention’s Evaluation of Genome Applications in Practice and Prevention (EGAPP) group and the Blue Cross Blue Shield Technology Evaluation Center (BCBS-TEC). For BCBS-TEC, in addition to conducting a text search, we reviewed all assessments in two of the 21 categories listed to identify relevant genomic and PGx assessments: “genetic testing” and “pharmacotherapy/therapy”. For EGAPP, all reports were reviewed. Each coverage policy and assessment was reviewed to identify the specific test, evidence considered (if provided) and the policy decision. In addition, we determined whether the test was cleared or approved by the Food and Drug Administration (FDA). For PGx tests, we also determined whether the package insert had been revised to include information about the impact of genetic variants on risk of adverse response or likelihood to respond and PGx testing.Overview of Coverage Policies. In our search of leading U.S. insurance companies, we identified a total of 41 coverage policies for genomic and PGx testing  to warfarin was not covered by any of the insurers and deemed investigational. The Oncotype Dx is covered by all insurers to assess breast cancer recurrence risk, but considered investigational with respect to colon cancer recurrence. Twelve of the 49 tests were covered by at least one company, with twice as many PGx tests than disease-related tests covered . Nine teBasis for Coverage Decisions. For coverage policies that included background information or reasons for their coverage decision, a review of key clinical studies of the test, clinical guidelines and availability of FDA-approved tests were often included. For several of the tests, while evidence of a strong association between genotype and disease risk or drug effectiveness were noted, the lack of evidence of clinical utility appeared to be the determining factor for coverage. In particular, tests that are not covered tended to lack evidence from prospective, randomized clinical trials. Evidence of cost-effectiveness was not specifically mentioned in any coverage policy. Discordant Coverage Policies. Of the tests reviewed by more than one insurer, five tests had discordant coverage policies: the Allomap test for cardiac allograft rejection risk, the Mammaprint test for breast cancer risk recurrence, CYP2C19 testing for use of clopidogrel, and EGFR and KRAS testing for erlotinib. The AlloMap test and CYP2C19 PGx test for clopidogrel were reviewed by all insurers except United Health. Aetna and Independence Blue Cross provided coverage for these tests, whereas Cigna and Humana considered both tests to be experimental/investigational and therefore, not covered. We compared the coverage policies for discordant coverage decisions. Humana does not provide any background information or rationale for their coverage policies, and therefore, comparisons were not possible. The AlloMap test evaluates expression of 20 genes associated with risk of cardiac allograft rejection following a heart transplant. The test is intended to reduce the number of endomyocardial biopsies performed by identifying patients who are not at risk of rejection, therefore ruling out those patients for biopsy. The BCBS-TEC assessment concluded that the AlloMap test did not meet their criteria for clinical use. Cigna considered a broader range of sources in formulating its policy for AlloMap than other insurers. For example, they commented on four peer-reviewed studies that Aetna did not consider ,12,13,14The CYP2C19 test is used to identify patients not likely to respond to treatment with the anti-platelet drug clopidogrel. The prevalence of loss of function mutations in the CYP2C19 gene ranges from 2%–5% in Caucasians and from 13%–23% in people of Asian ancestry . In 2010Comparison of the differing coverage policies between Cigna and Aetna again shows different data were considered. Cigna analyzed nine peer-reviewed articles on genetic testing in use with clopidogrel; Aetna analyzed only one of these studies in its corresponding review but cited (though not discussed) two other studies evaluated by Cigna. Overall, Aetna considered a smaller data set in formulating its policy, analyzing three peer-reviewed articles and citing four others without discussion. One article was cited by both payers without discussion. Five of the six studies evaluated by Cigna but not Aetna suggest an association between CYP2C19 genotype and response to clopidogrel treatment ,19,20,21One of the important drivers of uptake of genomic tests is insurance coverage. Based on the data collected from a small group of leading insurers, we find that while coverage of genomic and PGx testing is not common, some insurers have been more proactive in this area than others. Overall, 25 percent of the tests reviewed by major insurers are covered, with twice as many PGx tests covered compared to disease-related genomic tests. The variability of review and insurance coverage is consistent with other reports ,8. The let al. [et al. [While FDA test approval does not appear to be a leading factor in coverage decisions, inclusion of PGx information in drug package inserts does appear to be an important factor. Over the past decade, the number of labels with PGx-related information has increased, including some for commonly used drugs ,27. We fet al.  similarl [et al.  reported [et al. . For disGiven that policies are regularly reviewed and updated, we anticipate increased coverage of these tests as evidence is published and physicians become more comfortable ordering such tests, perhaps in response to patient requests, given the high level of interest in PGx testing . As we dAnother approach to increasing the evidence basis of genomic tests is for the Centers for Medicare and Medicaid Services (CMS) to apply the coverage with evidence development (CED) status to promising genomic and PGx tests ,31. ThisThe early adoption of tests is an important factor considered in coverage decisions, particularly in the absence of published evidence of clinical utility . TherefoIn summary, insurance coverage for disease-related genomic and PGx testing is low and variable, though some insurers are willing to provide coverage based on limited evidence of clinical utility. Expanding data collection efforts outside of traditional clinical trial study designs and promoting physician and patient awareness will likely lead to greater review and coverage of these tests.


We developed an evidence service that draws inputs from Health Systems Evidence (HSE), which is a comprehensive database of research evidence about governance, financial and delivery arrangements within health systems and about implementation strategies relevant to health systems. Our goal was to evaluate whether, how and why a ‘full-serve’ evidence service increases the use of synthesized research evidence by policy analysts and advisors in the Ontario Ministry of Health and Long-Term Care as compared to a ‘self-serve’ evidence service.We attempted to conduct a two-arm, 10-month randomized controlled trial (RCT), along with a follow-up qualitative process evaluation, but we terminated the RCT when we failed to reach our recruitment target. For the qualitative process evaluation we modified the original interview guide to allow us to explore the (1) factors influencing participation in the trial; (2) usage of HSE, factors explaining usage patterns, and strategies to increase usage; (3) participation in training workshops and use of other supports; and (4) views about and experiences with key HSE features.We terminated the RCT given our 15% recruitment rate. Six factors were identified by those who had agreed to participate in the trial as encouraging their participation: relevance of the study to participants’ own work; familiarity with the researchers; personal view of the importance of using research evidence in policymaking; academic background; support from supervisors; and participation of colleagues. Most reported that they never, infrequently or inconsistently used HSE and suggested strategies to increase its use, including regular email reminders and employee training. However, only two participants indicated that employee training, in the form of a workshop about finding and using research evidence, had influenced their use of HSE. Most participants found HSE features to be intuitive and helpful, although registration/sign-in and some page formats (particularly the advanced search page and detailed search results page) discouraged their use or did not optimize the user experience.The qualitative findings informed a re-design of HSE, which allows users to more efficiently find and use research evidence about how to strengthen or reform health systems or in how to get cost-effective programs, services and drugs to those who need them. Our experience with RCT recruitment suggests the need to consider changing the unit of allocation to divisions instead of individuals within divisions, among other lessons.HHS/FHS REB 10–267).This protocol for this study is published in Implementation Science and registered with ClinicalTrials.gov (The online version of this article (doi:10.1186/s12961-015-0066-z) contains supplementary material, which is available to authorized users. Health system policymakers make important decisions every day about the governance, financial, and delivery arrangements within which programs, services, and drugs are provided and about implementation strategies . For thePolicymakers also need to be alerted to the availability of new research evidence in their areas of interest to them. As outlined in the protocol for this study , until rAs detailed in our published study protocol , we planHowever, as we report in the results section in more detail, we terminated the RCT when we failed to reach our recruitment target of 148, which was the number of policy analysts and advisors working in the purposively sampled division at the time of publishing the protocol (this changed to 138 by the time we started recruitment). After the termination of the trial, we revised the original interview guide  from which the sample was drawn. Using the list of 78 individuals who met these criteria, we selected a purposive sample of 30 policy analysts and advisors, of whom 15 had originally agreed to participate in the trial and 15 had not. In selecting each half of the sample, we included a mix of participants from different branches and units within the division and from different types of positions . As reported in the results, all of those who participated in the qualitative interviews were those who had agreed to participate in the RCT. Based on this, we determined that additional sampling would be highly unlikely to yield additional participants given that, of those remaining in the sample frame, most had not responded and some had declined the invitation for the RCT phase of the study.We invited participants by email to take part in semi-structured telephone interviews. The interviews were conducted by MGW using an interview guide and audio-taped the interviews, and we transcribed the interviews verbatim. The first part of the interview guide explored the factors influencing participation in the trial. We included an initial set of probes for these factors, such as the time commitment involved, concerns about their HSE usage being monitored, and the trial’s perceived relevance to their work. The second part of the interview guide explored (1) participants’ usage of HSE, factors explaining usage patterns, and strategies to increase usage; (2) participation in training workshops and use of other supports; and (3) views about and experiences with key HSE features. We asked explicitly about eight of HSE’s key features: (1) registration and sign-in; (2) open search; (3) advanced search; (4) search overview page; (5) detailed search results page; (6) links to one-page summaries; (7) monthly evidence service; and (8) supplementary material/portals. We added the last probe to solicit feedback about the Evidence-Informed Healthcare Renewal Portal that was added after the trial was terminated and provides a set of policy-relevant documents related to healthcare renewal in Canada . Prior tWe used a constant comparative approach  to data From the 138 policy analysts and advisors who were working in the division at the time and hence invited to participate in the trial, only 59 responded to either of our three waves of study invitations (a 43% response rate). Of these, 21 agreed to participate  and 38 declined. Of the 38 who declined to participate, 16 were still in the same ministry but no longer in the same division, nine did not provide a reason, seven were on short-term contract, retiring or going on leave, and six had moved to a different ministry. Given our failure to reach our recruitment target in the division addressing health system issues most directly, we terminated the trial.Of the 30 policy analysts we invited to participate in the qualitative process evaluation, nine agreed to participate, of whom seven had agreed to participate in the trial and two had not. While limited in size, the sample included participants from six different units and covered a range of sectors , populations  and decision support . However, the sample was not as balanced in terms of level of position with eight participants in senior analyst roles and one in a junior role.While two of the study participants had not agreed to participate in the trial according to our study records, during the interview all nine reported having agreed to participate in the trial. Participants cited a number of factors that encouraged their participation in the trial, which included the relevance of the trial to their own work; familiarity with the researchers; personal view of the importance of using research evidence in policymaking; academic background; support from supervisors and the Assistant Deputy Minister; and the participation of colleagues.think people have a variety of different pressures on their time here and they don’t understand the importance of research so they put it down lower on the priority list.” The same participant indicated that, given these pressures, “to have the manager say this was actually something that is really a priority, helped us to prioritize that as something to do.”Three participants also reported that the time commitment had been a concern for them, and many noted that there are often a large number of tasks to accomplish and non-prioritized tasks are often ignored. For example, one participant noted that they “…a good thing for the policy and planning community” as they help to ensure that policymakers “get better access to evidence and tools.”One participant expressed concern about having to be careful what policy analysts and advisors said as a representative of the ministry, but none of the study participants were concerned about their HSE usage being monitored as part of the trial. In general, they suggested that trials such as this one are “with any policy initiative you have, especially with the Ministry of Health, there’s a huge push to have evidence. So when you’re writing different types of documents to support initiatives, you need to have options and they need to be supported with evidence.”Six of the participants had used HSE before the interview, but these same participants indicated that they used it infrequently or inconsistently. Despite this, almost all of the participants (n = 7) stated that they felt it would be useful for their work-related projects. Two participants indicated that HSE was a helpful resource because it helps policymakers meet their need for finding relevant research evidence to support and inform the development of policy initiatives. For example, one participant noted that “It frustrates me to no end that we continue to have all of these different partitioning[s] of evidence. Which is both good, because it allows us to focus but also bad because there is […] much less cross-boundary searching that we can do. I always have to end up going back to the general databases to search […] being able to specifically say, for example, I want to look for something that crosses the boundaries of public health and primary care and community care, that would be fantastic.”“When asked for reasons why HSE was used infrequently or inconsistently, all participants indicated that they already use multiple databases and one participant specifically indicated that they often prefer to stick to the ones they know. In addition, three participants noted that they are typically not required to directly seek research evidence themselves because units in the ministry already do this for them. One participant said that rather than having different databases addressing different healthcare topics , it would be better to more generally include evidence across healthcare boundaries. Specifically, this participant indicated that:When asked about what could be done to increase their use of HSE, four common themes emerged, namely reminders about the existence of the site, such as the monthly evidence service; encouragement of its use from management in general and supervisors in particular; incorporation into employee training; and promotion on the Ministry’s website.if there’s buy in from senior management here to help promote a tool like this or to encourage people to use it, that’s also a helpful way in terms of getting the attention of people who are doing policy work because often, it takes people getting direction from someone above to motivate them sometimes”“One participant highlighted that:I actually went to [the] workshop and he [the facilitator] probably talked about it and then just went, you know, out the other ear.” None of the participants had watched the tutorial videos provided through HSE, but three stated they thought they could be helpful. Two participants also said the tips on the site itself were useful.Six of the participants had participated in workshops on finding and using research evidence provided to ministry staff in which HSE is profiled as a key source for research evidence to address questions related to health systems. Two others had never participated but had read the material, and one had not participated or read the material. Of those who had participated, two indicated that they thought it had an impact on their use of HSE. Another two participants said the workshop was helpful because it taught them about quality ratings (like the ones provided in HSE) and another participant noted that the workshop helped them learn about the different search tools. Others indicated that attending the workshop did not influence their use of the website. For example, one said: “were all consistently very high quality compared to if I did my usual bibliography searches on the literature databases.”In general, most participants (six out of nine) said they found the features of HSE to be intuitive. Some said they found the information it contains more relevant for policymakers than most other search engines. One participant indicated that the documents found in HSE “Table We terminated the RCT – to our knowledge, the first attempt to evaluate the effects of an evidence service specifically designed to support health system policymakers in finding and using research evidence using such a design – given our 15% trial recruitment rate. Six factors were identified by those who had agreed to participate in the trial as encouraging their participation: relevance of the study to participants’ own work; familiarity with the researchers; personal view of the importance of using research evidence in policymaking; academic background; support from supervisors; and participation of colleagues. Most participants reported that they never, infrequently or inconsistently used HSE and suggested several strategies to increase its use, including regular email reminders and employee training. However, only two participants indicated that employee training, in the form of a workshop about finding and using research evidence, had influenced their use of HSE. Most participants found HSE features to be intuitive and helpful, although registration/sign-in and some page formats (particularly the advanced search page and detailed search results page) discouraged their use of HSE or did not optimize the user experience.The principal strength of our study is that we received in-depth feedback from a sample of policy analysts and advisors (a key target audience for us) about trial participation and HSE features and we have acted on many of their suggestions for improving HSE. The principal limitations of our study relate to sample size, both in terms of the very low recruitment rate for the trial (which resulted in its termination) and in terms of the somewhat low recruitment rate for the qualitative process evaluation, particularly among those who did not volunteer for the trial. In addition, it is likely that many of those who agreed to participate in the qualitative interviews were those already keen to use HSE in their work, which may have limited our ability to identify divergent views. However, we did observe the recurrence of common themes in each interview and with participants from a diverse set of units in the division, which leads us to believe that, if data saturation was not reached in all domains , we were likely close to achieving it.We have identified implications related to future trial recruitment and for efforts to support the use of research evidence by policymakers. For future trial recruitment, our study provides a helpful reflection about how best to recruit policy analysts and advisors in a trial within a particular division of a health ministry even when a team has the explicit support of an Assistant Deputy Minister, who is the divisional head, in the form of both a formal letter attached to our invitation and an internal email that was sent to staff of the division. One particular point for reflection is whether our ‘unit of allocation’ for the intervention should have been divisions and not individuals within a division. Policy development is a complex process that depends on the efforts of many policy analysts and its use at the individual level is therefore likely to vary depending on policy priorities at a given point in time. As a result, the more meaningful unit of analysis could be the use of an evidence service in a division, which would more accurately capture this variability at the individual level. In other words, the focus would instead be on studying the effect of our evidence service on the use of research evidence by all policy analysts and advisors within their division as the more meaningful level of outcomes.In addition to modifying our approach to analysis, changing the unit of allocation has significant implications for the levels at which informed consent should be obtained for a cluster RCT . SpecifiFor efforts to support the use of research evidence, our study supports others’ findings that essential components of such efforts include the need to ensure timely, multi-faceted yet focused communication ,12-14 anBased on the feedback received about HSE, we made several modifications to enhance the functionality of the advanced search page and the detailed search results page. On the advanced search page, we modified the layout to better highlight the available search functionality. Specifically, we have incorporated a short descriptive header for the three primary groupings of search functionalities , which are followed by a one-line description of what each can be used to do. In addition, we now highlight that the Evidence-Informed Healthcare Renewal Portal contains policy documents related to healthcare renewal in Canada. Next, we incorporated two new search limits. The first limit allows users to limit their search to documents with either a general focus  or a specific focus . The second limit allows users to search for documents that either focus on a particular country (or countries) or include at least one study that was conducted in a country (or countries).On the detailed search results page, we removed the columns outlining the type of question and the health system topics addressed, which were both identified by most participants as not being helpful when scanning results for relevant documents. We also now list the document title in the first column (instead of the type of document), which is then hyperlinked to the one-page summary, uses bolded text in a different colour (to make it stand out on the page) and notes whether the document has a general or specific focus in an icon below the title.The qualitative data we collected has informed the redesign of HSE, which will allow users to more efficiently find and use research evidence about how to strengthen or reform health systems or in how to get cost-effective programs, services and drugs to those who need them. We plan to continue to conduct periodic user testing to ensure that HSE is an optimal tool to identify research evidence about health systems, as well as to explore ways to increase trial recruitment rates among policy analysts, and more generally approaches to future evaluations.


Saccharomyces Genome Database, PomBase, the Protein Information Resource and others. Information on the development and use of ECO can be found at http://evidenceontology.org. The ontology is freely available under Creative Commons license (CC BY-SA 3.0), and can be downloaded in both Open Biological Ontologies and Web Ontology Language formats at http://code.google.com/p/evidenceontology. Also at this site is a tracker for user submission of term requests and questions. ECO remains under active development in response to user-requested terms and in collaborations with other ontologies and database resources.The Evidence Ontology (ECO) is a structured, controlled vocabulary for capturing evidence in biological research. ECO includes diverse terms for categorizing evidence that supports annotation assertions including experimental types, computational methods, author statements and curator inferences. Using ECO, annotation assertions can be distinguished according to the evidence they are based on such as those made by curators versus those automatically computed or those made via high-throughput data review versus single test experiments. Originally created for capturing evidence associated with Gene Ontology annotations, ECO is now used in other capacities by many additional annotation resources including UniProt, Mouse Genome Informatics, Database URL: Evidence Ontology Web site: http://evidenceontology.org An essential component of scientific research is the documentation of evidence-based conclusions resulting from investigations. Just as careful documentation of scientific methodology can allow other investigators to understand or reproduce an experiment, describing the scientific evidence associated with a given assertion or inference allows for its meaningful assessment. Many of the data that are used to support assertions are experimentally derived in a laboratory or field setting. However, assertions are also commonly made based on computational analysis of large data sets, implied by known biology and captured during literature curation, or even synthesized from investigator speculation. Because types of evidence can vary, it is essential to document this information to allow others to draw meaningful and appropriate conclusions. There are numerous practical benefits gained by capturing evidence. One simple but essential benefit is the ability to query a database for an annotation (an assertion about a gene product) based on a particular type of evidence. In addition, an effective evidence capture system can be used as a mechanism for filtering search results, for establishing computable rules about what data types should be associated with what types of evidence and to flag particular chains of inference that might require further review .A central goal of biological curation systems is to associate biological entities with descriptive information. This is achieved through curating annotations. Optimally, the associated information, i.e. the annotation, is represented with terms from a descriptive vocabulary. To document the evidence for a scientific conclusion (such as the assertion that ‘this protein has function x’), the assertion must be linked to an appropriate term that describes the evidence supporting the conclusion (such as ‘enzymatic assay’). It is possible to connect any number of evidence types with a wide range of experimentally or computationally supported assertions. However, for this to be practical, a structured, well-defined and expansive vocabulary that captures the types of evidence is needed. Ontologies are particularly well suited to this task. Ontologies allow for the detailed description of things that exist within a particular knowledge domain and provide a structural framework with relationships among descriptive terms. This enables categorization of descriptions at different levels of specificity and facilitates retrieval of data associated with ontology terms in an interpretable and computable fashion , 2. Therhttp://evidenceontology.org) contains around 600 terms arranged in a hierarchy with the root node being ‘evidence’ (ECO:000000), where ‘evidence’ is defined as ‘a type of information that is used to support an assertion’. An assertion is a statement about something that is thought to be true, for example, the assignment of a function to a protein , defined as ‘a means by which a statement is made about an entity’. For example, whether a curator arrives at an annotation by reading about a result reported in a paper or through the process of manually evaluating pairwise sequence alignment results, ECO can capture that a manual curation method has been used . [For a et al. ]. LikewiArabidopsis Information Resource (http://purl.obolibrary.org/obo/eco/legacy).The earliest versions of ECO were created from sets of evidence codes in use by the GO Consortium and its members. The GO evidence codes and the more extensive codes used by The Resource  and FlyBResource  providedhttp://code.google.com/p/evidenceontology), and an independent home page was created (http://evidenceontology.org). At the time of writing, there were ∼570 terms in ECO, and many organizations are now using ECO in their evidence capture systems , using the ontology editor OBO-Edit (https://code.google.com/p/evidenceontology/issues/list), researched and entered into the ontology as appropriate. Reasoning is performed to check term relationships and intra-ontology cross-product term validity. Releases are generated by the OBO Ontology Release Tool (http://code.google.com/p/owltools/wiki/Oor tIntro), and both OBO and OWL versions of the ontology are provided. The HermiT reasoner is run as part of the release cycle, to verify that the ontology is formally coherent, and to build the hierarchy. A continuous integration server is used to perform build and integration tests (http://build.berkeleybop.org/job/build-eco).ECO is developed in Open Biological Ontologies (OBO) format syntax, which is a restricted subset of Web Ontology Language  including (i) inconsistent mixing of ‘is_a’ relations based on experimental methodology, commercial platform, types of reagents used, measurement objective and analyte being measured; (ii) incomplete or entirely lacking term definitions; (iii) a lack of active development and Web presence; (iv) and a lack of interoperability, and possible overlap, with the Ontology for Biomedical Investigations (OBI) (https://code.google.com/p/information-artifact-ontology).A number of criticisms of ECO (revision 19) were noted at an OBO Foundry  workshopns (OBI)  and the To address these issues, and with the goal of adhering to general OBO Foundry principles for active development and interoperability , first gA primary target for ECO improvement was to make definitions more succinct to enhance clarity and interoperability with external ontologies, as well as to facilitate development and maintenance. To this end, definitions were edited to remove explicit references to the GO, and lengthy definitions that contained extraneous information such as usage notes and examples were shortened and formatted to conform to genus-differentia syntax. This was of particular importance because GO Consortium evidence codes are actually not directly equivalent to ECO evidence classes, but rather are analogous to ECO intra-ontology ‘evidence x assertion method’ cross products (described later). Because some long-standing ECO users include major model organism databases that perform GO annotation, as well as the GO itself, much of the GO-specific text that was removed from term definitions was retained, but appropriately rephrased, in comment fields. Such improvements allow for a term to be used in any annotation system, including one not concerned with GO evidence capture, while offering salient information to a user who is doing GO evidence capture.Early versions of ECO comprised a mixture of terms that conflated distinct concepts, including ‘origin’ of evidence , ‘type’ of information contributing to evidence , ‘evaluation’ or ‘assessment’ yielding evidence , ‘assignment’ of evidence in an annotation (e.g. by a human or a machine) and ‘inference’ drawn from evidence (e.g. a conclusion arrived at after weighing evidence). Such mixing of ideas was present in term names and definitions and was unclear in inheritances. To enhance conceptual clarity and usability, as well as to facilitate future development and interoperability with existing ontologies, it was essential to distinguish among the different interpretations of the meaning of evidence. Two axes were established for the organization of all ECO terms . The firused_in relation to connect evidence to assertion method   . This teused_in either manual assertion (ECO:0000218) or automatic assertion (ECO:0000203). Currently, the used_in relation is defined as ‘a relation connecting a piece of evidence to an assertion method, where that assertion method is supported by the evidence’ and is restricted to hold between these two ECO classes. In the future, a more general-purpose relation may be selected from an ontology such as the Relationship Ontology  and connected with the ‘is_a’ relationship. Although asserted as members of the ‘evidence’ hierarchy, they are differentiated by being Ontology , IAO or used_in ‘automatic assertion’ (ECO:0000203) and carries the dbxref ‘GOECO: IEA’. In another example dealing with experimental evidence, ECO:0000353 ‘physical interaction evidence used in manual assertion’ is analogous to the GO code ‘IPI’  and so contains the cross-reference ‘GOECO:IPI’.GO evidence codes have been used extensively to divide most annotations into two broad categories: those based on experimental evidence, and those based on computed or automatic evidence, the latter of which is often associated with high-throughput data. Automatically curated annotations can be identified in the GO annotation set primarily by the ‘IEA’ (inferred from electronic annotation) abbreviation. Such an evidence code is referenced by ECO through the common practice of assigning a database cross-reference (‘dbxref’), where a term in one ontology is linked to an analogous term in another ontology or other resource. In this particular case, the ECO term ECO:0000501 ‘evidence used in automatic assertion’ is the cross product of ‘evidence’ (ECO:0000000) and http://www.geneontology.org/doc/GO.references) to describe various curation methodologies that can be captured in the ‘Reference’ field of a GO annotation. A specific combination of a GO_REF and a GO evidence code is considered equivalent to a particular granular ECO class. For example, whereas the GO abbreviation ‘IEA’ by default maps to ECO:0000501 ‘evidence used in automatic assertion’, the ‘IEA’ evidence code in combination with GO_REF:0000002  for detTo amplify the benefits of experimental knowledge that curators capture, the GO Consortium is using a phylogenetic tree-based approach to generate manually reviewed, homology-based annotations for a broad range of species . It is bhttp://microbialphenotypes.org, manuscript in preparation) has been developed to standardize phenotypic information capture for diverse microbes. There are numerous classes of evidence associated with the study of microbes including evidence resulting from growth assays, motility assays, biochemical tests and antibiotic resistance tests. Just as it is now routine to make an assertion that a particular gene product has a particular function, one can also make an assertion that a particular genotype displays a particular phenotype. The phenotype can be captured using an OMP term, and the evidence for determining the phenotype will be captured with an ECO term. For example, if a mutation causes normally motile cells to have a decreased ability to swim, one will capture this by linking the OMP term ‘decreased motility’ to an identifier for the mutant strain. The evidence for this annotation would be captured with an ECO term such as ‘motility stab test evidence’. Thus, researchers will be able to track which types of evidence were used to assess which phenotypes. It is important to note that phenotypes can also be predicted from whole-genome or transcriptome analysis of individual strains, and ECO is well suited to track these computational evidence types, just as it does for automated annotation of gene function. The application of ECO to capturing evidence associated with phenotypic study, and the resulting annotation of mutant phenotypes with these ECO terms, will facilitate studies in a broad area of applied research. An implementation of ECO can be browsed at the OMP wiki .Characterization of phenotypes is important for many biomedical research applications including clinical identification of microbes, biotechnological applications and for determining protein function through genetic manipulation. The Ontology of Microbial Phenotypes ’. This evidence type encapsulates a chain of events, which is difficult and sometimes impossible to fully capture as an equivalence axiom using OBO format. To better model the experimental complexity, future ECO development will be migrated to OWL, which allows for greater expressivity. This will not affect the majority of ECO users, who will still have the option of either OBO or OWL versions.One advantage of the increased expressivity of OWL is the ability to directly model chains of evidence. Patterns involving sequences of arbitrary length can be easily modeled in OWL using nested class expressions—for example, an evidence chain involving sequence similarity based on mutant phenotype can be described with an expression ‘based on’ some (‘sequence similarity’ and ‘based on’ some ‘mutant phenotype’). Even if the need for this complex modeling is not immediately required, migrating the base version of the ontology to OWL will allow the use of tooling such as OWL reasoners to help maintain the ontology.Another promising area of development that will enable ECO to include complex terms lies in recent collaborative work undertaken with developers of the OBI . OBI is At present, ECO contains some evidence terms that arise from processes not yet instantiated in OBI. Conversely, OBI has numerous classes whose outputs are not represented in ECO. Where possible, ECO makes use of the OBI ID as definition provenance . We acknowledge that this is a weak form of integration, and we aim to continue to work with OBI to increase interoperability and synergistic growth where possible.The IAO is an offshoot of OBI designed to model pieces of information, including those pertaining to biomedical investigations. It may be the case that ECO will be conceived of as an extension to IAO, with the root node of ‘evidence’ being a subtype of information artifact.Currently, ECO is neutral with respect to upper ontologies, and could potentially be used in combination with different upper ontologies such as Basic Formal Ontology (BFO)  or the SWith respect to weighting types of evidence, clearly there is a perceived difference in quality between different types of evidence. For example, experimental assays are generally viewed as the gold standard for evidence and thus more reliable than other types of evidence such as computationally derived sequence-based methods. However, in fact when examined carefully, there is huge variation in quality and reliability in all types of evidence, both experimental and computational. It is not possible, based on evidence type or assertion method alone, to determine the quality of evidence. Thus, ECO evidence terms do not carry implicit assumptions about the quality of a particular type of evidence or measures of confidence in different types of evidence. Each method has inherent strengths and challenges for identification of underlying biological knowledge at a certain level of granularity. The kind of evidence, along with the selection of the annotation terminology that is consistent with the level of granularity uncovered by the method, combines to provide a level of confidence in the assertion.http://wiki.isb-sib.ch/biocuration/Workshop_follow-up). It was clear that there is a desire in the community to capture quality/confidence metrics. Although the best way to achieve this has not been decided, ECO will be part of the community effort to solve this challenge.Thus, an independent and orthogonal method must be developed to capture annotation evidence quality and confidence. Although this is not currently within the scope of ECO, we are eager to work with the broader community to establish ways in which this can be captured by making use of resources such as ECO. For example, at the 2012 International Society for Biocuration meeting, ECO developers participated in a workshop hosted by researchers from the Swiss Institute of Bioinformatics titled ‘Quality information in support of annotations’ (http://evidenceontology.org) and development site (http://code.google.com/p/eviden ceontology) that is being used by the community. Initial improvements to ECO included structural modifications to the ontology, batch term renaming, spelling corrections, term merges, term obsoleting, completing definitions and creating a new root class, as well as generation of internal cross products. Work on normalization continues. These changes were essential to promoting usability of ECO and have resulted in the growth of our user community. Several major annotation resources now use ECO for facilitating information retrieval and quality control. We continue to pursue several areas of development including collaborations with orthogonal ontologies such as OBI. ECO development will continue to be responsive to user needs, and we welcome user feedback and requests at the ECO tracker site (http://code.google.com/p/evidenceontology/issues/list).The past several years have seen ECO grow into an independent, robust and actively developed ontology. ECO has a centralized Web presence (


Objectives: The aim of this study was to review and compare current healthtechnology assessment (HTA) activities for medical devices across non-European Union HTAagencies.Methods: HTA activities for medical devices were evaluated from threeperspectives: organizational structure, processes, and methods. Agencies were primarilyselected upon membership of existing HTA networks. The data collection was performed intwo stages: stage 1–agency Web-site assessment using a standardized questionnaire,followed by review and validation of the collected data by a representative of the agency;and stage 2–semi-structured telephone interviews with key informants of a sub-sample ofagencies.Results: In total, thirty-six HTA agencies across twenty non-EU countriesassessing medical devices were included. Twenty-seven of thirty-six (75 percent) agencieswere judged at stage 1 to have adopted HTA-specific approaches for medical devices(MD-specific agencies) that were largely organizational or procedural. There appeared tobe few differences in the organization, process and methods between MD-specific andnon–MD-specific agencies. Although the majority (69 percent) of both categories of agencyhad specific methods guidance or policy for evidence submission, only one MD-specificagency had developed methodological guidelines specific to medical devices. In stage 2,many MD-specific agencies cited insufficient resources , lackof coordination (between regulator and reimbursement bodies), and the inability togeneralize findings from evidence synthesis to be key challenges in the HTA of medicaldevices.Conclusions: The lack of evidence for differentiation in scientific methodsfor HTA of devices raises the question of whether HTA needs to develop new methods formedical devices but rather adapt existing methodological approaches. In contrast,organizational and/or procedural adaptation of existing HTA agency frameworks toaccommodate medical devices appear relatively commonplace. Health technology assessment (HTA) seeks to provide policy makers with information on theclinical and economic value of health technologies , clinical procedures, and organizational systems used in health care) to inform theirreimbursement or coverage decisions .Several surveys have been undertaken to date that describe and compare the HTA activitiesacross organizations and countries –6. For eThe evaluation of non-drug technologies, in particular MDs, may pose different challenges tothose of pharmaceuticals. Compared with drugs, undertaking randomized controlled trials areoften more difficult, product modification occurs more frequently, pricing is more dynamic,and clinical outcomes are more depending on training, competence, and experience (“learningcurve”) of the operator ;10. It iwww.medtechta.eu), funded under the EU's 7th Framework Program (FP7). Under theconditions of this EU-FP7 funding, the MedTecHTA project was directed to undertake a survey ofMD activities in non-EU HTA agencies. The EU-FP7 funded ADVANCE-HTA project (www.advance-hta.eu) projectis responsible for a parallel survey of MD activity in EU HTA agencies.The aim of this study was to undertake a survey of non-European Union (EU) HTA agencies tocharacterize and contrast their activities for MDs in terms of: (i) organizational structure;(ii) processes, that is, standard operating procedures in use; and (iii) methods, that is,scientific methodologies and assessment in place. This study was undertaken as part of theMedTecHTA project ; Processes: how does the organizationconduct HTA and involve stakeholders? ; andMethods: what methodologies are used for an HTA? .Our assessment of HTA for MDs was based on the key principles for the improved conduct ofHTA proposed by Drummond et al. . HTAactWe undertook a two-stage data collection process. In stage 1, we performed: (i) a reviewof the literature to understand the HTA system in each country, (ii) a review of agencyWeb sites using a standardized questionnaire, and (iii) review and validation of thecompleted Web site questionnaires by a senior representative of the agency. In stage 2, weconducted semistructured telephone interviews with key informants of the agencies thatwere identified at stage 1 as having developed specific HTA activities for MDs .First, we reviewed published literature describing the HTA system in each of theincluded countries to understand the interactions and players involved from a macroperspective. This involved a Medline (PubMed) search ). From this information, we developed a “map” summarizing thelinks between HTA agencies and other key stakeholders  ;16.A questionnaire based upon a literature review of HTA of MDs and previous HTA survey;17;18 waFor verification of the collected data, a key strategic individual within each agency was e-mailed a copy of ourcompleted Web site questionnaire assessment and the country level HTA map. Afterverification, if applicable, these individuals were asked if they would participate in afollow up semistructured interview.Interviews were conducted in the subgroup of HTA agencies identified from Stage 1 tohave MD-specific processes in terms of their structure, processes, or methods. Theobjective of the interviews were: (i) to clarify items reported in the Web based datacollection process and (ii) to determine how the organizational framework mitigateschallenges associated with performing HTA on MDs. If individuals did not respond to theinitial invitation e-mail letter and the reminders, other key informants of the agencywere identified and asked if they would participate. Interviews were by telephone andsemistructured in format, lasting approximately 1 hour, and conducted by two members ofthe research team. We sought permission to audio-record at the outset of eachinterview.Data collected from the stage 1 analysis of agency Web sites is presented descriptively.Categorical and continuous data were summarized as frequencies and percentages or asmedians and ranges, respectively. Stage 1 results are aggregated across all agencies andpresented in a detailed tabular form. Given the objective of this study, aggregate resultsare also presented separately for “MD specific” and “non–MD-specific” agencies.Stage 2 audio-recorded interviews collected from MD-specific agencies were transcribed.Answers to open questions were reviewed to identify major topics that were recurrentlyreported by the interviewees. These topics were then mapped to the three HTA perspectivesoutlined above, that is, agency structure, processes, and methods. Individual interviewswere then searched for certain expressions reflecting either positive or negative opinions/recommendations. Those expressions werethen linked to the major themes and coded using NVivo 10 qualitative data analysissoftware. To summarize the frequency of item occurrence, the following categorization wasassigned: “some” (10–30 percent of respondents), “many” (31–65 percent), and “most”(66–100 percent) .A total of 135 HTA agencies were identified; after applying our exclusion criteria, 36agencies across 20 non-EU countries were included for analysis .Web site data collection was completed for all thirty-six HTA agencies between June andSeptember 2013; thirty-one (86 percent) data collection sheets were validated by anagency representative. Twenty-seven of the thirty-six agencies were considered to be“MD-specific” because they reported MD-specific information about either theirorganizational structure , processes , or methods  in stage 1 . The mosThe organizational structures of the MD- and non–MD-specific HTA agencies are compared inA summary of agencies' processes for conducting HTA is presented in All twenty-seven HTA agencies that were judged in stage 1 to be MD-specific wereinvited to participate in a follow up interview process (stage 2). Eighteen (67 percent)agencies agreed to be interviewed. In most cases, the interviews were conducted bytelephone with the same agency individual(s) who verified the Web assessmentquestionnaire. In one case, a telephone interview was not possible and instead responseswere obtained by e-mail. After an iterative process of qualitative data analysis, thefollowing seven broad themes were identified: (i) capacity; (ii) complex interventionand co-dependency; (iii) decision making; (iv) evidence; (v) coordination; (vi) decisionproblem; (vii) transferability. In a next step, these themes were subsumed under thethree dimensions of HTA used earlier in this report . A tabulated summary of the results of the semi-structured interviews isavailable in Supplementary Table 1.Some interviewees indicated that their organizational structure effectively evolves tomeet the needs of decision makers as it relates to HTA of MDs. By adapting theorganizational structure according to the objectives outlined in policy requirementsrather than by an HTA “codebook,” agencies contend that they are better positioned toefficiently inform key funders about the medical, social, economic, and ethical issuesassociated with introducing a new technology into the health care system. Oneinterviewee from an MD-specific agency purported:“The structure of the HTA has evolved to meet the funding programs; it directlyreflects that need and not the other way round (not build on an HTA-centric view). HTAserves the need of the policy requirement and its organizational structure fits withwhat is expected from it.”They find that this approach reduces the time needed to translate MD-specific evidenceinto the health system.Many interviewees cited insufficient resources  as alimitation, creating hurdles in the agency's ability to adapt to policy needs for HTA ofMDs. For example, one key informant explained that the lack of resources has atrickle-down effect on the way in which their agency is able to examine MD evidence:“Not enough [MD] experts – there is a very large gap there. So we only look atevidence in the sense of systematic reviews, meta-analysis.”Some interviewees indicated that they integrate external academic partners into the HTAprocess for MDs when they do not have staff with the appropriate skills to performevidence assessments. In fact, the use of external networks was even cited as an assetby some interviewees. For example:“I think some of the real operational issues related to medical technologies areimportant to understand – How this piece of equipment works in the hospital, who'susing it, what happens if it's in a different pair of hands? I think we don'tnecessarily have a lot of that expertise in house. I'd say one of our strengths isbuilding those networks.”Some interviewees noted the value of integrating subject matter experts in the HTAprocess for MDs. The input of experts fosters stakeholder buy-in on policy decisionsresulting from evidence appraisals. Furthermore, it supports the contextualization of MDevidence and the introduction of MDs into the health system.Another issue identified by many interviewees is that the introduction of MDs into thehealthcare system is diverse and not standardized:“For devices, our health care system is extremely fractured. Devices can comeinto our system at a regional level, hospital level, within a particular doctors'office, at a provincial level, or even at a national level. So, for devices we have anextremely fractured system for entry points. Because of that, we have different kindsof evidence requirement at different kinds of levels. I would go so far to say at somelevels there is no rigorous evidence assessment taking place.”Although many interviewees acknowledge that the procedure used to review devices,including evidence requirements, is inextricably linked to the point of entry into thehealth care system, they expressed discontent with the poor levels of evidence requiredfor pre-market access for MDs.Many also found fault with the level of harmonization between the regulatory andreimbursement bodies' evidence requirements. In consequence, this imbalance leads tounnecessarily prolonged review processes.“The fact of having an evaluation of sanitary registration for marketing thatdoes not use the same principles as are used for incorporation in the Public HealthSystem, are considerable obstacles in HTA for MD.”Another topic was how agencies manage complex interventions, that by definition includeMDs, and the evidence assessment process for co-dependent technologies, for example,delivery of drug using a device technology. Responses were suggestive in nature,including recommendations on how to improve the management of MDs in complexenvironments as well as harmonize the review process by having co-dependent technologiessimultaneously reviewed by drug and device assessment committees.Some agencies noted that the approval channels integrated into their standard operatingprocedures mitigate potential challenges associated with translating MD evidence intopolicy. For example, one key informant describes current review processes as sufficientfor advising decision makers on whether to adopt a new technology:“In health technology assessment term, I think there is not any raised problemof HTA of MDs. The applied HTA of MDs has been discussed and assessed by sub-committeewhich consist of qualified medical doctors in their specialty, and has beenre-discussed internally whether to accept, modify or reject the assessment ofsub-committee.”Most interviewees stated that there is a lack of methodological guidance for MDs.Furthermore, they identified several shortcomings with evidence requirements for MDs.“The main problem of health technology assessment is systematic literaturereview that search, analyze and contemplate existing studies […] The main problems ofliterature review are that any newly emerging technologies which have potentialbenefit for patients but lack of literature evidence will not be approved as newtechnologies.”The quality of evidence  was cited severaltimes as the main challenge for performing HTA of MDs.“The main challenge is that there is not enough good evidence […] I think thatthere is a broad societal question of how we will ever collect enough data to answerall the questions we need to answer. I think that that's really the biggestchallenge.”The quality of evidence was also described as being a limiting factor for the scope ofdevice assessments.“There are certainly again limitations on what we can do with the devices giventhe nature of the evidence base.”The MD industry was identified by some interviewees as a “conspirator” to poor evidencegeneration. As a result, agencies experienced challenges with obtaining the appropriateevidence for HTA of MD.Some interviewees identified human resource constraints as a great challenge whenattempting to apply scientific methods to critique MD evidence. The range of staffskills tend to be limited. An interviewee described how they are addressing skilldeficiencies such as the ability to appropriately evaluate various types of evidence ordata.“We've also recognized the need to put a real focus in terms of getting ourstaff comfortable with qualitative research. I'm not sure we'll get it to where itneeds to be in the next year, but at least getting people to understand whatqualitative research can bring to an assessment on the non-drug side and especiallywith respect to the non-clinical, non-economic of HTA. Within the team that we havehere, we're working to build awareness of what it is and why it's important and thento be able to take on some projects to get people actually using it and doing it andthen get their comfort level up.”Many interviewees identified the inability to generalize the findings obtained from MDevidence reviews to their local context as highly problematic.“The methodology of HTA of MDs can be really difficult. All we know is generallytranslated from drugs to devices but we know that differences exist, for instance theperformance of devices in clinical practice can be very different from that assessedin controlled setting.”This weakness was attributed to device-specific challenges, such as the learning curve.Interviewees believed that intervention evidence, typically generated in environmentswith highly skilled staff, may not be representative of the health system where thetechnology will be introduced:“Such another issue, an issue that probably exists here and elsewhere is thatthe early use and early evidence on devices tend to be in individuals with “the besthands problem”- so specialized academic centers that either develop technologiesthemselves or have used it for a number of years. So it's very difficult to thengeneralize what the findings might be in the community.”In addition to providing a detailed survey of the practices and methods of acomprehensive sample of non-EU HTA agencies, this study specifically sought tocharacterize and compare current HTA activities for MDs across these agencies. Inparticular, we assessed agencies in terms their organizational structure, processes, andscientific methods. All 36 HTA agencies identified within our sampling framework undertookthe evaluation of MDs. A substantive proportion  of HTA agenciesreported that they had developed specific approaches within their organizational structureor procedures for the assessment or appraisal of MD technologies (“MD-specific agencies”).For example, organizational developments included the allocation of specific staff to MDsassessment or the setup of a completely separate program or unit within the agency fordevice evaluation. Procedural developments could include convening a specific committee toappraise MDs evidence and provide policy advice. There appeared to be few differences inthe organization, process, and methods of between MD-specific and non–MD-specificagencies. Although several agencies had a methods guidance document or policy for evidencesubmission, only one MD-specific agency (Department of Science and Technology in Brazil)had developed specific scientific methodological guidance for the HTA of MDs .Our interviews confirmed several commonly cited challenges in the HTA of MDs –11.ThesTM catheter renal ablation therapy for treatment-resistanthypertension), specifically designed to facilitate parallel submission to Health Canada,for regulatory purposes, and to the Ontario Health Technology Advisory Committee  program recentlyestablished in Ontario , has beeommittee . Other oommittee .Whereas there have been several previous surveys of international HTA agency practice, webelieve this to be the first international survey of HTA practice to focus on MDsevaluation. Three previous studies have included aspects of their review of HTA practiceto the consideration of MDs. In 2012, Stephens et al. published a survey of thirty HTAorganizations representing sixteen countries including Europe, North and South America,and Australia . In accoThe main strengths of this study include the comprehensive and systematic approach to theselection of HTA agencies, high levels of response both to the request for verification ofour agency Web-site surveys and interview invitations, and quality assurance measures fordata collection .between agency comparison)was conducted. It is possible that a within agency analysis  may have been more sensitive to detecting differences. However,the data collection methods were not designed to undertake such a within agency comparisonof HTA approaches as the available Web-site material does not often differentiate betweenthe evaluation of differing health technologies.However, we acknowledge several potential limitations. As the scope of this survey waslimited by the funder of the project to consideration of non-EU HTA agencies, our findingsmay not be directly generalizable to the EU jurisdiction. A consequence of thecomprehensiveness of our survey was the inclusion of a heterogeneous group of HTA agencies. Although we largely took a summative (“average”) approach to thepresentation of results across agencies, where possible, we did report ranges. Theassessment of agency Web sites for some countries  was limited to text andmaterial available in English, excluding content in other languages. However, to minimizethe risk of this retrieval bias, key informants reviewed the accuracy of the country-levelanalysis and Web-based assessment form. An alternative approach that may have led to moredirect collection of data would have been to directly request agencies to self-complete aquestionnaire. Although this later approach has been undertaken by several internationalsurveys of HTA agencies, it has the major limitation of a low response rate. For example,the ISPOR and EUNetHTA surveys that both used questionnaires reported response rates ofonly 42 percent and 36 percent, respectively ;6. FinalOur findings raise some important implications. First, in contrast to the regulatoryrequirements that appear fairly consistent across international settings , we founThis survey provides a unique overview of organization and practices of non-EU HTAagencies. It is widely cited that MDs differ from other health technologies, inparticular, they can change rapidly, clinical outcomes often depend on training andexperience of operator, and pricing is more dynamic than drugs. The fact that all thenon-EU HTA agencies included in this survey undertake the evaluation of MDs and do soalongside to other health technologies such as pharmaceuticals, supports the principlespublished by Drummond and colleagues for good HTA practice: “HTA should include allrelevant technologies” . HoweverFor supplementary material accompanying this paper visit http://dx.doi.org/10.1017/S0266462315000185.click here to view supplementary materialAll authors declare no conflict of interest.


The widespread use of meshes for hiatal hernia repair has emerged in the era of laparoscopic surgery, although sporadic cases of mesh augmentation of traumatic diaphragmatic rupture have been reported. The indications for biologic meshes in diaphragmatic repair are ill defined. This systematic review aims to investigate the available evidence on the role of biologic meshes in diaphragmatic rupture and hiatal hernia repair. Limited data from sporadic case reports and case series have demonstrated that repair of traumatic diaphragmatic rupture with biologic mesh is safe technique in both the acute or chronic setting. High level evidence demonstrates short-term benefits of biologic mesh augmentation in hiatal hernia repair over primary repair, although adequate long-term data are not currently available. Long-term follow-up data suggest no benefit of hiatal hernia repair using porcine small intestine submucosa over suture repair. The effectiveness of different biologic mesh materials on hernia recurrence requires further investigation. Blunt or penetrating trauma of the abdomen and thorax may cause injury to the diaphragm . In the Relevant literature evidence is limited, mainly due to the rarity of the condition. Primary suture repair or covering the defect with a synthetic mesh has been the standard of care during the past decades . BiologiIn the presence of insufficient evidence, there is ongoing debate on the need of augmentation of the diaphragmatic hiatus during hernia repair . A numbeElectronic searches of the Medline database were conducted using the PubMed search engine. The following combination of terms and keywords was applied: (trauma OR traumatic OR posttraumatic OR rupture*) AND (diaphragm* OR phren*) AND (mesh OR implant). The search returned 141 reports. The last search was run in November 2014. Titles and abstracts were interrogated and clinical reports on the use of biologic material for closure of traumatic diaphragmatic defects were selected. The full texts of 17 articles were assessed for eligibility; three relevant reports were identified –17. The Similarly, Medline was searched to identify relevant clinical evidence using the PubMed interface up to November 2014. The keywords (hiat*) AND (hernia) AND (mesh OR implant) were used. Of a total of 309 records, 28 articles were selected for full text review based on relevant information from titles and abstracts. Twenty-two articles provided relevant outcome data on mesh-reinforced hiatal hernia repair with biologic meshes –39. The Two case reports and one case series reported on the use of biologic meshes in traumatic diaphragmatic rupture. Four chronic traumatic defects and two acute ruptures were repaired laparoscopically, or with a laparotomy or a combined (thoracotomy and laparotomy) approach using human acellular cadaveric dermis (HACD) or porcine small intestine submucosa (SIS). Two of the repairs were performed in contaminated surgical fields, one due to inflammation of the herniated gallbladder and one due to pleural empyema. No septic complications requiring prolonged hospital stay or reintervention were reported. Chest X-ray in five of these cases did not reveal recurrence within a 6- to 24-month follow-up period.A plethora studies reporting use of biologic mesh augmentation of the esophageal hiatus have been published since 2003. Most of these are retrospective industry-sponsored cohort studies. Both HACD and SIS meshes have been used, most commonly in a U-shape or a pantaloon fashion, placed in a retroesophageal position with the limbs of the mesh encircling the esophagus. The graft is anchored to the diaphragm and, in some cases, to the esophagus with non-absorbable sutures, tacks, or fibrin sealant, most commonly following suture repair of the crura or in a tension-free bridging fashion. A Collis gastroplasty has also been reported as a lengthening procedure in cases of a short esophagus , 32. AltThe best available evidence is provided by two well-designed RCTs –23, 39. In a recent double blind RCT that was sponsored by a national authority, suture mesh repair was compared with SIS or collagen-coated titanium mesh augmentation of the hiatus; similarMost authors have focused their interest on potential beneficial effects of biologic grafts in paraesophageal hernia. In a cohort study, Schmidt and colleagues compared suture repair and mesh augmentation with HACD in small hernias  . A benefLimited evidence exists investigating the role of biologic meshes in traumatic diaphragmatic repair. Low quality evidence (Level 4) suggests that this approach is feasible, at least in chronic cases. Biologic meshes have also been used in contaminated surgical fields with favorable results (Level 5). Because of the difficulties randomizing patients in the acute setting and the rarity of this condition, clinicians should be encouraged to publish their experience with biologic meshes in traumatic diaphragmatic rupture.p = 0.003)  . HoweverThe impact of type of biologic graft on hernia recurrence remains to be investigated. Further experimental and clinical research is required to assess new biologic implants in hiatal hernia repair. Although current data have shown SIS implants to be associated with high recurrence rates, other biologic materials have not been adequately investigated. Considering the rarity of cases with traumatic diaphragmatic defects, the effectiveness of biologic implants in such situations may be extrapolated from evidence derived from hiatal hernia repair. Future RCTs are required to investigate the role of biologic meshes in both paraesophageal and small hiatal hernias and evaluate their comparative efficacy to synthetic meshes.Conception and design: SA, FK. Acquisition and interpretation of data: SA, FG, RP. Drafting the work or revision for important intellectual content: SA, FG, RP, FK. Final approval: SA, FG, RP, FK. Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved: SA, FG, RP, FK.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.