The gaps observed between the use of research evidence and policy have been reported to be based on the different methods of using research evidence in policymaking by researchers and actual policymakers. Some policies and policymaking processes may therefore be particularly well informed by research evidence compared to others. The aims of the present article are to explore the use of research evidence in health-enhancing physical activity (HEPA) policies, identify when research evidence was used, and find what other types of evidence were employed in HEPA policymaking.Multidisciplinary teams from six EU member states analysed the use of research evidence and other kinds of evidence in 21 HEPA policies and interviewed 86 key policymakers involved in the policies. Qualitative content analysis was conducted on both policy documents and interview data.Research evidence was mostly used to justify the creation of HEPA policies and, generally, implicitly without citation. The policies analysed used many types of evidence other than citable research. The evidence used in HEPA policies was found to fall into the following categories: societal framework, media, everyday knowledge and intuition, research evidence, and other types of evidence.Research evidence seems to be the only type of evidence used in policymaking. Competition between the use of other types of evidence and research evidence is constant due to the various sources of information on the Internet and elsewhere. However, researchers need to understand their role in translating research evidence into policymaking processes. The health benefits of physical activity (PA) are supported by research evidence as well as the international public health community and its policies, including the World Health Organization (WHO) . RegularHowever, there are strong economic and moral, and occasionally political, arguments for making better use of research evidence in policymaking . EconomiPrevious international research on HEPA policies –22 has lInstitutions, interest groups, and networks interact with individuals and their ideas at different times and places during the policymaking process . In poliThis study is part of the EU-funded project Research into POlicy to enhance Physical Activity (REPOPA), with the overall aim of integrating scientific research knowledge, expertise, and real world policymaking processes. This could increase synergy and sustainability in promoting health and preventing disease, and promote physical activity in structural policymaking through different research and networking activities. The aims of the present article are to explore the use of research evidence in HEPA policies, identify when research evidence was used, and indicate which other types of evidence were used in HEPA policymaking. The study may be considered valuable in providing information on effective translation of research evidence into HEPA policymaking processes, which would ultimately lead to more effective HEPA policies.Each research team in the partner countries suggested relevant HEPA policies for consideration. Prior to the selection of HEPA policies for the study, the research team, e.g. principal researchers in each partner country, looked at the objectives, primary stakeholders involved, subpopulation specificity of policies, processes leading to release of the policies, and selection of documents used for the development of the policies. The policies were then categorised as national, regional, or local. Based on this primary inventory and in order to focus on the most recent developments, documents were selected according to the criteria of policies focusing on HEPA, being published by public authorities and representing HEPA policies in force at the beginning of 2012. The selected national, regional, and local HEPA policies ensured at least some variation in the scale and topic of the expected policy change: from major changes, such as governance structures and legislative perspective, to minor changes such as programmes, advice services, or building playgrounds. Some of the national, regional, and local policies belong to a ‘package’ of national, regional, or local policies created after elections for respective development purposes. They therefore also reflect wider policymaking processes and possibly the use of a wide base of research evidence. In the case of ‘package’ policies, the study concentrated on the specific policy documents for HEPA issues only. For the final set of HEPA policies, 21 policies from six European countries were selected with variation across the types of policies and policymaking processes. The topics of the sample of HEPA policies included in the analysis contained public health aspects , HEPA aspects , infrastructure , and places for PA  . The stuTo assess the use of research evidence in HEPA policies, content analyses of policy documents and semi-structured interviews were used. The content analyses of the policy documents and the interviews complemented each other to identify the various ways of using evidence in policymaking. The intention was to facilitate comparisons between countries in the use of research evidence in policymaking and take into consideration contextual differences through interviews. In each country, a policy was identified along with the content, stakeholders, and processes in relation to the relevant research. A series of questions was put to the stakeholders and policymakers to recall the use of research evidence. Further questions were allocated depending on the references to the use of research evidence in speeches, statements, guidelines, and similar background documents.The HEPA policy process was split into agenda-setting and policy development phases in order to identify the use of research evidence at different stages of the policymaking process on various topics and issues. The agenda-setting phase and the use of research evidence were mostly traced through interviews and the use of research evidence in the policymaking process through policies or other supporting documents. The agenda-setting phase and the use of research evidence were dependent on interviews, as most of the countries did not have records on the decisions made in the agenda-setting phase, whereas the policymaking phase was often described in policies or related background documents.Based on the literature review for the project proposal and its update upon acceptance of funding for the use of research evidence in policymaking, the study was undertaken using a qualitative descriptive approach inspired by political sciences , 31, pubThe content analysis of HEPA policy documents followed the ideas of Ritchie and Spencer  and consWhere appropriate, a published protocol and tool developed by Lavis et al.  and HannThe common guideline that was developed covered the criteria for the selection of policies, theoretical models of the policymaking phases based on Kingdon , the focDuring the HEPA policy document analysis, key informants for each policy were identified for face-to-face semi-structured interviews in order to verify the findings of the content analysis stage and gather information on any gaps in the uncited or implicit use of research evidence. The purposeful sampling of 87 informants in six countries was based on the selection criteria of the interviewees being directly involved in the policymaking process and able to report on the use of research evidence or other kinds of evidence in the policymaking process. In each policy, one of the interviewees was employed at the organisation responsible for the policy. The interviewees had been involved in policymaking processes of the policy on which they were interviewed and were policymakers, researchers, public sector officers, or other influential stakeholders. All the interviewees were contacted by email or phone by the research team in each country with basic information on the project and consent forms in the local language. The interviews were conducted by research team members with backgrounds in health and social sciences in the local language, recorded when accepted and transcribed for the analysis. An interview guide was developed by the REPOPA team led by the work package leader. In the guide, the questions for the policymaking process were split into agenda-setting and policy development phases. The questions followed the protocols , 15 withTo adapt the interviews for each context, each country team conducted between one and three pilot interviews to modify the questions, interview process, and language.The semi-structured interviews were selected as a method to gain complementary information and to facilitate the adaptation of questions to each policy and case. The stakeholder interviews for each policy verified facts identified during the content analysis of policy documents, gaps in the information gathered in the use of research evidence, and the needs of policymakers to use evidence in the policymaking process.In the semi-structured interviews, the interviewees were asked to recall the policymaking process period, review their files before the interview, and explain the policymaking process and the parties involved. The main issues in the interviews were how and why research evidence and other types of evidence did or did not enter into the policymaking, as well as the origin, influence, and characters of policymaking. The topics covered included policy changes, their significance, the use of research evidence and other types of evidence in bringing the content and change into the policy, the factors describing how and why the issues appeared on the policy agenda, and the factors influencing the way the policy developed.The selected HEPA policy documents and interviews were mapped, coded, and further analysed using the interview questions as guidance. Each HEPA policy and transcribed interview was reread, coded, sifted, charted, and sorted according to the key issues and themes, confirming the patterns of use of research and other evidence, and categorised into various topics and themes. The interviews were analysed by the country-based research teams using an interpretative approach derived from a content analysis of the policy . The codDemographic and statistical data (facts and reports used as a background or for prioritising policy areas)Non-systematically derived peer-reviewed scientific literature Systematically derived peer-reviewed scientific literature International, transnational, national, regional, or local standards and guidelinesKnowledge derived from community consultations, stakeholder workshops, and in-house consultationsNational reports Knowledge derived from expert consultations or policy briefs developed by research institutionsThe policies and interviews were further analysed to identify the use of evidence to support the policies. For the distinction between research evidence and other types of evidence, a list of evidence with modifications was developed based on Lavis, Ross, and Hurley . EvidencIn the content analysis stage, the policy documents were reviewed to identify the explicit use of research evidence. When research evidence was cited, the content was analysed to establish the way the citation supported the policy statements or position in addressing the issues. The research evidence was published as journal articles, book chapters, working papers, or reports typically produced by research institutes, universities, or other independent research units. Other types of evidence referred to readings, information from media, and interaction with peers or stakeholders. Involvement of people in hearings, working groups, meetings, and the like was also looked for in the policy documents and interviews.Finding patterns when research was used required a framework for determining the contexts in which policymaking occurred , 27, 40.The interviews complemented the understanding of the use of research evidence in the agenda-setting and actual policymaking phases of HEPA policies. Policy documents  providedBy contrasting and searching for internal patterns, connections, and explanations for themes, categories, or issues in the content analysis, it was possible to describe the HEPA policymaking process, involvement of stakeholders and use of research evidence, and other types of evidence for judging HEPA policies for, for example, various subgroups. Sometimes, research evidence on health determinants and efficacy of interventions was used to legitimise actions, with other types of evidence used to accommodate contextually salient factors such as culture, community and organisational values, resources, and political priorities, which defined the usefulness of evidence for HEPA policymaking and implementation.Explicit citable use of research evidence was mostly found in the justification of HEPA policies and when stating a specific study or publication as a trigger for a policy. It seemed that the visions and goals of the policies were not validated by research evidence but by other types of evidence. For the justification of the policies, instead of using peer-reviewed scientific articles, the policymakers used various types of secondary publications, such as national and international reports and recommendations. In other parts of the policy, such as target setting or actions for target groups, explicit research evidence was rarely used.Neither policy document analysis nor interviews provided information on whether explicit or implicit evidence or citable research evidence was really influencing policies or whether implicit evidence was informed by research evidence.The study found that most countries did not have routine reporting mechanisms for policy decisions during policymaking processes that would have been accessible for this study. Some countries had records available in the form of media communications, lists of members of the various committees, working groups or legislative commissions, and report databases. Theoretically, in the agenda-setting phase, priorities are set out for the policymaking process and in the actual policymaking development phase the research evidence should support the issues to be considered, chosen, and selected as part of the policy. Nevertheless, in the studied HEPA policies, the decisions made in the agenda-setting and policymaking phases were not recorded and therefore the information on the use of research evidence was based on what was found in the documents and their references and in the recall of interviewees.Where research evidence was used, it was identified in an ad hoc manner in the policymaking phase and consisted of epidemiological research, population studies or statistics, and case studies. Peer-reviewed research articles and research based on surveys, as well as monitoring, evaluation, and implementation studies were rarely used. When the HEPA policies used citable research evidence, it was not necessarily peer-reviewed scientific articles. Instead, various types of national and international reports and recommendations were used. In most cases, when paragraphs in the policy documents suggested being informed by scientific knowledge, explicit research evidence was seldom referenced.According to the interviews, the research evidence used in HEPA policymaking was based on previous strategies, programmes, recommendations, guidelines, alignments, traditions, political support, trends, legislation, or economics Figure . In the Policymakers obtained other types of evidence from experiences, such as earlier national or international good practices, projects or programmes, or information gathered from personal networks Figure . MeetingThe most challenging part of the different types of evidence used in the policymaking was the use of the Internet and media information for policymaking. The challenge was even stronger when the policymaking process had occurred some years ago, which was the case for most of the policies. According to the interviews, for the political level of policymaking, the media and its attention to certain topics during the policymaking process seemed to be an important source and legitimation of information and concern for taking up the issue in the policy.Based on the content analysis of policy documents and interviews, the types of evidence used were listed, the items from which were classified and categorised; these five broad categories, namely research evidence, other types of evidence, societal frameworks, media, and everyday knowledge and intuition, are depicted in Figure The aims of the study were to assess how, to what extent, and where research evidence and other evidence were used in HEPA policymaking. In this qualitative study of the content analysis of 21 policy documents and 86 stakeholder interviews from six European countries, it was found that most countries did not have routine reporting mechanisms for policy decisions using research evidence during the policymaking processes. This can be considered a weakness in the transparency of the policymaking processes. Other related reports were available, but they were not necessarily related to policies or policymaking processes as such, they did not reflect the chosen policies, or particularly explain, describe, or confirm the use of specific research evidence in policymaking. Overall, from the content analysis of the policy documents, it was established that there was a lack of citable research evidence use in policy documents as it was rarely explicitly expressed. This was also confirmed by the interviews with stakeholders involved in policymaking. It was found that implicit evidence, such as common knowledge, facts, and practices, were primarily used in the policies.However, when investigated in more depth, the study proved that the use of other types of evidence and broader societal framework information (both of which may or may not have been informed by citable research) was common. The use of non-citable research evidence was difficult to define, as many guidelines, standards, and recommendations by international and national organisations are based on citable research but without explicitly showing the citations used. In addition, public hearings and experiences from other sectors and colleagues from networks as other types of evidence confirmed the expectation that various types of evidence were used in policymaking. As shown in a previous study , 42, theThe content analysis approach showed that research evidence seems to be only one type of information used in policymaking, and competition between the use of other types of information and research evidence is constant due to various sources of information on the Internet and elsewhere. To navigate the use of evidence in policy and practice, it is therefore necessary to understand how ideas spread through systems, how decisions and policies are made, and how capacity is required to use evidence in policymaking . The useEvidence-informed policymaking is a contingent, complex system-like, non-linear, and emergent process of producing, managing, and implementing new knowledge. Most authors appear to agree with at least the possibility that evidence-informed policymaking can work as a virtuous cycle in order to improve policymaking, even if there are widely divergent ideas about what constitutes the component parts of the evidence-informed policymaking process, how this can and should be done, and what it should achieve . PolicymThe aim of the study, based on successive phases of policy analysis and stakeholder interviews, was to allow more in-depth identification of evidence use for HEPA policymaking and of the pattern of evidence use. One major limitation of this study was that the interviewees did not recall the policymaking process in detail, even if the policies were still in force. The evidence used in the policymaking process also remained unclear to some extent despite selection of the most recent HEPA policies to minimise recall bias. However, this study clearly shows that there is a need for future policy research to take into account the various types of evidence used in policymaking and to choose a study design that can minimise recall bias. A further study area is the use of the Internet and its sources of research evidence and other types of evidence in policymaking. Additional countries would have increased the validity of the results and a variety of approaches to HEPA and the use of research evidence. However, the selected policies were not necessarily from the same levels due to the non-existence or unavailability of similar policies across the six countries.The use of research evidence identified through this study will be relevant to other researchers and research units working with policymakers at national, regional, and local levels. Similarly, the findings yield useful approaches for consideration by policymakers who work with researchers in academic institutions. These research-policy nexuses are clearly intended for practical applications in policymaking. Not all research is meant for this purpose. Research evidence seems to be only one type of information used in policymaking. Competition between the use of other types of information and research evidence is constant due to various sources of information on the Internet and elsewhere. In addition to the research evidence, other types of evidence from good practices, experiences from networks, work-related knowledge, and public hearings were used in policymaking; for example, values, traditions, interests, and opinions also influence everyday knowledge for policymaking. Moreover, past policies, regulations, and other policies influence and are used to frame policies. However, the use of research evidence seems to depend on close contacts between policymakers and researchers as well as on easy availability of and access to information. Making better use of research evidence in developing HEPA policies requires researchers to understand their role in policymaking and in translating research into policymaking .Consolidating and revising HEPA policies to be more informed by research evidence at national, regional, or local level would strengthen systematic assessment of structures and institutional arrangements of HEPA policies and make them more effective. HEPA policies are embedded within a wider socio-economic context, and HEPA policies should therefore target wider determinants of physical activity and its impact on health. Using more explicit research evidence in policymaking could also offer alternatives and options for research evidence in more transparent ways.


We thank Schwartz for his thoughtful comments, which provide an opportunity for us to clarify some of the points in our commentary, in which we called for greater transparency about the values that influence policy-relevant research . His firSchwartz’s second point is that it is problematic for us to treat tort law and chemical regulations as equivalent contexts, in which weaker standards of evidence can appropriately inform policy. We agree that different standards of evidence may be appropriate in the two contexts. However, we caution against equating the standards of evidence expected in tort law with those expected in more traditional scientific contexts. The tort system requires only a preponderance of evidence (> 50% likelihood) to win a case; this is much weaker evidence than scientists typically demand when presenting or publishing results, and confusion about these differing standards has led to significant legal controversies .a) Advocacy organizations typically have much fewer resources than industry to generate policy-relevant research that serves their interests (b) a large body of evidence indicates that industry funding has important effects on research outcomes, whereas there is less information about how ideological ties affect research  ch e.g., .Schwartz’s fourth concern is that it is important to think about risk–benefit relationships when applying science to public policy. This is a very good point, but it needs to be considered in conjunction with the fact that scientists’ values and assumptions may influence their assessments of the evidence. It would be ideal if scientists could provide perfectly unbiased risk assessments to policy makers, who could then evaluate the risks versus the benefits in order to make policy decisions. Unfortunately, this picture is unrealistic; scientists’ views about the benefits associated with new technologies likely have implicit influences on their risk assessments. For example, evidence from the literature on risk perception indicates that people’s perceptions of risk are influenced by a number of factors, including the voluntariness of the risks, the fairness of their distribution, their familiarity, and the perceived benefits associated with them . Scienti


With the increasing recognition of health literacy as a worldwide research priority, the development and refinement of indices to measure the construct is an important area of inquiry. Furthermore, the proliferation of online resources and research means that there is a growing need for self-administered instruments. We undertook a systematic overview to identify all published self-administered health literacy assessment indices to report their content and considerations associated with their administration. A primary aim of this study was to assist those seeking to employ a self-reported health literacy index to select one that has been developed and validated for an appropriate context, as well as with desired administration characteristics. Systematic searches were carried out in four electronic databases, and studies were included if they reported the development and/or validation of a novel health literacy assessment measure. Data were systematically extracted on key characteristics of the instruments: breadth of construct , whether it was an original instrument or a derivative, country of origin, administration characteristics, age of target population (adult vs. pediatric), and evidence for validity. 35 articles met the inclusion criteria. There were 27 original instruments  and 8 derivative instruments . 22 indices measured “general” health literacy  while the remainder measured condition- or context- specific health literacy . Most health literacy measures were developed in the United States , and about half had adequate face, content, and construct validity . Given the number of measures available for many specific conditions and contexts, and that several have acceptable validity, our findings suggest that the research agenda should shift towards the investigation and elaboration of health literacy as a construct itself, in order for research in health literacy measurement to progress. Health literacy- one commonly-cited definition of which is “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” Although there are many currently available indices, it has been noted that they are not all of equal quality A key focus in recent research on health literacy measurement has been the development of self-administered indices. These do not require a trained research assistant or clinician to administer, and have been referred to in the literature as ‘indirect’ Therefore, to identify all available self-administered health literacy measurement indices, we undertook an overview of these measures. A primary aim of this study was to assist those seeking to use a health literacy index to select one that has been developed and validated for an appropriate context, as well as with desired administration characteristics. Furthermore, we aimed to inform and understand to what extent the development of additional health literacy measures is appropriate. Our purpose was to identify all available self-reported health literacy assessment measures and report their validity, their administration characteristics, and their intended uses.We conducted systematic searches of the following electronic databases: OVID Medline (1946- April 2014), OVID Embase , and CINAHL (1981- April 2014). The searches were undertaken in two phases: first, on 11 September 2012, then an update search was carried out on 8 April 2014. All references were imported into EndNote X4 .The search strategy was developed with a specialist librarian, with an emphasis on sensitivity to identifying studies reporting the development of condition- and specialty-specific health literacy measures, in addition to measures of general health literacy. Search terms used included literacy, health, assessment, indices, and measurement. The full search strategy is available as supporting information; see In order to capture the literature as broadly as possible, we included studies that reported the development and/or validation of a novel measure of health-related literacy. For inclusion, studies must have reported assessing the validity of a measure, as well as providing some description of its administration characteristics. Generic measures, as well as those focussing on more specialized health literacy such as “nutrition literacy” One author (BO) conducted the search; three authors  scanned abstracts, and selected full text articles for review. The remaining two authors  contributed to developing the data extraction forms and to analysis of the extracted data. We designed and used structured forms to extract pertinent information from each article. One author extracted data from each included reference into a data extraction form in Microsoft Excel; another author (either DG or IRC) independently extracted data from each reference as well. Disagreements were resolved by consensus.Calculations of means and standard deviations were undertaken in Microsoft Excel. Data were systematically extracted on key characteristics of the instruments: breadth of construct , whether it was an original instrument or a derivative, country, time for completion, number of items, age of target population (adult vs. pediatric), as well as evidence for validity. We extracted data relevant to assessing the face validity (the extent to which an index appears to measure what it is supposed to), content validity (how far items of which the index is comprised reflect the concept to be measured), and construct validity (the extent to which the index as a whole measures what it is supposed to) of included measures Our initial database search, carried out on 12 September 2012, identified 2399 references. An update search was carried out on 8 April 2014 identifying 769 references which had been published after the original search was carried out. Hand searching of reference lists of included papers resulted in an additional 45 references. After removing duplicate references, 2431 abstracts were scanned for inclusion and 2047 were excluded. 87 full-text articles were obtained for review, of which 52 were excluded because upon full-text review they did not report the development of a novel or newly validated self-report health literacy measure. Thus, 35 articles reporting the development and/or validation of a health literacy instrument were included , all of Of 35 instruments included in the review, there were 27 original instruments  and 8 derivative instruments , which were either modifications or short-form versions of original instruments . We clasMost of the available instruments to measure health literacy have been validated in the United States , with Australia and Japan being the next most frequent country of origin of health literacy indices . Two indices were validated in each of the following countries: the United Kingdom, Canada, and Germany. There were two measures initially validated in multiple countries: the ‘Graph literacy’ scale (in Germany and the United States) Most included studies reported the setting from which participants were recruited for validation. About a third of the instruments were validated in populations recruited from secondary and specialty care settings , while a quarter of studies recruited in primary care settings . The remainder of validation studies recruited from diverse non-clinical settings , such as schools and community centres, and shopping malls. Two studies The majority of available indices were initially studied and validated with populations between 18 and 65 years old. The mean age of participants in included studies ranged from 18 to 76 years. There were six indices validated in <18 year old populations: the eHealth Literacy Scale (eHEALS) 16 indices were assessed in included studies as having adequate face, content, and construct validity  . 18 indiOnly 16 studies  reported how long it takes to administer the instrument being reported; this ranged from 2 minutes  This review reports the administration characteristics and validity of 35 health literacy measures, and aims to assist selection of an appropriate index. The majority of included indices have been developed and validated in the United States. There is a clear trend towards the development of more measures in recent years. The instruments identified in this review are mostly intended for adults (with only six available for pediatric populations) and have been primarily tested in non-clinical settings. Several instruments take less than 5 minutes to complete, and there are many with adequate validity.Low health literacy is strongly associated with worse health outcomes The measures included in this review represent a cross-sectional evaluation of 10 years of health literacy measurement research. We focused on self-report measures because these require fewer resources than those for which examiners (or clinicians) must be trained and provided. Consequently, we believe there will be more of a future in self-administered measures, as they are more scalable and can more easily be incorporated into online surveys, which are becoming more prevalent in health research a priori definition of health literacy, ‘condition- or context- specific’ indices are developed in the course of identifying a particular area in which the authors (presumably) believe it will be more beneficial to measure specific deficits that can be ameliorated. Existing research has not found clear patterns of effectiveness in interventions that are designed to mitigate the detrimental effects of low health literacy It has been noted that there is underlying conceptual disagreement about what health literacy is and that this may be contributing to an “unstable foundation” in research in this area Complimentary to ‘mitigating’ the negative effects is the idea of ‘improving’ health literacy, which is widespread throughout available policy documents instruments reported in included studies was assessed; since the primary aim of this study was to assess the quality of these measures, it was felt that being inclusive regarding studies would allow this study to more adequately represent the available literature. Given that one of the conclusions of this study is that the need for additional health literacy measures should be questioned, this selection bias serves only to further the point that there are already many high-quality available general and specific indices available. Limiting this study to self-administered measures has the advantage of providing those looking for one of these measures an accessible, clear reference to guide selection of an index of this type, however it excludes many other health literacy measures, which may be of lesser quality. Conclusions of a review encompassing all the measures may prove less optimistic about the quality of available indices for health literacy measurement, which is what the most comprehensive available review including these measures identified This study has identified that new measures are being developed and published frequently. One important limitation is therefore that there are other measures which have been developed and published since the search was carried out which have not been included. In addition, given the clinical heterogeneity and diversity of health literacy measurement, it is possible that relevant studies available at the time the search was carried out have not been retrieved. In particular, since only one reference was included for each measure, it is probable that many of these measures were further validated in subsequent studies. Although the risk of bias was not determined for each individual study, the quality of Researchers and clinicians alike need to consider practicalities of administration, including how long a measure takes to complete, whether it is suitable for self-completion (either online or with paper and pencil), and in what other circumstances and populations it has been used. Our review draws attention to the fact that several health literacy assessments require a great deal of time, potentially exceeding the average length of a primary care consultation. In a large European study, for example, Deveugele et al Almost three quarters of available health literacy indices included in this review have been developed and validated in populations in the United States. Indices from the United States may not be fully transferable to another health system, although it should be noted that this review only included English language papers. Again, in contrast to proposed research agendas for health literacy Many successful interventions intended to mitigate the effects of low health literacy on outcomes have resulted in mixed effects Asking the right questions is critical to effective research, and doing so is necessary to eliminate “research waste” Checklist S1PRISMA Checklist.(DOC)Click here for additional data file.Appendix S1Search strategy for MEDLINE.(DOCX)Click here for additional data file.Appendix S2Characteristics of included indices.(DOCX)Click here for additional data file.


Cancer diagnosis in both dogs and humans is complicated by the lack of a non-invasive diagnostic test. To meet this clinical need, we apply the recently developed immunosignature assay to spontaneous canine lymphoma as clinical proof-of-concept. Here we evaluate the immunosignature as a diagnostic for spontaneous canine lymphoma at both at initial diagnosis and evaluating the disease free interval following treatment.Sera from dogs with confirmed lymphoma  and clinically normal dogs (n = 39) were analyzed. Serum antibody responses were characterized by analyzing the binding pattern, or immunosignature, of serum antibodies on a non-natural sequence peptide microarray. Peptides were selected and tested for the ability to distinguish healthy dogs from those with lymphoma and to distinguish lymphoma subtypes based on immunophenotype. The immunosignature of dogs with lymphoma were evaluated for individual signatures. Changes in the immunosignatures were evaluated following treatment and eventual relapse.Despite being a clonal disease, both an individual immunosignature and a generalized lymphoma immunosignature were observed in each dog. The general lymphoma immunosignature identified in the initial set of dogs (n = 32) was able to predict disease status in an independent set of dogs . A separate immunosignature was able to distinguish the lymphoma based on immunophenotype . The individual immunosignature was capable of confirming remission three months following diagnosis. Immunosignature at diagnosis was able to predict which dogs with B cell lymphoma would relapse in less than 120 days .We conclude that the immunosignature can serve as a multilevel diagnostic for canine, and potentially human, lymphoma.The online version of this article (doi:10.1186/1471-2407-14-657) contains supplementary material, which is available to authorized users. Clinical diagnosis of cancer is a complex process usually initiated by presentation of indicative symptoms. Suspected conditions are identified as possible differential diagnosis and a battery of blood tests, urinalysis, imaging tests, and biopsy are conducted before final diagnosis is made. Biomarkers have been identified for some cancers, but have limited use as a primary screening tool. A single blood test capable of diagnosing cancer with high sensitivity and specificity would enhance patient care by streamlining the diagnostic process. Non-Hodgkin lymphoma (NHL) is a spontaneously occurring neoplasm of particular interest. NHL newly affects approximately 70,000 people annually in the United States and has Lymphoma is one of the most commonly encountered canine neoplasms, generally affecting middle-aged to older dogs. Breeds reported to be at increased risk include boxers, bull mastiffs, Bassett hounds, Saint Bernards, Scottish terriers, Airedales, golden retrievers and English bulldogs. Typical6 initiating cancer cells release 1000 molecules each of a biomarker into two liters of blood at steady state, the concentration of this biomarker would only be 1.3 × 10-14 M[11 amplification of signal in only a week[A serological test would facilitate routine monitoring during an annual wellness examination, enable faster diagnosis when LSA is suspected and allow monitoring following treatment. Design of such a test is dependent on the identification of an appropriate biomarker. Ideally, this test would be applicable to early disease, but to do so it must overcome the "blood dilution" problem: that is, if 10 10-14 M, placing 10-14 M. Antibod 10-14 M, 14. Ariy a week. Furthery a week, 17.We have developed a technology termed immunosignatures which displays the circulating antibody repertoire upon an addressable, machine readable random peptide microarray (reviewed in). The raIn this study we assess the ability of the immunosignature to characterize the humoral response to canine LSA and investigate its clinical utility in diagnosing different subtypes of disease. Pretreatment serum samples from patients presenting with T cell and B cell LSA (LSA-T and LSA-B) are compared to healthy dogs. Serial serum samples from patients that experienced remission following chemotherapy and ultimately relapsed were investigated. Immunosignatures informative for each subtype of disease and their diagnostic efficacy are reported.The diagnosis and treatment of many cancers, including canine and human LSA, is complicated by the lack of a non-invasive serological test. Having demonstrated that the immunosignature is capable of simultaneously classifying human cancers including multiple subtypes of brain cancer, we hypoSerum samples were obtained from clinically normal client-owned dogs or dogs with histologically or cytologically confirmed LSA and stored at -80°C from the time of presentation, prior to any specific therapy, and were collected with owner consent and approval of the CSU Institutional Animal Care and Use Committee (Protocol #10-2007A). Samples were collected during routine visits under nominal clinical conditions. In a subset of patients, sera were collected serially from dogs with LSA at each subsequent recheck visit, up to and including the time of relapse.http://www.peptidearraycore.com). Two print runs having a quality control technical cross batch correlation of 0.67 were used for this study.The CIM10Kv2 random peptide microarrays used for immunosignatures have been described previously, 19. ThePatient serum was used to probe the CIM10Kv2 immunosignaturing microarray as described previously using a Tecan HS4800, 21. Pri-6 significance level , Standard deviation of 50%, and a 1.5 fold change (delta) between groups indicated a minimum of 11 samples were needed for each comparison made. All comparisons were adequately powered.Raw array images were aligned using GenePix (Molecular Devices) to produce a tab deliminated results file. Physical artifacts were removed by flagging the features as bad. Results files were evaluated in GeneSpring (Agilent) or Bioconductor R (3.0.1). For all analysis, the arrays run with the KPL and the Jackson Immuno Research conjugates were treated separately. The ComBat algorithm was used to minimize assay batch effects on per chip median normalized scores. BackgroInitially, the immunosignature was evaluated with a small set of dogs to determine the ability to distinguish LSA (either B or T cell) from healthy. Patient n = 21) and healthy dog (n = 11) sera were randomized and applied to the CIM10Kv2 array. Per chip median normalized values were ComBat normalized to remove batch effects and then compared between LSA and healthy dogs. A Student’s T-test selected 340 peptides having an FDR corrected p value less than 0.05 and a minimum 1.5 fold difference in intensity between classes. Reactivity is shown in the heatmap in Figure  and healTo test the predictive ability of the peptides identified above, additional LSA-B patients (n = 20) and healthy donors (n =22) were obtained. Serum from all LSA-B patients (n = 38) and healthy donors (n = 39) were randomized and used to probe the CIM10Kv2. Technical requirements necessitated that serum from all dogs be run on a second print run of the CIM10Kv2 and detected using a different secondary antibody due to product discontinuation by the original supplier. The 340 peptides selected above to separate LSA and healthy clearly separated the expanded test set of dogs, even though the print run and anti-IgG secondary antibody were different  and test 15%) sets. The training set was used to select peptides having a p value < 0.05 with FDR and a minimum fold change between classes of 10.0 fold. The peptides were then used to train an SVM and predict class membership of the test set. Over 10,000 randomizations into training and test sets, the median performance on the test set accuracy was 92 +/- 9.6%, sensitivity was 100 +/- 12.2% and specificity was 83 +/- 14.3%. The lower specificity was due to 3 healthy dogs that were consistently miscalled when included in the test set . This raises the possibility of an immunosignature for each LSA patient in addition to the general class immunosignature. Such an immunosignature could be either from the antibody species produced by the B cell or the immune response to the surface markers or other cancer related antigens of the LSA cell. In the case of LSA-T there could be a unique antibody response to the T cell receptor of the LSA clone. Pattern matching analysis was done to identify peptides uniquely recognized by each dog.To reduce the influence of recent vaccines or infections, the peptides with the least variability in healthy donors (bottom quartile ranked on CV) were analyzed. The pattern used was the per peptide median for all dogs except the LSA patient being queried for which the value was set at 5 fold above the per peptide median. Peptides matching the profile with a Pearson correlation greater than 0.90 were defined as unique to that individual. A heatmap of the unique peptides is presented in Figure Lymphoma is a clonal disease arising from the uncontrolled proliferation of a single lymphocyte. We haveTo determine if the immunosignature could be used to monitor patients for early signs of relapse, we obtained sera from 12 dogs with LSA at the time of diagnosis, 3 months following treatment at which point they were clinically in remission, and at the time of relapse. The sera was run on the CIM10Kv2 and evaluated for changes both personal and subtype immunosignatures. Comparison of the individual immunosignatures between diagnosis and remission indicated that a median of 72 percent of the immunosignature decreased following initiation of treatment  decreased at remission and 13 +/- 7 peptides increased from remission to relapse. Of the 173 peptides increased in LSA-T, a median of 35 +/- 20 peptides (20%) decreased upon remission and a median of 21.4 +/- 6 peptides (12%) increased between remission and relapse. Antibodies binding the subtype specific immunosignature were likely raised by a normal B cells against the LSA cell as part of the anti-LSA immune response. Clinical therapy either excises or chemically kills the LSA cell, removing the antigen that stimulated the normal B cell. Persistence of the immunosignature after removal of the LSA cell suggests that the normal B cell had differentiated to a long lived plasma cell and was unaffected by remission. Taken together, this indicates that the immunosignature can verify remission through the personalized signature, but other means are needed to detect recurrence.-4) than those probed with healthy donors. In prior studies we noted overall feature intensity increases in infection and as the concentration of antibodies applied to the array increases[Retrospective surveys of LSA-T and LSA-B identified median DFIs of 2.5 months for LSA-T and 6.74 months for LSA-B following multi-agent chemotherapy. In thisncreases. This suncreases, 27 and Since changes in the individual or class immunosignatures following treatment were not indicative of relapse, we sought to determine if the immunosignature at diagnosis could predict time to relapse. The LSA-B patients were divided into those that relapsed in under 120 days (n = 10) and those that had a delayed relapse of over 120 days (n = 23). A Student’s T-test identified 35 peptides with a p value <0.05 with FDR between classes. A heatmap of the selected peptides is presented in Figure It has been well established that most cancers are capable of generating detectable cellular and humoral immune responses, although they are insufficient to control the disease. This observation has led to the exploration of immune-based cancer therapies in humans and dogs, including in canine LSA–31. In tLymphoma is a clonal disease arising from an abnormally proliferating B or T cell. Despite the clonal origin of the disease, a LSA specific immunosignature was identifiable. In humans, certain variable region genes are prevalent in LSA–34 and sWe observed that the personalized immunosignature declined as the dog entered remission along with the common LSA-B signature fits with what is known of this cancer. The causal B cell clone that forms the LSA is present in abundance at diagnosis and is excised or chemically reduced to barely detectible levels at remission, yet antibodies against the LSA persist. Serum IgG in dogs has a half-life of 8 to 12 days, indicatConversion of a microarray-based assay to a diagnostic is reliant on the robustness of the platform to differences in technician, print batch and detection system. In our training and test set, we evaluated the sum effect of these conditions. The training set was initially run as one batch then repeated in the second batch with the test set. When the SVM was trained using either training set batch, the test set was predicted with >94% accuracy. This demonstrates that as a practical diagnostic assay, a single patient could be normalized to a co-run standard and the immunosignature compared to a database, thereby alleviating the impractical possibility of having to run a 60 sample training set with each assay. Further enabling the transition to a clinical diagnostic is the ability of antibodies eluted from dried blood spots to perform in the immunosignature assay.There are multiple other techniques that have been established for the determination of immunophenotype in canine LSA. These include immunohistochemistry, immunocytochemistry, flow cytometry, and PCR for antigen receptor rearrangement (PARR). While aThis study illustrates the immunosignature as an improvement over current veterinary serodiagnostics for LSA. As opposed to tests which either non-specifically indicate a cancer by measuring thymidine kinase activity or rely This paper demonstrates that the serum antibody repertoire produces a unique binding pattern or immunosignature for lymphoma on a random peptide array. The immunosignature is descriptive for B or T cell lymphoma and estimating duration of remission following treatment. Thus the immunosignature has the capability to provide multiple levels of prognostic information to the patient and clinician.The immunosignature of canine lymphoma: characterization and diagnostic application supplemental material.(DOCX 243 KB)Additional file 1:


Objectives: Health Technology Assessment (HTA) needs to address thechallenges posed by high cost, effective technologies, expedited regulatory approaches,and the opportunities provided by collaborative real-world evaluation of technologies. TheHealth Technology Assessment International (HTAi) Policy Forum met to consider theseissues and the implications for evidence production to inform HTA. This paper shares theirdiscussion to stimulate further debate.Methods: A background paper, presentations, group discussions, andstakeholder role play at the 2015 HTAi Policy Forum meeting informed this paper.Results: HTA has an important role to play in helping improve evidenceproduction and ensuring that the health service is ready to adopt effective technologies.It needs to move from simply informing health system decisions to also working actively toalign stakeholder expectations about realistic evidence requirements. Processes to supportdialogue over the health technology life cycle need to be developed that are mindful oflimited resources, operate across jurisdictions and learn from past processes.Collaborations between health technology developers and health systems in differentcountries should be encouraged to develop evidence that will inform decision making. Newanalytical techniques emerging for real-world data should be harnessed to support modelingfor HTA.Conclusions: A paradigm shift  issuggested where HTA adopts a more central, proactive role to support alignment within andamongst stakeholders over the whole life cycle of the technology. This could help ensurethat evidence production is better aligned with patient and health system needs and so ismore effective and efficient. The Health Technology Assessment International (HTAi) Policy Forum provides an opportunityfor senior people from public and private sector organizations using health technologyassessment (HTA) to support decisions or recommendations about product development andcoverage to meet for strategic discussions about the present state of HTA, its development andimplications for healthcare systems, industry, patients, and other stakeholders . Since 2trends in technology development, HTA and healthsystems.” Given the breadth of the topics, a background paper The intention was to create models for evidence production and “stress test” models fordrugs and devices using a “Fred Friendly” discussiThe meeting was conducted under the Chatham House Rule, whereby participants are free toshare information obtained at the meeting, but they may not reveal the identity oraffiliation of the person providing the information. This study summarizes the authors’views of the key thoughts and suggestions emerging from the discussion at the meeting andthe wider issues raised. While informed and strengthened by attendees’ comments on drafts,it is not a consensus statement from the meeting, nor can it be taken to represent the viewsof any of those attending the meeting or the organizations for which they work.The background paper outlined the current challenges faced by health systems and healthtechnology developers and highlighted a range of interesting initiatives to supportcollaborative or real-world evaluation of promising technologies involving healthcareproviders, payers, health technology developers, regulators, patient organizations,academics, and HTA organizations . MeetingThe opening presentations and discussions in the meeting augmented the themes identifiedin the background paper, highlighting that we appear to be at a disruptive time in healthservice delivery, with several pressures but a range of exciting opportunities.There are increasing financial pressures on all stakeholders and the desire to providepatients with earlier access to effective therapies where there is high unmet need.Therefore, issues of available evidence, uncertainty, and perceived value need to bebalanced.Waste in research needs to be reduced , andtheThere are exciting new developments in basic science that could lead to targeted, highlyeffective and curative treatments. Health systems are improving their electronic recordsand recording health outcomes, which can be analyzed using structured, sophisticatedanalyses in real-time . There aThere is a need and an opportunity to harness these developments and improve theeffectiveness and efficiency of evidence production for new health technologies to inputto HTA and inform decision making. However, new approaches need to take account of thelimited resources of all stakeholders and risks need to be shared to produce evidence thatis sufficient for different decision makers. This requires elimination of unnecessaryduplication and agreement on what evidence is needed, and when, recognizing thatcompromises may be required on all sides. To achieve this, there must be alignment ofstakeholders, not just at the point of a decision about reimbursement, but earlier whendecisions are made about which technologies to develop and what evidence to produce.Clinicians, managers, patients, and technology developers need to be involved to ensurethat the process to a coverage decision is not only efficient but that it is alsoeffective. To be effective, health services need to be organized to enable rapid andappropriate introduction of effective technologies and disinvestment of ineffectivetechnologies.This suggests an additional responsibility for HTA. The HTA function needs to act as afacilitator to help align objectives within and between stakeholder groups relating to thedevelopment and deployment of technologies. Some described this as HTA being a “broker,”bringing stakeholders together to create common understanding of issues and developsolutions. This would involve helping technology developers understand clinical andpatient needs, evidence generation requirements, and limitations and helping healthsystems understand the potential and implications of new technologies and possiblechallenges of implementation. Others questioned whether this was a role for the HTAfunction alone, pointing to the importance of regulators, clinicians, managers, andpatient organizations.In practice, how this facilitator or broker function could and should be organized islikely to be dependent on the organization of health systems, available resources and arange of other factors, and is an important topic for debate. So within this context,break-out groups at the meeting discussed how health technology evidence production couldbe organized over the life cycle of a technology to make it more effective and efficient,and to consider the role of HTA in this context.In the preclinical phase and phase I, it may be too early for dialogue amongstakeholders about specific evidence requirements. However, there could be helpfuldiscussions about unmet needs in clinical practice, products in the pipeline, place inthe clinical pathway, expected indications, potential size of market, anticipatedbenefits, regulatory approach, and forms of evidence likely to be required.•which patients should be treated•comparators •key outcomes (relevant for decision makers and patients) and method by whichthese can best be measured or estimated•expected benefits in terms of effectiveness (including duration of benefit) andassociated resource use•generalizability of evidence•organization of services to optimize delivery•anticipated budget impact•risk sharing and need for co-payments•management of uncertainty.Early in phase II, dialogue may consider the potential population to be treated.Later in phase II, predictive models of the value of the technology are needed thatseek to identify what evidence is needed, when. Specific issues around evidenceproduction in phase III that are important for decisions on adoption should bediscussed, including: After phase III, it will be important to report whether the evidence was producedaccording to plan. During regulatory approval and HTA and coverage decisions, dialogueis needed about how to manage uncertainty and fill evidence gaps in the light of theresults achieved in the confirmatory trials.In the breakout discussions about the evidence development pathways for drugs, it wasproposed that new approaches to clinical research and HTA are required, with more plannedinteraction between stakehholders before and during evidence production to alignexpectations and manage emerging risks. Any systems that are established for postmarketing and postadoption evidence collectionneed to make it easy for clinicians to contribute ; must have clear governance arrangements ; and ensure efficient and flexibleinteractions among the necessary stakeholders.Challenges that may arise include multiple views about the appropriate developmentpathway and evidence requirements among HTA agencies and regulators, so strategic choiceswill need to be made. Furthermore, drugs that are selected for adaptive or expiditedregulatory pathways ;10 may nDiscussion of evidence development for drugs with companion diagnostics noted thatcompanion diagnostics are linked to the drug development life cycle, aiming to reduceuncertainty by identifying patients that will benefit most from the drug. Key questionsarise about how companion diagnostics will fit into the clinical pathway, so in additionto the questions relevant for drugs, the following issues can be added:•technical feasibility •performance of the test•cutoff points for relevant subpopulations to determine treatment choices•organization of healthcare testing systems •ethical questions •regulatory requirements•responsibility for payment (given siloed budgets).Interestingly, it would seem that most of these issues are similar to the more generalissues raised for drugs in phase I and so could be addressed early in the life cycle.Given the low cost of diagnostics and the high cost of drugs, some members questionedwhether such a complex HTA system is needed for companion diagnostics. Others pointed tothe major impact that the proposed use of the diagnostic can have on feasibility, impactand value of the targeted therapy, and the consequent need to consider both the diagnosticand the drug in relation to one another in an HTA. A report by the UK Office of HealthEconomics suggests that a broad range of value elements and balanced analysis ofdiagnostic impacts is needed for companion diagnostics introduced at the launch of a newdrug .•development and use of many more medical devices than any other form of healthtechnology•rapid technology evolution and no obvious phase I, II, III delineations•optimization of use is more complex (depending on user and setting)•necessary infrastructure •regulatory approval that does not require clinical evidence for all forms ofdevices•health system use often determined at a hospital level and/or through aprocurement, business case route, which is not informed by HTA.Discussion emphasized that devices have a different context to drugs ;13,inclDespite these differences, the evidence conundrum is similar, with a need for moreefficient evidence production and assessment. This would ideally be achieved through ajoint advice process discussing regulatory and payer needs across jurisdictions, toidentify unmet needs, help design studies that are relevant for all key stakeholders, andidentify how outcomes from different sources can be linked.It was noted that some evidence requirements for devices can be identified early andinvolve purely scientific issues that can be documented in a protocol .Other topics require a different form of dialogue with stakeholders .•focus on the disruptive, innovative technologies with the aim of providing relevantevidence to health systems to optimize patient outcomes•involvement of relevant experts •a neutral convenor•payer and industry infrastructure to enable effective engagement.Different stakeholders will have differing views on what constitutes appropriate anduseful evidence. Hence, dialogue for evidence production of devices needs to have a clearprocess, with: In the plenary discussions, several key issues were identified of relevance to theoptimal role and functioning of HTA across all classes of technologies.HTA is undertaken to inform decisions, and it can and should inform a wider range ofdecisions than just those for reimbursement at the time of launch. It can also informearly decisions about whether to pursue development of a technology, and later decisionsin clinical practice and health service organization about how best to adopt a technologyand optimize its use. Hence, HTA should not be seen as an academic endeavor disconnectedfrom technology development and healthcare delivery. Rather, it should be seen asproviding a crucial interface between decision makers in health technology development,clinical practice, and health systems, bringing them, patients, and other key stakeholderstogether to agree on evidence requirements and then helping develop and evaluate thatevidence to support informed decisions by all parties . It shouThe provision of scientific advice on HTA requirements is seen as a key initiative tosupport effective and efficient evidence production ;16. It cMore recently, the term “early dialogue” has been used for discussions of evidence thatwill inform HTA and coverage decisons (in addition to regulatory processes) . This haThe workstreams in the Policy Forum meeting suggested an expansion of the role of thisdialogue from being “early”  to being a series ofdiscussions starting before many current dialogues and followed up at various points overthe life cycle of the technology. It also noted that dialogue beyond the scientificaspects of clinical trial design can be helpful. Discussions about clinical pathways,unmet needs, and health system implementation may also be valuable. The authors suggestthat this type of interaction might be better described by a system such as “dialogue onevidence, impact, and use.”It is interesting to consider how HTA agencies could build capacity and develop efficientprocesses in collaboration with one another, with regulators and other key stakeholders toprovide health technology developers with the broader range of advice at key points in thelife cycle of drugs and devices proposed in this study. Such developments would be greatlyfacilitated if the outcomes and learnings from existing advice processes could be morewidely shared and discussed, and this is starting to happen . FurtherHealth systems are continuing to implement better administrative databases  and deveThere seems to be a potential for better collaboration among health systems to work withtechnology developers to undertake international studies for evaluation of potentiallyinnovative technologies to produce good quality pragmatic data efficiently that can informHTA .The Fred Friendly stakeholder role play confirmed the need for dialogue about evidenceproduction for HTA, but showed that the form of evidence required will be intenselynegotiated and dependent on several factors. The value of nontraditional data sources andstudy designs to augment randomized controlled trials and the value of patient involvementthroughout the evidence production process are apparent. However, willingness to acceptearly evidence and the associated risks in expedited approvals will vary by health systemand could be dependent on their control over data collection, their capability toencourage and restrict health technology use, and payment systems.The discussion at the meeting stressed the need to revisit how value is judged anddecisions on adoption are made. This issue has come into focus recently for very high costtechnologies addressing high unmet needs and for high cost technologies addressing morecommon conditions, such as hepatitis C .There wHTA must be independent and objective, but it needs to develop more agile and adaptiveprocesses that help to broker alignment among technology developers and health systems. The goal is to create a commonunderstanding of health needs, the potential of new technologies to address needs andimprove outcomes in the real world, and the need for health systems to adapt to realizethose benefits. However, it needs to do this without increasing complexity, keeping itsimple and comprehensible to the public.This suggests that HTA needs to innovate and be prepared to play a more active role toinfluence evidence production and help facilitate dialogue among stakeholders to optimizetechnology development and use. Participants in the HTAi Policy Forum meeting referred tothis new approach as HTA 2.0. But does this mean a new paradigm for the HTA function, or forthe whole system?The discussion at the HTAi Policy Forum clearly identified the need for fundamentalre-thinking of a range of issues in addition to how evidence requirements can be agreed andaddressed. These include the need to review concepts such as value and affordability, theneed to consider costs and benefits more coherently across health and social care systems,and the need to manage services and systems so as to be more adaptable and responsive topromising new treatments. This requires engagement with payers, providers, patients,politicians, and the public to debate priorities, trade-offs, value, and acceptance of risk.It will need sophisticated architecture to deliver this and a champion at the highest policylevel.This could be described as a set of related paradigm shifts: HTA 2.0, Health SystemOrganization and Management 2.0, Innovation System 2.0, Public Policy 2.0. Or it could be asingle paradigm shift that embraces all these areas in a coherent manner: “Health InnovationSystem 2.0.” If it is the latter, how can all the key stakeholders be involved in discussionand development of it, and who could and should take the lead? These are some of the keyissues and questions that the HTAi Policy Forum hopes its discussion will stimulate othersto consider and debate, and to which it expects to return to at its 2016 meeting andelsewhere.As a first step in this wider discussion, the issues raised in this paper were discussed ata panel session at the 2015 HTAi Annual Scientific Meeting in Oslo and publicly reported.•HTA needs to help facilitate dialogue among stakeholders over the life cycle of thetechnology to develop a shared understanding of evidence requirements to demonstratevalue and enable rapid decisions about use of technologies.•Technology developers can improve dialogue by explaining the scientific rationaleunderpinning the creation of a technology, providing a clear overview of informationavailable, and posing clear questions to HTA and coverage bodies about future evidencerequirements and approaches to value determination.•Those involved in the wider health system such as patients, clinicians, and managersshould be encouraged to contribute to this dialogue to help clarify unmet needs,quantify risks, determine value, and ensure rapid adoption of effectivetechnologies.•All stakeholders need to engage in a discussion about the development of this newcollaborative approach to health innovation.For supplementary material accompanying this paper visit http://dx.doi.org/10.1017/S026646231500035.click here to view supplementary materialThe authors were funded by HTAi to organize and report on this meeting. In addition, KarenFacey and Chris Henshall have undertaken a range of other paid work relating to HTA withlife sciences companies, consultancies, not-for-profit organizations and governments.


A41L, B15R, C6L, or C12L open reading frames (ORFs) enhanced cellular immune responses in recombinant MVA (rMVA) by up to 2-fold. However, previously, we showed that deletion of the C12L, A44L, A46R, B7R, or B15R ORFs from rMVA, using MVA-BAC recombineering technology, did not enhance rMVA immunogenicity at either peak or memory cellular immune responses. Here, we extend our previous study to examine the effect of deleting clusters of genes on rMVA cellular immunogenicity. Two clusters of fifteen genes were deleted in one rMVA mutant that encodes either the 85A antigen of Mycobacterium tuberculosis or an immunodominant H2-Kd-restricted murine malaria epitope (pb9). The deletion mutants were tested in prime only or prime and boost vaccination regimens. The responses showed no improved peak or memory CD8+ T cell frequencies. Our results suggest that the reported small increases in MVA deletion mutants could not be replicated with different antigens, or epitopes. Therefore, the gene deletion strategy may not be taken as a generic approach for improving the immunogenicity of MVA-based vaccines, and should be carefully assessed for every individual recombinant antigen.Modified vaccinia virus Ankara (MVA) is a highly attenuated strain of vaccinia virus, which has been used as a recombinant vaccine vector in many vaccine development programmes. The loss of many immunosuppressive and host-range genes resulted in a safe and immunogenic vaccine vector. However it still retains some immunomodulatory genes that may reduce MVA immunogenicity. Earlier reports demonstrated that the deletion of the  B15R, encoding the IL-1β binding protein [A41L, encoding a CC-chemokine binding protein [Modified Vaccinia virus Ankara (MVA) is a highly attenuated strain of vaccinia virus, the smallpox vaccine. This attenuation was achieved by more than five hundred passages in chick embryo fibroblast (CEF) cells . The res protein , or A41L protein .A41L , B15R, or both [+ T cell responses upon stimulation with VACV-infected cells or MVA-specific peptides. The immune responses to the recombinant antigens were also increased to a similar extent. In the double deletion ΔA41L\B15R [+ T cell response to one of the pools (GPN-pool) in the rMVA mutant vaccinated group. This mutant also induced more CD8+ T cells than CD4+ T cells in the Env-pool. In another study, the deletion of C6L gene from MVA-B (ΔC6-MVA-B) increased memory CD4+ and CD8+ T cell responses, and although this mutant did not shift the immune responses towards particular peptide pools, the increased CD4+ or CD8+ T cells were only observed in one peptide pool (the GPN-pool), but not in the Env- or Gag-pool [C6L and K7R in the same MVA-B vaccine (ΔC6/K7-MVA-B) showed that the ΔC6-MVA-B, included as a control, had no impact on the peak immune responses [+ T cells, but not CD4+ T cells; and upon comparison with the ΔC6-MVA-B, the increase seemed to be mainly an effect of C6 absence and not due to the K7R deletion [+ T cells specific to the MVA vector or to encoded antigens were also reported upon the single deletion of F1L , N2L , or C12L (MVA008L) [It has been reported that deletion of some of these genes improved innate, adaptive memory cellular or humoral immune responses to the vector (MVA) or to the encoded recombinant antigens, and enhanced the protective capacity of MVA following a challenge with VACV WR. The deletion of the ORF  or both , showed 41L\B15R , the MVAGag-pool . AlthougMVA008L)  in rMVA A41L, B15R, or the double deletion of both had no impact on the antibody induction at the memory phase, the deletion of C6L or N2L increased the memory antibody-mediated responses by around 2-fold in the total anti-recombinant antigen antibody titres. Of note, the C6L deletion enhanced the anti-gp120 antibody titres [C6L deletion mutant, in a later study, along with the double deletion C6L/K7R mutant showed no significant difference in the anti-gp120 antibody titres by those two mutants compared to the control (MVA-B vaccine) [1L and C12L studies. Finally, the deletion of A35R showed small non-significant increase in the cellular immunogenicity but increased the anti-VACV antibody titres by 2-fold with isotype switching to more IgG and its subclasses IgG1 and IgG2a [ex vivo re-stimulation (e.g. VACV-infected cells vs. E3 or F2(G) peptides), and the time for harvesting the spleens to determine memory responses .In terms of improving humoral immunogenicity in these studies, while the deletion of y titres , but thend IgG2a . OverallC12L (MVA008L), B15R (MVA184R), A41L (MVA153L), and A46R (MVA159R) from MVA encoding HIV-1 clade C consensus Gag and Env immunogen, given twice with a 9 week interval, resulted in increased CD4+ and CD8+ T cells either after the prime (4-fold) or after the boost dose (5-fold); and around 25-fold increase in the anti-gp120 antibody titres [udg) gene, in addition to those four genes, resulted in a five-gene deletion rMVA mutant that did not further enhance the improved immunogenicity of the four-gene deletion mutant. The deletion of the fifth gene, udg, was expected to improve the immune responses, as a previous report by the same group showed that the deletion of udg gene alone in rMVA inhibited the late MVA gene expression, reduced the antigen complexity of MVA, and improved the elicited CD4+ and CD8+ T cells (2-fold) in a similar regimen, but with Gag gene from HIV clade B, which is a different recombinant antigen [Nevertheless, a study in macaques reported that deleting four genes;  antigen .C12L, A44L, A46R, B7R, or B15R, from MVA did not enhance the cellular immunogenicity of the TIP model antigen  using seven immunodominant MVA epitopes in two strains of mice at the peak cellular responses. Of the memory responses (day 56 post-MVA injection), only the ΔB15R-MVA showed a small (1.5-fold) but significant increase in the CD8+ T cell frequencies [M. tuberculosis [d-restricted murine malaria epitope (pb9) from the Plasmodium berghei circumsporozoite protein [Previously, we reported that the single deletion of quencies . Here, wrculosis  or the Trculosis  that enc protein . The MVA protein .+ T cell frequencies. This unchanged immunogenicity was observed using either the synthetic model TIP antigen or the naturally occurring 85A antigen. The responses at 28, 56, or 84 (long-term) days post-MVA injection showed no improved memory CD8+ T cell frequencies, except to one peptide (E3) at day 28. However, at day 84, or at day 56 in the prime-boost vaccination, this increase in E3-specific responses was very marginal and not statistically significant. This support the earlier studies showing that the increase could be observed with only few peptides or peptide pools.None of the derived rMVA mutants showed improved immunogenicity either to the MVA or to encoded TIP antigens. The large MVA deletion mutant (lacking fifteen genes) was tested in prime only or in prime-boost regimens and showed no improved CD8Our results suggest that the reported increases in MVA deletion mutants, which were small increases in many cases (around 2-fold), could not be replicated with different antigens and that the approach of gene deletion to improve MVA-vectored vaccines should be carefully assessed for every recombinant antigen rather than taken as a generic approach for improving MVA immunogenicity.+ T cells, specific to MVA-peptides (E3 and F2(G)) or to the TIP-specific peptide (pb9) were then measured. There were no significant differences in response to the MVA-E3, MVA-F2(G), or pb9 peptide between any of the mutant groups and the control  or with different recombinant MVA mutants . The presence of either 85A or TIP antigens were screened with PCR; and the absence of the MVA wild type was also confirmed by PCR.As there was no improved immunogenicity with various deletion mutants, we constructed a large MVA mutant, lacking fifteen genes (Δ15-MVA) described in 6 pfu via i.m route, and the immune responses followed for three months. First, all mice were bled at day 7 post-immunization and the PBMCs were screened in blood-ELISpot using the pb9 peptide, or the vector-specific E3 and F2(G) peptides. The CD8+ T cell immune responses were similar between the MVA-TIP and the mutant (Δ15-MVA-TIP). At day 28 post-immunization, five mice from each group were sacrificed, spleen harvested, and ICS was performed using the same three peptides. Only E3-specific CD8+ T cell responses were enhanced (in two independent experiments) in the mutant group while there were no differences in the recombinant epitope (pb9) or in the other vector-specific F2(G) peptide. The remaining of the mice were sacrificed at day 84 post-immunization and showed no improved CD8+ T cell frequencies, even the increased responses with the E3 peptide at day 28 was not observed at day 84, suggesting it was transient enhancement  10ancement .7 pfu, via i.p. route, with either the Δ15-MVA-TIP mutant or the control (MVA-TIP). Five mice were sacrificed at day 7 post-boost to determine the peak immune responses with the remaining mice sacrificed at day 56 post-boost to determine the memory responses. There were no improved CD8+ T cell responses either at the peak or at the memory responses to pb9, E3, or F2(G) peptide stimulations  and has been shown to induce CD4+ and CD8+ T cell responses and when used to boost BCG-vaccinated adult or infant humans [wt), non-mutant MVA-BAC-85A, and the MVA85A  was performed to determine the cellular immune responses using two 85A peptide pools, specific for CD4+ or CD8+ T cells, or E3 and F2(G) MVA-peptides specific for CD8+ T cells. The ICS did not show any difference in the frequencies of IFN-γ secreting CD4+ or CD8+ T cells between the mutant and any of the control viruses. The comparison of MVAwt and the MVA-BAC-85A showed similar vector-specific immune responses using CD8+ T cell-specific MVA peptides (E3 and F2(G) peptides). Moreover, the conventional MVA85A induced similar anti-vector and anti-rAg85A responses as the MVA-BAC-85A. Taking together, this supports our earlier observation [+ or CD8+ T cells by ELISpot and observed a similar result to the ICS  epitopes. The use of TIP model antigen, which harbours the immunodominant pb9 epitope, enabled us to follow the immune responses for three months, but did not show any improvement at the long-term memory CD8+ T cell responses (for three months). Many previous studies showed that the single deletion of C6L [N2L [C12 [A41L/B15R and C6L/K7R from MVA vectors enhanced the cellular immunogenicity in DNA-prime MVA-boost vaccination regimens. This regimen is also relevant to MVA clinical testing applications; MVA is used as a boosting agent in many vaccine clinical trials [A41L and B15R are deleted in the Δ15-MVA-TIP mutant, and therefore this mutant would be expected to elicit stronger cellular immune responses, especially in this regimen as reported before. However, our result did not show any improved immunogenicity with the Δ15-MVA-TIP in prime-only or prime-boost vaccinations. Garber et al (2012) reported that the deletion of four immunomodulatory genes enhanced MVA cellular immunogenicity in macaques, but the deletion of a fifth gene did not extend this improved immunogenicity [A41L or B15R, was also noted when the A41L or B15R were singly deleted, as we previously published [A41L or B15R, or both of them was reported to enhance MVA cellular immune responses, but this enhancement was 2-fold at best [A41L or B15R, or both, the inability to reproduce this improved immunogenicity could suggest that improving MVA immunogenic profiles could be very difficult to achieve through gene deletions, especially as MVA does not replicate in vivo, and its gene expression declines within 12 hours, and vanishes less than 48 hours post-inoculation in vivo [Here we investigated the effect of deleting some immunomodulatory or unknown non-essential genes from MVA, in different combinations , on the n of C6L , N2L [17N2L [C12 ; or the l trials . Thus, wer et al 12 reportgenicity . It coulublished . Moreove at best –13. Alth in vivo , 28.+ T and CD8+ T cell immunogenicity. Thus, we replaced TIP with 85A antigen in the large MVA deletion mutant (deriving Δ15-MVA-85A), confirmed that no other changes had occurred in the genome by sequencing, then tested it in mice, measuring both CD4+ T and CD8+ T cell responses with peptide pools by ICS or two strong individual peptides (specific for CD4+ or CD8+ T cells) by ELISpot. Here, three control viruses were included; the MVA85A vaccine that is used for clinical testing against TB, the MVA-BAC-85A , and the MVA wild type (MVAwt), which is non-recombinant and non-mutant control. The results showed a continued lack of improved immunogenicity even with this natural immunogenic antigen. Although we were able to test CD4+ T cell responses, there was no significant difference in the magnitudes of 85A-specific CD4+ T cell between the mutant group and any of the control groups.The concern over the use of non-natural model antigen like TIP should not apply to a “real” immunogenic antigen. The TB 85A is one of the most advanced candidates in clinical trials and showed strong CD4E3L, which is difficult to grow in CEFs[+ and CD8+ T cell responses to the rAg-specific peptide pools by ICS or to the individual strong peptides measured by ELISpot as well as similar CD8+ T cell responses to the vector-specific E3 and F2(G) peptides. Moreover, this was also supported by the result of MVAwt compared to the MVA-BAC-85A that showed similar responses to the vector specific E3 and F2(G) peptides. This MVA-BAC system was also shown to have no altering affect on the immunogenicity of MVA [+ T cells was the main readout as it would be more relevant for the improvement required for clinical testing of MVA-based vaccines.It could be speculated that the MVA-BAC system or deriving MVA in BHk-21 cell lines had some effect on these mutants, which could prevent them from enhancing the immune responses. The BHK-21 cell line was reported to allow for the replication of some MVA mutant lacking w in CEFs. This waw in CEFs. The BHKw in CEFs. The PKRy of MVA . The MVAC12L and C6L, which were reported to enhance cellular immunogenicity when deleted from MVA [C12L [C6L (data not shown) failed to enhance the immune responses to TIP.The ΔC11-K3/B15-MVA lacks from MVA , 18, didVA [C12L  or C6L (+ T cell responses. The memory responses at 28 or 84 (long-term) days post-MVA immunization showed no improved memory CD8+ T cell frequencies, except to one peptide (E3) at day 28. At day 56, in prime-boost regimen, this increase was observed but it was very marginal and insignificant. Even replacing TIP with TB 85A “real” antigen (Δ15-MVA-85A) and testing CD8+ as well as CD4+ T cells magnitudes did not detect any differences compared to various control viruses. This study suggests that the previously reported immunogenicity increases in MVA deletion mutant, which were small increases in many cases (2-fold), may not be replicated with different antigens. The approach of gene deletion to improve MVA-vectored vaccines should be carefully assessed for every recombinant antigen, and epitope (peptide), and may not be taken as a generic approach for improving MVA immunogenicity.In conclusion, none of the derived rMVA mutants showed improved immunogenicity either to the MVA antigens or to the encoded TIP antigen. Focusing on the large deletion Δ15-MVA-TIP mutant, neither prime only nor prime-boost regimens showed any improved CD8All animal procedures were performed in accordance with the terms of the UK Animals (Scientific Procedures) Act (ASPA) for the project licenses 30/2414 or 30/2889 and were approved by the University of Oxford Animal Care and Ethical Review Committee. All mice were housed at least 7 days for settlement prior to any procedure in the University animal facility, Oxford, UK under Specific Pathogen Free (SPF) conditions.d restricted murine CD8+ T cell epitope SYIPSAEKI (pb9) from the Plasmodium berghei circumsporozoite protein [Mycobacterium tuberculosis 85A antigen (TB 85A), deriving the Δ15-MVA-85A. The 85A was also inserted into two other viruses to serve as controls; first, into a non-mutated MVA-BAC using MVA-BAC recombineering (virus named MVA-BAC-85A); second, into MVA wild type using the conventional recombinant vaccinia virus construction .GalK recombineering [GalK resistance gene under a prokaryotic promoter. This was amplified with Phusion (Finnzymes) as a targeting DNA for recombineering by using long oligonucleotide primers (Eurofins MWG Operon) to add 50bp homology arms (matching two regions within the TK ORF) to the 5’ and 3’ ends. Primers were designed to insert the TIP-GalK cassette into the TK locus. These targeting constructs were used for MVA-BAC recombineering as previously described [GalK from MVA-BAC that contains TIP-GalK, leaving only the TIP into the TK locus. Therefore, GalK was then amplified with long oligonucleotide primers with 50 bp homology arms, matching the flanks of the targeted ORFs, deleting and replacing them with GalK. The colonies on GalK-positive plates were re-streaked three times before the recombineered MVA-BAC constructs are then confirmed by identity and purity PCR. The presence of the TIP was also confirmed by PCR and sequencing.Construction and generation of MVA-BAC and generation of rMVA deletion mutants using ineering  has beenineering . To geneescribed . Next, MGalK bacterial marker gene was only removed from mutants that required a multiple deletions such as the Δ15-MVA-TIP, for the mutants with only one gene or one region of deleted genes, the GalK was not removed. BACs and derived viruses were checked for identity and purity by PCR and the sequences of the homology arms and transgenes were confirmed at both stages. BAC-derived rMVAs were plaque-picked three times to ensure purity, as a precautionary measure, since the bacterial colonies were re-streaked three times prior to rescue. The viruses were amplified in 1500cm2 of BHK cell monolayers, purified over sucrose cushions and titred in BHK-21 cells according to standard practice, and purity and identity were again verified by PCR. Since MVA-BAC has a GFP marker gene under control of the Fowlpox virus p4B promoter [The recombineered MVA-BACs were rescued to recombinant MVA in BHK cells using a fowlpox virus helper as previously described . The Galpromoter , all theDNA samples of the Δ15-MVA-85A from bacterial LB culture (for MVA-BAC DNA) or from BHK-21-infected cells  were isolated using EndoFree Plasmid Maxi Kit (Qiagen) or standard phenol-chloroform DNA extraction, respectively. Samples were sent to the High-Throughput Genomics Group at the Wellcome Trust Centre for Human Genetics, University of Oxford, for MVA genome sequencing.6 pfu of rMVA. For the heterologous prime boost regimen, mice were immunized with 100 μg of DNA, given i.m. (50μL per mouse) followed eight weeks later with 107 pfu, given intraperitoneal (i.p.) of rMVA. Procedures were performed in accordance with the UK Animals (Scientific Procedures) Act 1986 under granted project licenses 30/2414 or 30/2889. Splenocytes were harvested seven, or 56 days post-MVA immunization for analysis by IFN-γ ELISpot or intracellular cytokine staining (ICS) and flow cytometry as previously described [+ or CD8+ T cell-specific 85A peptide pools for ICS. Two 85A-specific individual peptides were used for ELISpot; CD8+ T cell-specific p11 peptide, EWYDQSGLSVVMPVGGQSSF, and CD4+ T cell-specific p15 peptide, TFLTSELPGWLQANRHVKPT.Female BALB/c mice aged 6 to 8 weeks  were immunized intramuscularly (i.m.) in the tibialis muscles (50μL per mouse) or intradermally (i.d.) with a total of 10escribed , 31, usiescribed , 1μg/mL GraphPad Prism (GraphPad software) was used for statistical analysis and to plot data.


Leishmania infantum, Dirofilaria immitis and Angiostrongylus vasorum in dogs from the Algarve region, in southern Portugal.Leishmaniosis, dirofilariosis and angiostrongylosis are parasitic diseases of established importance in dogs worldwide. The aim of the present report was to determine all together levels of infection with or exposure to L. infantum were detected by the direct agglutination test; and antigens of D. immitis and of A. vasorum by an enzyme-linked immunosorbent assay and an immunochromatography test, respectively. Antibodies to L. infantum were found in 18.2% (31/170) of dogs, while D. immitis antigen was detected in 9.4% (16/170). None of 120 dogs tested out of the 170 under investigation had evidence of infection with A. vasorum. D. immitis infection was more prevalent in dogs aged 84–204 months than in those younger than 12 months. Three dogs were positive to both L. infantum and D. immitis.Serum samples were obtained from a total of 170 apparently healthy dogs. Antibodies to Dogs living in southern Portugal are at risk of leishmaniosis and dirofilariosis. This scenario should make the veterinary community, local dog owners and also tourists from non-endemic countries coming on vacation with their pets aware of the need of effective prophylactic measures, in order to protect animals and public health. Leishmania infantum  and heartworm disease caused by the nematode Dirofilaria immitis  are vector-borne zoonoses with importance in veterinary medicine and widely present in the Mediterranean [Leishmaniosis, dirofilariosis and angiostrongylosis are parasitic diseases of established importance in dogs worldwide. Leishmaniosis caused by the protozoan erranean -3. Both erranean  and 3.6-erranean , respectCanine leishmaniosis is a systemic chronic condition whose clinical manifestations usually include lymphadenomegaly, cutaneous alterations, loss of body weight, ocular signs, epistaxis, onychogryphosis and lameness . Canine Angiostrongylus vasorum , is an emerging disease with an increasing number of cases diagnosed in the last few years, and with a high established relevance in several regions of Europe [A. vasorum antigens or specific antibodies have recently been developed [Canine angiostrongylosis, caused by the nematode f Europe . The disf Europe . Albeit eveloped ,10. In Eeveloped ,12. In ceveloped .L. infantum and D. immitis) and parasites acquired by ingestion of intermediate hosts (A. vasorum) in dogs from the Algarve region, the southernmost part of continental Portugal, in view of the scarcity of epidemiological studies targeting these parasites all together in canine populations.The present report aimed at determining the levels of infection with or exposure to vector-borne parasites  as complying with the Portuguese legislation for the protection of animals (Law no. 92/1995).Leishmania antigen as previously described [D. immitis antigen was detected in 170 animals by using a commercial enzyme-linked immunosorbent assay kit . Detection of A. vasorum antigens was performed in only 120 dogs out of the 170 under investigation by using a commercial immunochromatography test .The direct agglutination test (DAT) for titration of specific antibodies was performed in 170 dogs using a standard freeze-dried escribed . The DATescribed . D. immip value < 0.05 was considered as statistically significant. Analyses were performed with StatLib and SPSS® 21 software for Windows.The exact binomial test established confidence intervals (CI) with a 95% confidence level. The chi-square or Fisher’s exact tests were used to compare percentages of positivity among categories of the same independent variables and also the total prevalence of each agent . A p valL. infantum were found in 31 (18.2%) dogs, at the dilutions of 1:400 (n = 5), 1:800 (n = 2), 1:1,600 (n = 2), 1:3,200 (n = 5), 1:6,400 (n = 6), 1:12,800 (n = 6), 1:25,600 (n = 4) and 1:51,200 (n = 1)  [Leishmania infection in the Algarve [Antibodies to :400 n = , 1:800 ( Algarve .D. immitis antigen was detected in 16 (9.6%) animals , which may be explained by a cumulative exposure of older animals to the parasite through an increased contact with the vectors [p = 0.128), D. immitis antigens were not detected in any of the animals receiving macrocyclic lactones; anyhow, this finding suggests the importance of using prophylaxis in areas where dirofilariosis is endemic.Prevalence of  vectors ,21. AlthD. immitis and A. vasorum in dogs experimentally infected with the latter has previously been reported for some heartworm antigen test kits [A. vasorum antigens were not detected in any of the dogs tested . The non-detection of this parasite, in comparison with the survey (1.8%) carried out in other coastal areas of Portugal [A. vasorum, for which the actual national distribution is poorly acquainted.Cross-reactivity between est kits . HoweverPortugal , might bPortugal ), with sL. infantum and D. immitis. Only three dogs were positive to both L. infantum and D. immitis. Co-infections with these two vector-borne pathogens are relatively frequent in dogs living in geographic areas where the presence of competent vectors for the different pathogens overlap [In the present study, no statistical association was found between positivities to  overlap ,24. Alth overlap .The findings of this study reveal that dogs from southern Portugal are at risk of acquiring two of the three parasites assessed (i.e. the agents of leishmaniosis and dirofilariosis). They also reinforce the importance to alert the veterinary community, local dog owners as well as tourists from non-endemic countries coming on vacation with their pets to the need of prophylactic measures, such as insect repellents and macrocyclic lactones, in order to defend animals and public health.