The goal of this study was to define the construct and establish the validity of disciplinary literacy, which has recently gained attention from the implementation of the Common Core State Standards (National Governors Association Center for Best Practices & Council of Chief State School Officers in Common Core State Standards for English language arts & literacy in history/social studies, science, and technical subjects [PDF]. Authors, Washington, DC,  Increasingly, educators have concerns with adolescent under-performance in literacy  rather than solely in the language arts classroom , with a focus on apprenticeship and problematizing the linear view of disciplinary literacy. Finally, we examined ELA, science, history and social studies, and mathematics and synthesized extant research that supports different literacies for each of these subject areas. This literature review served as the theoretical foundation on which to functionally examine how participants operationalize domain-specific literacy practices.d and capital D in order to demonstrate how people use language differently across contexts.Middle and secondary literacy has been theorized from literacy perspectives rather than disciplinary perspectives. Moje  advocateSnow and Uccelli  synthesiIntegral to disciplinary literacy and academic success is mastery of academic language; therefore, a discipline has a “need for precision through its texts or its language”  to reading to learn  and provide support from the literature about how each has its own specialized language, expectations, and means of constructing knowledge. In the context of secondary schools, the subject area of ELA is comprised of subdisciplines, such as literature, creative writing, and linguistics; science is comprised of subdisciplines, such as biology, physics, and chemistry; social studies is comprised of subdisciplines, such as history, economics, and psychology; and mathematics is comprised of subdisciplines, such as calculus, logic, and statistics. For ELA and social studies, we narrowed the focus to the subdisciplines of literature and history. We were able to find multiple studies that focused on literature specifically and history specifically at the secondary level, but we were not able to find a body of work centered on subdisciplines in math or science. For science and mathematics, we kept our focus more general on the subject areas. Our rationale is that the disciplinary literacy research on science asserts that scientific knowledge across subdisciplines develops in the same way by accruing “evidence for and against potential explanations of science phenomena”  the situating of literacy in the context of disciplinary knowledge, (2) the emergence of a cognitive argument for disciplinary literacies, and (3) the rise of pedagogical scaffolds for supporting literacy teaching and learning. Concurrent with educational reform and the publication of , Gagnon  describe, Gagnon .Reading Like a Historian curriculum of document-based lesson plans is premised on research which suggests that disciplines—in this case, history—have particular modes of thinking that cannot be assimilated as general skills and thus, require teaching materials designed narrowly to produce thinking and literacy skills within the discipline  What are teachers’ self-reported disciplinary literacy practices in these four areas?; and (b) Do disciplinary literacies in fact differ within these four areas?In this section, the procedures and methods used to develop, explore, confirm, and test a scale of disciplinary literacy are outlined. First, item development, content validation, and gathering of participants are discussed, followed by the method used to remove careless responders. Next, the cleaned dataset demographics are outlined, followed by a description of the exploratory factor analysis, preliminary factor structure, and method used for item retention. Finally, the procedures that were used to confirm the factor structure and test differences among disciplines are described.The online survey was deployed through Qualtrics with the survey development process consisting of scale development, item creation, content validation, and choice of response format.Our first task was to define the construct of disciplinary literacy through synthesis of relevant literature and expert judgment Hinkin, . To guidn = 8) and two focus groups of high school teachers (n = 8). The cognitive interview findings were used to clarified procedures, item wording, and response choices. This process resulted in 24 finished items.The content experts rated each item within the 100-plus-item pool for content relevance and item quality. Content experts rated each item as essential, useful but not essential, or not necessary. A content validity ratio (CVR) was then calculated for each item, providing a structured method to assess content validity Lawshe, . Items wFrequency scale anchors ask the individual to recall and cognitively compute how often a specific primed behavior has been engaged in  was assessed through exploratory factor analysis (EFA) on Pearson correlations. Principal axis factoring with direct oblimin oblique rotation was utilized to account for the possible correlation of latent variables, and estimated with maximum likelihood conducted in the psych package in R  measure was utilized to test sampling adequacy of items to yield distinct and reliable factors . The KMO values indicate that the sample adequacy was good (KMO = 0.86.), even “meritorious” according to Hutcheson and Sofroniou . The KMOTo determine the most adequate number of factors to extract, initially a scree plot was used Cattell, . No deciFor item elimination, Comrey and Lee’s  criteriasource literacy, the second as analytic literacy, and the third factor as expressive literacy. These three terms were chosen because they represent the factor in a comprehensive way. Source literacy, which is most directly related to the literature introduced by Wineburg  = 3.57, p < 0.001, source literacy F = 5.87, p < 0.001, and expressive literacy F = 12.83, p < 0.001. Since the assumption of homogeneity of variance was not met, Welch’s adjusted F-test was used to assess mean differences. For effect size measures, adjusted omega squared (ω2) values were calculated based on the F-tests. For post-hoc comparisons of literacy by subject areas, standard box-plots were conducted as well as Dunnett’s T3 procedure .X2  = 295.36, p < 0.001, CFI = 0.92, SRMR = 0.07, RMSEA = 0.09). In efforts to improve fit, two items from analytical literacy and one from source literacy were removed due to low factor loadings (≤ 0.45) and weak alignment to theory  was conducted with the second half of the randomly split original 820 cases to test the factor structure of the exploratory results. The proposed three-factor model originally revealed a marginal fit to the data (X2 (39) = 190.48, p < 0.001) and the 13 remaining items all displayed strong factor loadings (≥ 0.54). Overall, the revised model fit well  = 104.88, p < 0.001, CFI = 0.98, SRMR = 0.06, RMSEA = 0.05) with acceptable CFI, SRMR, and RMSEA values. The retained model is presented in Fig. The first two items described instructional practices more than disciplinary literacy practices and were unlike the other items. Removing these three items resulted in a better fit  = 65.52, p < 0.001, est. ω2 = 0.24, as well as analytic literacy Welch’s F  = 72.22, p < 0.001, est. ω2 = 0.21, and expressive literacy Welch’s F  = 222.47, p < 0.001, est. ω2 = 0.43. The effect of subject areas on the three literacies all passed Cohen’s  = 65.52, p < 0.001, est. ω2 = 0.24), which is graphically represented in Fig. p < 0.001), science , and history and social studies . Teachers of history and social studies were the highest in source literacy (M = 3.48), followed closely by those who taught ELA (M = 3.30). Teachers of history and social studies scored significantly higher in source literacy than those who taught science  and those who taught mathematics . Teachers of science did not significantly differ from those who taught ELA  with respect to source literacy.Teachers differed significantly with regard to their practice of source literacy  = 72.22, p < 0.001, est. ω2 = 0.21). Figure M = 3.93), with significantly higher reported levels than teachers of ELA  and history and social studies . Teachers of ELA reported the lowest levels of analytic literacy, less than those in science , history and social studies , and mathematics . Teachers of science did not significantly differ in their use of analytic literacy from those who taught mathematics , but both were higher in their use than those who taught ELA.The practice of analytic literacy also differed significantly across subject areas  = 222.47, p < 0.001, est. ω2 = 0.43). Figure M = 4.02), higher than teachers of mathematics , history and social studies (T3 = 0.83 p < 0.001), and science . Teachers of mathematics engaged in the lowest levels of expressive literacy, with significantly lower levels of expressive literacy than teachers of ELA , science , and history and social studies . Lastly, teachers of history and social studies engaged in significantly more expressive literacy than teachers of science .Post-hoc comparisons of expressive literacy revealed significant differences between all four subject areas (The research attempted to (a) define disciplinary literacies within four subject areas ; (b) assess teachers’ self-reported practices with respect to disciplinary literacies in these four core areas; and (c) empirically demonstrate how disciplinary literacies differ within these four areas. In the following discussion, we summarize and interpret the findings and relate the findings to existing research on disciplinary literacy.disciplinary literacies: source literacy, analytic literacy, and expressive literacy. After confirming this three-factor structure via CFA, differences by literacy and discipline were evaluated. Findings supported individual-level differences in which teachers engaged in some amount of each of the three factors. When teachers were grouped by subject area taught, patterns emerged suggesting that these disciplinary-specific teachers were more likely to engage in certain literacies. Results indicated that literacy practices are not limited to the ELA classroom, but instead are distributed among teachers of academic subjects with source, analytic, and expressive literacies practiced by varying degrees.The primary interest of this study was to investigate the construct of disciplinary literacy as teacher practice, operationalized through the areas of ELA, science, history and social studies, and mathematics. The findings supported the notion that disciplinary literacy is, in fact, a series of at least three Source literacy includes the reading skills of sourcing , contextualization , and corroboration .The idea of literacy as a distributed practice that emerged in this research aligns with the idea of a shared responsibility for literacy instruction as encouraged in state and national standards for ELA . Likewise, our participants comprised sixth—twelfth grade teachers because that is when the Common Core State Standards specify literacy standards by discipline. Although disciplinary literacy practices occur at the lower grades, the inclusion of elementary school teachers was beyond the scope of this study.Third, while we did not use stratified random sampling, we argue that our population is an approximation of a stratified sample. Our sample was approximately half middle school teachers and half high school teachers. In addition, our sample included participants from all four regions used by the US Census (Northeast, South, Midwest, and West; see Fig. The results of the study have implications for policymakers, practitioners, and researchers. Our study provides empirical support for policy, practice, and research already underway in addition to providing insight for future applications of disciplinary literacy.Policymakers would be interested in these results especially in relationship to the Common Core State Standards for literacy in ELA, science and technical subjects, and history and social studies. Results support adoption of literacy standards in ELA, science, and history and social studies and imply that literacy standards in mathematics are warranted. In the future, existing assessments might be improved by making more appropriate and sensitive measures, accounting for the different literacies employed in the disciplines. We do not dismiss the importance of content knowledge, agreeing with Moje , who staFor practitioners, the results can help inform the current content area or disciplinary literacy debate (e.g., Collin, should we expect of elementary students, middle grades students, and high school students? We anticipate that the results can be used to generate and test effective tools that classroom teachers use to guide students through the different literacies practiced within the four targeted disciplines of this study.The results of this study lead to additional questions that need to be addressed in future research. Construct validity is an ongoing process (DeCuir-Gunby, Overall, the current research sheds new light on disciplinary literacy and establishes quantitative justification for future work. For those researchers who are invested in establishing extended theories and practices for disciplinary literacy, our research suggests a promising context for fuller discussions as well as a useful base for guiding subsequent studies. Instruction in disciplinary literacies holds promise for improving adolescent literacy and has the potential to engage students in a community of practice that shares a common language and way of thinking.


This systematic review gives an overview of guidelines and original publications as well as the evidence on which the currently proposed indication criteria are based. Until now such a state-of-the-science overview was lacking.Websites of orthopaedic and arthritis organizations (English/Dutch language) were independently searched by two authors for THA/TKA guidelines for OA. Furthermore, a systematic search strategy in several databases through August 2014 was performed. Quality of the guidelines was assessed with the AGREE II instrument, which consists of 6 domains . Also, the level of evidence of all included studies was assessed.We found 6 guidelines and 18 papers, out of 3065 references. The quality of the guidelines summed across 6 domains ranged from 0.46 to 4.78. In total, 12 THA, 10 TKA and 2 THA/TKA indication sets were found. Four studies stated that no evidence-based indication criteria are available. Indication criteria concerning THA/TKA consisted of the following domains: pain (in respectively 11 and 10 sets), function (12 and 7 sets), radiological changes (10 and 9 sets), failed conservative therapy (8 and 4 sets) and other indications (6 and 7 sets). Specific cut-off values or ranges were often not stated and the level of evidence was low.The indication criteria for THA/TKA are based on limited evidence. Empirical research is needed, especially regarding domain specific cut-off values or ranges at which the best postoperative outcomes are achieved for patients, taking into account the limited lifespan of a prosthesis.The online version of this article (doi:10.1186/s12891-016-1325-z) contains supplementary material, which is available to authorized users. Total hip and knee arthroplasty (THA/TKA) have been widely performed since the 1970s. In 2009 over a million of THA and TKA were carried out in the United States . As a reThe rise in THA/TKA surgery has important implications for health care costs as well as capacity. As such, it is of utmost importance that patients are carefully selected, and that these procedures are optimally timed to achieve the best possible patient outcomes and that revision surgery is prevented thereby reducing costs and worse outcomes. However, large heterogeneity exists in the patients disease severity at the time of surgery , 7. ThisGuidelines concerning THA and TKA indications have been published and several studies regarding the appropriateness of THA and TKA have been conducted , 19–21. Websites of orthopaedic and arthritis organizations (English or Dutch websites) were independently searched by two authors for guidelines concerning primary THA/TKA for OA. When these websites cross-linked to guidelines from other organizations these were also included. All available guidelines published since January 1, 2000 were included. A librarian-assisted search strategy was performed on August 3 2014 to retrieve additional publications on THA/TKA indications. The following databases were searched: Pubmed, MEDLINE, Embase, Web of Science, the COCHRANE Librabry, CENTRAL and CINAHL. Searches were limited to English, Dutch and German language papers published since January 1, 2000 .The full-text articles were reviewed by MG and were included when the following criteria were met: studies reporting about indication criteria and/or appropriateness of decision tools for primary THA/TKA in OA. Papers involving guidelines on unicompartimental replacements, resurfacing or revision of THA/TKA were excluded if no separate indications for primary THA/TKA were provided. Also papers on prioritizing tools to reduce waiting times were excluded unless these reported on the appropriateness of surgery as all of these patients on the waiting list already have an indication and variables determining priority are not necessarily the same as variables determining an indication for surgery.The included papers where checked by a second author (SH). If disagreement existed the authors tried to reach consensus, when necessary a third author had the decisive vote (PM). When a guideline was also published as a scientific paper, only the guideline was included.➢ Ia evidence from meta-analysis of randomized controlled trials➢ Ib evidence from at least one randomized controlled trial➢ IIa evidence from at least one controlled study without randomization➢ IIb evidence from at least one well-designed quasi-experimental study➢ III evidence from at least one non-experimental descriptive study, such as comparative studies, correlation studies, and case-control studies➢ IV evidence from expert committee reports or opinions or clinical experience of respected authorities or bothThe following information was extracted from the guidelines by MG: orthopaedic or arthritis organization, publication date, indication criteria and the level of evidence on which indication criteria were based (see below). We extracted the following information from the publications: first author, publication date, country where the indication criteria were developed, the organization(s) that initiated the development of the criteria, study type, indication criteria and the level of evidence on which indication criteria were based. The level of evidence was scored according to the following the criteria :➢ Ia eviData extraction and level of evidence score was checked by SH.Guideline quality was assessed with the validated AGREE-II instrument  . This inThe AGREE-II consists of six quality domains: 1) scope and purpose, 2) stakeholder involvement, 3) rigour of development, 4) clarity of presentation, 5) applicability and 6) editorial independence. Each domain entails several questions which are rated from 1 (lowest score) to 7 (highest score), with 1 rated for items with no clear discussion or no specific information, 7 for exceptional reporting quality, 2–6 for items not fully meeting the AGREE-II criteria. Scaled domain scores were calculated using the following formula:The scores will always lie between 0 and 1, with scores closer to 1 indicating higher quality. The scaled domain scores from the two authors were averaged to obtain one quality score for each domain. We summed the scaled domain scores across the 6 domains to obtain 1 overall guideline score. The maximum summed score was thus 6, indicating high quality.Across guidelines and studies, 12 THA, 10 TKA and 2 THA/TKA indication sets were found.We found six guidelines concerning THA, of which three specific OA guidelines . Hence, it seems that guideline quality did not influence the main domains included in the indication sets.Our literature search yielded 3065 references Fig. , the fulOf the included 12 reviews, only 2 were systematic reviews  [n = 3) [The other 5 included original studies investigated decision tools to assess the appropriateness of TKA (n = 2) , 9 or TH [n = 3) , 11, 36 Two studies evaluated algorithms to assess TKA appropriateness , 9. The Riddle et al. modified the Escobar algorithm to attain a decision tool for US patients . They usQuintana et al. developed three THA appropriateness algorithms in OA patients , 11, 36.The other algorithm was based on the WOMAC as they wanted to develop a tool based on a disease specific instrument rather than on expert opinion . SurgicaIn this systematic review we examined the quality and evidence base of existing indication criteria and guidelines for primary THA and TKA in OA patients. Across guidelines and publications we found, 12 THA, 10 TKA and 2 THA/TKA indication sets. Only 6 guidelines included indication criteria for THA/TKA with differing quality. Overall quality of the guidelines summed across the 6 domains ranged from 0.46 to 4.78. Low scores were frequently attained in the editorial independence domain and the applicability domain. High scores were often attained in the clarity of presentation domain. In the additional 12 reviews and 6 original publications most indication criteria included the following three domains: pain, function and radiological changes. Frequently a prerequisite was that conservative treatment had been insufficient in controlling pain. However, domain specific cut-off values or ranges were mostly not reported. Also, it was often not stated if pain, functional disability and radiological changes should all exist, or which combinations of domain-specific deviations should be apparent to indicate THA or TKA. The level of evidence was low (level IV).We were not able to discriminate between high and poor quality guidelines as the AGREE-II has not given a set of rules to define a high quality guideline. Given the low scores in the applicability and the editorial independence domains, we advise guideline developers to pay more attention in reporting these issues. A limitation of the current study may be that the scoring of guidelines according to the AGREE-II is not completely objective, even though the manual clearly articulates how each item should be scored including the criteria and considerations for each item. However, the weighting of criteria and considerations in the overall scoring of the item is not mentioned, which could introduce inter-observer variability. To cope with this, the AGREE-II proposes to use more than one observer, which is why the guidelines were scored independently by two investigators and compared to reach consensus (with or without a decisive vote of a third investigator). As such, we tried to minimalize subjectivity.Another limitation of this study is that we restricted our search strategy to English, Dutch and German language papers. In addition, our website search of orthopaedic and arthritis organizations was restricted to English and Dutch websites. Therefore we may have missed part of the existing literature on indication criteria for THA/TKA.Irrespective of the quality of individual guidelines, the same domains concerning THA/TKA indications were reported across most guidelines. Based on the design of included studies, the highest level of evidence was reported by the OARSI and EULAR guidelines . The evidence on which indication sets were based came from studies investigating the effectiveness and safety of THA/TKA, but these studies did not specifically address THA/TKA indication sets. Therefore the evidence from these guidelines was rated as level IV evidence, so that the evidence on which indication criteria are based, is low quality evidence.Looking at other literature, most of the reviews also did not specifically focus on THA/TKA indications and none of the systematic reviews did. Moreover, in 2 of 3 reviews with THA/TKA indication as the main topic it was concluded that no conclusive evidence on THA/TKA indications are currently available. Furthermore, few original papers investigating THA/TKA indications were found, which may be partly due to the employed language restrictions, possibly resulting in language bias. Four of five original studies came from the same group and four were based on the RAND/UCLA method –11. AlthNonetheless, when indications were reported, the same domains were included. Hence, although evidence based studies are lacking, expert opinion seems reasonably consistent. This is promising as these domains may give clues to the targets on which future research for THA/TKA indications should focus. It seems evident that pain, function, radiological changes and failed conservative therapy should be part of future studies on THA/TKA indications. The research of indication criteria is, however, difficult. One of the difficulties is that pain and function are relatively subjective measures both when reported by the patient and when judged by the physician. This is illustrated by the fact that although consensus on the indication domains seems to exist, disease severity greatly varies at the time of surgery across different centres in Europe and Australia , 7. ThisIn conclusion, our current study gives an overview of the available evidence base of THA/TKA indication criteria in both guidelines and original studies. We showed that the currently available THA/TKA indication criteria are based on limited and low quality evidence. Hence, empirical research on this topic is needed, especially regarding domain specific cut-off values or ranges at which the best postoperative outcomes are achieved.


In traditional psychosis prediction research, the assumption is that a single “snapshot” of clinical disturbance at time point one (i.e. baseline) can reliably predict the future emergence of psychosis over time . This is a linear, static approach to psychosis prediction. However, the field increasingly recognizes that mental health behaves as a non-linear, dynamic system, common to other complex structures such as ecosystems, financial markets or the climate.Increasing evidence points toward the existence of generic “tipping points” in these complex dynamic systems. A tipping point refers to a critical threshold whereby a system shifts from one state into another. Evidence suggests there are universal early warning signals/resilience indicators , which predict close proximity to a critical tipping point.There is growing evidence for the presence of these early warning signals in psychopathology. This presentation will introduce theoretical concepts of tipping points and resilience indicators in the context of transitioning from at-risk mental state to frank psychosis.This new framework may represent a paradigm shift from static prediction approaches to dynamic, individualized models of psychosis prediction and may inform the development of new clinical identification tools and early and individualized interventions to prevent such transitions.


Background: Evidence-based guidelines published by health authorities for the promotion of health-enhancing physical activity (PA), continue to be implemented unsuccessfully and demonstrate a gap between evidence and policies. This review synthesizes evidence on factors influencing delivery, adoption and implementation of PA promotion guidelines within different policy sectors . Methods: Published literature was initially searched using PubMed, EBSCO, Google Scholar and continued through an iterative snowball technique. The literature review spanned the period 2002–2017. The realist synthesis approach was adopted to review the content of 39 included studies. An initial programme theory with a four-step chain from evidence emersion to implementation of guidelines was tested. Results: The synthesis furthers our understanding of the link between PA guidelines delivery and the actions of professionals responsible for implementation within health services, school departments and municipalities. The main mechanisms identified for guidance implementation were scientific legitimation, enforcement, feasibility, familiarity with concepts and PA habits. Threats emerged to the successful implementation of PA guidelines at national/local jurisdictional levels. Conclusions: The way PA guidelines are developed may influence their adoption by policy-makers and professionals. Useful lessons emerged that may inform synergies between policymaking and professional practices, promoting win-win multisectoral strategies. Worldwide trends of inactivity, which increase with economic development urge societies to invest in physical activity (PA) promotion to counteract the growing social and economic costs of multiple non-communicable diseases associated with insufficient PA levels . In lineStrong scientific evidence supports the relationship between built environment and trends toward physical (in)activity at population level ,8, despiTwo main types of guidelines for PA promotion have been produced. The first type encompasses official position statements or clinical and educational guidance documents underpinned by scientific evidence on the relationship between PA/exercise and health. Such guidelines consist of recommendations for reaching and maintaining health-enhancing levels of PA, targeting different age groups ,14, fromGuidelines are targeted to disseminate updated information and evidence-based recommendations in order to promote a change in behaviour toward a healthy lifestyle and adherence to recommended health-related practices. However, despite widespread circulation and publicity of clinical and educational health-related guidelines, they are often not applied ,22. A wiThe formulation of guidance to influence policymaking and professional practices is embedded in the broader process of knowledge translation . AccordiMultiple reasons explain why research findings and evidence-based guidance cannot be transferred in a simple linear way into policy ,31,32. TConcluding, an incomplete evidence base still exists to decide which guideline dissemination and implementation strategies may be effective; to this aim, a theoretical framework should drive the synthesis of evidence to enhance the adherence of different users . There iThe purpose of this realist review was to develop a conceptual framework of facilitators of and barriers to the delivery and adoption PA guidelines within different policy sectors. Specifically, we aimed to identify: (1) Mechanisms of advocacy through which PA entered onto the agenda setting and ways in which evidence was translated in PA guidelines; (2) Factors that may facilitate or hamper PA guidelines adoption within national plans or regulatory tools in different policy sectors and compliance by relevant practitioners. These issues were addressed using the realist synthesis approach , which iThe process from evidence emersion to the delivery of PA guidelines and to changing of regulatory tools and professional practices in multiple policy sectors was examined; thus conceptualising the guidance implementation process as a complex social intervention involving a complex implementation chain. Indeed, complex intervention programmes are by definition implemented by different stakeholders. Along the implementation chain, resources and opportunities are provided to and exploited by different decision makers. Based on this conceptualization, we used the “realist approach” of Ray Pawson  to revieTo map and articulate the key programme theories ,42 of PAThe general rationale of the initial programme theory is grounded on the evidence-based policy movement and on the key role attributed to guidelines as a tool to influence policymaking processes and, lastly, practitioners’ behaviour . SpecifiFor each step, possible realist explanations regarding mechanisms that underpin change in specific contexts were inferred from the reviewed literature by answering a number of specific sub-questions. To generate and prioritize the review sub-questions, we conducted a consensus process utilising the content expertise of policy makers and scholars of PA promotion. To this aim, a sheet was sent to four targeted experts from the Health-Enhancing PA (HEPA) Europe network working at national/European level to obtain suggestions about the consistency of the initial programme theory and judgments about the relevance of the related questions. The sheet included a set of questions centred on the promotion of PA for health. HEPA experts’ feedback underlined the need to monitor, at European and national level, the effectiveness of PA guidelines implementation.The sub-questions for each step were mainly expressed in terms of possible hypotheses to test Context-Mechanism-Outcome configurations. In a realist review, a Context-Mechanism-Outcome pattern configuration allows to retrace regularities and to understand “what works for whom in which circumstances” . For exaPrimary studies, reviews and evaluation studies were initially searched through PubMed, EBSCO, Google Scholar and, according to the realist synthesis approach, continued through the snowball technique with continuous iterative search. Policy documents were mainly retrieved from European Union, European Commission, WHO and WHO Europe websites. The search strategy involved intersections of the following terms: physical activity; physical inactivity; health-enhancing physical activity; physical exercise; active commuting; guideline; guidance; guidance implementation; guidance dissemination; health promotion guidance; plan; knowledge translation; built environment; evaluation; agenda setting; advocacy; delivery, adoption; implementation; multisectoral intervention(s); multisectoral strategy/-ies; policy; healthy environment; school; education; sport; transport; community; partnership; local authority.It is important to underline that for realist synthesis in general and for our review questions in particular, relevant data sources go beyond traditional disciplinary and sector boundaries. Seven hundred and seventy-six electronic references were collected considering all relevant research designs and methods; 535 references covering publication years 2002 to 2017 (date last searched: August 2017) were analysed. A first selection was performed excluding those publications not specifically addressing evidence emersion, building, delivery or dissemination of PA guidance. In a subsequent selection step, references were included according to their pertinence to the four propositions and specific sub-questions. At the end of the process, we selected 39 studies, among them seven systematic reviews, in order to respond to the sub-questions related to each step of the initial programme theory testing. The selection strategy is represented in The inclusion of data to inform programme theory development was guided by the RAMESES principles of relevance and rigour . The quaThe typical Context-Mechanism-Outcome realist configuration was adopted to compare and interrogate literature and synthesise evidence. According to the scope of this review, we attempted to identify recurrent patterns of contexts and outcomes, giving priority to the following contexts: (1) main policy sectors , (2) different governmental levels , (3) country areas with a special focus on the EU context. Successively, we sought to explain these recurrent patterns or demi-regularities through the mechanisms by which they occurred. All the assumptions, corresponding to each of the four steps of the initial programme theory, were tested through comparison with literature search results. How and Through which Mechanism of Advocacy did PA enter onto the Agenda Setting? For this first step concerning how clinical and epidemiological evidence influences policymaking processes and how it mobilises interest groups for PA, we chose to focus on the case of EU compared with the United States and Australia. The focus on Europe, contrasted with other main developed economies with analogous inactivity epidemics, was prompted by the actual interest of the European Union institutions for multisectoral PA promotion strategies ,10,11. TOur findings indicate that the current EU attention to obesity and physically (in)activity in strategic documents  depends The risk inherent in linking PA promotion mainly to obesity prevention and reduction is that to neglect evidence on the far broader role played by PA in health enhancement and maintenance independently of weight loss and, vice versa, by physical inactivity in the development of major non-communicable diseases . FurtherAre PA guidelines developed synthesising data meaningful for different policy sectors?Is there coherence among PA guidelines delivered by different authorities?Guidance delivered by public health authorities having scientific legitimation, plays a relevant role in orienting policymaking processes toward PA promotion . NeverthDifferences in PA guidelines among regional health departments of the same country sometimes emerged as regards purposes, evidence, focus of prescriptions and optimal amount of exercise . In 2006Are PA guidelines coherently integrated within national/regional plans for PA, physical education at school and sports promotion?Which are the determinants of effective PA guidelines adoption in the health sector at local government level?Is the perception of PA as a policy priority influenced by policy sectors different from health ?Did medical schools adopt curricula coherently with PA guidelines?The WHO Europe  highlighIn the sport sector, elite sport facilities and events are often prioritised at the expense of those for recreational sport-for-all based on the erroneous assumption that such prioritisation would increase general participation in sport. Indeed in Europe, there is a paucity of evidence supporting the hypothesis that “Sport for All” and elite sport benefit each other  and thatState funding for PA, adequate staffing and supportive state legislature seem relevant to the adoption of PA guidelines in the health sector . A mechaGuidelines dissemination seems to translate into increased adoption and implementation by state level practitioners more clearly and consistently than by local level practitioners. In The Netherlands, local government is required by national law to develop health policy at the municipal level. Nevertheless, local health policy did not appear to be of pivotal importance to the operations or organization of stakeholders mainly due to the dominance of a medical approach to health that slows the progress of integrated health policies . In NorwIn the United States, urban planners seem to play a more relevant role in PA promotion as compared to local health departments, being able to anticipate innovative, multisectoral win-win strategies for active lifestyle and commuting . A publiFinally, the so-called “Cinderella” status of PA in health care and public health  seems reDo school principals’ beliefs about compatibility of PA and academic achievement matter? Do medical doctors’ attitudes influence PA prescription? Do urban planners’ and administrators’ attitudes influence PA promotion interventions?School principals who have a supportive attitude to students walking to school seem more likely to believe that active travel benefits not only the health of students, but also their scholastic success . Thus, aLack of time to promote PA is the most often cited perceived barrier ,91,92,93The perception of “feasibility” by public officers and administrators of multisectoral policy guidelines seems a relevant mechanism to increase the potential of successful policy implementation to promote an environment more conducive to PA among children . FurtherEvidence-based PA guidelines delivered by international health bodies play a significant role for the dissemination of PA recommendations and the realization of multisectoral strategies for PA promotion. However, there is not a simple direct relationship between scientific evidence, delivery of guidance, adoption of policy measures to contrast physical inactivity trends and changes in professional practices. The general purpose of the present realist review was to develop a conceptual framework of facilitators of and barriers to the delivery and adoption PA guidelines within different policy sectors. We analysed the long chain leading from advocacy to adoption of PA guidelines and, specifically, (1) the mechanisms of advocacy and the ways of evidence translation into PA guidelines; (2) the factors that may facilitate or hamper PA guidelines adoption in different policy sectors and the compliance by relevant practitioners. Expected and unexpected outcomes are explained by a plurality of mechanisms triggered at each step of this complex implementation chain .Several barriers hamper PA guidelines delivery and adoption. They range from factors that limit the meaningfulness of evidence and the subsequent translation into coherent and useful recommendations for different target groups  to weak mechanisms of advocacy for PA promotion. The main weaknesses identified regarding the delivery and adoption of PA guidelines were : (a) PA A critical point of evidence-based guidelines is that a medical approach dominates “how” evidence is built and data gathering is fulfilled. It implies some limitations in research agendas about PA, involvement of relevant sectors of knowledge, evidence search techniques, evaluation methods and source of evidence narrowed to health-related databases. According to some scholars, broad conceptual and methodological gaps need to be addressed to progress research on public open space and PA and therefore to better design PA guidelines . When inThreats to the adoption of PA guidelines within national or local strategic plans and regulatory measures emerged. A key problem is that while the health sector, mainly at national or international level, usually provides leadership or stewardship for PA and publishes guidelines; it then also needs to engage other sectors in the quest for promoting PA. Indeed, the action on PA often falls in the domain of professionals belonging to other sectors such as education, sport urban planning and transport . Non-synPA guidelines seem more effective for urban planning and transport sectors if framed in broader terms  and inspired by a holistic view of health that encompasses other dominant concerns as quality of life and environmental sustainability. Effective PA promotion strategies can be implemented by coordinators of the built environment and assisted by academics with research expertise in architecture, planning and health . EffectiResults demonstrated that the adoption of, or adherence to PA guidelines are influenced by factors acting at different community and individual levels: previous experiences of effective PA promotion interventions in the community, competition among different priorities or other health promotion guidelines , as wellMany mechanisms mediate the long journey that goes from evidence emersion to PA promotion guidelines delivery and adoption by several sectors. Some of the mechanisms were not present in the initial programme theory, but emerged along the review. Though the role of scientific knowledge is widely recognised, the adoption of PA guidance varies and the goals are reframed according to each policy sector, its priorities and dominant organisational culture. For example, school principals and their beliefs about the compatibility of physical education with academic performance represents a key issue, while for urban planners at municipal level, active commuting and cycling are stronger priorities than PA “per se”. In the sport sector, PA policy competes with policies that prioritise investments for elite sport facilities and events. In conclusion, policy sectors other than health seem to play a pivotal role for ensuring effectiveness of health-enhancing PA promotion. Thus, it seems necessary to extract evidence and offer recommendations that are meaningful for other sectors.To our knowledge, this review represents the first attempt to employ the realist synthesis approach to analyse the delivery and adoption of PA promotion guidelines more broadly in several policy sectors. It may complement review outcomes focused on the uptake of clinical guidelines in the health sector obtained with a realist synthesis approach .However, in each policy sector, the concepts of “evidence” and “evidence-based policy” differ in terms of quality assessment and the relevance of evidence in specific circumstances . The preIn the present review, the realist synthesis approach was used to investigate variability in the adoption of PA guidelines in different circumstances, namely at different levels of governance and in different policy sectors ,101,102.As evidence and guidance delivery alone are not sufficient to influence practitioners and policy maker behaviours ,36,75, tOur conclusion is that PA guidance delivery should go beyond a narrow focus on the “know-do gap” , acknowl


Modified Vaccinia Ankara (MVA) is a live, viral vaccine under advanced development as a non-replicating smallpox vaccine. A randomised, double-blind, placebo-controlled phase III clinical trial was conducted to demonstrate the humoral immunogenic equivalence of three consecutively manufactured MVA production lots, and to confirm the safety and tolerability of MVA focusing on cardiac readouts.The trial was conducted at 34 sites in the US. Vaccinia-naïve adults aged 18-40 years were randomly allocated to one of four groups using a 1:1:1:1 randomization scheme. Subjects received either two MVA injections from three consecutive lots (Groups 1-3), or two placebo injections (Group 4), four weeks apart. Everyone except personnel involved in vaccine handling and administration was blinded to treatment. Safety assessment focused on cardiac monitoring throughout the trial. Vaccinia-specific antibody titers were measured using a Plaque Reduction Neutralization Test (PRNT) and an Enzyme-Linked Immunosorbent Assay (ELISA). The primary immunogenicity endpoint was Geometric Mean Titers (GMTs) after two MVA vaccinations measured by PRNT at trial visit 4. This trial is registered with ClinicalTrials.gov, number NCT01144637.Between March 2013 and May 2014, 4005 subjects were enrolled and received at least one injection of MVA (n = 3003) or placebo (n = 1002). The three MVA lots induced equivalent antibody titers two weeks after the second vaccination, with seroconversion rates of 99·8% (PRNT) and 99·7% (ELISA). Overall, 180 (6·0%) subjects receiving MVA and 29 (2·9%) subjects in the placebo group reported at least one unsolicited Adverse Event (AE) that was considered trial-related. Vaccination was well tolerated without significant safety concerns, particularly regarding cardiac assessment.The neutralizing and total antibody titers induced by each of the three lots were equivalent. No significant safety concerns emerged in this healthy trial population, especially regarding cardiac safety, thus confirming the excellent safety and tolerability profile of MVA.NCT01144637ClinicalTrials.gov  Despite the eradication of smallpox in 1980 , 3. The ® in healthy subjects. Prior MVA vaccination inhibits the replication of vaccinia virus (Dryvax®) in the skin [®; invented name in the EU IMVANEX®) [MVA has demonstrated protective efficacy in animal models, including non-human primates , as wellthe skin . In ordethe skin ,13,15,19MVANEX®) ,18,20,21The objectives of the phase III trial reported here were to demonstrate the equivalence of three consecutive MVA production lots in terms of induced antibody titers, and to confirm the favorable cardiac safety and tolerability profile of MVA.th Avenue, Seattle, WA; Washington University in St. Louis Human Research Protective Office, 600 South Euclid Avenue, St. Louis, MO; Marshall University IRB #1 / Office of Research Integrity, 401 11th Street, Huntington, WV; The University of Alabama at Birmingham IRB for Human Use, 701 20th Street South, Birmingham, AL.This randomised, double-blind, placebo-controlled Phase III trial was conducted at 34 sites in the US between March 18, 2013 and May 23, 2014. The study was approved by the following Institutional Review Boards: Quorum Review IRB, 1601 5Healthy, vaccinia-naïve, women and men aged 18-40 years were randomised into four groups: Groups 1 to 3 received two subcutaneous (s.c.) MVA injections from three different consecutively manufactured vaccine lots at weeks 0 and 4. Group 4 received two s.c. placebo injections in the same time interval. The active trial period (8-10 weeks) comprised five visits. A phone follow-up (FU) visit was performed 26 weeks after the second injection. The trial was conducted according to ICH-GCP standards and the principles of the Declaration of Helsinki. The protocol was approved by independent review committees for each site and all subjects provided written informed consent prior to participation.Treatment was assigned at trial visit 1 after re-confirmation of the subject’s eligibility. The randomisation scheme was 1:1:1:1 (3 MVA groups and 1 placebo group). Randomisation was stratified by clinical trial site. The unblinded trial statistician generated randomisation sequences (block randomisation with block size of eight) and the unblinded clinical trial staff assigned each participant to a treatment group using a secure interactive web response system.Everyone except personnel performing randomization, vaccine handling and administration, remained blinded to treatment throughout the study.8 TCID50. Placebo consisted of the vaccine formulation buffer, Tris-buffered saline (TBS), provided in liquid aliquots. One placebo dose (0·5 mL) contained 0·605 mg tris (hydroxymethyl)-amino methane and 4·09 mg sodium chloride.The MVA bulk drug substance was produced at Bavarian Nordic A/S  according to cGMP standard and the final drug product was filled, formulated and labelled at IDT Biologika GmbH . One dose (0·5 mL) of the liquid frozen formulation had a nominal virus titre of 1 x 10To evaluate the safety of MVA, solicited and unsolicited AEs were recorded at all visits. Serious Adverse Events (SAEs) were monitored up to the six months phone FU visit. Solicited AEs constituted a set of pre-defined, expected local reactions  and systemic symptoms . All solicited AEs were recorded on a memory aid during an 8-day period following each injection. Treatment-emergent unsolicited AEs were events not included in the memory aid and reported by the subject within a 29-day period after each injection. Safety laboratory tests were performed at screening and two weeks after each injection. Troponin I testing and electrocardiograms (ECG) were performed at screening and two weeks after the first injection. If clinically indicated, additional safety measures could be taken at any other scheduled or unscheduled visit. Any cardiac symptoms developed since the first injection, clinically significant ECG changes and Troponin I elevations ≥2 x upper limit of normal (ULN) were defined as AEs of special interest (AESIs). Any subject developing an AESI was requested to return to the clinical trial site in order to perform a thorough work-up, physical and cardiac examination comprising ECG, cardiac enzymes and/or echocardiogram. If necessary, further diagnostic tests and regular follow-ups could be performed until complete resolution or stabilization.Blood for immunogenicity serum collection was drawn at baseline prior to the first injection (week 0) and two weeks after the second injection (week 6). These time points are based on results obtained during the phase I and II development program, in which peak antibody titers were consistently observed two weeks after the second vaccination in vaccinia-naïve subjects ,16,19–22The primary objective was to assess the equivalence of three consecutively produced MVA lots by PRNT. The secondary objectives were to assess uncommon adverse reactions, with a particular focus on cardiac signs and symptoms indicating myo- /pericarditis, and to compare their frequency to those observed after placebo. Additionally, vaccinia-specific humoral immune evaluations utilizing an ELISA and a correlation analysis between total and neutralizing antibody titers were performed.Primary immunogenicity endpoint:GMTs after two MVA vaccinations measured by PRNT at trial visit 4.Secondary immunogenicity endpoints:GMTs after two MVA vaccinations measured by ELISA at trial visit 4.Vaccinia-specific PRNT and ELISA seroconversion rates at trial visit 410 transformed PRNT titers and the log10 transformed ELISA titers at trial visit 4.Pearson Correlation Coefficient between the logSecondary safety and reactogenicity endpoints:Occurrence, relationship and intensity of any Serious Adverse Event (SAE) at any time during the trial.Occurrence, relationship and intensity of any cardiac sign or symptom indicating a case of myo-/pericarditisOccurrence of any Grade 3 or 4 Adverse Events (AEs) probably, possibly or definitely related to the trial vaccine within 28 days after vaccination.Occurrence, relationship and intensity of unsolicited non-serious AEs within 28 days after each vaccination.Occurrence, relationship and intensity of solicited local AEs  during the 8-day period (day of vaccination and the following seven days) after each vaccination.Occurrence, relationship, intensity and duration of solicited general AEs  during the 8-day period (day of vaccination and the following seven days) after each vaccination.The Full Analysis Set (FAS) consisted of all randomized subjects who received at least one vaccination and for whom any data were available. For all analyses using the FAS, subjects were summarized according to the treatment that they received.The Immunogenicity Analysis Set (IAS) was a subset of the FAS comprised of the first 700 randomized subjects from each group.The Per Protocol Set (PPS) consisted of all subjects present in the IAS who received both vaccinations, completed Visits 1, 3, and 4, and adhered to all protocol conditions. Subjects with only minor protocol deviations were included in this dataset.Statistical analyses were performed using SAS version 9.1 .10 titer SD of 0.85 in all groups receiving MVA for the PRNT, the primary hypothesis of the PRNT GMT at Visit 4  has a power of > 80% to show equivalence for all three MVA groups, using an equivalence margin of Δ = 0.301 on the log10 scale. In order to account for a dropout rate of about 15%, which has been observed in previous MVA trials, serum collection was scheduled for a total of 700 subjects in each group.Assuming a significance level of 5%, an analyzable sample size of 600 per group for the PPS, and an expected logTo allow a proper safety analysis, 300 additional subjects were enrolled in each group to ensure that a total of ~ 1,000 subjects per group were available, giving a combined MVA safety population of 3,000 subjects. With this sample size the trial had a 95% chance of detecting any uncommon AEs, i.e., AEs with an incidence of at least 1/1,000.Immunogenicity analyses were primarily ran on the PPS and the IAS.10 titers for the calculation of geometric means and 95% confidence intervals (CIs). Titers below the detection limit were assigned a value of 1 for GMT calculation.The PRNT and ELISA titers were transformed into log10-transformed post baseline titers between pairs of lots. Equivalence among the three lots was demonstrated if for each pair of lots, the two-sided 95% CI for the log10 PRNT mean was between -0·301 and 0·301, i.e., if the PRNT GMT ratio of each pair of lots was between 0·5 and 2·0. For the ELISA, the two-sided 95% CI was -0·176 and 0·176, i.e. the ELISA GMT ratio was between 0·66 and 1·5.For lot equivalence, a two-sided 95% CI was calculated using the differences of the means of the log10 titers measured by both assays was calculated per group and for Groups 1–3 combined along with the associated p-values and 95% CIs.For the PRNT and ELISA correlation, Pearson’s correlation coefficient (with associated CI) between logThe trial was overseen by an independent data safety monitoring board  and is registered with ClinicalTrials.gov, number NCT01144637.A total of 5357 volunteers aged 18-40 years were screened, of whom 4005 were eligible for enrolment. All eligible subjects received at least one injection, had data available for analysis and were included in the FAS. Of these 4005 subjects, 2829 were included in the IAS. Of the 2829 subjects in the IAS, 2549 were eligible for the PPS .There were no statistically significant differences for any demographic characteristics among treatment groups .Vaccinations were generally well tolerated.Higher proportions of subjects vaccinated with MVA experienced solicited local AEs compared to Placebo .Injection site induration and injection site hematoma were the only unsolicited AEs experienced by more than 1% of the subjects and mostly occurred after the first vaccine administration (data not shown).A total of 18/3003 MVA-vaccinated subjects (0·6%) and 3/1002 placebo-vaccinated subjects (0·3%) were withdrawn from the trial due to an AE, most of which were assessed as unrelated or unlikely related to the trial vaccine, and had a mild to moderate intensity. Four subjects receiving MVA were withdrawn due to AEs corresponding to arthralgia, bundle branch block, pericarditis or pruritus, assessed as possibly related to the trial vaccine. The occurrence of two life-threatening events (a suicide and a suicide attempt) leading to withdrawal of subjects receiving MVA was assessed as unrelated to the vaccine. One subject in the placebo group experienced a seizure in temporal relation to the trial vaccination. This possibly related SAE of moderate intensity was the only Suspected Unexpected Serious Adverse Reaction (SUSAR) occurring during the trial. Injection site reactions and an AE of dizziness leading to withdrawal were assessed as definitely related to the vaccine. Overall, the AE safety profile across the three MVA groups was similar .A small proportion of subjects (< 1%) in the placebo group and the combined MVA groups experienced SAEs (p > 0·9999).A total of 8/3003 subjects (0·3%) in the combined MVA groups experienced AESI compared to one subject in the placebo group . Two subTwo out of the 3003 MVA-vaccinated subjects (0·1%) experienced AESI related to MVA : one in No subjects in the placebo group experienced AESI considered related to trial vaccine. All AESIs, except a tachycardia experienced by a subject in Group 2, occurred after the first vaccination.Immunogenicity data for the PPS are shown in This phase III trial was designed according to the recommendations issued by the US FDA to commercialize a non-replicating smallpox vaccine. The results demonstrate a consistent MVA manufacturing process and confirm the favorable cardiac safety profile of MVA in a large population.·5% after the second vaccination. Analogous findings were previously observed using the same MVA administration scheme [Comparable vaccinia-specific PRNT GMT values were observed two weeks following the second vaccination (week 6) in the PPS, demonstrating the equivalence of three consecutively produced MVA lots. MVA induced strong total (ELISA) and neutralizing (PRNT) antibody responses, with seroconversion rates of at least 99n scheme ,15,20–22This trial adds a large dataset of almost 2000 paired ELISA and PRNT titers to the immunogenicity database, exhibiting a significant correlation between total and neutralizing antibody titers that confirms the results of previous clinical trials . SimilarMost of the AEs reported after MVA administration in this trial comprised temporary local and systemic reactions of mild to moderate intensity that did not require additional treatment nor led to withdrawal from further vaccination. As expected, the frequency of solicited AEs observed in the three MVA groups was significantly higher compared to the placebo group. Their frequency, intensity and duration were equally distributed across the 3 MVA groups, showing lot equivalence. This profile was similar to those established during the phase I and II clinical trial programs, further demonstrating the excellent and reliable safety of MVA across different populations , 20–23.Concerning unsolicited AEs, including SAEs, no trends towards any specific events or system organ classes were observed. The overall number of unsolicited AEs and SAEs was low and the distribution was comparable across all treatment groups.Replicating smallpox vaccines are associated with a series of cardiac AEs of inflammatory nature, such as myo/pericarditis. MVA has exhibited a lack of such effects in almost 8000 vaccinees , 24. A lOnly three of the 3003 MVA-vaccinates showed an increase of Troponin I above normal limit following vaccination. One subject exhibited an elevated Troponin I value at baseline that did not classify as an AESI and therefore met the inclusion criterion for enrolment. This value later increased above baseline levels. The second subject showed increased Troponin I levels upon the second visit (14 days after the first vaccination), accompanied by normal ECG results. His Troponin I levels returned to normal 5 days later. The event was considered as resolved and assessed as unlikely related to the trial vaccine. The third subject had elevated Troponin I levels 15 days after vaccination. Further cardiac examination revealed signs suggesting athlete’s heart, which coincided with a history of marathon running. The event was later considered as resolved and assessed as unlikely related to trial vaccine. In summary, the frequency of Troponin I increase above ULN following MVA administration was below 1 in 1000.All AESI were transient without long-term cardiac damage according to the cardiology follow-up examinations. AESI reports were based on findings obtained through the targeted examinations including Troponin I testing and ECG readings. Epidemiologic figures for background incidence of Troponin I elevations are in the range of 0·43% to 1·5%, depending on the definition of the upper limit of normal –27. The This phase III trial further demonstrates the favorable cardiac safety profile of the non-replicating smallpox vaccine MVA. Replication-competent smallpox vaccines such as Dryvax and ACAM2000 are associated with a significant risk of myo-/pericarditis –10, 29, The overall AE profile observed in this trial coincides with reports from previous trials , 19–22.In summary, this trial has confirmed the excellent safety and cardiac profile of MVA, clearly differentiating MVA from replicating smallpox vaccines. In addition, the robust immunogenicity displayed by the different lots indicates a reliable and consistent manufacturing process.S1 File(PDF)Click here for additional data file.S2 File(DOCX)Click here for additional data file.S3 File(PDF)Click here for additional data file.S4 File(DOC)Click here for additional data file.


There are a variety of methods for priority setting in health research but few studies have addressed how to prioritise the gaps that exist between research evidence and clinical practice. This study aimed to build a suite of robust, evidence based techniques and tools for use in implementation science projects. We applied the priority setting methodology in lung cancer care as an example.We reviewed existing techniques and tools for priority setting in health research and the criteria used to prioritise items. An expert interdisciplinary consensus group comprised of health service, cancer and nursing researchers iteratively reviewed and adapted the techniques and tools. We tested these on evidence-practice gaps identified for lung cancer. The tools were pilot tested and finalised. A brief process evaluation was conducted.We based our priority setting on the Nominal Group Technique (NGT). The adapted tools included a matrix for individuals to privately rate priority gaps; the same matrix was used for group discussion and reaching consensus. An investment exercise was used to validate allocation of priorities across the gaps. We describe the NGT process, criteria and tool adaptations and process evaluation results.The modified NGT process, criteria and tools contribute to building a suite of methods that can be applied in prioritising evidence-practice gaps. These methods could be adapted for other health settings within the broader context of implementation science projects.The online version of this article (doi:10.1186/s12874-016-0210-7) contains supplementary material, which is available to authorized users. Setting priorities in health services delivery and research is highly relevant in an era of scarce resources. In many settings, health researchers and clinicians participate in collaborative efforts to determine how best to implement evidence into routine clinical practice. Priority setting offers an important opportunity for collaborators to determine what evidence is relevant to implement, how best to do this, and where the greatest gains can be made in changing clinical practice . There iFew studies have addressed methods that specifically prioritise existing gaps between research evidence and clinical practice . To dateThe NGT is an interpretive approach that engages stakeholders in interactive discussion to generate priorities , 7. NGT n = 42 participants) to fulfil our primary aim. Data outcomes from the focus groups are reported in a separate paper [Our group had separately developed a set of seven evidence-practice gaps in lung cancer on the basis of a scoping review of the literature  and examte paper .identify relevant health priority setting approaches or techniques that had been used in focus or small group settings;identify original research studies conducted in oncology settings, where oncology health professionals and/or consumers were the target audience for priority setting; and,Cochrane Agenda and Priority Setting Methods Group).locate relevant documents in the grey literature (i.e. reports and publications that may not be peer-reviewed) and web-based sources about group decision making tools used in health. This last search included checking the Cochrane Library  the most relevant technique; and, b) criteria to use in the rating tools. We formed an expert interdisciplinary consensus group comprising of cancer clinician-researchers and academic experts from multiple disciplines including nursing, psychology, health services research, epidemiology and clinical oncology. Results from the literature reviews were presented. To select the most relevant technique, the consensus group considered the relative advantages, disadvantages and/or the applicability of each one. They considered how optimal and practical each technique was in terms of equity of participation encouraged, time required to enact and the group size, stakeholder mix of disciplines and level of expertise required. To select the criteria for use in the rating tools, the consensus group focused on whether there was a clear definition for each criteria and how easily participants would be able to rate each gap against the criteria. The group discussed the criteria options until consensus was reached.n = 7, including two medical oncologists and 5 health service researchers) to test the tools and the priority setting process. Minor adjustments were made to the content and formatting of the tools based on participant feedback. Participants indicated a maximum time of two hours should be allocated for the priority setting process. The pilot test confirmed that the priority setting technique and criteria were acceptable and feasible within this time-frame.A pilot was conducted with key stakeholders (n = 42) to complete a brief 8-item survey to evaluate the process and provide feedback about whether consensus had been achieved.We asked the three lung cancer multidisciplinary teams  ; the DelUsing this same strategy described above, we identified seven journal articles and reports that listed relevant criteria options and three guides to consensus processes for selecting criteria , 18, 22.Through the consensus process, four criteria were selected against which the gaps could be rated. These were: 1) relevance to the local setting; 2) magnitude of the issue; 3) burden of suffering; and 4); amenable to change. Criteria 1-3  were grouped under the heading of ‘significance’, so that results could later be collated and summarised more efficiently.This tool is used to elicit individual responses from participants, rating the gaps against the criteria using a matrix scoring sheet Table . We adapThe same matrix tool can be used to elicit group responses for the purpose of discussing each gap and reaching consensus, and subsequently giving a rating to each one . A ‘DotmEach participant receives $100 fake ‘dollars’ to spend across the gaps. This seeks to allow participants to make a global decision on resource allocation and also confirms whether the gaps are reflected in how participants would most like to invest in each gap. This acts as a cross check with the priorities selected in the matrix.Table Step 1: Describe identified evidence practice gapsIn this step, an investigator team member (facilitator) gave a presentation about the approach used to review evidence about gaps in lung cancer care, read out a brief summary for each gaps and answered any participant questions.Step 2: Present local data/information about gapsThis step included a presentation about national, jurisdictional and local data gathered to support the gaps. Where available, this was tailored to the local site to include analysis of local health district or hospital data collected in registries or administrative databases. This step was an adaptation of Lomas’ second step in the ‘listening model’ .Step 3: Elicit feedback and include additional gaps identified by stakeholdersParticipants were asked to provide feedback about the seven gaps and whether they were relevant in the local hospital setting and surrounding community. We provided an opportunity for participants to nominate any additional gaps and this was considered as elicitation and inclusion of ‘practice-based evidence’. Additional gaps were only included if consensus was reached within the group that the gap should be added.Step 4: Individuals vote privately to prioritise gaps, using moderator-created criteriaThe criteria were described by the facilitator and participants had an opportunity to seek any clarification before privately rating each gap. Participants were then asked to rate each gap for across the criteria listed above without sharing these with others. This was considered an important step as this private rating of items allowed participants to make their own considered judgements prior to reaching group consensus. This step also ensured that every member of the group had read and considered each gap in detail.Step 5: Each participant selects the two most important gaps from the prioritised list.At the conclusion of step 4, each participant was asked to select the two most important gaps from on the matrix tool for the two criteria of ‘significance’ and ‘amenable to change’, according to their highest scores. This step replicates the original NGT step 5 but is modified by selecting only two most important items instead of five.Step 6: Focus group participants discuss ratings and moderator uses matrix tool as a tallyIn this step, participants broke into small groups (maximum of seven participants in each) and shared their ratings for each gap. We pre-assigned participants into small groups that consisted of a mix of professional groups. Each group was asked to nominate a spokesperson to report back to the larger group and provide feedback about the choices made. To achieve equity within small groups, each person spoke in turn about their top two gaps and provided any clarification for their choices. All responses were recorded by a group facilitator on an A4/US letter-sized printed sheet of the matrix tool in ‘Dotmocracy’ style , assigniStep 7: Whole group consensusAt the conclusion of discussion, small groups reformed back into a larger group, compared ratings for differences and anomalies and resolved these to reach group consensus. The moderator led discussion about the gaps against the criteria of significance and amenability to change.Step 8. Investment exerciseIn the final step, participants were asked to spend $100 fake dollars across each gap, configured as five $20 notes. Eight A3/US Ledger-sized envelopes were placed on a table  and participants were asked to spend their money in any way they might choose. Participants were able to configure the dollars in any way so long as all $100 was allocated . At the conclusion of the task, the investigator team tallied the dollars in each envelope and then reported back to the group about the proportions  spent on each gap.The modified NGT process developed for this project was subsequently used in focus groups and is described here as an exemplar. (Please note that the results of the focus groups are described in a separate paper ):Step 1:Tools: The matrix tool was easy to complete and the Likert scale enabled participants to examine how they had comparatively rated each gap. By first completing the matrix tool privately, individual participants had an opportunity to focus on the evidence gaps and consider their responses according to the criteria. Using the same matrix for the group consensus process facilitated discussion in the small group setting. The investment tool of $100 fake dollars allocated across the gaps was a useful way to double check priorities after reaching consensus. The research team also observed that the investment exercise was a fun and engaging way to end the group session, with participants interested in the outcome and whether it was reflective of whole group consensus.Criteria: We found that the ‘amenable to change’ criteria was particularly meaningful for participants, as it allowed them to consider whether their current health service was capable of making change in the ‘gap’. Participation was a useful exercise for reflecting on current practice and what would be desirable to change in the future.Process evaluation: Thirty (of 42) (71 %) participants completed the evaluation survey. Results are shown in Table As a result of testing the methodology with the three lung cancer teams, we observed the following.This study aimed to describe the methods for selecting and adapting a priority setting technique, selecting criteria and developing tools for implementation science projects. We sought to modify the NGT and relevant tools to allow the identified gaps to be rated according to agreed criteria. We believe this is the first study to demonstrate how to modify the NGT in setting priorities for evidence practice gaps within the context of implementation science projects. The selected technique, criteria, tools and process have drawn on evidence based approaches and we believe these will be highly relevant in other health care settings. A similar process was reported by Buckley and colleagues that assessed the effects of a research prioritization process to influence the research agenda in urinary incontinence . Their tWe consider that there is significant value in prioritising gaps in research evidence and where change should be targeted in clinical settings. Gaining stakeholder engagement is key in building effective relationships for adoption of intervention strategies . The advIt is worth reflecting on the modifications made to the NGT process. We modified the first two steps of NGT (generate and recording ideas) to become a presentation of an evidence-practice gaps summary, as the investigator team had already undertaken a significant amount of work to identify relevant gaps. By doing so, we sought to address the documented barrier of clinicians having insufficient time to read and assimilate research evidence . Step 2 Our modifications include additional opportunities for group discussion (Steps 6 and 7). In a subsequent study to test success factors about coordinated cancer in care , we founThe limitations of this study are that we modified the NGT process and tools with a specific purpose of testing evidence-practice gaps in lung cancer with multidisciplinary teams of clinicians working across three hospital settings. Whilst we have used a robust and evidence-based approach to reviewing existing priority setting techniques, it is by no means the only option. Researchers will seek to adapt techniques, criteria, tools and processes to suit their own research aims and local context. However, given the paucity of tools available to researchers working in implementation science , we consIn conclusion, this study shows that existing priority setting techniques and tools can be successfully adapted for use in implementation science projects. The modified NGT process and tools can be used by collaborations of researchers and clinicians who seek to identify priorities for research within a particular health condition. The modified NGT approach is practical, user-friendly and easy to conduct in small groups of clinicians. The study provides a methodology for how to prioritize evidence-practice gaps in other health settings, within the broader context of implementation science projects.


Bereavement can result in severe mental health problems, including persistent, severe and disabling grief symptoms, termed complicated grief. Grief rumination  is a malleable cognitive risk-factor in adjustment to bereavement. The Utrecht Grief Rumination Scale (UGRS) was recently developed to assess grief rumination. The present study aimed to develop and validate a German version of the UGRS.An online survey including measures of demographic and loss-related variables, grief rumination (UGRS), depressive rumination (brooding and reflection), and symptoms of depression, anxiety, and complicated grief, was administered online among 159 persons (87% women) who had lost a first-degree relative in the past three years. UGRS item analyses, a confirmatory factor analysis and associations of grief rumination with brooding, reflection and symptom levels were performed.d > 1.60).The internal consistency of the UGRS was good. The confirmatory factor analysis obtained a good fit for a model with five correlated grief rumination subscales. The UGRS contributed uniquely to the prediction of complicated grief symptoms even when controlling for symptoms of anxiety and depression, brooding, reflection, and demographic and loss-related variables. Discriminant validity of the UGRS was demonstrated by the fact that higher UGRS scores were found in participants with a higher likelihood of receiving a diagnosis of complicated grief  contains supplementary material, which is available to authorized users. Bereavement is a universal human experience. While most people react to a loss with intense pain and may even develop an increased risk of mental or physical health problems , a majorGiven the severe distress that may be experienced after bereavement, a thorough understanding of the malleable factors that may contribute to the development and persistence of mental health problems is important. Thought processes are both malleable (for a review: ) and areTwo types of rumination have been studied in some detail in adjustment to loss. The first investigations in this area were conducted in the mid-nineties. They focused on clarifying the role of depressive rumination after bereavement. Depressive rumination was defined as repetitively and passively focusing on depressive symptoms and on their possible causes and consequences . A frequIn contrast to depressive rumination, grief rumination is not limited to analysing feelings of depression, as negative post-loss emotions are not restricted to sadness or helplessness but may also entail many other emotions including yearning, anger or irritability . Thus, rA model that is currently often used to understand the negative consequences of grief rumination is the Rumination as Avoidance Hypothesis (RAH ). The RAGiven grief rumination’s potential theoretical and clinical relevance, the Utrecht Grief Rumination Scale (UGRS) was recently developed to specifically assess grief rumination . The UGRClearly, more international research is needed to distinguish potentially adaptive and maladaptive facets of rumination at different time points in the grieving process. We also need to elucidate the pathways via which rumination contributes to the development and maintenance of mental health problems and, specifically, complicated grief. As prerequisite to this long-term goal, the present study aimed to develop a German version of the UGRS, to investigate its psychometric properties , and to test its concurrent and discriminant validity by examining associations between the UGRS, reflection and brooding, and symptoms of anxiety, depression and complicated grief.n = 5; premature termination of the survey, n = 28; systematic data pattern, n = 3). The final sample therefore consisted of 159 bereaved participants, who were mostly female (87%) and aged between 18 and 77 years . Concerning time since loss, 23% indicated that the death had taken place less than six months ago, 21% that it had happened six to twelve months ago and most (55%) reported their loss dating back to between one and three years. Most participants (54%) had lost a partner/spouse, 37% had lost a child, 6% a parent and 3% a sibling. The mean Inventory of Complicated Grief (ICG) score in the present sample was 34.33 (±13.56); most participants (75%) demonstrated ICG scores higher than the established cut-off (> 25).Ethical approval was obtained from the Ethics Committee of the Department of Psychology of the Philipps University Marburg (Germany) and invitations to participate were posted on grief-related websites . Recruitment lasted from August to October 2016. People accepting the invitation followed a link to an online survey platform that offered information about the aims of the study, confidentiality and study eligibility criteria; after providing written informed consent, they could participate in the study. As an incentive, participants had the chance of winning one out of two 50€-vouchers for an online store. Inclusion criteria were age over 17 years and having lost a first-degree relative or partner within the last three years. Exclusion criteria were feeling too distressed by grief to answer loss-related questions or having experienced suicidal ideation in the last month. For the analysis in the present paper, participants who indicated that German was not their first language were excluded. A total of 195 participants gave informed consent; of these, 36 were excluded  and loss-related information. Loss-related variables comprised time since loss , relationship to the deceased , and cause of death .The English Utrecht Grief Rumination Scale (UGRS)  was tranThe UGRS consists of 15 items detailing various types of thought about the causes and consequences of the loss; participants indicate how frequently they have experienced each of these in the past month. Answers are rated on a five-point scale ranging from 1 (never) to 5 (very often). Item scores are summed to generate an overall grief rumination score. Furthermore, five subscales can be computed, consisting of three items each. ‘Reactions’  measures how frequently participants analyse their emotional reactions to the loss  The subscale ‘Injustice’  captures thoughts about the injustice of the loss (example item: ‘How often in the past month did you wonder why this had to happen to you and not to someone else?’). ‘Counterfactuals’  assesses counterfactual thinking about the events leading to the loss . The subscale ‘Meaning’  measures thoughts about the meaning and the consequences of the death . The subscale ‘Relationships’  assesses thoughts related to reactions from the social environment (example item: ‘How often in the past month did you think about how you would like others to react to your loss?’). Eisma et al.  showed tThe Response Style Questionnaire 10D (RSQ-10D) , 33 is tTo measure symptoms of anxiety and depression, the Hospital Anxiety and Depression Scale (HADS)  was usedThe Inventory of Complicated Grief (ICG)  was usedn = 5; HADS n = 6; RSQ-D n = 6.In the UGRS, no missing data were observed; single missing items in other questionnaires were replaced according to the respective questionnaire’s instructions . Very few participants failed to provide answers to single items, namely: ICG For the UGRS, standard item analyses were calculated to determine mean item scores and standard deviations, item difficulties, item-total correlations  and estimations of internal consistency when the item was omitted. Mean inter-item correlations, mean item difficulties and internal consistency (standardised Cronbach’s α) of the UGRS and its subscales were computed. Confirmatory factor analyses with maximum likelihood estimation were calculated to test two models previously reported for the factorial structure of the English and Dutch versions : The firVarious facets of validity were investigated. Convergent validity was assessed by inspecting zero-order correlations of UGRS and brooding. Divergent validity was assessed by calculating zero-order correlation of UGRS and reflection; z-tests compared the correlations of UGRS with brooding and reflection, respectively, since grief rumination should be less closely related to reflection. This prediction relies on the distinction between adaptive and maladaptive forms of repetitive thinking , 16, 40.d is reported as a measure of effect size. The data analysis was carried out with IBM SPSS statistics 24; for the confirmatory factor analysis, the SPSS AMOS version 21.0.0 was used . Unless otherwise stated, the α level was set to p = .05.To investigate the discriminant validity of the UGRS, we compared potential candidates for a complicated grief diagnosis with non-candidates for such a diagnosis. A potential candidate for a complicated grief diagnosis was defined as a participant with an ICG score above the cut-off of 25  who alsopi = .37 (item 10) and pi = .80 (item 2) with a mean difficulty of pi = .54. The item-whole correlations of the individual items with the total score ranged from ritc = .45 (item 10) to ritc = .64 (item 9); the mean item-total correlation was ritc = .57 and the mean inter-item correlation was r = .36 .In order to test the previously established five-factor structure , a confirUGRS x Brooding = .56, p < .001) was significantly higher than with reflection  as indicated by the Fisher z-test . Interestingly though, the subscales showed varying associations with the rumination sub-facets brooding and reflection. ‘Injustice’ and ‘Counterfactuals’ were associated with brooding but not reflection, while all other subscales demonstrated significant correlations with both rumination sub-facets  = 3.80, p < .001] and rUGRS x HADS-depression , demonstrated that the latter two were significantly smaller. Considering the UGRS subscales, their correlation pattern with complicated grief, anxiety and depression generally followed the sum score’s pattern.The UGRS sum score showed high correlations with symptoms of complicated grief symptoms as measured by the ICG. Correlations between the UGRS and indicators of anxiety and depression (HADS) were also significant. Using Fisher z-tests to compare the size of the coefficient F = 1.452; p = .232, adj.R2 = .011]. When entered in the next step, both symptoms of anxiety and depression contributed significantly to the prediction of ICG scores . In the third model, brooding but not reflection explained a significant additional amount of variance . UGRS scores were included in the model in the final step and improved the prediction of ICG scores significantly . In this last model, regression coefficients for anxious and depressive symptoms remained significant predictors, while brooding lost its previous significance. This model explained 72% of variance in ICG scores.The hierarchical multiple regression on symptoms of complicated grief investigated the concurrent validity of the UGRS . The UGRS sum score and its subscales differed significantly between the respective candidate groups for a diagnosis of complicated grief and groups of participants with the same time since loss but whose grief symptoms remained below the cut-off, with higher UGRS scores in the candidate groups  than more adaptive types (i.e. reflection). This replicates previous findings  even to d = 1.66 and d = 1.60) in both candidate groups when compared with the group with less likelihood of receiving a diagnosis of complicated grief. This speaks for the discriminant validity of the UGRS.Lastly, we investigated whether UGRS scores differed between participants with and without an increased likelihood of receiving a diagnosis of complicated grief. To this end, we classified participants according to the established symptom level cut-off in the ICG as candidates or non-candidates for complicated grief, applying different time criteria. The loss had to have happened at least six months ago (time criterion ICD-11) or twelve months ago (time criterion DSM 5). Irrespective of the time criterion, the UGRS scores were elevated strongly , psychoAdditional file 1:German Version of the Utrecht Grief Rumination Scale. (PDF 60 kb)