They also made less use of overall ERSs , avoidance strategies , and approach strategies . However, they made greater use of self-development strategies . Social presence was associated with greater use of all ERSs. This study showed that although a sense of social presence in a choir is not reduced by engagement in VCs compared to live choirs, there is a lowered use of ERSs when engaging in VCs. However, as the difference in use of ERSs is relatively modest, virtual musical experiences may still have a role to play in supporting those who cannot engage in live experiences such as people who are socially isolated.Over the past two decades, many musical experiences have become mediated by digital technology, including the distribution of music online, the generation of new content and participation in virtual musical experiences. However, whether virtual musical experiences lead to different experiences of social presence or differential use of emotion regulation strategies (ERSs) compared to live musical experiences remains un-researched. We compared the experiences of 1,158 singers in a virtual choir (VC) with the experiences of 1,158 singers from a live choir using propensity score matching based on a range of demographic, social and musical factors. Participants in VCs reported a slightly greater feeling of social presence than participants in live choirs [ Over the past decade, there has been a rise in the availability of virtual cultural experiences, from visiting national parks, to exploring archeological sites, touring monuments, interacting with heritage buildings, collaboratively contributing to digital art, and experiencing other cultures . In virtOne virtual activity growing in popularity is singing in a virtual choir (VC). These choirs use the internet as a participatory platform for individuals to get involved in online crowd-sourced music-making . Studiesn = 2,316) of adults.‘Presence’ can be categorized into spatial presence (the feeling of being in a particular location) and social presence (the feeling of being with others) . PresencSpecifically, we focused on the emotional regulation strategies employed when singing rather than the emotional responses themselves. The reason for this was that emotional responses to singing have been shown to be diverse, affected by a broad range of factors including specific features in the music itself and individual responses to that music  . So in tThe Great British Creativity Test: a citizen science project that includes responses from 47,924 adults (age 18+) on engagement in creative activities. Of these participants, 5,775 provided data on experiences of singing in a live choir. To account for differences in baseline characteristics between the two samples, we used propensity score matching to create a propensity-matched cohort of a total of 2,316 people . Of 2,991 adults (age 18+) who enrolled on a larger study, 1,257 provided data on experiences of singing in a VC. We compared the responses of this VC group with data in ople see . For bothttps://www.youtube.com/channel/UCrjq25xbdEiL8jH28FR38Ow. Throughout the process, participants can connect with one another through a bespoke social media platform. For this study, following involvement in the VC, participants were asked to complete an online questionnaire about their experiences. This questionnaire took approximately 10 min to complete. The comparable data about engagement in live choirs came from the Great British Creativity Test: an online Citizen Science project led by the BBC that asks about engagement with creative activities and ERSs when engaging in these activities. This questionnaire took approximately 20 min to complete  which includes a general factor as well as avoidance, approach, and self-development subscales, all scored from 1 to 5 with higher scores indicating greater use of the strategy . AvoidanSocial presence was measured through three items . These items were “When singing in a [virtual/live] choir: I feel a part of something bigger/I feel a sense of connection to other people (even people I don’t know) who are doing the same activity/I feel like I’m part of a community.” The average was taken of the three responses. The item had a Cronbach’s alpha of 0.85 for the entire sample (0.76 for VC participants and 0.86 for live choir participants).B < 25 (B = 4.6), Rubin’s R of 0.5-2 (R = 1.1) and a percentage bias of <10% for each covariate (bias = 0.2–4.0%)  for each individual based on demographic variables , social variables  and singing experience (number of years singing and frequency of singing). We then used nearest available Mahalanobis metric 1-to-1 matching method without replacement, using a caliper size of 0.25 using the Stata module PSMATCH2 . Success.2–4.0%) .t-tests and χ2 tests. For matched data, differences between groups were analyzed using paired t-tests, Wilcoxon signed-ranks test, and McNemar’s test. Pearson’s pairwise correlations were used to explore associations between presence and ERSs. To account for a sense of social presence, we categorized social presence into low (1–3) vs. high (4–5) and analyzed a sub-set of the sample for whom both participants within the pair scored low or high respectively. This provided a reduced sample of 66 pairs in which both partners scored low and 516 pairs in which both partners scored high. We then re-ran analyses of ERSs using this stratified sub-sample. Analyses were carried out using Stata v14 .For unmatched data (prior to the matching procedure), differences between groups  were analyzed using independent Demographic characteristics of participants before and after propensity score matching are shown in SE: 0.02) than participants in live choirs   see .Comparisons in use of ERSs between virtual and live choirs confirmed our hypothesis that participants in VCs made less use of ERSs overall than participants in live choirs see . SpecifiCorrelations support our third hypothesis, showing a strong overall correlation between social presence and use of ERSs, with moderate correlations between social presence and both avoidance and approach strategies, and strong correlations between social presence and use of self-development strategies. Comparable findings were shown in the entire sample and also within subgroups of VC and live choir see .To explore this relationship further, we graphed mean ERS responses by social presence rating . There wTo account for a sense of social presence, we categorized social presence into low (1–3) vs. high (4–5) and analyzed a sub-set of the sample for whom both participants within the pair scored low or high respectively. Sensitivity analyses involving pairs in which both participants reported either low social presence or high social presence showed that differential levels of use of ERSs for overall use, avoidance strategies and approach strategies between virtual and live were maintained. For self-development strategies, amongst pairs who both reported a sense of low social presence, there was no significant difference in use, while for pairs who reported a sense of high social presence, those in VCs in fact reported lower use of self-development strategies. However, it should be noted that the numbers in the ‘low sense of social presence’ group were very small so power may have been affected. Results from these analyses are shown in This study explored differences in perceptions of social presence and the use of ERSs amongst singers in live and VCs. We found that, amongst matched pairs of singers, singing in a VC was associated with a higher overall perception of social presence along with a lower overall use of ERSs, including specifically a lower use of avoidance strategies , a lower use of approach strategies , and a higher use of self-development strategies . Social presence was associated with greater use of all ERSs. We found no evidence that social presence moderated the overall use of ERSs or use of approach and avoidance strategies between virtual and live choirs. But we did find a moderating effect for the use of self-development strategies, which differed depending on how much presence an individual felt whilst singing in a VC.The finding that VCs are associated with a greater sense of social presence is in one sense surprising  given the fact that VCs involve low levels of both interaction and immersion . FurtherOur study also found differences in use of ERSs, partly supporting our original hypothesis that VCs are associated with lower use of overall ERSs and both approach and avoidance strategies. Whether certain ERSs are more or less healthy or adaptive than others has been fiercely debated. Some studies have suggested that avoidance ERSs predict negative emotions and are therefore unhealthy or maladaptive . HoweverHowever, contrary to our hypothesis, we found that singers in VCs make greater use of self-development strategies than singers in live choirs. There are a few possible explanations for this. First, in VCs each individual member of the choir is treated simultaneously as a group member and as a soloist. Instead of a voice merely being lost in the overall group sound, individuals in the Virtual Choir 5.0 could listen to their specific contribution in the process of submission and may therefore have felt more personal achievement. Second, singing in a VC gave individuals the chance to record and re-record their contribution until they were happy with it. This may have led to a greater sense of satisfaction with one’s performance than singing live and therefore increased one’s sense of self-esteem and confidence. Third, the decision to engage with a VC may be partly associated with a desire to improve self-confidence in singing.Finally, we found support for our third hypothesis that there is a correlation between perceived social presence and use of ERSs. This link between presence and ERSs found across our study echoes previous literature , and is spatial aspects of presence have a different relationship with ERSs remains unknown. It should also be noted that our measure of social presence was based on just three individual self-report items. As such it may lack discriminatory rigor in relation to other aspects of presence. Future work assessing presence in more detail using validated measures is therefore recommended. We also did not include a measure of technological capability so could not assess whether those in the VC were more advanced in their use of technology than those in the live choir and therefore whether this could have acted as an additional confounding variable. However, both virtual and live choir respondents had to undertake their surveys online, so a minimum technological ability was demonstrated in both groups. Similarly, we have evidence that those who sang in the VC had also sung in live choirs as over half of participants singing since before the internet was invented, thereby showing that these participants have experience of singing in live choirs too. While we matched participants closely on a range of demographic, social and musical variables, we did not compare the response to live vs. VCs within individuals. This remains a possible extension of the findings here for future studies. Additionally, this study focused on participants in a range of live choirs, but just one VC program: Virtual Choir 5.0. How VCs involving different combinations of immersion and interaction affect presence and ERSs remains to be explore. Finally, determining causality is challenging. Although we matched participants on a wide range of background demographic, social and musical factors, it remains possible that unmeasured characteristics served as confounding variables. For example, those who selected to join a VC may have been those who have had particularly strong emotional responses to live choirs. Similarly, it remains unknown how previous perceived benefits of singing for emotion regulation might feed into the motivation to engage in singing either in live or VCs.This study has a number of strengths including its large sample size, its direct comparison of engagement in live and VCs through matched questions across two different datasets, and its use of propensity score matching to help achieve exchangeability between the two groups. However, several limitations remain. First, this study asked people about their experiences in live and VCs so was observational rather than specifically involving experimental manipulation. On the one hand, this means that the study is arguably ecologically stronger than an experiment might have been in that it draws on people’s natural experiences of singing rather than controlled experiences. On the other hand, it remains possible that latent confounding variables that may have been imbalanced between matched pairs could have affected results, so future intervention studies are recommended. Further, in this study, we did not include a measure of physical space, so whether Over the past two decades, many musical experiences have become mediated by digital technology, including the distribution of music online, the generation of new content and the participation in digital musical experiences. However, questions concerning the nature and impact of such digital musical experiences remain broadly unanswered. This study showed that perceived social presence can still be high for virtual musical experiences, even when the technological sophistication involved is modest. Further, although there is a lowered use of ERSs when engaging in VCs, this difference is relatively small. So virtual cultural experiences may still have a role to play in supporting those who cannot engage in live experiences such as people who are socially isolated.University College London Research Ethics Committee. All participants gave informed consent.DF carried out the analyses and drafted the manuscript. Both authors designed the study critically appraised the manuscript and approved it for submission.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The handling Editor declared a shared affiliation, though no other collaboration, with the authors at the time of review.


Public health risks from urban floods are a global concern. A typhoon is a devastating natural hazard that is often accompanied by heavy rainfall and high storm surges and causes serious floods in coastal cities. Affected by the same meteorological systems, typhoons, rainfall, and storm surges are three variables with significant correlations. In the study, the joint risk of rainfall and storm surges during typhoons was investigated based on principal component analysis, copula-based probability analysis, urban flood inundation model, and flood risk model methods. First, a typhoon was characterized by principal component analysis, integrating the maximum sustained wind (MSW), center pressure, and distance between the typhoon center and the study area. Following this, the Gumbel copula was selected as the best-fit copula function for the joint probability distribution of typhoons, rainfall, and storm surges. Finally, the impact of typhoons on the joint risk of rainfall and storm surges was investigated. The results indicate the following: (1) Typhoons can be well quantified by the principal component analysis method. (2) Ignoring the dependence between these flood drivers can inappropriately underestimate the flood risk in coastal regions. (3) The co-occurrence probability of rainfall and storm surges increases by at least 200% during typhoons. Therefore, coastal urban flood management should pay more attention to the joint impact of rainfall and storm surges on flood risk when a typhoon has occurred. (4) The expected annual damage is 0.82 million dollars when there is no typhoon, and it rises to 3.27 million dollars when typhoons have occurred. This indicates that typhoons greatly increase the flood risk in coastal zones. The obtained results may provide a scientific basis for urban flood risk assessment and management in the study area. Typhoons are considered extremely devastating natural hazards worldwide, which have caused enormous property and human losses and severely impacted public health . Under tAt present, the dependence between bivariate variables of typhoons, rainfall, and storm surges has been analyzed. Zheng ,4 employExpected annual damage (EAD) is used as a flood risk model based on the probability function and flood damage function . Flood dIn this study, the joint risk of rainfall and storm surges during typhoons is investigated by integrating principal component analysis, copula function, urban flood inundation model, and flood risk model methods. The main objectives of this study are to find a good statistical description of multivariate joint probability distribution of typhoon–rainfall–storm surge and then to evaluate the impact of typhoons on the joint risk of rainfall and storm surges. This study would provide a theoretical basis for flood risk management in coastal zones and a reference for the investigation of urban floods caused by multiple hazard factors. The remainder of the paper is organized as follows. The study area and data are described in the next section. Haidian Island is located in the northern part of Haikou in Hainan Province, China, and is adjacent to the Qiongzhou strait . BecauseIn this study, daily rainfall data and daily maximum storm surge data during 1974–2012 were collected from the Haikou hydrological station, which are provided by the Haikou Municipal Water Authority. The continuity of the data was checked. The typhoon data are available from the best-track dataset by the Shanghai Typhoon Institute of China Meteorological Administration (CMA). The dataset contains information on each typhoon track every 6 h, including the time, location (latitude and longitude), maximum sustained wind (MSW), and minimum pressure near the typhoon center. The CMA typhoon best-track dataset is included in the International Best Track Archive for Climate Stewardship (IBTrACS) project , which iWhen the distance between the typhoon center and the study area is less than 500 km ,41, the The overall framework of the study is shown in The existing literature has focused on the dependence between typhoons, rainfall, and storm surges used to characterize typhoons by MSW ,10,11. Hn-dimensional observation sample matrix S with p eigenvalues, the principal component can be calculated with the following steps:Principal component analysis is one of the most widely applied tools for simplifying a dataset by reducing multidimensional datasets to a smaller number of dimensions than the original representation without changing the characteristics of the original data . For an S:Step 1. Establish observation sample matrix S to normalized matrix Y:Step 2. Convert sample matrix 1. Normalized matrix for positive indices:2. Normalized matrix for negative indices:Step 3. Calculate the correlation coefficient matrix of the normalized matrix:λ of correlation coefficient matrix R:I is the p by p identity matrix, |·| is the determinant operator, and λ represents the p eigenvalues ranked in decreasing order, Step 4. Calculate eigenvalues Step 5. Choose principal components m principal components mth principal components of the original variables, respectively, which are expressed as:y is the normalized matrix mentioned in Equations (2) and (3), and When the variance cumulative contribution rate of Step 6. Characterize samples by integrating principal components:The sample values can be integrated by the following formulas ,44,45. and three Archimedean copulas  were employed to build joint distributions of typhoon–rainfall, typhoon–storm surge, and rainfall–storm surge. Trivariate Gaussian copula, Gumbel copula, and Student’s t copula were employed to construct the joint distribution of typhoons, rainfall, and storm surges. The above copula functions are presented in T representing the typhoon, H representing rainfall, and Z representing the storm surge, the joint probability of at least one variable  exceeding its extreme values is denoted as The joint probability, co-occurrence probability, and conditional probability of typhoons, rainfall, and storm surges can be calculated by copulas. For  follows ,48:(11)PT, H, and Z all exceeding certain extremes is denoted as The co-occurrence probability of  follows ,48:(12)PH and Z exceed a certain extreme when T has exceeded the extreme is denoted as The conditional probability that PD is the damage caused by a flood of exceedance probability P, m is the number of probability increments, and iD is the average flood damage (mean of piD and Dpi−1) during probability increment ith interval. Flood damages are calculated by unit flood damage and maximum inundation depth. The inundation depth is calculated by PCSWMM, which is introduced in In this study, flood risk is defined as the product of probability and damages . The expIn order to establish the multivariate joint probability of typhoons, rainfall, and storm surges, typhoons should be quantified first. To quantify typhoons, characterization by principal component analysis was performed, integrating MSW, center pressure, and distance between the typhoon center and the study area. The results reveal that, on the basis of eigenvector loadings, the first principal component (PC1) with an eigenvalue of 0.093 is able to explain 59.1% of the total variation, whereas the second principal component (PC2) with an eigenvalue of 0.062 explains 39.3% of the variation, and both PC1 and PC2 explain 98.4% of the total variation . TherefoFrom the score coefficient matrix in T can be quantified by the following equation [From Equations (8) and (9), typhoon equation ,44,45:, while it is the smallest between rainfall and storm surges . The reason may be that rainfall and storm surges are affected by various weather systems, and typhoon is only one factor. Additionally, the angle of approach of a typhoon may also cause the low correlation between rainfall and storm surges . Even though the correlation between variables is low, it can have significant implications for flood risk estimates, and there are still many studies about their joint probability analysis [u was also evaluated, and the introduction of λu is described in the literature in detail [The quantitative correlation among variables was analyzed using Pearson’s correlation coefficient analysis ,6,7,8. Fn detail . As showD are lower than the critical values (D0.01 = 0.067). Furthermore, a comparison of kernel density estimations and empirical distributions of typhoons, rainfall, and storm surges are presented in The nonparametric kernel density estimation ,52 was u∪ is nearly two times the univariate occurrence probability of typhoons, rainfall, and storm surges, and flooding would easily occur when either of them exceeds the threshold. However, the flood design standard in China is only determined by univariate analysis , which would highly underestimate the flood risk in coastal regions.The joint probability and co-occurrence probability of typhoons, rainfall, and storm surges are calculated from Equations (11) and (12). From T, rainfall H, and storm surge Z all exceed a certain magnitude. i et al. .rred see . TherefoThe impact of typhoons on flood risk caused by rainfall and storm surges was estimated based on an urban flood inundation model and an expected annual damage model. In this study, expected annual damage (EAD) was then found to express the flood risk, which can be calculated by integrating the flood damage function with the probability function. Flood extent and depth are the most important indicators of flood damage, often denoted as depth-damage curves . HoweverH,Z|T) is much more than P. After calculation, EAD is 0.82 million dollars when there is no typhoon, and it rises to 3.27 million dollars when a typhoon has occurred. This indicates that typhoons greatly increase the flood risk in coastal zones. The main reason is the increase of the occurrence probability of rainfall and storm surges in typhoon conditions.Inundation depth was calculated by PCSWMM. The calculation steps of inundation depth are introduced in detail in In this study, the joint risk of rainfall and storm surges during typhoons was investigated by integrating principal component analysis, copula-based probability analysis, urban flood inundation model, and flood risk model methods. First, principal component analysis was used to quantify typhoons by integrating MSW, center pressure, and distance between the typhoon center and the study area. Following this, the Gumbel copula was found to be a robust and proper function for the joint probability of typhoons, rainfall, and storm surges. Then, the joint probability, co-occurrence probability, and conditional probability were indicated by the Gumbel copula. The results of joint probability indicate that ignoring the dependence between these flood drivers may inappropriately characterize the flood risk in coastal regions and can lead to underestimating it. For conditional probability, the co-occurrence probability of rainfall and storm surges increases by at least 200% during typhoons. Therefore, coastal urban flood management should pay more attention to the joint impact of rainfall and storm surges on flood risk when a typhoon has occurred. Furthermore, when a typhoon has occurred, The expected annual damage (EAD) increases from 0.82 million dollars to 3.27 million dollars. This indicates that typhoons greatly increase the flood risk in coastal cities.These messages are useful for practical design and planning, since all flood hazards, such as typhoons, rainfall, and storm surges, are considered. However, the study also has certain limitations. First, since the main objective of this study is to evaluate the impact of typhoons on the joint risk of rainfall and storm surges, the impact of typhoon size and typhoon conditions on rainfall distribution was not taken into consideration and needs to be explored in our future work. Second, the limited socioeconomic data in the study area restricted monetizing flood damage. Referring to Lian et al. , flood d


With an increase in the number of disciplines contributing to health literacy scholarship, we sought to explore the nature of interdisciplinary research in the field.This study sought to describe disciplines that contribute to health literacy research and to quantify how disciplines draw from and contribute to an interdisciplinary evidence base, as measured by citation networks.We conducted a literature search for health literacy articles published between 1991 and 2015 in four bibliographic databases, producing 6,229 unique bibliographic records. We employed a scientometric tool (CiteSpace [Version 4.4.R1]) to quantify patterns in published health literacy research, including a visual path from cited discipline domains to citing discipline domains.Journal of General Internal Medicine  and Patient Education and Counseling . The most recently cited journal added to the top 10 list of cited journals is the Journal of Health Communication (n = 989). Three main citation paths exist in the health literacy data set. Articles from the domain “medicine, medical, clinical” heavily cite from one domain , whereas articles from the domain “psychology, education, health” cite from two separate domains .The number of health literacy publications increased each year between 1991 and 2015. Two spikes, in 2008 and 2013, correspond to the introduction of additional subject categories, including information science and communication. Two journals have been cited more than 2,000 times—the Health Literacy Research and Practice. 2017;1(4):e182–e191.][Recent spikes in the number of published health literacy articles have been spurred by a greater diversity of disciplines contributing to the evidence base. However, despite the diversity of disciplines, citation paths indicate the presence of a few, self-contained disciplines contributing to most of the literature, suggesting a lack of interdisciplinary research. To address complex and evolving challenges in the health literacy field, interdisciplinary team science, that is, integrating science from across multiple disciplines, should continue to grow. The addition of diverse disciplines conducting health literacy scholarship has spurred recent spikes in the number of publications. However, citation paths suggest that interdisciplinary research can be strengthened. Findings directly align with the increasing emphasis on team science, and support opportunities and resources that incentivize interdisciplinary health literacy research. The study of health literacy has significantly expanded over the past decade. It represents a dynamic area of inquiry that extends to multiple disciplines. Health literacy emerged as a derivative of literacy and early definitions focused on the ability to read and understand medical instructions and health care information . This eaOver time, the definition of health literacy expanded to include the capacity to obtain, process, understand, and apply health information and services required to make informed decisions that allow health-enhancing actions at the individual, social, and environmental levels . This brAlong with this expanding definition came a focus on health literacy by researchers from an increasing number of fields including medicine , public Drawing from the multiple definitions, conceptual dimensions, and disciplines of health literacy scholarship, Sørensen et al.  developeResearch Question 1 (RQ1): What disciplines contribute to the field of health literacy as measured by published literature?Research Question 2 (RQ2): How do disciplines draw from and contribute to an interdisciplinary evidence base as measured by citation networks?To explore whether the published evidence base is reflective of and responsive to an integrated, interdisciplinary approach to health literacy, we examined trends in the health literacy literature over the past 25 years to answer the following research questions:“*health literacy*” or “*test of functional health literacy*” or “*rapid estimate of adult literacy*” or “*rapid estimate of adolescent literacy*” or “*newest vital sign*” or “*health information literacy*” (title/abstract)“1991/01/01 – 2015/12/31” (date of publication)English [language]journal article (publication type) and review (publication type)To provide a quantitative assessment of publication and citation patterns in the health literacy literature, we developed a keyword list to identify published journal articles over the past 25 years. The keyword list was developed through an iterative process and included health literacy and health information literacy, along with the most commonly used health literacy assessment tools. We decided to include the full name of the assessment tools as keywords , as many articles that only used the abbreviations  were not relevant to the study. We employed the wildcard “*” to capture relevant variations of a word, and quotation marks were used to identify phrases. We decided to develop and use our own search query to limit our data set to articles specifically related to health literacy. The final query included the following:We conducted a search in four different bibliographic databases: Scopus, PubMed, CINAHL, and Web of Science. Journal and review articles were regarded as relevant if any of the terms were found in the title, abstract, or keyword fields. The query of bibliographic records between 1991 and 2015 resulted in over 4,000 records from each database: Scopus , PubMed , CINAHL , and Web of Science . Using the article title as the common variable for merging, data from the four databases were merged and duplicates were removed to produce 6,229 unique bibliographic records. For a quality check, 100 of the 6,229 articles were randomly selected and reviewed for study relevance, and all articles were relevant.We employed scientometric approaches to answer the research questions  and to bFor RQ1, we used basic scientometric analysis to provide information on the disciplines that are contributing to the field of health literacy, including the number of published articles, cited journals, and subject categories. Each bibliographic database , assigns subject categories to indexed journals, and articles published in these journals automatically inherit the journals' subject categories. To illuminate significant changes in publication trends, we used a burst detection algorithm to capture sharp increases in subject categories . A subjeFor RQ2, we employed more advanced scientometric analysis, including network cluster analysis and citation path analysis, to better understand the interdisciplinary nature of published health literacy research. Network cluster analysis is a way to quantify how articles cite one another, with clusters forming around commonly and consistently cited article groups. Each node represents a cited article and the size of a node is proportional to its cited frequency. Nodes are grouped in the same lines by a clustering technique called smart local moving . ClusterFigure 1 depicts the number of records over time in the final data set . Health literacy has received increasing attention from the scientific community as measured by the published literature, with large spikes in 2008 and 2013.Table 1 describes the top 10 subject categories by frequency and density in our dataset. Density is defined as frequency divided by the number of years published . Frequency shows a snapshot of the cumulative impact of a subject category, whereas density provides an additional metric measuring the distribution of the impact over time. Among the subject categories that were first recorded in 1995, “public, environmental, and occupational health” received the most assignments  and the largest density (52.9). In general, most subject categories relate to medical and clinical domains. The most recent addition in the top 10 list was “information science and library science,” first appearing in 2003 (n = 232). This subject category has a relatively high density given its first occurrence is the most recent among the leading categories, suggesting that recent health literacy literature has been largely driven by scientific approaches in information and library sciences.Figure 2 describes the top 10 “bursting” subject categories, sorted by the beginning year of the burst. Burst detection is a way to measure the intensity of subject categories during a specific timespan. In this study, social sciences and information sciences have received bursting attention in recent years. Findings from Table 1 and Figure 2 indicate that an early focus in health literacy research was on medical and clinical domains, with little overlap with social science, information science, and communication. Scholars from information and library sciences, communication, education, and science and technology appear to have developed an emerging focus on health literacy in recent years.Table 2 displays the top 10 most frequently cited journals in our dataset, the year it was first cited, and the density of how many times per year a specific journal has been cited, from its first year cited. Results show that two journals have been cited more than 2,000 times—the Journal of General Internal Medicine  and Patient Education and Counseling . These two journals were first cited early in the dataset, in 1995 and 1994, respectively, and they also demonstrate the greatest densities. The most recently cited journal to be added to the top 10 list is the Journal of Health Communication (n = 989), first cited in 2001. This journal also demonstrates a relatively high density (65.9), given its first year of citation was 2001. This finding suggests a recent concentration in publications in health literacy scholarship in the Journal of Health Communication.In addition to subject category, examining what journals have been the most frequently cited can serve as a proxy for what disciplines are contributing to the knowledge base in health literacy research. Figure 3 summarizes the 10 largest clusters in terms of the number of member articles, label extracted from titles and abstracts, and mean year of published citations. The color legend at the top indicates that citations in cooler colors occur closer to 1990 whereas hotter ones occur closer to 2015. Among these clusters, Cluster 4 is the oldest and the label generated from this line of research is functional health literacy, an earlier focus in the field. Considering both the size and recency of member nodes as depicted in the visualization, there are four emerging research themes reflected by Clusters 1, 2, 5, and 6: health literacy, health information, providing high-quality care, and mental health literacy.Figure 4, there are three main citation paths in the health literacy data set. The base map depicts the interconnections of over 10,000 journals and these journals are clustered into regions that represent publications and citation activities at a domain level  from collective clusters of articles, as opposed to article-level citation paths, limiting our ability to analyze trajectories of landmark articles in health literacy. For this study, however, focusing on domain-level trajectories allowed for commentary on the larger field of health literacy, and not on individual articles. Second, our search term list may not have captured the full population of health literacy articles relevant to our study. To minimize this bias, we used an iterative process to develop the keyword list and examined additional articles that were included based on additional search term queries . We determined that the additional articles were not relevant to our study . Finally, we are operating under the assumption that subject categories, journal titles, domains, and citation networks are valid and reliable measures used to capture the true nature of discipline-specific health literacy scholarship. We believe that this is an accurate measure of discipline activity as there is a rich history and tradition of discipline-specific scholarship being published and disseminated in discipline-specific journals.This study is one of few initial assessments of citations in health literacy research. To provide additional insight into differences observed in health literacy citation paths, and to further understand the nature of interdisciplinary research, future work may wish to examine and compare the composition of study teams and authors. For example, research teams publishing in the medical domain may be composed of clinical researchers and clinicians, whereas teams publishing in the education domain may be composed of more diverse disciplines, including clinicians as well as disciplines outside of clinical research, thus leading to more citation paths. Future work may also want to expand the search terms, or look at specific populations (such as adolescents) or topics (such as diabetes) to make recommendations for future research directions.Finally, for scholars in the field who hope to increase interdisciplinary research, our findings support the need for more deliberate and strategic activities to help strengthen opportunities for interdisciplinary health literacy work . Develop


Evidence-to-decision (EtD) frameworks intend to ensure that all criteria of relevance to a health decision are systematically considered. This paper, part of a series commissioned by the WHO, reports on the development of an EtD framework that is rooted in WHO norms and values, reflective of the changing global health landscape, and suitable for a range of interventions and complexity features. We also sought to assess the value of this framework to decision-makers at global and national levels, and to facilitate uptake through suggestions on how to prioritise criteria and methods to collect evidence.In an iterative, principles-based approach, we developed the framework structure from WHO norms and values. Preliminary criteria were derived from key documents and supplemented with comprehensive subcriteria obtained through an overview of systematic reviews of criteria employed in health decision-making. We assessed to what extent the framework can accommodate features of complexity, and conducted key informant interviews among WHO guideline developers. Suggestions on methods were drawn from the literature and expert consultation.balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations—and the meta-criterion quality of evidence. It is intended to facilitate a structured process of reflection and discussion in a problem-specific and context-specific manner from the start of a guideline development or other health decision-making process. For each criterion, the framework offers a definition, subcriteria and example questions; it also suggests relevant primary research and evidence synthesis methods and approaches to assessing quality of evidence.The new WHO-INTEGRATE (INTEGRATe Evidence) framework comprises six substantive criteria—The framework is deliberately labelled version 1.0. We expect further modifications based on focus group discussions in four countries, example applications and input across concerned disciplines. Evidence-to-decision (EtD) frameworks help to ensure that all criteria of relevance in a given guideline development or other health decision-making process are considered in a systematic way.The WHO-INTEGRATE (INTEGRATe Evidence) framework is a new EtD framework that is rooted in the norms and values of the WHO, which are agreed on by all WHO Member States.The framework was developed to be applicable to all health interventions, although it is particularly well suited for decisions about population-level and system-level interventions at the global as well as national levels.quality of evidence; example questions and suggested methods are provided to facilitate uptake.The WHO-INTEGRATE framework offers structured definitions for each of the six substantive criteria as well as the meta-criterion As part of a more holistic approach, the framework is devised as a tool to facilitate structured reflection and discussion from the beginning to the completion of a guideline development or other health decision-making process; this entails prioritisation among criteria and subcriteria to ensure appropriate evidence collection and appraisal.Health decision-making at local, national, regional and global levels is complex,20 21quality of evidence (in relation to intervention benefits and harms), values and preferences (in relation to outcomes), balance of benefits and harms, resource implications, priority of the problem, equity and human rights, acceptability, and feasibility (table 10.1 of the WHO Handbook for Guideline Development).22Guidelines by the WHO provide recommendations for clinical practice, public health and health system strengthening, and are intended to support health decision-makers in prioritising among or selecting suitable clinical, public health or health system interventions. When formulating recommendations, WHO generally uses an EtD framework which encompasses eight criteria: WHO Handbook for Guideline Development et al, forthcoming). In particular, this framework can be applied across diverse types of health decisions and was developed following an iterative and multipronged process, combining a literature review, brainstorming and feedback from stakeholders,20 25Chapter 10 in the et al, forthcoming). First, the framework was developed using a pragmatic approach and lacks an explicit theoretical or conceptual basis. This makes it difficult to assess objectively whether the set of criteria is complete and organised in a meaningful way.However, a number of weaknesses were identified with the GRADE EtD frameworks . While there are many potential reasons for this, the current content and structure of the GRADE EtD framework may result in superficial use rather than indepth collection and assessment of evidence for the different criteria. In particular, guidance on how to frame questions for and collect evidence towards criteria beyond balance of health benefits and harms appears to be missing.A final and important concern relates to consistency in the application of the GRADE EtD frameworks within the WHO guideline development processes. While there are exemplar guidelines, where the WHO EtD framework has been employed as intended,This paper, one of a series exploring the implications of complexity in systematic reviews and guideline development, reports on the development of a new EtD framework that is rooted in WHO norms and values and suitable for a broad range of health interventions, including complex interventions and interventions delivered in complex systems.Develop an EtD framework that (a) is firmly rooted in WHO norms and values and reflective of the changing global health landscape, and (b) encompasses a comprehensive set of criteria suitable for clinical practice, public health and health system interventions.Explore the value of this framework in relation to (a) complexity in individual-level as well as population-level and system-level interventions, (b) the views of developers of the WHO guidelines , and (c) the views of users of the WHO guidelines .Facilitate uptake of the framework by emphasising the need for structured, evidence-based reflection and suggesting methods to populate the criteria with evidence in the context of decision-making under uncertainty.The paper addresses the following three objectives:The EtD framework developed out of this process is referred to as the WHO-INTEGRATE (INTEGRATe Evidence) framework version 1.0. It is proposed for use in the WHO guideline development as well as in other guideline development or health decision-making processes at the global or national level. It is intended to be used holistically—from the beginning of a health decision-making process to formulating recommendations or making a decision at the end of this process.In addressing these objectives, we followed a three-step approach, as illustrated in et al, forthcoming). An integrated analysis of the views of those developing (step 2b) and using WHO guidelines (step 2c) with respect to the WHO-INTEGRATE framework will also be published separately .This paper provides an overview of the research project with all of its constituent components. It presents the current version of the framework (WHO-INTEGRATE framework version 1.0) and its intended use. It also reports the detailed methods and findings for steps 1a, 2a and 3, as well as an overview of the methods and findings for steps 1b and 2b. A full account of the methods and findings of step 1b is currently in preparation .WHO Handbook for Guideline Development,WHO Handbook for Guideline Development)43 44 46 56During the development process, we focused on substantive criteria or what decisions are based on  rather than procedural criteria or how the decision-making process is conducted . This is consistent with the approach promoted by the In step 1b we refined the preliminary criteria derived from WHO and other related documents and supplemented them with a comprehensive set of subcriteria; we also developed definitions for criteria and example questions relating to each of the subcriteria. We conducted an overview of systematic reviews of criteria used in decision-making, priority setting and resource allocation processes for health to derive a comprehensive set of health-relevant criteria . We then compared the preliminary criteria developed in step 1a against this comprehensive set of criteria and subcriteria. To do so, one author (JMS), in a discussion with a second author (EAR), allotted the subcriteria obtained from the overview of systematic reviews to the preliminary criteria within the WHO-INTEGRATE framework. Subcriteria that did not fit were kept in a separate category. Any uncertainties were resolved in discussion with a third author (RB).et al, forthcoming), and any definitions or descriptions provided in the publications included in our overview of systematic reviews of criteria . Where appropriate, we also drew on additional key documents . Each definition (1) provides an overall definition of the criterion, (2) offers details and explanations regarding the subcriteria, and (3) gives guidance on how the criterion in question influences the recommendation.We then prepared definitions for each of the criteria using the above-described source documents for health norms and values, existing health decision frameworks  may be employed to describe different underlying concepts, relating to process versus outcome  and the point in time when these criteria apply . To enable better access to sometimes abstract constructs, we also developed example questions for each of the subcriteria, drawing on the same set of documents as above.As we prepared definitions, we also examined the extent to which the criteria and subcriteria relate to the intervention itself In step 2a we explored whether the WHO-INTEGRATE framework would be able to accommodate different types of health interventions and different features of complexity. We assessed to what extent the WHO-INTEGRATE framework would be able to accommodate features of distinct types of health interventions.In step 2b we examined the usefulness and relevance of the WHO-INTEGRATE framework and its criteria among those developing WHO guidelines. We conducted key informant interviews with individuals who had recently participated in a WHO guideline development process. In consultation with the Secretariat of the WHO Guidelines Review Committee, we purposively selected three ongoing or completed guidelines that had applied the current WHO EtD framework,We critically examined how to enable use of the WHO-INTEGRATE framework as intended, and generated a table linking the criteria with suggested methods for primary research, evidence synthesis and assessing quality of evidence. The current WHO EtD framework is intended to be used right from the planning stages of a guideline, to help derive relevant questions and structure the process, but in practice it is usually used at the end of a guideline process to help decide on the recommendations. To determine how the new framework could be used more holistically, we reflected on the literature reviewed in the context of developing the WHO-INTEGRATE framework and sought feedback from a large number of experts (see Acknowledgements). We specifically sought suggestions on how to use the framework during the early stages of the guideline development process and in a context-specific manner.et al, forthcoming). We also consulted a broad range of experts comprising other authors of papers in this series, selected guideline development organisations  and researchers with an interest in evaluating complex health technologies (see Acknowledgements).To make it easier for guideline panels to populate the criteria in the framework with evidence, we identified types of primary research, evidence synthesis methods and methods for assessing evidence quality that could inform each criterion. To accomplish this, two researchers (AP and EAR) reviewed the research questions and methods described or mentioned in the systematic review of health decision frameworks and the overview of systematic reviews .ethical principles primarily according to the public health ethics framework of Childress and colleagues. This framework inter alia includes the aspects of producing benefits, avoiding harms, maximising the balance between benefits and harms, as well as distributive justice and autonomy.Given the large number of biomedical and public health ethics frameworks,sustainability elements to capture the wide range of factors that promote conditions in which people can lead a healthy life and allow societies and individuals to develop and flourish; these sustainability elements also reflect the societal impact that interventions can have beyond health outcomes. Importantly, good health is both a precondition for achieving sustainable development and an outcome of sustainable development.65Acknowledging the importance of the social determinants of health and the SDGs, we derived goals  and six building blocks .19 38–40To capture the importance of feasibility of implementation as well as the impact of interventions on the health system, we used the WHO health systems framework with its four balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations. A seventh criterion, quality of evidence, represents a metacriterion that applies to each of the six substantive criteria. All seven criteria are relevant to health decision-making and the formulation of recommendations as part of the guideline development process. Each criterion may apply to interventions targeting individuals, populations or systems, or any combination of these levels.et al et al balance of health benefits and harms, and large cost of disease to health system features under financial and economic considerations.While priority of the problem featured in both the health decision frameworks included in our systematic review . Recurrent aspects addressed by the subcriteria focused on the health outcomes and benefits of the intervention, health benefit for individuals and the benefit for society as a whole, the societal importance of the disease, economic considerations, quality or uncertainty of evidence, as well as population priorities, priorities within the health system and stakeholders’ interests and pressures. Feasibility criteria were concerned with the available budget, the capacities within the health system, technological complexity and acceptability of the intervention within society. Some systematic reviews were primarily concerned with interventions that would benefit vulnerable or marginalised populations . In many reviews, normative criteria such as ethics, justice or fairness were mentioned without clear definitions or contextualisation. This comprehensive list did not yield any further criteria beyond the seven presented in Our overview of systematic reviews yielded more than 30 systematic reviews that contained several thousand criteria and subcriteria currently used in decision-making  than individual-level interventions. Some subcriteria may take on a different meaning when applied to individual-level versus population-level and system-level interventions . Broadly speaking, most features of complex interventions apply to both individual-level and population-level/system-level interventions but are more salient for the latter. In contrast, many features of complex systems only apply to population-level and system-level interventions.An earlier paper in this seriesbalance of health benefits and harms , and it may even have unwanted social consequences by stigmatising those unable to afford soft drinks . Raising taxes on only one sugar-sweetened product may lead to increasing the sugar content of other sugar-sweetened products  or have implications on agricultural production patterns nationally and internationally , illustrating the complexity of downstream implications of a ‘simple’ intervention. Drawing on the same worked example, Box 1balance of health benefits and harms would warrant the most attention. A complexity perspective on the same intervention would not start off with a preconception about a single criterion being most influential but carefully examine all criteria. For illustration purposes, this complexity perspective would examine acceptability among and likely reactions from different groups of stakeholders , and pay specific attention to the response from vendors and producers of soft drinks , which may dampen the expected effect of the tax intervention, in terms of changes in consumption patterns, perceptions of the intervention and changes in social norms.A simple perspective on raising taxes on soft drinks would emphasise the linear impacts of this intervention on consumption of sugar-sweetened beverages (intermediate outcome) and different measures of childhood obesity (ultimate outcome of interest); with this perspective, the criterion This complexity perspective would also encompass potential negative impacts on health equity, equality and non-discrimination , explore positive or negative social, environmental or economic impacts , adopt a societal perspective in estimating the financial and economic impacts of the intervention , and pay attention to feasibility and health system considerations .The last column of et al, forthcoming).The key informants we interviewed had been involved in developing three very different guidelines—the WHO recommendations on antenatal care,10.1136/bmjgh-2018-000844.supp1Supplementary file 1human rights and sociocultural acceptability and health equity, equality and non-discrimination. Two participants questioned the added value of the new EtD framework, since any guideline development process led by an experienced methodologist would automatically address the details covered in the subcriteria. Several participants were concerned about the workload that the use of the WHO-INTEGRATE framework might add to the guideline development process.Most participants commented positively on the WHO-INTEGRATE framework and highlighted the value of a criterion assessing societal implications, as well as the broader and more detailed specification of the criteria Specific remarks were made in regard to (1) missing criteria and subcriteria; (2) the hierarchy and order of criteria and subcriteria; (3) overlap and redundancies between criteria and subcriteria; (4) the precise wording and definitions of criteria; (5) the need for (more) guidance on how to use and interpret criteria and subcriteria; (6) the challenges of identifying and synthesising the required evidence; (7) resource, time and skill implications for the guideline development process; as well as (8) procedural aspects for using the framework in the guideline development process.In response to these concerns and suggestions, we made several modifications, including changing the name and definition of several criteria and subcriteria to improve clarity and reduce overlap. We also expanded the example questions for the subcriteria to improve understandability and facilitate the development of specific questions for a given guideline. Moreover, we added suggestions on how to prioritise among criteria and subcriteria in a problem-specific and context-specific manner. Finally, we emphasised the importance of incorporating the voices of those directly affected by the recommendations into the guideline development process.not intended as a ‘tick-box exercise’; there must be prioritisation of the most relevant criteria and subcriteria depending on the questions addressed by a given guideline, and the time and resources at disposition. It would be impossible and probably unnecessary for every guideline development or health decision-making process to examine all subcriteria. This flexibility can, however, lead to misuse, as stakeholders may disproportionately  or unduly  influence the decision-making process. Safeguards can be put in place through explicit procedures, in particular in relation to the composition of guideline panels or other decision-making groups. The WHO-INTEGRATE framework is also not an algorithm for integrating evidence across different criteria: making decisions under uncertainty and agreeing on trade-offs across criteria and subcriteria and among (and within) diverse stakeholder groups remain a core task for a guideline panel.The WHO-INTEGRATE framework is intended to improve transparency in health decision-making by supporting a structured process of reflection and discussion in a problem-specific and context-specific manner. To be most effective, this process must begin at the start of a guideline or other health decision-making process and must take evidence into account. The WHO-INTEGRATE framework is All criteria are important and should be reflected on, but their relevance varies depending on the type of health decision and the decision-making context. In contrast, not all subcriteria are always relevant. At the start of a guideline or other decision-making process, an appropriately composed guideline panel or other decision-making group needs to discuss which of the subcriteria are applicable and useful in relation to the nature and specific characteristics of the intervention see ; this grA systematic weakness in many guideline development and other health decision-making processes is that consumer participation is obviated and guideline panels often substitute their own values and views for those of patients/beneficiaries. The voices of patients/beneficiaries and other relevant stakeholder groups can be incorporated through direct participation or representative surveysThe guideline panel will also need to decide how best to populate the criteria with evidence and whether a formal evidence synthesis or a more pragmatic approach is warranted for each. This decision will be influenced by the relevance of criteria and subcriteria in relation to a specific intervention or decision, and by the likely types and quantity of evidence available, as well as time and resource constraints. At the end of the process, the guideline panel will need to reassess the criteria and relevant subcriteria in light of the assembled evidence and make a judgement regarding each criterion.et al, forthcoming) did not offer insights for operationalising frameworks, for example by specifying research questions or suggesting methods for primary research or evidence synthesis. The GRADE EtD frameworkThe WHO-INTEGRATE framework represents a new comprehensive EtD framework that is rooted in WHO norms and values. It offers an explicit conceptualisation of each criterion and a rationale for including relevant concepts as criteria or subcriteria. The WHO norms and values apply across all WHO Member States and settings, and the new framework should, in principle, be relevant for health decision-making at global, national and subnational levels. It reflects a broad understanding of health and its determinants and takes account of complex interventions and complex systems perspectives. It emphasises sustainability and the interconnectedness between health and other sectors, inherent in the SDGs. While the framework is conceived for individual-level, population-level and system-level interventions, it is likely to be particularly well suited for public health and health system interventions characterised by complexity and/or approached from a complexity perspective. The WHO-INTEGRATE framework is intended as a tool to facilitate structured reflection and discussions from the beginning of a guideline development or other health decision-making process. This has ramifications in terms of the need to prioritise among criteria and subcriteria and the need to collect evidence for each. The framework supports this process by offering structured definitions for each criterion and example questions for each subcriterion, and by suggesting methods for primary research, evidence synthesis and assessing the quality of the evidence.balance of health benefits and harms). In contrast, criteria with a strong normative foundation  were much less developed in the GRADE EtD framework; notably, the criterion societal implications, which has its roots in the recognition of the multisectoral determinants of health, is absent from the GRADE EtD framework. There are also more fundamental differences. While the GRADE EtD framework emphasises the efficacy/effectiveness of interventions and their potential harmful impacts, there is no inherent weighting of criteria in the WHO-INTEGRATE framework: guideline panels must decide in a context-specific and problem-specific manner which criteria and subcriteria are most relevant. Moreover, in contrast to the narrower certainty of evidence concept in the GRADE EtD framework, the WHO-INTEGRATE framework has deliberately adopted a broad quality of evidence concept that applies across all criteria and is not linked to a prespecified grading system. For several criteria (and/or subcriteria) GRADEThere are many similarities between the WHO-INTEGRATE framework and the widely used GRADE EtD framework. As stated in our methods, we deliberately attempted to stay as close as possible to the GRADE EtD framework, thus building on established terms and concepts  were derived. While there is some conceptual overlap at the level of the criteria , there are no significant redundancies among the subcriteria . Cross-linkages among the criteria are emphasised in the definitions and example questions.In developing the WHO-INTEGRATE framework, we combined a principles-based approach with an overview of systematic reviews of decision criteria and thus ensured a solid, comprehensive normative foundation. We were explicit and transparent as to how criteria see  and subcnot comprise procedural criteria but is intended to be embedded in a clearly specified health decision-making process as described, for example, in the WHO Handbook for Guideline Development.et al, forthcoming) and suggest that these be reviewed separately to inform guideline development and other health decision processes.80–82Solely adapting the substantive criteria may be insufficient to overcome limitations in guideline development or other decision-making processes.human rights and sociocultural acceptability) or absent from the literature , merits research to unpack them and, where applicable, provide a more detailed normative justification. We anticipate constructive input from and exchange with relevant disciplines, in particular public health ethics but also sociology, environmental sciences, economics and many others. Future collaborative research is expected to lead to a WHO-INTEGRATE framework version 2.0. This may advance the criteria and subcriteria and their normative foundations, as well as methodological approaches to populate these criteria with evidence.The WHO-INTEGRATE framework is a highly interdisciplinary framework: each criterion, especially those criteria that are less developed in current EtD frameworks .To examine the value of the WHO-INTEGRATE framework to potential users, we conducted empirical qualitative research. Insights from interviews with key informants in relation to their recent experiences with developing WHO guidelines led to several refinements in the wording of the criteria and subcriteria and highlighted the importance of providing example questions as well as suggested methods. We expect that the second empirical qualitative research component, focus group discussions in Nepal, Uganda, Germany and Brazil, will yield additional insights from different perspectives and possibly further modifications to the framework. An integrated analysis of the views of WHO guideline developers and users will be published separately (Stratil Several of our key informants expressed concern about the potential workload resulting from collecting evidence for each of the criteria and, in particular, for the many subcriteria in the WHO-INTEGRATE framework. Both the process of prioritisation and the process of collecting evidence—through high-quality evidence synthesis or more pragmatic approaches—need to be tested in practice. We anticipate sharing worked examples and developing additional guidance on how to implement the framework in practice.The WHO-INTEGRATE framework represents a comprehensive EtD framework rooted in WHO norms and values that is, in principle, suitable for individual-level, population-level and system-level health interventions that may or may not be characterised by complexity. It offers structured definitions for each of the six substantive criteria as well as the meta-criterion quality of evidence; example questions and suggested methods are provided to facilitate uptake. Importantly, this framework is intended to be used from the beginning and throughout a guideline or other health decision-making process, whether this process takes place at the global, national or subnational level. In working towards version 2.0, we welcome learning from the experiences of those applying the framework, as well as from researchers in disciplines concerned with the included criteria or subcriteria.


Evidence-informed practice is fundamental to the delivery of high quality health care. Delays and gaps in the translation of research into practice can impact negatively on patient care. Previous studies have reported that problems facing health care professionals such as information overload, underdeveloped critical appraisal skills, lack of time and other individual, organisational and system-level contextual factors are barriers to the uptake of evidence. Health services research in this area has been restricted largely to the evaluation of program outcomes. This paper aims to describe the implementation process of an educational initiative for health care professionals working in midwifery, neonatology or obstetrics aimed at disseminating evidence and enhancing evidence-informed clinical care.We designed and implemented an educational initiative called Evidence Rounds for health care professionals working in the women and children’s division of an urban hospital in Ireland. It consisted of three core components: (1) group educational sessions examining evidence on topics chosen by staff (2) a dedicated website and (3) facilitation, enablement and support from a knowledge translation professional. We evaluated user engagement in the educational program by monitoring attendance figures and website analytics. We followed up with staff at 3, 16 and 21-month intervals after the last educational session to find out whether evidence had been implemented. We use Lavis’s organising framework for knowledge transfer and the Template for Intervention Description and Replication (TIDieR) checklist to describe the educational program and document the implementation process.Six educational sessions presented by 18 health care professionals took place over a nine month period with 148 attendances of which 85 were unique . During the period spanning from one month before, during and one month after the running of the group sessions, 188 unique visitors, 331 visits and 862 page views were recorded on our website.Audit and feedback processes can provide quantitative data to track practice outcomes. Achieving sustainable educational programs can be challenging without dedicated resources such as staffing and funding.The online version of this article (10.1186/s12909-019-1489-y) contains supplementary material, which is available to authorized users. Evidence-informed practice is central to the delivery of quality care and is associated with improvements in patient outcomes. Emparanza and colleagues  demonstrThere are multiple barriers to evidence uptake and evidence-informed practice. Information overload –5 is a bThere is a need for evidence-informed, theory-based educational and knowledge translation initiatives aimed at HCPs to promote evidence-informed practice and the implementation of evidence where appropriate. There is also a knowledge gap regarding the implementation process of these type of initiatives. One of the most frequently used theories in research looking at the adoption of evidence based practice and implementation science is Everett Rogers’ diffusion of innovations theory 1983). Rogers identified four key elements instrumental to the adoption of an innovation; the innovation itself, communication channels (hereby referred to as modes of delivery), time, and the social system. He categorised stakeholders into five groups according to how they adopt innovations over time; innovators, early adopters, early majority, late majority and laggards [83. RogerGrimshaw and colleagues , highligInformed by this evidence, Evidence Rounds featured a multifaceted strategy centred around educational meetings and focused on increasing the reach of evidence and the motivation and ability to use and apply evidence. We also took an interprofessional approach, by involving multiple professions  and working with opinion leaders. We designed the initiative to address individual and organizational level factors and adapted it when necessary throughout the implementation process. We arranged for a local catering service to provide food at each session. Our description of the implementation of Evidence Rounds adds to the literature on educational initiatives in applied health services research. There is a general paucity in the existing literature of studies that provide insight into how contextual factors have influenced dissemination and implementation efforts.Evidence Rounds was based loosely on an intervention conceived by Jacqui Le May, former Head of Knowledge Services at University Hospitals Coventry and Warwickshire, NHS Trust in the United Kingdom (UK). There, members of the Clinical Evidence Based Information Service (CEBIS) team run Evidence in Practice Groups to examine evidence in various departments within the hospital. Topics and questions are linked to a specific patient cases, series of patient cases or other general topics. As well as incorporating the best available evidence into our group sessions, we used evidence from key findings of systematic reviews and other research to inform the design and implementation of the initiative.The goal of Evidence Rounds was to bridge the gap between evidence and practice through an educational initiative aimed at HCPs. The objectives were to disseminate the best available evidence to HCPs on topics of their choosing during group sessions; to promote evidence-informed practice through the provision of an in-person group platform for staff to discuss the implications of the evidence, the barriers and facilitators to its implementation and, to enhance evidence-informed practice by identifying and assigning resulting actions where appropriate.The aims of this paper are to describe the process of planning, designing and implementing this multi-component educational initiative, to report data on quantitative performance indicators monitoring engagement during the implementation process and to provide follow up information regarding the implementation or lack of implementation of the evidence. The second paper in this two-part series reports the findings of focus groups and interviews about Evidence Rounds with HCPs who attended or presented at the group educational sessions .In Fig. What should be transferred to decision makers? To improve the likelihood of evidence uptake, HCPs were invited to select topics or clinical questions relating to treatment or diagnostic interventions. A member of staff who later confirmed with colleagues their agreement on her chosen topic suggested the topic for the first group session at a planning meeting. For subsequent sessions, a collective decision was made at group sessions about the topic to be covered in the next session. Sometimes, several suggestions were considered before a decision was made. At the request of one HCP, a topic suggestion sheet was passed around during sessions to accommodate staff who were reluctant to propose topics in front of their colleagues. HCPs were asked to submit suggestions based on gaps they perceived in their knowledge of the evidence or where there was a perceived gap between the evidence and their own practice. Topics were not limited to those known to have clear and conclusive evidence and suggestions covering controversial treatments, those that had conflicting evidence findings, or a lack of evidence, were encouraged. Our aim was to transfer the best available, most up to date, relevant and applicable evidence. A list of sub-questions or topics for each educational session is featured in Table To whom should research knowledge be transferred? Our target audience consisted of HCPs working in the neonatal and obstetric departments in the women and children’s division of an urban hospital in Ireland. We took a multi-disciplinary and interprofessional approach to maximise the potential for the dissemination and implementation of evidence and to promote collaboration with the ultimate goal of implementation of evidence where appropriate. We also invited staff members outside of key departments when deemed appropriate to the topic. For example, laboratory staff were invited to attend the fourth session: antenatal screening for group B streptococcus. When these staff were identified, invitations were extended through the presenting HCPs. The implementation team also invited students who were on placement in the departments during the time of the sessions.By whom should research knowledge be transferred? We took a team approach to the transfer of knowledge. Three HCPs presented at each session with representatives from both medical and nursing and midwifery staff in each session. Staff from the neonatal and obstetric departments presented when the topic covered both disciplines. To recruit HCPs to present, staff were asked to volunteer during group sessions or previous presenters contacted individuals they perceived as suitable candidates. The KT professional who is an author on this paper (AC) introduced each session, discussed the literature search process, the breadth of the literature on the chosen topic, and directed discussion to decide on the next topic.How should research knowledge be transferred? The KT strategy involved both active and passive methods of promotion, communication and dissemination. In line with Rogers’ diffusion of innovations theory [P values and confidence intervals, identifying appropriate critical appraisal tools, sourcing images to put into presentations (in compliance with licensing and copyright restrictions), providing feedback on presentation slides, populating reference sections, extracting key information and data, providing guidance on selecting papers for inclusion etc. During the discussion forum, obstacles to the implementation of evidence were identified to increase the likelihood that they would be addressed and plans for change could be tailored [s theory , we acces theory  who may tailored .The organising framework for knowledge transfer strategies conceived by Lavis et al.  was used5)With what effect should research knowledge be transferred? The main aims of Evidence Rounds were to provide an educational program that disseminated evidence to health care professionals and promoted evidence-informed practice. We undertook process evaluation by capturing and monitoring data for key indicators throughout the initiative. Firstly, we distributed sign-in sheets at group sessions to record attendance figures. We wanted to track neonatal and obstetric staff attendances and identify potential patterns. Secondly, we monitored usage analytics on our dedicated website. Both informed us of the penetration of Evidence Rounds to the HCP community within the department. Thirdly, our focus groups and interviews provided self-reported data on how the HCPs were receiving the educational initiative and how they viewed it in relation to their own evidence-informed practice. Using this data, we identified individual, organisational and intervention level barriers and facilitators to attending and presenting at Evidence Rounds. We were better able to understand the complexity of the behaviours and gauge opinions on whether and how Evidence Rounds was promoting evidence-informed practice for them. These results are published in the second paper of this two part series [t series . Fourthlt series , 33. Fort series . WritingAt the initial planning meetings, we emphasized that we did not intend on imposing the Evidence in Practice Groups model from the UK on staff at our hospital. Baumann recommends taking an adaptive approach to implementation because no single intervention will be a perfect fit in all settings . ProactiWe took measures to plan for sustainability  such as developing tools that could be handed over easily. For example, we chose a web hosting platform that allowed us to build the website and create content using high quality templates without the need for coding or programming skills. Our choice was deemed the most likely option to promote sustainability because at the end of the period of support from the KT specialist, it could easily been handed over to a HCP lacking advanced technical skills of website design and administration/maintenance. We also linked in with library staff to confirm that they would be willing to design and conduct future searches, had conversations with key people, discussed it during our focus groups and interviews and offered guidance during a handover period. We planned to assess sustainability by following up with the implementation team to find out whether the educational initiative had continued to be delivered.We employed the Template for Intervention Description and Replication (TIDieR) checklist, to complement the reporting of the initiative . This rewebsite analytics captured by our website hosting platform. We report figures spanning the period from one month before the first group session, during the group sessions and one month after the last group session:unique visitors defined as the number of visitors visiting for the first timevisits defined as the number of browsing sessions and can involve multiple page viewspage views defined as the number of times a webpage from our website was fully loaded by a browserthe total number of HCPs and other attendees who attended each Evidence Round session the total number of HCPs who presented at an Evidence Rounds session.We collected and report a number of quantitative measures:We contacted the 5 HCP members of the implementation team three, 16 and 21 months after the initiative ended to find out whether Evidence Rounds had led to the implementation of research findings.Six Evidence Rounds group sessions were run over a 9-month period. There was a total of 148 attendees of which 85 were unique . See Table Seventeen HCPs who worked at the hospital presented during the period of implementation. One external HCP , was asked to present at a session because he authored two relevant papers that were selected for inclusion in the presentation (session number 6).Between 01/06/2016 and 29/04/2017, 188 unique visitors, 331 visits and 862 page views were recorded on the website. See Fig. In Table Follow up with the implementation team also confirmed that the educational program was not sustained beyond the period of support from the KT Specialist.We would like to acknowledge that our study has several limitations. Firstly, the six educational sessions were carried out over nine months. It is unlikely that this was a sufficient duration of implementation to allow for the initiative to realise its full potential, become fully integrated or adopted by staff that Rogers  might deIn terms of implications for practice, the issue of sustainability is important to consider. Despite the steps we describe in the Methods section aimed at increasing the sustainability of the initiative, it was not sustained beyond the period of support from the KT professional. Without a nominated person or team with dedicated professional hours and taking into consideration the time spent planning and developing, we were aware that there was reduced potential to sustain the initiative at our busy hospital setting. Ideally, future initiatives will have a longer period of implementation to allow for appropriate capacity building and so that they have a better chance of integration and becoming accepted and adopted by staff.laggards. Involving an Information Specialist or Librarian or someone who has knowledge of appropriate databases and other online resources and is experienced in carrying out systematic and detailed literature searches is essential. They can help to address issues of information overload and reduce the workload of HCPs involved in presenting.A key learning point for us has been that initiatives like Evidence Rounds are only as strong as the people involved. We recommend collaboration and partnership with the target audience starting from the planning stages and continuing throughout. The multi-disciplinary and interprofessional approach worked very successfully for Evidence Rounds and according to informal feedback and our focus group and interview data  it was hAdaptation and adherence to a small number of core components was a fundamental of the initiative. Baker et al. , found tFurther studies are required to assess the effectiveness of Evidence Rounds, similar educational initiatives including those implemented in settings in the developing world. Evaluation could include pre and post-testing of knowledge of topics the initiative addressed, impact on HCP behaviour and patient care outcomes. More studies are needed to better understand and identify additional underlying mechanisms and contextual factors that influence educational programs. Additional research is also needed to understand how a social media strategy might be optimised for use in the delivery of similar initiatives.Evidence Rounds presents a novel educational initiative to support a knowledge translation strategy targeted at HCPs. It moves beyond the journal club model that was familiar to our target audience. It was designed and implemented based on feedback obtained by proactively engaging with staff. We have helped address the need for more research that provides a detailed account of the implementation of knowledge translation strategies , 22. We Additional file 1:Quick guide for presenters (DOCX 59 kb)Additional file 2:Sample poster promoting Evidence Rounds (PDF 8151 kb)Additional file 3:Sample certificate of attendance (PDF 103 kb)Additional file 4:Sample certificate of participation (PDF 103 kb)