As health resources and services are increasingly delivered through digital platforms, eHealth literacy is becoming a set of essential capabilities to improve consumer health in the digital era. To understand eHealth literacy needs, a meaningful measure is required. Strong initial evidence for the reliability and construct validity of inferences drawn from the eHealth Literacy Questionnaire (eHLQ) was obtained during its development in Denmark, but validity testing for varying purposes is an ongoing and cumulative process.This study aims to examine validity evidence based on relations to other variables—using data collected with the known-groups approach—to further explore if the eHLQ is a robust tool to understand eHealth literacy needs in different contexts. A priori hypotheses are set for the expected score differences among age, sex, education, and information and communication technology (ICT) use for each of the 7 eHealth literacy constructs represented by the 7 eHLQ scales.A Bayesian mediated multiple indicators multiple causes model approach was used to simultaneously identify group differences and test measurement invariance through differential item functioning across the groups, with ICT use as a mediator. A sample size of 500 participants was estimated. Data were collected at 3 diverse health sites in Australia.3. Ability to actively engage with digital services having the strongest effect , followed by 1. Using technology to process health information , 5. Motivated to engage with digital services , and 7. Digital services that suit individual needs . However, the effects were only partially mediated by ICT use. Higher education was associated with higher scores in 1. Using technology to process health information  and 3. Ability to actively engage with digital services , with the effects mostly mediated by ICT use. Higher ICT use was related to higher scores in all scales except 2. Understanding health concepts and language and 4. Feel safe and in control. Either no or ignorable cases of differential item functioning were found across the 4 groups.Responses from 525 participants were included for analysis. Being older was significantly related to lower scores in 4 eHLQ scales, with By using a Bayesian mediated multiple indicators multiple causes model, this study provides supportive validity evidence for the eHLQ based on relations to other variables as well as established evidence regarding internal structure related to measurement invariance across the groups for the 7 scales in the Australian community health context. This study also demonstrates that the eHLQ can be used to gain valuable insights into people’s eHealth literacy needs to help optimize access and use of digital health and promote health equity. As heTo understand eHealth literacy needs, a useful and valid measurement of eHealth literacy is needed. With the introduction of this concept in 2006, the study by Norman and Skinner  developeUsing a grounded validity-driven approach , the stuUsing technology to process health informationUnderstanding of health concepts and languageAbility to actively engage with digital servicesFeel safe and in controlMotivated to engage with digital servicesAccess to digital services that workDigital services that suit individual needs Each eHLQ scale has 4 to 6 items relating to a 4-point ordinal scale, ranging from strongly disagree to strongly agree. The results are 7 scale scores with a range of 1-4, calculated by averaging the item scores within each scale with equal weighting. Initial validity testing of the eHLQ involved extensive discussion of the test content in the Australian and Danish contexts by an international multidisciplinary team experienced in questionnaire development and cognitive interviewing with community members from different cultural and educational backgrounds to ensure that the items were understood as intended. The items were then administered to 475 Danish participants randomly approached by trained interviewers in the broader-community locations, including libraries, workplaces, hospitals, nursing homes, health centers, and an outpatient clinic. Bayesian confirmatory factor analysis supported the seven-factor model, with all items loading strongly on their relevant factors and no statistically significant cross-loadings. Composite scale reliability (ranging from 0.75 to 0.87) demonstrated good internal consistency. Item response theory analysis confirmed that there were no disordered thresholds, and differential item functioning (DIF) testing established evidence of measurement invariance for age and sex . The eHLStandards for Educational and Psychological Testing (the Standards) [According to the andards) , the autandards) . This stEvidence of relations to other variables refers to an analysis of the relationship between the eHLQ scores and other variables with which the scores are predicted by theory or past research to be associated. The evidence may include association of the scores with certain demographic groups, relationships with predicted outcomes, or relationships between the scores and other external instruments that measure the same construct ,28. As tTo evaluate known-groups validity, hypotheses based on theoretical and empirical evidence need to be set up and tested. As this is an emergent field of research, studies on the predictors of eHealth literacy are limited, and inconsistent results are common . NeverthPeople aged above 65 years are less likely than the younger generation to have had the chance to familiarize themselves with ICT either at school or at work ,35. CombH1: Age is negatively related to the scores on all latent variables representing the 7 scales.Technology is traditionally perceived as a male-dominated domain, with men usually reporting higher levels of digital skills than women ,44. Howe1. Using technology to process health information, 2. Understanding of health concepts and language, and 5. Motivated to engage with digital services.H2a: Being female is related to higher scores on the 3 latent variables representing the scales H2b: Sex is not related to score differences on the 4 latent variables representing the other 4 scales.Many studies have found education to be a predictor of ICT use and skills ,35,38,53H3: Education is positively related to the scores on all latent variables representing the 7 scales.Higher ICT use is frequently found to be related to better digital skills and higher likelihood of searching for health information and using web-based health information and health apps ,43,53,632. Understanding of health concepts and language.H4: ICT use is positively related to the scores on the 6 latent variables representing scales 1, 3, 4, 5, 6, and 7 but not to the score on the latent variable representing Considering that Australia is a multicultural country, whether the nation’s main language—English—was spoken at home was also included for analysis. However, because of the limited studies on the eHealth literacy of ethnic minorities  and becaA cross-sectional survey was conducted across 3 health sites in Victoria, Australia, in 2018. The 3 sites included a private primary care medical clinic and a not-for-profit community health organization located in metropolitan areas as well as a private primary care medical clinic in a regional area. These sites were selected because they represented a mix of advantaged, disadvantaged, culturally diverse, metropolitan, and regional areas to ensure that the sample would capture people with different eHealth literacy levels. People attending the health sites were invited to participate if they were aged 18 years or older, with or without any health conditions, and were able to complete the eHLQ in paper-based format, web-based format, or face-to-face interview. The option of offering interviews allowed people with lower literacy who were not interested in reading to feel comfortable to participate in the survey, another strategy to ensure that people with potentially lower eHealth literacy were included. The exclusion criteria included people currently experiencing significant cognitive or mental health issues or too clinically unwell as deemed by their treating health care professionals and those with insufficient fluency in English to complete the survey because no family member or carer was present to assist them. The study was approved by the Deakin University Human Research Ethics Committee . Potential participants were provided information about the study, including that participation was voluntary. Returning the completed questionnaire was regarded as implied consent.Demographic data collected for analysis included age, sex , education , language (spoke English at home or not), and ICT use. The classification of education into 4 categories was somewhat arbitrary. Other Australian studies of eHealth literacy and internet use only included the 3 education categories of secondary school or less, certificate or diploma, and university or higher ,65. GiveTo evaluate known-groups validity, an important prerequisite for hypothesis testing is evidence of measurement equivalence or invariance across the groups , which rStandards, the main concern is uniform DIF because it can lead to “systematically different responses to a particular item” [To evaluate measurement invariance, DIF is a common statistical observation that signals whether an item is functioning differentially across the grouping variables ,72. The ar item” .measurement model that describes the relationship of the latent variables and their observed variables and a structural model that describes the links among the latent variables [Standards, is to identify systematically different responses [To ensure that DIF was considered, analysis using the multiple indicators multiple causes (MIMIC) model approach was chosen for this study. The MIMIC model is a type of structural equation modeling (SEM), which contains a ariables . Therefoariables . This apariables . In a MIariables ,78. The ariables ,74,79. Aariables , and theesponses .frequentist approaches such as maximum likelihood [approximately zero rather than exactly zero as in traditional SEM [Furthermore, a Bayesian approach was used for the MIMIC model in this study. It has been argued that the Bayesian approach is a better reflection of substantive theories because it is less restrictive; in addition, it does not rely on data with normal distribution and performs well with a small sample size ,81. A fekelihood ,81. Of ponal SEM . The samdf of the inverse-Wishart distribution and gradually lowering the df parameter to find a more flexible model [PPP) and 95% CI for the difference between the observed and replicated chi-square values. PPP<.05 and positive 95% CI indicate a misfit, and PPP of approximately .50 and a value of zero falling close to the middle of the 95% CI indicates an excellent fit [For a Bayesian MIMIC model, informative small variance priors are applied to the DIF paths. A prior variance of 0.01 or 0.02 means that 95% of the variation lies within the ranges of ±0.20 or ±0.28 ,82. In alent fit . Models For the MIMIC model, age, sex, education, ICT use, and language were included as covariates. However, given that research indicated that age, sex, and education were also associated with ICT use and skills ,32,43, IPPP=.49, 95% CI for the difference between observed and replicated chi-square values –11.81 to 12.25), and number of devices was identified with the highest loading (0.81) on ICT use  for ICT use, a final setup for the MIMIC model was to identify the single best indicator for ICT use. It has been argued that the single best indicator is sufficient for developing theoretically sophisticated models . This wa ICT use . Hence, df of the inverse-Wishart distribution=200, 150, 100, 80, and 60. The results would inform the use of the prior for residual covariance for the MIMIC model testing. Next, 3 models using the chosen prior combined with informative priors for DIF paths  were fitted to the data. Model estimation was performed with 50,000 iterations.Descriptive statistics were conducted using SPSS, version 25.0 (IBM Corporation) , and theTo determine the group differences, direct effect, indirect effect, and total effect produced in the Mplus outputs were examined. A significant total effect indicated significant group differences. A significant direct effect indicated that group differences existed independent of any mediating effect, whereas a significant indirect effect indicated that group differences were mediated through ICT use. Mplus produced the results of one-tailed P values and indicated that P<.025 was significant. However, because the hypotheses for age, sex, education, and ICT use were directional, P<.05 was considered significant, whereas P<.025 remained significant for either a positive or negative effect for language because no hypothesis was set a priori. A further calculation of mediation proportion was undertaken to gain deeper understanding of the extent of mediation. Mediation proportion refers to the portion of effect on an outcome explained by an intermediate variable. It is calculated by dividing the indirect effect by the total effect . It shouPPPP) from the model of interest needed to be examined. The PPPP is a value used “for the evaluation of hypotheses specifying small variance priors for the parameters of interest” [PPPP>.05, that is, it can be considered nonsignificant, the estimates of the DIF are considered approximate zero and are thus ignorable. Hence, if the model of interest has a variance prior of 0.01 and the PPPP is nonsignificant, then estimates within the range of ±0.20 could be considered ignorable [For the evaluation of DIF, a significant direct effect from the covariate on the observed variable, that is, the questionnaire item, indicated the presence of DIF -92, and nterest” . It is agnorable .2. Understanding of health concepts and language: mean 2.95, SD 0.41), but they might not always use technology for health . Although the participants were generally comfortable with the privacy and security of digital health systems , they were less likely to consider that the systems met their individual needs .A total of 525 responses were included for analysis. The mean age of the participants was 56.8 (SD 18.6) years. Of the 525 participants, 320 (61%) were women, 175 (33.3%) had a university education, and 162 (30.9%) spoke a language other than English at home. Ownership of digital devices was generally high, but of the 525 participants, 66 (12.6%) did not have a mobile phone and 153 (28.8%) did not have a computer or laptop, whereas 133 (25.6%) did not search for any web-based information . Age also had quite a strong negative effect on 1. Using technology to process health information, with a total effect of –0.32 . A total effect of –0.21 (posterior SD 0.09) was found for both 5. Motivated to engage with digital services and 7. Digital services that suit individual needs, with P=.01 and P=.02, respectively. For all 4 latent variables with significant total effect, approximately two-thirds was a direct effect.This hypothesis was supported for only 4 of the expected latent variables. Being older was most strongly related to lower scores in 1. Using technology to process health information, 2. Understanding of health concepts and language, and 5. Motivated to engage with digital services.H2a: Being female is related to higher scores on the 3 latent variables representing the scales H2b: Sex is not related to score differences on the 4 latent variables representing the other 4 scales.5. Motivated to engage with digital services, the estimate was –0.12, which was less than ±0.2 and was considered a too small effect [H2a was not supported, whereas H2b was supported. Sex was not related to the score differences in any of the latent variables. Although a significant total effect was found for the scale l effect  and therH3: Education is positively related to the scores on all latent variables representing the 7 scales.1. Using technology to process health information , with 59% of the effect mediated by ICT use, and 3. Ability to actively engage with digital services , with 56% of the effect mediated by ICT use.This hypothesis was supported only for 2 of the 7 expected latent variables. Higher education was associated with higher scores in the latent variables representing 2. Understanding of health concepts and language.H4: ICT use is positively related to the scores on the 6 latent variables representing scales 1, 3, 4, 5, 6, and 7 but not to the score on the latent variable representing 4. Feel safe and in control. Higher ICT use had the strongest relation to higher scores in 3. Ability to actively engage with digital services, with a direct effect of 0.42 , followed by 1. Using technology to process health information, with a direct effect of 0.38 . The other latent variables with significant positive effect included 5. Motivated to engage with digital services , 6. Access to digital services that work , and 7. Digital services that suit individual needs .This hypothesis was supported except for the latent variable representing 2. Understanding of health concepts and language, the effect size was ignorable because the estimate of –0.15 was considered too small [No group differences were found between the participants who spoke English and those who spoke a language other than English at home. Although a significant total effect was found for the latent variable representing oo small .This study used a Bayesian mediated MIMIC model approach to collect evidence based on relations to other variables to evaluate the eHLQ as a tool to understand eHealth literacy needs, using data collected from known-groups validity in the Australian community health context. Hypotheses for the expected score differences for age, sex, education, ICT use, and speaking English at home or not were supported for some of the eHealth literacy latent variables represented by the relevant eHLQ scales but not all. The results also confirmed measurement invariance across 5 demographic groups. This is important because the presence of measurement invariance indicates that when the eHLQ is applied to compare population groups with different demographic compositions in the Australian health setting, unbiased estimates of mean group differences are obtained, which provide reliable data to researchers, clinicians, and policy makers.Standards, the authoritative validity-testing reference, suggests that the expected score differences among groups can be an important source of evidence for validity [Although the validity , it was t test or analysis of variance or relevant nonparametric tests to establish group differences. However, these methods do not take into account DIF as a potential confounding factor. Without establishing measurement invariance across demographic groups, any apparent group differences detected cannot be ascertained. Apart from being able to detect group differences as well as DIF in using a MIMIC model, this SEM approach accounts for measurement errors, and the use of the Bayesian approach also allows for an evaluation that is a better reflection of the real world, whereas the inclusion of a mediator adds valuable information to the underlying mechanism of the group differences detected. Unlike t tests or analysis of variance, age was not divided into arbitrary groups but was treated as a continuous variable. Therefore, the results of this study, in fact, provide new and unbiased insights into the predictors of eHealth literacy.This study is based on a contemporary statistical method, the Bayesian mediated MIMIC model, rather than methods frequently used in previous and current eHealth literacy research specifically and psychometric research in general. This could also be one of the reasons why the findings from this study are somewhat different from those of current studies. Common practice in testing for known-groups validity uses statistical methods such as the independent sample one-tailed or two-tailed 3. Ability to actively engage with digital services having the strongest effect, followed by 1. Using technology to process health information, 5. Motivated to engage with digital services, and 7. Digital services that suit individual needs. This result is generally consistent with existing studies. However, an interesting result from the mediated MIMIC model is that most of the effect of age was not mediated by ICT use, indicating that ICT use may have a limited role in these 4 domains of eHealth literacy. A possible factor is the cognitive, motor, and sensory decline associated with aging, as frequently suggested in the literature [The findings in this study indicated that being older had lower scores in 4 scales, with terature -42. Suchterature  not only1. Using technology to process health information and 3. Ability to actively engage with digital services. Unlike in the case of age, most of the effects were mediated by ICT use, confirming the role of ICT use in enhancing certain aspects of eHealth literacy. As such, providing ICT education and training is likely to benefit people with lower education in increasing their ability to engage with digital services and enhancing use of technology for health. Such a finding suggests that other efforts are required to address the eHealth literacy needs concerning privacy, motivation, or access to suitable digital services. In contrast, why education was not related to 2. Understanding of health concepts and language in the digital context warrants further investigation.Another result of interest is the relationship between education and the eHealth literacy domains because education is only positively associated with the scales Sex having no relationship with any of the eHLQ scale scores suggests that sex may not be a good predictor of eHealth literacy. This may be due to the narrowing gap in education between the sexes in recent years and the fact that technology use has become an indispensable part of modern-day life for most people. Although the hypothesis of ICT use is mostly supported, the results show that ICT use is not associated with the eHealth literacy domain of feeling safe and in control. This again reiterates that technical skills in ICT training are inadequate to address all eHealth literacy needs. These findings also call for assessment of eHealth literacy using unbiased multidimensional questionnaires such as the eHLQ so that eHealth literacy needs can be clearly identified and addressed.Although the aim of this study is to collect evidence based on relations to other variables, the statistical method used also established the robustness of the internal structure of the eHLQ in terms of measurement invariance across 5 demographic groups. The presence of DIF indicates that a questionnaire item is not measuring what is intended, and thus the resulting scores may be biased . Given tA limitation of this study is that the hypotheses were based on limited empirical findings of eHealth literacy predictors such that the results may need to be interpreted differently. Although the MIMIC model approach has the advantage of evaluating both group differences and DIF simultaneously, only uniform DIF was tested, and nonuniform DIF was not investigated. Nevertheless, uniform DIF is a more important threat to validity because it can lead to systematic score differences on account of group characteristics . In examWith health resources and services increasingly delivered through digital technologies, eHealth literacy has become an essential capability in the digital age. This study provides robust validity evidence of the eHLQ in the Australian community health setting. The evidence demonstrates that the tool can be used by health care providers and policy makers to gain unbiased and valuable insights into people’s diverse eHealth literacy needs so that tailored health interventions can be effectively developed in similar settings. The eHLQ can also be used to align the demand of any eHealth system with the eHealth literacy needs of users to optimize access and use of digital health among users and promote health equity.


Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of experts to use when formulating recommendations or making decisions. While extensively used for clinical and public health recommendations, EtD frameworks are not in widespread use in environmental health. This review sought to identify, compare and contrast key EtD frameworks for decisions on interventions used in clinical medicine, public health or environmental health. This information can be used to develop an EtD framework suitable for formulating recommendations for interventions in environmental health.We identified a convenience sample of EtD frameworks used by a range of organizations. We searched Medline for systematic reviews of frameworks. We summarized the decision criteria in the selected frameworks and reviews in a qualitative manner.Fourteen organizations provided 18 EtD frameworks; most frameworks focused on clinical medicine or public health interventions; four focused on environmental health and three on economic considerations. Harms of interventions were examined in all frameworks and benefits in all but one. Other criteria included certainty of the body of evidence (15 frameworks), resource considerations (15), feasibility (13), equity (12), values (11), acceptability (11), and human rights (2). There was variation in how specific criteria were defined. The five identified systematic reviews reported a similar spectrum of EtD criteria.The EtD frameworks examined encompassed similar criteria, with tailoring to specific audience needs. Existing frameworks are a useful starting point for development of one tailored to decision-making in environmental health.JPB Foundation.The online version contains supplementary material available at 10.1186/s12940-021-00794-z. It is widely recognized that environmental pollution is an important determinant of health and that interventions, in particular policy recommendations, are a key means by which population health can be improved. The formulation of trustworthy and impactful recommendations and policies on environmental health interventions is a complex task, which requires the identification of all relevant data and evidence, their critical appraisal and synthesis and translation into a recommendation or policy. The processes and methods for hazard identification and risk assessment of environmental substances based on systematic reviews of the evidence have advanced considerably in the last decade –4. TheseThe final step in the process of formulating evidence-informed recommendations and policies involves the translation of data and evidence on various decision factors into an explicit recommendation or policy. Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of technical experts or policy-makers to accomplish this step , 11. TheIn environmental health, once hazards are identified and risks assessed, organizations may want to examine mitigating and prevention interventions and make recommendations and policies based on systematic reviews of research evidence and other data and information. An EtD framework suitable for environmental health interventions will facilitate this process. However, such frameworks have not been widely used in this field.The objectives of this review were to identify, compare and contrast key EtD frameworks for interventions in clinical medicine, public health and environmental health and to summarize the main decision criteria across these frameworks. The identification of these criteria will inform the development of an EtD framework for use in formulating recommendations regarding mitigating and prevention interventions related to exposure to harmful substances in the environment.In order to identify relevant, existing EtD frameworks, we took two approaches: a search of the peer-reviewed literature for systematic reviews of EtD frameworks, and identification of frameworks used by a range of organizations which make decisions or formulate recommendations in clinical medicine, or public or environmental health.The focus of this paper is on the substantive criteria for decision-making with respect to interventions, including both normative criteria (what should be done) and feasibility criteria (what can be done) . CriteriThere are an increasing number of organizations which use systematic and transparent approaches to synthesize evidence in environmental health for hazard identification , risk asTo identify relevant EtD frameworks, we searched Medline via PubMed for systematic reviews of EtD frameworks, templates or tools published in English and indexed in the 10-year period up to 7 January 2021. The complete search strategy is found in Two persons (SLN and MTA) screened the titles and abstracts, and full-text versions were retrieved for studies which potentially fulfilled inclusion criteria. Consensus was achieved between the two reviewers for final inclusion in the review.In addition to the systematic review described above, we examined a broad range of organizations including clinical and public health guideline development groups in academia and the private sector, healthcare provider professional organizations, governmental agencies and international organizations. Given the purpose of this review, we focused particularly on organizations that work in the field of environmental health. Organizations were identified based on author knowledge of prominent guidelines in public health and clinical medicine, by snowballing (reviewing the origin of or basis for identified frameworks), and by conferring with a broad international network of guideline developers based on our experiences and contacts.We used this pragmatic approach for several reasons. First, because of the dominance of the GRADE  EtD framework over the last 15 years , 17, manFor each systematic review identified in our search, a single author (SLN) extracted key information describing the focus of the review, time period searched, and the main findings, including an overview of the criteria identified across the included frameworks. We did not extract data on each individual framework included in each review.For the EtD frameworks used by key organizations, we extracted data on organizational characteristics, the methods used to develop the framework, the funders and any declared interests of the developers of the framework, how evidence should be used to inform each criterion in the framework, quality assessment of individual studies and of the body of evidence, the specific EtD criteria, and the types of conclusions or recommendations formulated. One author extracted these data (SLN) and a second checked them (MTA); disagreements were discussed and consensus reached. Data were extracted into a template in Excel .For the key organizations, the primary decision criteria for each EtD framework were extracted from the main published reports, and specifically from the identified template or list of criteria. Criteria which are mentioned only in the narrative text accompanying a framework were not extracted unless the text suggested that they were consistently applied in decision making.Data were summarized in a narrative, qualitative manner. Given that the included frameworks were a convenience sample, descriptive statistics and statistical comparisons are not meaningful. Quality assessment of the frameworks was not performed as there is no standard for such an assessment.The study funder (JPB Foundation) had no role in the study design; data collection, analysis or interpretation; in writing the report; or in the decision to publish.The bibliographic database search for systematic reviews of EtD frameworks yielded 399 citations, of which four fulfilled inclusion criteria , 18, 19 Fourteen organizations that use EtD frameworks for recommendation formulation or decision-making were examined in detail Table . One orgTwo organizations focus primarily on clinical care: the Scottish Intercollegiate Guideline Network (SIGN)  and the For the organizations that developed de novo EtD frameworks and described the process for developing them, all used an iterative approach based on an examination of other organizations, with input from experts in guideline methods and evidence synthesis Table . The mosIt was difficult to discern the expertise of contributors to framework development; most groups appeared to consist mainly of academic, generalist guideline methodologists with either clinical or public health experience. Social scientists led work on WHO-INTEGRATE  and contEighteen frameworks were examined in detail: one from each of the 14 organizations, except for CalEPA and GRADE, where two , 23 and “Priority of the problem” was explicitly included in the EtD frameworks based on GRADE , 34, 37,The meaning of the criterion “values and preferences” varied over time and across publications. This phrase encompasses two different constructs: the relative value that persons affected by the recommendations place on the outcomes of the intervention and the preferences such persons have regarding the intervention options. The GRADE Working Group initially combined these two constructs  with a dAll of the frameworks report that the EtD criteria should be informed by evidence obtained from a variety of sources, with an emphasis on systematic reviews of research evidence Table . Twelve Nine of the included organizations provide specific guidance on the nomenclature for the types of recommendations formulated based on the EtD criteria Table . The mosFive organizations provide explicit guidance on the situations where recommendations cannot be formulated due to insufficient evidence or a close balance between benefits and harms Table  11, 20,, 32, 36.GRADE includes EtD frameworks for four different purposes: i) clinical recommendations, individual perspective; ii) clinical recommendations, population perspective; iii) coverage decisions; and iv) health system and public health recommendations/decisions.  These ACIP’s EtD framework is derived directly from GRADE , while NThe WHO-INTEGRATE EtD framework version 1.0 , 41, firThe frameworks that focus on interventions in environmental health were generally less well developed and lacked specificity compared with the clinical and public health frameworks , 21, 22 Additional details on each of the included frameworks are provided in Additional file A number of EtD frameworks have been developed in a variety of clinical, and public and environmental health disciplines, and there is significant overlap in the criteria used to inform recommendations or decisions across these frameworks. Benefits and harms are almost universally included, and an assessment of certainty or quality of the body of evidence and some measure of resource use were included in most frameworks examined. Other decision criteria such as values, equity, feasibility, and acceptability were variably included, while only two frameworks encompassed human rights. There was variation across frameworks in terminology, definitions and presentation of EtD criteria. The five systematic reviews of EtD frameworks describe decision criteria that were similar to those presented in the frameworks from key organizations.The 18 frameworks identified for 14 key organizations focused mainly on clinical medicine or public health interventions. The GRADE framework is the most well developed, with extensive information and guidance on its methods. A number of the key organizations examined have adopted GRADE, often with minor modifications , 34, 37.The process and methods for developing EtD frameworks were often superficial and poorly reported. Only WHO-INTEGRATE  and the GRADE has dominated guideline methods in health care and public health over the last 15 years. This has led to productive collaborations, standardization of processes and methods, and the development of detailed methods and guidance. On the other hand, there are instances where intellectual dominance by a single group – i.e., monopolies of knowledge – can suppress innovation and slow down development processes .EtD frameworks largely focus on, and are optimal for, single-component interventions with simple linear pathways from intervention to outcomes. In addition, most EtD frameworks do not consider the context in which the intervention is delivered. With the exception of WHO-INTEGRATE, EtD frameworks do not explicitly or even implicitly incorporate a complexity perspective including the inter-relationship between the intervention and the context or environment in which it is delivered. Only the GCPS, the USPSTF, and WHO-INTEGRATE recommend the use of visual depictions of the relationships among the intervention components: such approaches facilitate examination of multiple key questions linked across the causal pathway for intervention effects, as well as the consideration of contextual factors which may be important for decision making.Guidance on if and when to identify, synthesize and integrate evidence on criteria other than benefits and harms of the intervention into the decision-making process was provided only for WHO-INTEGRATE . In our Ideally, EtD considerations are discussed early in the guideline development process, and strategic decisions are made as to the types of information and evidence that will be needed to inform key considerations. It may not be necessary, feasible, or even possible, to systematically examine evidence on all EtD considerations. The participation of a range of stakeholders, including persons and communities potentially affected by the intervention, is important both early in the planning phase and later when decisions are made. Their perspectives and experiences with issues such as acceptability and feasibility, for example, are critical for formulating credible and impactful recommendations or policies.It was difficult to compare specific EtD criteria across frameworks due to significant variation in approaches to lumping and splitting criteria across frameworks, as well as in the terminology and definitions used. While most frameworks presented broad categories of criteria , only WHO-INTEGRATE  providesThere is also variability in how criteria are defined and whether operational definitions or guidance are provided. The criterion “values and preferences” is particularly problematic and guideline developers continue to use the phrase “values and preferences” with unclear and variable meaning.The GRADE criterion of “priority of the problem” is also problematic. It was initially included in the GRADE framework to facilitate prioritization across interventions, such as at the national or sub-national levels. However, in most guideline development scenarios, the problem on which interventions and comparators are focused has already been determined to be of high priority. Thus, examination of the burden of disease and other priority considerations is irrelevant at the stage of recommendation formulation.The criteria related to resource use vary considerably, and may include cost, affordability, infrastructure needs, personnel training, and measures of economic efficiency . This variability is due in part to the different perspectives of the sponsors of the EtD framework or of the target audiences for the organization’s products. For example, the USPSTF is prohibited by national legislation from considering cost or cost-effectiveness in their recommendations. On the other hand, for the UK’s NICE, cost-effectiveness is a critical part of their decisions. There is also variability in the extent and specificity of the guidance on how to incorporate resource use into decision-making. While well-developed for NICE , and reaThe criterion of “equity” is poorly described and little operational guidance is provided across the frameworks except for WHO-INTEGRATE which provides several sub-criteria for consideration under the main criterion of “health equity, equality and non-discrimination” .Group decision making when formulating recommendations in guidelines is rarely a simple, linear process. While EtD frameworks support the normative aspect of decision making in terms of how expert panels “should” or “ought to” develop recommendations by providing a structure for decision making –49, manyWhile acknowledging the complex process of formulating recommendations in guidelines, EtD frameworks are a valuable tool and have led to vast improvements over “free-for-all” meetings where decision making criteria were selected in an ad hoc manner, and it was often unclear which considerations actually contributed to final decisions and their relative weight, what evidence was examined, and how and why the final recommendation or decision was arrived at. EtD frameworks can be combined with any one of a number of approaches for achieving group consensus on the direction, strength, and wording of a recommendation or decision, including formal approaches such as the Delphi approach and the Nominal Group Technique, or voting if consensus cannot be reached.Applying approaches for clinical medicine interventions to other scientific fields, including environmental health, is challenging. There are important differences both in the assessment of the quality (certainty) of the body of evidence, and in translating evidence to recommendations or decisions. Woodruff and colleagues  note thaThe approach taken for this review and analysis has a number of limitations. The methods for identifying existing frameworks were limited: the systematic review included only reviews and focused only on English language literature accessible through PubMed. Furthermore, many EtD frameworks are not published in the peer reviewed literature: rather, they are found on organizational web-sites and/or in organizational procedure manuals. For the review of key organizations, only a convenience sample was examined and EtD frameworks with important additional considerations or novel approaches may have been overlooked. However, it is unlikely that this approach missed key or significantly different frameworks in view of the dominance of GRADE and our consultation with experts on EtD frameworks.Our analysis also has limitations related to the nature of the available data. Published information on the methods for developing frameworks and for operationalizing them was generally sparse, except for GRADE and WHO-INTEGRATE. For some frameworks it was difficult to identify exactly which criteria were routinely included in the decision-making process. While the publication might present a list or table of criteria, additional criteria might be mentioned in the adjacent text. However, the degree to which the guideline group or other decision maker addressed these additional criteria was often unclear.EtD frameworks are an extremely useful tool for recommendation formulation and decision making in healthcare and other scientific fields. They enable decisions to be made based on research evidence and with explicit consideration of a range of constructs, and they facilitate clear articulation of the rationale for decisions.The principles underpinning evidence-informed, transparent and impactful decision making are the same across scientific fields, including environmental health. The GRADE EtD framework, along with WHO-INTEGRATE with its focus on contextual issues and inter-relationships, provide a useful starting point for consideration for decision-making for interventions in environmental health. Significant modifications will be needed, however, given the nature of the evidence base and the complex context in which environmental health interventions are designed, implemented, regulated and evaluated. The process for developing an EtD framework for environmental health interventions requires a broad range of experts, including not only environmental health scientists, but evidence synthesis and guideline methodologists, public health generalists, human rights and ethics experts, social scientists, and economists, among others. An iterative development process will be needed: a draft framework should be pilot-tested, revised and evaluated. The framework’s utility and impact on decision making, and the quality and impact of the resultant recommendations need careful evaluation, with the results used to develop future iterations.Additional file 1.


The COVID-19 public health crisis has produced an immense and quickly evolving body of evidence. This research speed and volume, along with variability in quality, could overwhelm public health decision-makers striving to make timely decisions based on the best available evidence. In response to this challenge, the National Collaborating Centre for Methods and Tools developed a Rapid Evidence Service, building on internationally accepted rapid review methodologies, to address priority COVID-19 public health questions.Each week, the Rapid Evidence Service team receives requests from public health decision-makers, prioritizes questions received, and frames the prioritized topics into searchable questions. We develop and conduct a comprehensive search strategy and critically appraise all relevant evidence using validated tools. We synthesize the findings into a final report that includes key messages, with a rating of the certainty of the evidence using GRADE, as well as an overview of evidence and remaining knowledge gaps. Rapid reviews are typically completed and disseminated within two weeks. From May 2020 to July 21, 2021, we have answered more than 31 distinct questions and completed 32 updates as new evidence emerged. Reviews receive an average of 213 downloads per week, with some reaching over 7700. To date reviews have been accessed and cited around the world, and a more fulsome evaluation of impact on decision-making is planned.The development, evolution, and lessons learned from our process, presented here, provides a real-world example of how review-level evidence can be made available – rapidly and rigorously, and in response to decision-makers’ needs – during an unprecedented public health crisis.The online version contains supplementary material available at 10.1186/s12874-021-01436-1. Coronavirus disease 2019 (COVID-19) is an urgent public health crisis requiring prompt decision-making due to rapidly evolving policy and practice needs. Public health decision-makers are always challenged with integrating research into decision-making . An analKnowledge syntheses  represent the highest levels of research evidence , summariThe six National Collaborating Centres for Public Health were created by the Public Health Agency of Canada (PHAC) in 2005 to strengthen public health in response to the 2003 Severe Acute Respiratory Syndrome (SARS) epidemic . The NCCAs COVID-19 unfolded, the NCCMT heard from public health decision-makers at all levels of government  about the lack of time and human resources to find answers to key questions. These decision-makers ranged from managers responsible for front line public health staff at local health departments, to members of federal government advisory committees responsible for key recommendations and policies related to various aspects of the pandemic response. To address this need – and with encouragement from our funder to focus and reallocate resources – the NCCMT pivoted to completing RRs within 5–10 business days based on priority questions from public health decision-makers.In reviewing NCCMT’s established RR protocol , the teaHere we describe in detail the methods the NCCMT has used to conduct RRs as part of our Rapid Evidence Service (RES), including how these have evolved to ensure feasibility, accuracy, and efficiency as the evidence landscape changed. Our process Fig.  may be uThe initial protocol built upon the five steps in the NCCMT’s RR Guidebook : defininThe RES requires a team with methodological and organizational expertise Table  in: condLimited resources were initially allocated to the RES; questions were restricted to two per week. Questions were prioritized from a list of urgent topics by the NCCMT Scientific Director, Operational Lead, and RES Scientific Lead, primarily based on team expertise and capacity. As demand grew, 60 % of NCCMT staff resources were re-allocated to the RES. Questions initially came from PHAC. As awareness of the service grew, requests came from decision-makers at all levels across Canada and indirectly through involvement in the COVID-19 Evidence Network to support Decision-Makers (COVID-END) . As requA weekly team meeting was implemented to assess progress on current reviews and assess capacity for new reviews, review the week’s schedule for review completion, assign staff tasks, and prioritize new questions and updates to previously completed reviews. Decisions as to which questions to accept were made through team discussion at the weekly team meeting and were based on several factors including: the urgency and relevance to Canada; in-house content expertise and capacity; and availability of evidence, determined by a preliminary scan of databases. Once accepted, an RR Lead is assigned. The Search Lead first looks to see if a recently completed review on the topic exists by scanning COVID-19 RR repositories and websites of organizations known to conduct rigorous syntheses. This step was introduced after we identified duplication of efforts between our team and others  or PS  format and identify inclusion/exclusion criteria and preliminary search terms. Unless the question is very clear or is an update to a previously completed review, the team will confirm the refined question and proposed criteria with the requestor. This usually involves narrowing the scope of the review to a question that is feasible to answer in the given time frame.Searching for evidence involves developing a search strategy and screening results for inclusion. The RR Lead, RES Coordinator, and RR Search Lead/Staff collaboratively develop a strategy for each question including databases to search, search terms and parameters for each database, and whether grey literature will be included. Our initial search strategy included 14 databases or websites, ten of which were developed specifically for COVID-19 Table . The seaSearches are conducted in English; peer-reviewed, preprints, and non-peer reviewed reports are included. When titles and abstracts for non-English publications are available in English, and are sufficient to determine eligibility for inclusion, we use in-house expertise  or Google Translate. Depending on the question, we may consider a search for data from various public health jurisdictions , to supplement scientific evidence or when this may enhance our ability to answer a question. For example, in a review on COVID-19’s role on substance use, overdoses, and substance-related deaths, we supplemented the search of scientific literature with available surveillance data across Canada .Initially, most COVID-19-specific databases and repositories did not have functions to export all references into reference management software. To accommodate this, RR Search Staff entered potentially eligible studies, based on title and abstract screening, into an Excel spreadsheet for full text screening by the RR Lead. As the functionality for many databases evolved, we began to export references into Rayyan – an open access systematic review screening software  – for scTitle and abstract screening are done by a single reviewer, as per other RR guidelines , 29. FulFor feasibility and timeliness, we prioritize guidelines and/or high-quality syntheses, when available. If a recent high-quality synthesis is available, we will consider excluding single studies or only including single studies after the last search date. To gauge quality, we look for whether a comprehensive search strategy is described and included evidence is critically appraised. If both criteria are met, we appraise the synthesis using AMSTAR 1 (A MeaSurement Tool to Assess systematic Reviews) . A revieFor questions where there were few systematic or rapid reviews or single studies identified, we may consider expert opinion or opinion-based guidance documents. These may include interim guidance documents from reputable organizations  that provide policy recommendations for a given aspect of pandemic response that was created through an expert panel for example. Although these are typically considered at the bottom of the hierarchy of evidence, given the novelty of the SARS-CoV-2 virus, and speed at which a response was required, these were sometimes considered the best available, or only available, evidence. These documents also underwent data extraction and critical appraisal as detailed below.The RES first focused exclusively on new questions. As evidence continued to evolve, there was a need to update completed reviews and a strategy for identifying evidence that moved from preprint to publication stage was required. While some updated preprints were easily identified during searches, there were instances where substantial changes to the paper  made it difficult to identify when a preprint had been modified. While many entries into preprint servers are updated within four weeks of publication of peer-reviewed versions, this occurs inconsistently. We now systematically review previously included evidence to determine if it has been updated and if results have changed when completing an update. This includes a targeted search for: (1) updated versions to other included RRs ; (2) publication of preprint manuscripts and associated changes to data or interpretation ; and (3) updates to surveillance data or grey literature sources . While this adds an additional step to the search process, it ensures we are including the most current evidence.We created a standard template to build the RR document for data extraction. Key information  are added and sent to staff for data extraction. A single team member extracts data and summarizes key findings relevant to the specific research question; this is double checked by the RR Lead. Specific information depends on the research question, but typically includes study design, quality of included single studies (syntheses only), setting, population characteristics, interventions/exposure, and key outcomes. Any results that are not relevant to the research question are not extracted. Study limitations are noted to inform key findings and recommendations.We critically appraise evidence using AMSTAR 1 for systematic reviews and Joanna Briggs Institute critical appraisal tools for other study designs , 31. SomIn initial RRs, we reported on the number of studies of low, moderate, and high quality to report overall quality of evidence. But we were concerned that, although studies were appraised as high methodological quality, they were based on designs that had inherently high risk of bias ; thus, overall confidence in the evidence was low. In response, we adapted the GRADE approach , 34: an Results are synthesized narratively due to variation in methodology and outcomes across included studies. Following data extraction, critical appraisal, and GRADE, the RR Lead completes the final synthesis for the Executive Summary. Early RR versions did not include an overall synthesis; this was added in response to requests from decision-makers for a high-level summary of key points, overview of evidence, and knowledge gaps, to be presented first. This revised layout more closely aligns with recommendations for communicating evidence to policymakers, including using a “graded entry” approach (1:3:25 page format), which allows users to access their preferred level of detail  –37.RRs are reviewed internally by the RR Lead, RES Scientific Lead, and NCCMT Scientific Director. For partnered RRs , partneA tailored knowledge translation plan is developed for each RR depending on the topic and target audiences. When the review is requested by an organization, it is shared immediately upon completion. All RRs are freely available to download from the NCCMT website . In SeptFrom May 2020 to July 2021, the NCCMT’s RES team has answered more than 31 distinct questions and completed 32 updates to previously completed reviews (ranging from 1 to 16 updates per question) as new evidence emerged. Preliminary data from web analytics to assess engagement shows our RRs are accessed by all Canadian provinces and territories and 99 countries worldwide. Metrics collected from September 2020 to July 2021 indicate that each review is typically accessed 216 times within the first week, with wide variation in access across reviews. For example, our review on the role of schools and daycares in COVID-19 has been our mostly highly accessed . BetweenA primary challenge to any RR is balance between speed and rigor. This issue is even more pronounced in the context of COVID-19 given the massive amount of data and the urgency with which evidence is needed to inform decisions. The streamlined approach of RRs  will always introduce some degree of bias, so it is important to establish and follow a transparent process. The evolving evidence landscape has necessitated many changes to our typical RR methodology; we anticipate further changes may be needed. For example, in addition to accepting and addressing new and emerging public health questions, we have committed to maintaining a living RR of the role of schools and daycares in COVID-19 transmission , given tWe are aware of other Canadian and international organizations conducting RRs on a number of topics related to COVID-19; many report adaptations to the RR process that are similar to ours , 43–46. Although wide variations in RR methodology have been reported both in response to the COVID-19 pandemic, and in general , overallFrom January 1-April 30, 2020, there were > 6,000 COVID-19-related manuscripts posted across preprint servers with > 250 preprints posted weekly . While tOur organization is fortunate to have been well-prepared, in terms of staff expertise and funder support, to rapidly respond to public health needs in this time of crisis. Given specialized skills and a dedicated, nimble team, it was possible to pivot from previous workplans and use rigorous methods to develop RRs in a much faster timeframe than has been reported pre-pandemic (average 3.2 months , range 1While many international RR groups focus specifically on clinical or treatment-related questions –52, few Most of our efforts have been to conduct knowledge syntheses on priority public health questions and to broadly share findings with decision-makers. As the urgency to complete reviews has diminished somewhat, there is opportunity to expand our knowledge translation to diverse audiences and to identify new ways to support implementation of evidence into policy and program decisions. We also now seek to formally evaluate our process and its impact on public health decision-making.This overview provides a real-world example of how internationally accepted RR methods can be modified to meet the emergent needs of public health decision-makers in the unprecedented context of the COVID-19 pandemic. As countries around the world continue to grapple with ongoing issues including vaccine rollout, variants of concern, public distrust, fatigue with pandemic-related restrictions, and social and economic inequalities, there has never been a more important time to work collaboratively and in partnership with decision-makers to ensure the best available evidence is available to inform policy decisions and program planning.Additional file 1.