Without adequate reporting of research, valuable time and resources are wasted. In the same vein, adequate reporting of practice guidelines to optimise patient care is equally important. Our study examines the quality of reporting of published WHO guidelines, over time, using the RIGHT  reporting checklist.We examined English-language guidelines approved by the WHO Guidelines Review Committee from inception of the committee in 2007 until 31 December 2017. Pairs of independent, trained reviewers assessed the reporting quality of these guidelines. Descriptive data were summarised with frequencies and percentages.We included 182 eligible guidelines. Overall, 25 out of the 34 RIGHT items were reported in 75% or more of the WHO guidelines. The reporting rates improved over time. Further, 90% of the guidelines reported document type in the title. The identification of evidence, the rationale for recommendations and the review process were reported in more than 80% of guidelines. The certainty of the evidence using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system was assessed in 81% of the guidelines assessed. While 82% of guidelines reported funding sources, only 25% mentioned the role of funders.WHO guidelines provide adequate reporting of many of the RIGHT items and reporting has improved over time. WHO guidelines compare favourably to guidelines produced by other organisations. However, reporting can be further improved in a number of areas. Clinical practice guidelines are statements that include recommendations intended to optimise patient care . A trustIn 2017, the RIGHT  Working Group published the RIGHT reporting checklist for guideline reporting . RIGHT hWHO develops guidelines on a wide range of clinical and public health topics for use by various stakeholders worldwide. While the quality of WHO guidelines has been studied –13, to dWe obtained a list of all documents approved by the WHO Guidelines Review Committee (GRC), a quality oversight body for WHO guidelines. We included only guidelines that were submitted to the GRC in English. This list encompassed documents approved by the GRC from its inception in 2007 through 31 December 2017. Because not all documents in the list provided by WHO appeared to be guidelines, we developed staged eligibility criteria. First, we included only documents that contained specific recommendations for clinical practice or public health policy. Second, we included only documents with original recommendations, i.e. those not referencing another (source) guideline. If the document had such a reference, it was considered a derivative product and was excluded from our analysis as derivative products are not reasonably expected to report the same level of detail as the original guideline. Third, we excluded several documents that contained specific recommendation(s) but appeared to be derivative products although they did not reference a source guideline.http://www.who.int/en/).When a guideline was updated, we included the newest edition and examined prior versions for additional information. We downloaded all documents pertaining to each guideline from the WHO website  of independent, trained researchers and any disagreements were resolved through discussion with a third researcher (XW). One author (XW) also verified all data extraction and any discrepancies were further discussed with the original extractors. The following data were extracted: (1) basic information about the guideline, including topic, developers and publication year; and (2) all content corresponding to items in the RIGHT reporting checklist .We rated each of the 34 (out of 35) RIGHT items with dichotomous options — ‘reported’ or ‘not reported’. The ‘reported’ option was used when the relevant information was provided in the guideline, whereas ‘not reported’” indicated that the relevant information could not be found or was unclear. RIGHT item #13a  is not applicable at the level of the guideline because it must be assessed for each individual recommendation; we therefore indicated ‘not applicable’ for all guidelines for this item and removed this item from both numerator and denominator when calculating overall proportions.Summary statistics (frequencies and percentages) are provided for each guideline characteristic and for each RIGHT item. To analyse reporting trends, we explored the mean number of reported items per guideline by year. We identified ten items related to four domains ) as key items because of their presumed importance in assessing the quality or trustworthiness of guidelines (hereafter referred to as ‘key items’). We then analysed trends in the reporting of these key items and compared them between guidelines developed by WHO alone versus by WHO in partnership with other organisations and between guidelines that reported using GRADE to assess the certainty of the evidence and/or formulate the recommendation versus those that did not. Because we included all WHO GRC-approved guidelines from 2008 to 2017, we report descriptive statistics only; our goal was not to infer or predict the characteristics of other cohorts of guidelines. We extracted and analysed the data using Excel 2016.n = 30), had been updated (n = 18), or were as yet unpublished or were published in 2018 (n = 9). Table We obtained a list of 239 records approved by the GRC from its inception until 31 December 2017. A total of 182 guidelines fulfilled eligibility criteria, while 57 were excluded because they were not guidelines according to our criteria  were mentioned in only 39%, 25% and 1%, respectively , recommendation(s) (n = 27), guidance (n = 15), policy statements (n = 5), and a variety of other terms . All the guidelines reported the publication year, either in the title or subtitle, elsewhere on the cover page, or on the copyright page. A glossary or a list of acronyms or abbreviations was provided in 132 (73%) guidelines, while 63 (35%) reported both a glossary and a list.Overall, more than 50% of WHO guidelines reported all the items in this domain and 164 (90%) reported the document type in the title. However, the descriptor for the document type varied considerably, including guideline(s)  guidelines clearly reported the aim(s) or specific objectives of the guideline. Some guidelines combined the aims and objectives while others distinguished them , 15; 172A total of 156 (86%) guidelines stated that the recommendations were based on systematic reviews, including 39 based on existing systematic reviews and 39 on new systematic reviews, while 78 used both. However, only 71 (39%) indicated the methods used to identify existing systematic reviews. All 148 (81%) guidelines that described the approach for assessing the certainty of evidence used the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach.Overall, 121 (66%) guidelines reported both the strength of recommendations and the certainty of evidence, whereas 18 (10%) reported only the strength of recommendations and 2 (<1%) reported only the certainty of evidence. Regarding the rationale for recommendations, many guidelines reported considering various important factors in addition to benefits and harms of the intervention without mentioning the methods used to collect this information. For example, 148 guidelines (81%) reported considering values and preferences while only 32 (18%) reported the methods used to collect this information. Similarly, 171 (94%) reported that cost and resource implications contributed to the recommendation but only 68 (37%) noted the sources for these data.A total of 168 (92%) guidelines reported information about the external review of the draft final guideline; 116 (64%) guidelines reported the peer reviewers, 123 (68%) described the review process and 56 (31%) mentioned management of the feedback, while only 33 (18%) reported all 3 of these components. We did not collect specific information from each guideline on quality assurance because we only included GRC-approved WHO guidelines. All such guidelines have undergone an independent quality assurance process, which is indicated on the website where these guidelines were sourced [The funding sources was reported in 149 (82%) guidelines, including 46 (25%) that stated the role of funders in specific stages of development and 2 that mentioned funding for the dissemination and implementation of recommendations. Declarations of interest for all contributors were reportedly obtained in 177 (97%) guidelines, of which 37 declared no COI. Of the 140 guidelines with a COI noted among contributors, 122 (87%) reported the management of COIs.“A limitation of the search method was its restriction to guidelines in English. Also, there was no grey literature search” [The sites where relevant documents could be accessed and gaps in the evidence were respectively reported in 98% and 84% of guidelines. On the other hand, only 2 (<1%) guidelines reported limitations, for example,  search” .We observed an increase over time in the percentage of items reported with a more remarkable increase in key items than in non-key items Fig. a. The trWe compared the reporting of key items between guidelines developed by WHO alone and by WHO in partnership with other organisations and found that reporting of all key items slightly favoured those developed by WHO alone, except for reporting of COIs, where 100% of partnered guidelines reported COIs Fig. a. Among This is the first comprehensive assessment of the reporting of WHO guidelines. We examined 182 WHO guidelines using the RIGHT reporting checklist. On average, 25 RIGHT items were reported in each guideline and 75% of guidelines reported more than 70% (25 out of 34) of items. The items in the basic information, background, recommendations, and review and quality assurance were reported in a high percentage of guidelines. On the other hand, the process of evidence preparation, the role of funders and the limitations of the guideline were reported in less than 50% of guidelines. We noted an increase in overall reporting completeness over time.WHO guidelines compare favourably to prior analyses of guidelines developed by other entities — WHO adhered to a median of 79% of RIGHT items, while cohorts of Croatian and European practice guidelines both adhered to fewer than 50% of RIGHT items . A studyAlthough most items were reported in many WHO guidelines, there is significant room for improvement. First, labels for guidelines varied, with more than 10 different terms used to represent a ‘guideline’. This could confuse users as to the nature of the document and could make it difficult for end users to retrieve and for indexers to categorise these documents ; this inSecond, items included in the RIGHT domain of evidence were reported at relatively low rates compared with other domains. Several studies using the Appraisal of Guidelines for Research and Evaluation (AGREE) II tool –25 to evThird, while information about funders and declarations of interest was provided in more than 80% of the guidelines, the role of funders and the management of COIs were insufficiently reported. Similar findings have been noted previously in WHO guidelines . FurtherThe reasons for the improvements noted in the reporting of items related to evidence, recommendation, funders and COI management over the last 10 years are unclear; however, the increased focus on adequate reporting in the health sciences may be having an impact. Improvements over the last decade in the quality of processes and methods used in WHO guidelines have also been reported  and may The organisations involved in guideline development may influence reporting  and we fOur findings have implications for guideline developers in other organisations, suggesting common areas where improvements are needed, regardless of the nature of the guideline producer. In addition, end users of WHO and other guidelines need to be alert to suboptimal reporting when they assess, select and implement guidelines. Just as reporting has improved for clinical trials following the publication and widespread uptake of CONSORT , we hopeWHO guidelines have rather unique processes, requirements and specifications  such as Some information could only be found in an older version of a WHO guideline. For instance, the main recommendations and list of abbreviations and acronyms in a 2015 HIV guideline  could onSome sub-items in the RIGHT reporting checklist comprise multiple components. For example, item 11b concerns reporting the use of existing systematic reviews in guidelines, including search strategies, selection criteria, risk of bias and updating information. We found that many guidelines reported some but not all of the listed criteria and thus it was difficult to provide a single assessment of this sub-item. This issue has been noted for other reporting tools such as the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement , 37. As There are several limitations to our study. First, we included a limited cohort of WHO guidelines, including only guidelines approved by the GRC, where the first version was published in English. WHO produces some guidance documents that are not submitted to the GRC and produces guidelines in other languages (in addition to the translation of many English-language guidelines into other languages). Second, we had to adapt RIGHT for item 13a regarding clear, precise, actionable recommendations because we could not make an overall judgement for each guideline as each individual recommendation should be evaluated on its own merits , 41 and This is the first paper to investigate the reporting quality of WHO guidelines and our team is now working on a follow-up study, which will include the assessment of WHO guidelines published after 2017 with further analysis comparing the quality of WHO guidelines published before and after RIGHT was published.The majority of WHO guidelines included most RIGHT reporting items and reporting quality has improved over time. Further improvements are needed in the reporting of limitations in guideline development, the role of funders, and the identification and assessment of the evidence. These findings are likely relevant to guidelines produced by other organisations, although some RIGHT items may need to be modified when applied to guidelines produced by organisations with unique formats such as WHO.


Dempster–Shafer (DS) evidence theory is widely applied in multi-source data fusion technology. However, classical DS combination rule fails to deal with the situation when evidence is highly in conflict. To address this problem, a novel multi-source data fusion method is proposed in this paper. The main steps of the proposed method are presented as follows. Firstly, the credibility weight of each piece of evidence is obtained after transforming the belief Jenson–Shannon divergence into belief similarities. Next, the belief entropy of each piece of evidence is calculated and the information volume weights of evidence are generated. Then, both credibility weights and information volume weights of evidence are unified to generate the final weight of each piece of evidence before the weighted average evidence is calculated. Then, the classical DS combination rule is used multiple times on the modified evidence to generate the fusing results. A numerical example compares the fusing result of the proposed method with that of other existing combination rules. Further, a practical application of fault diagnosis is presented to illustrate the plausibility and efficiency of the proposed method. The experimental result shows that the targeted type of fault is recognized most accurately by the proposed method in comparing with other combination rules. Multi-source data fusion is the process of combining data obtained from different sources to make robust and complete evaluation on the certain system. As is known, single data source cannot provide sufficient information to detect a complex system in a full scale. By contrast, multi-source data fusion presents comprehensive and credible results after integrating groups of data that reflect various features of the system ,2. ThereDempster–Shafer evidence theory (DS evidence theory), firstly proposed by Dempster  and lateIn particular, by taking advantage of the belief entropy to quantify the information volume of the system and belief divergence to measure the difference among multi-source data, the credibility and the information volume, as two important factors of evidence are integrated to allocate the weight on the original evidence. In this way, the weight of untrustworthy evidence is declined, so that the influence of conflicted evidence is controlled more strictly. The main steps of the proposed procedure are concluded as follows. Firstly, the credibility of evidence is calculated according to their similarity with the average evidence. Besides, the belief entropy is applied to calculate the information volume of each piece of evidence. After that, the weight is allocated on the evidence based on their credibility and information volume before the weighted average evidence is generated. Finally, classical DS evidence is used to fuse the modified evidence multiple times and the final result is obtained. A practical application of fault diagnosis is given to prove the efficiency of the proposed method.The following parts of this paper are organized as follows. In In this section, several preliminary theories are briefly introduced, including DS evidence theory, belief entropy and Belief Jenson–Shannon divergence.Dempster–Shafer evidence theory is an extension of the Bayes probability theory. Comparing with probability theory, DS evidence theory can not only assign the probability on one single element, but also on the subset of the universal set ,58. BesiDefinition 1 (Frame of discernment).Θ is an exhaustive and finite set of all possible, independent and exclusive values of variable x, indicated by:Assume Θ is denoted as a frame of discernment. The power set of Θ is where position ,66,67,68Definition 2 (Basic probability assignment).Θ, assume a mapping m: On the frame of discernment then function m is called mass function or basic probability assignment (BPA).A [A is called a focal element of m [In DS evidence theory, thesis A ,70,71. Ient of m ,73.Definition 3 (Belief function).The belief function is a mapping which represents the total belief on A ,75. BeliDefinition 4 (Plausibility function).The plausibility function is a mapping which represents the undeniable degree on A ,77. PlauDefinition 5 (DS combination rule).Assume wheren pieces of BPAs are fused simultaneously, which satisfies the mathematical communication law and the associative law. In Equation BJSStep 1-3. Since the similarities of the pieces of evidence are negatively correlated with their divergences, if the divergence between two pieces of evidence is higher, they have lower similarity. The divergence between Step 1-4. If a piece of evidence is highly similar to the average BPA, it means that the evidence is supported by most of the other pieces of evidence and it is more reliable, thus it gains high credibility. Thus, the credibility weight of the pieces of evidence is determined by normalizing their similarity with the arithmetical average BPA. The credibility weight Ed*Step 2-2. To avoid assigning zero weight to the evidence whose belief entropy is zero, the information volume in Step 2-1 is modified as follows:Step 2-3. Calculate the information volume weight (Step 3-1. Based on the credibility weight and information volume weight of evidence, the weight of each piece of evidence is adjusted as follows:Step 3-2. Normalize the modified weight as follows:Step 3-3. Generate the modified evidence by calculating the weighted average sum of BPAs, which is defined as:Step 3-4. DS combination rule is used based on , then thIn this section, the proposed method is compared with other existing combination rules by a numerical example in  and the Assume the frame of discernment Step 1-1. Calculate the arithmetical average BPA.Step 1-2. Calculate the BJS divergence between each piece of evidence and Step 1-3. Calculate the similarity degree of each piece of evidence.Step 1-4. Calculate the weight of credibility.Step 2-1. Calculate the belief entropy of each piece of evidence.Step 2-2. Adjust the information volume of each piece of evidence.Step 2-3. Calculate the weight of information volume.Step 3-1. Adjust the weight of each piece of evidence.Step 3-2. Normalize the modified weight of each piece of evidence.Step 3-3. Generate the modified evidence.Step 3-4. Use the classical DS combination rule to fuse the modified evidence four times, and the result is shown in B, while the remaining four pieces of evidence all support A. In this case, the major belief after fusing should be allocated on A since the According to C wrongly. The remaining combination rules all present the reasonable results and majorly support A. The incredible evidence C, but as the later credible pieces of evidence join the fusion process, these combination methods all turn to assign their beliefs mainly on A. However, in the real situation, a slight increase of accuracy is significant to improve the performance of the system [A among these combination rules. Therefore, the proposed method is relatively effective because it takes two important factors of evidence—the credibility and information volume—into consideration, so that the weight of the incredible evidence is controlled more strictly.As shown in e system ,55,56. CAn automobile system was excessively used and it caused shortage of power. According to the records in the database, there were three possible faults that may lead to this problem: low oil pressure, air leakage in the intake system and a stuck solenoid valve, which are denoted as Step 1-1. Calculate the arithmetical average BPA.Step 1-2. Calculate the BJS divergence between each piece of evidence and Step 1-3. Calculate the similarity degree of each piece of evidence.Step 1-4. Calculate the weight of credibility.Step 2-1. Calculate the belief entropy of each piece of evidence.Step 2-2. Adjust the information volume of each piece of evidence.Step 2-3. Calculate the weight of information volume.Step 3-1. Adjust the weight of each piece of evidence.Step 3-2. Normalize the modified weight of each piece of evidence.Step 3-3. Generate the modified evidence.Step 3-4. Use the classical DS combination rule to fuse the modified evidence four times, and the result is shown in based on . After fIn this application, As shown in lication ,55,56. WDempster–Shafer evidence theory is widely used in information fusion field due to its efficiency in handling uncertainty and imprecision. However, some counter-intuitive results occur when the evidence is highly in conflict. To address this shortcoming, a novel multi-source combination method is proposed in this paper based on BJS divergence measure and the improved Deng entropy. Not only the credibility but also the information volume of the evidence is taken into consideration to allocate the weight on the original evidence before fusing the modified evidence, so that the influence of untruthful evidence is controlled more seriously, resulting in the higher attention on the credible evidence when fusing. Next, the proposed method is compared with other existing combination rules in a numerical example. The result shows that the proposed method achieves the highest accuracy of 99.74% among these combination rules. Furthermore, the proposed method is applied in an application of fault diagnosis to identify the type of fault in an automobile system. Among the combination rules which successfully recognized the correct type of fault, the proposed method shows the highest degree of accuracy of 99.34% with lower computational complexity since the sizes of two vectors are both


As the basis for managing the risks of chemical exposure, the Chemical Risk Assessment (CRA) process can impact a substantial part of the economy, the health of hundreds of millions of people, and the condition of the environment. However, the number of properly assessed chemicals falls short of societal needs due to a lack of experts for evaluation, interference of third party interests, and the sheer volume of potentially relevant information on the chemicals from disparate sources.In order to explore ways in which computational methods may help overcome this discrepancy between the number of chemical risk assessments required on the one hand and the number and adequateness of assessments actually being conducted on the other, the European Commission's Joint Research Centre organised a workshop on Artificial Intelligence for Chemical Risk Assessment (AI4CRA).The workshop identified a number of areas where Artificial Intelligence could potentially increase the number and quality of regulatory risk management decisions based on CRA, involving process simulation, supporting evaluation, identifying problems, facilitating collaboration, finding experts, evidence gathering, systematic review, knowledge discovery, and building cognitive models. Although these are interconnected, they are organised and discussed under two main themes: scientific-technical process and social aspects and the decision making process. The four elements of CRA, i.e. hazard identification, hazard characterisation, exposure assessment and ultimately risk characterisation , depend Similar efforts are being reported from the International Collaboration for the Automation of Systematic Reviews (ICASR) 2A JRC-organised workshop, titled “AI4CRA – Artificial Intelligence for Chemical Risk Assessment”, was held in Ispra, Italy on 4–5 April 2018. The participants were scientists from both the AI and the CRA fields from Aalto University (Finland), Lancaster University, Leeds University (UK), the European Food Safety Authority (EFSA) and the JRC. The participants explored how, and to what extent, applying AI methods could support both the quantitative  and the qualitative  aspects of CRA.•identifying problems•gathering evidence•systematic review•knowledge discovery•supporting evaluation•finding experts•facilitating collaboration•process simulation•cognitive modellingThe following topics were identified as prime candidates to be examined more closely when further efforts are undertaken to make AI an enabling technology in CRA:3•Scientific-technical evaluation process•Social aspects and the decision making processIn the weeks following the workshop, the authors of this manuscript continued their discussion about the most promising areas that could be supported, facilitated or even made possible via the integration of AI. Based on the topics identified in the workshop, and following some regrouping, merging and additional input, two major overarching themes were identified:While the first theme touches upon more content-related issues like actually identifying problems to be addressed with CRA, gathering the scientific evidence via e.g. AI-supported systematic review and general knowledge discovery, the second focuses on the social interaction between actors and stakeholders by enhancing the human-led evaluation process and introducing the notion of simulating and exploring human expert judgement:The level of AI-readiness of the individual topics , which are gaining visibility as a useful systematic analysis product. In brief, evidence maps use systematic review methods to identify the amount and type of evidence available to address a particular topic 4.1.2The collation of relevant evidence and evaluation of its quality and suitability for use in the assessment process are crucial steps at the beginning of a CRA.AI could be trained to crowd-source multi-disciplinary expertise, and add complementary and useful knowledge normally not considered in a specific context.New streams of data from biomonitoring and epidemiological studies can contribute to the overall picture In line with Wilson’s statement that “we are drowning in information, while starving for wisdom” Not only scientific literature, study reports and submitted dossiers should be considered for CRA, but other valuable “unofficial” sources of information should also be included and mined, such as “grey” literature and other information from the web and social media, e.g. twitter or blogs. These could contribute to identifying trends or emerging risks, by providing information for example on actual use (or misuse) of, and exposure to, chemicals.AI could contribute to building good quality data for CRA. In addition to analysing evidence more efficiently, AI could identify data gaps and help to formulate the corresponding research questions. AI might contribute to setting up the experimental design in the most efficient and effective way to close the knowledge gaps.The main obstacle to keeping on top of research is that scientific papers are being published at ever increasing rates. To keep abreast of all research which is relevant to a topic or policy area is impossible without some means of automating the aggregation and summarisation of the content. Without technology, research synthesis efforts remain limited and incomplete.Example 2: systematic reviewIn the near term, AI could dramatically increase the amount of conducted systematic reviews (SRs), which in the long run could lead to a culture of evidence-use based primarily on the systematic gathering and appraisal of evidence streams. This would reduce the risk of a 'single study' influence on policy decisions, and ease the process of obtaining scientific consensus. They are both vital for areas where facts are heavily contested and opinions highly polarised.An SR aims to provide a complete summary of current evidence which is relevant to answering a research question. They are characterised by the effort made to minimise the risk of bias in the evidence review process, emphasising sensitive search strategies, comprehensive inclusion criteria, and transparent and reproducible approaches to critically appraising existing evidence and synthesising it into summary results .While the transparency and the minimised risk of bias of an SR has a strong appeal as a technique for developing the methodological quality of CRA, they are lengthy, labour-intensive projects to undertake, which require a great deal of specialised skill across multiple disciplines to complete successfully. This is proving a significant barrier to their uptake in regulatory contexts, where time-frames are often relatively short, and capacity often limited in comparison to the requirements of the methodology.AI stands to shift this balance in many ways, potentially at every stage of the review process. AI can facilitate each individual step see  of an SRExample 3: evaluating complex exposure scenariosAI could help with complex evaluations, for example to conclude on possible effects of combined exposure to chemicals and to simulate exposure to chemicals for a multitude of different possible exposure scenarios and probabilities – across different sectors and pathways, such as via food, consumer goods, occupational and environmental exposure, and for different populations in terms of geography and susceptibility, and over different periods of time. Similarly, in the direct assessment of chemical mixtures, even in the ideal case that the effects were known for all individual chemical components, the sheer number of possible combinations of several compounds in many possible concentrations is too high as to allow a simulation of the mixtures and measurement of the effects of all these theoretically possible combinations in a laboratory In the near term, AI machine-reading tools will enable the characteristics and results of hundreds of thousands of studies relevant to assessing chemical health risks, currently locked away in individual manuscripts, to be converted into composite, queryable databases. These databases can be set up with flagging systems for regulators, automatically identifying new evidence streams which should be analysed, and they should be queryable in terms of concepts essential to their decision-making process. This way, risk managers can be confident that they are acting on all relevant evidence, not just part of it, and their reasoning for focusing on one part of the evidence base rather than another can be made fully transparent.4.1.3As defined in the Commission Communication Information from distinct sources might give new insights when brought together. However, so far there might be no awareness about the existence of and/or connections between these different sources. AI could help to screen and systematically map the “universe of information”, extracting information also from non-easily processable sources, or hidden in scanned documents or graphs, and subsequently to combining them. Pattern recognition techniques can help to make connections and identify relationships, which would be too complex to be recognised by human screening and processing, and support discovering correlations. Furthermore, very different types of data/information have to be integrated to enable conclusions and decision making. AI could help by making sense of data and building knowledge. An example is the Adverse Outcome Pathway (AOP) approach for systematically describing mechanisms of chemical toxicity, and the possibility to find new AOPs or build interconnected AOP networks, based on many scattered pathways or fragments of pathways AI could be trained to identify other substances that could inform the CRA, if identification of similarity in chemical structures, biological behaviour or nature of the available data, would be made. Once enough data have accumulated in the chemical risks database, AI could be used to build multidimensional models of patterns latent within the data (such as trends in toxicity), and compare the “profiles” of different chemicals. By comparing such profiles, risk assessors will be able to base predictions of health risks such as carcinogenicity on the basis of all the available evidence, including hidden similarities.The authors recognise multiple initiatives exist in this space, e.g. at ICASR, US EPA, NTP, who are working on operationalising crowd sourcing options with a focus on ensuring interoperability of tools, onboarding of AI, development of training data sets for model development, mapping of various vocabularies into an ontology framework that can be used to help with AOP development and other evidence synthesis activities, and incorporation of tenants of systematic review into the AI pipeline. As a result of the workshop, JRC is reaching out to various actors to encourage further collaboration and streamlining of efforts.4.24.2.1After aggregating the evidence, the next step in a CRA is evaluation. The compiled lines of evidence are assessed, considering the quality of the data, their reliability and associated uncertainties, as well as the relevance for the problem formulated, in order to make an overall judgement. The assessment needs to integrate results from different types of approaches and should be performed in an unbiased way while weighing their respective contributions and handling possibly conflicting results. Evaluation is typically an expert judgement driven process, with expert knowledge and experience being required to make decisions regarding the available data and the conclusions based on them.Different groups of experts are making CRAs in different legal frameworks based on the data made available at a certain point in time. In hindsight, it is often difficult to understand whether and what data was actually considered and how different data was weighed to lead to final conclusions and recommendations regarding the possible or restricted uses of a substance. The discrepancy between decisions taken by different expert groups can lead to mistrust in the resulting CRAs and expert recommendations. Generally, AI could play a role in carrying out the evaluation process in a more neutral, consistent and transparent way.AI could learn how to perform a CRA which could then be peer-reviewed by real experts, and this would be the basis for an iterative learning process for both the machine and the experts.4.2.1.1In terms of expert panel work, there are several ways in which AI could support the CRA process. For example, since the success of a group decision process depends on the dynamics of the group, as well as the complementary strengths and areas of expertise of its members and their knowledge of the topic, and on avoiding potential bias due to the experts’ background, finding an optimal mix of experts is a crucial task which AI could support. A selection based on limited or biased selection criteria, or only based on already established networks, can reduce the chance of finding the right balance of experience, competence and different views which are beneficial for a fruitful discussion. AI could be used to create a mapping system that takes into consideration not only the potential experts’ publications on the topic, but parameters that are normally not considered and difficult to retrieve manually, such as membership in other similar panels, presentations at conferences, presence in committees, etc. Combined with a broad and up-to-date search for the information to map the expertise, this process could reduce the risk of bias and broaden the pool of possibilities, not relying on personal experience or potentially biased judgement criteria of the selectors.4.2.1.2Furthermore, AI could monitor or guide the process in an interactive Weight of Evidence (WoE) and thus help to rationalise the decisions made in the process. A possible direction is to use AI as an external objective observer. It could automatically create and maintain an organised, up-to-date record of topics discussed, and provide further context by linking to relevant information from available sources. This AI-driven external observation could take the form of a dynamic ‘mind map’, generated on the fly as the work of the group progresses, following and facilitating the process in a transparent way. AI would have the ability to cope with large amounts of data regarding individual chemicals, similar chemicals, chemicals in combination, experience made in earlier risk assessments, recognising differences between different evaluations of one chemical or of similar chemicals in different contexts.AI as an instant feedback instrument could also point out all possible variables and their impact, the pros and cons of different options that the group may be faced with, as well as the consequences from the judgements made for the final outcome. This would include tools to explain consequences of decisions thereby informing the weighing of economic and health related impacts.In terms of evaluating uncertainties in CRA, AI can take the probabilistic assessment to a next level by having uncertainties explicitly at the core of the risk assessment methodology.Another direction is a more active involvement of AI in the group decision process, by directly interacting with the members of a group. The contribution of AI could even consist of being a full member of the expert team and contributing an additional expert opinion, drawn from the available data. Going beyond the role of making evaluation more efficient, AI could develop into an overarching universal expert entity in itself, by learning about human expert decisions based on the available evidence. AI would build a collective memory of and accumulate the experience from many experts and learnings from previous cases. Furthermore, AI could derive the implicit knowledge from experts in the first place. This could be valuable for industry and regulators because the potential loss from key experts missing risk assessments could be mitigated. The accumulated knowledge from all AI-assisted group decision processes could act as a shared memory or knowledge resource.4.2.24.2.2.1Once the scope of a CRA is defined, the CRA process is based on the four steps of hazard identification, hazard characterisation, exposure assessment, and risk characterisation . Since t4.2.2.2Human processes, especially those related to decision making and government, are often assumed to be rational and logically structured. The calls for transparency of decision making are motivated by the need to scrutinise the process, i.e. to verify how much it follows this clear, predefined, logical pathway of reasoning. Certainly, this assumption of rationality is prevalent in strongly codified areas such as risk assessments, but also holds true in other complex co-decision processes where many sources of knowledge, actors and interests are involved.However, the assumption that decision making is mostly rational and strongly relies on an objective analysis of facts is often discussed as simplistic It is unlikely, however, that the decisions are made in a completely haphazard way. There must be underlying mechanisms of how they happen, with patterns of interacting factors contributing to particular results. Therefore, it can be beneficial to better understand our cognition and group dynamic in the context of chemical risk assessment, science advice and decision making. As risk assessment processes and precautionary principle are often contested, a constructive rethinking of them would benefit from a better understanding of the factors at play and their interconnections.For complex processes, a single person, or even a team routinely involved in them, will have difficulty in systematically auto-analysing these patterns of human thinking, interacting, and decision making. This is where AI comes as a potential solution: it could help model cognitive and decision-making processes 1.Where the rationality is strongly expected and clearly codified, AI observations could reveal the actual complications of these assumptions, pointing out areas worth improving in the process.2.Where the process is already not fully clear , they give a tool to understand what is going on. This could be highly interesting information e.g. for researchers interested in providing their input to policy effectively, as they often struggle with an overly simplified policy cycle model.The objectives would slightly vary, depending on scenarios:Models produced by AI could provide empirical evidence of the functioning of decision-making processes. Through such cognitive models, weaker and stronger points of complex processes can be identified and a more thorough reform can be foreseen. If focused specifically on scientific support, it could help map the best ways of achieving impact with evidence.In practical terms, it would require a sufficient number of observable cases , which requires a lot of trust on the side of various institutions who manage or participate in these processes.Full exploitation of all benefits of AI for CRA cannot be a short term operation: Prior to use for decision-making, users will need to see evidence in the quality of AI for accurate collection of information reported in studies  as well as from data sources that may not be reported in single studies (high throughput screening results). Also, the development of AI-based evidence integration approaches will have to be done with awareness of structured frameworks being used in environmental health for weight of evidence .5The workshop participants concluded that AI can support and facilitate important processes and activities in the multifaceted CRA discipline. While the introduction of AI to support CRA will not be a trivial undertaking, the current problems that CRA is facing  will not be solved by conventional means. “More of the same” will not lead to better and reliable CRA, but AI offers the opportunity to support CRA in innovative ways. These include supporting the gathering and analysis of information as the basis for the CRA, facilitating different aspects of expert evaluation, including AI as an additional expert around the table, to simulating all possible assessment pathways and the range of possible outcomes, as well as creating new knowledge and cognitive models of decision-making.AI assisted innovation in evidence gathering could therefore build an automated system of transforming data into knowledge, with much more efficiency. Freed up brain power could be used to identify important gaps in knowledge and frame the right questions. Altogether this leads to a more robust evidence base for policy making, and hopefully to a more transparent method in the decision making process.Using AI is not only beneficial for humans, but in turn can help develop better AI. For example, it would allow developing better systems for human-AI collaboration and AI-based argumentation in the CRA process, building on the complementary strengths of human and machines. In particular, cognitive models not only help AI understand humans better  but also help developing human-understandable and transparent algorithms that can explain reasons (including emotions) behind their actions.It is expected that further exploration of the topics covered in this workshop could lead to executable plans for pilot projects, with a view to eventually increasing the efficiency and effectiveness of the CRA process. However, the only way to fully realise the applications of AI for CRA is global and discipline-crossing collaboration. The AI method development and deployment needs (including associated user dashboards) are too large and complex to be developed by any single entity. The development of AI should include a focus on interoperability of tools to best take advantage of the talent and resources available in both the environmental and biomedical sciences.The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.


Evidence-based practice is among core competencies of health care professionals (HCPs). However, the levels of evidence-searching capability may differ among various disciplines of HCPs as they receive different education and trainings for various durations in medical schools and teaching hospitals.This study aimed to compare the evidence-searching capability among different disciplines of HCPs and identify which aspects need to be reinforced. From a teaching hospital, we recruited 80 HCPs of various disciplines and compared their evidence-searching capability by using a validated scale. To examine if sex and education levels affect evidence-searching capability, we performed a multiple linear regression analysis with collinearity diagnostics.P < 0.05). No collinearity was detected between discipline and age nor level of education. Except for the 2nd formative assessment item , participant’s levels of education did not affect evidence-searching capability. Age was associated with lower evidence-searching capability in five formative and the summative assessment items.Physicians and pharmacists performed significantly better than other disciplines in the seven formative assessment items and the summative item (all We found a better evidence-searching capability among physicians and pharmacists than other HCPs who may require more training on evidence-searching skills. Also, evidence-searching skills training should be provided to HCPs irrespective of age and education levels. Over the past few decades, evidence-based medicine (EBM) has become an important issue in clinical practice, medical education, and clinical research. Also, evidence-based practice has been listed by the US Institute of Medicine as one of five core competencies of health care professionals (HCPs). EBM aims to apply the best evidence in resolving clinical uncertainty and efficiently provide effective health care . The praThere are various disciplines of HCPs including physicians, registered nurses, pharmacists, allied health professionals such as respiratory therapists and dietitians. To maintain the normal operation of health care industry, administrative staff, clinical teachers, and research staff are also needed. They receive different education and trainings for various durations in medical schools and teaching hospitals. Also, the college entry requirements vary among different disciplines of HCPs, with physicians usually the strictest. Therefore, the levels of evidence-searching capability may differ among various disciplines of HCPs. The objective of this study was to compare the evidence-searching capability among different disciplines of HCPs and to identify which aspects need to be reinforced in EBM education.This study was a sub-analysis of a previous study in which a scale for measuring evidence-searching capability has been developed and validated . From a We compared the evidence-searching skills between different disciplines of HCPs and analyzed the 15 items and 1 summative rating item, respectively, by using the Kruskal-Wallis test with post-hoc Dunn’s test. To examine if age group and level of education affect evidence-searching capability, we performed a multiple linear regression analysis with collinearity diagnostics. Collinearity refers to near perfect linear combinations of two variables and multicollinearity involves more than two variables, leading to unstable regression estimates with high standard errors. We used variance inflation factor (VIF) to evaluate multicollinearity, with a VIF of > 10 indicating multicollinearity . We consn = 9). The data on the evidence-searching capability of the 80 participants, including 23 men (28.8 %) and 57 women (71.3 %), were collected and analyzed. The disciplines and education levels of participants are presented in Table P < 0.05). By using the post-hoc Dunn’s test, physicians and pharmacists did not differ in these item . However, both physicians and pharmacists differed from registered nurses, allied health professionals, and others in most of these items .The data on the 15 formative assessment items and 1 summative assessment item of participants are shown in Table P = 0.04), participants’ levels of education did not significantly affect evidence-searching capability. Senior age (> 35 years) was negatively associated with evidence-searching capability in five formative assessment items  and the summative assessment (S1) item.As shown in Table This study found a higher level of evidence-searching capability of physicians and pharmacists than other disciplines of HCPs, especially in the building up of search terms, prioritized use of secondary databases, use of both Medical Subject Headings (MeSH) term and free text in searching, and devising search strategy by using appropriate Boolean operators. Also, physicians and pharmacists had a better overall ability in evidence-searching than other HCPs. Therefore, more EBM trainings including evidence-searching capability in the aforementioned skills should be provided for HCPs, especially those disciplines other than physicians and pharmacists.The better evidence-searching capability found in physicians and pharmacists may be related to the different levels of contact with latest evidence. When compared to other disciplines of HCPs, physicians and pharmacists are in greater demand for up-to-date evidence on clinical trials as newly developed drugs are springing up over the past several decades. Therefore, physicians and pharmacists may have more experiences in searching databases than other HCPs.Compared to HCPs of younger age, senior HCPs should have possessed better evidence-searching capability because of having a better understanding of the context of the clinical PICO question. By contrast, one previous study suggested younger age as a facilitating factor for evidence-based practice . Also, oThere are several limitations in this study. First, this pilot study was a sub-analysis and the participants were recruited from a single teaching hospital. However, the HCPs received education and training from various medical schools and teaching hospitals. Therefore, our findings may represent a miniature of EBM education. Also, one previous nationwide survey of 605 physicians and 551 registered nurses in 61 hospitals across Taiwan found that physicians possessed better self-rated evidence-acquiring skills than registered nurses . Second,Our study suggested a higher level of evidence-searching capability among physicians and pharmacists than other HCPs. HCPs other than the two disciplines may require more training on evidence-searching skills, especially in the building up of search terms and strategy as well as prioritized use of appropriate secondary databases. Also, our study illustrated that senior age and graduate education did not increase evidence-searching capability. Therefore evidence-searching skills training should be provided to HCPs irrespective of age and education levels.


People who plan ahead typically fare better during the response and recovery phases of a disaster. However, problems arise when the needs, wants, abilities, and resources of vulnerable people are not adequately considered. The lack of alignment between the literacy demands of existing materials and the literacy skills of many vulnerable subgroups limits their ability to understand and effectively use potentially life-saving information. Existing health literacy models that have demonstrated effectiveness in changing health behaviors and improving outcomes is a first step to reducing disaster-related morbidity and mortality in low resource and low literacy areas. This presentation will 1) describe how interdisciplinary collaborations can be used to address this public health issue, 2) explain how health literacy techniques can be applied when developing disaster materials, and 3) present research data on a social marketing campaign to improved disaster preparedness of older adults. Part of a symposium sponsored by Disasters and Older Adults Interest Group.


Uncertainty is integral to evidence-informed decision making and is of particular importance for preference-sensitive decisions. Communicating uncertainty to patients and the public has long been identified as a goal in the informed and shared decision-making movement. Despite this, there is little quantitative research on how uncertainty in health information is perceived by readers.The aim of this study was to examine the impact of different uncertainty descriptions regarding the evidence for a treatment effect in a written research summary for the public.We developed 8 versions of a research summary on a fictitious drug for tinnitus with varying degrees (Q1), sources (Q2), and magnitudes of uncertainty (Q3). We recruited 2099 members of the German public from a web-based research panel. Of these, 1727 fulfilled the inclusion criteria and were randomly presented with one of these research summaries. Randomization was conducted by using a centralized computer with a random number generator. Web-based recruitment and data collection were fully automated. Participants were not aware of the purpose of the study and alternative presentations. We measured the following outcomes: perception of the treatment effectiveness (primary), certainty in the judgement of treatment effectiveness, perception of the body of evidence, text quality, and intended decision. The outcomes were self-assessed.P=.25 and P=.73), but we found a global effect for Q3 (P=.048). Pairwise comparisons showed a weaker perception of treatment effectiveness for the research summary with 3 sources of uncertainty compared to the version with 2 sources of uncertainty (P=.04). Specifically, the proportion of the participants in the group with 3 sources of uncertainty that perceived the drug as possibly beneficial was 9% lower than that of the participants in the group with 2 sources of uncertainty . The proportion of the participants in the group with 3 sources of uncertainty that considered the drug to be of unclear benefit was 8% higher than that of the participants in the group with 2 sources of uncertainty . However, there was no significant difference compared to the version with 1 source of uncertainty (P=.31). We did not find any meaningful differences between the research summaries for the secondary outcomes.For the primary outcome, we did not find a global effect for Q1 and Q2 (Communicating even a large magnitude of uncertainty for a treatment effect had little impact on the perceived effectiveness. Efforts to improve public understanding of research are needed to improve the understanding of evidence-based health information.German Clinical Trials Register DRKS00015911, https://www.drks.de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00015911RR2-10.2196/13425 Uncertainty pervades health care and is integral to evidence-informed decision making. Helping patients and consumers to understand and deal with uncertainty has been identified as one of the goals of the shared decision making and informed choices movement . UnderstQuantitative research on how to communicate uncertainty regarding the benefits and harms of treatments to patients and the public is limited. A systematic review by the Agency for Healthcare Research and Quality identified 8 controlled studies with 9 comparisons, including 6 randomized trials . Four ofFrom our experience in providing evidence-based health information to the German public, enabling informed decisions often requires providing information on other types of uncertainty. For example, a patient with subacromial pain syndrome may wonder whether surgery using subacromial decompression may help with his/her condition. Two randomized, sham-controlled studies have shown no benefit of this treatment. However, few people in these trials had a hooked type III acromion. Thus, there is uncertainty whether this subgroup of patients may benefit from such a treatment. Whether such information is presented or not may impact readers in different ways and affect their choices. We are not aware of any studies investigating whether the perception of treatment effectiveness depends on the degree, type, and magnitude of uncertainty presented in written health information.In order to test the main way in which we communicate uncertainty, we conducted a study addressing the following 3 questions.Q1. Degree of uncertainty: Do members of the public perceive treatment effects differently depending on the certainty of the treatment effect (higher versus lower), which are expressed by different wordings (“studies show” or “studies indicate”)?Q2. Type of uncertainty: Do members of the public interpret uncertain treatment effects differently depending on the type of uncertainty ?Q3. Number of sources of uncertainty: Is there an additive effect when multiple sources of uncertainty are presented (1-3 sources of uncertainty)?We hypothesized that higher degrees of uncertainty expressed through different wordings or a larger magnitude of uncertainty reduce the perceived effectiveness of a treatment, the certainty in this judgement, and the intention to use the treatment. We further hypothesized that uncertainties due to vested interests/publication bias reduce perceived effectiveness to a larger extent than other sources of uncertainty. Lastly, we hypothesized that the ratings of text quality decrease with an increasing magnitude of uncertainty.We investigated the 3 research questions by using 8 variations of a research summary for a fictitious drug for tinnitus that was set in a hypothetical treatment decision scenario. We designed the experiment as a randomized superiority study with 8 parallel groups allocated in an equal ratio (between-group design). This trial has been registered in the German Clinical Trials Register (DRKS00015911). The study protocol has been published elsewhere and includes the technical details of the study conduct .We recruited members of the German public from a web-based research panel. Panel members were eligible if they were at least 18 years of age and able to read and write German. There were no other restrictions. We used a quota to ensure equal representation of different age groups (below and above 45 years) and sex. Once a quota cell was full, enrolment for this quota was closed.After enrolment, the participants were provided with a short introduction to the study and an informed consent sheet. Panel members who agreed to participate were asked for information on age, sex, and educational degree based on the German school system . We then asked them to imagine having tinnitus and having unsuccessfully tried several treatments. Participants were then randomly presented with one of the 8 versions of the research summary. These contained brief information on the medical condition and a short summary of the evidence for a fictitious new tinnitus medication (“Oroxil”). After the presentation of the research summaries, we collected data on the different outcomes by using a questionnaire developed for the purpose of this study. We allowed participants to refer to the research summary as needed while answering the questions. Before the conclusion of the experiment, participants were asked about their profession and history of tinnitus.We developed 8 variations of the research summary based on standards and use of language in providing evidence-based health information to consumers through Germany’s statutory health website . Accordicertain treatment effect and version B a possible but not a certain treatment effect (indication of effect). Version B1 was identical to B, but it included an additional statement on the need for further research. The wordings are based on the methods for the assessment of treatment benefits developed by the German Institute for Quality and Efficiency in Health Care (IQWiG)  and B4 [imprecision]). The results of the sensitivity analysis using ANOVA were consistent with those of the Kruskal-Wallis tests for the secondary outcomes. The results of the sensitivity analyses including straightliners and speeders were consistent with the main findings. The detailed results of the secondary outcomes are presented in The variations of the research summary had little impact on the secondary outcomes. There was a significant overall effect of the perception of effectiveness), we did not find any statistically significant differences between alternative wordings for different degrees of uncertainty (Q1) and between different sources of uncertainty (Q2). However, there was a significant effect of the number of sources of uncertainty (Q3). As the differences between the groups in Q3 were small and the P value was just below the significance level, we believe that this result should still be interpreted cautiously. Furthermore, we did not see a dose-response effect of the number of sources of uncertainty on the perception of effectiveness. A possible explanation is that indirectness, which was added in the version of the research summary with 3 sources of uncertainty, is of particular importance to people. This seems plausible, as it is a tangible source of uncertainty, easy to understand, and relates to them personally. However, this type of uncertainty did not decrease the perception of treatment effectiveness in Q2 to a statistically significant degree.For the primary outcome .The different presentations of uncertainty had no meaningful effect on the secondary outcomes, including the intention to take Oroxil. While the Psychological research suggests that discounting cues  prior to the provision of information can reduce the impact of that information . Thus, oThe results may also be explained by the neutral choice of language in the research summaries, thereby resulting in subtle differences between some of the variations. While stronger wordings may be needed to convey the uncertainties, we do not generally recommend the use of a more partial style of language as this would counteract the aims of providing balanced information and supporting informed decision making. This said, there are exceptions to this rule, for example, in case of highly implausible treatments such as homeopathy or new high-risk treatments lacking good evidence. On a more critical note, use of words may be unsuitable to convey the quality of evidence in general. While we identified little experimental research addressing similar questions, 1 study with physicians did not show an effect of different wordings such as “might” or “suggest” on conveying the strength of the recommendation within guidelines .Lastly, it is possible that in a competitive choice situation, where there are 2 treatments, uncertainty information may matter more (ceteris paribus) because it helps to discriminate drugs from each other. This may be particularly true for the outcome intended treatment decision, where, in our study, participants in all groups tended to opt for taking the drug. A possible explanation for the lack of differences between the groups is that the scenario suggested that there are no other proven treatments for tinnitus.BRCA-positive gene mutations looked at ribbons to illustrate levels of evidence (4 grades from bronze to platinum). In this study, the users assumed that if data are presented using numbers and graphs, people will automatically consider them to be of sufficient quality [On an average, 52.4% 832/1588) of all the participants considered the treatment to be possibly helpful, and about 30.4% (483/1588) considered the benefit of the drug to be unclear, including the group without any expressed uncertainty (A). A possible explanation for this result is that the use of numbers to communicate the response rate for the treatment may have diluted the effects of the uncertainty descriptions. This raises the question of whether communicating certainty in the quality of evidence in parallel with probability information is the actual challenge for communicators rather than communicating uncertainty. The explanation seems likely since the public is still not used to such presentations and probably has difficulties in distinguishing between uncertainty from limitations in the quality of evidence (the certainty in an effect estimate) and uncertainty in terms of dealing with probabilistic outcomes (the magnitude of an effect). This has been shown in a qualitative study conducted to improve Cochrane plain language summaries . Users i88 of all quality . These e quality .A possible but, in many cases, an inappropriate solution would be to avoid the use of numbers. Presenting quantitative information on treatment effects has become a standard requirement for evidence-based health information and decision aids because probabilities on the benefits and harms support realistic expectations and allow patients to weigh the pros and cons . FurtherLastly, providers of evidence-based health information should carefully consider whether the quality of the evidence is sufficient for an effect to be presented at all. While fine-graded levels of evidence such as in the GRADE system are useful to health care professionals, they may be too nuanced for patients and the public. Instead, it may be more sensible to use a higher cut-off for presenting treatment effects  while labelling low or very low quality of evidence to be unclear. This seems reasonable as the true effect may substantially differ from the study estimate for evidence of low or very low quality according to the current definition within the GRADE system .While there are many possible solutions to dealing with uncertainty, it is unlikely that one will fit all scenarios in practice. How uncertainty is presented may depend on many factors such as the format and aims of the health information, the target group, the severity of the condition, the risks of the intervention, and the availability and number of treatment options. Thus, and maybe most importantly, the judgements made in this process should be carefully reflected.The main strengths of our study are the large sample population based on a sample size calculation for a conservative effect, few missing data, and a robust protocol-driven methodology. Furthermore, we used a research summary that is concordant with the current standards of evidence-based health information, including the use of probabilities to communicate benefits and harms in order to allow weighing the pros and cons. The weaknesses include the use of a hypothetical scenario and restriction to 1 (relatively benign) condition and treatment. Furthermore, the choice of the size of the presented treatment effects is debatable. We decided to use a small absolute effect since large treatment effects are rare in practice . In our In order for uncertainty descriptions to be useful in conveying limitations in the quality of evidence, the public has to be aware that not all research is of equal quality and that the size of an effect and the confidence in the effect are separate concepts. Therefore, it would be helpful to develop or enhance and implement interventions that explain basic research concepts and skills. A notable recent effort in this direction has been the Key Concepts for Informed Health Choices Project -26. It mIn conclusion, providers of written evidence-based health information or decision aids should not assume that consumers and patients will understand the intentions and meaning of uncertainty descriptions without further explanation or support. The most likely explanation for this is that the public does not distinguish between the concepts of quality of evidence and magnitudes of effect. The challenge in communicating uncertainty should not be used as an argument to withhold uncertainty information from patients and the public. Future research should aim to understand how the public or patients process information regarding limitations of evidences in order to understand why the degree and types of uncertainty seem to have little impact on the perceptions of effectiveness or treatment intentions.


Background: The Informed Health Choices (IHC) Key Concepts are principles for evaluating the trustworthiness of claims about treatment effects. The Key Concepts provide a framework for developing learning-resources to help people use the concepts when treatment claims are made, and when they make health choices.Objective: To compare the framework provided by the IHC Key Concepts to other frameworks intended to promote critical thinking about treatment (intervention) claims and choices.Methods: We identified relevant frameworks from reviews of frameworks, searching Google Scholar, citation searches, and contact with key informants. We included frameworks intended to provide a structure for teaching or learning to think critically about the basis for claims, evidence used to support claims, or informed choices. For a framework to be included, there had to be a description of its purpose; a list of concepts, competences, or dispositions; and definitions of key terms. We made independent assessments of framework eligibility and extracted data for each included framework using standardised forms.Results: Twenty-two frameworks met our inclusion criteria. The purpose of the IHC Framework is similar to that of two frameworks for critical thinking and somewhat similar to that of a framework for evidence-based practice. Those frameworks have broader scopes than the IHC Framework. An important limitation of broad frameworks is that they do not provide an adequate basis (concepts) for deciding which claims to believe and what to do. There was at most some overlap between the concepts, competences, and dispositions in each of the 22 included frameworks and those in the IHC Framework.Conclusions: The IHC Key Concepts Framework appears to be unique.  Our review has shown how it and other frameworks can be improved by taking account of the ways in which other related frameworks have been developed, evaluated, and made useful. Claims about what people can do to improve or protect their health (treatments) are ubiquitous. They are found in the mass media, advertisements, and everyday personal communication. Some are based on trustworthy evidence. Many are not, and many people have difficulties determining which claims to believe and act on. Acting on untrustworthy claims and not acting on ones that are trustworthy can result in unnecessary suffering and waste.4. The aim of the IHC project is to help people, particularly primary and secondary school students, learn to assess treatment claims and make informed health choices5.In response to these challenges, we developed the Informed Health Choices (IHC) Key Concepts as the first step in the IHC project6. People in other fields have found the IHC Key Concepts relevant for assessing claims about the effects of other types of interventions7. This includes agricultural, educational, environmental, management, social welfare, economic, international development, nutrition, policing, and veterinary interventions.We use ‘treatment’ to refer to any intervention or action intended to protect or improve health8. We first published the framework in 20151 and have continued to update it yearly. The current (2019) framework includes 49 concepts in three groups  for critical thinking14. Similarly, there are different definitions and frameworks for scientific thinking (reasoning and literacy)18, epistemic cognition and meta-cognition20, health literacy23, and various aspects of evidence-based health care26. There is also overlap across these different framework categories, some of which have been grouped together as frameworks for “productive thinking”12.There are many other frameworks that include concepts, competences, or dispositions that are relevant to thinking critically about treatment claims, comparisons, and choices. These include critical thinking frameworks, logical fallacies and argumentation frameworks, cognitive frameworks, frameworks for scientific thinking, and frameworks related to evidence-based health care. For each category of frameworks there are disagreements about definitions and what is included. For example, learning to think critically is widely held as an aim of educationDefinitions of terms that we use in this paper are shown inThe objective of our review was to systematically compare the IHC Key Concepts Framework to other frameworks that are relevant to teaching and learning how to think critically about treatment claims, evidence, and choices. We examined similarities and differences between the IHC Key Concepts Framework and other frameworks - particularly in the context of primary and secondary school education - including:The purposes and definitions of key termsThe elements included and domains in which they are groupedHow the frameworks have been developed and evaluatedHow the frameworks have been used to develop curricula, teaching and learning resources, and assessment tools27.We conducted a systematic review of frameworks relevant to teaching and learning to think critically about treatment claims, evidence used to support those claims, and choices. The protocol for the review is published on our websiteWe included frameworks that are intended to provide a structure for teaching or learning to think critically about at least one of the following:The basis (justification) for claims or arguments about the effects of interventions and the reliability of those justificationsThe extent to which evidence used to support claims about the effects of interventions (comparisons) is fair and reliableChoices about what to do in order to achieve a goalTo be included, the sources for each framework had to include:a description of the purpose of the framework;a list of the framework’s elements; anddefinitions of the key terms used to describe the purpose of the framework, its elements and domains .Frameworks that are modifications of another framework were considered together with the framework that had been modified.Frameworks for Thinking: A Handbook for Teaching and Learning12 and frameworks with which we were already familiar35. We searched for other relevant frameworks using Google Scholar between October 2018 and June 2019 using the search strategies found inExtended data File 1. We supplemented these searches by conducting citation searches and contacting key informants for each category of the frameworks.We began by considering 41 frameworks reviewed inExtended data File 2). We discussed disagreements and reached a consensus. Frameworks that were assessed for inclusion by both authors and then excluded are listed with the reasons for exclusion inOne review author (ADO) initially screened frameworks for possible inclusion. Both review authors then independently assessed full-text articles for each potentially relevant framework using an eligibility form :We recorded independently the following information for each framework, using a data collection form , and findingsWays in which the framework has been used as the basis forCurriculaTeaching and learningAssessment toolsWe compared the data that each of us had collected, discussed disagreements, and reached a consensus.Based on this information, we assessed independently:strengths and weaknesses of how each framework had been developed and evaluatedstrengths and weaknesses of how each framework has been or could be usedany other strengths or weaknessesWe compared our assessments, discussed disagreements, and reached a consensus.1. We summarised key characteristics of the included frameworks in tables.2. Using Venn diagrams, we mapped the extent to which the purposes of the different frameworks overlap with those of the IHC Key Concepts Framework.3. We compared the concepts, competences and dispositions in each framework with those in the IHC Key Concepts Framework. We considered separately any elements that could not be categorised as concepts, competences or dispositions.4. We reflected on our assessments of the frameworks and identified implications for how we might improve the IHC Key Concepts Framework, and its usefulness.We conducted these analyses independently and then compared our analyses, discussed disagreements, and reached consensus.Extended data File 1). In addition, we screened the reference lists in the articles that we retrieved. We identified over 80 frameworks and assessed 35 of these for eligibility based on one or more full-text articles  framework and theCochrane Risk of Bias Tool. The purpose and background of each of the included frameworks are shown inWe included four frameworks on critical thinking, three on logic and argumentation, four on cognition, four on scientific thinking, and seven on evidence-based healthcare. We grouped several frameworks together for five types of frameworks - logical fallacies, cognitive biases, epistemological models, systems thinking, and health literacy. We also considered related frameworks together with thetaxonomy of critical thinking dispositions and abilities59 and Baron’smodel of the good thinker75. Ennis’ goal is for students to learn to think critically about what to believe or do. Baron’s goal is for students to learn to think more rationally, that is, in a way that helps them to achieve their goals. Both those goals are broader than that of the IHC Key Concepts Framework, which is to enable people to make informed decisions about which claims to believe about the effects of things they can do (interventions) for their health, the health of others or for other reasons, and about what to do to achieve their goals4. The purposes of the two other critical thinking frameworks that we included  were also somewhat like the purpose of the IHC Key Concepts Framework.We summarise our comparison of the included frameworks to the IHC Key Concepts Framework in7. Because achieving our goals depends on what we do (actions), deciding what to believe about the possible effects of our actions and what to do is at the centre of critical thinking. However, critical thinking also applies to many other types of beliefs, such as beliefs about religion, history, or art.4. Our formulation of that goal was influenced by how Ennis and Baron formulated their goals. We have adapted Baron’s definition of a “good decision”36 to explain what this means: a good decision is one that makes effective use of the information available to the decision maker at the time the decision is made. A good outcome is one that the decision maker likes. The aim of thinking critically about treatments is to increase the probability of good outcomes (and true conclusions), but many other factors affect outcomes aside from critical thinking.The goal of the IHC Key Concepts Framework is “To enable people to make good decisions about which claims to believe about the effects of things they can do for their health, the health of others or for other reasons, and about what to do to achieve their goals”evidence-based reasoning framework87 was developed as an analytic tool intended as a foundation for assessing students’ ability to reason from evidence in writing and classroom discussions. The relationship between argumentation – critical evaluation of arguments – and the IHC Key Concepts Framework is illustrated in149, theEvidence-based practice (EBP) core competencies26,GRADE160, and critical appraisal tools179.The purpose of one of the logic and argumentation frameworks that we included had a somewhat similar purpose to that of the IHC Key Concepts Framework. The148, which divides health literacy into functional, interactive, and critical health literacy. The IHC Key Concepts Framework is most closely related to critical health literacy, as illustrated inHealth literacy also has a broader focus than the IHC Key Concepts Framework. This is most clearly illustrated by Nutbeam’s frameworkGRADE framework overlaps substantially with the IHC Framework with respect to critical thinking about evidence of intervention effects and decisions about what to do, as illustrated inGRADE framework is designed primarily for judgements by authors of systematic reviews, guideline developers, and policymakers.The84 and cognitive biases96 are both highly relevant to the IHC Key Concepts Framework. However, there is little similarity between the purposes of either of those types of frameworks and the purpose of the IHC Key Concepts Framework 32; principal components and Rasch analysis143; systematic reviews188; an agreement study189 and an assessment of the effect of training on reliability182; and an assessment of usability using focus groups and online surveys183. Two frameworks were evaluated both formally and informally, were found to be useful, and are widely used: theGRADE framework189 and theCochrane Risk of Bias Tool185.Key findings of formal and informal evaluations of the included frameworks are summarised inframework for K-12 science education, had clear inclusion criteria for one of three dimensions (“core ideas”). We judged the elements to be coherent in five frameworks, distinct in nine, and organised logically in eight. There were no inappropriate elements in seven frameworks and no missing elements in two. Overall, theevidence-based reasoning framework86 was the only framework that we assessed positively for all five criteria . That framework is a relatively simple analytic model of arguments about scientific ideas.Our assessment of the elements  in the 22 frameworks is summarised inExtended data File 4. We found evidence that most of the frameworks were being used. For four  we found little evidence of use. Two had only been available for one or two years , and we were uncertain about their use. Twelve of the frameworks appeared to be intended primarily for teachers and students, and we found learning resources based on 14 of the frameworks.Information about how the 22 frameworks have been used is summarised inevidence-based practice core competences) appeared to be intended primarily for curriculum developers26. We found at least some evidence that six other frameworks were used for curriculum development, including three of the critical thinking frameworks. We found evidence that 12 of the frameworks were used as the basis for one or more assessment tools. Other ways in which the frameworks have been used or have been proposed for use include: self-teaching; by parents, institutions, and government; by employers developing training programs; professional development; establishing norms or standards; developing ways of protecting against cognitive biases; theory development; intervention design; policy advice; and reporting standards.Nine of the frameworks appeared to be intended primarily for researchers. One  for thinking critically about claims about the effects of interventions and decisions about what to do. As noted by Dewey: “It would be impossible to over-estimate the educational importance of arriving at conceptions: that is, meanings that are general because applicable in a great variety of different instances in spite of their difference. They are known points of reference by which we get our bearings when we are plunged into the strange and unknown. Without this conceptualizing, nothing is gained that can be carried over to the better understanding of new experiences”framework for K-12 science education, and little overlap in the competences. That framework places little focus on applied science, practical understanding and use of science by non-scientists, and what children will remember and make use of in their daily lives. This may be the case for many national science curricula.We did not find any overlap between the IHC Key Concepts and those included in theOur review has helped us to clarify the goal of the IHC Key Concepts Framework and led us to add four new concepts, 10 new competences, and four new dispositions. In addition, we have identified ways in which we can improve the methods we use to further develop and evaluate the IHC Key Concepts Framework and make it more useful.190, epistemic cognition42, health literacy22, assessments of the certainty of evidence and recommendations or decisions193, causal inference162, critical appraisal29, and assessment of the risk of bias185. Moseley and colleagues12 conducted a comprehensive review of frameworks for thinking, which overlaps with and informed our review. However, we are unaware of other reviews with the same scope as this review, whether in terms of the included frameworks or the data that were collected for each included framework.Previous systematic and unsystematic reviews have reviewed different types of frameworks with similar purposes, including frameworks for cognitive biasesWe used explicit inclusion criteria for frameworks and two review authors independently collected data from included frameworks using a data collection form. Both the eligibility assessments and the data collection required judgement. Although we frequently disagreed, most of our disagreements were minor and all our disagreements were easily resolved. We did not conduct an exhaustive search for relevant frameworks. There may be other frameworks that meet our inclusion criteria. It is possible that other frameworks could add to our findings, but unlikely that they would otherwise substantially change the findings of this review.12. We have systematically considered 22 frameworks that are relevant to supporting critical thinking about claims about the effects of interventions (actions), comparisons (evidence used to support those claims), and decisions about what to do. We have found that the IHC Key Concepts Framework is unique and that it can be improved by building on the ways in which other related frameworks have been developed, evaluated, and made useful. Much of what we have found can also inform the development and evaluation of other frameworks.As defined by Moseley and colleagues: “Framework is a general term for a structure that provides support”All data underlying the results are available as part of the article and no additional source data are required.https://doi.org/10.18712/NSD-NSD2817-V1197.Norwegian Centre for Research Data, Enabling Sustainable Public Engagement in Improving Health and Health Equity,This project contains the following extended data:- File 1: Search strategy- File 2: Critical thinking frameworks eligibility form- File 3: Critical thinking frameworks data collection form- File 4: Use of the frameworks- File 5: PRISMA checklistCreative Commons Attribution 4.0 International license (CC-BY 4.0).Data are available under the terms of thehttps://doi.org/10.18712/NSD-NSD2817-V1197.The PRISMA checklist for ‘Comparison of the Informed Health Choices Key Concepts Framework to other frameworks relevant to teaching and learning how to think critically about health claims and choices: a systematic review’,http://nsddata.nsd.uib.no/webview/index.jsp?v=2&submode=ddi&study=http%3A%2F%2Fnsddata.nsd.uib.no%2Fobj%2FfStudy%2FNSD2817&mode=documentationInformation about this dataset can be found in English here: Thanks to the authors for this important and interesting paper. The Informed Health Choices (IHC) project aims to support people to evaluate the trustworthiness of health claims and to make informed choices. The IHC Key Concepts provide a framework for designing curricula, learning resources and evaluation tools. The framework has been updated and extended regularly. It comprises several concepts, competences and dispositions in different groups. Objective of this systematic review was to compare the IHC Key Concepts Framework to other frameworks that are relevant to teaching and learning how to think critically about treatment claims, evidence and choices. The frameworks were mainly identified from reviews of frameworks and by searching Google Scholar. Twenty-two frameworks were included. Two authors independently extracted information on purposes, definitions of key terms, included elements, methods of development and evaluation, and the way the frameworks were used to develop curricula, learning resources and assessment tools. Strengths and weaknesses of each framework were assessed. The authors described the frameworks in detail and displayed differences and similarities in comparison to the IHC Key Concepts Framework. They concluded that the IHC Key Concepts Framework is unique and that it can be improved by taking account of the ways in which other related frameworks have been developed, evaluated, and made useful. The findings can also be used to improve other frameworks. A possible limitation of the manuscript is the exclusively search in Google Scholar. Underlying algorithms are unknown and searches cannot be replicated reliably. The authors themselves stated that they may have missed frameworks. We agree with them that it is unlikely that additional frameworks would substantially change the findings of the review.Are the rationale for, and objectives of, the Systematic Review clearly stated?YesIs the statistical analysis and its interpretation appropriate?Not applicableAre sufficient details of the methods and analysis provided to allow replication by others?YesAre the conclusions drawn adequately supported by the results presented in the review?YesReviewer Expertise:Evidence-based health information, informed decision-making, trainings in evidence-based medicineI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Thanks for inviting me to review this interesting systematic review. The authors of this article systematically review and compare frameworks relevant to teaching and learning of critical thinking about claims, evidence, and choices. This is a great article that is tackling a very interesting issue related to the concepts of critical thinking. The authors elegantly visualised the overlap between IHC frameworks and other frameworks on a group of Venn diagrams and included a list of definitions and terminologies that are very useful. The authors provide a detailed transparent description of their methods and results – very impressive.Authors might consider describing how they collected the data regarding the use of these frameworks as the basis for curricula/teaching & learning/ assessment tools – as this usually goes beyond the framework publications.Authors mentioned that as a result of this work, they made a few modifications/additions to the original IHC concepts – authors might consider describing/justifying these changes. I would like to congratulate the authors on this great work. A couple of comments:Are the rationale for, and objectives of, the Systematic Review clearly stated?YesIs the statistical analysis and its interpretation appropriate?Not applicableAre sufficient details of the methods and analysis provided to allow replication by others?YesAre the conclusions drawn adequately supported by the results presented in the review?YesReviewer Expertise:Evidence-based HealthcareI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The authors of this paper have done a good job in conducting a systematic review of frameworks that overlap with, or potentially overlap with, the IHC key concepts framework. The Informed Health Choices (IHC) Key Concepts Framework may help build an early foundation for thinking critically about health claims and learning to make decisions informed by evidence. It provides a potential means of going beyond the delivery of information, to the development of critical thinking skills. There are however other frameworks that are relevant to thinking critically about treatment claims, comparisons, and choices. In this paper the authors position the theoretical underpinnings of the IHC Key Concepts Framework in existing frameworks within the fields of health literacy and critical thinking. Systematic review methods are detailed and appropriate as are process for decision making. The authors did not conduct an exhaustive search. It is possible therefore that additional frameworks may have been excluded. the authors acknowledge this limitation and I agree with them in that it is unlikely that additional frameworks would substantially change the findings of their review. Comparisons between the IHC framework and the included frameworks in the review are tabulated clearly and the accompanying figures demonstrating relationships between, for example, critical thinking and the key concepts framework are welcomed. It is refreshing to see that the findings of the review led the authors to revise their framework by adding concepts, competencies and dispositions. Although this paper is focused largely on the context of primary and secondary school education, it has learning for use of the IHC framework outside of these contexts and also for the development and evaluation of other frameworks. The paper also offers a useful structure for cross comparisons of frameworks in any setting or context.Are the rationale for, and objectives of, the Systematic Review clearly stated?YesIs the statistical analysis and its interpretation appropriate?YesAre sufficient details of the methods and analysis provided to allow replication by others?YesAre the conclusions drawn adequately supported by the results presented in the review?YesReviewer Expertise:Evidence syntheses, randomised trials, maternity careI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.