The purpose of this article is to close the gap in frameworks for the use of evidence in the mental health policy agenda-setting in low- and middle-income countries (LMICs). Agenda-setting is important because mental health remains a culturally sensitive and neglected issue in LMICs. Moreover, effective evidence-informed agenda-setting can help achieve, and sustain, the status of mental health as a policy priority in these low-resource contexts. A scoping ‘review of reviews’ of evidence-to-policy frameworks was conducted, which followed preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines. Nineteen reviews met the inclusion criteria. A meta-framework was developed from analysis and narrative synthesis of these 19 reviews, which integrates the key elements identified across studies. It comprises the concepts of evidence, actors, process, context and approach, which are linked via the cross-cutting dimensions of beliefs, values and interests; capacity; power and politics; and trust and relationships. Five accompanying questions act as a guide for applying the meta-framework with relevance to mental health agenda-setting in LMICs. This is a novel and integrative meta-framework for mental health policy agenda-setting in LMICs and, as such, an important contribution to this under-researched area. Two major recommendations are identified from the development of the framework to enhance its implementation. First, given the paucity of formal evidence on mental health in LMICs, informal evidence based on stakeholder experience could be better utilized in these contexts. Second, the use of evidence in mental health agenda-setting in LMICs would be enhanced by involving a broader range of stakeholders in generating, communicating and promoting relevant information. Key messagesThere are multiple frameworks for understanding, strengthening and assessing the role of evidence in health policy-making, including to some extent agenda-setting. However, there is paucity of frameworks specifically for mental health agenda-setting in low- and middle-income countries (LMICs).Current frameworks predominantly, and often implicitly, emphasize (i) research evidence and omit informal evidence such as expert advice and community narratives and (ii) professional actors such as researchers, policymakers and practitioners, rather than communities and civil society.Effective use of evidence for mental health agenda-setting in LMICs should be cognisant of (i) less abundance of formal research evidence; (ii) the stigmatization surrounding mental health, which may exclude those affected from participation in policy-making and agenda-setting processes, (iii) different kinds of policy actors that have unequal power in these processes.A meta-framework is presented with specific application to mental health agenda-setting in LMICs, along with practical recommendations for its implementation. It comprises the concepts of evidence, actors, process, context and approach, which are linked via the cross-cutting dimensions of beliefs, values and interests; capacity; politics and power; and trust and relationships.Evidence-informed policy-making occurs when governments base their policies and plans on the best available information , particuRecognized as a global development priority . TargetsThe Stages Heuristic Model is the prevailing conceptualization of the policy-making cycle , para 3 Many LMICs lack a stand-alone mental health policy, and one-quarter (25%) of World Health Organization (WHO) member states do not have a stand-alone mental health policy or plan . ExistinSubstantial headway is being made in conceptualizing the intricate relationship between evidence and policy . Theory An initial scoping search revealed numerous frameworks for understanding, strengthening and assessing the role of evidence in health policy-making more generally. Yet, only one framework focused specifically on mental health agenda-setting in LMICs: the EVIdence To Agenda setting (EVITA) framework . HoweverIn the current article, we reframe the focus to encompass a broader range of evidence, informal as well as formal. This is poignant for mental health policy-making due to the widely documented lack of formal mental health research evidence on many key topics relevant to government planning, including health provision and the creation of health policy . This evIn focusing on evidence, we also acknowledge that it is one of many influences on policy decisions, which are taken by policy actors who typically bring their agendas and interests and are engaged in complex power interplay . ExploriThe context is especially important in relation to mental health  given heHence, it is appropriate to survey frameworks from the wider health policy literature for insights . HoweverThe aim of this article is to report results of a ‘review of reviews’ of evidence to health policy frameworks to glean insights into mental health agenda-setting in LMICs. Our review sought to answer the following research question: What can be learnt from health evidence-to-policy frameworks for the use of evidence in mental health agenda-setting in LMICs? The first objective of this review was to review the applicability of current theorizations and frameworks for evidence-informed policy-making to mental health agenda-setting in LMICs. A second objective was, as a result, to propose a meta-framework for the role of evidence in agenda-setting for mental health policy-making in LMICs. We hope that this article will be of interest and relevance to policymakers, practitioners and researchers who are interested in advancing the understanding of and improving evidence-informed mental health agenda-setting and improving evidence-informed policy-making more generally.Given the existence of multiple reviews of health evidence-to-policy frameworks  and PsychINFO, followed by citation search for further publications. The HMIC database includes grey literature , which iThe BeHEMoTh framework  was usedOur inclusion criteria were the existence of theoretical/conceptual frameworks in the review; focus on the role of evidence and the process of health policy-making; published in English and published in or after 2004. Although we are primarily interested in the agenda-setting stage of the policy-making cycle and/or LMIC contexts, our scoping search suggested that such a narrow focus in the first instance would yield insufficient results to elicit meaningful findings. We followed the definition of frameworks as structures that describe the relationship between variables . ResultsResults were first screened by titles and abstracts, then full texts. Results were single-screened, and the screener brought unresolved questions regarding individual papers to the team for discussion and joint decision-making. Screening mainly filtered out results that were not reviews of theoretical/conceptual frameworks due to the difficulties in designing a search with high specificity. There was minimal ambiguity and any unclear decisions were discussed among all researchers.Limited tools exist specifically for assessing the quality of reviews of frameworks or theory. Therefore, an adapted version of GRADE-CERQual was used (Supplementary Information 2). GRADE-CERQual provides an assessment of confidence in the evidence from systematic reviews of qualitative research or syntheses of qualitative evidence  and A MeThe developers of GRADE-CERQual recommend assessing the primary studies included in each review (All included reviews were scored independently by the first author and an independent assessor. Inter-rater reliability was high: 15 of the 19 reviews were given the same rating (80%). Three reviews were discrepant in only a single rating, and one review was discrepant in two ratings. All disagreements were resolved at a consensus meeting. About half the reviews  were awarded a high level of confidence in their findings: the highest-level possible (Supplementary Information 3). Only one review (5%) was awarded the lowest level of confidence (very low). Following the GRADE-CERQual approach, no review was excluded. However, the score was considered in interpreting results of our analysis. Few reviews sufficiently assessed and/or documented the quality of the frameworks or the primary studies. Additionally, few reported the source of funding of the frameworks, or studies. Frameworks produced outside of academia, for example, by nongovernmental organizations, may be more likely to use a broader definition of ‘evidence’ beyond research.For the review papers included in the sample, the full article was analysed with the narrative synthesis of the reviews and any frameworks produced from the reviews as data. Data were extracted by a single author into the data extraction table. For the analysis of our data and synthesis of results, we were guided by the ‘best fit’ framework synthesis approach  and nonsystematic narrative reviews . An adapted version of GRADE-CERQual that was developed to provide an assessment of confidence in the evidence from systematic reviews of qualitative research or syntheses of qualitative evidence , and critiqued existing frameworks of (at least some part of) the evidence-to-policy pathway. The level of details provided on included frameworks varied greatly as did the level of analysis. Some reviews presented a list of available frameworks, some provided a categorization and some identified common factors. Seventeen focused on ‘explaining’ and ‘strengthening’ the use of evidence in health policy-making . Of thesTheories explain the relationship between variables, whereas frameworks are structures to describe the relationship between variables . Thus, iThe theories underlying individual frameworks included in each review were rarely presented. N = 5, 56%)       ; Two Com 4, 44%) ; Theory  5, 56%) ; Social  6, 67%) ; Complex 2, 22%) . All sixThere was broad consensus among the reviews that few frameworks have been applied and tested for any context , the fraInterestingly, power and political context feature strongly in the frameworks identified by The findings from our thematic analysis will now be presented under the five key concepts . The specific relevance of these findings for mental health policy agenda-setting in LMICs will also be highlighted. Findings emerged from our analysis related to evidence in four key areas: nature, perception, supply and demand, and use.‘Evidence’ was often used interchangeably with related terms such as ‘knowledge’ and not explicitly defined. Different types of evidence were identified including ‘tacit’, ‘implicit’, and ‘explicit’ , ‘informImportant characteristics identified for evidence, although not specifically for agenda-setting, surrounded the context of its intended use and include ‘relevance’, ‘applicability’ and ‘salience’ . AccordiAs one review noted, evidence is encountered often in a social context and is open to debate and interpretation . ReviewsFor mental health agenda-setting in LMICs, the influence of stigma may mean that formal research evidence is actually viewed as more robust than informal evidence, based on personal experience, that comes directly from communities . CommuniSupply and demand was often framed as the, dynamic, mismatch between the availability of evidence and demands of policymakers . An areaDifferent uses of evidence were recognized including ‘conceptual’, ‘direct’, ‘tactical’, ‘political’, ‘imposed’ and ‘procedural’ . MoreoveActors are individuals and groups directly or indirectly involved in policy-making. Interestingly, actors were the concept least featured by the reviews . Three kThe three predominant categories of actor identified were ‘researchers’ (producers of evidence), ‘policymakers’ (users of evidence) and ‘intermediaries’ (knowledge brokers). Some reviews acknowledged that their classifications were a gross simplification . Other rCharacteristics of actors received attention in many frameworks. ‘Knowledge’ and ‘capacity’ were discussed within the context of the ability and power to use evidence  often lacked detail, possibly because they appear less tangible and more difficult to assess than other contextual factors with regard to policy-making (Meso-level (organization-level) factors centred on two components: ‘capacity’ covering ‘resources’ and ‘support’ and ‘motivation’ encompassing ‘culture’ and ‘leadership’ (The predominant macro-level (systems-level) contextual factors included in the reviews were ‘political and economic’  and thisPolicy process is the way in which policies are made and enacted, often conceptualized by the stages heuristic model as the stages of agenda-setting, development or formulation, implementation and evaluation  used to strengthen the role of evidence in policy-making varied. Approaches could be categorized according to ‘effort (passive or active), direction (push or pull)and linkage  . The revTailoring approaches, including communication, to the intended audience was deemed critical .Also important is the person delivering the message  ‘nature’ of evidence on the topic and time for which use it is most suited (for different purposes and audiences and at different times); how the evidence is (2) ‘perceived’ and whether it is deemed to constitute robust and useful evidence by stakeholders and the level of (3) ‘demand’ for such evidence by stakeholders and the ease with which it can be ‘supplied’.Alongside evidence, the four other key concepts in the framework  represent barriers and facilitators arising from the environment in which evidence is to be used. ‘Actors’, ‘process’ and ‘approach’ form a triangle linked to the factors relating to evidence .actors engage with agenda-setting as individuals and through their organizational role. ‘Approach’ sits at the intersection of ‘micro’ and ‘macro context’ because ‘approach’ involves individuals seeking to have a systemic impact, often through their organizational role. ‘Process’ sits at the intersection of ‘meso’ and ‘macro context’ because ‘process’ involves organizations and therefore individuals within these organization, working within systems.‘Context’, while distinct, permeates all other concepts and is therefore displayed as a triangle housing the framework. The outer sides of the triangle denote the three interlinking sublevels of ‘context’: ‘micro’ , ‘meso’ (organizations) and ‘macro’ (systems). The positioning of actors, process and approach at the corners of the triangle highlights the most pertinent links between these concepts and the sublevels of context. ‘Actors’ sit at the intersection of ‘micro’ and ‘meso context’ because Arrows, indicating cross-cutting dimensions  link the five concepts . Whilst initially falling under the five concepts, as the analysis proceeded it became clear that these dimensions were apparent under several, if not all, of the concepts. Only the most pertinent links are displayed in the framework; all the concepts link together in complex, myriad ways. Furthermore, all of the concepts are linked through each other, for example, capacity influences the approach via actors. The use of double-headed arrows indicates the bi-directional influence, which are now explained.‘Actors’ can ‘perceive evidence’ differently due to the nature of their personal, professional and/or cultural positioning with respect to that ‘evidence’. On the other hand, the ways in which ‘actors’ relate to ‘evidence’, such as the role they play in the policy process, can also be influenced by their ‘perception of evidence’. The agenda-setting ‘process’ influences the ‘demand’ for and consequent ‘supply of evidence’. On the other hand, the ‘supply’ of evidence can also influence the agenda-setting ‘process’. Appropriate ‘approaches’ to strengthening the use of evidence in agenda-setting are influenced by the intended ‘use of evidence’ in agenda-setting. On the other hand, ‘approach’ can also influence how ‘evidence is used’. Evidence, such as community narratives may, for example, be used to drive interest in mental health policy issues, leading to demands for further evidence.‘context’ influences all aspects of ‘evidence’ in policy agenda-setting, the predominant influence is via the ‘beliefs, values and interests’ of actors as individuals (‘micro context’) and through their organizational role (‘meso context’). On the other hand, the ‘beliefs, values, and interests’ of ‘actors’ also influence the ‘context’ in which agenda-setting is undertaken. For mental health, the surrounding stigma can influence public opinion and the level of political attention received.While The extent of ‘trust’, and nature of the ‘relationships’ between ‘actors’, influences the extent to which ‘approach’ can be effective in strengthening the ‘use of evidence’. Some approaches may be dependent upon trusted relationships, for example, the use of knowledge intermediaries to share evidence. On the other hand, the kind of ‘approach’ used can influence the extent of ‘trust’ and nature of the’ relationships’ developed between ‘actors’. Developing relationships between actors, for example, by providing informal networking opportunities, may be an integral component of an approach, The policy ‘process’ is inherently ‘political’, and deciding the ‘approach’ needs to take into account the ‘power’ dynamics at play. On the other hand, the ‘approach’ taken can influence the distribution of ‘power’ in the policy ‘process’. ‘Actors’ capacity’ is a key determinant of their involvement in the policy ‘process’. On the other hand, involvement in the policy ‘process’ can magnify ‘actors’ capacity’ to engage, such as through increasing their experience and skills.In this article, we reported the analysis of (reviews of) frameworks that explain evidence-informed policy-making and proposed a resultant meta-framework for evidence-informed agenda-setting for mental health policies.Our meta-framework complements the existing body of knowledge and advances the literature through collating in a novel way a vast body of relevant information. One area we advance the knowledge on evidence-informed policy-making is a deeper understanding of the context of, and approach to, evidence-informed mental health policy-making. For example, issues such as trust, power and capacity permeate across the micro, meso and macro levels of the context and can often be intrinsically linked. We also advance the theorization of evidence-informed policy-making through highlighting the distinction of approach and process of evidence use, which need to be examined separately, offering useful practical insights for stakeholders working towards strengthening the use of evidence. Effective approaches may incorporate a broader range of activities and actors than is necessarily apparent from a focus on how policies are made and the flow of evidence. The additional focus on how the concepts are linked together offers an insight into the more diverse, and often indirect, ways in which evidence may be used to inform health policy-making.Although frameworks by their nature are a simplification, a criticism is that current frameworks treat the use of health evidence in policy as a ‘black box’ . A singlOur meta-framework aims to consider and incorporate some of this complexity through the four cross-cutting dimensions. Reviews have noted the increasing value in including ‘software’ elements of health systems  alongside their ‘hardware’ elements . However, the social and political context of decision-making, the next layer in representing the complexity of health policy and systems (According to the findings of this review, we can observe three key issues important in aiding application and in advancing evidence-informed mental health agenda-setting in LMICs.First, our findings call for greater attention to be given to informal evidence, evidence based on personal experience, e.g. expert opinion and stakeholder consultations .Our study complements, and extends, the existing EVITA framework for mental health agenda-setting in LMICs  by expanThe limited focus on the role of informal evidence also often extends to policy analysis of existing policies, which often focuses on formal research evidence (Second, our review highlights the importance of communities. Frameworks mostly focus on the ‘two communities’ of researchers and policymakers, and, increasingly, intermediaries who attempt to bridge this gap . PolicymWhile we reinforce the importance of capacity and relationships between different policy actors acknowledged elsewhere e.g. . Other sNascent frameworks are beginning to include a broader array of actors, including advocacy coalitions and included enactors, or those actors who are engaged in either research or policy processes . We arguInvolvement of a greater range of actors in promoting the use of evidence in agenda-setting would be expected to lead to a more indirect flow of evidence from researchers to policymakers, broadening the range of potential approaches. Recent attention to the importance of communities for strengthening the use of evidence for global health policies has been evoked by the COVID-19 pandemic (Bidirectionality should be a key component of their inclusion in frameworks, given the importance of genuine engagement (Widening the range of actors considered in frameworks is particularly important for LMIC settings. As argued by Different actors, however, often do not have the same power. People with lived experience of mental conditions, recognized as important participants, may face barriers to engaging in policy processes due to their health status (Third, although our results do not directly highlight this, our reflections on the results of our review highlight the importance of distinguishing policy agenda-setting from routine practices such as service delivery. Policy is often grouped with practice by reviews e.g.  and framAccordingly, from the key issues identified for mental health agenda-setting in LMICs in relation to the meta-framework, a list of five accompanying questions was developed . The queDue to the large number of existing frameworks, and the diverse terminology used, it is possible that some relevant reviews, and frameworks, were missed. However, although the reviews differed slightly in their focus, they had broadly similar findings. Due to the large number of frameworks included within the reviews, it was not possible to analyse all the frameworks individually, and the analysis of the authors of the reviews had to be relied upon. To mitigate this, individual sources were followed up, where needed. The large proportion of shared findings between the reviews also suggested robustness of the analysis of the reviews.Only English language reviews were included, and consequently, some relevant reviews may have been excluded, a limitation exacerbated by most of the included reviews themselves only including frameworks from English language publications. As highlighted by Our review has built upon the multitude of evidence-to-policy frameworks by collating the literature in a novel way. Consequently, our resultant meta-framework enables a deeper understanding of the context of and approach to evidence-informed mental health agenda-setting, for which there has been limited attention to date. Only one framework was found that focuses on this aspect, which we build upon and extend. Furthermore, we advance theory by distinguishing between approach and process of evidence use, which is of use for stakeholders working to strengthen the use of evidence. The current health frameworks were critically analysed from the perspective of mental health agenda-setting in LMICs to develop recommendations of how current frameworks could be further developed to be tailored to this specific context. Our expanded focus on what constitutes evidence and whom it is used by for mental health agenda-setting in LMICs, in addition to the conventional contribution of formal research, offers unique insights for strengthening the use of evidence. Our expanded focus aims to consider facilitating and accruing the contribution of the context and voices of individuals and stakeholders towards shaping and impacting the policy agenda.czad038_SuppClick here for additional data file.


Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts. The success of prompt-based models (GPT-3.55 and recently ChatGPT) brings new hope for medical evidence summarization, where the model can follow human instructions and summarize zero-shot without updating parameters. Medical evidence summarization refers to the process of extracting and synthesizing key information from a large number of medical research studies and clinical trials into a concise and comprehensive summary. While recent work has analyzed and evaluated this strategy for news summarization6 and biomedical literature abstract generation7, there is no study yet on medical evidence summarization and appraisal.Fine-tuned pre-trained models have been the leading approach in text summarization research, but they often require sizable training datasets which are not always available in specific domains, such as medical evidence in the literature. The recent success of zero- and few-shot prompting with large language models (LLMs) has led to a paradigm shift in NLP researchIn this study, we conduct a systematic study of the potential and possible limitations of zero-shot prompt-based LLMs on medical evidence summarization using GPT-3.5 and ChatGPT models. We then explore their impact on the summarization of medical evidence findings in the context of evidence synthesis and meta-analysis.We used Cochrane Reviews obtained from the Cochrane Library and focused on six distinct clinical domains—Alzheimer’s disease, Kidney disease, Esophageal cancer, Neurological conditions, Skin disorders, and Heart failure Table . Domain 8. They summarize the key methods, results and conclusions of the review. An abstract does not contain any information that is not in the main body of the review, and the overall messages should be consistent with the conclusions of the review. In addition, abstracts of Cochrane Reviews are freely available on the Internet . As some readers may be unable to access the full review, abstracts may be the only source readers have to understand the review results.In our study, we tackle the single-document summarization setting where we focus on the abstracts of Cochrane Reviews. Abstracts of Cochrane Reviews can be read as stand-alone documentsAuthor’s Conclusions section, the last section of the abstract, as the human reference summary for the Cochrane Reviews in our study. This section contains the most salient details of the studies analyzed within the specific clinical context. Clinicians often consult the conclusions first when seeking answers to a clinical question, before deciding whether to read the full abstract and subsequently the entire study. It also allows the authors to interpret the evidence presented in the review, assess the strength of the evidence, and provide their own conclusions or recommendations concerning the efficacy and safety of the intervention under review.Each abstract includes the Background, Objectives of the review, Search methods, Selection criteria, Data collection and analysis, Main results, and Author’s conclusions. The average length of the abstracts we evaluated can be found in Table 5 (text-davinci-003) and ChatGPT9. To evaluate the models’ capabilities, we designed two distinct experimental setups. In the first setup, the models were given the entire abstract, excluding the Author’s Conclusions (ChatGPT-Abstract). In the second setup, the models received both the Objectives and the Main Results sections from the abstract as the input (ChatGPT-MainResult and GPT3.5-MainResult). Here, we chose Main Results as the input document because it includes the findings of all important benefit and harm outcomes. It also summarizes the impact of the risk of bias on trial design, conduct, and reporting. We did not evaluate GPT3.5-Abstract because our pilot study indicated that ChatGPT-MainResult performs generally better than ChatGPT-Abstract. In both settings, we used [input] + “Based on the Objectives, summarize the above systematic review in four sentences” as a prompt, emphasizing the importance of referring to the Objectives section for aspect-based summarization. We decided to summarize the review into four sentences since it is close to the length of human reference summaries on average. The average words of the generated summary for each system are 107, 111, and 95 for ChatGPT-MainResult, ChatGPT-Abstract, and GPT3.5-MainResult, respectively. More details can be found in Supplementary Table We assessed the zero-shot performance of medical evidence summarization using two models: GPT-3.510, METEOR11, and BLEU12, comparing them against a reference summary. Their values range from 0.0 to 1.0, with a score of 1.0 indicating the generated summaries are identical to the reference summary. Our findings reveal that all models exhibit similar performance with respect to these automatic metrics. A relatively high ROUGE score demonstrates that these models can effectively capture the key information from the source document. In contrast, a low BLEU score implies that the generated summary is written differently from the reference summary. Consistent METEOR scores across the models suggest that the summaries maintain a similar degree of lexical and semantic similarity to the reference summary.To evaluate the quality of the automatically generated summaries, we employed various automatic metrics Fig. , includi13 and the percentage of novel n-grams in the summary with respect to the input. Compared to human-written summaries, those generated by LLMs tend to be more extractive, exhibiting significantly lower n-gram novelty  Coherence; (2) Factual Consistency; (3) Comprehensiveness; and (4) Harmfulness and the results are presented in Fig. To obtain a comprehensive understanding of the summarization capabilities of LLMs, we conducted an extensive human evaluation of the model-generated summaries, which goes beyond the capabilities of automatic metricsCoherence refers to the capability of a summary to create a coherent body of information about a topic through connections between sentences. Figure Factual Consistency measures whether the statements in the summary are supported by the source document. As illustrated in Fig. Comprehensiveness refers to whether a summary contains comprehensive information to support the systematic review. As shown in Fig. Harmfulness refers to the potential of a summary to cause physical or psychological harm or undesired changes in therapy or compliance due to the misinterpretation of information. Figure Supplementary Fig. Figure 14. Moreover, there is no off-the-shelf automatic evaluation metric specifically designed to assess the factuality of summaries generated by the most recent summarization systems15. We believe this likely extends to the absence of a tailored factuality metric for evaluating medical evidence summaries generated by LLMs as well. In our study, we observed similar results for three model settings when using automatic metrics, which fall short of accurately measuring factual inconsistency, potential for medical harmfulness, or human preference for LLM-generated summaries. Therefore, human evaluation becomes an essential component to properly assess the quality and factuality of medical evidence summaries generated by LLMs at this time, and more effective automatic evaluation methods should be developed for this field.Are existing automatic metrics well-suited for evaluating medical evidence summaries? Research has demonstrated that automatic metrics often do not strongly correlate with the quality of summariesWhat causes Factual Inconsistency? We categorize factual inconsistency errors into three types of errors using an open coding approach on annotators’ comments  plus conventional medical treatment (CMT) compared to CMT alone causes a higher risk of short‐term stroke and death. However, we found that the generated summary conveys low-quality confidence.First, through our qualitative analysis of the annotators' comments, we discover that auto-generated summaries often contain Fabricated errors arise when a statement appears in a summary, but no evidence from the source document can be found to support or refute the statement. For instance, a summary states that exercise could enhance satisfaction and quality of life for patients with chronic neck pain, but the review does not mention those two outcomes for patients18. Interestingly, in our human evaluation, we did not find ChatGPT producing any fabricated errors.Attribute errors refer to any errors on non-key elements in the review question  and may arise in summaries under three circumstances: (a) Fabricated attribute: This error occurs when a summary incorporates an attribute for a specific symptom or outcome that is not referenced in the source document. For example, a review17 draws conclusions about a population with intracranial artery stenosis (ICAS), but the summary states the population with recent symptomatic severe ICAS, where “recent” and “severe” cannot be inferred from the document and would impact one’s interpretation of the review; (b) Omitted attribute: This error occurs when a summary neglects an attribute for a specific symptom or outcome, such as neglecting to specify the subtype of dementia discussed in the review, leading to overgeneralization of the conclusion16; and (c) Distorted attribute: This error occurs when the specified attribute is incorrect, like stating four trials are included in the study while the source document indicates that only two trials are included19.Finally, We observed that ChatGPT-MainResult has the lowest proportion of all three types of errors when compared to the other two model configurations. Moreover, it is important to note that LLMs generally generate summaries with few fabricated errors. This is a promising finding, as it is crucial for generated statements to be supported by source documents. However, LLMs do display a noticeable occurrence of attribute errors and misinterpretation errors, with ChatGPT-Abstract and GPT3.5-MainResult displaying a higher incidence of the latter. Drawing inaccurate conclusions or conveying incorrect certainty regarding evidence could lead to medical harm as shown in later sections.19, our domain experts do not think this could bring medical harm.What causes medical harmfulness? We further identify three reasons that could potentially cause medical harmfulness: misinterpretation errors, fabricated errors, and incomprehensiveness ). Notably, we did not find any instances of medical harmfulness resulting from attribute errors in the summaries we analyzed. However, given the limited number of summaries we examined, we cannot definitively conclude that attribute errors could never cause harm. Our study suggests that medical harmfulness caused by LLMs is mainly due to misinterpretation errors and incomprehensiveness. Although our human evaluation showed that LLMs tend to make relatively few fabricated errors when completing our tasks, we cannot exclude the possibility that such errors could lead to harmful consequences. However, not all summaries with these errors would bring medical harm. For example, although the summary makes a significant error by misspecifying the number of trials in a studyHow do human-generated summaries compare to LLM-generated summaries? We observe that human-generated summaries contain a higher proportion (28%) of fabricated errors, resulting in more factual inconsistency and potential for harmfulness in human references. However, it is essential to approach this finding with caution, as our human evaluation on human-generated summaries relies solely on the abstracts of Cochrane Reviews as a proxy. There is a possibility that statements deemed to contain fabricated errors could, in fact, be validated by other sections within the full-length Cochrane Review. For example, the severity of intracranial atherosclerosis (ICAS) of the studied population  since examining the entire Cochrane Review is a time-consuming process. Therefore, it is possible that some of the errors identified in human reference summaries may actually be substantiated by other sections of the full-length review.Our evaluation of LLM-generated summaries focused on six clinical domains, with one designated expert assigned to each domain. Such evaluation requires domain knowledge, making it difficult for non-experts to carry out the evaluation. This constraint limits the total amount of summaries we are able to annotate. Further, we chose to use only the abstracts of Cochrane Reviews to evaluate human-generated summaries  and ChatGPT. GPT-3.5 is built upon the GPT-3 model but has undergone further training using reinforcement learning with a human feedback procedure to provide better outputs preferred by humans. ChatGPT has gathered significant attention due to its ability to generate high-quality and human-like responses to conversational text prompts. Despite its impressive capabilities, it remains unclear whether ChatGPT can generalize and perform high-quality zero-shot summarization of medical evidence reviews. Therefore, we seek to investigate the comparative performance of ChatGPT and GPT-3.5 in summarizing systematic reviews of medical evidence data.Author’s Conclusions as the input (ChatGPT-Abstract). The second setup takes both the objectives and the Main results sections of the abstract as the model input (ChatGPT-MainResult and GPT3.5-MainResult). The objective section outlines the specific research question in the PICO formulation that the review aims to address, while the Main results section provides a summary of the results of the studies included in the review, including key outcomes and any statistical data while highlighting the strengths and limitations of the evidence.To evaluate the capabilities of the models, we have designed two distinct setups for input. In the first setup, the models take the whole abstract except for the [input] + “Based on the Objectives, summarize the above systematic review in four sentences” to prompt the model to perform summarization, where we emphasize the purpose of the summary by providing the Objectives section. We use the Author’s Conclusions section as the human reference (see explanations in the “Introduction” section) and compare it against the models’ generated outputs.In both settings, we use 10 measures the overlap between the generated summary and the reference summary, focusing on the recall of the n-grams. METEOR11 measures the harmonic mean of unigram precision and recall, based on stemming and synonym matching. BLEU12 measures the overlap between the generated summary and the reference summary, focusing on the precision of the n-grams. These metrics have values ranging from 0 to 1, where 1 indicates perfect matching between the generation and the reference, while 0 indicates no overlap. In addition, we selected two reference-free metrics. Extractiveness13 measures the percentage of words in a summary from some extractive fragments of the source document, where extractive fragments are a set of shared sequences of tokens in the document and the summary. The percentage of novel n-grams signifies the proportion of n-grams in the summary that differ from the original source document.We select several metrics widely used in text generation and summarization. ROUGE-L24. Factual consistency measures whether the statements in the summary are supported by the systematic review. Medical harmfulness refers to the potential of a summary that leads to physical or psychological harm or unwanted changes in therapy or compliance due to the misinterpretation of information. Comprehensiveness evaluates whether a summary contains sufficient information to cover the objectives of the systematic review. Coherence refers to the ability of a summary to build a coherent body of information about a topic through sentence-to-sentence connections.We systematically evaluate the quality of generated summaries via human evaluation. We propose to evaluate summary quality along several dimensions: (1) Factual consistency; (2) Medical harmfulness; (3) Comprehensiveness; and (4) Coherence. These dimensions have been previously identified and serve as essential factors in evaluating the overall quality of generated summariesAuthors’ conclusion section; (2) ChatGPT-MainResult; (3) ChatGPT-Abstract; and (4) GPT3.5-MainResult. The order in which the summaries are presented is randomized to minimize potential order effects during the evaluation process. We utilize a 5-point Likert scale for the evaluation of each dimension. If the summary received a low score on any of the dimensions, we further asked participants to explain the reason for the low score in a provided text box for each dimension. This approach enables us to perform a qualitative analysis of the responses and identify common themes to define a terminology of error types for medical evidence summarization where none exists. In addition to evaluating the quality of the summaries, we also asked participants to indicate their most and least preferred summaries and to provide reasons for their choices. This approach enables us to identify specific subcategories of reasons and gain insights into the potential of using model-generated summaries to assist in completing the systematic review process.To assess the quality of the generated summaries, we include six domain experts, with each annotating summaries for a specific topic. During the annotation process, participants are presented with the whole abstract of the systematic review, along with four summaries: (1) the U test25. The response categories of a 5-point Likert item are coded 1–5 which were used as numerical scores in the Mann–Whitney U test for differences. The p-value reflects if the responses of the summaries generated by the two models are different, assuming the null hypothesis means there is no difference between the results generated by the two models. We used 1000 bootstrap samples to obtain a distribution of the Likert scales and reported p-values.The 5-point Likert scales between models were assessed by the Mann–Whitney Further information on research design is available in the SUPPLEMENTAL MATERIALReporting SummaryDATA SET


Cirrhosis and its associated complications can lead to changes in the composition of intestinal micro-ecosystem, resulting in dysbiosis of the intestinal microbiota. These changes may be related to the altered composition of BAs. The BAs transported to the intestinal cavity through the enterohepatic circulation are hydrolyzed and oxidized by intestinal microorganisms, resulting in changes in their physicochemical properties, which can also lead to dysbiosis of intestinal microbiota and overgrowth of pathogenic bacteria, induction of inflammation, and damage to the intestinal barrier, thus aggravating the progression of cirrhosis. In this paper, we review the discussion of BAs synthesis pathway and signal transduction, the bidirectional regulation of bile acids and intestinal microbiota, and further explore the role of reduced total bile acid concentration and dysregulated intestinal microbiota ratio in the development of cirrhosis, in order to provide a new theoretical basis for the clinical treatment of cirrhosis and its complications.Bile acids (BAs) are synthesized in liver tissue from cholesterol and are an important endocrine regulator and signaling molecule in the liver and intestine. It maintains BAs homeostasis, and the integrity of intestinal barrier function, and regulates enterohepatic circulation  The liver is a key hub for the crosstalk between host metabolism and gut microbiota, and the homeostasis of BAs and the micro-ecosystem plays an important role in maintaining human health . The hosBAs are amphiphilic detergents, signaling molecules synthesized in the liver from cholesterol, and are synthesized mainly through two pathways: the classical pathway and the alternative pathway  Chiang,. The claIn the alternative pathway, 27-hydroxycholesterol is oxidized from cholesterol via mitochondrial steroid 27-hydroxylase (CYP27A1) and then further hydroxylated by oxysterol 7α-hydroxylase (CYP7B1) to form CDCA, most of which is converted to α-muricholic acid (α-MCA) and β-MCA in rodents . These BIn the brains of rodents and humans, cholesterol 24 hydroxylase, a cytochrome P450 (CYP46A1), catalyzes the synthesis of 24S-hydroxycholesterol, androxylate which is converted to CDCA through the formation of intermediates by several enzymes. moreover, (25R)-26-hydoxycholesterol enters the brain in large quantities through the circulation and is likewise converted to 3β-hydroxy-5-cholenoic acid .FXR is a nuclear hormone receptor involved in glucose, energy and lipid metabolism and plays a central role in regulating BAS homeostasis, FXR is mainly expressed in the liver, ileum and kidneys . BAs werThe effect of FXR on BAs has been demonstrated in FXR knockout (KO) mice, where both BAs synthesis and BAs pool size were increased in KO mice . FXR regMost BAs absorption in the liver, as well as reabsorption in the intestine, relies on active transport across cell membranes, and the circulation of BAs in the gut-liver axis is mainly dependent on four transport proteins  Tranah . The firThe enterohepatic circulation is a very important BAs recycling system, most of the BAs molecules (about 95%) are reabsorbed daily into the circulating BAs pool, this reabsorption occurs mainly in the ileum and only a small fraction (about 5%) is excreted into the feces  Chiang,.In the distal ileum, BAs are reabsorbed by ASBT into enterocytes, where BAs can activate FXR and induce the expression of fibroblast growth factor 15/19 (FGF15/19). Mouse FGF15 and human FGF19 have been considered as BAs induced intestinal factors, which are released to the portal vein and then circulated to hepatocytes to activate the FGF receptor 4 (FGFR4)/b-Klotho complex. By activating the activation of c-Jun N-terminal kinase dependent pathway of mitogen activated protein kinase (MAPK) pathway, they inhibit the expression of CYP7A1, reduce the synthesis of BAs in the liver and affect the size of BAs pool . The decThere are two main pathways of feedback inhibition of BAs synthesis by FXR, one is that BAs activate intestinal FXR-FGF15/19 signal and inhibits CYP7A1 in hepatocytes. Another way is that in hepatocytes, FXR activation induces the expression of a small heterodimer partner (SHP) and inhibits the transcription of CYP8B1 and CYP7A1 genes in hepatocytes. The intestinal FXR-FGF15/19 signal of SHP mainly inhibits CYP7A1, while the hepatic FXR-SHP signal mainly inhibits CYP8B1 . FXR actBacteroides (22%), Actinobacteria (17%) and Proteobacteria (1%)  . Environria (1%) .Bacteroides, Clostridium, Lactobacillus, Bifidobacterium and Listeria  and conventionally fed FXR −/− mice, the results showed that the increase of MCA level in GF mice explained the decrease of FXR dependent gene signal in ileum and BAs excretion in feces of GF mice. In the presence of intestinal flora, the decrease of BAs pool in mice may also reflect the decrease of bile acid reabsorption in the distal ileum and the increase of BAs excretion in feces . In the le acids . Intestile acids . In concLiver cirrhosis is the end stage of chronic liver disease. “Decompensated cirrhosis” is the more advanced stage of chronic liver disease, which is characterized by variceal bleeding, ascites, HE spontaneous bacterial peritonitis (SBP) or jaundice. Patients without these complications are called “compensated” cirrhosis . There ide novo BAs synthesis was inhibited. Importantly, this was positively correlated with alcohol-related Model for End-stage Liver Disease  were common in patients with liver cirrhosis at the family level. Proteobacteria and Fusobacteria were relatively overgrown and the proportion of beneficial Bacteroides, Lactobacillus and Bifidobacterium was significantly reduced. These changes become more serious with the progression of liver cirrhosis. When progressing to HE, the gut microbiota dysbiosis may occur . It was proved that the child Turcotte Pugh (CTP) score was positively correlated with Streptococcaceae, and the significant reduction of Clostridium resulted in pro-inflammatory response, which was negatively correlated with CTP score  were found to be enriched in the cecum  was performed to observe changes in the characteristics of the fecal microbial community in patients with liver cirrhosis. TP score . In addiA levels . In a sthe cecum . The prepatients .HE is a common neuropsychiatric complication characterized by hyperammonemia in the course of cirrhosis. It has previously been proved to be related to the increase of ammonia and BAs levels in the blood . Blood aStreptococcus salivarius and Proteobacteria. They catalyze the conversion of urea to ammonia. However, impaired liver clearance leads to ammonia accumulation and in synergy with inflammation exacerbates nerve damage triggering HE (Altered intestinal microbiome and intestinal permeability may occur with associated dysfunctional bidirectional brain and intestinal actions . The cliering HE . This ilEnterobacteriaceae positively correlated with fecal CDCA in humans and were strongly associated with the development of HE, while Ruminococcaceae positively correlated with DCA (Escherichia coli and Staphylococcus spp (Bacillariophyceae, Porphyromonas and Enterobacteriaceae are directly related to poorer cognitive performance and increased inflammation in OHE. In addition, Alcaligenaceae and Escherichia coli were significantly more frequent in cirrhotic patients with HE compared to the group without HE, and Alcaligenaceae and Porphyromonadaceae were positively associated with cognitive impairment, whereas Fusobacteriaceae, Enterobacteriaceae and Veillonellaceae were positively associated with inflammation (Altered microbiota function is likely to be mechanistically related to cognitive impairment as confirmed by rifaximin, an effective antibiotic for HE with broad-spectrum antimicrobial activity, and reduced endotoxemia and improved cognitive function in its treatment of patients with cirrhosis and MHE, accompanied by reduced harmful metabolites, altered BAs composition and intestinal microecology . Enterobwith DCA . DCA is with DCA . Severalccus spp . Other sammation . These sammation .BAs, a center factor in the regulation of the intestine-liver-brain axis, has a beneficial effect on BAs homeostasis and microbiota to some extent, but it also affects the permeability of the intestinal barrier and the blood-brain barrier, which can cause complications such as HE. Altered gut microbiota diversity and disturbed BAs metabolism in cirrhosis have led to a focus on key pathological mechanisms of dysbiosis and altered barrier permeability as potential targets and approaches for the treatment of cirrhosis. More personalized treatment strategies, such as the use of targeted probiotics, antibiotics, and fecal transplants, can improve cirrhosis, hepatic cholestasis status, and neurological function of the brain by modulating specific flora to affect bile acid composition. In conclusion, a deeper understanding of the complex relationship between BAs signaling and microecology is expected to help us use this knowledge in the clinical setting to improve future strategies for the treatment of cirrhosis.


Health and welfare technologies (HWTs) are interventions that aim at maintaining or promoting health, well-being, quality of life, and increasing efficiency in the service delivery system of welfare, social, and health care services, while improving the working conditions of the staff. Health and social care must be evidence-based according to national policy, but there are indications that evidence for HWT effectiveness is lacking in related Swedish municipal work processes.This study aimed to investigate whether the evidence is used when Swedish municipalities procure, implement, and evaluate HWT, and if so, the kinds of evidence and the manner of their use. The study also aimed to identify if municipalities currently receive adequate support in using evidence for HWT, and if not, what support is desired.An explanatory sequential mixed methods design was used with quantitative surveys and subsequent semistructured interviews with officials in 5 nationally designated “model” municipalities regarding HWT implementation and use.In the past 12 months, 4 of 5 municipalities had required some form of evidence during procurement processes, but the frequency of this varied and often consisted of references from other municipalities instead of other objective sources. Formulating requirements or requests for evidence during procurement was viewed as difficult, and gathered evidence was often only assessed by procurement administration personnel. In total, 2 of 5 municipalities used an established process for the implementation of HWT, and 3 of 5 had a plan for structured follow-up, but the use and dissemination of evidence within these were varying and often weakly integrated. Standardized processes for follow-up and evaluation across municipalities did not exist, and those processes used by individual municipalities were described as inadequate and difficult to follow. Most municipalities desired support for using evidence when procuring, establishing evaluation frameworks for, and following up effectiveness of HWT, while all municipalities suggested tools or methods for this kind of support.Structured use of evidence in procurement, implementation, and evaluation of HWT is inconsistent among municipalities, and internal and external dissemination of evidence for effectiveness is rare. This may establish a legacy of ineffective HWT in municipal settings. The results suggest that existing national agency guidance is not sufficient to meet current needs. New, more effective types of support to increase the use of evidence in critical phases of municipal procurement and implementation of HWT are recommended. Health and welfare technology (HWT) is described as technology-based interventions that aim at maintaining or promoting health, well-being, quality of life, and increasing efficiency in the service delivery system of welfare, social, and healthcare services, while improving working conditions of the staff . ExampleFollow-up and evaluation of implemented interventions, both regarding the intervention itself and the implementation process, is highly prudent when faced with insufficient research-based evidence . MunicipThis study’s objective was to answer the following questions: is evidence used in Swedish model municipalities’ procurement, implementation, and evaluation of HWT? If so, what kinds of evidence are used and how? Do these municipalities have the support they need to constructively use evidence when procuring, implementing, and evaluating HWT? If not, what support is desired?The setting for the study was Swedish municipal health and social care administrations. Of Sweden’s 260 municipalities, 10 of these were designated as model municipalities for HWT use by SALAR, and the study participants were recruited from this pool. The contact details of the appropriate civil servants in the model municipalities were provided by SALAR, and a request was sent by email to all regarding participation in the study. Officials of 5 municipalities agreed to participate in the study and, following a telephone-based information session regarding the study procedures, signed informed consent forms regarding their participation as representative municipal officials.The population of the participating municipalities ranged from 57,000 to 125,000 inhabitants per municipality in March 2020. The municipalities were distributed geographically across the country. Overall, 2 of these were classified as “larger” and 3 as “smaller” according to national demographic classification levels .An explanatory sequential mixed methods design was used in this study, with a quantitative and a qualitative phase .Quantitative data were collected first via a web-based survey in November and December 2021. The questionnaire was constructed using multiple-choice questions and individual open questions with free text  accordinSemistructured interviews were then conducted with the same participants who responded to the initial web-based survey. The interview guides  containeAccording to the explanatory sequential mixed method, the quantitative and the qualitative data were analyzed separately and then amalgamated by interconnecting the data in the final analysis step . For quaFor qualitative data, content analysis was performed according to the method described by Graneheim and Lundman . The priThe study was exempted from ethics approval requirements as participants were engaging in their roles as official municipal representatives responding to requests for information, and the methods were not classified by the national research ethics agency as human research attempting to affect the participants physically or psychologically. The study nonetheless used procedures that were in accordance with the Helsinki Declaration on research involving human participants. Participants provided informed consent (in Swedish) for the use of data under the Swedish right-to-information act, including that they were able to opt out of the study at any point. Participants’ contact information as municipal representatives was obtained from public sources in order to recruit and communicate during the conduct of the study. This information was not used in the analyses, publication, or its presentation however, and not stored after the completion of the study. No compensation was provided for participation in the study.All municipalities (N=5) completed the quantitative phase in its entirety. All had implemented at least one HWT intervention in the past year, including digital alarm systems, digital medication reminders, and automated medication dispensers.According to the survey responses, 4 municipalities had required evidence of effectiveness to some extent when procuring HWT interventions, although the frequency of such requirements across procurements was uncertain. The evidence was gathered mainly via referrals from other municipalities, CE marking, or via the supplier's own analyses. Two municipalities had an established model or process for the specific implementation of HWT interventions. Three municipalities had a plan for systematic follow-up and evaluation of HWT interventions. Three municipalities indicated that their municipal work unit desired support in how to procure, implement, and evaluate HWT interventions from an evidence perspective .All studied municipalities responded that they conducted follow-up and evaluation of HWT interventions via staff directly using the technology. In total, 4 municipalities conducted follow-up and evaluation of interventions via other employee roles, and 2 municipalities via the supplier. Three municipalities reported that they used the results from follow-up and evaluation of HWT interventions to make necessary adjustments to improve the effectiveness of the technology, although this was done rarely, sometimes, and often in the respective municipalities. Two respondents did not know if the municipality was conducting follow-up and evaluation of interventions. Three municipalities disseminated results of their own evaluation of HWT intervention effectiveness, most commonly through internal reports and via the supplier.All municipalities (N=5) completed the qualitative phase in its entirety. Ten categories emerged from the content analysis: (1) evidence via references from other municipalities when procuring HWT, (2) evidence via documentation from suppliers when procuring, (3) evidence via benefits realization when procuring, (4) difficulties obtaining evidence when procuring, (5) Plan-Do-Study-Act (PDSA) as a model used when implementing HWT, (6) organizational models when implementing, (7) value and benefits when evaluating implemented HWT, (8) survey use during follow-up and evaluation of HWT, (9) desire for practical tools and knowledge transfer regarding use of evidence in HWT-related processes, and (10) desire for support networks with other municipalities regarding evidence use and HWT.The categories related to procurement (1 to 4) confirmed survey results that other municipalities were the most common source of evidence during procurement, and lacking a reliable source created difficulties. One civil servant described this accordingly:The interviews revealed that those who worked directly with HWT were not involved in the procurement process to any meaningful extent, and they often did not have sufficient information about what was included in this process. One civil servant explained:Municipalities expressed the need for evidence that was relevant to their own settings, and in particular, evidence regarding effect and benefit realization for the user, during the procurement process, but this kind of evidence was generally difficult to obtain.The categories related to implementation (5 and 6) involved the use of implementation models in general, specifically the PDSA and organizational models. The PDSA model was considered more fit-for-purpose in enabling evidence use, although it was unclear if it was used for this purpose. The organizational model was described as having a clear structure with a steering group and external frameworks to be formed, but it lacked guidance for how projects should be implemented:Other municipalities expressed the need for an established implementation plan that could be integrated into regular work processes:The categories related to follow-up and evaluation (7 and 8) primarily addressed surveys and value or benefit evaluation. Evaluation via survey, including questionnaires to staff and users following implementation of HWT interventions, preferably in close collaboration with first-line managers, was considered potentially useful. The intention was that decision makers could use the data to follow working environment conditions and adjust working methods if deemed necessary. While national agency guidance on value and benefit realization was mentioned, respondents found it difficult to evaluate qualitative benefits, especially from the users' perspective, as well as long-term benefits. It was considered easier to evaluate economic benefits, time savings, and short-term benefits:Municipalities in some cases had begun developing a framework for working with benefit evaluation and benefit calculations if one did not already exist. In one such municipality, adaptation of an externally developed evaluation model was being considered, in which they mentioned the potential for evidence production:The categories regarding desired support (9 and 10) elucidated the desire for follow-up at different levels and in different ways, from the citizen or user perspective to personnel to the IT department. Three municipalities had indicated that this support was desired for their own organizations, while all municipalities had suggestions for support mechanisms that were not specific to their own organization. Support that was desired or suggested for using an evidence-based perspective is described in Structured processes that consider evidence during procurement, implementation, and evaluation of HWT are partially or entirely lacking in Swedish municipalities that are leading the deployment of such technologies. These municipalities desire more support in using an evidence-based approach when procuring and implementing HWT.The 3 municipalities that reported using a process or model for the HWT implementation did not clearly describe how evidence was used when describing it, and the remainder described the deficits in, or absence of, any implementation model (including non-HWT specific ones) in their organizations. While follow-ups and evaluation processes existed in 3 municipalities, they varied in their methodology and were described as difficult and inadequate. Most municipalities desired support for their own organizations in using evidence across the addressed aspects of HWT. All municipalities suggested tools or methods for this kind of support across a broad spectrum of purposes and types, however.Both quantitative and qualitative results showed that evidence was requested by 4 of 5 municipalities during procurement but only infrequently due to difficulties in formulating what and how it should be requested or required. In the cases where evidence was requested, references from other municipalities regarding evidence were considered most often, while supplier-provided evidence (when presented) was considered less trustworthy. Assessment of any provided evidence was most often conducted by personnel administratively responsible for the procurement process and in some cases the intended staff users of the procured technology. It was, however, reported that there was a gap between these staff groups. These principal results demonstrate that although there is often interest in using an evidence-based perspective in municipal HWT–related processes, municipalities completely or partially lack structured processes and competency to use or generate evidence within these. This falls in line with previous assessments of knowledge about general scientifically based approaches in social service activities . This maa priori evidence in their own organizations appears to be largely missed due to variability in implementation methods. Logically, it seems difficult to ensure the use of evidence in the absence of such structured evidence-generating processes.The municipalities participating in this study were nationally designated “model” municipalities representing sector-leading HWT use, which would suggest that it is even less likely that other nonmodel municipalities use evidence-based approaches when using HWT interventions. The observed deficits in the use of a structured, evidence-based approach during key HWT-related processes run contrary to national mandates for municipal health and social care services for using the best available knowledge and conducting evidence-based practice ,24. It iTwo key research directions appear appropriate in amending such shortfalls. First, robust evidence from rigorous, controlled HWT intervention studies should increase. Second, the generation of RWE from deployed HWT interventions in health and social care settings should increase to demonstrate effects and be more readily disseminated. HWT-related research collaborations between municipalities, academia technology developers, care providers, and users should be encouraged to proceed in these 2 directions.Previous research has highlighted the use of HWT despite the lack of evidence for effectiveness or safety -6, and tPrevious research also demonstrates that municipalities lack evaluation and specific plan for follow-up of HWT ,26. AddiTransferability and generalizability of results are largely limited to publicly funded settings that use HWT. The low number of participating municipalities may reduce the certainty in comparing the variation of responses with other municipalities. Current results related to public procurement of HWT may be less relevant for countries with health systems different from Sweden or other Nordic countries regarding financing, organization, and roles.The deficits in or lack of structured processes for considering evidence during procurement, implementation, and evaluation of HWT in leading Swedish municipalities may call into question the effectiveness of HWT interventions. It could also lead to persistent negative effects on working conditions, user health and well-being, and organizational efficiency. The cost-benefit of implementing many HWT interventions would thus be greatly reduced or produce deficits. Considering the lead times to implementation and the typically long life span of implemented technologies, the risk of creating a legacy of ineffective HWT interventions is considerable. Targeted research funding to conduct robust research based on experimental evidence and RWE for HWT interventions is justified. Generation of effective national guidelines and recommendations to support evidence-based procurement and implementation of HWT-related processes should be prioritized to provide better support to municipalities in their work.


However, few works have examined the doping behavior of those 2D monolayers. This study synthesizes WSxSey monolayers using the CVD process, in which different heating temperatures are applied to sulfur powders to control the ratio of S to Se in WSxSey. Increasing the Se component in WSxSey monolayers produced an apparent electronic state transformation from p-type to n-type, recorded through energy band diagrams. Simultaneously, p-type characteristics gradually became clear as the S component was enhanced in WSxSey monolayers. In addition, Raman spectra showed a red shift of the WS2-related peaks, indicating n-doping behavior in the WSxSey monolayers. In contrast, with the increase of the sulfur component, the blue shift of the WSe2-related peaks in the Raman spectra involved the p-doping behavior of WSxSey monolayers. In addition, the optical band gap of the as-grown WSxSey monolayers from 1.97 eV to 1.61 eV is precisely tunable via the different chalcogenide heating temperatures. The results regarding the doping characteristics of WSxSey monolayers provide more options in electronic and optical design.Transition metal dichalcogenides (TMDs) have attracted much attention because of their unique characteristics and potential applications in electronic devices. Recent reports have successfully demonstrated the growth of 2-dimensional MoS However, the calculations of density functional theory claim that WS2 was p-type and would be transformed into n-type WSxSey when enough Se element was added to the WSxSey [Transition metal dichalcogenides (TMDs) have recently attracted considerable research attention due to their atomic monolayer structure, moderate carrier mobility ,2, direceristics . The syny alloys . Note thcrystals ,31,32, tositions ,39,40,41 process . In addie WSxSey . These ixSey monolayers using tungsten oxides, selenium, and sulfur powders as the sources for the CVD process, in which different heating temperatures are applied to sulfur powders for S/Se ratio modulation. The optical band gap of the as-grown WSxSey monolayers from 1.97 eV (WS2) to 1.61 eV (WSe2) was precisely controlled through the tunable S/Se ratio. The red shift of the WS2-related peaks in Raman spectra involved n-doping behavior in WSxSey monolayers, whereas the blue shift of the WSe2-related peaks indicated p-doping characteristics. The electronic state transformation of WSxSey monolayers could be tuned from p-type WS2 toward n-type WSe2 by systematically controlling the S/Se ratio as recorded through UPS measurements. The reported observations of the doping characteristics in these WSxSey monolayers have implications for electronic and optical design [In this study, we report the synthesis of WSl design ,43,44,452 and WSe2 triangles were synthesized by modifying the processes described in our previous work. In brief, the WO3 powders  were placed in a quartz boat located in the heating zone center of the furnace. The S powders  were placed in a separate quartz boat on the upper stream side. The sapphire substrates for growing WS2 were located downstream close to the WO3 powders. The central heating zone was first heated to 500 °C at 10 °C/min with an Ar/H2 flowing gas  and kept for 20 min. The furnace was then heated to 925 °C at a ramping rate of 25 °C/min and kept for 15 min. The sulfur was heated separately by a heating belt to 160 °C when the furnace reached 650 °C. After growth, the furnace was slowly cooled to room temperature. For WSe2 growth, the Se powders  were substituted for S powders as the source of the Se element, and the same growth process was followed, except for the heating belt being heated to 270 °C during growth.Crystal WSxSey, the sulfur and selenium powders were separately but simultaneously placed in two quartz boats on the upper stream side. During WSxSey alloy growth, the selenium powders were continuously heated to 260 °C by a heating belt and, for the various proportion of ingredients, the heating temperature of the sulfur powders was increased from 80 °C, 90 °C, 100 °C, and 110 °C to 120 °C. The location of the WO3 powders (0.3 g) and the growth procedure are identical to the conditions previously described. After growth, the furnace was allowed to cool to room temperature.To synthesize WS−1 was used as a reference for wavenumber calibration. The atomic force microscope (AFM) images were performed using a Veeco Dimension Icon system. The photoluminescence (PL) and differential reflectance spectra were measured with a homemade microscopy system. For PL measurements, a 532 nm solid-state laser was focused to a spot size < 1 μm on the sample by an objective lens . The PL signals were collected through the same objective lens, analyzed by a 0.75 m monochromator, and detected by a liquid-nitrogen-cooled charge-coupled device (CCD) camera. The apparatus for differential reflectance measurements was basically the same, except that the light source was replaced by a fiber-coupled tungsten–halogen lamp. Chemical configurations were determined by X-ray photoelectron spectroscope . XPS measurements of the samples were performed with an Mg Kα X-ray source. The energy calibrations were made against the C 1 s peak to eliminate sample charging during analysis.Raman spectra were collected with an NT-MDT confocal Raman microscopic system (laser wavelength 473 nm and laser spot size ~0.5 μm). The Si peak at 520 cm2 and WSe2 monolayers has been reported in our previous studies. Briefly, the triangular WS2 and WSe2 flakes are fabricated through the vapor phase reaction of WO3 with S and Se powders, respectively; similar methods have been demonstrated by many other groups [xSey monolayers in a hot-wall furnace is illustrated in xSey monolayers, S and Se powders were introduced simultaneously into the furnace during the growth process. Moreover, the Se powder was maintained at 260 °C, while the S powder was heated incrementally from 80, 90, 100, and 110 to 120 °C to modulate the S/Se ratio, labeled as Ts = 80 °C, Ts = 90 °C, Ts = 100 °C, Ts = 110 °C, and Ts = 120 °C. xSey monolayers on sapphire substrates, which are at different locations far from the WO3 precursor. First, 3 precursor, indicating that the nucleation density and the precursor density are low. Close to the WO3 precursor, small isolated triangles grew generally and then merged together with a lateral size of ~50 µm, shown in xSey domains closest to the WO3 precursor merged together and formed a continuous complete film. Some bilayer flakes are still observed on the top of the continuous monolayer film, which is attributed to the nucleation assisted by the grain boundary or particles, as shown in xSey flake was confirmed [The growth of crystalline WSr groups ,47,48. Tonfirmed .2, WSe2, and WSxSey monolayers. The PL peak positions for the pristine WS2 and WSe2 are, respectively, located at 2.0 and 1.64 eV [xSey monolayers gradually decreased from 1.91, 1.88, 1.83, and 1.76 to 1.70 eV, respectively. Therefore, a tuneable PL emission position can be easily achieved based on the modulation of the S/Se ratio in WSxSey monolayers by controlling the heating temperatures of the S and Se precursors. The strong emission from A excitons without B excitons for WSxSey compounds is in good agreement with the direct band gap emission in a monolayer, consistent with pristine TMD monolayer materials [xSey monolayers indicates a uniform distribution of S and Se in the compound domain. Otherwise, two apparent characteristic peaks, respectively belonging to MoS2 and MoSe2, would present simultaneously in the PL spectrum due to distinguishable components of MoS2 and MoSe2 in the WSxSey monolayer, shown in 2, WSe2, and WSxSey monolayers. Two distinct A and B excitonic absorption peaks for WS2 (WSe2) monolayers are at 2.01 and 2.40 eV (1.65 and 2.07 eV), respectively, resulting from the spin–orbital splitting of the valence band [xSey monolayers at Ts = 120, 110, 100, and 90, to 80 °C are located at 1.93 and 2.33 eV, 1.89 and 2.29 eV, 1.84 and 2.26 eV, and 1.79 and 2.22 eV to 1.73 and 2.14 eV, respectively. An absorption spectrum represents the energy required for electrons to be excited from the valence band (EV) to the conduction band (EC). Therefore, the optical band gap energy (Eg) can be determined from the absorption coefficient near the absorption edge shown in 2, WSxSey at Ts = 120, 110, 100, 90, and 80 °C, and WSe2 monolayers is 1.97, 1.88, 1.85, 1.79, 1.75, 1.67, and 1.61 eV, respectively. Hence, the optical band gap energy of WSxSey alloys could be controlled precisely in the range between that of the WS2 and WSe2 monolayers. 1.64 eV , attribu 1.64 eV ,50. Whenaterials . Furthernce band . The two2, WSe2, and WSxSey monolayers on sapphire substrates are shown in 2 shows two distinct characteristic peaks of 1g at 359.8 and 420.7 cm−1, respectively, due to the in-plane and out-of-plane vibration models of the atoms [1g at 251.7 and 262.7 cm−1 for pristine WSe2 were also confirmed [xSey monolayers, both WS2-related and WSe2-related characteristic peaks were simultaneously observed in the spectrum. In addition, the shifts of the WS2-related and WSe2-related characteristic peaks were seen in opposite directions. Compared with pristine WS2, a slight red shift for the WS2-related peaks of 1g at 358 and 415.7 cm−1 was revealed after Se doping in pristine WS2 at the Ts = 120 °C stage. Meanwhile, the intensity of the WS2-related 2-related 1g peaks. Note that an unidentified peak at 265 cm−1 could be attributed to the vibration from the degenerate xSey monolayers is increased by the Ts = 120 °C to Ts = 80 °C processes, the WS2-related A1g peak shows a red shift trend from 415.7, 413.8, 412.7, and 407 to 406 cm−1. However, the position of the WS2-related 2 (a = 3.13 Å for WS2) [2 monolayer would suffer compressive strain arising from Se atom doping. Therefore, the red shift of the WS2-related peaks does not result from the compressive strain arising from Se atom doping in the WSxSey monolayers. The increased red shift of the WS2-related peaks may be attributed to the effect of changing carrier concentrations on phonon vibrations arising from the increased amount of selenium in the WSxSey monolayers, where relevant investigations have been reported on the Au nanoparticle-decorated MoS2 [2/graphene stacks [2-related peaks, a blue shift of WSe2-related −1 was also observed for sulfur doping in pristine WSe2 at the Ts = 80 °C stage. Furthermore, as the amount of sulfur in the WSxSey monolayers increases from Ts = 80 °C to Ts = 120 °C, the WSe2-related 2- and WSe2-related Raman peaks was apparently revealed, suggesting that doping behaviors for S and Se atoms in WSxSey monolayers may result in different changes in carrier concentrations or strain. However, as previously mentioned, according to the smaller lattice constant for WS2 [2-related peaks does not result from the stretching strain arising from S atom doping in the WSxSey monolayers. Hence, the strain is not the main reason for the shifts of both WS2- and WSe2-related peaks in the Raman spectra, indicating that the carrier concentration is the key factor. Consequently, according to prior work [2-related Raman peaks through the increase of the Se element in WSxSey monolayers may be associated with the change of carrier concentration toward the n-type, while the blue shift of WSe2-related Raman peaks via the S atom increase in WSxSey monolayers could involve an increase in hole carrier concentrations. The doping behaviors for carrier concentration will be discussed further below. A multilayer-related peak at 307 cm−1 is not observed, indicating that these WSxSey materials are monolayers [2-related 1g peaks and the WSe2-related The Raman spectra of as-deposited WShe atoms . The chaonfirmed . However defects ,55. Alth defects , compres defects . Due to for WS2) , the WS2ted MoS2 ,53 and M for WS2 , the bluior work ,56,59, tnolayers . The RamxSey monolayers, an optical micrograph and the corresponding PL and Raman mapping of a triangular WSxSey flake are shown in xSey triangle with a lateral size of ~10 µm, grown using the Ts = 120 °C process. The corresponding peak intensity and position mappings of PL, WS2-related A1g Raman mode, and WSe2-related xSey monolayer is much stronger than the Raman signal, indicating superior crystallinity and lower defects in the as-grown WSxSey monolayers. In addition, the homogeneous Raman intensity mappings shown in xSey flake. The tiny variation within ±2 cm–1 for both Raman position mappings in 2-related and WSe2-related materials were uniformly distributed over the whole WSxSey flake, indicating good mixing and consistent components for the W, S, and Se elements. Therefore, the mapping results show that WS2-related and WSe2-related materials are uniformly mixed to form an homogeneous alloy monolayer.To confirm the homogeneity of the WS2, WSe2, and WSxSey monolayers were characterized using X-ray photoemission spectroscopy (XPS). 2 with two characteristic peaks at 32.8 and 35.0 eV, attributed to the doublet W 4f7/2 and W 4f5/2 binding energies for W4+, whereas WSe2 presents a slight red shift at 32.1 and 34.3 eV due to weaker electronegativity [3/2 and S 2p1/2 orbital of divalent sulfide ions (S2−) are observed at 162.1 and 163.3 eV [2− at 54.3 and 55.1 eV are assigned to the Se 3d5/2 and Se 3d3/2 binding energies [3 at 35.8 and 38.0 eV are also observed in all samples, possibly resulting from incomplete-reaction WO3 precursors or oxidation from residual oxygen in the chamber [3/2 and S 2p1/2) gradually become less evident, while the two selenium doublet peaks  and  become more prominent. The magnitude of each profile was normalized for easier comparison. By changing the sulfur heating temperature from 120 to 80 °C, various stoichiometries from the WS1.87Se0.31 to WS0.88Se1.39 for WSxSey monolayers could be controlled precisely, specifically listed in xSey monolayers could be accurately modulated by controlling the precursor heating temperature. In addition, the chemical stoichiometry of these WSxSey monolayers is chalcogen-plentiful; that is, the ratio of (S + Se)/Mo is greater than 2, which could be attributed to lower WSxSey formation enthalpies evaluated by first-principle calculations [The surface composition and stoichiometry of the as-deposited WSgativity . The dou163.3 eV . The douenergies ,47. In aF) was explored using ultraviolet photoemission spectroscopy (UPS). The Au layer was used as a reference to ensure that the Fermi energy was located at 0 eV [F (EF-EV) for WS2, WSxSey at Ts = 120, 110, 100, 90, and 80 °C, and WSe2 monolayers is 0.735, 0.835, 0.885, 0.90, 0.91, 0.91, and 0.96 eV, respectively, acquired by linearly extrapolating the leading edge of the spectrum to the baseline shown in onset, where hν is the incident photon energy (21.2 eV) and Eonset is the onset level related to the secondary electrons, as shown in 2, WSxSey at Ts = 120, 110, 100, 90, and 80 °C, and WSe2 monolayers is 4.31, 4.17, 4.06, 4.03, 4.00, 3.96, and 3.95 eV, respectively. In addition, the optical band gaps of the WS2, WSxSey, and WSe2 monolayers were discussed above in F-EV for the WS2, WSxSey, and WSe2 monolayers are listed in F for the WS2, WSxSey, and WSe2 monolayers are shown in F-EV energy of the WS2 is 0.735 eV, which is smaller than half the band gap energy, indicating that the WS2 monolayer is a p-type semiconductor material, consistent with other reports [xSey monolayers , the EF-EV energy increased, and the EF moved from the valence band toward the conduction band, demonstrating the transformation of electronic states from p-type to n-type. In addition, n-type WSe2 was identified according to the EF position near to the conduction band, consistent with other reports [xSey monolayers are, respectively, in xSey monolayers. In addition, the electronic state model could be tuned to p-type or n-type by modulating the Se concentration in the WSxSey monolayers. Therefore, the WSxSey monolayers can be adjusted as p-type or n-type semiconductors by systematically modulating the S/Se ratio in the process.The energy level alignment with respect to the Fermi energy (E at 0 eV . The val reports ,64. With reports ,66. The xSey monolayers were synthesized using tungsten oxides, selenium, and sulfur powders as the sources in the CVD process, in which different heating temperatures for the selenium and sulfur powders are applied, respectively, to control the S/Se ratio. The tunable band gap of the as-grown WSxSey monolayers changed from 1.97 eV to 1.61 eV with different chalcogenide heating temperatures, consistent with findings in other literature of 626.6 nm to 751.9 nm. The red shift for WS2-related Raman peaks arising from an increase of the Se element in the WSxSey monolayers was associated with an increase in electron concentration, whereas the blue shift for the WSe2-related Raman peaks was related to enhanced hole concentration. The homogeneous element distribution within a WSxSey flake was identified by PL and Raman mapping. The chemical stoichiometry for the WS2, WSxSey at Ts = 120, 110, 100, 90, and 80 °C, and WSe2 monolayers was, respectively, WS2.20, WS1.87Se0.31, WS1.66Se0.40, WS1.54Se0.48, WS1.12Se1.00, WS0.88Se1.39, and WSe1.77, indicating good control of the S/Se ratio via the chalcogenide heating temperature. With an increase of the Se element in the WSxSey monolayers, the work function changed from 4.31, 4.17, 4.06, 4, and 3.96 to 3.95, demonstrating the electronic state transition from p-type to n-type. The study of doping characteristics in those WSxSey monolayers via different chalcogen heating temperatures provides useful implications for electronic and optical design.WS


Central Nervous System Diseases are a leading cause of disability worldwide, posing significant social and economic burdens for patients, their families, caregivers, and society as a whole. In an ageing population, these diseases have a substantial impact and are expected to continue to increase . ConsequIn the present special issue, we will try to gather evidence on whether and how Virtual Reality (VR)-based interventions may help to mitigate impairments, increase engagement in activities, and improve quality of life. VR represents a paradigm shift in rehabilitation due to its capacity to provide meaningful and realistic experiences that adapt learning principles to the intervention. Learning is improved when tasks are relevant and repetitive, and when task difficulty is gradually increased . VR permDue to research progress and the decreasing cost of virtual technology, VR has steadily become a valuable tool for assessment and intervention in clinical rehabilitation for several conditions . VR has Nevertheless, the literature has reported conflicting evidence, from no significant improvement when VR is compared to conventional therapies, to an enhancement of rehabilitation outcomes when it is used alone or as an additional treatment. Recently, Voinescu et al.  reviewedMany factors should be taken into account to objectively assess the efficacy of VR in neurorehabilitation. A large body of literature supports the importance of technology components underpinning VR treatments, as well as the role of personalized VR interventions as an effective treatment. In general, personalized VR systems are recommended over commercially available VR systems , particularly for upper limb extremities, body function, and activity . ConsequA key factor to differentiate between different VR solutions is the level of immersion and presence they provide. According to their technology, VR systems are often categorized as immersive, semi-immersive, or non-immersive, although these boundaries are sometimes nuanced . In geneAnother topic of our Special Issue is how VR-based interventions can be integrated with other strategies and technologies. For instance, comparing VR standalone interventions with VR interventions delivered in combination with conventional therapy, or combining VR with other solutions. These combinations can enhance the effectiveness of interventions by leveraging the benefits of different technologies. Some examples of technologies employed alongside VR include robotics, different kind of sensors, and brain–computer interfaces (BCI) . The intVR-based neurorehabilitation should be considered from a multidisciplinary perspective. Clinical expertise is critical to tailor VR interventions. At the same time, feedback from experts in technology and engineering who contribute to VR systems’ design, development, and customization is also needed. Psychosocial and behavioral aspects that may impact patients’ engagement should be considered as well. Finally, a patient-centered approach requires input from patients, caregivers, and their families. This multidisciplinary perspective should be considered in the basic research and applied clinical studies for VR-based neurorehabilitation development.With this Special Issue, we aim to provide new evidence regarding how VR can improve neurorehabilitation outcomes. Nevertheless, we are aware of some challenges regarding the current status of this field of research. In particular, the novelty of the topic and the heterogeneity of patient populations result in a lack of standardized protocols, guidelines, and measures for evaluating the effectiveness, safety, and usability of VR interventions. It will be worthwhile to provide studies that compare and synthesize research findings and help to establish evidence-based practices in the field. Along the same line, VR technologies are rapidly evolving, and there is a wide range of VR hardware, software, and platforms available in the commercial market, which can differ in terms of quality, capabilities, and costs. It is essential to analyze such variability in terms of standardization, reproducibility, generalizability of research findings, and cost-effectiveness. It is also essential to provide evidence about the effectiveness and sustainability of VR-based interventions over time. As a general recommendation, for the articles published in this Special Issue we would like a final section entitled “The Potential Impact of Virtual Reality on Clinical Practice” to be included in the submitted manuscripts. This will be helpful for neurorehabilitation professionals in their daily work providing clinical care to their patients.