Complementary and alternative medicine (CAM) is used by about half of all patients with cancer. Guidelines are an important tool to introduce evidence-based medicine into routine cancer care. The aim of our study was to assess methodology of the statements and recommendations concerning CAM.A systematic assessment of all S3 guidelines published until November 2018 was done. Methodology of all statements and recommendations concerning CAM which were declared as evidence-based was evaluated with respect to international standards. According to the AMSTAR-2 instrument search strategy including filters, searched databases, restrictions to the research question and description of the included studies were examined. In case of adaptations from other guidelines, all underlying guidelines were examined as well.After examining 212 guidelines, 82 evidence-based statements and recommendations regarding CAM could be identified. Four were derived by adaptation, 78 by a de-novo search. Only 11 of 78 (14%) fulfilled all assessment criteria. In 18 (19%) cases no information on search strategy was attainable in any document affiliated to the guideline, in 35 (45%) cases information on search strategy was superficial and in 54 (78%) cases the referred evidence was not presented in adequate detail.Concerning CAM statements and recommendations within S3 guidelines quality of evidence processing has several shortcomings. Guideline adaptions often lack transparency and traceability.The online version of this article (10.1007/s00432-020-03238-2) contains supplementary material, which is available to authorized users. With a worldwide growing number of publications, evolving new study designs and statistical methods, guidelines become more important for treatment planning of service providers and information and decision-making of patients. Guidelines are an essential tool for promoting quality and transparency of medical care. High quality guidelines are systematically developed by independent institutions and scientific working groups around the world. The German Guideline Program in Oncology (GGPO), launched in cooperation by the Association of the Scientific Medical Societies in Germany (AMWF), the German Cancer Society (DKG) and the German Cancer Aid (DKH) in 2008, is the most important national program on cancer guidelines in Germany. The methodology of guideline development in this program is consistent with international standards  is used by about half of all patients with cancer  independently. Any deviation has been discussed and solved mutual. If necessary, a third expert was involved (JH). The first step was a systematic search of the following vocabulary and root words (here translated to English):“alternative”, ”complementary”, “integrative”, “acupuncture”, “anthroposophy”, “autogenic training”, “biological”, “Chinese”, “enzyme”, “exercise”, “folate”, “homeopathy”, “hypnosis”, “massage”, “meditation”, “mind-body”, “movement”, “naturopathy”, “physiotherapy”, “phytotherapy”, “plants”, “qigong”, “relaxation”, “sports”, “tai chi”, “TCM”, “vitamins”, “yoga”.Additionally all directories of the guidelines have been screened for any CAM-interventions. All relevant text passages were examined for statements and recommendations.Provide a search strategy or at least key words with comprehensive vocabulary?Search at least two databases?Use appropriate restrictions to the research question ?[Item 4:] Did the review authors use a comprehensive literature search strategy?[Item 8:] Did the review authors describe the included studies in adequate detail? (Are evidence tables presented?)All statements and recommendations in the considered guidelines that were declared as “evidence-based” were examined for their methodological quality. For evaluation items 4 and 8 of the AMSTAR 2 instrument  of the evidence-based statements and recommendations were adapted from other guidelines and 78 (95%) were based on de-novo searches by the guideline authors. The results of the methodological examination of all 82 evidence-based CAM- statements and recommendations are presented in detail in the supplement. An overview is presented in Table Altogether, in only 25 of the 78 statements and recommendations (32%) derived from de-novo searches detailed information on the search strategy could be found. In 35 cases (45%) only superficial information and in 18 cases (23%) no information on the search strategy was attainable in any publication affiliated to the guideline. In 59 cases (76%) the search was performed on at least two databases. There was no case of performing the search on only one database (0%), but in 19 (24%) cases no information about the used databases was given. The search restrictions were adequate in 20 cases (26%).They were only partly adequate in 35 cases (45%), because vocabulary of search terms was too narrow, recommendations exceeded the searched topic or selection of study types was not conclusive. In 23 cases (29%) search restrictions were clearly inadequate and for 24 of the 78 statements and recommendations (31%) the referred evidence was presented in detail, meaning evidence tables or comparable systematic presentations of relevant details of the included studies were displayed. In 27 cases (35%) evidence tables lacked important information on e.g. study population characteristics, study size or concrete results. In 27 cases (35%) no detailed information on included studies was found, furthermore an assessment of the bias risk of included studies was missing as well.Only eleven of the 78 statements and recommendations (14%) fulfilled all the assessment criteria. Eight further statements (10%) met the criteria almost completely (one of the four criteria was only fulfilled partly), 26 cases (33%) fulfilled all criteria in general only partly and 33 cases (42%) failed to fulfill the criteria.Adaptations of CAM statements or recommendations took place in four different guidelines [on stomach cancer by GGPO ], the adapted guideline was six years old Weimann . It was In the second case of a guideline adaptation [guideline on breast cancer ] a recommendation concerning nutrition was derived from a guideline which was only two years old Runowicz , but theIn the third case, concerning a recommendation on vitamin D to prevent osteoporosis in any cancer type [guideline on supportive therapy (45: recommendation number 10.54)], the adapted guideline considered only men and woman more than 60 years old and the original statement restricted the recommendation of vitamin D supplementation to people at high risk of falling or fractures with low exposure to sunlight  guidelines, transparency and traceability are often lacking. In one case we had to search nine different versions of a guideline and other underlying guidelines to get information about the search strategy and the included evidence, which in the end were not described in detail anywhere. To deal responsibly with time and financial resources and to enable timely updates of guidelines, it is necessary to adapt previous guidelines and to refer to systematic reviews. However, the process of adaptation may lead to methodological faults being handed on from one guideline to another. Our suggestion is that in a guideline adaptation the methodology of the underlying guideline should not only be evaluated (e.g. with the meanwhile established DELBI procedure), but also the exact details of the methodology  should be repeated. If no details on the search strategy can be found, a de-novo search should be done.Supplementary material 1 (DOCX 67 kb)Below is the link to the electronic supplementary material.


The increasingly rapid rate of evidence publication has made it difficult for evidence synthesis—systematic reviews and health guidelines—to be continually kept up to date. One proposed solution for this is the use of automation in health evidence synthesis. Guideline developers are key gatekeepers in the acceptance and use of evidence, and therefore, their opinions on the potential use of automation are crucial.Relative Advantage, Compatibility, Complexity, Trialability, and Observability. Eighteen interviews were conducted with individuals who were currently working, or had previously worked, in guideline development. After transcription, a multiphase mixed deductive and grounded approach was used to analyze the data. First, transcripts were coded with a deductive approach using Rogers’ Diffusion of Innovation as the top-level themes. Second, sub-themes within the framework were identified using a grounded approach.The objective of this study was to analyze the attitudes of guideline developers towards the use of automation in health evidence synthesis. The Diffusion of Innovations framework was chosen as an initial analytical framework because it encapsulates some of the core issues which are thought to affect the adoption of new innovations in practice. This well-established theory posits five dimensions which affect the adoption of novel technologies: Compatibility in the Diffusion of Innovations framework). Participants were also concerned with Relative Advantage and Observability, which were discussed in approximately equal amounts. For the latter, participants expressed a desire for transparency in the methodology of automation software. Participants were noticeably less interested in Complexity and Trialability, which were discussed infrequently. These results were reasonably consistent across all participants.Participants were consistently most concerned with the extent to which an innovation is in line with current values and practices (i.e., If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.The online version contains supplementary material available at 10.1186/s13643-020-01569-2. As guidelines increasingly incorporate an evidence-based medicine approach, the systematic reviews which are a crucial component of this evidence have become overwhelmed by the rate of publication of new evidence . With neGiven this, there is increasing interest in the use of automation in the completion of systematic reviews –6. WhileThe literature on the topic of automation in health evidence synthesis is somewhat lacking. Previous publications largely focus on two main areas: potential applications of automation and the validity or accuracy of automation tools . The uptThere has not yet been significant primary research into barriers and facilitators to uptake of automation in health evidence synthesis contexts , 8. ConsRogers’ Diffusion of Innovations is a highly applicable framework for the analysis of these views and the adoption of automation in health evidence production more broadly . This thComplexity is how easy an innovation is, or is perceived to be, to comprehend and to put into use.Complexity: Compatibility refers to an innovation being in line with existing values and practices, and the needs of potential future users.Compatibility: Trialability refers to the ability of users or potential users to experiment with an innovation prior to adopting it.Trialability: Observability is the degree to which potential users may examine the results of an innovation.Observability: Relative Advantage refers to how much better an innovation is, or is perceived to be, than the system it is replacing.Relative Advantage: The characteristics of an innovation that impact its diffusion are described as follows:These five elements collectively influence potential adopters’ decisions towards the adoption of an innovation. In distinct contexts, and with distinct populations, some characteristics may play a greater role than some others. For example, farmers considering whether to adopt a new system of irrigation may hold different values from pilots considering a new autopilot software. Understanding the comparative role of these characteristics in the context of systematic reviews’ and health guidelines’ potential use of automation should prove useful in describing the current state of adoption, as well as considering future research foci and organizational norm-setting.To what extent do the opinions of guideline developers of automation of health evidence synthesis fit into the Diffusion of Innovations framework?Within the Diffusion of Innovations themes, what important concepts were identified by participants?The goal of this research was to gather data from guideline developers regarding their attitudes and perceptions of automation, and specifically automation applied to health evidence production. Research questions were as follows:Ethical approval was granted from the University College London Institute of Education prior to commencement of the study.Participants were invited to participate by email sent to existing personal and professional networks for guideline developers and systematic review researchers. These networks included but were not limited to Guidelines International Network (GIN), National Institute for Health Care and Excellence , and the National Health and Medical Research Council . Participants were required to be developers of health policy or clinical practice guidelines and/or to have firsthand experience with guideline development. Participants whose experience was limited to systematic reviews or other research not including guideline development were excluded.Potential participants were invited to participate in a semi-structured one-on-one interview conducted via phone or via Skype with Ms. Arno, with the session to be audio recorded with participant consent. None personally knew Ms. Arno prior to the completion of the interviews. Participants were provided information on her background as a PhD student at the University College London studying the adoption of automation in health evidence synthesis. An interview instrument was developed by the lead author and validated by the research team prior to the interviews and was applied in all interview sessions, with variation in follow-up questions as relevant according to participants’ responses.Following verbatim transcription of the interviews, they were provided to the participants for validation. They were then analyzed using QSR NVivo 12 .A thematic approach, as outlined by Braun et al. , was appThe analysis took place in five stages, described in more detail below.Complexity, Compatibility, Trialability, Observability, and Relative Advantage).First, Rogers’ Diffusion of Innovation framework was used as the top-level deductive codes, using a line-by-line verbatim assignment of transcripts to one or more of the five themes , a codebook—a document containing all data belonging to a code or theme—was generated for each of the five themes. These codebooks were then examined with an open coding method.The codebook of each Diffusion of Innovation theme was reviewed across all transcripts together to identify the shared patterns among the grounded open codes. Each individual verbatim code was grouped with others with similar meaning and content, forming preliminary explanatory themes.Following the formation of these themes, a further review process was undertaken to reconsider how the themes fitted together. In addition, outlying codes were identified as those that either had not been grouped with codes from other transcripts or those that had relatively few grounded codes grouped together.A matrix was generated comparing each of the top-level framework themes against the data-driven themes. This approach not only allowed for an overview of the relative significance of each overall theme—thus addressing research question 1—but also to examine this significance through different lenses. For instance, isolating the framework to selected grounded codes might give a different impression of the results of the framework analysis.These matrices were finally used to analyze the data in relation to the first research question  and to expand upon these data in relation to the second research question .Twenty individuals responded to the email invitations. Eighteen interviews were conducted and varied in length from approximately 30 min to approximately 80. The remaining two respondents were deemed ineligible due to the lack of firsthand guideline development experience. Five participants were male, and 13 were female. Half of the participants had between 5 and 10 years of experience in evidence synthesis; five participants had between 10 and 20 years of experience; two participants had more than 20 years of experience, and two had less than 5 years of experience. Ten participants’ current primary affiliation was with a university, seven with a government body, and one in the private sector. Eleven participants were based in Australia, with the remaining seven based either in the USA or in the UK. In addition to NICE and NHMRC, organizations represented included the Agency for Healthcare Research and Quality , the Joanna Briggs Institute , the World Health Organization (WHO), and private consultancies. No participants withdrew from the study, and no repeat interviews were required.Compatibility was the most prominently discussed theme across all participants. Relative Advantage and Observability were also given substantial attention from participants’ discussions, though to a lesser extent than Compatibility. Trialability and Complexity were the least discussed themes among all participants.Interview transcripts demonstrated high consistency in the distribution within the Diffusion of Innovations thematic framework. Following initial coding (stage 1), Compatibility theme was consistently far stronger than any of the other four themes in the deductive framework being applied in this analysis. Some examples of values were a “rigorous” approach to the evaluation of evidence and careful construction of questions. Relating to automation more specifically, participants highlighted a need for human and organizational involvement.How you synthesize it, how you pull it together is kind of key. Participant 3I think it would be a shame if humans weren’t involved in [synthesis]. Participant 9All participants discussed their values as guideline developers at length, both within the context of the potential use of automation and independent from it. Emphasis on the Compatibility theme which further detailed participants’ desire to match new practices with the values which underpin current practices: ability to double-check and transparency as accountability.Two sub-themes were identified within the ability to double-check the output of automation with human researcher input. These discussions often cited the rationale that current practices usually involve a human double-checking the work of another human and posited that newer workflows should therefore maintain this pattern.Most participants indicated the importance of the I can see it could be done. But surely it would need to be checked by someone anyway. Because even if it’s done by a human with vast experience, it’s always important to have a second person to check it. Participant 5At the minute the standard is for two operators. So you’d want it to have been checked by a second method, if not person. So that would be my only thing – reproducibility. Participant 7Some, but not all, participants indicated that reproducibility was the underlying reason for the double-checking status quo. It is possible that views of participants would be different should rigorous research alter overall perceptions of the reproducibility of automated screening and extraction; this is further discussed as a contextual factor in subsequent sections.Several participants wanted to ensure that any automation methods used in synthesizing evidence were freely accessible and transparent to examination. Many emphasized that they are accountable to stakeholders who need to be sure they have not missed any information, and therefore require the ability to freely examine methods used, including any automation.A group of experts can apply judgement to that body of evidence, and needs to know they can trust the evidence that you’d found. Participant 12The key part of working with a face to face committee … Is you have they have to have total confidence in what the technical team has done. Participant 16The trustworthiness of evidence in general is integral to the professional culture of guideline development and was emphasized by the participants. Trustworthiness and methods to verify it therefore extend to new tools that use automation, in the view of participants, in the form of transparency and validation.Relative Advantage, participants focused on the freeing up of human resources, and to a lesser extent on time and cost saving. When prompted to discuss ML directly , participants tended to more frequently discuss ideas relating to the Relative Advantage of automation. Participants were interested in freeing time and money, but contingent upon the automation perfectly matching perceived human quality.When discussing In research time is always limited and you know there’s never enough grant money to help employ staff … by having a machine do it, it would be cost-effective, and spare the researchers’ time to do other research-related tasks. Participant 17The primary advantage specified in the discussion with participants was the potential to free up human resources for rededication to additional tasks within the health evidence ecosystem.No matter how quickly a guideline’s done, everybody always wants it faster and to be of high quality. So anything that can improve on that would be welcome, I think. Participant 11Some participants also identified that automation might potentially save time and/or save money. Strikingly, no participants indicated an openness to any trade-off between accuracy and time.Participants communicated that they would like to see evidence prior to implementing new practices, as well as a sustained ability to cross-examine the behavior of the technologies.I think at the moment it has a potentially high level of risk of being incorrect. But I don’t really know enough about it. I’d need to be convinced about it I think to consider it. Participant 9If the whole process were done by some machine or machine learning application, I think it would need to be properly trialed. Participant 5As long as there was clear data to support that … machine-learning is a reliable method, but you know, better than or equal to humans doing it. Participant 17The need for rigorously produced, disseminated, and easily accessed evidence was clear in the data. Several participants expressed an openness to automation being integrated into evidence synthesis, on the condition that accuracy has been demonstrated.I do think it’s been well demonstrated for the screening aspects, for the hit rates of what gets included and what doesn’t, and how correct it is. Participant 11One notable outlier indicated they were already convinced of automation’s abilities within the specific context of screening. This unusual case raises the possibility that these results would be different given further evidence production and dissemination.Compatibility: ability to double-check, the latter discussed that guideline developers believe the ability to check methods should be available as a matter of principle, while Observability: personal need for double-checking discusses that guideline developers want to do such checking themselves.Participants often wanted an established and ongoing method of observing the inner workings of the ML processes, frequently described as a desire to “check” what ML had done. While similar to the previous theme of The thing that’s sort of a little bit distressing from a novice point of view with machine-learning is not feeling like I have a way to check it… I’d need some way to be confident …. [I’d need] a way to check the algorithms. Participant 3This need to be able to continually check how the machine learning has processed information could be interpreted as a desire to maintain control over the evidence synthesis process. As previously discussed, guideline developers must convince other stakeholders of their recommendations’ integrity, so personal quality control fits in with the cultural expectations of guideline development.Whenever you try and really change things, I think there’s a degree of skepticism anyway…I think that might just be the nature of human beings. Participant 9If they have to learn the process, and if it’s hard, then that sort of discourages them. Participant 18So unless the technology offers a value add that’s substantial enough to overcome the learning curve…however much time it takes to do that has to not be more time than you’re gonna save. Participant 3Selected participants identified that the learning process would need to be simple if researchers were to adopt ML. They also expressed a preference for familiarity over the novel.Upon re-examination of how the data informed the deductive framework (described in step 5 of the “Methods” section), several contextual factors were identified.I’ve done a very little bit with machine-learning. Participant 3It’s just my concern would be that I’ve not had any experience with it. Participant 7I haven’t had much to do with machine-learning. Like I’ve kind of heard about it. Participant 17I think that’s something I have no personal experience with. Participant 11To be honest I actually haven’t had much experience with it. Participant 8Yeah, I don’t know, I don’t really understand that process. Participant 5Participants nearly always offered disclaimers prior to commenting, indicating they felt they did not have sufficient experience with automation technologies to be able to comment at their desired level of expertise. These data were of significant interest as they demonstrated a current lack of robust knowledge of the capabilities of automation within the target population.It would be very difficult to train a machine to make the sort of value decisions that we have to make. Participant 10I’m still a bit nervous about some of the interpretation of that…it just might be a distrust about it, I think? Participant 13How can a computer apply judgement? …There’s judgement required when it comes to things like quality or – they are not things I expect to be evidence that could be accurate. Participant 12I don’t think it could fully replace a human … I think there can be subtleties between how things can interact… I think there’s always going to be some sort of human element. Participant 9I don’t know if we’re there yet. Maybe we’ll get to the point where we can do that, but to do that, like quality rating, or to do a level of evidence, or strength of evidence… I mean there’s still a lot of value judgements in that. And I don’t know how much machine learning could help with that at this point. Participant 3Overall skepticism or mistrust towards automation, both towards current technologies and anticipated future ones, was clear in the contributions from participants. They particularly expressed doubt over the ability of a machine to mimic human judgment calls they felt are currently essential to well-formulated health guidelines.Figure Guideline developers demonstrated deeply held core beliefs about evidence synthesis methodology and perceived quality. These will be central in the potential adoption of automation to health evidence synthesis.Relative Advantage and Compatibility the most important themes, playing the most significant roles in the adoption of automation. Considering the data presented in this study in relation to Thomas’ [A 2013 paper commenting on the reasons for the slow uptake of automation posed the broad question: “why is [automation] not yet widely used?” [Relative Advantage was important, it was secondary to the far more prominent discussion of Compatibility. Further, the identified sub-themes of Compatibility focused more on automation’s fitting in with the values behind current practices, rather than on fitting in with existing infrastructureThe most significant reason appears to be that key stakeholders of EBM are not persuaded that automation is compatible with their guiding values, principles, or standards. While Observability to be critical to uptake, as well as Compatibility specifically in reference to fitting into existing practice.The preceding points not only represent a shift from the hypotheses presented in the literature, but also from the focus of previous discussions at relevant conferences. The International Collaboration for the Automation of Systematic Reviews (ICASR), formed in 2015, is a global network endeavoring to successfully integrate all the parts of automation of systematic review production together. In the notes of the third ICASR meeting in 2017, the group concluded that the “most pressing needs at present are to develop approaches for validating” automation and integration with existing system architecture . Stated Compatibility plays a significant role, it also demonstrates that alignment with values is more highly prioritized than alignment with current practice and system architecture. In addition, the “most pressing need” may not be validation (Observability), but instead the demonstration and communication of methodological standards and cultural coordination (Compatibility).As in the previous case discussed, this research has provided some evidence to support this assertion but suggests a slight redirection in which priorities would be best suited to the promotion of automation adoption. While evidence gathered in this study reinforces that Guideline developers anticipate that automation will be most useful in redirecting person-time rather than replacing it. They highlighted that a critical  part of their professional contribution is the nuanced judgments applied to the presented evidence, often derived from lived experience. Automation could contribute to an improvement in guideline quality by providing additional resources  to more difficult aspects of guideline development, and not simply by cutting costs and workload.For example, external stakeholders might believe the current vision is automated reviews devoid of valuable human control and input, that is, a general autonomous artificial intelligence system. That view, however, was neither represented nor sanctioned at the meeting. Therefore, improving the terminology associated with systematic review automation to reflect the goal more accurately is likely valuable.Contributions from participants in this study relating to the reluctance to relinquish human judgment align with notes from the previously mentioned ICASR meeting . They stThis study provides evidence in support of this proposition: participants were wary of automation in part due to the idea that it might remove crucial human judgment in the process of guideline development; notably, encouraging complete and total replacement is “neither represented nor sanctioned.” Given that participants in this study echoed this fear, it raises the question of why guideline developers hold this view, and how to best communicate a more accurate representation of the goals of promoters of automation in systematic reviews and guidelines.This observed anticipation of automation allowing for refocusing of effort is what should be expected if the results of this study are situated in the historical evidence and context. From the late nineteenth century onwards, there have been repeated waves of automation of production and consequent population-level job panic . With eaAs in the previous section, perhaps proponents of automation have a choice to steer the general conversation to clarify that expert opinion will not be superseded but will instead be made less costly and more available by freeing up person-time and other resources. An enabling environment for the promotion and adoption of automation in a manner that redirects rather than replaces research effort could be an effective strategy in building consensus among guideline developers, as key stakeholders of the evidence synthesis process, in accepting, implementing, and promoting automation practices.A potential limitation of a study of this kind is the construction of the sample and whether a different sample might lead to different conclusions. Purposive sampling was used to efficiently target potential participants from the specified group of interest. Our use of purposive sampling resulted in a sample of 18 participants from a relatively small number of organizations. One question to ask therefore is whether our approach has biased us towards identifying similar respondents, which is a possibility when using this method . A more It might be expected that purposive sampling used in this manner  would result in respondents with similar views to the investigators. This was not observed, however. The generally low awareness of the capabilities of ML, and moreover the aims of integration of ML into EBM, indicates that participants most likely did not hold similar opinions to the researchers.One clear skew in the resulting sample was the inclusion of only 28% male participants. Despite this, the views of participants did not appear to vary according to gender. It is possible that this balance is representative of the current balance in the field of guideline development; for example, according to data available from NICE, they employ 68.63% women and 31.37% men .An additional result of the sampling technique was the potential over-representation of Australian professionals. Five participants, however, had direct experience in low-resource settings and/or originated from countries other than their current base, broadening the potential perspectives for this analysis. Nevertheless, any use of these results should be tempered by awareness of the strong Australian, UK, and US representation in the data collected.Diffusion of Innovations describes the typical categories of innovators (personas) and provides an approximation of the expected proportions of each category. These are shown in Fig. Finally, as previously outlined in the “Background” section, The finding that many of the participants perceived themselves to be inexperienced with automation in the context of evidence synthesis raises the question of where the field currently resides within this adoption curve. The evidence of this study suggests that the field is in very early stages of broader adoption  with only a small minority taking on the use of this new technology.Observability: need for evidence, while certainly an outlier in the context of this study, may fall under an innovator or early adopter persona while other participants may fall under late majority or laggards. Additional analysis and/or data collection using persona categories as the deductive framework could build upon the results of this study. However, until a later stage of diffusion is reached, it may be difficult to find sufficient contributors within each category.Once a later stage has been reached, subsequent studies may well return different results. For example, the case from participant 11 in While guideline developers are a crucial group within the field of evidence-based medicine, they are far from the only one. As mentioned in the “Background” section, patients, caregivers, consumers, healthcare professionals, and researchers are also key stakeholders in evidence synthesis. Therefore, it would be logical to repeat this study with different population groups. Systematic reviewers could be considered a high priority for consultation, as these individuals will be using automation software directly, as opposed to guideline developers who act as gatekeepers of the output  of such software.Patient stakeholders are also an important group to consult. Patients are often involved in guideline panels, and there have been recent pushes to include more consumers and patients in the development of health guidance . Health Future research should be prioritized and proceed in parallel to the forms of validation highlighted by participants as crucial to their decision making. Select examples of automation have long been available for evidence synthesis, and several prominent organizations are encouraging automation uptake. Despite this reality, they are not being integrated into workflows at a large or even a medium scale. The data from this study show hesitation from a key stakeholder group, and additional data relating to other user stakeholders will be helpful in identifying barriers and facilitators for these groups.Compatibility with professional cultural values is the most significant consideration for guideline developers in the potential adoption of automation. Participating guideline developers identified increased availability of person-time as a primary Relative Advantage, and desired rigorous validation (Observability) to occur both prior to adoption and on an ongoing basis. A lack of knowledge around ML among participants is a contributing contextual factor to the slow uptake of automation, along with a generalized anxiety towards relinquishing human control to a computer. This contextual factor means that future studies may return different results if and when the evidence synthesis field reaches a later stage in the adoption curve. The data demonstrated an inaccurate perception that nuanced human judgment is likely to be removed from evidence synthesis by automation.Analyzed via the lens of the Diffusion of Innovations framework, the results of this study strongly demonstrate that The creation and dissemination of empirical evidence that systematically demonstrates automation’s alignment with the values and standards of guideline development and EBM should therefore be prioritized. In addition, disseminated evidence and communications around automation tools may benefit from focusing on the combination of human and ML effort, rather than the replacement of human insight. Finally, proponents should prioritize communication of transparency in automation methods and on strengthening automation competency and familiarity among EBM professionals.Additional file 1. COREQ  Checklist.Additional file 2. Interview Instrument.


Social presence, or the feeling of being there with a “real” person, is a crucial component of interactions that take place in virtual reality. This paper reviews the concept, antecedents, and implications of social presence, with a focus on the literature regarding the predictors of social presence. The article begins by exploring the concept of social presence, distinguishing it from two other dimensions of presence—telepresence and self-presence. After establishing the definition of social presence, the article offers a systematic review of 233 separate findings identified from 152 studies that investigate the factors  that predict social presence. Finally, the paper discusses the implications of heightened social presence and when it does and does not enhance one's experience in a virtual environment. Since its conceptualization, virtual reality (VR) has been touted as a novel communication medium that would radically change the way people interact with each other  impact perceptions of social presence.technological qualities and psychological experiences afforded by mediated communication. Immersion can be defined as a medium's technological capacity to generate realistic experiences that can remove people from their physical reality , it is probable that the technological features of the environment will influence the level of social presence a person feels. At the same time, however, this perspective offers a more nuanced view of social presence; while the immersive qualities  can impact social presence, individual communication strategies as well as contextual differences have a significant effect on social presence.While both telepresence and self-presence have received academic focus, social presence has been considered to be particularly important within virtual environments with social actors . This is due to the impact of social presence on social influence. Studies have shown that social presence is associated with a variety of positive communication outcomes, such as persuasion and attraction  Conference and the IEEE Conference on Virtual Reality. We chose these outlets by selecting and expanding upon the outlets chosen in a recent meta-analysis on presence conducted by Cummings and Bailenson  used at least one self-report measure of social presence (or the synonymous concept of co-presence); if social presence and co-presence were measured separately, we considered both measures in our review, (2) included experimental manipulations and/or questionnaire items  that were used to assess the predictors of social presence, and (3) conducted quantitative analyses to determine whether a predictor significantly influenced perceptions of social presence.Studies that measured social presence with related but distinct constructs  were not included, as they do not uniquely measure the extent to which one feels as if she is present with a sentient being. Similarly, concepts that share theoretical similarities with, but do not uniquely measure social presence, were excluded. One significant concept that was not included due to this criterion was plausibility illusion Slater, . PlausibFinally, while we are aware of the strengths of behavioral and physiological measures and limitations of self-report measures Slater, , we did Considering that social presence was initially considered to be an inherent quality of a communication medium  might influence social presence, it compares media that vary across multiple features, which makes it difficult to isolate the affordance(s) that influenced perceptions of social presence. This camp of research is well-aligned with the traditions of social presence theory  also found that participants felt the lowest level of social presence when communicating via text-based CMC compared to “richer” forms of media, when given the same amount of time , these results suggest that linear increments of immersion do not necessarily lead to corresponding increases in social presence. Considering that two of the studies  the presence or absence of a visual representation and (2) the visual realism of the virtual representation. Visual realism consists of photographic, anthropomorphic, and behavioral (or communicative) realism  affects social presence. Behavioral realism can be operationalized by the complete absence or presence of nonverbal behavior (animations) or how much the virtual human's nonverbal behavior is consistent with actual humans . Studies generally show that behavioral realism is a powerful predictor of perceived social presence. These positive effects are most consistently found when the avatar's or agent's behavior indicates awareness of their communication partner's presence . For example, von der Pütten et al.  found thand behavior of the visual representation show strong support for consistency effects . The opposite effect was found for avatars low in behavioral realism. In a separate study, Bailenson et al.  the presence of a visual representation and (2) a more behaviorally realistic visual representation enhance social presence. In contrast, while both photographic and anthropomorphic realism can enhance perceptions of social presence, this effect appears to be contingent on certain boundary conditions, including consistency with the level of behavioral realism.While real-time virtual communication between actual people is usually characterized by high levels of interactivity, the level of interactivity afforded by a computerized agent can vary. As such, studies that explored the impact of interactivity on social presence generally looked into how an agent's interactivity influences social presence. Considering that social presence is dependent on how strongly one feels that he or she is talking with an intelligent being that is aware of his or her presence Biocca, , it is uDue to the significance of touch in physical interactions, a lot of effort has been—and continues to be—made to introduce interpersonal touch through haptic devices in virtual environments. The current review identified haptic feedback as one of the most commonly studied immersive qualities that influence social presence, apart from visual representation and interactivity. With the exception of one study  to technology-mediated contexts to explore their relevance in interpersonal perceptions  or a computerized algorithm . Studies that explore the impact of (perceived) agency on social presence generally introduce the virtual human as an actual person or a computerized character prior to the interaction. Approximately half of the studies surveyed in this paper found that people felt higher levels of social presence when the virtual human was thought to be controlled by an actual person rather than by a computer program. For example, participants felt greater social presence when they believed that the Rapport Agent they were interacting with was a real person compared to when they thought it was an artificial intelligence  as a contextual factor that influences social presence, and found that increasing the number of identity cues enhances feelings of social presence , Hauber et al.  desirable for a virtual environment to adopt immersive and contextual qualities that will increase social presence.There are several limitations in the current study. First the research that was reviewed often used different measures of social presence, which limits their comparability. It is thus important for researchers to note the different measures used  in addition to the number of participants in Table credibility of a scenario, and thus is not dependent on the sensory capabilities of a virtual environment  of a limited set of trackers. Considering the fact that both gesture and posture have a significant influence on person perception in FtF contexts  influences social presence, it is less probable that these features will influence plausibility illusion, or how believable they find the virtual human's behavior to be. Lending some support to this conjecture, Pan et al.  found thIn addition, future studies should offer a more holistic view of social presence by considering the different dimensions that impact social presence. Just as studies found that increasing the behavioral realism of a virtual human that had low photographic realism did not lead to increased social presence (Garau et al., Lanier  noted thCO, JB, and GW contributed to the conception and design of the study. CO organized and reviewed the database of social presence studies. CO wrote the first draft of the manuscript. CO, JB, and GW contributed to the manuscript revision, read and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


CD8 T cells play a crucial role in providing protection from viral infections. It has recently been established that a subset of CD8 T cells expressing Tcf1 are responsible for sustaining exhausted T cells during chronic lymphocytic choriomeningitis virus (LCMV) infection. Many of these studies, however, have been performed using T cell receptor (TCR) transgenic mice, in which CD8 T cells express a monoclonal TCR specific for the LCMV glycoprotein. To investigate whether the Tcf1+ and Tcf1- repertoires are naturally composed of similar or different clones in wild-type mice exposed to acute or chronic LCMV infection, we performed TCR repertoire sequencing of virus-specific CD8 T cells, including Tcf1+ and Tcf1- populations. Our analysis revealed that the Tcf1+ TCR repertoire is maintained at an equal or higher degree of clonal diversity despite harboring fewer cells. Additionally, within the same animal, there was extensive clonal overlap between the Tcf1+ and Tcf1- repertoires in both chronic and acute LCMV infection. We could further detect these virus-specific clones in longitudinal blood samples earlier in the infection. With respect to common repertoire parameters , we found minor differences between the virus-specific TCR repertoire of acute and chronic LCMV infection 40 days post infection. Overall, our results indicate that the Tcf1+ population emerging during chronic LCMV infection is not clonally distinct from the Tcf1- population, supporting the notion that the Tcf1+ pool is indeed a fuel for the more exhausted Tcf1– population within the heterogenous repertoire of LCMV-specific CD8 T cells. Viral infections represent a global health problem and often invoke a potent immune response that can potentially result in severe immunopathology to the host if not counter-balanced by immunomodulatory circuits , 2. PersA population of CD8 T cells expressing the transcription factor T cell factor 1 (Tcf1) has been recently demonstrated to play an important role in the immune response during chronic LCMV infection. Absence of this population of Tcf1+ CD8 T cells resulted in reduced viral control and a decrease in virus-specific T cell clonal expansion . SpecifiWhile multiple studies have recently investigated the transcriptional landscape of defined virus-specific Tcf1+ CD8 T cells in both acute and chronic infections , less atTcf7GFP, which expresses GFP under the control of the Tcf7 promoter that encodes Tcf1  and CD44 expression (in case of both acute and chronic infection) (hi CD8 T cells lacking specificity toward the gp33−41 epitope (GP33-) were sorted from mice with chronic LCMV infection to include cells with other LCMV-specificities. While this sorting strategy resulted in a discrepancy between infection cohorts for those T cell populations unspecific to gp33−41, it nevertheless allowed the comparison between Tcf1+ and Tcf1– repertoires across repertoires skewed toward LCMV-specificity. Overall, our experimental pipeline resulted in 16 distinct T cell populations (8 of which with known LCMV-specificity to the gp33−41 epitope) for TCR repertoire sequencing ranging from hundreds to hundreds of thousands of cells  repertoire in nine mice with acute or chronic LCMV infection; mice were of the transgenic Tcf1 reporter strain [des Tcf1 ]  . Specifiof cells . FACS anof cells . As expeof cells , the pooof cells . Furthere 40 dpi . FACS anr2 = 0.004, Recent work has demonstrated that the Tcf1+ CD8 T cell population sustains the Tcf1- population during chronic infection . This waOur initial analysis suggested that the highest degree of clonal diversity in GP33+ TCR repertoires was in the lungs . This anAfter observing the comparable clonal diversity in the Tcf1+ and Tcf1– TCR repertoires, we hypothesized that these repertoires would use similar germline V-gene elements. We therefore quantified the percentage of unique clones using a particular V-gene in the virus-specific Tcf1+ and Tcf1- spleen/LN populations in chronically infected mice . Both re2 fold-change of V-gene usage between the Tcf1+ and Tcf1– within the same organ and infection cohort (2 fold-change >1/−1. While there were two V-genes up-regulated in the lungs (TRV31 and TRBV5) in the Tcf1+ compared to the Tcf1– repertoires of chronically infected mice, this upregulation was not similarly observed in the spleen (We next explicitly quantified whether certain TRB V-genes were more or less present in either the Tcf1+ or Tcf1– repertoires for either infection cohort. To relate the expression level to V-gene (up- or down-regulation), we plotted the mean V-gene usage against the logn cohort . Overalle spleen . Similare spleen .Surprised by the high degree of similarity in TRB V-gene usage across all of our samples, we questioned whether similar patterns of germline gene usage would be detected in the blood, particularly prior to infection. Quantifying the percent of unique clones aligning to each V-gene revealed a substantial (>40% of unique clones) portion of the unsorted blood repertoire consisted of TRBV1 and TRBV31 before viral infection. While these two genes were present in the sorted populations 40 dpi, neither consisted of over 20% of the repertoire on average as seen in the blood repertoires of both infection cohorts before any viral infection . We hypoAfter observing the consistencies involving V-gene usage and CDR3 length between the Tcf1+ and Tcf1– repertoires, we next quantified the extent to which the two repertoires shared common clones . We hypothesized that the two repertoires would share a high proportion of clones given the hypothesis that the Tcf1+ pool sustains the Tcf1– population. We first calculated the percent of Tcf1– clones found in the Tcf1+ repertoires of either the same mouse or other mice within the same cohort . This anDespite the aforementioned mouse-specific overlap between the Tcf1– and Tcf1+ populations, we were surprised that only ~50% of the clones were common to both repertoires. This prompted us to question whether restricting our analysis on the most expanded clones of both Tcf1+ and Tcf1– repertoires would increase the clonal overlap between these populations. We thereby quantified the fraction of the top 10 most expanded clones in either the Tcf1+ and Tcf1– repertoires. This analysis demonstrated a much stronger trend of mouse-specific overlap, in that ~4–6 of the top 10 GP33+ clones were common across the Tcf1+ and Tcf1– repertoires of the same mouse, but only ~0–1 of the top 10 clones were common to different mice within the same cohort , suggestAfter observing a high degree of clonal overlap between the Tcf1+ and Tcf1– repertoires within the same organ, we next determined the extent of clonal overlap between the spleen/LN, lungs and blood repertoires. We hypothesized that that there should be a high overlap between the Tcf1– lung repertoire and the Tcf1+ central memory pool, as it has been shown that these Tcf1+ virus-specific T cells reside primarily in secondary lymphoid organs , 19. To We subsequently investigated whether the most expanded clones found at the terminal time point could be detected in the blood TCR repertoires of the same mice for both chronic and acute infection cohorts. Indeed, our time-resolved sequencing showed that many of top 10 most expanded clones from both Tcf1+ and Tcf1– of spleen/LN repertoires were observed at earlier time points . Given bThe majority of our analysis until now has focused on the differences between the Tcf1+ and Tcf1– repertoires, regardless of infection cohort. However, the observation that TCR repertoires were more similar to the blood 40 dpi in chronically infected mice  lead us 2 fold-change of V-gene expression between chronic and acute infection. We had expected more pronounced differences in V-gene usage in the lung relative to the spleen/LN repertoires based on the aforementioned clonal overlap analysis  vs. the loganalysis . There wanalysis . This ananalysis  focused analysis , 5, we danalysis . For allanalysis . There wanalysis . Indeed,analysis . There wWhile normally expressed in naïve and memory CD8 T cells, Tcf1 has been recently demonstrated to distinguish a memory-like population of T cells which play a role in resolving both chronic infection and cancer due to their ability to feed into the pools of more differentiated /exhausted CD8 T cells and due to their ability to primarily respond to check-point blockade. In the case of chronic LCMV infection, initial clonal expansion of virus-specific T cells in Tcf1 knock-out mice were comparable to wild type mice but were unable to resolve viral infection . These pOverall, our findings support a model in which the Tcf1+ population sustains the Tcf1– repertoire during chronic LCMV infection, particularly demonstrated by an equal or greater number of unique clones in the Tcf1+ population  and that almost all of the most expanded Tcf1– clones were found in the corresponding Tcf1+ counterpart. These findings were further supported by similar V-gene usage, CDR3 length and the extent of clonal expansion between Tcf1+ and Tcf1– repertoires. Furthermore, there was a trend in chronically infected mice of increased clonal expansion in the Tcf1+ repertoire relative to the Tcf1- counterpart , again cIn contrast to the antibody repertoires following LCMV infection, we observed minor differences in the TCR repertoires between acute and chronic cohorts, including the upregulation of the TRBV30 in both spleen/LN and lung repertoires in chronic infection  and viruDespite previously reported differences in clonal expansion kinetics of virus specific CD8 T cells in regards to chronic vs. acute LCMV infection , we saw Tcf7GFP mice . 18–20 week old female LCMV clone 13 was produced as previously described . In brieTwo hundred microliter of blood was sampled from the leg vein at indicated time points  during the course of infection. Upon sacrifice 40 dpi, blood was sampled via intracardiac bleeding. Blood was collected in heparin-coated microtainers (BD) and centrifuged for 5 min at 2,000 g to separate plasma and cells. Pellets were resuspended in 50 μL PBS and mixed with 1,500 mL Trizol LS (Ambion) for future RNA extractions. Spleens and six lymph nodes  were pooled together. Mice were perfused with PBS to remove blood associated cells before extracting the lungs. Single cell suspensions from spleens and LN pools were prepared by meshing the tissue through a 70 μM cell strainer. Lungs were first cut into small pieces, incubated with Collegenase I and DNAse I for 30 min, meshed through 70 μM strainers then followed by a 30% percoll gradient. Cells were incubated with fluorescently conjugated antibodies for 30 min at 4°C. Antibodies used in the experiments; CD8-PerCP , CD44-PE , PD-1-PECy7 . MHC Class 1 tetramers for gp33-41 was prepared and conjugated to APC as previously described (RNA was extracted using the Direct-zol RNA MiniPrep kit (Zymo) according to manufacturer's instructions without the use of DNAse. First strand cDNA was synthesized in a total volume of 20 μl using 11.5 μl of RNA, 0.5 μl oligo(dT) primers , 1 μl dNTPs , 1 μl 0.1M DTT (life technologies), 1 μl RNAsin Plus RNAse inhibitor , 1 μl Superscript III  and 4 μl 5x Superscript III buffer for 10 min at 50°C, 10 min at 25°C and 60 min at 50°C. Polymerase was inactivated by incubation for 5 min at 94°C. TCR sequencing libraries were then prepared in a two-step PCR approach amplifying the TCR-β chain as previously described  using 192 fold-change ratios were calculated based on the mean proportion of repertoire for each V gene within a given infection cohort. When relating log2 fold-change to mean V-gene usage, mean V gene usage was calculated across all mice included in the comparison, regardless of cohort. Repertoire parameters such as Shannon evenness and Jaccard indices were calculated as previously described (Paired-end sequencing fastq files were processed using the MiXCR software (v3.0.1)  with reaescribed , 25. Briescribed . We thenescribed . The Jac*p ≤ 0.05, **p ≤ 0.05, ***p ≤ 0.005, ****p ≤ 0.0005). The nonparametric Wilcoxon-matched pairs signed rank test was used for paired data . All analyses were performed using GraphPad Prism 8. All error bars represent the standard error of the mean.All statistical significance for unpaired data was calculated using the nonparametric Kolmogorov-Smirnov test Unless otherwise mentioned, statistical significance .AY, IS, AP, MB, FW, NO, SW, and KP performed the experiments. AY, IS, SR, and AO designed the study. AY performed the repertoire analysis. AY, IS, SW, SR, and AO wrote the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Among all diseases globally, mental illnesses are one of the major causes of burden. As many people are resistant to conventional evidence-based treatments, there is an unmet need for the implementation of novel mental health treatments. Efforts to increase the effectiveness and benefits of evidence-based psychotherapy in psychiatry have led to the emergence of virtual reality (VR)–based interventions. These interventions have shown a wide range of advantages over conventional psychotherapies. Currently, VR-based interventions have been developed mainly for anxiety-related disorders; however, they are also used for developmental disorders, severe mental disorders, and neurocognitive disorders.This meta-review aims to summarize the current state of evidence on the efficacy of VR-based interventions for various psychiatric disorders by evaluating the quality of evidence provided by meta-analytical studies.A systematic search was performed using the following electronic databases: PubMed, PsycINFO, Web of Science, and Google Scholar (any time until February 2020). Meta-analyses were included as long as they quantitatively examined the efficacy of VR-based interventions for symptoms of a psychiatric disorder. To avoid overlap among meta-analyses, for each subanalysis included within this meta-review, only one analysis provided from one meta-analysis was selected based on the best quality of evidence.The search retrieved 11 eligible meta-analyses. The quality of evidence varied from very low to moderate quality. Several reasons account for the lower quality evidence, such as a limited number of randomized controlled trials, lack of follow-up analysis or control group, and the presence of heterogeneity and publication bias. Nonetheless, evidence has shown that VR-based interventions for anxiety-related disorders display overall medium-to-large effects when compared with inactive controls but no significant difference when compared with standard evidence-based approaches. Preliminary data have highlighted that such effects appear to be sustained in time, and subjects may fare better than active controls. Neurocognitive disorders also appear to improve with VR-based approaches, with small effects being found for various clinical outcomes . Finally, there are insufficient data to classify VR-based interventions as an evidence-based practice for social skills training in neurodevelopmental disorders and compliance among patients with schizophrenia.VR provides unlimited opportunities by tailoring approaches to specific complex problems and individualizing the intervention. However, VR-based interventions have not shown superiority compared with usual evidence-based treatments. Future VR-based interventions should focus on developing innovative approaches for complex and treatment-resistant symptoms that are difficult to address with traditional treatments. Future research should also aim to gain a better understanding of the potential factors that may mediate VR outcomes to improve treatment. Mental illnesses are one of the predominant causes of burden among all diseases globally . It has VR-based treatments have been developed for many psychopathologies, particularly for anxiety-related disorders, and for developmental disorders, severe mental disorders, and neurocognitive disorders ,24. As tA search was independently conducted by 2 graduate students (LD and ML) on PubMed, PsycINFO, Web of Science, and Google Scholar electronic databases, from each database’s inception to February 2020. Search terms were chosen to be inclusive of VR , mental disorders , and interventions . The search syntax was tailored for each database. See Meta-analyses were included as long as they quantitatively examined the efficacy of VR-based interventions for the symptoms of psychiatric disorders. To maximize the number of meta-analyses, we did not restrict the search to any specific psychiatric population or any age group. It is noteworthy that a problem with meta-analyses is that they may overlap when many have been conducted on a particular disorder and a particular type of subanalysis for the disorder . To avoid this issue, for each subanalysis included within this meta-review, only one analysis provided from one meta-analysis was selected based on the best quality of evidence. The inclusion of the meta-analyses was generally based on (1) the year of publication, (2) the number of included studies, and (3) the quality of the included studies . To ensure consensus, discussions on the inclusion of meta-analyses were held with a senior researcher (SP). As a meta-analysis only requires a minimum of 2 studies , we chosQ statistics, 2I index), and publication bias  were recorded. Refer to Data were extracted using a standardized form by LD and ML. Key information related to the sample, effect sizes , outcome measured, control group, timeline , confounding factors , heterogeneity , have become typical in VR implementation as exposure is undeniably a key element that must be addressed in these disorders ,46-48. T anxiety ,54. In t anxiety ,56.A meta-analysis by Parsons et al  found anA meta-analysis by Parsons et al , comprisEvidence based on the meta-analysis by Cardoş et al  includinIn summary, meta-analytical evidence shows that VR-based interventions may be effective for specific phobias. The quality of evidence ranged from low to moderate quality, with better quality evidence provided for aviophobia, which comprised a larger number of RCTs with a larger sample size. The presence of heterogeneity and publication bias was evaluated for aviophobia. When compared with active controls, the results suggested better aviophobia outcomes for VR-based therapies than classical evidence-based interventions, with no significant superiority over other exposure-based therapies. However, for arachnophobia, significant superiority was found for VR when compared with active controls. Finally, the effects of VR for aviophobia remained stable in time, indicating that VR might fare better than active controls in the long term.First, the meta-analysis by Parsons et al  includedIn summary, these meta-analyses on panic disorder with or without agoraphobia showed that VR-based interventions are efficient. Evidence has been evaluated to be of low to moderate quality, with the quality of evidence being lower owing to a lack of consideration of heterogeneity, publication bias, and moderating factors. Better quality evidence was provided for comparison with classical evidence-based interventions, which showed that VR was no better than these interventions at posttreatment. At follow-up, there was a small superiority observed favoring VR over standard interventions.First, the meta-analysis by Kampmann et al  observedIn summary, overall evidence was evaluated as low-to-moderate quality: most meta-analyses included a limited number of trials and moderator analyses, and did not report heterogeneity or publication bias. Medium to large effects were observed for VR-based interventions for social phobia. Nevertheless, no significant difference existed between the VR-based interventions and standard treatment. The overall beneficial effect of VR interventions was maintained in the long term, although no significant difference was observed with active controls.Trauma- and stress-related disorders (such as posttraumatic stress disorder) may develop by directly experiencing, witnessing, or repeating exposure to aversive elements of a traumatic event . Although many show resilience following exposure, up to one-third of those confronted with a traumatic event will subsequently develop clinically relevant posttraumatic symptoms  . It is wFirst, a meta-analysis by Deng et al  found a In summary, the meta-analyses on posttraumatic stress disorder showed small-to-moderate effects for VR. Evidence was generally graded as low-to-moderate to moderate quality. No significant difference was found with standard evidence-based interventions. Moreover, improvements in VR were maintained in time.VR-based treatments for the symptoms of individuals with severe mental disorders have multiplied in recent years. Although there are very limited studies on the effects of VR for those with mood disorders ,62, thisIt is worth noting that VR interventions included in the meta-analyses did not target depression as a diagnosis per se and did not generally aim at the reduction of depressive symptoms as a main outcome. First, an overall small effect was observed for VR-based interventions at posttreatment in the analysis by Kampmann et al   may benefit from VR-based interventions that promote simulations of functional learning, the transfer of learned functions to daily life, and relaxation .A meta-analysis by Kim et al  analyzedIn summary, low to low-to-moderate quality evidence indicated that VR interventions may positively affect various clinical outcomes among patients with cognitive impairment and thus improve cognitive and routine functions. However, these VR-based interventions were not compared with active controls.This meta-review aimed to summarize the current state of evidence on the efficacy of VR-based interventions for psychiatric disorders by evaluating the data provided by meta-analytical studies. Cumulating evidence on various anxiety disorders and posttraumatic stress disorder showed that VR-based interventions displayed overall medium-to-large effects in comparison with inactive controls. However, there was globally no significant difference in comparison with standard evidence-based approaches at posttreatment, apart from significant differences with classical evidence-based interventions (g=0.353 in favor of VR) for aviophobia. With limited evidence on the superiority of one over the other, these findings suggest that both VR-based and standard evidence-based therapies are as effective for anxiety-related disorders at posttreatment. Furthermore, although results may not be dissimilar among interventions in the short term, preliminary data on aviophobia and panic disorder have highlighted that the effects of VR appear to be sustained in time, and subjects may fare better in the long term than with active controls. This suggests that although the effects of conventional treatments diminish in time, the effects of VR appear to be maintained, leading to longer-lasting positive outcomes. Such differential outcomes may be explained by the advantages of VR over classical and in vivo exposure-based interventions, comprising a more flexible and personalized approach where the therapist can better control the content of exposure , exposure rhythm, and repetition of scenarios -70.VR interventions have also shown promise in the treatment of other disorders included in this meta-review. First, although there are only a few VR interventions that have been developed specifically for individuals with mood disorders ,62, VR-bAs observed, meta-analyses serve as a useful tool to provide a global overview of the benefits of VR for patients affected by psychiatric disorders. The quality of evidence was evaluated as being quite variable, ranging from very low to moderate quality. Several reasons account for the lower quality of evidence. First, many meta-analyses included a limited number of RCTs within their analyses, thereby also lacking large sample sizes. Meta-analyses with a larger number of randomized trials were provided for aviophobia , posttraFurthermore, numerous psychiatric symptoms and disorders that are treatable by VR interventions have not been examined by meta-analytical investigation. For example, although we retrieved only one meta-analysis on compliance for schizophrenia , the pasIn the above efficacy studies, some key aspects remain to be further investigated. With the rise of personalized medicine, future research should be encouraged to achieve a better understanding of factors that may play a role in VR outcomes and help explain different effects from usual treatment. These factors may include patient characteristics  and the severity of the disorder ; certain patients may indeed be more susceptible to better respond to these VR approaches. For instance, a meta-analysis by Cardoş et al  on the sVR provides opportunities to go over and beyond traditional interventions and allows tailoring approaches to each individual, thereby possibly improving efficacy and the maintenance of skills. With variable quality of evidence, meta-analytical literature suggests positive outcomes in the VR treatment of psychiatric conditions, mainly anxiety-related disorders. VR-based interventions are better than inactive controls and generally show similar effects when compared with evidence-based approaches for these disorders. Preliminary findings also suggest that the effects of VR may be long-lasting. Furthermore, VR has shown efficacy for the treatment of depressive symptoms and neurocognitive disorders. However, support for the use of VR in the treatment of social skills in autism and compliance in schizophrenia is lacking. There are also numerous VR studies that were not included in meta-analyses that targeted other psychiatric symptoms and disorders ; these have also shown prefatory beneficial outcomes. Nevertheless, more research is necessary in the field of psychiatry to establish high-quality evidence with the use of gold-standard evidence from well-designed RCTs comprising large samples. As current VR treatments have not clearly shown superiority over conventional treatments, future VR-based interventions should focus on developing innovative approaches for complex and treatment-resistant symptoms that are difficult to address with traditional treatment. Research is also warranted to evaluate the aspects enabling the better use of VR and examine the specificity of VR-based interventions. As soon as more studies become available, systematic meta-regression analyses could statistically examine the influence of certain variables on the efficacy of VR for improving personalized patient care.