Clinical practice guidelines aim to support clinicians in providing clinical care and should be supported by evidence. There is currently no information on whether clinical practice guidelines in laparoscopic surgery are supported by evidence.We performed a systematic review and identified clinical practice guidelines of laparoscopic surgery published in PubMed and Embase between March 2016 and February 2019. We performed an independent assessment of the strength of recommendation based on the evidence provided by the guideline authors. We used the ‘Appraisal of Guidelines for Research & Evaluation II’ (AGREE-II) Tool’s ‘rigour of development’, ‘clarity of presentation’, and ‘editorial independence’ domains to assess the quality of the guidelines. We performed a mixed-effects generalised linear regression modelling.P value = 0.0071) and whether the strength of recommendation was ‘strong’ by independent evaluation  were the only determinants of difference in rating between the guideline authors and independent evaluation.We retrieved 63 guidelines containing 1905 guideline statements. The median proportion of ‘difference in rating’ of strength of recommendation between the guideline authors and independent assessment was 33.3% . The ‘rigour of development’ domain score .The online version contains supplementary material available at 10.1007/s13304-021-01168-3. Laparoscopic surgery has become the preferred technique in many fields of surgery. According to the National Cancer Data Repository (NCDR), there was an increase in the use of laparoscopy from 10 to 28.4% between 2006 and 2008 for colorectal interventions in the United Kingdom.–5. HowevThere are varied definitions of clinical practice guidelines. The current Institute of Medicine defines clinical Practice Guidelines as “statements that include recommendations intended to optimise patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” . GuideliGuideline developers usually use a formal recommendation scheme to provide a grade of recommendation for each guideline statement. This is done to provide the guideline users a sense with information on the reliability and quality of the guidelines . The mosThe Appraisal of Guidelines, Research and Evaluation (AGREE) research team developed a tool to assess the methodological quality of clinical practice guidelines in 2003 . This waThe plethora of recommendation schemes available to develop guidelines means that there is variation in the grades of recommendations in the different guidelines. Variation in the grades of recommendations may be because of the lack of standardisation in developing the guidelines . This vaIn a pilot study, we reviewed four guidelines related to laparoscopic surgery containing 191 guideline statements: in 38–74% of guideline statements, the grades of recommendation of the guideline authors were different from those made independently using the supporting information provided by the guideline authors . There hWe hypothesised that the differences in grades of recommendation of the guideline authors from those made independently using the supporting information could be related to the rigour of development, clarity of presentation, and/or editorial independence domains of AGREE-II.Of the six domains of AGREE-II, the three domains: rigour of development, clarity of presentation, and/or editorial independence are the domains most likely to be related to disagreement in the grade of recommendation between the guideline authors and an independent assessment based on the supporting information provided. One of the key aspects of the domain ‘stakeholder involvement’ (‘patient’s views and preferences’) is a part of developing the grade of the recommendation, and, therefore, was incorporated into developing the strength of recommendation. The remaining two domains scope and purpose and applicability are not related to the strength of the recommendation, as these domains cover the generalisability of the guidelines, likely barriers and facilitators to implementation, strategies to improve uptake, and resource implications of applying the guideline.The objectives of this research were to find the extent of differences between the grades of recommendation made by the guideline authors and an independent assessment of the same guideline statements based on the supporting evidence provided by the guideline authors and explore the reasons for these differences (‘differences in grading’).We followed the PRISMA guidance in the reporting . A list https://zenodo.org/record/3660007. The major deviations from the protocol include hierarchical logistic regression to account for the correlation between guideline statements within the guideline and including the strength of recommendation as a variable to explore reasons for the differences in grading.A study protocol is available at We searched PubMed and Embase using free text and controlled vocabulary terms (MeSH). We modified the search strategy for laparoscopy from a Cochrane systematic review  and thatRelated to any laparoscopic surgery.An evidence summary or citation of studies for each statement was presented to allow reclassification of statements using GRADE.A clear recommendation instrument was implemented with well-explained criteria for each grade.Each statement in the guidelines received a specific recommendation rather than an overall recommendation for the entire guideline.st March 2016 and 28th February 2019.Published between 1We included the guidelines for the systematic review if they met the following inclusion criteria:We included all guidelines that met the above inclusion criteria and did not have any specific exclusion criteria. We did not apply any language restrictions. We translated the guidelines which were not published in English using translation software (Google translate).Two reviewers among the research team independently screened the references and selected the guidelines for full-text evaluation and inclusion. We resolved any differences through discussion.Citation (each guideline was provided a unique id).Scheme used for making the levels of recommendations .Guideline statements.Level of recommendation for each guideline statement as stated by the authors.Supporting evidence for each guideline statement.Two reviewers among the research team (the first author and one of the next four authors) independently collected the following data from each included guideline.We converted each scheme of recommendation used in the guidelines to the current GRADE method of recommendation, similar to the conversion table that we used for the pilot study . The conBalance between desirable and undesirable effects.Quality of the evidence.Costs or resources utilised.Values and preferences.Two trained reviewers independently assessed the supporting evidence and calculated the level of recommendation for each guideline statement. We used the supporting evidence as stated by the authors to arrive at the recommendation level and did not search for any additional citations or information. In other words, we did not check whether the guideline authors had misunderstood or misquoted the information from the citations, or whether they excluded some key citations on the topic to support their own views. We then graded the results as ‘Strong’ or ‘Weak’ for each guideline statement by considering the following four factors used to arrive at a recommendation as per GRADE guidance , 28–31.BTwo review authors (the first author and one of the next four authors) gave scores for each item in the domains for each guideline statement independently according to the instruction manual of the AGREE-II Tool. We gave all guideline statements in a guideline the same score for some domains such as ‘editorial independence’, while we gave different scores for different guideline statements within the same guideline for the rigour and clarity domains. The scores of each item in the domain were then scaled using the following equation according to the instructions in the AGREE-II Tool Manual :\documenModel A Fixed-effects for the three AGREE-II domains , the system used for grading the recommendation , and the grade of recommendation by independent assessment.Model B Same as model A, but in addition, random-effects for guideline ID  to account for potential correlation of domain scores within guideline statements for a set of guideline statements developed by a group of guideline authors.We summarised the characteristics as median and interquartiles. We did not log-transform the data as the log transformation did not improve the normality of distribution. We used a mixed-effects generalised linear regression modelling using binary distribution and logit link in the GLIMMIX procedure of Statistical Analysis System (SAS) software (version 9.4). Initially, we ran the following two models.Model C Fixed-effects for the ‘rigour’ AGREE-II domain and the grade of recommendation by independent assessment.Model D Same as model C, but in addition, random-effects for guideline IDBased on the results of these analyses, we ran two further models.P value of < 0.05 as statistically significant. The data and codes used for analysis are available in Supplementary file S2.We chose the best model based on model fit (corrected Akaike Information Criterion (AICC)) and the area under the receiver-operating characteristics curve (C-statistic). When the results of two models were similar, we chose the simpler model . We considered a n = 1940) and Embase (n = 2850). We excluded 1179 duplicates and 3501 clearly irrelevant references through review of titles and/or abstracts. Of the 110 retrieved references, all were identified as clinical practice guidelines. We further excluded 47 references due to a variety of reasons: the majority either lacked recommendation systems or instruments or had unclear grading criteria for the recommendation instrument. We listed the detailed reasons under ‘Characteristics of excluded studies’ (Supplementary file S1 Appendix 4). In total, 63 clinical practice guidelines fulfilled the inclusion criteria. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram is shown in Fig. We identified a total of 4790 references through the electronic searches in PubMed n = 940 and EDetails of the clinical practice guidelines used in the included studies are available in Supplementary file S1 Appendix 3.In total, we included 63 guidelines consisting of 1905 statements. The number of statements from each guideline varied from 2 to 152 statements (median and interquartile: 22 ). ElevenThe ratings of the strength of recommendation by the guideline authors and the independent assessment (by some authors of this manuscript as detailed in the methods) are available in the Supplementary file S3 (Sheet 'Guidelines'). A summary of the disagreements between the guideline authors and independent assessment is shown in Supplementary file S3 (Sheet 'Disagreements') and in graphical format in Supplementary file S1 Appendix 7. The median of the proportion of disagreement of recommendations between guideline authors and independent evaluation was 33.3% .The scaled scores for the AGREE-II tool domains for each guideline statement are available in the Supplementary file S3 (Sheet 'Guidelines'). The median and interquartile scaled score for the ‘rigour of development’, ‘clarity of presentation’, and 'editorial independence' domains were 0.43 , 0.72 , and 0.63 , respectively.The output from the analysis is available in Supplementary file S4.P value < 0.001. The correlation coefficients suggest that there was very weak correlation between the three domains assessed.The Spearman correlation coefficients between the assessed AGREE-II domains are listed in Table C-statistic = 0.8167) compared to the model without the guideline ID as random-effects .In the initial analysis, the model including guideline ID as random-effects  had better model fit , the AICC and C-statistic were 2226.60 and 0.7190, respectively. In the hierarchical logistic regression analysis including only rigour of development and grade of recommendation by independent assessment as fixed-effects and guideline ID as random-effects (model D), the AICC and C-statistic were 2091.05 and 0.8151, respectively. The fit statistics of model D were similar to that of model B, indicating that the remaining variables (other than the variables included in model D) contributed very little to explaining the difference in rating by the guideline authors and independent assessment. Therefore, we have presented the results of model D. The results based on model B were similar to those of model D.Based on the results of model D, the odds ratios of 'difference in rating' (of the strength of recommendation between the guideline author and independent assessment) were 0.06 (95% CI 0.01–0.48) per unit increase in the rigour of development  and 0.09 (95% CI 0.06–0.13) of strong versus weak recommendation by independent assessment . The results of model B were similar: the odds ratios of 'difference in rating' were 0.11 (95% CI 0.01–0.90) per unit increase in the rigour of development and 0.08 (95% CI 0.06–0.12) of strong versus weak recommendation by independent assessment.The odds ratios of the different independent variables from different models are shown as forest plots in Fig. Our research identified 63 guidelines related to laparoscopy consisting of 1905 guideline statements. A number of classification systems for grading guidelines were used; the most common classification system used was the GRADE system.The strength of recommendation of approximately a third of the guideline statements was different between the guideline developers and an independent assessment based on the evidence provided by the guideline developers. Regression analysis showed that this was related to the rigour of development  and the strength of recommendation  of the AGREE-II system of assessing clinical practice guidelines. This indicates that the guideline authors were systematically overrating the strength of the recommendation .The systematic overrating of the strength of recommendation has major implications in terms of uptake of the treatment , use of resources (to implement the recommendation), medical negligence lawsuits, and in further research performed to address the uncertainty. When the recommendation is strong, the GRADE working group has suggested that the clinical practice guideline developers use terms such as 'clinicians should…' or 'should not…' (depending on whether the intervention should or should not be recommended to patients); when the recommendation is weak, the GRADE working group has suggested clinical practice guideline developers use terms such as 'clinicians might…' . This obIn medical negligence lawsuits, when a clinician has failed to recommend a treatment option which has a strong guideline recommendation, this would be considered a breach of duty of care and could lead to prosecution and suspension from the GMC. A flawed systematic overrating of the strength of evidence and poor rigour of development (of the guideline) may result in a major injustice for a clinician who interprets the evidence better than the guideline developers.Flawed systematic overrating of the strength of evidence can also impair research performed to address the uncertainties: this may result in perpetuating the wrong beliefs and further impairment of shared-decision making and appropriate resource-use.While the impact of the systematic overrating of the strength of evidence can be understood, the reason for this systematic overrating of the strength of evidence has not been investigated in this study. Some potential reasons for guideline authors getting the guidelines wrong include an unbalanced guideline developers panel with healthcare professionals favouring new treatments, vested interests of the guideline developers, lack of formal consensus methods to develop guidelines, opaque and inconsistent methods for rating evidence, failure to capture the impact of differing patients’ values and perspectives, publication bias, and absence of adequate peer-review procedures . Based oThe major strengths of this research include a thorough and systematic search of the literature to understand the prevalence of the misclassification of the strength of recommendations in laparoscopic surgery guidelines, duplicate and independent study selection and data extraction to minimise errors, and use of the appropriate analysis to identify the potential reasons for misclassification of the strength of recommendation in clinical practice guidelines.The most important limitation is that we used the supporting evidence as stated by the guideline authors to arrive at the recommendation level and did not search for any additional citations or information. In other words, we did not check whether the guideline authors had misunderstood or misquoted the information from the citations, or whether they excluded some key citations on the topic to support their own views. If the supporting evidence used by the guideline authors does not take into account all the available evidence or if the interpretation of the supporting evidence by the guideline authors is found to be wrong, the 'rigour of development' scores will be lower than that estimated by our research team and the difference in rating  will be even more than what we found in this research. This is likely to strengthen the association between poor rigour of development and misclassification of the strength of recommendation. Therefore, the findings of this research should be interpreted as minimum effect (rather than the true effect) of poor rigour of development on the misclassification of the strength of recommendation. In other words, what we found in this research is likely to be only the 'tip of the iceberg'.The other major limitation of this research is that we converted the strength of recommendation from other classification systems of grading to the GRADE system of classification of recommendations. This conversion was done by the author team and not by formal consensus methods. However, it should be noted that our research did not demonstrate any evidence that the disagreement between the strength of recommendation between the guideline authors and independent evaluation was related to the guideline classification system used. Besides, we have provided reasons for this conversion and the data used for analysis in the supplementary material, which will allow other researchers to develop their own conversion methods to analyse the impact of different conversion methods on the results of our research.There have been previous systematic reviews assessing the quality of clinical practice guidelines in other areas of healthcare , 24. TheThese findings are applicable only for clinical practice guidelines related to laparoscopy. We are unable to comment on whether these findings are applicable in clinical practice guidelines in other fields of healthcare.There were considerable differences between the grades of recommendation made by the guideline authors and an independent assessment of the same guideline statements based on the supporting evidence provided by the guideline authors. These differences are associated with poor rigour of development of the clinical practice guideline. Guideline authors systematically overrated the strength of the clinical recommendations .Supplementary file1 (DOCX 150 kb)Supplementary file2 (SAS 89 kb)Supplementary file3 (XLSX 409 kb)Supplementary file4 (PDF 2192 kb)Supplementary file5 (DOC 64 kb)Below is the link to the electronic supplementary material.


BFBMA). Strength of evidence and trialling varied between drugs. Median evidential strength was extreme for ADHD medication (BFBMA = 1820.4), moderate for antipsychotics (BFBMA = 365.4), and considerably lower and more frequently classified as weak or moderate for antidepressants for depression (BFBMA = 94.2) and anxiety (BFBMA = 49.8). Varying median effect sizes , sample sizes , and numbers of trials  might account for differences. Although most drugs were supported by strong evidence at the time of approval, some only had moderate or ambiguous evidence. These results show the need for more systematic quantification and classification of statistical evidence for psychotropic drugs. Evidential strength should be communicated transparently and clearly towards clinical decision makers.Approval and prescription of psychotropic drugs should be informed by the strength of evidence for efficacy. Using a Bayesian framework, we examined (1) whether psychotropic drugs are supported by substantial evidence , and (2) whether there are systematic differences across drug groups. Data from short-term, placebo-controlled phase II/III clinical trials for 15 antipsychotics, 16 antidepressants for depression, nine antidepressants for anxiety, and 20 drugs for attention deficit hyperactivity disorder (ADHD) were extracted from FDA reviews. Bayesian model-averaged meta-analysis was performed and strength of evidence was quantified (i.e.  Psychiatric disorders can be treated with various psychotropic drugs. With a wide variety of drugs available, choosing the most appropriate one can be difficult, highlighting the importance of good evidence. Clinicians must be able to trust that there is strong evidence that the drug is effective. In the USA, drugs must be approved by the Food and Drug Administration (FDA) before they can be marketed. Although many aspects of a drug's profile must be considered in the approval process, the statistical evaluation of efficacy plays a central role. The FDA states that substantial evidence for efficacy is provided by ‘at least two adequate and well-controlled studies, each convincing on their own’  and negative  trials. As such, crucial information like the number of trials conducted before obtaining two positive trials is ignored  has been suggested as a measure to quantify evidence holistically  and the alternative hypothesis (i.e. a treatment effect) by comparing the relative likelihood of the observed data under either hypothesis  of 30 indicates the observed data are 30 times more likely to have occurred under the alternative hypothesis than under the null hypothesis  indicates the observed data are five times more likely to have occurred under the null hypothesis than under the alternative hypothesis. Finally, a BF around 1 indicates equipoise .In some cases, the current statistical evaluation process may lead to suboptimal decisions. The assumption that at least two independent randomised controlled trials (RCTs), or even fewer, provide substantial evidence of drug efficacy has been questioned  is to exist. Ideally, effect sizes should be clinically meaningful and the strength of evidence sufficiently large that the drug can be considered effective with relative certainty. There is a great body of literature regarding effect sizes of psychotropic drugs  medication. We examine whether the current evaluation process generally leads to psychotropic drugs supported by substantial evidence . To determine whether there are systematic differences across drug groups in terms of strength of evidence, we compare them across the disorder groups and investigate whether trial programme characteristics (e.g. effect sizes and sample sizes) are related to the strength of evidence per drug within each drug group.This study involved publicly available trial-level data. No ethical approval was needed.Study information, details regarding prior knowledge of the data, and the analysis plan were preregistered at OSF before data analysis but after knowledge of the data. Deviations from the preregistration are reported in the online Supplementary material.Nrtrials = 43, ntreatment = 9937, ncontrol = 4303), 16 antidepressants approved for depression , nine antidepressants approved for anxiety , and 20 drugs approved for ADHD .Data sources were not identified by a systematic search. Instead, we obtained data for psychotropic drugs approved by the FDA. We used data extracted for previous meta-analyses, supplemented by data extracted by ourselves. Data on antidepressants for depression were obtained from Turner et al.  and de VWe included data for all available short-term, placebo-controlled, parallel-group, and cross-over phase II/III clinical trials. We excluded studies concerned with relapse or discontinuation of the medication, long-term extension trials, and studies without a placebo control group, as these do not qualify as ‘well-controlled’ trials , using the ‘BayesFactor’ (0.9.12–4.2)  or imputed . To determine the strength of evidence that an effect exists, we calculated Jeffreys–Zellner–Siow BFs . For parallel group trials, we calculated the corrected Hedges We implemented Bayesian model-averaging , in the form of model-averaged BFs (BFBMA). A total of 63 Bayesian meta-analyses were performed. For nine drug-disorder combinations supported by a single two-arm trial, we did not conduct a Bayesian meta-analysis, but used the individual BF and effect size instead.To conduct a Bayesian meta-analysis, prior distributions were assigned to the model parameters : sertraline approved for PTSD (BFBMA = 0.7), vilazodone (BFBMA = 0.5), and bupropion approved for depression (BFBMA = 2.7). Four (5.6%) meta-analytic BFBMAs indicated only modest evidence for a treatment effect (3 < BFBMA < 10): Daytrana for ADHD (BFBMA = 8.3), sertraline approved for SAD (BFBMA = 7.3), and paroxetine (BFBMA = 4.4) and paroxetine CR (BFBMA = 4.4) for PD. Ten (13.9%) BFBMAs showed moderately strong evidence for treatment effects (10 < BFBMA < 30), including five antidepressants approved for anxiety, two antidepressants for depression, two antipsychotics, and one ADHD medication. The majority of drugs (76.3%) were supported by strong pro-alternative evidence (BFBMA > 30).BFs and pooled effect sizes, individual trial BFs and effect sizes, sample sizes, and number of trials) are presented in online Supplementary Table S1. Summary results are presented in BFs were heavily right-skewed, we report the median instead of the mean.Detailed results , followed by antipsychotics for schizophrenia (BFBMA = 365.4). Median strength of evidence was lower for antidepressants for depression (BFBMA = 94.2) and for anxiety (BFBMA = 49.8). Similarly, variability in BFBMA differed between disorders. The largest variance was found in ADHD (8.3–2.3 × 1015), followed by schizophrenia (26.7–8.0 × 105). For antidepressants approved for depression (0.8–3.3 × 105) and anxiety (0.7–1.3 × 105), the range was the smallest.Although the median meta-analytic BFs are presented in online Supplementary Table S1 . Individual BFs differed  between the four disorder groups. The median individual BF was the highest for ADHD (BF = 185.0), followed by antipsychotics (BF = 21.0) and anxiety (BF = 6.8). The median individual BF was the lowest for depression (BF = 1.3). Trial strength of evidence varied for all disorder groups ,ESlog( = 0.492), whereas the scatterplot of sample sizes and individual BFs does not show a clear pattern ,Nlog( = 0.056), although this may be due to confounding with other factors (such as effect size).The relationships between individual n = 20, ES = 1.74, BF = 189.4 and n = 39, BF = 3.70 × 1014). This was the case for parallel trials for depression and schizophrenia, as well .We observed variation in sample size for trials concerning ADHD medication (20–563). This can be partially explained by the inclusion of cross-over trials, which by design have comparatively small sample sizes. Trials were generally randomised, controlled, parallel group trials ; however, crossover trials were sometimes performed for drugs approved for the treatment of ADHD. Smaller sample sizes did not necessarily correspond to ambiguous evidence. For instance, two cross-over trials for ADHD indicated very strong pro-alternative evidence with small sample sizes (BFs were found for depression (BF = 1.3), which might be explained by the small effect sizes (ES = 0.27). As illustrated in Fig. 3a, antidepressants for depression commonly displayed the smallest effect sizes, corresponding to the smallest individual BFs. In contrast, antipsychotics and ADHD medication showed larger effect sizes  corresponding to stronger evidential strength.The lowest individual BFs, as the number of trials for each ADHD drug was very small. In contrast, very strong evidence for depression was generally achieved through a large number of trials, despite small studies and very low individual BFs. Larger numbers of trials corresponded to a greater proportion of trials being deemed questionable or negative by the FDA. For example, for paroxetine for depression, 16 trials were mentioned, of which nine were deemed questionable or negative. Our Bayesian re-analysis suggested evidence for an additional trial to be ambiguous. Nonetheless, the meta-analytic BF suggested very strong pro-alternative evidence for the drug to treat depression (BFBMA = 10 267.8).The very strong evidence obtained for most ADHD medications appeared to be primarily due to high individual BF) tends to point towards either zero (in case the null hypothesis is true) or infinity . However, we do not observe this relationship across drugs in practice (rBF),Nr. trialslog equal more evidence for the more probable hypothesis. In other words, with accumulating data the evidential strength . In a few instances, drugs were approved despite ambiguous statistical evidence (i.e. 1/3 < BFBMA < 3). Sometimes approval was based on other considerations. For example, bupropion SR (sustained-release) was approved based on bio-equivalence with immediate-release bupropion, despite negative efficacy trials for bupropion SR (U.S. Food and Drug Administration ‘Bupropion SR’, can lead to inconsistent decision making in practice. Under the Bayesian framework, endorsement of this drug would not have been recommended.Using a Bayesian approach allowed us to identify cases in which psychotropic drugs were approved with moderate or even ambiguous evidence for its efficacy. Approximately, a quarter of all meta-analytic For the purpose of drug development and endorsement, Bayesian meta-analysis offers several advantages over classical, frequentist meta-analysis, suggested by the FDA (U.S. Department of Health and Human Services et al., both pooled effect sizes and evidential strength. Effect size estimates from the current analysis are similar to those from previous meta-analyses. Combining pre- and post-marketing studies, effect sizes for methylphenidate for ADHD, antipsychotics, and antidepressants approved for depression were estimated to be 0.77, 0.51, and 0.38, respectively (Leucht et al., Bayesian meta-analysis yields BFs to estimate the strength of evidence. The network meta-analyses conclude with rankings based on efficacy and safety data. We offer additional insight into the strength of evidence for efficacy. Sometimes, our rankings align, lending further support to the efficacy of the drug. For example, based on effect size, Huhn et al. (BFs advise caution. For example, based on effect size, Huhn et al. (BFs can help to refine rankings based on efficacy and safety data. For example, Cipriani et al. (BFs offer a valuable source of information for clinicians. Prescription and use of psychotropic drugs has steadily increased over the past few decades (Ilyas & Moncrieff, Our study adds novel information to previous research by using n et al.  ranked rn et al.  ranked oi et al.  performeBFs offer a valuable source of information when effect sizes are highly comparable between drugs. For instance, the APA concludes that many antidepressants are equally effective (American Psychological Association, BF could be used as an additional criterion, as antidepressants vary substantially in evidential strength (see also Monden et al., Moreover, BF qualitatively in the present analysis.Adopting a Bayesian framework enabled us to capture differences in evidential strength between disorders and drug groups. Nonetheless, the results should be considered in light of a few limitations. First, although we used information from the FDA-registered trials, limiting the influence of reporting bias, we were confined to data from approved drugs and pre-marketing studies. Consequently, we cannot speak to the process and statistical evidence of non-approval, or strength of evidence after post-marketing studies. As such, the current results only reflect the evidential strength at the time of approval but are not necessarily accurate reflections of the current state of evidence. Second, some values were unavailable and had to be imputed, which might have introduced extra noise. Nonetheless, imputation did not seem to be associated with increased between-study heterogeneity as indicated by comparable posterior probabilities of the random-effect model between drugs for which test statistics were available and drugs for which test statistics were imputed. Finally, Bayesian analysis is dependent on the choice of prior. Although this is an often-heard critique, we mostly restricted our analyses to default priors to ensure comparability of our results across drug groups. Furthermore, our sensitivity analysis indicated that different choices for the scale parameter of the prior did not change interpretation of the BFs.The main strength of our study is that, to our knowledge, we have performed the first large-scale comparison of evidential strength between several disorders. Previously, Bayesian methods have been proposed for and discussed in the context of the drug development and endorsement (Burke, Billingham, Girling, & Riley, BF as a measure of evidential strength might offer a valuable, additional source of information and helps to set up a consistent and transparent standard for evaluating strength of evidence of efficacy in the approval process of psychotropic drugs.Taken together, the present analysis offers interesting insights into the evidential strength within and across different psychotropic drugs. We observed large differences in evidential strength and trialling between disorders. Although the majority of re-analysed drugs was supported by substantial evidence, we also observed cases where the current approval process led to endorsement despite ambiguous statistical evidence. Moreover, evidential strength differed greatly between drugs and across disorder groups. Lower evidential support for efficacy was observed more frequently for antidepressants. Differences in evidential strength might be a consequence of different standards in trialling. The


Collaborative virtual environments are being used in various applications ranging from online games to complex team training scenarios. The key to the success of such environments is the ability of the participants to form a shared mental model of the collaborative task being performed. Poor quality of service can deteriorate user performance and quality of experience, leading to a disruption of this mental model. While the effects of quality of service have been analyzed for traditional desktop environments, these effects remain unclear in collaborative virtual environments during user-to-user interactions. Here, we analyze the role of latency and packet bursts, two common problems in collaborative applications, on both simulator perception and actual task performance in a collaborative fire-fighting simulator. This exploratory study indicates that large latencies have a significant (p < 0.05) impact on the quality of experience, but not task performance. In contrast, packet bursts have a much larger impact on both the quality of experience and performance. Additionally, the network role, such as whether a user is a client or server, showed a significant (p < 0.05) impact on task performance in conditions impaired by packet bursts. For example, CVEs in gaming can promote teamwork through shared objectives, encouraging participants to share resources and take on specific roles. Other collaborative applications include team training, which is a critical component of many professions such as emergency response and surgical teams. To accomplish these collaborative tasks, it is critical for participants to construct a shared mental model (SMM)9. In psychology, an SMM is a shared understanding of the environment and tasks among a team of participants, and this shared understanding is often critical for improving performance in collaborative tasks. However, this SMM can be disrupted by poor quality of service.With the rise in popularity of multiplayer gaming and consumer head-mounted display (HMD) systems, multi-user virtual reality (VR) applications have become ubiquitous. VR collaborative virtual environments (CVEs) are virtual reality applications where users can interact with one another to accomplish a larger goal. VR CVEs enable the creation of a wide range of entertainment and team-training applications18, jitter (the variance of latency)19, and packet loss. Unstable internet connections and various sources of delay, such as infrastructure and application-level code, are common causes of poor QoS. Even as internet infrastructure and standards improve, mitigating the effects of latency remains a significant technical challenge. For example, both LTE round-trip times20 and Wi-Fi21 can add hundreds of milliseconds of latency to a networked application. Unfortunately, because of the real-time nature of video games and 3D interactive simulations, latency and jitter have been shown to have a substantial impact on task performance and QoE in these applications22. In developing countries, these conditions can be even worse, limiting the market penetration of collaborative VR applications and raising questions of fairness among participants with differing QoS23.Computer networks that provide the backbone to CVEs are subject to unpredictable drops in performance. Network connection quality, also known as quality of service (QoS), can vary significantly due to underlying network infrastructure and network congestion. Poor QoS can deteriorate user performance and the quality of experience (QoE), the user’s perceptions of how the environment affects his/her performance, in networked applications. QoS is characterized by multiple factors that affect QoE: latency (time delay from sending to the reception of a packet)9. When the QoS worsens, each instance of the shared virtual world diverges. In turn, the SMM can deteriorate, making it more difficult for participants to complete tasks that require team work. SMMs are formed through team learning behaviors including construction, co-construction, and constructive conflict9. Participants engage in “construction” when they perform actions or make decisions such as tossing a fire extinguisher in our simulator. “Co-construction” involves other participants building upon construction behaviors, such as catching the extinguisher. “Constructive conflict” occurs when team members do not agree on the SMM, which can occur due to interruptions in network connectivity. In our study, we explore how different types of QoS disruptions can shift the team learning behavior.Collaborative tasks benefit from the construction of SMMsWhile user performance and QoE as a result of QoS has been studied in competitive and collaborative games, as far as we are aware, there has not been a comprehensive analysis of the effects of both latency and packet bursts in HMD-based VR tasks requiring user-to-user collaborative interaction. We developed a multi-user simulator as a platform to collect data on the effect of various QoS metrics in VR CVEs. We then conducted a study to gather data on user behaviors as affected by these network conditions.What effects do high latencies have on both the perception and actual task performance of users?Does a user’s role (client/server) in the network have an impact on QoE or task performance?Can latency offset the effects of packet bursts?By answering these questions, we can better understand user tolerances and preferences in VR CVEs. These answers provide insight into the potential directions that could be taken when designing networking architectures for collaborative applications. Optimal network designs can reduce the degradation of SMMs under poor network conditions, leading to better QoE and task performance.Through our user study, we provide answers to the following research questions in the context of collaborative VR environments with person-to-person interactions:25. To assist in future analysis, we recorded the trajectory data of the extinguisher, hands, and head of each user. We also recorded fire extinguishing time and metadata, including who has authority over the fire extinguisher at a given time. We simulated different network conditions using software and conducted a user study to answer our research questions.To measure the effects of QoS on SMMs, we developed a VR simulator inspired by a real training scenario for extinguishing a fire in an operating roomIn our simulator, three participants work on a collaborative task in a virtual environment, and each participant is represented by an avatar Fig. . Each paEach fire spawns sequentially; once one fire is extinguished, another spawns until all nine fires are extinguished. A few rules dictate the fire spawning behavior: (1) the first fire is always the same (fire 0); (2) for every three fires, all three colors must be represented ; (3) two sequential fires cannot have the same color. These constraints were created to ensure that the virtual fire extinguisher was tossed from one user to another after extinguishing each fire and to ensure that each participant was actively engaged throughout the trial. This also creates an environment in which the users need to understand how they need to coordinate to fulfill each others’ tasks and complete the shared objective of extinguishing all of the fires. This randomized fire placement was designed to minimize the ability of participants to learn the order of who would throw the extinguisher next and anticipate the next throw, which forces the users to maintain a high level of situational awareness.11. Latency tolerance generally depends on the application type and associated mechanics. For example, Claypool and Claypool10 found that first-person avatar, third-person avatar, and omnipresent games have latency tolerances of 100 ms, 500 ms, and 1000 ms, respectively.For the user study, we tested five different conditions Table . In thes26. In our studies, we evaluate the effect of these “packet bursts” . Our goal in evaluating packet bursts is to measure the effect of “lag,” a loosely defined term for the responsiveness of an application often used by gamers to quantify the QoS28. Unlike uniform jitter, the effects of packet bursts have not been extensively studied.While some network conditions such as latency are well-defined and straightforward to emulate, other conditions such as jitter are not. Jitter does not follow a uniformly random distribution across real-world networks and can be largely masked by packet reordering schemes; Qin et al. showed that packet loss that fails to model burst loss might not be perceivable by users29. Similar effects can also occur in UDP, such as through burst loss. Similar to latency, the values are along one direction.Packet bursts, which simulates a cumulative effect that could be caused by network issues such as packet coalescing, low update rates, and packet congestion, is defined by two values: the amount of time to block all incoming/outgoing packets and the chance of this blockage to occur for a given packet. Because the chance is applied per packet, more packets sent during a frame contributes to a higher chance of triggering a data throttle. In our simulation, the server sends out 15 packets, so there is a high chance of a trigger occurring each frame. However, because the packets are sent sequentially, an individual object may not be throttled each frame. Additionally, once the throttle time period ends, all blocked packets are sent. This closely simulates the hitches caused by TCP correction during packet loss27. We take a more systematic approach by introducing this freezing mechanism at the network level, and we determined parameters for packet bursts that appeared to best emulate lag through preliminary testing. Our questionnaire responses regarding lag confirmed that our packet bursts settings were a reasonable proxy for lag  for our study. Prior to the start of the study, each participant was asked to fill out a demographic questionnaire. None of the participants reported having prior motion sickness in VR. Eleven (55%) of the participants reported having prior VR experience. Eight (40%) of the participants regularly play video games each week.The study protocol was approved by the Rensselaer Institutional Review Board (IRB). Signed informed consent was obtained from each of the participants in the study. All methods were performed in accordance with relevant guidelines and regulations. At the start of each study group, we introduced the participants to the simulator. We allowed each participant to learn the controls through a test trial without impaired network conditions. This practice trial was designed to mitigate the practice effect and to familiarize the participants with the controls and the virtual environment.Three participants were selected to participate in five trials (with each network preset); each trial consisted of each user extinguishing their three fires for a total of 9 fires extinguished per trial. The presets were selected through block randomization to mitigate the order effect. After the five trials, the participants either changed positions or were replaced by a waiting participant. Each participant participated in 15 total trials at the three different positions. Each study group took no longer than 2 h.At the start of each trial, the participants were asked to return to the starting circles on the ground corresponding with their starting locations. P1 always started the simulator and extinguished the first fire. After each trial, the three participants were asked to fill out a questionnaire about their perception of the network conditions; each participant completed 15 post-trial questionnaires in total.https://www.imstk.org/). We utilized iMSTK’s Vulkan backend30 to render the VR environment efficiently. To simulate the rigid body dynamics, we used Nvidia’s PhysX framework and advanced the physics simulation at 60 Hz. Each computer in the networked simulator runs a redundant copy of the scene, so each is responsible for computing physics and animation.We constructed our networked simulator using the Interactive Medical Simulation Toolkit (iMSTK)  and are in a single room, allowing all participants to share the same tracking area and thus use the same tracking hardware.Our network design only uses the UDP protocol, and each computer sends update packets at a fixed frequency, which in our case was 30 Hz. UDP does not include functionality to ensure the redelivery of dropped packets. Thus, to ensure critical information such as simulation state changes and object authority are updated on the server, each computer sends packets for objects they control. However, in our study, it is unlikely for a significant number of packets to be lost because the computers were connected through ethernet in the same room. In the fire extinguisher scenario, all simulation states are tied to the user who possesses authority over the extinguisher. The state is then redundantly sent to the server until another user acquires the extinguisher.While UDP lacks some useful features that TCP provides, it can more easily be customized with features implemented at the application layer. One of the most important features is the ability to broadcast packets, sending them to multiple network endpoints at a given time through a single port. Another benefit is the ability to ignore lost or out-of-order packets. Because our simulation environment operates in real-time, receiving packets in a timely manner is a priority. Out-of-date packets are simply discarded if newer packets have already arrived. We use a redundant state design, so dropped packets are less of a concern; for example, if a user requests authority, that request is embedded in all future packets until authority is given to another user. With this approach, authority can still be updated even if the initial authority request packet is lost. On the other hand, TCP will resend lost packets and packets that are out-of-order to ensure correct sequencing of packets, which can lead to increased latency and grouping of packets that behave similarly to our packet bursts preset.During each frame, the server sends 15 packets, each 512 bytes, to prevent fragmentation, to the clients. Each client will send less than 15 packets, but the exact number depends on the authority of the various networked objects. Each object is encapsulated within a single packet. Additionally, a ping packet is sent from each client to the server, and a ping packet is sent from the server to the clients. The ping packet contains information about the current state of the networking environment, such as local time and client ID number, and it is also used to initially connect the clients to the server.https://github.com/jagt/clumsy), which intercepts packets and buffers them to simulate various network conditions such as latency and packet bursts. We configured it to affect all inbound and outbound UDP traffic through the simulator’s port. Because both clients are not directly connected to one another, the networking preset values are effectively doubled when interacting between the two clients , as experienced by the users, were measured using a questionnaire that was provided at the end of each trial. Each trial consisted of participants P1 (red), P2 (blue), and P3 (green), where P1 served as the central server role. We study the latency presets  and the packet burst presets (500 ms + 15 ms@25% and 500 ms + 30 ms@50%).Errors were measured by the number of drops, or missed catches, made by the receiver of the fire extinguisher. We found that catching the extinguisher became very difficult in poor network conditions due to the unpredictability of delivered packets. Measuring the number of drops presented a few challenges. First, the trajectory of the extinguisher diverged among different participants. For example, if P1 threw the extinguisher, then P1 controlled the physics calculations such as gravity or bouncing off a wall. With latency present, the extinguisher would appear to have been dropped by the receiving client from P1’s perspective. These differences in user’s perspectives make it difficult for teammates to effectively coordinate their behaviors and understand the performance of the team. Second, in some cases, particularly those with severe packet bursts, when and if the extinguisher made contact with the ground became difficult to judge due to the application’s rejection of outdated packets; rejected packets resulted in missing positions along the extinguisher’s trajectory.To accurately count the number of drops, we created a replay system to allow us to visualize synchronized trajectories; the trajectory that was used was the receiving user’s trajectory to ensure that we were viewing what the receiver was viewing. Additionally, we only counted a successful catch if the receiving user retained control. If the user briefly caught the extinguisher but then dropped it almost immediately, it was considered a drop. Finally, if a user mishandled the extinguisher during the throw and dropped it but later picked it up and reattempted the throw, the throw did not count as a drop.To analyze the main effect of latency, we conducted a one-way repeated-measures ANOVA comparing the performance between latencies of 0 ms, 500 ms, and 1000 ms . The mean time for completing a single trial was 59.77 s, with a standard deviation of 7.57s. Between the three latency presets , we found the effect of latency on completion time to be significant (p = 0.001) through a one-way repeated ANOVA  as perceived by the users through a questionnaire about perceived network performance that was completed by each user at the end of each trial. We asked users to rate five subjective questions on a Likert scale (1 lowest − 5 highest); Table Each group evaluated the simulator worse in accordance with a decline in QoS. Both latency and packet bursts had a significant effect (p < 0.0005) on the response for all five subjective questions. Increasing the latency had large effect sizes, with a mean of Although there was a significant change in perception for different network presets, we did not observe a significant effect on questionnaire responses due to network role (p > 0.05). Users appeared to give consistent feedback, despite their difference in positioning.Through our study, we were able to determine how latency and packet bursts affected a user’s perception and behavior. Latency had little impact on the average number of errors made by each user. However, increased latency led to worse perception of network conditions, although not to the same extent as the packet burst presets. Lastly, latency increased completion time, although this is largely due to the added delay in network communication. While the exact simulation differs for each user, the overall movement behavior and intent was preserved, resulting in the preservation of the SMM through refining their approximate interpretation of the virtual environment .41, aviation45, and robotics47. In these situations, mistiming of information retrieval and situation assessment between team members can lead to improper responses.On the other hand, the packet burst conditions had a larger impact on both QoE and actual task performance. The added completion time was likely due to error correction methods . Additionally, the drop rates and QoE were much worse as compared to the baseline condition of 500 ms. Packet bursts will cause the trajectory of remotely-controlled objects to appear choppy. While the fire extinguisher is moving through the air, the flight path of the extinguisher effectively loses some of its frame updates, which makes predicting the current position difficult. An increase in the chance of packet bursts has the effect of increasing the uncertainty of where the extinguisher will be in the next update. These significant changes in simulation state lead to constructive conflict of the SMM. Although participants diverge significantly in their interpretation of the correct state, participants are able to adapt to these diverging conditions at the expense of task efficiency. This can have a critical effect on decision making in high-stakes, fast paced, team-based virtual environments such as surgery23. If the packet bursts or related conditions  were to occur, additional buffering and interpolation should be a more favorable solution as opposed to trying to further minimize latency if doing so were to introduce more packet bursts.While it would be best to minimize latency to below the values selected in our trial, adding latency can be applied rather liberally to stabilize the simulationAdditionally, the perception of the difficulty in interacting with collaborators did not have as large a decline as the perception of lag, but this trend is not as evident in the packet burst scenarios. Consistent with the number of errors for increased latencies, increasing latency does not appear to inhibit actual performance in collaborative tasks.Since one of the users hosts the simulation server, the roles are asymmetric in nature; the connections from the host PC are different from the connections to the client PCs. We found that asymmetric roles impacted task performance but not QoE. While all three roles gave similar responses to the QoE questions, it was observed that both the users at the client computers (P2 and P3) dropped the extinguisher, on average, significantly more than the user at the server computer (P1). The number of drops, particularly with more packet bursts, was significantly different for P2 and P3 compared to P1.Although P1 is also affected by the network conditions that P2 and P3 encounter, interactions between P2 and P3 are worsened by the added connections. For example, P2 must send data to P1, and then P1 must send packets to P3. Therefore, the network conditions can be worse since the packet bursts have a cumulative effect over the multiple connections. In a dedicated server setup, however, the roles would be effectively symmetric, at the cost of adding an additional network connection between one of the user’s computers.The authority model has some drawbacks in terms of presenting a consistent state. State consistency becomes a problem in any multiplayer environment that relies on users independently calculating part of the simulator state. From a user’s perspective, latency may not be apparent between some of the users. For instance, from P2’s perspective of P3 tossing the fire extinguisher to P1 may be that P1 is not experiencing latency with respect to P3. However, this is not the case; the latency is merely masked because P2 is receiving P1’s  view of the current scene.Other interactions more clearly show latency. For example, P1 tossing the extinguisher to P3 will appear to have latency from P2’s perspective. This occurs because P1 is merely acting as a pass-through point for the fire extinguisher, and P3’s acquisition of authority of the fire extinguisher will not be updated by the server until after the latency period  has passed. The latency creates a visual artifact for this example Fig. ; it is cImmersive collaborative VR applications have the potential to become the future of team training applications. However, poor quality of service can inhibit the formation of shared mental models and worsen individual and team performance, limiting dissemination to people and societies without access to reliable or high-speed internet. In this work, we show that (1) latency decreases QoE but not task performance, (2) packet bursts affect task performance and QoE more than large latency, and (3) network role significantly impacts task performance.18, we found users can tolerate at least 1000 ms of latency with an authority model. On the other hand, our results are consistent with the range of tolerable latencies that were introduced for omnipresent games (1000 ms)3. While the application presented in this paper is not omnipresent, the types of user interactions are more resilient to latency. Buffering multiple frames and then interpolating them can reduce the effects of more sensitive conditions such as high packet loss, packet coalescing, and packet bursts, and provide better user performance at the cost of adding additional latency.Since latency seems to have little effect on performance in our designed task, similar future applications should be able to exploit the finding that latency has little impact on performance, while packet bursts can have a large impact. While Vlahovic et al. found latency values as low as 200 ms can affect performance in networked PvE cooperative VR gamesThere exist multiple limitations in our study that we look to address in future work. The first was the combination of latency and packet burst values. While it would be ideal for testing each of the three distinct latency values against each of the three packet burst values for each of the three distinct positions, to do so would have required a group commitment too long to request of participants. Instead, we based the testing of throttle values at 500 ms as an upper threshold to see the impact that it would have on QoE and QoS. The success of the experiments performed in this study, while rudimentary, provides baseline results to compare against subsequent, more complex studies. Future studies should investigate the effect that occurred at those other combinations or looking at a finer scale. Future research will also explore the impact that latency and packet bursts have on more complex interactions in VR, such as soft-body and fluid simulation, as those may have profound effects on the ability of someone to interact within a virtual environment.Furthermore, while asymmetric connections do not show a significant difference in the perception of network conditions, it can penalize participants who are not hosting the game session. Depending on the application and evaluation criteria, performance can vary significantly across users in different networking roles Video .Supplementary Information.Supplementary Video.


This systematic review aims to investigate the evidence in applying a co-design, co-productive approach to develop social prescribing interventions. A growing body of evidence suggests that co-production and co-design are methods that can be applied to engage service users as knowledgeable assets who can contribute to developing sustainable health services. Applying the Preferred Reporting Items for Systemic Reviews and Meta-Analyses (PRISMA) guidelines, a systematic literature search was conducted. Peer-reviewed articles were sought using electronic databases, experts and grey literature. The review search concluded with eight observational studies. Quality appraisal methods were influenced by the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) Framework approach. A narrative thematic synthesis of the results was conducted. The evidence suggests that a co-design and co-productive social prescribing can lead to positive well-being outcomes among communities. Barriers and facilitators of co-production and co-design approach were also highlighted within the evidence. The evidence within this review confirms that a co-production and co-design would be an effective approach to engage stakeholders in the development and implementation of a SP intervention within a community setting. The evidence also implies that SP initiatives can be enhanced from the outset, by drawing on stakeholder knowledge to design a service that improves health and well-being outcomes for community members. Many long-term health issues have their route in socioeconomic and psychological issues that medical interventions cannot always sufficiently surmount . Social In addition, patients with long-term physical conditions have also reported becoming self-sufficient in managing their conditions as SP interventions can also connect individuals with groups and organisations that aim to establish healthy lifestyle behaviours ,8. FurthHowever, when assessing SP interventions it is imperative to be cognisant of the limited evidence base which mainly consists of small-scale evaluations  that areSince SP is a person-centred intervention, it was decided to ascertain the effectiveness of applying a co-produced and co-designed approach to its development. The term co-production is generally understood to mean a mutual relationship between service providers, service users and their families and communities . The terPrevious studies have stated that a co-productive and co-designed approach is necessary in the development of interventions that seek to improve community well-being outcomes. Examples of such interventions include healthy aging programs ,20 non-mAs yet, no systematic review has examined the evidence on SP interventions that apply a co-productive and co-designed approach to improve well-being. This review aims to examine evidence of such interventions within community settings. The objective is to review the evidence base to establish current standards in SP that engage communities in co-design and co-production leading to improvements in well-being as well as examine barriers and facilitators to SP intervention development. Community well-being outcomes will also be assessed as an indicator of the SP interventions’ effectiveness.The protocol for this review was registered on the University of York, Systematic Review database, PROSPERO . The PatThe following databases were searched: Web of Science; CINAHL; ASSIA; PsycINFO; PubMed incorporating MEDLINE; The Cochrane Library . Targeted searching was also conducted on the CRD database. Additional search strategies included hand-searching the key journals within the database search results, targeted searching of grey literature on Google and Google Scholar, and enquiring personal contacts within the field. The inclusion criteria were all papers relating to SP interventions that apply a co-productive or co-designed approach to improve well-being outcomes in a community setting. There was no restriction on study type. In this present review well-being is defined as people’s feelings, how they function on a personal and social level and their own overall evaluation of their lives . CommuniArticles were initially screened by two reviewers (G.T. and M.L.) for relevance against the eligibility criteria based on their titles. Two reviewers independently assessed the remaining studies by their abstracts and keywords, and all reviews considered relevant were obtained in full. A consensus was reached and documented on all articles meeting the inclusion and exclusion criteria. Disagreements were resolved through a discussion with the third reviewer (L.H.S.). See The clarity of the study’s aims and objectivesRisk of bias , performance bias, detection bias, attrition bias and measurement bias)Indirectness Tests of significance and their resultsPublication bias  effect? Is there any chance of funding bias?)Included studies were subdivided into themes and two reviewers independently extracted data from the final included papers. Data extraction forms were created for the review and piloted on the final included studies. Final data extraction criteria included: Study characteristics; Participants characteristics; intervention content and context; data collection methods and outcomes. All final included papers were critically appraised for methodological quality. Quality appraisal methods were influenced by the GRADE Framework . FollowiFollowing assessment of all the above factors an overarching quality level was determined for each study using GRADE levels. Quality appraisal outcomes are presented in The included studies objectives and data collection methods are presented in All studies ,36,37,38n = 3) studied a co-produced and co-designed approach to the development of a SP intervention to improve well-being within a community setting. These studies considered the dynamics and characteristics of the collaboration between service providers and service users and their communities. The remaining studies (n = 5) analysed the community outcomes and perspectives of a SP intervention that applied a co-produced or co-designed approach to improve the community’s well-being. A thematic synthesis [Two themes emerged among the studies. A proportion of the studies  brought convenience and consistency ,38.The evidence highlights the importance of evaluating the intervention from the outset. A lack of evaluation meant that GPs and health professionals were less likely to continue their contribution to the co-production of the intervention in the long term, due to healthcare professionals’ responsibility to prescribe effective and unharmful resolutions . HoweverAnother observation in each study was that collaboration depended on adequate provision of the necessary resources. Necessary resources included the investment of time to develop the collaborations. The evidence indicates that for SP to work, healthcare professionals should be ready to adapt a more holistic model of health which entails making time to assess patients’ well-being and become acquainted with community resources of support. GPs reported that they did not always have the time to a fully assess patients’ well-being and therefore, could not make referrals . SimilarIn terms of physical resources, concerns about the lack of consistency within the third sector organisations capacity were reported within the evidence. The VCS organisations were often dependent on short-term funding, which resulted in an “unintended unreliability” . The eviOther publications identified in the SR focused on wider community outcomes and/or their perspectives of co-produced co-designed SP interventions within community settings. Five studies were included under this theme which consisted of four case studies ,33,34,37An increase in confidence was a common well-being outcome reported within the studies. This was mainly as a result of a reduction social isolation as the SP intervention motivated participants to join social groups and build a social network of support ,34,36. TAs well as giving individuals this sense of control, being able to co-produce or co-design with the SP intervention provider also meant that participants felt the staff were approachable, which encouraged their participation . The eviThe evidence also suggests that the reciprocal relationships established between service users and service providers were particularly beneficial in creating positive well-being outcomes. Participants reported that being able to help others within a similar situation was rewarding and empowering as it led them to realize the strengths and weakness in themselves and others ,34. TherThe studies also provided insight into the possible obstacles that prevented service users from participating in the SP interventions. The reported obstacles were mainly due to the individual’s personal situations. One of the most apparent obstacles within the evidence was lack of transport options ,33. In aThe evidence also demonstrated the power of creating opportunities for SP practitioners to reflect on participants outcomes and perspectives. Reflecting on such findings through an action learning framework was shown to enable practitioners to explore participants to achieve positive well-being outcomes and the implications for their own practice . EstabliThis systematic review set out to examine the evidence in developing SP interventions that apply a co-productive, co-designed approach to improve well-being outcomes in a community setting. The evidence demonstrates that co-production and co-design can be an effective way of engaging stakeholders in the development of a SP intervention to improve well-being in a community setting. Consistent with other studies of co-produced and co-designed community well-being interventions  it was rThe evidence indicates that positive well-being outcomes were achieved as a result of such an approach. Well-being outcomes across the included studies were reported to have been an increase in confidence, empowerment, and self-sufficiency as well as reduction in social isolation. Positive well-being outcomes were reported among individuals with long term conditions ,33,36, mHowever, the SR evidence also indicates clearly that there are facilitators and barriers that can influence the success of the co-production and co-design of a SP intervention within community settings. Effective leadership was reported essential in advocating the necessary mutuality between co-producers ,38 a finThis review also found that a context with adequate resources is also vital to the sustainability of co-produced and co-designed SP interventions. Similar to other studies of co-produced and co-designed health interventions  this revA key strength of the present study was the explicit inclusion and exclusion criteria that were applied to discover relevant studies that could achieve this study’s aim and objectives. In addition, a second and third reviewer were consulted during all stages of the review process to increase the robustness of the review and reduce the risk of bias. However, despite attempts to avoid publication bias, the current review only searched for studies published in English due to limited translating resources. Therefore, it must be acknowledged that the search strategy may not be comprehensive. The articles found in the review are mostly qualitative, however the quality assessment criteria were influenced by tools designed to evaluate quantitative studies. It is therefore recognized that this aspect could also produce a bias in the interpretation of the results.As previously discussed, all included studies were of a low-quality standard. In addition to the previously stated limitations, the exact number of participants within some studies is unknown  as well This systematic review has demonstrated that taking a co-designed, co-produced designed approach to SP intervention development, implementation and evaluation has the potential to positively engage with stakeholders and encourages buy-in and utilisation among end users and sense of ownership for the intervention. Stakeholder engagements also leads to a SP intervention that is effective and efficient in meeting the health needs as well as improving health outcomes of end-users. The evidence presented within the current review therefore suggests that a co-designed and co-produced also contributes to the development of sustainable healthy communities as it leads to interventions that are tailored to community needs and available community resources. This finding reinforces previous studies that have highlighted that engaging providers and end-users in the development of SP interventions is key to sustainable SP interventions . As a reHowever, this systematic review also indicates that the current evidence base of co-designed, co-produced SP interventions is limited and consisting of low-quality studies. Future recommendations for SP research include high quality studies that include all stakeholders to further confirm what makes such an approach effective within the development of SP interventions as well as limiting its success. The evidence also demonstrates a lack of effective evaluation of co-produced and co-designed SP interventions. Consequently, to improve the quality standards for research and reporting future co-design and co-production of SP interventions should also build evaluation in from the inception phase onwards to the implementation. As the field of SP is an emerging area, there are many examples of frameworks that can be used to facilitate such steps such as a Realist Evaluation  and SociThe evidence strongly suggests that a co-production and co-design would be an effective approach to engage stakeholders in the development and implementation of a SP intervention within a community setting. The results of this review also indicate that SP initiatives can be enhanced from the outset, by drawing on stakeholder knowledge to design a service that improves community members health and well-being outcomes. Taken together, the facilitators and barriers of co-production and co-design highlighted within the evidence suggests how to efficiently implement such an approach to the development of a SP intervention within a community. When a co-production and co-design is successfully applied, the evidence illustrates that it can improve well-being outcomes, and communities feel empowered by this patient-centred approach. However, caution must be applied since this review consists of only a small number of low-quality studies. Therefore, SP interventions that apply a co-productive, co-designed approach to improving well-being outcomes in a community setting require more, high quality research to further investigate which mechanisms of such an approach lead to better well-being outcomes for communities.