The factors influencing tourist loyalty are widely highlighted in the literature. However, we find that the relationship between some influencing factors and loyalty is still inconsistent, and we don’t yet know the strength and magnitude of the relationships. To address this issue, this study examined a meta-analysis of the five factors  influencing tourist loyalty and its sub-dimensions.2. The results were obtained by combining multiple single effect values into the combined effect value.The samples included articles from major academic databases, including Web of Science, Wiley Online, EBSCO, SAGE, Taylor and Francis, and Elsevier. Studies written in Chinese were retrieved from CNKI.com. We used the following keywords for retrieval: loyalty, behavioral intention, recommendation intention, word-of-mouth, revisit intentions, intention to revisit, willingness to recommend, and similar related terms. Conceptual and empirical studies published between January 1989 and September 2021 were extracted. To test whether there was publication bias, we used Fail-Safe-Number (FSN) to verify the stability of the results. The homogeneity test of the selected statistical model was based on the Q test and IWe developed 21 hypotheses and proposed a theoretical framework and analyzed 114650 accumulated sample sizes from 242 independent empirical studies. Among the 21 hypotheses proposed in this paper, the remaining 20 hypotheses have been proved except for hypothesis H6.The findings showed that the five factors had varying degrees of positive and significant relationships with tourist loyalty and its sub-dimensions. In the descending order of effects, the five factors are degree of satisfaction, quality of experience, perceived value, perceived quality and motivation. We discussed the significance of the meta-analysis, theoretical and practical implications for destination marketing. Tourist loyalty plays an essential role in maintaining the competitiveness of the destination market. Today, the global tourism industry is rapidly expanding and highly competitive. Despite the impact of the current COVID-19 pandemic outbreak on tourism, once life returns to a new standard, event organizers will face a new market where they need to make more efforts to grab market share. Consequently, managers must make elaborate plans and develop destination tourism products to attract tourists and maintain market competitiveness . The empThe factors influencing tourist loyalty are still inconclusive, and we don’t yet know the strength and magnitude of relationships between different factors and loyalty –7. UsingThe results of the same factors that affect loyalty are inconsistent, reflected in the indirect or even negative influence on loyalty –14. For Among the many studies on tourist loyalty, the influencing factors included in each study and their degree of influence are different. We cannot identify the most decisive factor affecting loyalty, which presents difficulties for policymakers. In loyalty research, satisfaction has been repeatedly included in the study and is considered the main influencing factor –18. But With most studies widely confined to two or three factors, we cannot figure out the strength of different influencing factors on loyalty. For example, Kohsuke M et al.  studied Thus, to clarify the relationship between satisfaction, motivation, perceived value, perceived quality, experience quality, and tourist loyalty, and to probe into the intensity order of the relationship between these five factors and loyalty. First, we systematically reviewed the factors and the concept of loyalty and proposed the possible relationship between different factors and loyalty. Then, we established the theoretical framework of five-factor tourist loyalty. The relevant data were then extracted and a meta-analysis was conducted. We extracted and summarized 242 empirical and conceptual studies and evaluated the average effect value between tourist loyalty and its influencing factors. The most significant advantage of meta-analysis is that quantitative methods provide a high degree of objectivity when studies on the same subject reach different or even contradictory conclusions. Consequently, the meta-analysis is useful in eliminating the source of errors in the course of the analysis, and in determining the relationship and intensity of variables . Lastly,The concept of loyalty has had a longer involvement in the marketing literature, dating back 50 years. However, its use in tourism research has only become popular and developed in the last 20 years , 30. LoyIn the tourism industry, most of the literature defines loyalty as the willingness of the tourist to return and recommend to others –47. The Attitudinal loyalty refers to the positive intention of tourists to visit a specific destination again or visit similar destinations, and tourists recommend destinations to those seeking travel advice through word of mouth. Compared to behavioral loyalty, attitudinal loyalty is focuses on the psychological and emotional state of customers, and their intention to repurchase and recommend a particular product or service . AlthougAlthough tourist loyalty has been studied extensively, there are still unsolved questions about how to keep visitors loyal to one destination . For thiMotivation is one of the basic concepts of human behavior . In tourStudies on tourism motivation posit that understanding tourist motivation is the foundation of tourism development. Motivation, especially push-pull motivation, has been identified as an antecedent of tourist loyalty , 76. MaiBy exploring the relationship between motivation factors and loyalty, this paper posits the following assumptions:H1. Motivation has a positive effect on tourist loyalty.H2. Motivation has a positive effect on revisit intention.H3. Motivation has a positive effect on recommend intention.H4. Pull motivation has a positive effect on tourist loyalty.H5. Pull motivation has a positive effect on revisit intention.H6. Pull motivation has a positive effect on recommend intention.H7. Push motivation has a positive effect on tourist loyalty.H8. Push motivation has a positive effect on revisit intention.H9. Push motivation has a positive effect on recommend intention.One of the main objectives of destination managers is to maximize visitor satisfaction, due to the strong relationship between customer satisfaction and loyalty . As one However, there are some controversial points that satisfaction only affects one aspect of loyalty. For example, empirical research by Brigne E et al.  showed tH10. Satisfaction has a positive effect on tourist loyalty.H11. Satisfaction has a positive effect on revisit intention.H12. Satisfaction has a positive effect on recommend intention.Perceived value is the trade-off between tourists perceived benefits and costs  of the tourism products or services provided . The conPerceived value has always been a necessary prerequisite for the investigation of tourism behavior, especially in the prediction of tourist intentions , 86. ManH13. Perceived value has a positive effect on tourist loyalty.H14. Perceived value has a positive effect on revisit intention.H15. Perceived value has a positive effect on recommend intention.Research on the perceived quality of tourism destinations draws on service quality theories from Parasuraman A P et al.  and GrönAccordingly, to explore the relationship between perceived quality and loyalty, we propose the following assumptions:H16. Perceived quality has a positive effect on tourist loyalty.H17. Perceived quality has a positive effect on revisit intention.H18. Perceived quality has a positive effect on recommend intention.The quality of the experience refers to the final product of tourist feelings and perceptions of a destination . It is aH19. Experience quality has a positive effect on tourist loyalty.H20. Experience quality has a positive effect on revisit intention.H21. Experience quality has a positive effect on recommend intention.Based on the above 21 research hypotheses, we propose the basic theoretical framework illustrated in This research used meta-analysis, a quantitative literature analysis method, to discuss the composition and functions of factors affecting tourist loyalty . The greSeveral identical empirical articles exist in the field of tourism. Through the quantitative integration of meta-analysis, these empirical results can increase the sample size and improve the effect of statistical test, which is conducive to drawing more general, comprehensive, and clear conclusions . We adopThe protocol for the study was registered with INPLASY in March 2022 and its unique identification number is INPLASY202230114 . The Prewww.webofknowledge.com), Wiley Online (www.wiley.com), EBSCO (www.ebsco.com), SAGE (www.sagepub.com), Taylor and Francis (www.tandfonline.com), and Elsevier (www.elsevier.com). Studies written in Chinese were retrieved from CNKI.com (www.cnki.net) and other major scholarly databases. We used the following keywords for retrieval: loyalty, behavioral intention, recommendation intention, word-of-mouth, revisit intentions, intention to revisit, willingness to recommend, and similar related terms. Conceptual and empirical studies published between January 1989 and September 2021 were extracted. To expand the sample size, we conducted a secondary manual search to find relevant missing literature from the references of articles extracted from the databases. By soliciting assistance from tourism scholars, we obtained some unpublished research literature on tourist loyalty. This helps to reduce the impact of publication bias and to include as many relevant studies as possible.The samples included articles from major academic databases, including Web of Science  between independent variables and loyalty.Eliminate the literature with an ill-defined concept of research variables.Avoid duplication by classifying an article as the same study if it was published in multiple stages, repeated, or had the same sample.Exclude articles with less than three effect values of all components of the two variables.Based on the above criteria, 242 articles were extracted from 215 English papers and 27 Chinese papers , which pAfter obtaining the sample literature, the empirical studies were coded independently by the two authors of this article to ensure the reliability and independence of the data. Coded data include qualitative information and quantitative information. Qualitative information includes research description items, such as author , title, publication year, publication journal, research method, method, research object, etc.; and quantitative information includes sample size, variable reliability, effect value, etc. To ensure the accuracy of the coding content, the included documents are coded independently and compared. After the coding, the two authors of this article made a comparison and found the reasons for the inconsistencies in the coding information. After discussion, the authors modified and processed the data to resolve the discrepancies.The detailed steps are as follows: ① Identify statistics that can characterize the relationship between variables, such as the correlation coefficient, the regression coefficient, and the path coefficient. ② Convert all single statistics into a unified effect size (ES)—correlation coefficient r.③ Calculate the total effect size, which refers to the statistics that reflect the relationship between variables and do not depend on a single study. Data processing is mainly implemented by The Comprehensive Meta-analysis software (CMA 3.0).rx is the effect size of the xth study, zx represents the Fisher’s-Z score of the xth study.nx = sample number, k = number of effect size.First, we extracted the direction and sample size of the correlation coefficient reported in the literature. If the literature did not report the correlation coefficient, we converted the t-value, the P-value, the β value, and other convertible indicators into the correlation coefficient . Second,To test whether there was publication bias, we used Fail-Safe-Number (FSN) to verify the stability of the results . The los2. The Q-value obeys the chi-square distribution with the degree of freedom of M-1, where M is the number of effect sizes ),Theoretically, the study developed and tested a more comprehensive model of tourist loyalty than any previous study. The results extend our current understanding of the development of loyalty and strongly suggest that future loyalty models should include multiple factors. By incorporating more sample data from prior studies, the study proposed the five-factors theoretical framework on tourist loyalty based on previous studies. The five influencing factors are integrated into one single literature, which is impossible to achieve in prior studies. This study solved the convergence and divergence in previous studies and produced a more generalizable finding. Although previous tests of these 21 hypotheses produced inconclusive conclusions and different magnitudes, meta-analysis results allow us to draw more definitive conclusions about the relationships. With the advantage of meta-analysis and more empirical studies, this study confirmed that the five dependent variables had significantly positive relationship with loyalty and their sub-dimensions. In the descending order of effects, the five factors are degree of satisfaction, quality of experience, perceived value, perceived quality, and motivation.Practically, the results of this study are of great significance for the management of destination markets and loyalty. Managers should improve tourist loyalty in many aspects. First of all, the construct most studied by researchers is satisfaction. In the meantime, the results of meta-analysis showed that satisfaction was the best predictor of loyalty, which is consistent with most studies e.g. , 116). T. T116]).Second, experience quality is second only to satisfaction in all factors, highlighting its importance in tourist loyalty and sub-dimensions. Research on the quality of experience is an excellent response to Hanafiah et al. . The finThird, the inclusion of perceived value in the meta-analysis is a good response to the constructive argument of previous scholars , 118, whFourth, this study found that motivation and loyalty and its sub-dimensions are positively related. Our study shows that among the five factors that affect loyalty, motivation has the least impact, reaching a weak to moderate relationship. The results also show that motivation has a weaker association with loyalty than that of service quality, which is consistent with the findings of Paudyal et al. . From thThere are some limitations to this study. Using only English and Chinese sample literature may result in insufficient sample comprehensiveness and representativeness. Nevertheless, the limitations of data extraction and other conditions had been calculated in the study and FSN showed a high level of validity. Furthermore, due to the problem of error analysis and selective reporting in some samples, we were unable to fully use the original data, which reduced the sample size of this research and the accuracy of the results. Therefore, the sample base needs to be further improved in subsequent studies. The heterogeneity test supported the existence of moderating factors that determined the magnitude of effect sizes. Since this study did not analyze the mediating and moderating roles of variables, future studies can explore how the size of the would be influenced by including these variables.Although the existing literature mainly discussed the influence of only five factors on tourist loyalty, in reality, all factors belong to the tourist perception of the destination. Accordingly, there must be mutual relations and effects among all factors. Therefore, subsequent research should explore the influence of interaction and transformation among all influencing factors on tourist loyalty.S1 Protocol(PDF)Click here for additional data file.S1 Checklist(DOCX)Click here for additional data file.S1 Data(XLSX)Click here for additional data file.


The purpose of this paper is to outline the protocol for the research program “UserInvolve,” with the aim of developing sustainable, service user involvement practices in mental health services in Sweden.This protocol outlines the knowledge gap and aim of the UserInvolve-program. It further provides an overview of the research infrastructure, with specific focus on the organization and management of the program as well as the design of the six underlying research projects. These six research projects form the core of the UserInvolve-program and will be carried out during a six-year period (2022–2027). The projects are focused on examining articulations of experiential knowledge in user collectives, on four specific user involvement interventions  and on developing theory and method on co-production in mental health research and practice.The knowledge gained through the co-production approach will be disseminated throughout the program years, targeting service users, welfare actors and the research community. Based on these research activities, our impact goals relate to strengthening the legitimacy of and methods for co-production in the mental health research and practice field. In SwedClose collaborations between public sector and SU movement actors distinguishes the Swedish context for SU involvement . In the To address the knowledge gaps about strategies for involving SUs and co-producing in mental health research and practice, a research program is proposed with the aim of developing co-produced, cross-disciplinary, and practice-relevant research on SU involvement. This is a 6-year program, funded in May 2021 by FORTE  [grant number: 2021–01427], with the explicit ambition to promote a transformative change of mental health services through a co-production approach . This prThis paper provides an outline for a 6-year research program focused on the issue of SU involvement in mental health services in Sweden. Special attention is directed to community mental health services provided by the local municipalities. The program aims to examine strategies for SU involvement at the individual-, organizational- and system level, and develop sustainable SU involvement through co-production processes involving the research community, welfare actors and SU organizations. The specific aims are to:Explore how experiential knowledge can be articulated, and how it can be legitimized within mental health practices.Examine barriers and success factors in the implementation of SU involvement methods in mental health practice.Examine and develop co-production methods for integrating SU involvement in research and mental health practice.2.The theoretical starting point of this research program relates to co-production, knowledge generation and more specifically, the power relations in the creation of knowledge and in SU involvement practices in general. Communities of Practice, CoP, is a social theory based on collective processes , 28. Kno3.The overarching aim and research activities have been planned for in collaboration with both SU organizations and service providers. SU involvement does not only constitute the study object of this program, but methods of co-production are also at the core of the program design. In the development of the program’s underlying research projects, co-production strategies have been planned for at all phases, involving representatives of SU organizations and mental health services at both local and national levels.3.1.The setting for the research program is a co-production platform between researchers, welfare actors and The Swedish Partnership for Mental Health (NSPH). The program design, its aim, and the research questions have been developed in close collaboration with NSPH. The national NSPH is an umbrella organization consisting of 13 different mental health organizations whose membership base primarily includes both service users and family members. Its main focus is to drive change on the political level and to develop methods, such as peer support and user focused monitoring, for improving SU involvement in decision making on all levels. In addition to the overall collaboration with the national NSPH, collaborations with the regional NSPH-organizations and other local SU groups are established in the different research projects. This cooperation with the national, regional, and local SU organizations is an important resource for the co-production processes of the program.The researchers in the program represent five universities in Sweden and are also all affiliated with the Centre for Evidence-based Psychosocial Interventions (CEPI), a national multi-disciplinary research centre with the explicit ambition of integrating SU perspectives in research. CEPI, established in 2007, constitutes an arena for collaboration spanning several national research environments. In addition to national and international research collaborations, CEPI has established close collaborations with national stakeholders, public service providers and SU organizations. CEPI is specialized in research on psychosocial interventions for people with psychiatric disabilities in a community mental health context and has a strong focus on recovery perspectives and SU involvement. The centre has further been leading in developing SU involvement in research, for instance through organizing SU panels for research feedback and evaluation, and by integrating a centre-based SU council. In addition to conducting research, CEPI arranges education and training activities and offers implementation support to welfare actors in the mental health sector. The program will employ CEPI as an infrastructure to build new expertise and strengthened research environments on co-production and SU involvement.3.2.The UserInvolve program has established a program infrastructure for the organization and management of the program , consistAn innovative feature of the program management is the integration of a coordinator function for SU involvement. These two coordinators are affiliated with NSPH and are members of the program group. Their main responsibilities are to ensure that the right competence is present in the projects and to make both internal and external communication readily accessible. It is likely that the coordinator role will evolve to reflect the changing needs and demands of the program.3.3.The program contains six underlying projects . Project3.3.1.Swedish government official reports (GoR)  address Project 1 is focused on mapping and analyzing activities in autonomous  SU arenas, both online on social media platforms and in physical organizations. Based on both these forms of mobilizations, we will attend to how SUs collectively share experiences and, by way of that, construct a shared knowledge base and political standpoints in the realm of accessible welfare provisions. N/ethnographic methodology will be used, in three different sites, including the collection of observational fieldnotes combined with focus groups and qualitative interviews.3.3.2.SDM is an evidence-based method recommended in national guidelines since 2012 but has still not been implemented to any greater extent. SDM is a collaborative process, that creates conditions for SUs and professionals to jointly formulate goals and make informed decisions that consider the knowledge of the SU, the provider and available scientific knowledge . SDM hasThe project is designed as a multi-centre study with a participatory approach  explorin3.3.3.Peer support is a service provided by a trained individual with lived experience of mental illness and offers support and hope during treatment to aid in the long-term recovery of mental health SUs . A largeThe project consists of three parts. The first will explore stakeholders’ experiences of the current state of peer support by conducting a national survey and interviews. Secondly, a compilation of the experiential knowledge will be used to subsequently inform, develop, and iterate a scaled-up peer support intervention with inspiration from the international intervention , 49 in c3.3.4.UFM is a systematic and independent method of reviewing care and support services, with the entire process performed by people with lived experience of mental illness , 51. TheThe project consists of three parts. The first is focused on mapping and analysing Swedish UFM reports, compiled in a database by NSPH. The second part is a multi-case process study of five UFM processes. The third part is a follow-up study focused on the outcomes and significance of UFM.3.3.5.Since the 1990s, Swedish mental health SU organizations have participated in government projects and commissions of inquiry  These orThe project consists of two parts. The first is focused on systemic SU involvement at the national level, through mapping and analysing policy documents, as well as national actor’s initiatives regarding SU involvement. The second part of the project takes an explorative approach to map and analyse activities at a systemic local level.3.3.6.This project is divided into three parts and takes an overarching approach to the knowledge generated in the program, in order to develop theory and methodology in relation to SU involvement in research and in mental health practices. The first part is focused on the development and validation of questionnaires and interview guides that will be applied to develop and evaluate the co-production activities within the program. Part two focuses on empirically investigating and developing co-production methods in mental health research. In this part, our focus will be to explore strategies for integrating co-production in mental health research, examining both barriers and success factors. In the third part we aim to draw on insights generated in the program at-large in order to develop theory that can be applicable to a range of user involvement practices.3.4.Our co-production ambition is reflected at both the project and program level. Program level co-production arenas, such as the strategy group, the user research coordinators and Forum UserInvolve are described in 3.5.The UserInvolve program formally started in January 2022 through the formation of a program group, the development of the program’s infrastructure and an action plan for running the program until the end of 2027. The program is proceeding according to the time plan presented in 3.6.A number of activities will be focused on establishing a communication and dissemination platform for the research program. The co-produced ambition of the program means that we have the goal of creating an ongoing dialogue, where research results are continuously reported, feedback is received, and new research ideas are discussed. The communication and dissemination will be tailored to the needs and preferences of different audiences to maximizes the impact of the research. Thus, we will use varied dissemination activities, co-produced with our stakeholders in the program and projects, for potentially achieving greater impact than exclusively researcher-driven dissemination activities.The main arena for communication and dissemination will be the digital seminar format “Forum UserInvolve” see . This seThe involved researchers will present at scientific peer-reviewed conferences  as well as public conferences . To enable a broader public reach, social media activities will also be integrated in the program and its strategies for knowledge dissemination. A UserInvolve Facebook page will serve as a forum for disseminating the research results to a broader public, but also as a communication platform for the network on SU involvement issues. A podcast produced by NSPH will serve as an additional forum for broader dissemination of the research results.Collectively, these strategies aim to contribute to the formation of a network on SU involvement issues, assembled in collaboration with SU organizations and mental health representatives, thereby strengthening translation of the knowledge generated in the program into sustainable practice.4.At a general level, the UserInvolve-program can contribute to the quality development of mental health services by examining how the SUs opinions, values and knowledge can be integrated into practice. Motives for increased SU involvement are related to democratic participation, redistribution of power, and to service system adjustments . SU invoUserInvolve is focused on development, change and implementation work. It supports the implementation of four specific interventions for SU involvement and through these examines how SU involvement can be integrated in mental health practice. Our ambition is thereby to generate knowledge of more general components that are relevant to a range of involvement strategies. The involvement interventions examined may be relevant to further employ in both health- and social care services. The program will contribute with knowledge of how user involvement is enacted in these organizational contexts, but also of specific challenges associated with co-production in these sectors. A multitude of SU involvement initiatives are currently emerging in Sweden, but many of these assume the form of short-term projects. It is therefore relevant to examine and develop sustainable implementation. We further regard the co-production processes in the program as crucial for accomplishing such transformative changes in mental health practice. In addition, the program makes a contribution by studying the effects of SU involvement interventions. Evidence of such effects have been called for in prior studies, and the UserInvolve-program examines individual level outcomes as well as outcomes at an organizational level.The co-production structures of the program are built on ambitions to generate an evolved understanding of how SU involvement can be enacted and what significance it has for individual SUs and for SUs as a group in society. SU movement and welfare actors participating in UserInvolve have a key role in this, by contributing with their knowledge perspectives to the exploration of interventions, and of barriers and facilitating factors to sustainable implementation. Through our co-production approach, we further have the ambition to develop networks for collaborative knowledge processes or communities of practice, that extend beyond the program years. The program will further contribute to co-production theory and methodology development, by developing instruments to follow up co-production processes and impact and by discerning more general principles regarding the sustainable integration of SU involvement in mental health systems.In conclusion, the program has both conceptual and instrumental benefits for SU involvement policy and practice. Conceptually, the program has the ambition of contributing to general awareness, literacy and readiness for SU involvement among providers and enabling SUs to be empowered and supported as they attempt to participate in their care. The contributions of the program that regard co-production theory and methodology can further be relevant to the research community, but also to other actors that have ambitions to integrate a co-production approach. Instrumentally, the program intends to develop SU involvement strategies and improve implementation processes. This would contribute to an extended and refined repertoire of strategies for mental health research and practice. The co-production structures of the program further involve central government authorities and may thereby contribute to a policy-level impact based on their experiences and the knowledge generated in the program.The ethical considerations in these six underlying projects are based on their specific methodological procedures. These projects have, or will, all undergo individual ethical review.UM: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. HN: Writing – review & editing, Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Writing – original draft. U-KS: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. DR: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. UB: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. AG: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Resources, Writing – review & editing. MJ: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Resources, Writing – review & editing. EA: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. KG: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Writing – original draft, Writing – review & editing. PE: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Writing – original draft, Writing – review & editing. FN: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Writing – original draft, Writing – review & editing. SL: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Writing – original draft, Writing – review & editing. PS: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Visualization, Writing – original draft, Writing – review & editing.


In recent years, there has been a global advancement in the offering of canine-assisted interventions (CAI) in prisons. However, these programs have focused primarily on the benefits to the dogs involved and not on the impact on the participants. The authors of this perspective study have been running a CAI program with therapy dogs, called PAWSitive Support, in a Canadian federal prison since 2016. Thoughts from the program facilitators and interviews with prison staff indicate that the program, and specifically the therapy dogs, provides a unique and integrated source of comfort, support, and love for participants. These benefits are consistent with those seen in CAI programs outside of prisons. Unique to the prison setting appears to be an improvement in participant–staff relations. The therapy dogs have helped participants to experience comfort and consequently express their emotions. This seems to contribute to their recognition of support within the prison system and specifically developing trust with staff. Additionally, the dogs have helped to create an experience of the feeling of love within the prison, interpreted as the feeling of being cared for, which is rare for this population. The authors suggest that the integration of a therapy dog intervention in prison could be a novel harm reduction strategy to address issues related to prisonization and associated mental health concerns, including substance use. This consideration can offer unique insight into the field of forensic psychiatry about providing compassionate care to patients. During a presentation on canine-assisted interventions (CAIs) for the International Forensic Psychiatry Lecture Series in 2022 , Dr. DelThere is evidence to suggest that interacting with animals can have a positive impact on human health –9. This Therapy dogs involved in CAIs are friendly companion animals who, alongside a human handler, have passed a test enabling them to visit with individuals who do not have regular access to a dog and can benefit from their comfort, support, and love –19. WhilProgram evaluation research has demonstrated that CAIs implemented in correctional facilities can have a positive effect on both the institutional atmosphere and participant behavior , 26–30. In the realm of substance use research, companion animals and CAIs have been found to increase clients' experiences of connection and care with another being. These feelings of connection, support, love, and comfort can be seen to influence addiction and mental health outcomes including benefiting positive thinking patterns, improved emotional states, and positive attachments and style of attachment , 39–41. Historically, prisons in Canada, as in many other places, have been designed with a focus on punishment rather than rehabilitation , 45. ThiA CAI that involves therapy dogs has the potential to provide prisoners with a distinctive form of comfort, support, and love. Although there is no standard definition of compassionate care at present in healthcare, there seem to be important elements including an aim to reduce the pain, suffering, and harm experienced by a patient and treating patients with compassion, kindness, respect, and dignity –54. ThesPAWSitive Support is a canine-assisted learning program that aims to improve the wellbeing of participants through human–canine interaction. It was first delivered at the request of the Warden of Drumheller Institution in response to an increasing number of illicit drug overdoses. The program was developed by Dr. Colleen Dell from the University of Saskatchewan and Dr. Darlene Chalmers from the University of Regina and is grounded in an evidence-based experiential learning approach . The proThe program is offered over 4 intensive days and includes a follow-up session 6 months later. The 4-day program was designed around space availability. The follow-up sessions were less structured in nature than the initial program and allowed the program participants to reconnect with the program dogs, facilitators, and each other. After completing the program, the participants were invited to return and act as peer mentors. In this role, the peer mentors guided new program participants through the process, offered their insights or ideas on interacting with the program dogs, and supported the current participants in reflecting on the impact of the program on their experiences within prison. The first group of participants consisted of five individuals with a history of substance use, most of whom were serving life sentences. All participants liked dogs, were assessed as non-violent and non-aggressive at the time of the intervention, and had no history of animal abuse. The dogs involved in the program—Kisbey, a loving mature boxer, Subie, an emotionally attuned boxer, and Anna-Belle, a stubborn but affectionate English bulldog—were selected for their friendly personalities and ability to provide comfort, support, and love to those they interact with. They also worked as therapy dogs in their local community. The dogs were assessed for their health, they had basic obedience skills, and their welfare was paramount throughout the program's offering. Two program facilitators, Dr. Colleen Dell and Dr. Darlene Chalmers, led the program activities.The two program facilitators conducted eight interviews and a focus group with five prison staff members  who were associated with the case files of the first five PAWSitive Support participants. The eight individual interviews were conducted between one staff member and one facilitator, with the focus group having both facilitators present. These took place immediately after the program began and then approximately every 6 months over a 2-year period after the program ended to determine program impacts. The interviews and focus groups were semi-structured to allow for insights from the staff members to be collected beyond the experiences or knowledge of the facilitators and varied in length from 24 min to 89 min. The transcripts of these interviews were analyzed using flexible inductive coding methods to identify the program's overall impacts, following the methods outlined by Deterding and Waters . Ethics Three main themes emerged from the analysis, which were described as prison-specific therapy dog-facilitated experiences of comfort, support, and love. This was not surprising, given that these are common therapy dog program outcomes outside of a prison context. Unique here is that these are staff-identified insights, whereas the majority of the extant literature focuses on participant experiences. Thus, focusing our perspective article on how these three concepts are perceived uniquely within a prison-specific context by staff can be a potentially important contribution to forensic psychiatry and allied professions.According to the staff, the program has a temporary impact on the prison environment for the participants. They explained that the prison environment is typically characterized by hypervigilance, fear, and projected strength as participants feel the need to hide any signs of vulnerability to stay safe and maintain status. However, the presence of the program dogs helped ease these tensions and alter the prison environment. Staff observed that participants were able to “let their guard down” and “show vulnerability” without fear of judgment. One staff member shared the following:Staff 2: So this morning when we were sitting outside, Participant 1 and Participant 2 were sitting chatting and Participant 1 said, and it was just something that came out, and he said “I don't even feel like I am in prison right now…I feel like I am cured.”Another staff member similarly shared the following:Staff 1: Same with Participant 5. He's like, he's got gang affiliations, he's quite high ranking in his gang, so for him to let that down [his guard] a little bit, is big.Additionally, the staff observed that the sense of comfort facilitated by the non-judgmental comforting presence of the dogs allowed participants to experience and express emotions that they would not normally, and for some, had not done in a very long time.Staff 3: So that's some empathy, which again would be huge for Participant 4. Cause that's a big issue for him. Cause he can't show it, you know, you get so used to being that, you know look at me, the big tough guy… he probably, probably a place he hasn't been since I would say he was a child.Participating in the program offered participants a sense of comfort to explore their emotions and feelings with reduced fear of judgment. The culture and norms within the prison, alternatively, can create a disconnection between a participant and their emotions. This is especially important to substance-using individuals as disconnection from emotions and difficulty regulating them can lead to increased substance use , 60. In In forensic psychiatry, comfort is a key element to build rapport and trust with patients and is crucial for effective treatment –64. Animhttps://colleendell.com/top10.Our team has developed a Top 10 Resource for service providers, including in forensic psychiatry, to access evidence-informed outcomes that recognize the beneficial role of animals in the lives of individuals facing substance use concerns. Recognition of the positive benefits of human–animal interaction by service providers is a key, although under-recognized, way to build comfort with program clients. The resource is available at The staff members observed that having PAWSitive Support therapy dogs in the prison contributed to creating a supportive environment, which resulted in stronger relationships between the participants and prison staff. One staff shared (Staff 1) “More than anything probably [the program] is what motivated him but like, I think this helped him, like I said with his anxiety and with probably staying sober… and definitely helped with his relationship with staff.”The staff members noted that the presence of therapy dogs during the program increased the participants' recognition of being supported within the institution, which in turn facilitated the development of trusting relationships between the participants and the staff.Staff 1: He seemed to have more trust with me. Like cause when he came to me about getting out of the gang, it was like he came to the door… then he closed the door and “what do you think about maybe a transfer, so I can maybe distance myself, you know?”Staff also thought that the program participants' trust with them was strengthened when they saw that prison staff supported the program, such as when one of the therapy dogs was permitted to visit a participant in the segregation unit.Facilitator 1: Instead of fighting the institution… [the participants began] seeing it as something that was supporting them, and even… taking [therapy dog] to see [Participant 4] in] Seg yesterday,Staff 1: You took [therapy dog]? Oh my gosh, that must have been so big for [Participant 4] …. I can't believe you got in there, that is amazing.The therapy dogs provided a positive and supportive alternative to the traditional prison environment. This had a positive impact on the participants' relationships with the institutional staff. Research has shown that supportive relationships and strong therapeutic alliances can reduce the risk of substance use for patients in recovery , 69. Suphttps://colleendell.com/pawsitivecanine.Our team has created a resource in collaboration with the Canadian Centre on Substance Use and Addiction to help professionals incorporate animals into their practice to create a more supportive and client-centered environment. This resource includes common definitions in the field, a video series, and a resource guide on how to become animal-informed as a practitioner. This resource also incorporates the voices of people with lived experience of substance use whose animals supported them in their recovery, reflecting the harm reduction tenant of respecting the expertise of people with lived and living experience . You canThe staff reflected that participants in the PAWSitive Support program expressed gratitude for being a part of it and had positive feelings toward the prison for allowing them to take part in it. They felt that the institution cared for them and that they mattered. This is not a common sentiment within the broader culture of prisons.Facilitator 1: One thing that came out really strong from everybody… was that they saw [Correctional Service Canada] as supporting and caring about them. Which was surprising to me, is not how I was really thinking about it but to allow the program into here, and to allow them to participate in it, they saw as, maybe CSC really does care.Additionally, the staff observed that the program participants' interactions with the therapy dogs were non-judgmental, trusting, and loving. It provided them with an opportunity to see themselves beyond the externally and internally imposed labels of “criminal,” “lifer,” or “gang member.”Staff 4: [Participant 4] was very surprised, somebody would do that for him, because he's a bad guy, he's in prison, “well those dogs don't know that I've done bad things, and they still love me,” they're jumping on them, they're licking them, they're hugging them, and to have that love with no strings attached, because they can't do anything for you, they can't do anything for the dogs, and they know that, but still the dogs love them. And you guys took time out of your world and your life, but there's a lot of other things you could be doing, and they see that too, like “wow, they actually came in here, and wanna spend this time with us, wow, maybe I'm not so bad. Maybe I am okay.”Evidence-based literature tends to avoid discussing the concept of love. However, within the context of the PAWSitive Support program, institutional staff identified love as the participants' feeling of being cared for and valued. The concept of “mattering” is explored more commonly in literature than love. Research indicates that not feeling like one matters can contribute to increased criminal activity, mental health issues such as depression and suicidal tendencies, and a greater likelihood of substance use –79. Not www.colleendell.com), where access to our front-line experiences, research outcomes, and participant testimonials are regularly shared. This includes access to the Animal Memories magazine our office co-led in the development of, and which the Commissioner of Correctional Service Canada identified as “a remarkable collection of inmate stories about the importance of animals in their lives”  and the Internet (r lives” . You canOffering compassionate care in a prison setting is challenging, and establishing trust between participants and staff and within an institutional environment can seem impossible. One possible way to enhance the quality of care in forensic psychiatry is to take into account the insights of the Drumheller Institution staff and thoughts from PAWSitive Support facilitators who work with participants in this prison canine-assisted intervention program. According to the staff, the program's therapy dogs create a distinct and integrated sense of comfort, support, and love for participants—all elements of compassionate care. This, in turn, helps to establish some trust between participants and staff as well as the institution. Improving staff-participant relations is beneficial to creating a therapeutic alliance that supports participants' overall wellbeing and addresses the negative impacts of prisonization, such as substance use. By integrating this knowledge about human–animal interaction and insights into compassionate care into its practice, forensic psychiatry may be able to create a more positive and supportive environment for its patients, leading to better therapeutic outcomes, harm reduction for prisonization, and an improved standard of care.It must also be noted that ensuring the welfare of the participating program dogs is crucial to a program's success. This is central to the offering of the PAWSitive Support program. Future research should consider collecting data over a longer term and include pre- and post-intervention measures, including mental health and substance use-specific outcome indicators. At the same time, moderating factors on outcomes need to be fully accounted for, such as participants' mental health symptoms and length of incarceration.The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.The perspective study involving human participants was reviewed and given ethics exemption by the University of Saskatchewan Human Research Ethics Board given its evaluation focus. The patients/participants provided their written informed consent to participate in this study. The animal study was reviewed and approved by University of Saskatchewan Animal Research Ethics Board. Written informed consent was obtained from the owners for the participation of their animals in this study.MG wrote the original draft of this manuscript. MG and CD contributed to the conceptualization and editing of the manuscript. CD, DC, GR, and MM all contributed sections of the manuscript. All authors contributed to manuscript revisions and approved the submitted version.


Nutrition  and evaluation of the Program  were the main themes analyzed. Regarding the participant/object analyzed in each study, users  were the main actors investigated. Future studies should consider investigating the effectiveness of the actions developed in the program centers, especially physical activity and healthy eating practices.The aim of this scoping review was to map the literature related to the “Academia da Saúde” Program, including the objective and rationale of the studies, activities carried out in the program’s centers, as well as the actors involved in these actions. The search for evidence was conducted in the MEDLINE, LILACS, Web of Science, Scopus, COCHRANE, and SciELO databases. Additional evidence was investigated in the Catalog of Theses and Dissertations of the Coordination of Improvement of Higher Education Personnel (CAPES-Brazil) and in the Brazilian Digital Library of Theses and Dissertations, in addition to manual searches in the references of the studies/documents. Out of 642 initial records, the information synthesis was composed of 74 studies/documents ( Furthermore, sources of evidence published in Portuguese, English, and Spanish were considered. Additionally, there were no restrictions on the years of data collection and publication, topics addressed, or strategies adopted in data collection and analysis. Evidence derived from review studies, monographs, thesis, dissertations, non-scientific/non-technical texts from the internet, editorials, essays, or studies not available for full access in the investigated data sources were excluded.2.3.Two authors independently examined each database to identify potential studies/documents. After extracting the studies/documents from the databases, du-plicate information was excluded, and then studies/documents that did not meet the inclusion criteria previously reported were excluded after reading the titles and abstracts. Subsequently, these selected studies/documents were read in full to select the texts that would be included. Disagreements between the two reviewers were resolved through a consensus meeting. A third reviewer was consulted when disagreements/doubts regarding the studies/documents were not resolved by the reviewers. The EndNote® soft-ware version X6– was used to manage the studies/documents verified, as the functions provided by the software allow the identification and exclusion of duplicate studies, and the organization of information from each database.2.4.The search for available evidence related to the theme was conducted during February 2022 in the MEDLINE, LILACS, Web of Science, Scopus, COCHRANE, and SciELO databases, based on the search strategy applied in MEDLINE: [Health gym(Text Word)] OR [Health academy(Text Word)] OR [Academia da Saúde Program(Text Word)] OR [Health Academy Program(Text Word)] AND [Brazil(Text Word)]. Additional evidence was investigated in the Catalog of Theses and Dissertations of the Coordination for the Improvement of Higher Education Personnel (CAPES)2.5.Two authors conducted the extraction and synthesis of information derived from the included studies/documents, using a standardized data spreadsheet. The extracted descriptive information included characteristics of the studies/documents . Given that the present scope review aimed to map the literature on the Program regardless of quality, the risk of bias was not assessed .The descriptive synthesis of the studies/documents included in this study followed the guidelines suggested by JBI , which eAfter extracting the descriptive information from the included studies/documents and considering the diversity of investigated topics, it was decided to synthesize such information based on how the object of investigation of each analyzed study was related to the “Academia da Saúde” Program. In this context, seven major categories were defined: (i) information from participants was used in the investigation of different objectives and related aspects; (ii) methodological description, evaluation, and/or results of intervention programs conducted with users; (iii) spatial distribution of program units and their relationship with the food environment; (iv) quantitative and/or qualitative synthesis of actions performed, health indicators, and socioeconomic descriptors related to the program; (v) understanding and description of meanings, knowledge involved, and foundations required for the conception and/or execution of activities carried out in the program; (vi) understanding of users’ perspectives on the program and activities developed; (vii) evaluability and/or implementation of the program.In addition to the seven major categories adopted to synthesize the diversity of investigated topics, we decided to synthesize the information considering the topics of the studies/documents according to the analyzed objects. The topics from which the information was extracted are as follows: (1) nutrition; (2) nutrition and physical activity; (3) nutrition, physical activity, and anthropometric variables; (4) nutrition and anthropometric profile; (5) physical activity; (6) program evaluation; (7) local-level instrument evaluation ; (8) local-level program description and evaluation; (9) methodological study; (10) healthy lifestyle; (11) intervention evaluation conducted within the program context; (12) reduction of healthcare expenditures; (13) health education. Regarding the object of investigation in each analyzed study/document, eight distinct categories were identified: (a) users; (b) managers; (c) Physical Education professionals; (d) territory; (e) Health Academy Program; (f) users and territory; (g) managers and professionals; (h) managers, professionals, and users.3.n = 214), screening title and abstract (n = 232), and analyzing full-text articles (n = 66), 54 studies were included for information synthesis. In addition to the database search, the Catalog of Theses and Dissertations, and the BDTD were investigated to identify theses and dissertations related to the topic of interest in this review. Out of the initially identified 76 records, 27 documents were excluded for not meeting the inclusion criteria, and 29 were excluded as the analyzed information was subsequently published as a scientific article, resulting in a total of 20 documents. Therefore, a total of 74 studies/documents were included considering the different sources of information , with a growing number of publications from the biennium 2016–2017 , and involving up to 99 participants . Regarding the nature, design, and analysis of the information derived from the studies/documents, 73.0% (n = 54) were scientific articles, 64.9% were analyzed in a cross-sectional design, and 60.8% (n = 45) were conducted in a quantitative way, respectively.The descriptive characteristics of the included studies/documents are described in n = 02; 2.7%) and evaluating and/or implementing the program  were the basis for investigations conducted in the remaining studies/documents analyzed.Regarding the purpose of the included studies/documents, seven domains were identified . Approxin = 24; 32.2%) and Evaluation of the Program  as their main themes. Other investigated topics were related to Nutrition and Physical Activity , Nutrition, Physical Activity and Anthropometric variables , Nutrition and Anthropometric Profile , and Physical Activity . Other topics were also investigated, including Instrument evaluation at local level , Health Education , Description and Evaluation of the Program at the local level , Methodological study , Healthy lifestyle , Evaluation of a Program conducted within the context of the “Academia da Saúde” Program , and Reduction of health expenses ; (ii) managers ; (iii) Physical Education professionals ; (iv) territory ; (v) “Academia da Saúde” Program ; (vi) users and territory ; (vii) managers and professionals ; (viii) managers, professionals, and users  users  can be found in 4.As previously evidenced by studies with similar objectives and design to the present scoping review , 90, NutThe evidence synthesis revealed a high number of studies that aimed to evaluate the “Academia da Saúde” Program. It is important to highlight that this type of study allows the investigated information to be used as a management tool, contributing to decision-making regarding the actions carried out by the Program . In addi7, the synthesis of information conducted in this review indicated that a large part of the studies did not address the investigation of these themes or did not report how the actions carried out in the poles translated into results for the users of the program. In this sense, in addition to systematically reporting on how activities are developed with users , it is suggested that future studies related to the “Academia da Saúde” Program aim to investigate the relationship between activities developed in the poles and the themes for which the program was created , in order to identify successful practices and, consequently, enable their replication.Although the “Academia da Saúde” Program was created with the objective of promoting physical practices, regular physical activity, healthy eating, health education, interdisciplinary monitoring, as well as contributing to the production of healthy and sustainable lifestyles and care for the general populationAlthough the extensive literature review conducted in different databases to identify the literature related to the “Academia da Saúde” Program  is a strong point of the present study, some limitations need to be declared: (i) given the characteristics of the scoping review , the studies included in this review were not evaluated in terms of methodological rigor; (ii) although the strategy adopted to gather available information was comprehensive in terms of sources, it is possible that relevant studies related to the topic may not have been published in the searched databases.We concluded that Nutrition and Evaluation of the Program were the most evaluated topics by the studies, with program users being the main actors in these investigations. Future studies should consider investigating the effectiveness of the actions developed in the program’s poles, especially physical activity and healthy eating, which are the assumptions for the creation of the “Academia da Saúde” Program.DS: conceptualization, methodology, formal analysis, visualization, and writing–original draft. TL: investigation and writing–reviewing and editing. LG: investigation and writing–reviewing and editing. All authors contributed to the article and approved the submitted version.DS was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior–Brazil (CAPES)–Finance Code 001 and he is supported in part by National Council for Scientific and Technological–CNPq, Brazil .The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.https://www.frontiersin.org/articles/10.3389/fpubh.2023.1227899/full#supplementary-materialThe Supplementary material for this article can be found online at: Click here for additional data file.


Drug overdose deaths are the leading cause of unintentional death in the United States, and opioid-related mortality is the primary contributor (75.2%). Among opioid-related mortalities, opioids are most commonly taken with benzodiazepines (30.1%) and antidepressants (13.4%). The utility of a retrospective drug utilization review (DUR) program initiated by a commercial health plan for members taking potentially unsafe opioid and central nervous system (CNS) combination therapy is currently unknown.To determine the effectiveness of a retrospective DUR program on potentially unsafe opioid and CNS combination therapy.This research is a pre-post study utilizing pharmacy claims data from 2.6 million commercially insured members enrolled in a health plan in the Midwest. Members were required to be at least aged 18 years as of August 30, 2013, and continuously enrolled from May 2, 2013, through February 15, 2014. Members with 1 or more paid claims for an opioid at least 200 morphine equivalent dose (MED) daily and a concurrent supply of another opioid, benzodiazepine, or antidepressant from May 2, 2013, through August 30, 2013 (120-day preintervention period) were targeted for the retrospective DUR program. These exclusion criteria were applied: members belonging to commercial groups requiring permission on claims data analyses, missing or invalid prescriber information, or presence of pharmacy claims indicating human immunodeficiency virus or acquired immunodeficiency syndrome during the 2 years prior to the pre-intervention period. Prescribers of high-dose opioids received a mailing (intervention) containing a member-specific letter, medication profile, and satisfaction survey to determine the prescriber-perceived clinical value of the program. To assess the effectiveness of the retrospective DUR program, criteria was reapplied to identify members still meeting criteria 120 days postintervention . Paired samples t-test was used to compare pre-post results.Of 2,236,243 eligible members aged 18 years and older, 980 met DUR criteria. Prescribers for these members subsequently received a mailing regarding potentially unsafe opioid and CNS combination therapy. A total of 671 prescribers were sent a mailing regarding these 980 members. Among the 980 members meeting DUR criteria, distribution of prescriber specialty was family medicine (25.9%), physical medicine and rehabilitation (14.4%), internal medicine (13.0%), pain (9.2%), anesthesiology (7.0%), other (8.8%), and unknown (21.7%). High-dose opioids most commonly identified by the DUR were oxycodone extended release (27.6%), morphine sulfate extended release (17.7%), and fentanyl patch (13.1%). After reapplying DUR criteria to identify members still meeting criteria 120 days after the DUR, 528 members remained, representing a 28.1% reduction in high-risk opioid use. Survey response rate was 23.6% (231 of 980 surveys returned). The majority (62.3%) of respondents reported that this retrospective DUR program was useful in their daily practice.A 28.1% reduction in potentially unsafe opioid and CNS combination therapy was observed after implementing a retrospective DUR program targeting high-risk opioid use. Among members remaining high risk after the DUR, the change in total unique opioids and total daily MED was nonsignificant. Members remaining at high risk after the DUR can be targeted for further interventions such as care management and member education regarding fraud, waste, and abuse. A majority of prescribers (90.5%) self-report using their states’ prescription monitoring programs when prescribing controlled substances. What is already known about this subjectFrom 1990 to 2010, drug overdose deaths increased from 3.4 per 100,000 to 12.4 per 100,000, an increase of approximately 300%.Opioids are primarily responsible for the increase in drug overdose mortality, accounting for 75.2% of cases.Among opioid-related mortalities, opioids are most commonly taken with benzodiazepines (30.1%) and antidepressants (13.4%).What this study addsA commercial health plan implemented a retrospective drug utilization review (DUR) program targeting high-risk opioid use and observed a 28.1% reduction in such utilization.Among a commercially insured population, high dose opioids most commonly identified by a retrospective DUR program were oxycodone extended release (27.6%), morphine sulfate extended release (17.7%), and fentanyl patch (13.1%).The majority (90.5%) of prescribers self-reported using their states’ prescription monitoring programs when treating patients with controlled substances.2 From 1990 to 2010, drug overdose deaths increased from 3.4 per 100,000 to 12.4 per 100,000, an increase of approximately 300%.4 Opioids are the most common drug class involved in drug overdose deaths, with patients taking opioids in three fourths of cases.1 Doses greater than 100 morphine equivalent dose (MED) daily are associated with 11 times greater risk for opioid-related overdose death.5 Additionally, concomitant use of benzodiazepines and antidepressants are most often involved in opioid-related mortalities, 30.1% and 13.4%, respectively.6Drug overdose deaths are the leading cause of unintentional death in the United States and increased for the eleventh straight year in 2010.7 To combat the drug overdose epidemic, the CDC recommends the following: prescription drug monitoring programs (PDMPs), patient review and restriction programs, health care provider accountability, laws to prevent prescription drug abuse and diversion, and better access to substance abuse treatment. Retrospective drug utilization review (DUR) programs are a mechanism to monitor prescription claims information and have been successful in many different patient populations.8 However, the value of a retrospective DUR program for potentially unsafe opioid and central nervous system (CNS) combination therapy is unknown. A retrospective DUR program can identify members on potentially unsafe opioid and CNS combination therapy and contact their prescribers to reassess therapy and determine if dose reductions, discontinuation of potentially unsafe agents, or substitution for less dangerous medications are necessary. Third-party payers’ pharmacy and medical costs could decrease, since these members tend to be on multiple medications and high utilizers of medical resources.12 Hansen et al. (2011) reported that nonmedical use of prescription opioids costs the U.S. economy $53.4 billion, mainly due to lost productivity ($42 billion), followed by criminal justice costs ($8.2 billion), drug abuse treatment ($2.2 billion), and medical complications ($944 million).13The Centers for Disease Control and Prevention (CDC) classify drug overdose as an epidemic.The objective of this study was to determine the effectiveness of a retrospective DUR program among members whose claims history suggested potentially unsafe opioid and CNS combination therapy. This DUR identified members who were taking high doses of opioid analgesics concomitantly with other opioid analgesics, benzodiazepines, or antidepressants and encouraged their prescribers to reassess combination therapy.This pre-post study utilized pharmacy claims data from 2.6 million commercially insured members enrolled in a Midwest Blue Cross Blue Shield plan. To be eligible for analysis, members were required to be at least aged 18 years as of August 30, 2013, and continuously enrolled from May 2, 2013, through February 15, 2014. The study timeline is shown in Pharmacy claims were queried to identify members with an opioid claim at least 200 morphine equivalent dose (MED) daily and a concurrent supply of another opioid, benzodiazepine, or antidepressant from May 2, 2013, through August 30, 2013 (120-day preintervention period). Opioid, benzodiazepine, and antidepressant claims were identified using American Hospital Formulary System (AHFS) pharmacologic therapeutic classification codes .The following exclusion criteria were applied: members belonging to commercial groups requiring permission on claims data analyses, missing or invalid prescriber information, presence of pharmacy claims indicating malignancy, or human immunodeficiency virus (HIV) or acquired immunodeficiency syndrome (AIDS) during the 2 years prior to the pre-intervention period. HIV and cancer drugs were identified by AHFS codes 8:18.08 Antiretrovirals and 10:00 Antineoplastics, respectively. Claims for days’ supply less than 20 were excluded to capture chronic therapy because these members were at highest risk for opioid overdose.Identified members were targeted for the retrospective DUR program. The DUR involved a provider-directed intervention for the most recent prescriber of a high-dose opioid for the identified member. In October 2013, prescribers received a mailing via the U.S. Postal Service, containing a member-specific letter, medication profile, and satisfaction survey to determine the prescriber-perceived clinical value of the program  to assess the effectiveness of the retrospective DUR program. A 120-day preintervention and postintervention time period was selected to capture 90-day fills. Members were categorized based on their combination therapies. For example, opioid/antidepressant indicated that a member took at least 1 opioid and at least 1 antidepressant. Categories were as follows: (a) opioid/opioid, (b) opioid/benzodiazepine, (c) opioid/antidepressant, and (d) opioid/benzodiazepine/antidepressant. Statistical testing data for member characteristics are presented as means with standard deviations or frequencies with percentages. Paired samples t-test was used to compare differences from preintervention to postintervention. A 15 The study sample consisted of 528 members aged 18 years and older, who were continuously enrolled from May 2, 2013, through February 15, 2014, with an opioid claim of at least 200 MED daily and a concurrent supply of another opioid, benzodiazepine, or antidepressant during the preintervention and postintervention periods. Although 980 members were targeted for the retrospective DUR program, 246 members discontinued the Blue Cross Blue Shield pharmacy insurance and were excluded from postintervention analysis.Members meeting DUR criteria pre- and postintervention were the focus of this study’s analysis, since they represented the highest-risk members taking opioid and CNS combination therapy. A total of 206 members no longer met DUR criteria in the postintervention period, representing a 28.1% reduction in high-risk opioid use . Of the P = 0.213). Change in average total MED per day was non-significant . Among these high-risk members, the level of opioid exposure remained unchanged and did not increase. Average member age was 52.8 years, and 52.3% of the study cohort was male . The aveThe survey was sent to 671 unique prescribers managing pain therapy for the 980 members targeted for the retrospective DUR program. Of this number, 231 surveys were returned, representing a response rate of 23.6%. 16 As a result of the MassHealth DUR program, a prior authorization was implemented for all high-dose, long-acting opioid analgesics.This study showed that a retrospective DUR program targeting commercially insured members can result in a 28.1% decrease in potentially unsafe opioid and CNS combination therapy. High-dose opioids most commonly identified by the DUR were oxycodone extended release (27.6%), morphine sulfate extended release (17.7%), and fentanyl patch (13.1%). Such findings are consistent with the MassHealth DUR program in Massachusetts, which identified the necessity for a management strategy regarding long-acting opioid analgesics .17 Such strategy is more common among Medicaid plans, and lock-in programs could potentially benefit the highest-risk members still meeting our DUR criteria postintervention. Connecticut’s DUR Board observed a savings of $3.7 million and $2.4 million in 2009 and 2011, respectively, when examining results of its retrospective DUR programs and lock-in programs combined.18A targeted retrospective DUR program can be one of many different approaches that collectively address the opioid mortality epidemic. Additional strategies include lock-in programs in which controlled substance claims of a high-risk member is reimbursed only if a specific prescriber and pharmacy is used.Among the 980 members meeting DUR criteria in our study, distribution of prescriber specialty was family medicine (25.9%), physical medicine and rehabilitation (14.4%), internal medicine (13.0%), pain (9.2%), anesthesiology (7.0%), other (8.8%), and unknown (21.7%). Our retrospective DUR program may serve as a selection tool for prescribers who may benefit from academic detailing, as well as identify members that can be targeted for care management and member education regarding fraud, waste, and abuse. Although pain specialists were managing pain therapy for 9.2% of members targeted for the DUR, they were the largest group among survey respondents (36.7%). Pain specialists were very responsive regarding a retrospective DUR program targeting high-risk opioid use.9 Pergolizzi et al. (2014) found that monthly health care costs were significantly greater by $609 among patients exposed to a potential drug-drug interaction of clinical significance compared with those not exposed.19 Summers et al. (2011) found significantly higher ($667) average 6-month health care costs among patients exposed to opioid drug interactions compared with patients without such drug interactions.20 Although these studies focused on opioid drug interactions related to the cytochrome P450 pathway, such findings provide insight on costs associated with opioid drug interactions.Our retrospective DUR program identified members taking potentially unsafe opioid and CNS combination therapy. Interactions of such drug combinations can potentially result in respiratory depression and drug overdose death. The literature has shown that drug interactions involving opioids are associated with higher health care costs.In addition to retrospective DUR programs that can identify potentially unsafe opioid and CNS combination therapy, prescribers and pharmacists can utilize PDMPs, which identify controlled substances that members are obtaining from pharmacies. Health care professionals are able to determine if a member is receiving potentially unsafe drug combinations that otherwise may be unknown. Academic detailing efforts can be employed to encourage prescribers to use these resources when prescribing controlled substances. Our survey results showed that 90.9% of respondents reported routine use of their state’s PDMP when prescribing a controlled substance. Prescribers are utilizing PDMPs to appropriately manage their patients.The current study is not without limitations. First, the study used pharmacy claims data, which do not provide information regarding adherence or whether the member is actually taking the medication. Patient-specific characteristics captured by medical claims were not included, which could potentially illuminate factors that may be driving therapy change. Second, data are limited to a commercial population in the U.S. Midwest; therefore, findings may not be generalizable to Medicare/Medicaid populations or other geographic regions. Third, members often elect to pay cash for controlled substances, and this study was not able to capture cash transactions not billed through insurance. Fourth, response bias may be present among survey respondents, since the intervention and survey were administered by the health plan providing reimbursement to physicians. Fifth, there was no direct comparison group. Rather, the control was the historical claims experience of each member. The 28.1% decrease in potentially unsafe opioid use cannot be attributed solely to the intervention; the decrease may have occurred because of other factors that may impact opioid use, including disease severity and concomitant alcohol use. This limitation is inherent to pre-post study designs, but pre-post design in this study was used because the intervention was implemented to all members meeting criteria for potentially unsafe drug therapy in the health plan’s commercial population. Lastly, survey response rate was 23.6%. Future research in this area should focus on the patient and prescriber factors that are predictive of positive postintervention changes to opioid therapy.Commercial health plans can reduce risk of adverse effects and death associated with high-dose opioid analgesics used in combination with other CNS sedating agents. A 28.1% reduction in potentially unsafe opioid and CNS combination therapy was observed after implementing a retrospective DUR program targeting high-risk opioid use. Because the drug overdose epidemic is the leading cause of unintentional death in the United States, managed care organizations are well positioned to use multiple strategies such as care management and academic detailing, as well as member education regarding fraud, waste, and abuse, as follow-up to a retrospective DUR program for a comprehensive approach to further manage and reduce high-risk opioid use.