Syringe services programs (SSPs) hold promise for providing buprenorphine treatment access to people with opioid use disorder (OUD) who are reluctant to seek care elsewhere. In 2017, the New York City Department of Health and Mental Hygiene (DOHMH) provided funding and technical assistance to nine SSPs to develop “low-threshold” buprenorphine services as part of a multipronged initiative to lower opioid-related overdose rates. The aim of this study was to identify barriers to and facilitators of implementing SSP-based buprenorphine services.We conducted 26 semi-structured qualitative interviews from April 2019 to November 2019 at eight SSPs in NYC that received funding and technical assistance from DOHMH. Interviews were conducted with three categories of staff: leadership , staff , and buprenorphine providers (six interviews). We identified themes related to barriers and facilitators to program implementation using thematic analysis. We make recommendations for implementation based on our findings.Programs differed in their stage of development, location of services provided, and provider type, availability, and practices. Barriers to providing buprenorphine services at SSPs included gaps in staff knowledge and comfort communicating with participants about buprenorphine, difficulty hiring buprenorphine providers, managing tension between harm reduction and traditional OUD treatment philosophies, and financial constraints. Challenges also arose from serving a population with unmet psychosocial needs. Implementation facilitators included technical assistance from DOHMH, designated buprenorphine coordinators, offering other supportive services to participants, and telehealth to bridge gaps in provider availability. Key recommendations include: (1) health departments should provide support for SSPs in training staff, building health service infrastructure and developing policies and procedures, (2) SSPs should designate a buprenorphine coordinator and ensure regular training on buprenorphine for frontline staff, and (3) buprenorphine providers should be selected or supported to use a harm reduction approach to buprenorphine treatment.Despite encountering challenges, SSPs implemented buprenorphine services outside of conventional OUD treatment settings. Our findings have implications for health departments, SSPs, and other community organizations implementing buprenorphine services. Expansion of low-threshold buprenorphine services is a promising strategy to address the opioid overdose epidemic. Opioid overdose deaths in New York City (NYC) more than doubled between 2010 and 2019, despite the availability of evidence-based opioid use disorder (OUD) treatment , and opiWhile low-threshold buprenorphine services can be provided in traditional medical settings , syringe.In NYC, 15 SSPs are currently funded by the NYC Department of Health and Mental Hygiene (DOHMH) to provide harm reduction services to approximately 16,000 participants per year. In 2017, DOHMH launched an initiative to support SSPs in developing low-threshold buprenorphine services as part of a multipronged initiative to reduce opioid-related overdoses. Our objectives were to provide the first qualitative evaluation of buprenorphine service implementation at SSPs. First, we categorized program characteristics. Then, we sought to identify barriers to and facilitators of implementing SSP-based buprenorphine services. We use these data to make recommendations for implementation of SSP-based buprenorphine servicesIn 2017, DOHMH funded nine SSPs to develop buprenorphine services. Funding generally covered staff time, including consultants and subcontractors, and program supplies and equipment. Awards to programs ranged from $87,000–218,000 yearly (USD). DOHMH additionally provided technical assistance to SSPs, which consisted of informational sessions on regulatory compliance (two-part training) and quarterly learning communities (five sessions) over 15 months (December 2018–February 2020). Program managers and at least one other staff member involved in buprenorphine services were required to attend compliance and learning community sessions. Topics covered are listed in Box DOHMH also funded a harm reduction organization to conduct annual staff trainings to prepare staff to counsel SSP participants on treatment options. Programs were required to develop policies and procedures for buprenorphine services, for which DOHMH offered funding to hire consultants. Programs could request funding for an electronic health record (EHR)  in their budget proposals. Programs were encouraged to request individualized DOHMH technical assistance for any challenges they encountered in program development. Clinical mentoring was available from a harm reduction-experienced buprenorphine provider. All SSPs reported on buprenorphine services provided using an existing online reporting system developed by DOHMH for SSPs to report on other harm reduction services provided.This evaluation was deemed to not be human subjects research by the DOHMH Institutional Review Board. Two academic physicians (AJ and AF) unaffiliated with DOHMH evaluated SSPs’ experiences with implementing low-threshold buprenorphine services with support from DOHMH. AJ completed a total of 26 semi-structured qualitative interviews from April 2019—November 2019 at eight of the nine SSPs in NYC that received funding. One SSP was excluded because it developed buprenorphine services in collaboration with an academic medical center, so their experiences were not generalizable to the eight other SSPs. Interviews were conducted with three categories of staff: leadership , staff , and buprenorphine providers (six interviews). In the staff category, all but one SSP staff member were employed by the SSP prior to development of buprenorphine services. SSP leadership selected staff and provider interviewees after researchers contacted them and explained the objectives of the study.Semi-structured interviews followed a standard script that was developed in collaboration with DOHMH. Interview guides were tailored to the program role of the interviewee Table . Twenty-Findings from interviews are summarized describing prominent themes. We describe program characteristics without quotations for brevity. Data on barriers and facilitators of program implementation are supplemented with direct quotations from interviewees that provide context or highlight critical points. We provide recommendations for program implementation based on the findings from this study. Throughout the manuscript, we refer to people who use SSP services as “SSP participants”.Stage of buprenorphine development: Of the eight SSPs included in the evaluation, five developed new buprenorphine programs and were active at time of interview, one SSP had a buprenorphine program established prior to DOHMH funding, and two SSPs initiated the development of new buprenorphine programs, but had to stop due to setbacks . Characteristics of the six active buprenorphine programs are highlighted in Table Buprenorphine providers were nurse practitioners, physician assistants, and physicians , employed part-time by the SSP or full-time by the organization’s medical clinic. One SSP partnered with an addiction medicine fellowship program to host a rotating addiction medicine provider-in-training. SSPs employed between one and three buprenorphine providers contracted for a set number of hours per week.Programs varied in their use of electronic documentation. Organizations that had medical clinics used existing EHRs. At one program, the provider used their own cloud-based EHR which they also used in their private practice. Two programs used paper charts. One program used the SSP's existing data management software. All programs used electronic prescribing, in compliance with state and federal regulations.. Buprenorphine coordinator roles included providing education and orientation to buprenorphine services; conducting eligibility screening; monitoring participant engagement; providing navigation services; coordinating with buprenorphine providers; supervising buprenorphine peer specialists and navigators; and SSP duties unrelated to buprenorphine treatment. Examples of navigation services included: making appointment reminder calls; contacting participants who were due for refills; helping with pharmacies and insurance authorizations; and providing psychosocial support .The six active programs all employed buprenorphine coordinators who were nurses, medical assistants, social workers, or other SSP staff with informal training in buprenorphine treatmentSSP staff promoted buprenorphine services using fliers, brochures, and conversations with existing participants at office and mobile sites. Three programs formally involved peers  as buprenorphine champions or specialists to engage SSP participants who expressed interest in buprenorphine. These peers conducted community outreach at mobile sites and served as point-persons for other SSP staff members who identified SSP participants interested in buprenorphine. Interested participants were then connected with buprenorphine coordinators for more in-depth counseling and an introduction to buprenorphine services.Buprenorphine can displace other opioids from opioid receptors and cause severe withdrawal symptoms if taken too soon, thus participants must wait until they are in moderate opioid withdrawal to take the first buprenorphine dose. Most programs used a “home induction” approach, where the provider instructed participants when and how to take the first dose of buprenorphine at home . Some prPrograms performed urine toxicology testing at different frequencies, ranging from every buprenorphine visit to random intervals. Use of urine toxicology testing varied depending on the provider. No provider reported routinely stopping treatment for opioid-positive urine toxicology tests. Some providers increased the frequency of visits or spoke with participants about alternative treatments if they had multiple opioid-positive tests. All buprenorphine providers required that participants have buprenorphine-positive urine toxicology tests to continue treatment.There were numerous barriers to providing buprenorphine services at SSPs. These included: staff knowledge and skills gaps, difficulty hiring and retaining buprenorphine providers, managing tension between harm reduction and traditional OUD treatment philosophies, and financial constraints. Challenges also arose from serving a population with unmet psychosocial needs.Program staff members reported that their leadership needed additional guidance at the beginning of implementation, particularly sites that did not have existing clinical infrastructure. Although leaders were experienced in managing nonprofits, many lacked experience building health service programs. Specifically, leaders lacked requisite knowledge regarding provider recruitment and contracting, malpractice insurance requirements, creating clinical policies and procedures, regulatory requirements, and electronic health records.Hiring buprenorphine providers and (2) Comfort with harm reduction or “low-threshold” treatment principles.Hiring buprenorphine providers:(1) Programs found it challenging to identify buprenorphine providers who were experienced with buprenorphine treatment and willing to work part-time and in a harm reduction context. Covering malpractice insurance was prohibitively expensive for SSPs, and finding buprenorphine providers who had their own malpractice insurance was difficult, limiting the pool of potential candidates. Programs posted job listings online and asked personal or professional connections to advertise positions. Programs affiliated with medical clinics benefitted from established clinician recruiting teams. SSPs not affiliated with medical clinics hired buprenorphine providers for one to twelve hours per week, due to financial constraints, which was another challenge. However, finding the right provider was difficult: “We really don't have a provider that really understands the population.”—Program Coordinator (Program 2).Comfort with harm reduction or “low-threshold” treatment principles:2) Programs had difficulty finding harm reduction-oriented buprenorphine providers, and few buprenorphine providers had previous experience working in harm reduction settings:“My approach has changed over time. Because when you study something at the beginning you, you're trying to kind of follow it to the letter –and I didn't have the concept of harm reduction either before I came. So I kind of learned as I have been here. —Buprenorphine provider (Program 4).Provider-related challenges fell into two main categories: (1) “I think finding a provider that understands and truly practices harm reduction with their client is something that's rare. I feel like every provider has a tendency to be abstinence-based and use fear tactics when talking to their clients or their participants, finding someone that will implement [harm reduction] into their care has been a challenge.” —Program Manager (Program 3).Program staff recognized this challenge::“I told you I'm a little bit of a control freak… And I'm like that with the buprenorphine because it is a controlled substance… And number one, I don't want to get myself in trouble… and number two, I also don't want to be so lackadaisical that someone else could hurt themselves… I'm responsible… I'm not giving it to you for you to hurt yourself.” —Buprenorphine provider (Program 4).Some buprenorphine providers expressed concerns about their participant's continued opioid use. More than one provider were reluctant to provide buprenorphine prescriptions to participants who were also taking benzodiazepines. Individual buprenorphine providers had different practices around continuing to prescribe buprenorphine to participants who missed appointments. Buprenorphine providers also expressed concerns about their legal liability and risks to participantsHarm reduction staff did not always agree with provider practices that conflicted with harm reduction principles or deviated from their understanding of clinical guidelines, but they were uncomfortable communicating this to buprenorphine providers: “It's a little difficult as to how we manage because we don't want to disrespect the doctor.” —Program Manager (Program 2).Some staff members suggested that DOHMH should train buprenorphine providers in harm reduction principles, as they felt they had limited authority to give buprenorphine providers feedback on their practice. The state department of health offered a learning community for SSP buprenorphine providers, but attendance was voluntary, and buprenorphine providers often were unable to attend due to conflicting clinical schedules.“And then we had [redacted], who’s a wonderful fit. She was with us I think for six months, she was really great. She was the one that was out in the mobile unit, was able to engage a lot of people into the program… She’s a harm reductionist, like she understood opioid use disorder in a way… that most prescribers that I’ve talked to have not understood it.” —Program Manager (Program 3).When programs were able to find a harm reduction-oriented provider, this was a major facilitator to program implementation. Harm reduction-oriented buprenorphine providers were able to engage with SSP participants and work effectively in non-traditional settings:What has been the biggest challenge with the program to date? (Interviewer)“– well, it's about moving from not offering bupe into making it widely available without sending the message that you are being abstinence based.… But, when we talk to our clients (we) say this is an option… it all depends on how this relates to your life and to the things that you want to do.” —Leadership (Program 5).SSPs historically have not offered OUD treatment, and some programs noticed philosophical differences between traditional, abstinence-based treatment, and harm reduction approaches. Program leadership discussed concerns that offering buprenorphine (bupe) services would imply that the organization expected participants to stop non-prescribed opioid use.“I think it's sometimes, it's sometimes knowing that someone is going in the path that they want and all of a sudden, (they have) a big relapse. So that, emotionally for the harm reduction team as much as they want to keep the philosophy, it just really bothers the team… So that's a challenge, that it's hard to see, but because you're getting to the same level of the clients and you're not being pushy about it, but ask(ing) them what they want –– right, it gets a little bit more frustrating.” —Program Manager (Program 6).Staff members also found it difficult navigating their roles as harm reductionists and helping people engage in buprenorphine services. One staff member spoke about how offering buprenorphine treatment changed their expectations for participants, leading to disappointment if participants resumed using non-prescribed opioids, which typically would be understood differently from a harm reduction perspective:“You know, with buprenorphine basically you have to understand that the clients don't really know too much. And basically the messaging [about buprenorphine] has to be really specific. It has to be something that might catch [participants’] attention.” —Leadership (Program 4).Interviewees reported challenges with staff knowledge about buprenorphine at the beginning of program implementation. The annual staff training provided was perceived to be geared toward a medical audience, which was too technical for frontline staff. Even staff members closely involved in the program primarily learned about buprenorphine informally. For example, some staff members had personal experience with buprenorphine treatment, and others learned on the job from working closely with a provider or another buprenorphine program staff member. Programs also identified a need for refresher trainings for staff. Frontline staff desired training to help communicate quickly and effectively to SSP participants about buprenorphine:Unique characteristics; (2) Unmet service needs; and (3) Participants’ prior negative experiences with buprenorphine.SSP leadership and staff perceived participant challenges in the following categories: (1) Unique characteristics: Some programs observed that unique characteristics of their participants tempered interest in buprenorphine services. One program served a young population, whom they perceived as lacking interest in OUD treatment. Another program was located near a methadone program, and most SSP participants were already enrolled in methadone treatment. Programs serving populations without stable housing noted the unique challenges of buprenorphine in this population:“… they knew that buprenorphine was something that was mostly prescribed to specific populations meaning, you know, white America that were fully housed… and it was really, they didn't find it to really be like for them… so if I start this… I get a prescription where do I keep it, where do I store it, where do I put it. For clients who are chronically homeless that… certainly becomes a challenge…” —Leadership (Program 5).(1) To help address challenges with storage, some participants received small quantities of medication and returned multiple times per week for renewal of prescriptions. Other participants found pill boxing buprenorphine helpful to improve their adherence.Unmet service needs: A common theme was that buprenorphine alone did not meet all of participants’ needs. Staff perceived that participants required supportive services related to basic needs and buprenorphine to be successful in treatment. Services identified included peer navigations services, vocational training, and housing:(2) “We need other resources, viable resources that we can present to the clients for them to be adherent and stable in their life. I mean… some form of housing vouchers and even like some clothing, meals… employment, they just need realistic options – I think like right now, they don't really see a way forward… Okay, I'm getting this medication and stuff, I'm taking Suboxone, but these other things in my life ain't getting right… I think they need something to see in the future and I don't think they're really seeing it.” —Buprenorphine Coordinator (Program 4).History of negative experiences with buprenorphine: Many SSP participants reported having had past experiences of precipitated withdrawal when taking buprenorphine and were reluctant to try it again. In response, staff attempted to dispel misinformation about buprenorphine and counseled participants on how to avoid precipitated withdrawal when starting buprenorphine.(3) “…The salary ranges have been a pretty big issue…[employees] could also leave here and make $10,000 more somewhere else… Buying supplies can somewhat be a challenge… making sure we have enough MetroCards for our clients. Because we give them train fare for all their medical appointments… I think if we can give clients incentives – not for their medical appointment, but for like meeting with my staff that would help engage them into care, but we can't afford – incentives right now.” —Program Manager (Program 3).The primary financial constraint for programs was hiring buprenorphine providers. Most programs only had funding to hire a medical provider for a limited number of hours per week and could not afford buprenorphine providers without their own malpractice insurance coverage. Lack of funding also made it difficult to retain buprenorphine providers and mental health professionals in some organizations. SSPs not associated with medical clinics reported that electronic health records were unaffordable. SSPs not affiliated with a medical clinic were also unable to bill insurance for medical services and thus relied exclusively on grant funding to sustain the program.Sustainability was particularly difficult for programs operating out of mobile vehicles. Upkeep and cost of repairing mobile vehicles was a barrier for sustainability. Finding buprenorphine providers who were interested and skilled in working in a mobile setting was also challenging.SSPs that were part of organizations with medical clinics had the greatest capacity to provide regular services and same-day buprenorphine treatment. Other programs were able to successfully contract with part-time buprenorphine providers when these buprenorphine providers had their own malpractice insurance (either independently or through another organization) and were willing to extend their availability via telehealth. Provision of remote services via telehealth helped bridge gaps when in-person hours were unavailable. Two programs arranged for telephonic follow-up if participants came to the SSP when the provider was not available in-person. One program compensated their provider (using grant funding) for an additional 1.5 hours per week for telehealth visits to attend to participants who had been unable to attend in-person appointments and facilitate prescription refills. Buprenorphine coordinators were crucial to maintaining continuity of care in programs with limited provider hours..Technical assistance from the DOHMH was the key in several areas, particularly in developing policies and proceduresAt the beginning of the initiative, as part of the funding requirements, programs were asked to create policies and procedures which included protocols for starting buprenorphine, follow-up intervals for participants, and laboratory testing tailored for their organization and participants. However, many programs struggled, having little experience creating clinical protocols. Two SSPs hired consultants using funds provided by DOHMH, but other programs could not identify consultants with the necessary expertise. DOHMH later provided templates and individualized assistance to SSPs to develop their own policies and procedures, a key facilitator to programs that did not hire consultants.“And [DOHMH staff member] was very helpful and responsive… it was helpful to have conversations because we would identify and then look at issues that hadn't been thought of in advance… I was focused to some extent on… risk management for the organization, right; making sure that we were not going to have the state health department… breathing down our necks because we were providing services in some way, that, you know, was considered too broad… And then there were questions around, you know, insurance and whose insurance covered what, if it was under the individual, their malpractice. Policy and procedures, questions, data questions…” —Leadership (Program 1).The DOHMH also assigned a single staff member as the point person to answer questions that arose during the implementation process. This point person assisted programs with a range of challenges, including finding buprenorphine providers and addressing medicolegal concerns .“So for the most part, our clients are kind of honest in telling us things… And I told them like if you're using, you know, I ain't going to stop you from getting prescribed… I mean, that's not what I'm here for.… How could I help you maintain your adherence to Suboxone and stop you from using—and some like just need to talk.” —Buprenorphine Coordinator (Program 3).DOHMH encouraged SSPs to designate a dedicated buprenorphine coordinator. Programs who followed this advice reported that it was a facilitator of program success. Buprenorphine coordinators gained participants’ trust, perhaps more easily than buprenorphine providers:“…Me and the providers got together and we started identifying clients that were high risk of failure or at risk of failure, for whatever variety of reasons, and then we'll come in, collaborate with the doctor and the client at the same time – and work out a plan as to okay, this is how we can work this client through this part of his life to become stable on Suboxone.” —Buprenorphine Coordinator (Program 3).At one program, coordinators collaborated closely with buprenorphine providers to identify ways to support participants:“…And we do everything possible, calling insurances, walking to the pharmacy – so like every step of the way – we make sure you get [buprenorphine] and nothing happens in between from the van to the pharmacy… We have that urgency like you're here now, we're getting this for you now.” —Buprenorphine Coordinator (Program 2).Several SSPs offered more support services than typically can be provided in a doctor’s office. Buprenorphine coordinators and peers provided a variety of navigation and support services. At one SSP, peer navigators (supervised by the buprenorphine coordinator) accompanied participants to healthcare appointments, conducted home visits, and, when needed, delivered MetroCards the day before appointments. The following quote details some of the auxiliary supports offered at another SSP:“Well, I've been fortunate that the pharmacy we deal with is actually pretty good and – with the population that we serve, you know, there's always that that, that stigma… and they have been looked at differently, not him (pharmacist)… He greets them, he speaks to them like folks.” —Buprenorphine Coordinator (Program 6).Establishing a relationship with a local pharmacy able to stock and dispense buprenorphine was a key facilitator for four programs. Staff could be confident that buprenorphine would be in stock , participants would be treated respectfully, and pharmacies would help troubleshoot insurance problems. Pharmacies affiliated with federally qualified health centers were able to provide discounted medication through the 340B Drug Pricing program . One phaTaken in total, these interviews provide key lessons learned for implementing low-threshold buprenorphine services at SSPs. Below we summarize our recommendations for key stakeholders  based on the barriers and facilitators we identified in this report Table .Table 3RThis study aimed to identify barriers to and facilitators of implementation of SSP-based low-threshold buprenorphine services and make recommendations for implementation. We found that most programs successfully implemented at least some buprenorphine services despite experiencing challenges related to the novelty of providing buprenorphine services onsite and finding buprenorphine providers. Programs with pre-existing clinical infrastructure had many advantages in implementing and sustaining buprenorphine services. Many SSPs throughout the USA do not have this advantage and would benefit from support from public health agencies for developing clinical infrastructure, selecting and training providers, and training staff. Overall, SSPs are promising sites to expand access to low-threshold buprenorphine services.At SSPs, buprenorphine providers are generally not onsite full-time, therefore, having dedicated staff who can provide continuity is crucial. As such, we recommend having a dedicated buprenorphine coordinator to facilitate program implementation and ongoing management. This has been demonstrated in HIV treatment settings and is a key component of the “Massachusetts Model” of office-based buprenorphine treatment , 19. SimOther program characteristics that differed between sites may also facilitate implementation. SSPs were able develop successful programs within drop-in centers, mobile units or in partnership with established community health centers. However, not all programs were able to hire a harm-reduction oriented provider, which was an essential component of successful programs. Programs also differed in their involvement of peers. Few programs formally involved peers in buprenorphine services. When they were formally involved, peers served as participant navigators, provided other supportive services, and played critical roles in engaging participants. Training peers in buprenorphine and involving them in implementation of buprenorphine services could be an important strategy to improve the reach of buprenorphine services.Establishing successful SSP-based buprenorphine services will also require confronting philosophical differences between OUD treatment and harm reduction. Heller and colleagues described these differences in reference to implementing HIV care at SSPs, highlighting that traditional medical models are hierarchical, center around physician expertise, and expect patients to be compliant with prescribed treatment plans . Our finImplementing buprenorphine services at SSPs also requires additional attention to financial sustainability. Programs were funded by a large city health department as part of a major multi-sector strategy to reduce overdose deaths. Significant financial support is needed to hire buprenorphine providers and pay for malpractice insurance. Innovations in the malpractice market are necessary to make contracting with individual buprenorphine providers more feasible for SSPs. Until then, in places where there are multiple SSPs or community organizations that wish to implement buprenorphine services, organizations may be able to partner with a medical clinic and provide funding for them to lend a part-time buprenorphine provider. Alternatively, health departments could employ buprenorphine providers to work in SSPs. In some states, SSPs may be able to bill for medical services, including buprenorphine services. Health departments and SSPs should explore whether SSPs billing for buprenorphine services would create a viable revenue stream and increase program sustainability or whether the start-up costs and staffing costs for billing would be too high. In some states, policy changes may be required to allow SSPs to bill for health services. Buprenorphine treatment is highly cost effective to society due to reductions in patients’ use of emergency health services and criminal-legal involvement . Thus, aInterviewing individuals from multiple programs at different stages of development provided a diversity of models and perspectives on barriers and facilitators throughout the implementation process. Interviews were conducted with individuals with varying roles at the SSPs, including buprenorphine providers, leadership, buprenorphine coordinators, and other SSP staff. Finally, members of the study team were from outside DOHMH, reducing some potential biases in the study.We used strictly qualitative methods, so data were not collected systematically on process measures such as number of staff trainings SSPs held or number of SSP participants approached about buprenorphine treatment. The study was conducted up to two years after implementation, introducing recall bias and reducing the opportunity to act on program feedback in a timely manner. The study interviewed SSP staff but not participants, so may have missed important perspectives of those most impacted. Finally, we only interviewed SSP stakeholders, not DOHMH staff, so the perspective of the funders was not formally examined.Our finding that offering buprenorphine services may change SSP participants’ and staff members’ perceptions of the SSP’s harm reduction mission deserves additional investigation. While we did not examine participant perspectives in this study, some SSPs reported concern that participants would question the organization’s commitment to harm reduction after they started offering buprenorphine services. Staff members also reported shifts in their expectations for participants who engaged in treatment, expressing hopes for consistent adherence to buprenorphine, abstinence from non-prescribed opioids, and greater stability in participants’ lives. Some staff were concerned that such change in expectation would compromise their non-judgmental stance toward a participant’s substance use. It is important to support staff and organizations in exploring their understanding and practice of harm reduction and an evolving understanding of harm reduction principles and OUD treatment applied in new contexts. Lastly, this study examined implementation of low-threshold buprenorphine services, but understanding SSP participants’ experiences with such services will be an important area of future study.Despite encountering challenges, eights SSPs in NYC have implemented buprenorphine services with DOHMH support, serving a population at risk for opioid-related harms that may be reluctant to seek treatment elsewhere. Lessons learned from this study can be used to support SSPs and other community organizations in developing and improving buprenorphine services. Over time, SSPs have adapted to community needs in providing sterile syringes, distributing naloxone, and now improving access to lifesaving OUD treatment. SSPs are valuable community resources that improve the health of people who use drugs.


The gap between demand and supply of kidneys for transplantation necessitates the use of kidneys from extended criteria donors. Transplantation of these donor kidneys is associated with inferior results, reflected by an increased risk of delayed graft function. Inferior results might be explained by the higher immunogenicity of extended criteria donor kidneys. Normothermic machine perfusion (NMP) could be used as a platform to assess the quality and function of donor kidneys. In addition, it could be useful to evaluate and possibly alter the immunological response of donor kidneys. In this study, we first evaluated whether complement was activated during NMP of porcine and human discarded kidneys. Second, we examined the relationship between complement activation and pro-inflammatory cytokines during NMP. Third, we assessed the effect of complement activation on renal function and injury during NMP of porcine kidneys. Lastly, we examined local complement C3d deposition in human renal biopsies after NMP.NMP with a blood-based perfusion was performed with both porcine and discarded human kidneys for 4 and 6 h, respectively. Perfusate samples were taken every hour to assess complement activation, pro-inflammatory cytokines and renal function. Biopsies were taken to assess histological injury and complement deposition.Complement activation products C3a, C3d, and soluble C5b-9 (sC5b-9) were found in perfusate samples taken during NMP of both porcine and human kidneys. In addition, complement perfusate levels positively correlated with the cytokine perfusate levels of IL-6, IL-8, and TNF during NMP of porcine kidneys. Porcine kidneys with high sC5b-9 perfusate levels had significantly lower creatinine clearance after 4 h of NMP. In line with these findings, high complement perfusate levels were seen during NMP of human discarded kidneys. In addition, kidneys retrieved from brain-dead donors had significantly higher complement perfusate levels during NMP than kidneys retrieved from donors after circulatory death.Normothermic kidney machine perfusion induces complement activation in porcine and human kidneys, which is associated with the release of pro-inflammatory cytokines and in porcine kidneys with lower creatinine clearance. Complement inhibition during NMP might be a promising strategy to reduce renal graft injury and improve graft function prior to transplantation. Normothermic machine perfusion (NMP) is a preservation technique that recently has been introduced to assess organ quality prior to transplantation. The increasing gap between demand and supply resulted in the necessity to transplant kidneys with a lower graft quality and has resulted in an increased risk of delayed graft function and inferior renal function compared to standard criteria donor kidneys , 2. The Kidneys are equipped with a sophisticated localized immune system. Previous studies have shown that severe injury, such as brain death or ischemia–reperfusion injury, induces a robust local immune response. As part of the local immune response, the complement system is activated , 5–7. Ton = 20) were exposed to a warm ischemia time of 30 min to induce ischemic injury. Subsequently, kidneys were cold flushed with 180 ml of cold (4°C) saline  and stored for 24 h by either static cold storage (SCS) or with hypothermic machine perfusion (HMP) with a mean arterial pressure of 25 mmHg with no active oxygenation or with 21% or 100% oxygen  with a mixture of 95% O2/5% CO2 at a flow rate of 0.5 L/min. During NMP, there was a continuous supply of nutrients, which was administered at a rate of 20 ml/h (n = 2) kidney-free NMP control group where NMP was performed with heparinized pig blood for 4 h without a kidney.In the preclinical phase, porcine kidneys  and autologous heparinized whole blood were used, as described previously . In shor% oxygen . Kidneys% oxygen Figure 1ex situ NMP for kidney regeneration (PROPER) trial registered in the Dutch Trial Register as NL8446. The PROPER trial is a Dutch initiative from three University transplant centers—The University Medical Center Groningen (UMCG), Leiden University Medical Center (LUMC), and Rotterdam Erasmus Medical Center (Erasmus MC)—to introduce and clinically evaluate prolonged NMP. As part of the PROPER trial, the current study investigated the role of complement activation during NMP in human discarded kidneys. The ten analyzed kidneys were included between January 1, 2019 and August 1, 2019. The included kidneys were retrieved from donation after brain death (DBD) and donation after circulatory death (DCD) donors, but discarded postretrieval. Reasons for discarding are described in via SCS or with HMP. As a perfusion solution, matched banked red blood cells (RBCs) were used, which were washed prior to NMP with 2 L of 0.9% NaCl by using a CellSaver  and afterwards diluted with 0.9% NaCl (via an oxygenator (Medos Medizin AG) with a mixture of 95% O2/5% CO2 at a flow rate of 0.5 L/min. During NMP, there was a continuous supply of nutrients. Specifics on infusion and the perfusion solution can be found in Human discarded kidneys were included within the Prolonged .9% NaCl Table 1.g for 20 min. Plasma was stored at −80°C.Perfusate samples during NMP were taken at baseline, 30 min, 1 h, 2 h, 3 h, and 4 h. Samples at time points 5 h and 6 h were only taken during NMP of human discarded kidneys. Samples were collected in EDTA tubes  and stored on ice and centrifuged at 4°C, 2,500 × Creatinine concentration was measured in perfusate and urine, using routine procedures at the clinical chemistry lab of the University Medical Center Groningen. Creatinine clearance (L/min/100 g) was calculated with the following formula ((urine creatinine concentration (mmol/L) * urine flow (ml/min)/perfusate creatine concentration (mmol/L))/kidney weight (g))) *100.Renal biopsies were taken after 4 h of NMP. Paraffin sections (4 μm) were stained with hematoxylin-eosin (H&E). Histological injury was scored on the basis of two criteria: proximal tubular cell necrosis and proximal tubular cell edema. Histological injury was scored blinded by two independent examiners under the supervision of a pathologist. Proximal tubular cell necrosis and proximal tubular cell edema were scored as described previously : (1) no 2O2. Next, the sections were incubated with primary antibody against C3d . Subsequently, sections were incubated with a goat-anti-rabbit-FITC antibody. Finally, sections were incubated with 4’ 6-diamidino-2-phenylindole (DAPI) to stain the nuclei and mounted with Citifluor. Tissue images were acquired using Leica confocal microscope . Sections were quantified by two independent examiners. An immunofluorescence density (ID) score was calculated based on (1) the intensity of the staining in the renal cortex (scored from 1 to 5) and (2) the percentage area of the renal cortex with C3d deposition (scored from 0 to 100).Renal biopsies were taken after 6 h of NMP. Frozen sections (4 μm) were fixed with acetone and endogenous peroxidase was blocked by incubating all sections in 30% HTo measure complement activation products in the perfusate samples, complement activation products at the level of C3 were measured. In the perfusate samples retrieved from NMP with porcine kidneys, C3a was measured using a highly specific porcine ELISA described earlier, using antibodies reacting with the C3a fragment . The terC3 and C3d were measured in perfusate samples after NMP of human discarded kidneys. C3d was measured as described previously . BrieflyTo detect and quantify pro-inflammatory cytokines, tumor necrosis factor (TNF), interleukin-6, and interleukin-8 in the perfusate samples of porcine kidneys, we used commercial porcine immunoassay kits according to the manufacturer’s instructions . For the detection and quantification of interleukin-6 and interleukin-8 in perfusate samples of human kidneys, we used commercial human immunoassay kits according to the manufacturer’s instructions .U post-hoc test. Differences between two groups were tested by a Mann–Whitney U test. Spearman correlation coefficients were calculated to determine which cytokines were significantly associated with complement C3a and sC5b-9 levels. P < 0.05 was considered significant. Data are presented as median ± interquartile range (IQR).Statistical analysis was performed using GraphPad Prism Software v8.3.1. Differences between time points were tested using Kruskal–Wallis test followed by a Mann–Whitney Complement activation was assessed by the quantification of perfusate levels of C3a and sC5b-9 during 4 h of NMP. C3a levels increased after 2 h of NMP with a steep increase to 4 h NMP and were significantly increased compared to C3a perfusate levels at 30 min of NMP  were formed based on the median value of sC5b-9 after 4 h of NMP (median 105 AU/ml) and the creatinine clearance for these two subgroups was calculated. Kidneys with sC5b-9 perfusate levels above 105 AU/ml after 4 h of NMP had a significant lower creatinine clearance than kidneys with sC5b-9 levels below this value . Kidneys with high sC5b-9 perfusate levels had significantly more proximal tubular cell necrosis and edema than kidneys with low 2C5b-9 perfusate levels  or based on the cold ischemia time (shorter versus longer than 15 h) (data are not shown).To investigate the role of different types of donors, C3d/C3 ratio during NMP of kidneys retrieved from DBD D donors Figure 8In addition, C3d deposition was examined in human renal biopsies taken after 6 h of NMP .The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


In situ simulation (ISS) programs deliver patient safety benefits to healthcare systems, however, face many challenges in both implementation and sustainability. Prebriefing is conducted immediately prior to a simulation activity to enhance engagement with the learning activity, but is not sufficient to embed and sustain an ISS program. Longer-term and broader change leadership is required to engage colleagues, secure time and resources, and sustain an in situ simulation program. No framework currently exists to describe this process for ISS programs. This manuscript presents a framework derived from the analysis of three successful ISS program implementations across different hospital systems. We describe eight change leadership steps adapted from Kotter’s change management theory, used to sustainably implement the ISS programs analyzed. These steps include the following: (1) identifying goals of key stakeholders, (2) engaging a multi-professional team, (3) creating a shared vision, (4) communicating the vision effectively, (5) energizing participants and enabling program participation, (6) identifying and celebrating early success, (7) closing the loop on early program successes, and (8) embedding simulation in organizational culture and operations. We describe this process as a “longitudinal prebrief,” a framework which provides a step-by-step guide to engage colleagues and sustain successful implementation of ISS.The online version contains supplementary material available at 10.1186/s41077-023-00243-6. In situ simulation is conducted in the actual care environment  and servIn situ simulation (ISS) is the nexus of rival priorities in healthcare systems. The competing demands of clinical performance, efficiency, patient safety, and applied learning often clash when they intersect in frontline ISS. Despite more than 10 years of demonstrated benefits to healthcare systems –15, ISS When they are not adequately tuned in to the competing priorities, they can be blind sided when their well-intentioned efforts to start and sustain ISS programs are met with reluctance, fear, resentment, or outright refusal –18.Health system leaders who have committed space, simulation equipment, staff time and faculty development to improve quality, and safety or staff engagement via these programs may wonder about the return on investment.This produces two challenges for simulation practitioners:Simulation leaders must therefore design for impact across nested organizational levels, in which individuals are nested in groups, nested in departments, nested in organizations which are part of health systems, and influenced by the external environment. A case for simulation that is persuasive at the team or sub-unit level, to be successful, may need to demonstrate impacts at higher levels. And research on change leadership that includes simulation will benefit from a multi-level lens .The simulation literature has focused intently on prebriefing — information sharing for participants that play a role in psychological safety to build engagement for participants, primarily with a focus on during a session that typically immediately follows the prebriefing. Critical actions suggested for prebriefing include the following: clarifying objectives, equipment functionality, roles of the participants and faculty, confidentiality of the session, and expectations of participants during the simulation , 20, 21.Prebriefing ISS teams may pose additional challenges including complexity of scheduling participants within a clinical environment and safety risks related to using real or simulated equipment in clinical areas , 18. WhiThis paper therefore explores the question, “How did three disparate ISS programs address the challenge of linking in situ simulation to the concerns and goals of health system colleagues?” We present a retrospective description of our evolving practice of managing and leading ISS programs and apply Kotter’s theory of change leadership  to categThe author group convened based on dialogue prompted by postings on an online journal club about in situ simulation . Authors S. E., S. B., S. J., and K. B. described establishing programs at three locations, encountering comparable institutional and learner barriers, and using similar strategies to aid implementation. These anecdotal comparisons of successes inspired a more formal exploration of the processes to see if we could identify a framework to guide others in ISS implementation. We inductively analyzed emergent practices that authors S. E., S. B., S. J., and K. B. developed and adapted as they rolled out ISS programs in three settings in North America and Australia. Program A, in a large health system in Queensland, Australia, employed ISS to evaluate and improve code team performance at an urban academic medical center on the east coast of the USA, utilized ISS to improve performance on obstetrical emergencies; program B, at an urban academic medical center on the east coast of the USA, utilized ISS to improve performance on obstetrical emergencies; and program C used ISS to improve adherence to national resuscitation guidelines in a neonatal intensive care unit at a Midwestern United States Medical Center. Analyzing programs from three unique contexts allowed the author team to consider knowledge gained from their own programs as insiders and also view the other programs as outsiders in alignment with comparative qualitative methodology . ReportiMost research reported in biomedical and health professions education literature is “outsider” research conducted from an observer perspective . In thisTo better compare our ISS implementation narratives, each site wrote their respective approach without any specified form/structure other than removing all identifiable information. In reconstructing timelines and developing their narratives, the authors reviewed documentation from their own institutions, such as emails, meeting minutes, instructor notes, flyers for participants, and reports of successes shared within our institutions or at professional organizations/conferences. All narratives and subsequent analyses were written on Microsoft Word documents and shared between the authors in a secure online portal. These narratives were then reviewed as a group and analyzed initially using thematic analysis , 31 combData analysis followed the six-step thematic analysis process as outlined by Braun and Clarke . AuthorsThese themes were then compared against Kotter’s change leadership steps to adapt prescriptions for the simulation community based on the authors’ collective experiences. The final phase of thematic analysis is producing the report , and theSeveral techniques were employed to enhance the trustworthiness of our data analysis . During We would like to note there might be some limitations to our conclusions drawn via these methods because of our insider status and commitment to change leadership. Our research team was highly motivated to identify themes and patterns to clarify the ISS work we had done. While this served as an engine for the work, it subjected us to “motive-driven cognition,” i.e., we were at risk for arriving “at a particular conclusion, attempt to be rational, and to construct a justification of [our] desired conclusion that would persuade a dispassionate observer”  (p.272).In his seminal work “Why Transformation Efforts Fail” , Kotter Establish a sense of urgency in ISS implementation mirrors Kotter’s first step since successful implementation of ISS programs requires impetus to overcome organizational complacency. Simulation change agents often find that “no urgency-no buy-in.” Potent drivers of change can include “pain points” such as poor performance on benchmarked quality indicators, financial impacts, trends from morbidity and mortality reports, and poignant stories of patient outcomes [outcomes , 35. ExpForm a powerful guiding coalition, emphasizes that sustainable change requires a strong, thoughtfully composed team [Kotter’s second step, sed team . It is esed team , 37. Thised team . An orgased team . Our anaCreate a vision [After program goals and scenario objectives are established, the third step is a vision . This sta vision . All auta vision , and expCommunicate the vision, Kotter advocates for key elements to conveying the message, including simplicity, repetition, and use of multiple forums [In e forums . One aspEmpower others to act on the vision often involves removing structural barriers and providing training as needed to accomplish change [h change . Our shaPlan for and create short-term wins is another essential step for successful ISS program implementation. Delivering evidence that culture change efforts yield positive results rewards early adopters, encourages further participation, and motivates managers to support program continuation [inuation . Part ofConsolidate improvements to produce still more change. During this phase, the coalition uses momentum from short-term wins to address more challenging organizational issues [The next adapted step in longitudinal prebriefing is l issues . Gains fInstitutionalize new approaches [The final step proaches  involvesOur author group’s insider status impacted the understanding of context, organizational processes, and relationships to develop the Kotter simulation-specific adaptations presented. As an example of the importance of insider insights into context, the initiation of program A was catalyzed by the need to hardwire protocols to address obstetric emergencies, identified by a series of unfavorable clinical events. This “insider” perspective on the impetus behind program A’s development allowed for a nuanced understanding of the context under which external and internal organizational priorities created urgency, details that might otherwise be missed by an outsider trying to gain understanding of those priorities (and how they evolved) retrospectively.Program B is an example of how insiders’ understanding of organizational processes may provide an advantage over traditional “outsider” research. The authors involved with this program had deep familiarity with the organization and its processes of cardiac arrest response, since they practiced within the organization. This afforded them a practical understanding of the various code team compositions and the clinical governance structures surrounding code team responses. This familiarity ensured simulations could be tailored to specific unit and code team requirements and facilitated rapid authorization of process improvements suggested following simulation activities.While reflecting on the implementation of the cardiac arrest ISS program, this “insider” perspective on organizational processes allowed clear elucidation of what processes were no longer adequate, where “business-as-usual” needed change, and the steps required to hardwire that change.As “insiders,” the authors also have a unique understanding of both healthcare team’s structures and relationships. Preexisting relationships that simulationists had with nursing educators, residency coordinators, and hospital leadership accelerated the implementation of the program C. Nursing and physician educators expressed discomfort with conducting the simulations and agreed to take a simulation instructor course based on positive relationships with the simulation leads. These courses increased the NICU educators’ familiarity with simulation and built trust to within the ISS coalition. Based on negative previous experiences or unfamiliarity with the equipment, some nurses and pediatric residents working in the NICU expressed reluctance to participate in the simulations. The simulation leads leveraged the trust with the NICU nurse and residency educators to offer mini briefing sessions to familiarize participants with the equipment and define expectations. This positive exposure to simulation decreased the reluctance to participate and built trust with the ISS program leads.The conclusions and recommendations presented here are based on both analyzing our own initiatives as insiders  and analyzing each others’ initiatives as outsiders . We appreciate that there are threats to the trustworthiness of our conclusions. As seen through the lens of outsider research, S. B., S. J., S. E., and K. B.’s insider status, and particularly their personal investment in the success of their programs, and lack of blinding to outcomes introduce crucial biases. While these threats to our interpretations of our practice-based data exist, rather than seeing them as defects, we argue that they are assets. The inside researchers’ investment in change and richly contextualized understanding where it succeeded and failed complements outsider research by highlighting the lived experience and dilemmas of taking action in real-life contexts.Historically, simulation centers and programs were built on the old adage of “build it and they will come,” but such advice can be detrimental for ISS programs. Our insider research clearly highlights that “how you build it” matters, and time should be spent “playing the long game” in considering change leadership principles for successful ISS implementation. The longitudinal prebrief provides a road map for successful ISS organizational integration. The generalizability of the change leadership framework we propose here is limited by location of the initiatives: North America and Australia. It would be useful to explore how and if change leadership in simulation would be different in more hierarchical or collectivist cultures .Beyond application to the three programs described, the authors have since utilized “The Longitudinal Prebrief” to implement other ISS programs in different contexts and locations. All authors have provided support for other teams initiating ISS programs and activities across geographically dispersed health facility locations by providing guidelines and mentorship that included Kotter’s leadership steps adapted for ISS. Authors S. B. and S. J. utilized this model to coach an ISS program implementation in a smaller regional facility. A key factor to this process was gaining organizational motivation by first spending time understanding the specific clinical and workforce need and connecting the ISS program as a solution to the identified needs. Similarly, author J. R. utilizes the guiding principle “no urgency-no program” when consulting with groups that wish to utilize ISS. Author S. E. developed a policy of ensuring all clinical educators involved in ISS were part of the guiding coalition by having them assist with program design and complete simulation instructor training. Author K. B. strengthened institutional buy-in by co-developing “no-go considerations” for ISS activities that empowered unit leaders and published the framework to provide guidance for others . ComponeWhile not specifically naming Kotter’s change leadership principles, many ISS program descriptions report elements of the process we describe in successful implementation of programs for interprofessional health teams. Wheeler et al. described the successful implementation of an ISS program for the deteriorating patient in a pediatric hospital . Change Failing to “play the long game” in simulation program initiation is the norm, rather than the exception. Focusing on short-term gains, putting out fires, and focusing on the urgent at the expense of the important set simulation program leaders up frustration and even burnout. We therefore have differentiated “prebriefing” that is designed primarily as a process temporally proximate to an upcoming simulation session and designed to establish an engaging learning environment for current learners, from a longitudinal briefing process that focuses on building program legitimacy via connections with the politics and priorities of the larger organization. To succeed in the long game, a more comprehensive approach is required to engage colleagues at all levels and ensure that organizations can implement and sustain simulation programs . This loWe modified a well-known organizational change leadership framework to clarify the specifics of organizational engagement needed for successful implementation of ISS. We cataloged and thematically analyzed our program descriptions to adapt Kotter’s framework and provided recommendations for each step. While our approach is limited by the benefits and biases of describing our own experiences, we believe the framework could provide a structured approach for others implementing ISS programs. To test the soundness of this approach in other contexts, we hope that additional examples of activities relevant to each step (and their relevant success or failure) might be reported by other authors to build collective knowledge for best practice ISS implementation. We encourage those who are developing new ISS programs or expanding current programs to experiment with a “longitudinal prebriefing” to their program planning and implementation.Additional file 1. Supplement 1. Descriptions of 3 different in situ simulation programs and examples of longitudinal prebriefing related to each change leadership step.Additional file 1. Supplement 2. Phases of thematic analysis.


The rural areas of southwestern Indiana are challenged with insufficient healthcare workers in the long-term care industry. The University of Southern Indiana (USI) Geriatric Workforce Enhancement Program (GWEP) is collaborating with state agencies to address shortages in licensed administrators, activity professionals and social service designees. These roles, vital in the nursing facility, hold educational requirements through the Indiana State Department of Health (ISDH) and the Indiana Professional Licensing Agency (IPLA). USI gerontology faculty addressed workforce challenges in five phases: 1) Evaluation of state educational requirement and comparison to current USI gerontology courses; 2) Development of new coursework to support state requirements; 3) Expansion of preceptor pool; 4) Proposal to state agencies; 5) USI curriculum modifications. Faculty evaluated the educational and training requirements for these roles and compared the requirements with current courses. Preceptors were needed for students’ field experiences; a preceptor pool was developed through funded training. The requirements and course alignment were included in the program proposal to the state agencies. Upon ISDH, IPLA and University New Program Development Committee approval, additional instructors were recruited. Results of this initiative include a: 1) Fully approved and launched Administrator-in-Training (AIT) residency embedded in curriculum at undergraduate and graduate level; 2) Fully approved Activity Professional Certificate Program; 3) Pool of 10 certified preceptors; and 4) Onset of the social service designee certificate program. This collaborative partnership developed between USI, ISDH and IPLA ensure state agency requirements are met academically for students entering the long-term care workforce.


Between October 2020 and March 2022, FDA’s Center for Drug Evaluation and Research (CDER) completed two pilot programs to assess the quality management maturity (QMM) of drug manufacturing establishments. Mature quality systems promote proactive detection of vulnerabilities, prevent problems before they occur, and foster a culture that rewards process and system improvements. A CDER QMM program may help to advance supply chain resiliency and robustness and mitigate drug shortages. One pilot program evaluated seven establishments located within the U.S. that produce finished dosage form products marketed in the U.S. A second pilot program evaluated eight establishments located outside the U.S. that produce active pharmaceutical ingredients used in drug products marketed in the U.S. The execution of these pilot programs afforded FDA the opportunity to learn important lessons about the establishment QMM assessment process, scoring approach, assessor behaviors, and perceptions of the assessment questions, reports, and ratings. Many of the participating establishments reported that the QMM pilot assessments helped to identify their strengths, weaknesses, and new areas for improvement which they had not previously identified through internal audits or CGMP inspections. There has been a great deal of interest in the outcomes of CDER’s QMM pilot programs and this paper describes, for the first time, the lessons CDER learned and will continue to heed in the development of a QMM program.The online version contains supplementary material available at 10.1208/s12248-022-00777-z. Current Good Manufacturing Practice (CGMP) requirements establish systems that assure proper design, monitoring, and control of manufacturing processes and facilities, including effective quality systems. Manufacturers can exceed these standards by implementing advanced quality management practices that promote sustainable compliance and enable reliable supply chains. Quality management practices mature as companies expand their focus from meeting the standard of compliance with CGMP to a continual emphasis on proactive process and system improvements. Advanced quality systems increase proactive detection of vulnerabilities, prevent problems before they occur, and foster a culture that rewards process and system improvements. Drug manufacturersInvesting in mature quality management practices reduces the likelihood of disruptions or issues that lead to losses associated with poor quality and can lead to higher revenues, greater customer satisfaction, and operational efficiencies , 2. OverThe United States (U.S.) Food and Drug Administration’s (FDA’s) Center for Drug Evaluation and Research (CDER) is developing a program that includes a rating system to incentivize drug manufacturers to improve QMM at their establishments. This program aims to encourage establishments to invest in a culture of quality and promote the adoption of advanced manufacturing technologies and better tools to measure overall performance. QMM assessments are not regulatory inspections and will not be used to evaluate compliance with CGMP. Rather, these assessments will be used to determine how well an establishment leverages product and process understanding at all stages of the product lifecycle, uses advanced analytics, and engages staff at all levels of the organization to drive data-driven, timely, risk-based decisions to achieve quality and business objectives.Quality Management Maturity: Essential for Stable U.S. Supply Chains of Quality Pharmaceuticals  used in drug products marketed in the U.S. The first pilot program evaluated seven establishments located within the U.S. that produce finished dosage form (FDF) products. The second pilot program evaluated eight establishments located outside the U.S. that produce APIs. Establishments were eligible to participate if their surveillance inspections within 5 years were classified as no action indicated (NAI) or voluntary action indicated (VAI). Participating establishments included manufacturers of prescription drug products, over-the-counter drug products, APIs, and intermediates used to make APIs.CDER conducted these pilot programs between October 2020 and March 2022 using two contractors who were provided with a non-exhaustive list of topics to cover including supply chain management, inventory management, risk management, quality culture, and customer experience. The contractors were tasked with developing assessment protocol questions, a rubric to rate each question on an ordinal scale, and an overall QMM rating for each establishment. Contractors were selected based on their capability to recognize best practices, distinguish maturity levels using objective indicators to minimize inter-rater variability, and identify continual improvement opportunities.The pilot program’s QMM assessments were intended to determine: (i) the level of integration of the quality system and quality objectives with business and manufacturing operations at an establishment, (ii) the agility of an establishment in responding to unexpected changes , and (iii) the resilience of an establishment’s business and production processes.FDA staff served as pilot program observers to listen and learn from the pilot process. FDA routinely met with the contractors to reinforce the expectations for the QMM program and provided feedback to allow the contractors to fine-tune their assessment approaches. After the pilot programs concluded, both contractors gave FDA a report detailing their findings and recommendations for a future QMM program. CDER issued follow-up surveys to each of the participating establishments for feedback on the program, assessment tools and methods, and the utility of the reports. While there was no direct fee to participate in the pilot programs, follow-up surveys revealed that participants estimated their total time commitment was approximately 100 hours, though the amount of time and effort for a QMM assessment varied considerably depending on the number of staff participating in the assessment meetings.The QMM assessment approaches used by the two contractors were different. The use of a different contractor with a distinct strategy for each pilot increased the number of ideas generated and the potential for learning lessons related to conducting assessments. The use of different approaches limited direct comparison between the outcomes of the two pilot programs, but it did provide CDER the opportunity to analyze and implement the best aspects of each. Each assessment was conducted by at least two assessors provided by the contractor. Due to the ongoing COVID-19 public health emergency, all pilot assessments were conducted virtually.Leadership and governanceContinual improvementStakeholder engagement and satisfactionKnowledge managementWorkforce engagementOperationsIn the FDF pilot, the assessment was performed in two stages. During the first stage, participating establishments received a self-assessment protocol with 24 questions. The participants rated their own establishments for each question using a rubric that defined five maturity levels. The second stage of the assessment was a facilitated discussion that focused on practice areas needing further clarification. These discussions provided an opportunity for the assessors to engage directly with establishment personnel and make requests for additional information and supporting documentation. The FDF pilot covered six practice areas:SustainabilityRisk managementexceeding regulatory requirements and adopting best practices)Compliance  in favor of establishing a CDER QMM program.will not evaluate product quality. Rather, QMM assessments evaluate an establishment’s culture, behaviors, attitudes, and enhanced quality management practices. Establishments with high QMM are more likely to produce a robust supply and experience fewer quality-related disruptions.Some stakeholders have voiced confusion about the nomenclature of the program and the ratings. For example, some stakeholders have confused FDA’s Quality Metrics (QM) Program with QMM. The Quality Metrics Program is separate from the QMM pilot program. Quality Metrics are an objective way to measure, evaluate, and monitor the product and process lifecycle . Some haStakeholders have inquired about how long a QMM rating will last. FDA is determining whether other quality indicators of an establishment’s operations  will be used in addition to the QMM assessment to support confidence in the rating . These fThere has been much interest surrounding the potential incentives that might promote voluntary participation in a future CDER QMM program. Separate from the pilot programs, CDER’s 2-day QMM Workshop provided opportunities for stakeholder engagement. An audience poll of over 200 stakeholders at the 2022 CDER QMM Workshop asked, “What would the biggest potential benefit be for sites that participate in a QMM program?” Fig. . RespondExecuting two QMM pilots provided FDA with insights into how to design and implement a future QMM assessment protocol. The program will continue to be developed with stakeholder input. CDER is committed to providing stakeholders with additional information on the QMM program as it develops, including the implementation timeline, how ratings will be determined, and the measures of program success that track year-to-year progress on a continual improvement journey.A QMM program may enhance supply chain resiliency and robustness and mitigate drug shortages. All participating establishments should benefit from the identification of targeted areas for continual improvement. The overall supply chain should also benefit as manufacturers with higher QMM focus on continual improvement and are therefore more likely to embrace advanced manufacturing technologies that enhance the capability, robustness, and commitment to quality by the pharmaceutical industry . The potESM 1(DOCX 15.9 kb)


Grocery stores were the most common setting of use (59%); 29% used at farmers’ markets; 11% used in both settings. User characteristics varied by store setting based on demographics, program experience, fruit and vegetable purchasing and consumption patterns, and social dynamics related to use. Our findings support comprehensive awareness-raising efforts and tailored implementation of incentive programming that attends to diverse segments of SNAP consumers to promote equity in program reach.Structural barriers, such as food costs, reduce access to healthy foods for populations with limited income, including those benefitting from the Supplemental Nutrition Assistance Program (SNAP). Nutrition incentive programs seek to address this barrier. Evaluations of SNAP-based incentive programming often focus on one setting . We examined use patterns, characteristics, and preferences among 253 SNAP consumers with access to incentive programming at both a farmers’ market and a grocery store located within five miles of their home. Cross-sectional survey data were collected in 2019 in two Ohio cities. Despite geographic access, 45% of those surveyed were not using the incentive program; most non-users (80.5%) were unaware of the program. Program users compared to non-users had higher household incomes ( Food insecurity is a persistent and pressing public health inequity in the U.S., with 10.5% of households affected nationally in 2019; trends did not improve throughout 2020 during the COVID-19 pandemic when food assistance benefits were increased . Even wiThe Food Insecurity Nutrition Incentive (FINI) Program, now known as the Gus Schumacher Nutrition Incentive Program (GusNIP), was established in the U.S. in the 2014 Farm Bill to provide financial incentives to SNAP recipients for purchasing fruits and vegetables . EvaluatOhio is a rich context for examining implementation of SNAP-based nutrition incentive programming. Ohio’s nutrition incentive program, Produce Perks, has evolved over the past decade from single-site operations at farmers’ markets into a statewide, networked, multi-food retail setting program . Like thMost extant research is focused on the potential impacts of incentive program implementation at either farmers’ markets ,19,20,21Our study was conducted by a collaborative team that included an academic research center and a statewide non-profit organization that supports implementation of Ohio’s nutrition incentive program. We performed a cross-sectional survey from July–October 2019 in two cities in Ohio: Akron and Cleveland. In 2019, the populations of Akron and Cleveland were about 198,000 and 385,000, respectively, and both cities resided in counties that had over 13% of persons living in poverty . These cWe conducted participant recruitment by partnering with the statewide non-profit organization that manages nutrition incentive programming and the statewide social services agency that operates SNAP benefits. The research team worked with these partners to distribute two targeted mailers to SNAP recipients within the study boundaries in both cities. The mailers sent by the county-based social service agencies included: (1) a promotional flyer about Produce Perks with the location and details of a grocery store and year-round farmers’ market located near their home, information about how to use the incentive program, and resources to find more information about Produce Perks; and (2) a study recruitment flyer listing basic eligibility criteria and information for the SNAP recipient to contact the research team if interested in learning more about the study. Additional recruitment efforts included in-person tabling at all four promoted farmers’ market and grocery store sites throughout the data collection period and distribution of recruitment flyers to local organizations, community events, apartment buildings, and food pantries.We used a six-item screener to determine eligibility for this cross-sectional survey. Screener questions were asked by a trained research assistant in person or over the phone. Individuals were eligible if they were: (1) currently receiving SNAP benefits; (2) residing within the defined study boundaries; (3) responsible for at least half of the food shopping for their household; (4) ≥18 years of age; (5) English-speaking; and (6) the only member of their household enrolled in the study. To determine whether a potential participant resided within the defined study boundaries, they were asked to provide their home address, which was then entered into ArcGIS to confirm that it was within the study boundaries. If a participant did not want to provide an address or the address provided was outside of the study boundaries, they were not eligible for the study. Those who were eligible were offered more information on the study, as well as an opportunity to provide consent. We screened 412 people for eligibility, of whom 145 (35.2%) were screened at the in-person events and 267 (64.8%) were screened by phone . Most ofWe developed a structured survey guide for this study . Variables that differed by Produce Perks use status were identified by the Wilcoxon–Mann–Whitney test, the chi-squared test, or Fisher’s exact test as appropriate. Variables that differed by location of use were identified by the Kruskal–Wallis test or Fisher’s exact test as appropriate. The non-parametric tests  were used when the assumptions needed for parametric tests were unreasonable to make . Statistical significance was set at ositives . The varositives , as descA total of 253 SNAP recipients completed the cross-sectional survey . Most paWhen asked why they did not use Produce Perks, 80.5% of the non-users reported that they did not know the incentive program existed. Less frequently cited reasons for not using the program included: confusion about how the program works (8.8%), inconvenient store location (6.2%), transportation hurdles (3.5%), and high food prices at participating stores (2.7%). No participants indicated that their lack of incentive program use was due to a distaste for fruits and vegetables or that their store lacked the produce they wanted. Among the 90 people who did not know about the program until they participated in the study, 76.7% said that they were very likely to use the program in the next 6 months.The 140 users were asked what motivated them to use the Produce Perks program. The most common motivations for using the incentive program included: the program helps you save money (74%), it allows you to buy more fruits and vegetables (61%), and it is easy to use (14%).p < 0.1 (p < 0.001). Produce Perks users supported more people with their SNAP (p = 0.005) and received more monthly SNAP benefits (p = 0.039). With respect to spatial temporal factors, users lived closer to their preferred store (p = 0.020) and perceived purchasing fruits and vegetables as more convenient (p = 0.005) than non-users . Produce Perks users were also more likely to have purchased or received food from a farmers’ market (p = 0.034), specialty store (p = 0.038), or mobile pantry/free fruit and vegetable drop-off (p = 0.044) in the past year. Additionally, Produce Perks users were more likely to have a car for shopping (p = 0.051) and were less likely to use somebody else’s car for shopping (p = 0.048). Regarding social factors, users reported that more of their friends or family had used Produce Perks (p < 0.001) and had shopped at a farmers’ market (p = 0.054) in the last 6 months. In terms of personal factors, Produce Perks users had more children in their household (p = 0.034). though the median number of children across both groups was less than one. They were also less likely to have moved in the last 12 months (p = 0.050) and had higher educational attainment (p = 0.061). Finally, when food security was analyzed as a dichotomous variable , Produce Perks users were less likely to report experience of food insecurity (p = 0.044) in the past year. Overall, the strongest statistical findings and likely predictors of Produce Perks use were having a higher income and knowing more people who use the incentive program.Multiple variables across four of the five domains of access differed between Produce Perks users and non-users at p < 0.1 : economip < 0.1 (p < 0.001) and had the lowest education levels (p = 0.025). This group reported purchasing fruits and vegetables less frequently (p < 0.001) and they had less confidence in their ability to prepare fresh greens (p = 0.008) as well as root vegetables (p = 0.018). Additionally, grocery store-only Produce Perks users made more frequent trips to their primary store (p = 0.009), knew of more friends and family shopping at the same grocery store (p = 0.054), and had fewer adults in their household (p = 0.067). They also reported that they ate beans (p = 0.062) and other vegetables  less frequently (p = 0.007). Finally, the grocery store-only users reported a similar variety of fruits and vegetables at their primary store compared to the farmers’ market, while the two groups who utilized farmers’ markets reported a greater variety at their farmers’ markets compared to their primary store (p < 0.001). Those using Produce Perks at farmers’ markets only (n = 41) were older (p = 0.033), had used Produce Perks for the longest amount of time (p < 0.001), and were also less likely to be Black (p = 0.056). The group who used Produce Perks at both retail stores and farmers’ markets was the smallest group (n = 15). Users of Produce Perks at both grocery stores and farmers’ markets were redeeming the largest amounts of Produce Perks incentive money (p = 0.030). This group was the youngest (p = 0.038), most educated (p = 0.025), and had the highest levels of confidence in preparing fresh green vegetables (p = 0.008). There were no significant differences in food security across the three user types, when specifying food security as a dichotomous variable (p = 0.703).When Produce Perks users were compared by their setting of use, several variables differed between the three Produce Perks use types at p < 0.1 : groceryThe results of this cross-sectional research add to the field by offering a holistic view of nutrition incentive use and non-use among people receiving SNAP with access to programming in both farmers’ market and grocery store settings near their residence. Our results offer insights that could be used to expand the dietary and food security benefits of nutrition incentive programming. Three key lessons learned from our research include: (1) insufficient awareness of nutrition incentive programming was a critical yet modifiable impediment to its use; (2) multi-setting approaches are needed to engage diverse audiences of SNAP recipients in nutrition incentive programming; and (3) synergistic strategies are needed to reach those who may benefit the most from nutrition incentives.Despite having geographic access, 44.6% of the participants reported that they had not used the SNAP-based nutrition incentive program in the past year. The overwhelming response as to why SNAP recipients had not tried the program was because they did not know it existed. However, when informed about the incentive program, most participants indicated that they were very likely to use it in the next six months. These findings highlight the importance of conducting dissemination research to explore how best to promote SNAP-based nutrition incentive programming to reach diverse groups by testing the impact of different messengers , channels of communications , and messages. Our findings provide evidence for dissemination efforts that leverage social networks to raise awareness about SNAP-based incentive programs, an approach that has been effective in other research .Additionally, our findings reinforce the importance of implementing SNAP-based incentive programs in multiple settings within a community to align with different food shopping routines of SNAP consumers. Much of the prior research on SNAP-based incentive programming focused on farmers’ market settings demonstrating several benefits, such as improvements in spending flexibility among households, fruit and vegetable purchasing and consumption, and opportunities for social connectivity ,19,20,21Lastly, we found several segments of SNAP participants who may benefit from increased use of nutrition incentives to achieve programmatic goals of reducing disparities. The first segment included those experiencing food insecurity and other financial constraints related to low income. Despite having a high need to stretch food budgets, we found that the SNAP-based nutrition incentives were not widely used by this group. While lack of awareness about the program is a fundamental problem, these households may also have broader structural constraints or stressors influencing food purchasing that are not surmounted by the monetary value of a nutrition incentive. The second segment included those less confident about purchasing and preparing and those who less regularly consume fruits and vegetables. We found this segment was more likely to use incentives at grocery stores compared to farmers’ markets. The perceived value of the incentive program may be less for households not already integrating fruits and vegetables into their food routines. Reaching these two segments may require novel and synergistic strategies for program implementation and evaluation. Integrating SNAP-based incentive program implementation within broader efforts to reduce food insecurity, such as workforce development programming to promote job security or housing programs to promote stability, should be explored. In addition, findings point to the need for strategies that link culturally relevant nutrition education, skill building, and cooking classes with SNAP-based incentive program implementation to better reach those contemplating integration of fruits and vegetables into their food routines.Our study has some limitations. The sample may not be reflective of SNAP consumers in non-urban settings, SNAP customers in other urban settings, those living in communities without any nutrition incentive programming, and non-English speaking populations. Additionally, the sample was mostly comprised of older women, whose perspectives may not reflect those of younger groups or men. The sample size was modest and thus reduced statistical power to rigorously compare some sub-groups of nutrition incentive program users. Findings were based on self-reports that could be subject to recall bias. We note that our study was designed to highlight the factors shaping nutrition incentive use choices, and we therefore restricted our attention to areas where a choice existed. As such, our study was designed to tease out determinants of choice rather than generate a representative assessment of SNAP-based incentive program use in the two cities included in the study. Finally, we used a cross-sectional study design, and while the reported associations generate critical hypotheses, they do not confirm causal relationships.This study contributes key insights into the factors influencing use of SNAP-based nutrition incentive programs in urban contexts. Nutrition incentives represent a valuable resource to reduce cost barriers related to the purchasing of fruits and vegetables, yet our findings highlight that more efforts are needed to effectively raise awareness about these programs in communities. Overall, our research sheds light on the need for tailored implementation of incentive programming in diverse food retail settings that attends to different segments of SNAP consumers to promote equity in program reach and impact.