As the most common diseases globally, oral and dental diseases are closely related to people's behavior. The present study is aimed at developing a program to prevent primary teeth decay in children using the intervention mapping approach.  This study protocol is aimed at using the intervention mapping approach as the planning framework. The study consists of six steps of intervention mapping, including needs assessment based on the relevant literature review, development of an objectives matrix for changing people's behaviors and environmental factors, program preparation, program implementation, and program evaluation to develop a family-centered program.  The use of the intervention mapping approach helped us identify the outcomes and functional objectives, develop an appropriate intervention program, and evaluate the program.  The intervention mapping approach is an appropriate guide to developing a systematic and evidence-based program. Oral health plays a key role in the overall health of the body , 2. The Tooth decay is a global disease, and few populations are immune to its effects . It affeFamilies are advised to supervise their children's toothbrushing with fluoride toothpaste for up to 7 years to prevent tooth decay in the UK. 31% of five-year-old children have dental caries in this country, increasing to more than 41% in deprived areas .In this regard, communities with no prevention program have poor healthcare systems and/or are at a low level of socioeconomic status, general health, and standard of living . The valParents are advised to supervise their children's toothbrushing process from about six months to 7 years of age; this can lead to a 15% reduction in tooth decay , 14.13]13] Most According to some studies, fear, anxiety, psychological problems and low self-efficacy, parental education levels, number of children, family economic status, and awareness of the importance of toothbrushing can affect oral health behaviors –18.Some studies have focused on behavioral change at the individual level, but toothbrushing in children under the supervision of parents is an interpersonal behavior influenced by a wide range of factors , 20. GivThe present study was conducted to develop a family-centered program using the intervention mapping approach. As described by Bartholomew et al., the intervention mapping protocol provides a framework for developing, implementing, and evaluating health promotion programs  and is kThe protocol has six steps. The first step pertains to needs assessment, the second to fourth steps include the initial development of the intervention, the fifth step consists of planning for implementation, and the sixth step includes the evaluation and modification of the intervention. Activities in each mapping step will guide the following step , 26.The intervention mapping approach defines the pathway of identifying the needs or the health problems that determine the solution .In the first step, before starting to plan the program, the problem of health, quality of life, and the behavioral and environmental causes of the problem, and their determinants were addressed using the preceding model . This stConsequently, individual discussion and consultation sessions were held for the working mothers. The results obtained from the review of the relevant literature indicated that the individual (the mothers) and the environmental  levels were the program's most important ecological levels. The preliminary study of the needs assessment group on the causes of health problems showed that some factors such as fear and anxiety, psychological problems and low self-efficacy levels of the individuals, fatigue, laziness and impatience, depression, lack of time and high cost of living, parental education level, mother's occupation, birth order, number of children, family economic status, awareness, attitude, and health literacy of the mothers affected oral health behaviors , 28–31.Mothers and environmental behaviors were categorized into 8 and 3 behaviors, respectively. In addition, the cognitive determinants related to individual and environmental behaviors such as awareness, attitude, perceived severity, perceived benefits and perceived barriers, cues to action, perceived self-efficacy, behavioral intention, subjective norms, and social norms were also identified Tables  and 2.The second step of intervention mapping pertains to creating a change objectives matrix as the main tool of intervention mapping. This step examines the importance and possibility of behavior changeability .Functional objectives are specified in this step. These objectives guide the health intervention program and are expected effects of the health intervention program. The most important part of the functional objectives is what the patients need to do to turn the desired behavior into a habit.The change objectives in the intervention program reflect the changes needed in sociocognitive determinants for achieving the functional objectives. Therefore, each functional objective may include several change objectives .In other words, the second step is aimed at focusing on changing individual behavior and the environmental actors' behavior. The draft of the functional objectives based on the theory of planned behavior  was first drawn up by the workgroup preparing the matrix based on the individual and interpersonal outcomes of the program. Then, a 10-member group of pediatricians, dentists, and health educators was asked to validate the draft. After receiving their views, the functional objectives were revised. Finally, after writing the change objectives and creating individual and environmental level matrices, the logical pattern of program change was completed Tables .The third step of intervention mapping included the creation of program ideas, modification methods, and practical applications . In thisThe program was developed in the fourth step of the intervention mapping. Steps 1 to 3 completed the fourth step. The end product of this step was a comprehensive plan for the intervention program. This plan helped the program designers develop creative materials for supporting health education and health promotion programs. In this step, the pretest was carried out, the practical applications were experimentally implemented, and program materials were developed with the participation of the recipients and the implementers. In this task, the planning team and the design workgroup focused on the cultural relevance of the program since planning that lacks a cultural basis will jeopardize the project. In this step, attention was paid to all the cultural aspects, whether superstructure or infrastructure, in the community. They were accurately identified, and attempts were made to utilize them in designing the program positively. Since being continuously in touch with the participants is a method to enhance adaptation of the program to the culture, close attention was paid to interpersonal channels.Several ideas must be considered to find a good one .Brainstorming is the most appropriate way of getting ideas from the group members. The program's field, sequence, and components, which constitute its units, were then determined to provide 12 hours of training in 4 weeks in the family-centered program.Due to the role of face-to-face training and its advantage over delivering lectures, and in order to attract the participation of the mothers faced with lack of time (due to having children under the age of one), 2 groups of 15 members were formed in this program, and 12 hours of training were divided between the two groups. The training was then presented to them in a completely identical manner. In addition, the local language was used in the training sessions to make the program more effective.In the family-centered program, the graphic designer participated and the printing house and the available films in the Department of Oral and Dental Health in the Ministry of Health and Medical Education were used.The main documents used in this study included educational pamphlets (to increase awareness in and provide action guide for the mothers), educational clips (to increase the perceived intensity of the mothers), replicas (to increase the mothers' awareness), lectures and group discussions , and modeling (to increase perceived self-efficacy). The initial design of the pamphlet was prepared by the research team and finalized by the graphic designer and then the printing house, and the educational films on oral health behaviors were prepared by the Department of Oral and Dental Health which were shown to the mothers. The replicas in the Dental Department of Ilam University of Medical Sciences were also used. In addition, four phone calls per month were made to the mothers to augment the action guides for them .The program's implementation plan was the fifth step of the intervention mapping. In this step, the potential implementers of the program were identified, the outcomes and the functional objectives of the program were stated, the matrices of change objectives were created to be used in the program, and the implementation interventions were designed. In this step, two 2-hour meetings were held between the planner, the health educators, and the head of the Comprehensive Urban Health Center No. 6 to review program adoption, implementation, and sustainability to coordinate their actions as much as possible.The activities during this step provided a basis for evaluating the efficiency and effectiveness of the study so that the matrices provided in Step 2 were used to develop evaluation scales for effectiveness the matrices developed in Step 5 to develop the questions and the process evaluation scales.This research assessed the determinants of oral health behaviors in the mothers for program evaluation.The program's effectiveness was evaluated using a randomized controlled trial (RCT) (IRCT20190716044227N1). This study was conducted from May to December 2019 with the approval of the Ethics Committee at the Faculty of Medical Sciences in Tarbiat Modares University with the ethics code IR.MODARES.REC.1398.02.According to previous studies, with type I error of 5%, and the power of the study of 95%, the sample size was calculated to be 60 subjects. To this end, 60 mothers with children aged six months to one year were randomly selected among those visiting the Comprehensive Urban Health Center No. 6 in Ilam City (western Iran). They were then randomly divided into the two 30-member intervention and control controls.The inclusion criteria were having a healthy child of 6 months to one year and a health record and being literate. Women who were pregnant women or had an underlying disease were excluded from the research.Ilam City is divided into Districts 1 and 2 according to city divisions. District 1 has 9 Comprehensive Health Centers and 3 Urban Health Bases, and District 2 has 2 Comprehensive Health Centers and 4 Urban Health Bases. The total population of District 1 is 110,099, of which 55,141 are female. The Comprehensive Urban Health Center No. 6 included the largest population among the urban centers and, as the only educational center in the province, had suitable educational space. Therefore, it was selected for implementing the training program. In this stage, the mothers were selected using simple random sampling from those invited to participate in the study at the Comprehensive Urban Health Center No. 6. The participants were then randomly allocated to the intervention and control groups. The data were collected using a written questionnaire at baseline and 6 months after the end of the program.t-tests were used to evaluate the program's effectiveness. The chi-squared test evaluated the two groups in terms of the underlying variables and the studied determinants.The collected data were analyzed in SPSS 12. The independent and the paired-sample t-test regarding the study determinants before implementing the training program. No statistically significant differences were found between the two groups in this respect either. Before implementing the training program, the intervention and control groups were examined for demographic variables. No statistically significant differences were observed between them in this regard . The twoThe present study is aimed at developing and evaluating a program to prevent primary tooth decay in children based on the intervention mapping protocol in Ilam Province located in western Iran. Our literature review showed no published study in Iran on developing oral and dental interventions using IM. This randomized controlled trial showed the usefulness of the intervention program in improving the intention to adopt oral and dental hygiene behaviors in mothers. Our findings showed a significant increase in the mean awareness score after the training intervention. The educational strategy used to improve the awareness level of mothers included lectures, group discussions, educational pamphlets, and the use of replicas to familiarize the participants with the structures of the mouth and teeth. Consistent with the present research results, the findings of various studies have shown an increase in the score for awareness after training interventions –37.The mean score for attitude increased in this research in the intervention group after the training intervention through group discussions and lectures. This result agrees with those of the study by Bourke in Australia .McGowan proposed the balanced decision-making method as the most appropriate technique for improving the attitude of patients with diabetes or cancer. In this method, patients are encouraged to list the advantages and disadvantages of behavior change and make short- and long-term decisions .Another result of the present study was a significant increase in the mean score for perceived self-efficacy in the mothers after implementing the training program. This increase can be attributed to the employed educational strategies as modeling, verbally encouraging performance of minor functions, and expressing successful experiences by the participants were used in this research.Self-efficacy is an important predictor of compliance. People who show the most change usually have higher self-efficacy in performing certain behaviors .The present study's findings show a significant increase in the mean score for perceived benefits, perceived intensity, and cues to action after implementing the training program. The training strategy included making four phone calls to the mothers per month, lectures, and group discussion. Similar studies reported that discussion was the most effective method in increasing perceived intensity .According to the findings, the mean behavioral intention of the mothers increased significantly six months after the end of the program. Modeling was used as the training strategy for improving the intention of adopting oral health behavior. To this end, the mothers of the children with white and healthy teeth were invited to share their experiences with the other mothers.Achterberg et al. used the operational planning method as a suitable theoretical technique for changing the sociocognitive determinant of intention. The operational planning method is aimed at changing behaviors, i.e., when patients intend to make a plan to change their behavior, operational planning focuses on creating new behaviors similar to self-management in patients with chronic health problems. Operational planning can include changing the existing behavior or stopping unwanted behavior .Another finding of the present study was the significant decrease in the mean score for the perceived barriers in the mothers of the experimental group compared to the control group. These changes may indicate the effectiveness of the training program on the mothers' perceived barriers in improving their intention to adopt oral health behaviors.The training strategies used to reduce perceived barriers included lectures and group discussions. The simplicity of toothbrushing and using appropriate methods for breastfeeding infants were stated, the possible barriers to each of these behaviors were discussed, and the opportunity for group discussion was provided. These strategies eventually led to overcoming barriers such as lack of time, fatigue, and boredom.Lack of time was one of the major barriers that were mentioned in relation to observing oral health. Extensive training can minimize this barrier because upgrading lifestyle requires focusing on the quality of life improvement .Some studies have shown that coping planning effectively changes the sociocognitive determinant of barriers. The use of this method requires the determination of high-risk situations. High-risk conditions can stop the patient from changing the behavior to the desired one even if this change has been planned. In addition to determining the high-risk situations, learning to correctly cope with such situations is necessary. Coping planning focuses on the sociocognitive determinants of barriers and barrier control skills and mentions the need to continue behavior change. In practice, those patients change their behavior that encounters barriers to behavior change. Coping planning makes the patient feel slightly uncomfortable in behavior change .Another study result was the high mean score for the mothers' subjective norms in the experimental group compared to the control group. The employed training strategy for improving the subjective norms included using the modeling method, employing group discussion, and evoking the inner feelings and a sense of responsibility in the mothers regarding their children's adoption of oral health behaviors.Investigating the effect of a training program based on the theory of planned behavior on improving oral health behavior in pregnant women visiting urban health bases in Ashkhaneh City in Iran.” The results showed that the training intervention was not effective on the abstract norms of the women. In addition, there was a significant difference between the intervention and control groups in this regard after the training intervention. This finding, which is inconsistent with the present study results, can be attributed to the lack of participation of a large number of spouses in the training sessions, insufficient training for the companions, and low quality of the training intervention [Ebrahimipour et al. conducted a study entitled “rvention .The present study's findings showed a significant increase in the mean score for oral norms related to oral health in the mothers of the experimental group compared to the control group. These changes indicate the effectiveness of the training program in improving the social norms related to oral health in mothers. Using the method of showing educational films was the training strategy used to improve the subjective norms.Factors determining and upgrading oral hygiene behaviors: a study based on the theory of planned behavior,” both groups of the participants in the Caribbean and Nepal emphasized the impact of social norms on adopting oral health behaviors. The Caribbean participants reported no social pressure on them to adopt oral health behaviors. They felt that good control and self-management could effectively develop oral health behaviors [In a comparative study conducted in the Caribbean and Nepal entitled “ehaviors .Coster and Norman acknowledged in their studies that informing patients without any support  was not effective and did not influence their health .This research had some strengths and weaknesses. In relation to its strengths, it was the first clinical trial performed on Kurdish-speaking Iranian mothers to improve oral health behaviors. In addition, considering the children's age, the intervention was performed at the interpersonal and environmental  levels and focused on the mothers. As for its weaknesses, it was probably accompanied by some errors because a questionnaire was used as the research instrument. It was also time-consuming due to the nature of using intervention mapping in designing interventions. As reported by other researchers who used the intervention mapping protocol, this study showed that intervention mapping, while being complex and time-consuming, was flexible and could greatly help improve the program's effectiveness –50. The The present study provided comprehensive information on developing interventions to improve oral health behavior. It also forms the cornerstone for implementing similar programs in different parts of Iran and even in other countries.


The immediate past presidents and current president of the Northern California and Nevada Medical Library Group write to raise awareness of the American Library Association Spectrum Scholarship Program, share their approach to supporting Scholars in their region, and encourage Medical Library Association (MLA) chapters and MLA at large to build stronger infrastructures to support Black, Indigenous, and People of Color librarians who are in school and recently graduated. As the immediate past presidents and current president of the Northern California and Nevada Medical Library Group (NCNMLG), we would like to publicly thank the Medical Library Association (MLA) for honoring our organization with the 2021 Chapter Project of the Year Award for our sponsorship, mentorship, and financial support of American Library Association (ALA) Spectrum Scholarship Program. We would like to take this opportunity to raise awareness of this scholarship program, share our approach to supporting Scholars in our region, and encourage our colleagues, MLA, and the MLA chapters to build stronger infrastructures to support Black, Indigenous, and People of Color (BIPOC) librarians who are either in school or recent graduates.Since 1997, ALA has awarded more than 1,300 Spectrum Scholarships “to American Indian/Alaska Native, Asian, Black/African American, Hispanic/Latino, Middle Eastern and North African, and/or Native Hawaiian/Other Pacific Islander students to assist them with obtaining a graduate degree and leadership positions within the profession and ALA” . The SpeIn 2018, members of NCNMLG, including Spectrum Scholarship Program alumni and previous Spectrum Scholarship jurors, proposed that our chapter provide not only financial support but also professional scaffolding for a Spectrum Scholar in our region. Our executive board then unanimously voted to annually sponsor, in perpetuity, one Northern California or Nevada Spectrum Scholar interested in health sciences librarianship , making In the same spirit as the Spectrum Scholarship framework , NCNMLG There are many ways our sponsored Scholars can engage with our organization and members. In addition to encouraging their attendance and participation at our membership and executive board meetings, NCNMLG invites all Scholars to attend chapter meetings so that they can network with librarians, become familiar with professional medical library conferences, and learn about current health sciences library trends and issues. The chapter also provides the Scholars with multiple mentoring and job shadowing experiences with chapter volunteers, tailored to their professional interests and goals. Costs associated with these opportunities are covered by NCNMLG to reduce barriers to participation. We are currently accommodating Scholars remotely due to the COVID-19 pandemic and the associated library shutdowns, meeting over Zoom and encouraging virtual conference attendance. We continually adjust our support offerings in response to informal feedback from experienced librarians, current and past Spectrum Scholars, and chapter members, as well as established best practices for supporting recently graduated BIPOC librarians. We are currently developing a standardized data collection tool to document and share Scholars' voluntary feedback after they participate in our program. While this type of data collection is strategically important, we recognize we must be mindful not to place an undue burden on Scholars' time and expertise. It is important to note that not all students are interested in the extra activities, nor are all students able to participate in additional offerings beyond financial support. NCNMLG does not require students to take advantage of these engagement opportunities; they are provided to the students with encouragement.Our chapter is proud to have recently reached two milestones regarding our support of the Spectrum Scholarship Program. First, the NCNMLG executive board unanimously voted in 2020 to create a Spectrum Scholar subcommittee to recruit more chapter members in the fundraising, mentoring, and advocacy efforts involved in fully engaging with Scholars. This development will ensure long-term, sustainable administrative assistance so that the Scholars will be provided with an abundance of support while they are in the program. Second, because we are always seeking out opportunities to raise additional funds for the program, the proceeds from the joint 2021 Mountain Pacific Health Science Libraries Conference (MPHSLC), of which NCNMLG was a collaborating chapter, were donated to the Spectrum Scholarship Program and provided funding for an additional designated MPHSLC Scholar for 2021–2022. Coordinating this donation on behalf of MPHSLC further highlighted the need to distribute the work associated with supporting the Spectrum program; relying on a single person to handle the administrative load is unsustainable and tasking a committee with these responsibilities ensures timeliness, consistency, and scalability.Like in most other areas of librarianship, health sciences librarians are overwhelmingly white and do not accurately reflect the many populations and professions with which we frequently work. Winning the 2021 MLA Chapter Project of the Year Award has motivated NCNMLG to think even bigger about these issues of equity and inclusivity and to explore possibilities for future collaborations to fund and support more BIPOC library students who are preparing for careers in health sciences.NCNMLG encourages other MLA chapters to engage with students who live in their region and receive an MLA/NLM-sponsored Spectrum Scholarship or the MLA Scholarship for Underrepresented Students. This support does not need to be financial in nature, although financial support is important. Offering mentorship and job shadowing opportunities, providing scholarship recipients gratis chapter membership, and including the students in chapter meetings are all ways a chapter can provide a welcoming and supportive environment. If there are chapters that already have a regional student scholarship or grant, we urge those chapters to consider also providing those students with network-building and professional opportunities in addition to financial support. An aspirational goal for NCNMLG is to be able to offer more substantive professional opportunities to Scholars, such as paid internships, and we encourage other chapters to think big when it comes to what can be accomplished through their organization.MLA has shown continued commitment to diversity within health sciences librarianship through their longstanding financial support of the Spectrum Scholarship Program and the Scholarship for Underrepresented Students, the creation of the Diversity Task Force in 2017 , and theNCNMLG's support of Spectrum Scholars is just one example of what can be done, and the ideas we offer here are just some options for ways to make impactful, positive change in our organization. Overall, we want to encourage other chapters and professional groups within the MLA community to closely examine the work they are doing to support BIPOC students and librarians to make our profession a more inclusive and equitable one. We recognize that not all chapters may be able to provide financial support and professional development opportunities, but we hope that the work we have done thus far as a chapter can serve as inspiration and a roadmap for those who are able to move forward to address DEI issues in this manner. Looking to the future, NCNMLG plans to explore collaborations with other MLA chapters to sponsor additional Spectrum Scholars and to take advantage of any opportunity to expand on this successful and nationally recognized program. We will continue to incorporate feedback from the Scholars and other stakeholders to ensure that our support remains constructive and rewarding for everyone involved. A specific chapter goal is to provide assistance and collaborative opportunities for Scholars who are interested in sharing their experiences with the broader health sciences library community. We appreciate the iteration, self-reflection, and adaptation required for meaningful DEI efforts and strive to ensure that our chapter practices these critical actions. We hope that our efforts will lead to more engagement with the Spectrum Scholars within our chapter, other chapters, and MLA as a whole.


The COVID-19 pandemic necessitated rapid changes in the delivery of care across public primary care settings in rural Guatemala in 2020. In response, a hypertension program implemented within the public primary care system required multiple adaptations, providing an illustrative example of dynamic implementation amidst changing context in an under-resourced setting. This study describes the evolvability of an evidence-based intervention  and one of its main implementation strategies (team-based collaborative care) during the COVID-19 pandemic and discusses implications for health equity and sustainability.This convergent mixed methods analysis assessed implementation across five Ministry of Health districts during the initial phase of the pandemic. Qualitative and quantitative data were collected, analyzed, and integrated, informed by the RE-AIM  Framework’s extension for sustainability, and its contextual enhancement, PRISM . For RE-AIM, we focused on the “Implementation” domain, operationalizing it qualitatively as continued delivery and adaptations to the EBI and implementation strategy, and quantitatively as the extent of delivery over time. We conducted 18 in-depth interviews with health providers / administrators (n=8) and study staff (n=10) and performed a matrix-based thematic-analysis. Qualitative results informed the selection of quantitative implementation summarized as behavior over time graphs. Quantitative implementation data and illustrative quotes are presented as joint displays.In relation to implementation, several organic adaptations hindered delivery, threatened sustainability, and may have exacerbated health inequities. Planned adaptations enhanced program delivery and may have supported improved equity and sustainability. Salient PRISM factors that influenced implementation included “Organizational perspective of the EBI”, “Fit” and “Implementation and sustainability infrastructure”. Facilitators to continued delivery included the perception that the EBI is beneficial, program champions, and healthcare team organization. Barriers included the perception that the EBI is complicated, competition with other primary care activities, and temporary suspension of services due to COVID-19.Multi-level contextual changes led to numerous adaptations of the EBI and implementation strategy. Systems thinking approaches may shed light on how a program’s sustainability and its equitable delivery are influenced by adaptations over time in response to dynamic, multi-level contextual factors.NCT03504124 To shed light on the nature of successful implementation efforts, implementation researchers have called for an improved understanding of the interplay between interventions, implementation strategies and multi-level context . ImplemeAlong these lines, sustainability, the extent to which an intervention continues to be delivered and continuously delivers its intended benefits over an extended period of time, is an essential focus of implementation science ,3. WithiSince 2018, our team has been studying the effectiveness and implementation of a multi-component hypertension program delivered within the public primary care system in rural Guatemala through a hybrid type 2 cluster randomized control trial (cRCT) . This efWe used a convergent mixed methods design to assess implementation of the multicomponent hypertension program during the initial months of COVID-19 . Data coThis study was approved by the Guatemalan National Ethics Committee, INCAP Institutional Review Board (IRB), Columbia University Irving Medical Campus IRB, and the University of Colorado IRB.”As shown in We conducted this study in five of the 18 intervention health districts (clusters) in the parent study. Guided by field study staff, we selected five health districts in the three Health Areas with the longest experience implementing the program representing a wide range of program implementation (e.g. low and high implementers) To comprehensively assess implementation from different perspectives and levels, we conducted interviews with people who were closely involved in program implementation at the health area administration , municipal health districts  and health posts (community-level). We also included Ministry of Health (MoH) staff responsible for health management and service provision, and INCAP study staff responsible for supporting program implementation and gathering data. Study staff were invited to participate by APA, while MoH staff were invited to participate in the study by research assistants. All participants provided verbal informed consent prior to initiating the qualitative interviews.As shown in the Interviews were conducted by APA and ALG in Spanish; audio recorded over the phone; between July and September 2020; 9–12 months after the program was launched; and 3–5 months after the start of the lockdowns due to the COVID-19 epidemic We collected quantitative data corresponding to key indicators of program implementation. Specifically, we focused on program delivery of the EBI over time at the healthcare facility level , measured as the proportion of observed vs expected number of hypertensive medications per time period. We also assessed the delivery of the implementation strategy over time at the healthcare facility level, measured as the percentage of the observed vs the expected number of team meetings per time period. MoH providers in charge of delivering the program collected quantitative data on paper-based forms that were previously co-designed and piloted by the research team and MoH staff. Every three months, study staff reviewed and entered the paper-based data into a ReDCap electronic database hosted by INCAP ,18.To analyze qualitative data, we followed the Rapid Identification of Themes from Audio Recording (RITA) methodology and the matrix-based approach to thematic-analysis ,20. RITAOur quantitative data analysis focused on the RE-AIM “implementation” domain, operationalized as the extent of program (EBI and implementation strategy) delivery over time. We analyzed facility-level indicators of program delivery to shed light on important metrics of program implementation over time and across districts. After selecting the indicators, we built behavior-over time graphs to explore the level of program delivery across districts and over time, particularly before and during-COVID-19 .Finally, we converged qualitative and quantitative results to see if they reached the same conclusion ,22 . We Implementation of the protocol-based hypertension treatment (EBI) evolved, before and during-COVID-19, as indicated by the availability of hypertensive medications at healthcare facilities . BOT gra.The implementation of team-based collaborative care evolved over time as indicated by the completion of team meetings“Physicians did not buy into it, they did not participate in the program and some staff perceived the program as “additional workload”. Professional nurses ended up taking on responsibilities that had been assigned to physicians.”– Data collector, HD1We asked participants to describe the program and share their perceptions about it. We found that the perspectives of healthcare providers / administrators (i.e. the implementers) appeared to influence program implementation (delivery and adaptations) and the intention to sustain it. Perspectives varied widely between health districts and type of healthcare provider / administrator. As mentioned above, in HD5, the lack of support from leadership delayed the launching of team-based collaborative care meetings ( implementation strategy) and other program elements See . In a di“I think that the MoH can continue [implementing the program], as can help patients and prevent [hypertension] - patients’ relatives are trying to understand how to improve their health without medications, they want to start taking action. Patients go back home and share their experiences with their relatives on how to prevent chronic diseases – diet, salt consumption, weight, exercise, many habits that should be avoided, like eating junk food. This is very important and should be sustained. The MoH should take it on as its own program”– Health district administrator, HD3If healthcare providers and administrators perceived the program as beneficial and feasible, they appeared to support its institutionalization (with implications for sustainment) within the MoH beyond the study period.We asked participants to explain how the characteristics of health districts influenced program implementation (fit), which we categorized into barriers and facilitators related to fit. A major barrier related to fit was competition with other MoH primary care program activities. Two other barriers that we observed were: insufficient and overburdened healthcare staff and the temporary suspension of healthcare services during the initial part of the COVID-19 pandemic. Facilitators related to fit were: previous experience providing services for chronic diseases, the presence of program champions and strong leadership, and the extent of organization, collaboration and communication within the district team. At the time that the interviews were conducted, the number of COVID19 infections were on the rise and providers were increasingly becoming involved in the pandemic response, which presented an overarching challenge for program fit across health districts.“During the initial phase [of COVID-19], services for patients with chronic diseases were suspended, all at once we had 13 COVID cases, we were afraid.” -Professional nurse, HD3“Remember that an auxiliary nurse is responsible for 22 other MoH programs, and now, with this pandemic it’s even worse.” -Professional nurse, health area administration for HD1, 2Across all health districts, MoH providers and administrators described the difficulties of being responsible for delivering 22 other MoH programs at the primary care level, in addition to the hypertension control program. Providers described the tensions between delivering care for acute problems and maternal and child programs as compared to care for chronic conditions, such as hypertension. Historically, maternal and child health programs have been prioritized over chronic diseases. Often, providers had to choose between delivering one program over the other one. For example, a data collector at HD2 described that providers had to miss meetings for the team-based collaborative care in order to respond to acute health problems (e.g. an undernourished child), reducing the delivery of this implementation strategy. Moreover, the COVID19 pandemic response initially led to the suspension of healthcare services for chronic diseases, and later created more responsibilities among the primary care team, reducing program fit for the district.“At HD1, this isn’t a new program. Since 2011 we’ve been following-up with patients with chronic diseases, monthly or annually, as part of the Inclusive Health Model , which is supported by the Institute for Inclusive Health. […] Implementing this program has not been very hard for our district. This new program improved our approach to managing hypertension.” – Professional nurse, HD1 administratorThe program fit was facilitated by the particular health district’s experience providing care for chronic cardiometabolic diseases (e.g. diabetes mellitus 2 and hypertension). MoH administrators responsible for overseeing multiple districts, highlighted that districts with previously existing chronic disease programs had capacities and facilitators to implement the hypertension program. For instance, the healthcare team was already trained in chronic diseases, accustomed to and organized in such a way that allowed them to follow-up with chronically ill patients. In contrast, other healthcare teams were only organized to provide care for acute health problems (see previous theme). Providers within districts with previous experience providing services for chronic diseases (HDs 1–3) agreed with the administrators and indicated that the hypertension program was an improvement over their previous program, rather than a completely new approach to treating hypertension.“The health area administrator  really supports the program and she has made sure that hypertensive medications are available, which motivates patients to participate in the program.” - Data collector, HD2“Other health district directors would say ‘Let’s only implement this program for a short time’, but instead she [referring to the HD medical director] motivates us, and we motivate the auxiliary nurses to deliver the hypertension program. […] Professional nurses trained by the study team trained other professional nurses responsible for other health posts. Now the hypertension program is being implemented within the entire health district.” – Professional nurse, HD3Most participants (MoH and study staff) highlighted that strong leadership and program champions enhanced the program fit to the health districts. Program champions were MoH leaders, such as health area or health district administrators, who strongly supported the hypertension program and mobilized their teams to implement it. They also actively adapted the EBI / implementation strategy and ensured that key infrastructure elements were available, which enhanced the program’s fit to their health district see  below. FWe asked participants to reflect on what would be needed for the program to be implemented and sustained over time. We identified two primary components of the implementation and sustainability infrastructure, which are described below: human resources, and equipment and medications. These and other aspects of the implementation and sustainability infrastructure are presented as resources and processes in “Our main challenge to deliver this program is that we need more human resources.”– Health area nurse, HD1, 2“The program can be implemented as planned, but there needs to be a supervisory team from the MoH in charge of monitoring the program, supervising and training [healthcare providers].”– Research assistant, HD3The need for additional human resources to deliver the program was a recurring theme in all the health districts. Participants identified the need for additional providers to deliver care for chronic diseases like hypertension, in addition to the rest of the MoH primary care programs (e.g. childhood immunizations). Some participants suggested that health districts would need at least three auxiliary nurses per health post, instead of one or two. Moreover, to sustain the program beyond the study period, participants suggested creating the role of a local-level chronic disease program coordinator, who would be in charge of helping patients navigate care by scheduling their health coaching sessions, monitoring the availability of hypertension medication at health facilities, and coordinating between providers based at health centers (e.g. physicians) and those based at the health posts (e.g. auxiliary nurses). Providing ongoing training, supervision and support to auxiliary nurses was also identified as a need.“This program has been possible just because INCAP provided us with blood pressure monitors, weight balances…” – Health area nurse, HD 4, 5Participants identified the need for equipment (e.g. blood pressure monitors) and recognized that prior to the start of the program such equipment was not available or working at most health facilities. Most participants highlighted that ensuring the availability of hypertensive medications had been challenging during program implementation, and particularly during COVID19. In order to sustain the program beyond the study period, hypertensive medications would need to be available, which would require health districts and health areas to improve their medication supply chain.“It is easier with patients who have a relative who is willing to help them, if they know how to read and write, and if they live closer to the health services. Patients without family support may not receive hypertensive medications” -Data collector, HD2“Some female participants required husband authorization to enroll in the program. Some were not authorized and did not enroll in the program” – Data collector, HD1To identify instances of inequitable program implementation and reach, we asked participants to describe differences in program implementation between facilities, communities and individuals. Multiple participants provided examples of differences in program implementation between and within health districts. Those health districts or health facilities which approved the program and had sufficient infrastructure (e.g. human resources) seemed to deliver the program more successfully than their counterparts. Participants identified individual and community characteristics that may have led to inequitable implementation and reach of the program, including poverty, rurality  predominantly Maya-speaking populations, and in communities that had lack of support from community leaders. Participants described individual-level characteristics that may have led to inequitable participation, including working in the agricultural sector, lacking formal education, lacking family support to engage in program activities, and being a female with an authoritative husband.“The program should be expanded beyond the patients who are enrolled, for everyone - this is really important. This is a pilot, which allows us to evaluate the results and make recommendations to the MoH so that they know what to do. But we need to go beyond recommendations, we also need to deliver training workshops, buy equipment and supplies, hire staff designated to this [program]” –Professional nurse, HD4“It is possible to continue delivering the program within the new COVID19 reality. First of all, we need HTN medications. Second, we need training to provide health coaching sessions – we already have that, and we need to strengthen it. Third, we need to continue delivering health coaching sessions, but now as part of patient clubs.” – Health area nurse, HD3We asked participants to reflect on the program’s sustainability beyond the study period, considering the context of COVID-19. While participants identified threats to program sustainability, most of them stated that it was feasible to sustain the program. Health administrators showed interest and commitment to continuing implementation of the program beyond the study period and health providers recognized the importance of doing so. As described above, participants identified the need to meet with essential infrastructure elements for the program to be sustained. Participants also reiterated the need to recognize and integrate the hypertension program within the MoH primary care programs. In addition, for the program to be sustainable, health districts would need to ensure transportation for providers to visit patients in the most mountainous and rural areas. Finally, to continue implementing the program in the context of COVID19, it would need to undergo planned adaptations to improve its feasibility and reach, such as delivering health coaching sessions in a group setting, rather than individually.Our mixed methods analysis of mid-course implementation and context of a new hypertension control program within Guatemala’s public primary system during the initial months of the COVID-19 pandemic led to two major conclusions. First, we documented a close interplay between the rapidly evolving context, which influenced the extent of delivery and adaptations to the EBI and the implementation strategy. Second, we identified sources of unequal program implementation and potential threats to program sustainability as well as opportunities for improving health equity and sustainability.We observed the close interplay between contextual factors, EBIs and implementation strategies pointed out by others . We founSimilar to other implementation assessments conducted during COVID-19, we found that the COVID-19 emergency further stretched an overburdened and under-resourced primary care system , 9, threFollowing calls to utilize a health equity lens in implementation assessments , this stOur study has several strengths. First, we assessed program implementation over time focusing on two different phases (before and during COVID-19), which allowed us to understand how rapidly changing contextual factors led to both the EBI and the implementation strategy’s evolution. Second, we applied widely recognized implementation frameworks, RE-AIM/PRISM, coupled with an explicit focus on health equity and sustainability. Third, we utilized a mixed methods approach, interviewed participants at different levels of the health system , and included a range of health districts, all of which allowed for a deeper understanding of the program’s implementation within the public primary care system in Guatemala. However, our results need to be interpreted in light of certain limitations. First, our analysis only included five of the 18 districts that implemented the program. However, we purposefully selected those with the most implementation experience and those representing high and low implementers. It is important to note that the rest of the health districts included in the study and districts across Guatemala have many more distinct characteristics that could change their implementation experience. Additionally, this particular assessment did not include perspectives from program recipients (i.e. patients), although our study team has captured patient perspectives in a different analysis .This study contributes to calls to advance our understandings of sustainability and health equity in implementation science and provides a rich empirical example of application of complementary implementation science theories and frameworks. Our study also provides urgently needed information on how to assess multi-component programs in settings with limited resources and under rapidly changing contexts, such as the COVID-19 pandemic. We conducted a rich mixed methods assessment of a program implementation over time, showing the close inter-relationships between context, EBI and implementation strategy, and their influences on health equity and sustainability. These findings point to the need for robust mixed methods and more rapid assessments, and systems science approaches that help understand the dynamic relationships between contextual and implementation factors over time.


Substance use is a steadily worsening crisis, yet there is limited evidence on the effectiveness of brief educational programs related to substance use disorders (SUD). To address this, the Just Five digital program launched in 2019 with six educational lessons about SUD, each five minutes long. Just Five is delivered through various organizations, including employers as part of workplace wellness, health plans that offer it to members, and to community-based organizations and populations they serve. This study evaluated the effectiveness of the program across 10 organizations in the initial launch. A cross-sectional survey was given to Just Five viewers to establish changes in perceived knowledge and attitudes toward SUD. Results were analyzed among the overall sample; a sub-analysis compared results within the three types of organizations taking part. Between December 2019 and September 2021, 2,749 baseline and 397 follow-up surveys were collected. Significant improvements were observed across almost all questions asked, with the largest improvements in respondent confidence to recognize signs of addiction ; knowledge about ways to reduce the risk of addiction ; and knowledge of how to help someone with addiction . All trends were replicated in sub-analyses. Substance use has severe societal impacts, and there is a need for effective programs that can be quickly implemented across various environments at large scale. Just Five is a potentially effective tool to increase perceived knowledge and improve attitudes toward SUD. In the United States, substance use is an ongoing crisis that is steadily worsening, particularly since the start of the COVID-19 pandemic. At present, drug overdoses are the top cause of accidental death . ExpertsDespite the clear impact of SUD on society, there is limited evidence on the effectiveness of brief adult educational interventions. However, a review of the available literature suggests brief interventions can have an impact on a person’s actions and attitudes around substance use, particularly related to alcohol consumption . Many prThe CDC provides guidance on effective Workplace Health Models, suggesting that the workplace is one important setting for health protection, health promotion, and disease prevention programs, but does not mention SUD . There ijustfive.org. More information about the lessons can be found in In response to this gap in evidence, the Just Five digital SUD education program was created and launched in 2019 by Shatterproof, a national nonprofit organization dedicated to reversing the addiction crisis in the U.S. Just Five is designed for companies interested in educating their employees about substance use disorders. The program, offered in English and Spanish, includes six lessons, all five minutes or less in length, designed to educate viewers with clinically accurate information about addiction and how to support their own or a loved one’s health. Each organization or company receives a personalized landing page with both the Just Five educational program, as well as other employer or community resources. To encourage participation, the program is delivered through an open website, without a requirement to enroll or enter any identifying information to access the lessons. Lessons 1–3 focus on general addiction education, including the science of addiction, risk factors for addiction, and specific information on opioids. Lessons 4–6 provide actionable steps for those impacted to be able to identify signs of addiction, access evidence-based treatment, and support loved ones through to recovery. All lessons are cited and include additional resources for those interested in reading more about a topic. The program is sponsored and promoted by various types of organizations, including: 1) employers who provide Just Five to their employees through employer wellness programs, 2) health plans who provide Just Five to their subscribers, or 3) through community-based organizations who offer Just Five to the general community and members they serve. Organizations are provided with a Just Five toolkit that offers suggestions on how to integrate the program into other health educational initiatives. More information about the Just Five program can be found on the website To establish knowledge and attitudes toward SUD before engaging in Just Five, two online surveys were offered to each individual participating in the program, one immediately. before the first lesson, and one immediately after the final lesson. Eligibility criteria for the survey included being over 18 years old, receiving an invitation to take part in the Just Five program from either an employer, health plan, or community organization, and being English-speaking . Personally identifiable information, including demographic information, was not collected from respondents, and all respondents were identified with a unique identifier for analysis. Before participating in the surveys, participants were provided with an informed consent statement, informing them that their responses were voluntary and confidential. Respondents were not required to answer questions and could quit the survey at any time. Given that the survey presented no more than minimal risk of harm to participants and no identifiable information was collected, the need for documented written consent was waived by the Institutional Review Board Advarra, which deemed this study to be exempt from review. Their submission of survey answers was considered consent. Immediately after completion of the final module, individuals were requested to complete the follow-up survey. The surveys consisted of questions adapted from existing research on substance use and stigma –23, and Due to privacy restrictions from the organizations taking part in the Just Five program, it was not possible to link responses from the baseline and follow-up datasets. Therefore, the sample likely contained both longitudinal and cross-sectional respondents. Incentives for survey participation varied, depending on the policy of the organization. One community-based organization provided $5 coffee cards for survey participation, while all others did not provide any incentive, per organization policies. Given that the Just Five program enrolls organizations and participants on a rolling basis, surveys were delivered at differing times for each organization, depending on the organization’s enrollment into the program, or new staff or plan members start date . Analysis was conducted on baseline surveys across 10 organizations. Demographic information was not available for all respondents taking part in the program and was therefore not included in the analysis. Data were analyzed both among the overall sample of survey responses and were also segmented according to the three types of organizations that took part in the program. Data were segmented by employers who offered Just Five to their employees through a wellness program , health plans who made the program available to health plan subscribers , and community-based organizations who offered the program to the general communities they serve .Questions were all asked on a five-point Likert scale, from “strongly agree,” “agree,” “neither agree nor disagree,” “disagree” and “strongly disagree.” For the purposes of analysis, “strongly agree” and “agree” were combined into one category (Agree), and “strongly disagree” and “disagree” were combined into one category (Disagree). Negative questions were asked to prevent potential response set bias. For negative questions, disagreement is presented in the tables below, and noted where relevant. For analysis, organizations were segmented into three categories, based on the type of individual they reached. Data were compared using a two-sided Pearson Chi-square test using IBM SPSS Statistics software and significance was assessed at a p-value of ≤.05.A total of 2,749 baseline surveys were collected between 12/15/2019 and 9/29/2021 and 397 follow-up surveys were collected between 2/4/2020 and 9/28/2021. Health Plan respondents comprised 42.8% of all survey respondents , followed by Employer respondents with 32.0% of surveys , and CBO respondents with 25.2% of surveys . Respondents showed significant improvements from baseline to follow-up, across almost all questions asked . The larRespondents were then categorized by the type of organization that took part in the Just Five program  to examine whether improvements in questions among the overall sample held true for each type of organization engaged in the program . OverallAcross several questions, results showed significant improvements among Health Plan and Employer groups, and non-significant improvements among CBOs. This included disagreement that a lack of moral strength plays a part in addiction, with around 65% at baseline and more than 80% at follow-up among Health Plan and Employer respondents. The belief that people can fully recover showed 69% agreement at baseline, with 74% at follow-up for Health Plan and 84% for Employer respondents. Finally, these two groups showed significant improvements in beliefs that people are not at fault for their addiction, with 60% at baseline to more than 80% at follow-up.Other questions showed varying trends. CBO and Health Plan respondents showed significant improvements on beliefs about the effectiveness of treatment (89% at baseline to 93–95% at follow-up), Health Plan respondents showed significant improvements in believing that it is not ok for teenagers to try drugs and alcohol , and CBO respondents showed significant improvements on beliefs that virtually anyone can become addicted .Awareness and education around substance use disorders (SUD) is critical to improve the health and wellbeing of individuals and communities. This evaluation of the Just Five program suggests that the program may be a useful tool to increase perceived knowledge and improve attitudes toward SUD across various contexts—as part of an employer wellness program, when sponsored by a health plan to its subscribers, and when promoted by community-based organizations. Results from three questions were particularly notable: the largest increases were shown in feeling confident around recognizing the signs of addiction , knowing ways to reduce the risk of addiction (which increased 32.4 percentage points) and knowing how to help if an acquaintance is living with addiction (which increased 43.0 percentage points). These dramatic improvements suggest that respondents may be more able to support loved ones after taking the Just Five program . AlthougAlthough substance use severely impacts individuals, organizations, and society, there is a gap in evidence on programs that can be effectively delivered across various environments. Within the workplace, Just Five can fill a critical gap in SUD education as a part of employer wellness programs. In the past, employer wellness programs have focused on promoting health screenings, chronic disease prevention and management, flu shots, and mental health and stress management . These pIn the past several decades, health plans have placed an increasing emphasis on wellness programs as a way of supporting the health of their member base and reducing costs . To dateFinally, the flexibility and digital nature of Just Five also allows for its delivery to the public through community organizations. There are few brief, evidence-based substance use programs are available to reach adults in the community. Community substance use education programs are usually geared toward school-aged students through longer-term curricula focused on prevention , 33. Or,This study carries with it some limitations. Due to restrictions from the organizations taking part in the Just Five program, it was not possible to determine differences between longitudinal and cross-sectional respondents. Therefore, is it not possible to definitively associate changes to participation in the program. It is possible that survey respondents who took the follow-up survey already had better views toward addiction, or that survey respondents answered the surveys according to a response bias. For most organizations that participated in the program, there was a large drop-off from baseline to follow-up, likely due to the fact that most organizations taking part in the program could not incentivize the surveys. Particularly when analyzed by the type of organization involved in Just Five, this low response rate may have limited the ability to find significant improvements over time. Future researchers should undertake a longitudinal study to link changes more definitively to the program. Despite these limitations, we feel this study fills important gaps in evidence on brief interventions to address addiction.The Just Five program shows great potential as an important brief adult awareness program on substance use. The evaluation of 10 sites enrolled in the Just Five program showed significant increases in perceived knowledge, attitudes and levels of confidence in recognizing and helping with substance use issues. This suggests that the program may be an important addition to the current evidence base around tools for adult substance use education. As the United States now shifts attention away from the pandemic, it is important to have as many tools available as possible to help fight the growing substance use crisis.S1 File(DOCX)Click here for additional data file.


Opioid addiction and overdose is a public health problem in the United States and is expected to remain with substance use increasing due to the COVID-19 pandemic. Communities that approach this issue through multi-sector partnerships experience more positive health outcomes. Understanding motivation for stakeholder engagement in these efforts is essential to successful adoption, implementation, and sustainability particularly in the shifting landscape of needs and resources.n = 9). The Consolidated Framework for Implementation Research (CFIR) guided data collection and analysis. Surveys (n = 8) examined perception and attitudes on the program; motivations and communication for engagement; and, benefits and barriers to collaboration. Stakeholder interviews (n = 6) explored the quantitative findings in more detail. Surveys were analyzed using descriptive statistics and a content analysis with deductive approach was conducted for stakeholder interviews. The Diffusion of Innovation (DOI) Theory guided recommendations for communications to engage stakeholders.A formative evaluation was conducted on the C.L.E.A.R. Program in Massachusetts, a state heavily impacted by the opioid epidemic. A stakeholder power analysis identified appropriate stakeholders for the study (n = 5) were familiar with the C.L.E.A.R. Program. Despite the many strengths of the program and existing collaboration, based on the coding densities of each CFIR construct stakeholders identified crucial gaps in the services the program provided and noted that the overall infrastructure of the program could be enhanced. Opportunities for strategic communication to address the stages of DOI align with the gaps identified in the CFIR domains to result in increased agency collaboration and expansion of services into the surrounding communities to ensure sustainability of the C.L.E.A.R. Program.Agencies represented a range of sectors and the majority .The online version contains supplementary material available at 10.1186/s12889-023-15229-2. Opioid addiction and overdose is a pervasive public health problem in the United States , and MasRecognizing this, in 2010 a national drug policy was developed in the United States that sought to create a “balanced public health and public safety approach”. At the core of this policy was the recognition of addiction as a disease which should be treated in lieu of punishment . AccordiPresently, the COVID-19 pandemic has not only propelled the importance of addressing opioid overdose but highlighted the need to engage multiple sectors for success. Rates of overdose are growing with an increased reliance on substances due to the interruptions and stressors to daily life . CommuniThe Community and Law Enforcement Assisted Recovery (C.L.E.A.R.) Program, established in 2014 and not since formally evaluated, is a community-based program local to Winthrop, MA that aligns public health and safety to assist those who have experienced opioid overdose with accessing recovery resources. Aligned with principles outlined in the National Drug Control Strategy  the C.L.Public health and public safety partnerships have been highlighted for almost two decades and heightened social awareness coupled with COVID-19 impacts have elevated critical need for multi-sector collaboration in communities , 33. WitFormative evaluations allow planners to uncover strengths and limitations of the program and provide suggestions for further implementation and expansion . Using qThis evaluation was conducted on the C.L.E.A.R. Program developed in Winthrop, MA, a town of approximately 18,600 people north of Boston. In 2016, Winthrop ranked 17th out of 351 municipalities in MA for opioid-related fatalities per capita  . Since tThis study aimed to 1) identify stakeholders for inclusion in formative evaluation, 2) identify facilitators and barriers to adoption and implementation of the program, 3) offer strategies for communication to foster engagement, and, 4) build on the literature for use of a framework to guide evaluation and communication approaches. The formative evaluation of the C.L.E.A.R. Program is a necessary step in identifying barriers to implementation and subsequent recommendations for practice  and is wIn order to understand acceptance and implementation of the C.L.E.A.R. Program, mixed methods formative evaluation data collection, analysis, and results were organized according to the CFIR , a frameWhile the CFIR contains a variety of constructs that could ultimately be applied to analysis of data, we utilized a menu of constructs approach . In thisThe success of a program, especially a multi-sector program, relies on multiple stakeholders moving through a continuum from awareness of the problem, initial use of the innovation or program to address the problem, and ongoing application of the program for sustainability  to resulThe goal of this mixed methods formative evaluation was to explore implementation of the C.L.E.A.R. Program and the perception of need across stakeholders especially given the shifting context introduced by COVID-19; identify areas for improvement in collaboration, engagement, and communication; examine data needs for comprehensive system-level approaches; and, inform communication strategies to promote the goals and expected outcomes of the C.L.E.A.R. Program internally to stakeholders for collaboration.The formative evaluation included an intensive stakeholder analysis to determine the priority individuals to engage in the stakeholder interviews. A survey was administered to assess perspectives, responsibilities, and current C.L.E.A.R. Program collaboration status of each priority stakeholder and was guided by the CFIR domains. The semi-structured interviews were guided by CFIR in addition to the results from the survey and the C.L.E.A.R. Program logic model Fig.  53]. Re. Re53]. An analysis was performed to determine the stakeholders most relevant to the C.L.E.A.R. Program and therefore targets of the formative evaluation. Stakeholders, defined as an individual or organization influenced by or able to influence the C.L.E.A.R. Program  or who hFifteen stakeholders were identified through this process – five stakeholders with high interest and high power, eight with low interest and high power, one with low interest and low power, and one with high interest and low power. Stakeholders’ perceived needs , benefits , and resistance  were also assessed during the analysis. Of the 15 identified stakeholders, nine were recommended by the evaluation team for follow-up contact with oversampling of the categories most heavily concentrated by stakeholders  and with the most influence .n = 9) identified in the stakeholder analysis and represented private and public agencies in Winthrop and the surrounding counties. Some of these agencies were directly involved in substance use prevention efforts, mental health services, and treatment resources and others were more indirectly involved as first responders or youth-serving institutions. The primary contact for the C.L.E.A.R. Program within the agency was determined to be the stakeholder for this evaluation. Of the nine stakeholders contacted, six were available for an interview (66.7%) and eight (88.9%) responded to the survey.Follow-up contact was recommended to represent agencies that had either high power, high interest, or both . Within each area, the content of the questions corresponded to the appropriate CFIR construct. Accordingly, the interview guide was organized by 1) awareness and attitudes ; 2) collaboration ; 3) engagement (factors that foster agency connection with opioid overdose programs); and, 4) communication (successful outcomes of collaboration and communication of the outcomes as well as supports needed for sustainability). Each section of the interview used probes as appropriate from the survey results and aligned with CFIR constructs  using descriptive statistics to examine continuous and categorical data. Interviews were designed, conducted, and analyzed in accordance with the consolidated criteria for reporting qualitative research (CORE-Q) . The evan = 4) serving at their agency for more than ten years and none serving less than four years. Funding sources varied for each agency with the majority coming from federal sources (25%), local sources (20.8%), and state sources (20.8%) (data not shown).There were a total of eight survey responses with seven different agencies represented (one agency had two respondents). Stakeholder respondents represented public safety (37.5%), public health (25%), behavioral health (12.5%), community-based agency (12.5%), and public schools (12.5%). Job responsibilities ranged from agency leadership (24%), advocacy (12%), public information (12%), community health (8%), health educator (8%), preparedness (8%), office/administrative (8%), and other (20%) with half of respondents (n = 7), local resources , informal discussions/meetings on the opioid epidemic , training on the opioid epidemic , discussion forums/meetings on the opioid epidemic  , and technology/tracking systems .The majority of respondents strongly agreed there is an opioid epidemic nationally (87.5%) and that there is an associated burden of the opioid epidemic within Massachusetts (87.5%) and within Winthrop (62.5%). In addition, most strongly agree that responding to the epidemic requires more resources than are currently allocated (87.5%). Half of respondents (50%) report that the protocol in their community for responding to the opioid epidemic needs to be updated or better communicated with 50% reporting it is in a steady state and 50% reporting it is in a resurgence. Agencies reported using a variety of resources to respond to the epidemic including state resources  of agencies were somewhat familiar with the C.L.E.A.R. Program, 25% (n = 2) were very familiar with the program, and 12% (n = 1) were not that familiar with the program. Four respondents (50%) reported that they don’t currently collaborate with the program but have in the past, two (25%) reported that they do not currently collaborate nor have they in the past, one (13%) reported currently collaborating with the program, and one reported being unsure (13%). Of the seven agencies that do not currently collaborate with the C.L.E.A.R. Program or who are unsure, 66% are extremely likely and 34% are moderately likely to collaborate in the future.More than half , newsletters , and toolkits .Of the three agencies that have not collaborated with the C.L.E.A.R. Program or are unsure, all (100%) reported strengthened ties to the community, streamlined data systems for tracking and outreach, building trust within the community, and building capacity within the community as possible benefits. Resources they would find helpful in collaborating included reports with both qualitative and quantitative data . Resources to foster ongoing collaboration with the program included flyers, memorandums of agreement, social media, reports, infographics, websites, newsletters, and toolkits. All agencies reported that the C.L.E.A.R. Program adds value to the efforts of their agency, fills gaps in their agency, and fits within the structure of their agency, and all agreed there is a strong need for the program Table .Respondents were asked about communication channels used within their community for emergency efforts and reported Facebook (75%), email (63%), text messaging (50%), print media (38%), automated phone calling (38%), and a hotline or call center (25%). These findings are important to consider in making recommendations for communicating about the C.L.E.A.R. Program.Six of the eight survey respondents were interviewed with one respondent not available. The qualitative interview findings are presented by CFIR constructs and summarized into phases aligning with DOI for practical application. For each of the five CFIR domains  the hierarchy of inner constructs that comprise the domains  are depicted with the size of the box indicating the number of times a construct was referenced in the interviews. For example, implementation climate is the most densely coded construct in the inner setting domain with 35 codes, which is demonstrated by the larger surface area of that box Fig.  indicatiActual and perceived barriers such as cost, evidence strength, resources, patient needs, external policies, and champions, are reflected in the responses. In addition to the constructs presented in Fig. [A] clear gap is safe locations for people to use drugs. C.L.E.A.R. Program delivers naloxone. Gap would be working closer with folks who are abstinence-minded, very hard line about recovery, maybe some antiquated views about naloxone distribution to address abuse. It [naloxone use] just needs to be normalized.This domain addresses factors that influence whether the intervention is implemented successfully. While this domain can include factors and components specific to the intervention that are actual barriers or facilitators including cost and evidence strength. It largely addresses perception of the factors influencing an intervention’s successful implementation, which includes the decision to adopt given the idea is perceived as new, innovative, and/or necessary. Evidence strength and quality and relative advantage had the strongest coding densities Fig.  with sta[The current partners] “are a tight group  so the confidentiality among us is very strong……pact between us that allows sharing of information.” Knowing each other separates us from the average town or city collaboration. We all live in the community so that’s another level of understanding [and passion].This domain addresses external factors impacting the intervention such as agency networks, communication, policies and incentives, barriers and facilitators to meeting needs of the population, and competitive pressure to implement an intervention . Patient[Certain people have] tried to break down the stigma a bit but I don’t know if the culture has shifted. “It’s a huge undertaking…it’s the town culture, not a department of public health culture”.This domain addresses internal factors of an organization that impact the implementation of an intervention including the internal communication, norms and values, structure and system, capacity for change and receptivity of individuals, and commitment within the agency to the intervention . In thisRight now Winthrop [to my knowledge] is the only one without a naloxone program (OEND program). So when overdose occurs it’s not just about providing access to recovery but looking at stages of change and meeting them where they are – if they want to use then [the program should] help them use in safety. Help [those in need] build up readiness to get to the next level. This is a big gap that law enforcement can’t do… They also identified a need for clearer communication to stakeholders and community partners to enhance collaboration and program reach.This domain addresses the thoughts, perceptions, knowledge, confidence, beliefs, and commitment to change of individuals that comprise the organization . Ultimat“Some type of platform, that we can all go in and have access to… Success stories, where are they at”.This domain addresses implementation and sustainability through consideration of engagement strategies , established plans for implementing the intervention, delivery of the intervention according to the established plan, and communication and feedback about the intervention and its implementation. Observability, or the opportunity to see the intervention in practice, can prompt ongoing use . In the The findings presented are specific to the context and program within Winthrop and will be used for revision and improvement of program components, new and sustained engagement of collaborators, adaptations and enhancements given the shifting substance use landscape, and determination of resource allocation particularly with the increased need yet taxed public health and public safety infrastructure. The findings, however, provide valuable insights to other communities that are planning or currently implementing substance use programming and offer insights to the evolving nature of a multi-sector program that relies on high quality collaboration and communication to effectively meet objectives and respond to shifting needs of its recipients in a resource-constrained environment.This mixed-methods formative evaluation aimed to uncover how existing and future stakeholders could be leveraged and more fully engaged to further the reach and efforts of an opioid overdose program at a time when expansion and enhancements are necessary to meet the shifts resulting from COVID-19. This study found that stakeholders agree that the work of the C.L.E.A.R. Program is important and that the public health and public safety relationship is strong, but there are crucial gaps in the services provided such as prevention efforts and safe injection sites Table . The stuLimited evidence on the design, implementation, and evaluation of community-based opioid-related programs exist  and methConducting a formative mixed methods evaluation of the C.L.E.A.R. Program allowed insight to the implementation of the program as designed but also as it was being adapted to fit the COVID context . In partAreas of improvement in the design and implementation of the C.L.E.A.R. Program align with the CFIR and strategies to address those areas can be organized according to the DOI.Within the intervention characteristics domain there were reported gaps in the program to meet the needs of a larger geographic area. Redesigning the program means examining the current infrastructure of the collaborating agencies to determine how they provide support to the program or identify the enhancements needed , particuWithin the inner setting domain, it was reported that effective collaboration entailed prioritizing external communication with partners not just within the immediate catchment area of the program but in areas that affect both implementation and the stigma that surrounds opioid use. Communication encouraging collaboration with the C.L.E.A.R. Program requires careful messaging to the target population and stakeholder agencies. This messaging should be pre-tested to ensure sensitivity and that it fits the context .Gaps uncovered in the characteristics of the individuals domain can be addressed through communication focused on sustainability that is appropriate, consistent, and tailored to the program’s success, challenges, and areas of expansion. This is important for stakeholder investment in both their agency’s long-term engagement of the program but also in understanding the importance of their individual contribution to successful outcomes . This caAs outlined, considering implementation determinants organized by CFIR domains  results Effective communication relies on a target population’s ability to receive communication and for that communication to affect some sort of change, whether through adoption of a new behavior, uptake of new information, trialability of a new program, or commitment to ongoing implementation . InformaProgram advertising through various communication efforts raises awareness of the program for program recipients and stakeholders that make the program stronger and align with implementation strategies focused on communication including developing educational materials, distributing educational materials, tailoring strategies, and using mass media  or impleUtilizing strategic communication with both new and existing stakeholders to introduce and increase multi-sector collaboration is important to programs such as the C.L.E.A.R. Program so the varying needs of the target population can be addressed through availability of a variety of appropriate recovery services , 79. In The sustainability of a program is key to addressing a public health problem and is the last stage in the diffusion process . OngoingTailored communication to stakeholders and the community is used to 1) create awareness of the program to people who could benefit from it and organizations that will strengthen it, 2) foster collaboration between the established program and meaningful stakeholders, and 3) engage the community and stakeholders on an ongoing basis to move the program forward . AligninOur study demonstrates the use of the CFIR to identify strengths and gaps in an existing opioid overdose program and opportunities for expansion and improvement particularly as the post-COVID-19 public health field is shifting. The combination of the DOI to inform strategic communication to address the gaps and promote the strengths for stakeholder adoption, program implementation, and multi-sector sustainability further enhances the findings and recommendations of this study. In addition, the study utilized methodology that focused on mixed methods to inform immediate and sustainable changes , 82 fromAlthough DOI has been widely used in the public health field , much ofThis mixed methods study builds on existing literature by exploring the facilitators and barriers to a multi-sector community-based opioid overdose prevention program through use CFIR to systematically identify the factors impacting implementation and DOI to recommend communication strategies to ensure engagement, use, and continuation of the program. With existing programs, such as the C.L.E.A.R. Program, formative evaluation findings can result in necessary adaptations that ensure more widescale adoption and sustainability. This is particularly important when the context around an existing intervention change resulting in shifts in resources, priorities, and stakeholders. This is emphasized with the already severe opioid epidemic being exacerbated by the COVID-19 pandemic.Specifically, this study found crucial gaps in the services offered and catchment area reached by the C.L.E.A.R. Program but that existing resources and agencies could fill those gaps if appropriately engaged. Additionally, with ongoing collaboration from these agencies, the overall infrastructure of the existing program could be enhanced leading to expansion and sustainability. These findings were further explored through tangible recommendations on ways to communicate to all stakeholders – collaborating agencies and program targets – depending on whether the stage of diffusion was adoption, implementation, or sustainability. While this study explored multiple perspectives in this arena, future work in the implementation and evaluation of community-based opioid prevention programs should consider patient-centered perspectives and needs and how those needs change over time. This will allow continuous redevelopment and revision of substance use programs and policies that aim to address individual needs through agency cross-collaboration leading to targeted, sustainable approaches that improve outcomes for those who are most at risk of opioid overdose.Additional file 1.


The QUIC (Quick UDP Internet Connection) protocol has the potential to replace TLS over TCP, which is the standard choice for reliable and secure Internet communication. Due to its design that makes the inspection of QUIC handshakes challenging and its usage in HTTP/3, there is an increasing demand for research in QUIC traffic analysis. This dataset contains one month of QUIC traffic collected in an ISP backbone network, which connects 500 large institutions and serves around half a million people. The data are delivered as enriched flows that can be useful for various network monitoring tasks. The provided server names and packet-level information allow research in the encrypted traffic classification area. Moreover, included QUIC versions and user agents  provide information for large-scale QUIC deployment studies. Specifications Table•The dataset •The provided dataset is useful for research in the computer networks field. The real-world nature of data allows researchers to validate and evaluate network traffic classifiers and prepare their algorithms for various traffic phenomena and connection errors that appear in ISP-like networks. Flows in the dataset include packet metadata sequences, which are the standard data input for various tasks in encrypted traffic analysis.•The provided dataset can be used for (i) the design and real-world evaluation of QUIC web services classifiers •There are no public QUIC datasets of comparable size.•Previous studies 1The QUIC protocol is gaining adoption across service providers and consumer software. Its usage as the transport protocol in HTTP/3 suggests its future mass usage. However, despite its apparent importance, no extensive QUIC datasets exist. Therefore, we created the CESNET-QUIC22 dataset containing one month of QUIC network traffic transmitted through the backbone lines of CESNET2, which is the Czech national research and educational network that serves half a million people. The real-world nature of the provided traffic and its extensive size create a unique and comprehensive dataset that enables research of the novel yet crucial QUIC protocol.2The dataset consists of network flows describing encrypted QUIC communications. Flows are extended with packet metadata sequences, packet histograms, and with fields extracted from the QUIC Initial Packet, which is the first packet of the QUIC connection handshake. The extracted handshake fields are the Server Name Indication (SNI) domain, the used version of the QUIC protocol, and the user agent string that is available in a subset of QUIC communications.2.1Sequences of packet sizes, directions, and inter-packet times are standard data input for traffic analysis. For the packet sizes, we consider payload size after transport headers (UDP headers for the QUIC case). Packet directions are encoded as We also derive three fields from each packet sequence: its length, time duration, and the number of roundtrips. The roundtrips are counted as the number of changes in the communication direction (from packet directions data); in other words, each client request and server response pair counts as one roundtrip.2.2Flows in the dataset also include standard flow statistics, which represent aggregated information about the entire bidirectional flow. The fields are: the number of transmitted bytes and packets in both directions, the duration of the flow, and packet histograms. The packet histograms include binned counts of packet sizes and inter-packet times of the entire flow in both directions.2.3The dataset flows are delivered in compressed CSV files, which are organized as shown in Moreover, various dataset statistics, such as feature distributions see  and valu3The data collection was performed using the monitoring infrastructure of the CESNET association, which is the operator of the CESNET2 national research and educational network. Most of the connected entities are large public organizations such as universities, campuses, high schools, research centers, hospitals, and municipal offices. The CESNET2 network has around half a million users and spans the whole Czech Republic; its topology is shown in The monitoring infrastructure of the CESNET2 network follows the traditional IPFIX monitoring approach described in Hofstede et al. The data capture workflow is visualized in 3.1Traffic volumeWe prioritized web services with larger traffic volumes so that the dataset covers the majority of real network traffic. The selected web services cover 84% of all QUIC traffic in the CESNET2 network.DiversityWe selected diverse web services to capture various types of QUIC traffic. The dataset services can be divided into 17 categories. The distribution of service categories, the number of samples, and the traffic volume per category are shown in For building the dataset, we decided to select a diverse and representative subset of web services that could be observed in the CESNET2 network. The selection of the services was based on the following criteria:The selected services were recognized using Server Name Indication (SNI) domains that were extracted from the QUIC connection handshake. To find the domains associated with a service, we searched its online documentation. In some cases, we found a docpage with “whitelist domains for network and firewall settings”, which contained all the service’s domains. In other cases, we used Netify’s Application Lookup Tool.On top of the selected web services, we also captured background traffic—i.e., traffic not belonging to none of the selected services. The background traffic allows network classifiers to be evaluated in open-world scenarios, where the classifier needs to deal with traffic classes that were not available during training. We decided to split the background traffic into three classes. The Google Background class contains the traffic of not-selected Google services, Facebook Background contains the traffic of not-selected Facebook services, and Default Background contains the traffic of the remaining QUIC web services that are present in the CESNET2 network. This separation allows the evaluation of more granular and detailed open-world scenarios.3.2Each monitoring point was installed with ipfixprobe,The flow exporting process was set with an active timeout of 5 min and an inactive (idle) timeout of 65 s. Long connections are split when the connection duration is longer than the active timeout, and a flow record is exported even though the actual connection is not terminated yet. If no packet is observed within the inactive timeout period, the connection is considered terminated, and a flow record is exported. Using active and inactive timeouts for splitting connections is standard practice for flow-based network monitoring Exported flows from each monitoring point were transmitted using the IPFIX protocol to a single flow collector, which performed additional processing.3.3Flows were collected using the IPFIXcol2We performed flow filtration using a NEMEA filtering module.Next, flows were assigned a web service (or background) label using the SNI–service mapping described in 3.4Since our goal is long-term flow capture from a large backbone network, we need to use data sampling to maintain a reasonable size of the dataset. Instead of using one sampling ratio for all traffic, we decided to use a dynamic sampling ratio for each service to soften the class imbalances in the dataset. Each service is sampled at a different ratio, depending on the amount of traffic  it generates. This dynamic sampling approach ensures that even less-prevalent services are represented in the dataset with a sufficient amount of samples.Services are sorted based on the amount of their traffic. The top 5% of services are sampled with a 1:15 ratio; the bottom 60% of services are not sampled at all. The rest 35% are sampled with a ratio ranging between 1:2 and 1:9, depending on their prevalence. Moreover, the background traffic classes see  are all 3.5To protect the privacy of CESNET2 users, we anonymized client IP addresses using the Crypto-PAn (Cryptography-based Prefix-preserving Anonymization) algorithm Maintaining the privacy of Internet users is our prime concern; hence, the research and dataset collection were done with extreme carefulness. The indisputable advantages of real traffic generated in a production network come with the cost of potential privacy violations of real users. Therefore, we used automatic data processing with immediate data anonymization. With this, we declare that we did not analyze or process deanonymized data and did not perform any procedures that could lead us to users’ identities.Jan Luxemburk: Conceptualization, Data curation, Methodology, Software, Visualization, Validation, Writing – original draft, Writing – review & editing. Karel Hynek: Conceptualization, Methodology, Software, Validation, Writing – original draft, Writing – review & editing. Tomáš Čejka: Conceptualization, Project administration, Funding acquisition, Supervision. Andrej Lukačovič: Software, Validation. Pavel Šiška: Software, Validation.The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.