Current rates of burnout among physicians are alarming when compared to nonphysician U.S. workers, and numerous interventions have been introduced to mitigate the issue. However, no interventions have specifically targeted the 30% burnout rate among physician program directors. The complex and demanding role of program directors necessitates building relationships, solving crises, securing jobs for residents and maintaining well-being of trainees. The aim of this study is to investigate the impact of emotional intelligence (EQ) on burnout levels among program directors.A cross-sectional survey was administered from May 17 to June 30, 2021 to program directors and assistant/ associate program directors at an academic medical center in south-central Pennsylvania. A self-report questionnaire was used to collect data. The survey included an open-ended question along with the Trait Emotional Intelligence Questionnaire- Short Form (TEIQue-SF), Copenhagen Burnout Inventory (CBI), and demographic questions. All data were analyzed using SAS Version 9.4.Of the 109 program directors and assistant/associate program directors invited in the survey, 34 (31.20%) responded. The findings indicate that there is a moderate inverse association between EQ and burnout, suggesting EQ as a protective factor against burnout. We also found that program directors who were considering leaving their position demonstrated higher levels of burnout compared to those who did not. Results from the open-ended question suggest that perceived lack of support, micromanagement, criticism, and extra duties with less payment were among the reasons program directors and associates were considering steeping down from their position. The results showed no association between EQ skills and years of practicing.Burnout among program directors and assistant/associate program directors is not as alarming as rates of burnout among physicians-in-training. However, despite high level of EQ skills and low burnout level, nearly 43% of program directors were considering leaving their position. Nurturing EQ skills may be useful in improving retention and reducing turnover among medical leaders. The term “burnout”, which is defined by the World Health Organization (WHO) as a long-standing work-related stress , is becoAn additional example of an area for personal intervention is emotional intelligence (EQ), which has been defined as the ability to monitor, understand, and guide one’s own thinking and actions . EQ skilThe association between burnout and EQ has been studied previously, with most research affirming an inverse association between the two, i.e., higher EQ skills correlate with lower levels of burnout , 15. DesThe primary objective of the study is to evaluate the association between EQ skills of residency and fellowship program directors, including assistant/associate program directors, and level of burnout. A cross-sectional survey including fellowship and residency program directors and assistant/associate program directors at an academic medical center in south-central Pennsylvania was conducted from May 17, 2021, through June 30, 2021. Recruitment involved obtaining a list of fellowship and residency program directors and assistant/associate program directors from the director of graduate medical education at the institution and contacting the prospective participants via email. Program leaders were sent a link containing the study’s consent material; if they agreed to participate, they were taken to the survey. The survey was administered through REDCap, a secure online web application  self and learners, little room for additional growth in this role, not enough time given to focus on program/department/institutional issues related to GME” and “criticism of those around me” were contributing factors to burnout. These sentiments regarding lack of professional support in addition to interpersonal challenges presented in a number of open-ended responses from program leaders and highlight the impact that these particular factors have on the work-related burnout that these leaders experience.The role-specific challenges have been found to be related to frustration with regularity requirements, lack of appreciation and support from the department chair and hospital administration, low satisfaction with colleague relationships, and lack of training in dealing with residents’ problems , 25. The“more work and no increase in pay” to be emotionally draining  and, by extension, program leaders. Future research should aim to evaluate the impact that the changes that occurred due to the COVID-19 pandemic may have on these associations moving forward.


Families going through conflictive divorce processes are at increased risk of developing mental health problems. The Egokitzen program is a group intervention for parents who have undergone a divorce process, funded by the public administration. Budgetary constraints cause funding institutions to be interested in the effectiveness and economic efficiency of these programs. Therefore, the objective of this research is to carry out an efficiency analysis of the Egokitzen program, implemented by family visitation centers in Spain, through a cost-benefit analysis, to determine whether the positive impact on symptomatology (measured using CBCL and SCL-90 instruments) is translated into a positive economic impact for society. A sample of 382 parents participated. Costs will be first identified and valued; secondly, benefits achieved with the program will be identified through a prevalence analysis and, finally, the cost-benefit comparative analysis will be carried out. Additionally, a sensitivity analysis will be performed. The results obtained in the analysis indicate that for every euro spent on this program, the public administration and society save 3.10 euros in future interventions through medical costs and productivity losses. The study has practical implications for public administration, organizations, and the family visitation centers that implement the program. The United Nations has focused on the 17 most important global problems through the Sustainable Development Goals (SDG). Specifically, promoting good health and well-being is one of these goals SDG3). Given that mental health problems affect 25% of the world’s population and that problems such as anxiety or depression are one of the leading causes of disability in many countries, mental health care is one of the major challenges facing European health systems [. Given tIn this sense, families, both parents and children, who go through conflictive divorce proceedings are a population at risk for the development of mental health problems. Studies show that divorces create psychological problems, through externalizing and internalizing manifestations, both in parents and children ,5,6,7.This reality is significantly prevalent, given that the number of divorces at the European level amounts to almost one million families per year. In Spain, this figure reached 95,254 divorces during 2018, declining to 77,200 in 2020 . Of the total of divorced couples, more than 60% at the European level and 45% at the Spanish level had at least one child . It is tStudies at the international level have obtained results confirming that conflictive divorce is a fundamental stressor in children, with the potential to cause long-term health problems . The desThe Egokitzen program is a group intervention for parents who have undergone a divorce process. The nature of the program is preventive, psycho-educative, and with a systemic orientation of family functioning. The program achieved good results in a university laboratory context  and the Therefore, the objective of this research is to carry out an efficiency analysis of the Egokitzen program through a cost-benefit analysis, to determine whether the positive impact on internalizing and externalizing symptomatology has a positive economic impact on society.The Egokitzen program intervention is performed in a group format. Only one of the members of the divorced couple participates in the group. Each parent group participates in 11 weekly one-and-a-half-hour sessions, with a dynamic methodology . These sessions focused on three areas: divorce and its impact, interparental conflict, and parenting. Before the initiation, the facilitators who are in charge of the parent groups receive a 5-h instruction and the necessary documentation to work with the groups.The study was carried out with the collaboration of 13 family visitation centers (FVCs) that are located in 10 of the 19 Spanish autonomous communities. FVCs were part of the National Federation of Family Visitation Centers  which brought together 33 centers at national level. Up to 13 centers chose to participate in the research. The study focused on families with children with high interparental conflict, who attended the FVC by judicial referral. Interparental conflict was measured considering hostility, detachment and escalating distress through the judicial referral report and interviews conducted by the professionals of the FVCs (case history). The inclusion criteria for the study were as follows: the parents were legally separated, did not live in the same household, did not present an active restraining order for gender-based violence, and did not have severe pathological disorders. The FVCs obtained permission from the relevant public administrations to carry out the program. The FVC staff invited the users who met these conditions to participate in the Egokitzen program. The interventions of the study took place between 2016 and 2019.The research used a quasi-experimental study. Participants were divided into the intervention group (IG) and the wait-list comparison group (CG). Descriptive information was collected from the parents by the FVC staff and data of the main variables of the study through the tests completed by the parents. Both groups of parents (IG and CG) completed the pre- and post-tests, and the IG group completed the questionnaires after 6 months and 12 months.We used the following instruments to measure the symptomatology of parents and children.Symptoms Checklist (SCL-90  in the SThe Child Behavior Checklist (CBCL ) was appThe final sample of the study included 197 people. Initially, the FVCs contacted 1538 users who met the above-mentioned conditions to participate in the study. After explaining the program, 450 people (29.3%) showed an interest in participating, of whom 382 (84.5%) attended a personal interview.From the initial sample of 382 participants who, after the interview, signed informed consent and completed the evaluation protocol, 112 were part of the CG, and 270 participated in the IG. Of the 112 control participants, 73 people (attrition = 35%) completed the post-treatment questionnaire. Of the 270 intervention participants, 197 (attrition = 24%) responded to the test after the intervention. Of the 197 people, 105 completed the questionnaire at six months and 69 also completed the questionnaire 12 months after the intervention see . Some ofOf the parents who participated in the study, 61% were mothers see . The timThe study was approved by the Ethics Committee of the University of Deusto (ETK-7/16-17). Participants were informed of the study from the outset and signed the consent document. None of the people involved in the intervention received an incentive to participate in the study.The cost-benefit analysis was carried out in three consecutive phases see . In the The resources required to carry out the training of the FVC workers are the trainers who taught the training course, trainers’ transportation, course materials, and course participants (FVC workers), considering this as an opportunity cost because while receiving the formation, they were not carrying out other normal activities of their position. The resources required to carry out the interventions are: FVC professionals, materials , and administration services for program management. In the second phase, the effects achieved through the implementation of the Egokitzen program were valued. First, the main benefits of the program were identified, and then, quantified and, thirdly, valued in monetary units. In the case of the Egokitzen program, the identified benefits were a statistically significant improvement in the parents’ depressive symptomatology and the children’s anxiety when the parents’ global symptomatology improved, as can be seen in Martínez-Pampliega et al. . Situatip) exceeded the standardized score (T) of 18.2 in mothers and 14.04 in the fathers were considered to have high levels of depression. This coincides with the 85th centile of those standards. Concerning child anxiety, we observed how many IG parents’ global symptomatology improved. Based on weighted data from the global symptomatology subscales included [The effects were then quantified. Based on the parents’ depressive symptomatology, a prevalence analysis was performed to observe how many IG parents’ depressive symptomatology improved or worsened compared to the control group. Based on the Spanish population standards of the SCL-90 , we estaincluded , we estaThird, the effects were valued in monetary units. Several studies have calculated the costs of depression in different countries, through the savings both of direct  and indirect (loss of productivity) costs produced by a reduction in symptomatology ,24. In tThe final impact of the program was determined in the third phase taking into account the benefits achieved and the costs required. This shows whether the funds invested in the Egokitzen program had a positive economic impact on society.Finally, a sensitivity analysis was carried out to analyze the effects of the global outcome of the project, changes both in the costs and in the estimation of benefits.First, the costs were calculated. p) had a score higher than the 85th centile (T) (18.2 in the mothers and 14.04 in the fathers), indicating very high depressive symptomatology, to determine whether they improved after the intervention, and on those parents where the opposite occurred and they worsened after the intervention (p < T at pre-test and p > T at post-test). The prevalence analysis in To assess the benefits of the program economically, first, a prevalence analysis was performed. The previous study  indicateAs discussed in the methodology section, the study of Salvador-Carulla et al.  was usedConcerning improvements in children’s anxiety when the parents’ overall symptomatology improved, first, we determined which parents could be considered to be suffering from a very high overall symptomatology  at pre-test and a score below that at post-test; that is, their symptomatology improved. Once the families where parents’ improvement were identified, we proceeded to study how many children improved their anxiety symptoms between pre-test and post-test, and how many worsened. As explained in the methodology section, data from the study of Ander Gustavsson et al.  were useWhen performing the sensitivity analysis, the assumptions used in the study were modified to make them more demanding both for the costs and benefits.As noted in the costs section, the cost with the greatest impact is that of human capital. For the study, in the cost of staff per hour, it was considered that the intervention was carried out by FVC professionals within their working day. In the absence of FVCs, this kind of service should be provided by private cabinets of psychology, whose cost is higher than the one in the FVCs. In this case, the cost of personnel would be twice as much. Even so, the result would still be positive. For each euro invested, 1.66 would be recovered see .On the side of the benefits, assuming that the negative effects of depressive symptomatology lasted only one year instead of two, and those of anxiety 3 months instead of half a year , the result would still be positive. For each euro invested, 1.55 would be recovered see .The objective of this study is to carry out an efficiency analysis of the Egokitzen program through a cost-benefit analysis, to determine whether the positive impact on the internalizing and externalizing symptomatology generated by the program leads to a positive economic impact on society. The results obtained show that the program is not only effective, as it achieves a positive impact on the symptomatology of both parents and children, but is also efficient. The economic valuation made of the profits obtained exceeds the costs of carrying out the intervention, obtaining a return of EUR 3.10 for each euro invested in the program. According to the analysis carried out, this means that by investing 1 euro in the Egokitzen program, the public administration and society save 3.10 euros in future interventions through medical costs and productivity losses.The study has clear practical implications. It is shown that this program is not only beneficial in terms of improvement of participants’ symptomatology but also in economic terms, as it represents net cost savings. This conclusion is relevant to the public administration working with budgetary constraints, specifically for those managers who decide which programs should be supported. Cost-benefit analysis can help them in this decision-making.It also has implications for businesses and society, as, on the one hand, the symptomatology of citizens improves, thereby reducing sick leave, improving worker productivity, and contributing positively to the competitiveness of companies and organizations.Third, the program has implications for FVCs because during the time spent by professionals , they would have carried out monitoring or accompanying activities not aimed at reducing the psychological impact of divorce. Therefore, there is an improvement in the utilization of the available resources. This has also meant greater recognition and added value for the centers and their professionals.Fourth, variables such as parents’ age, gender or the support of their intervention group do not appear to be related to the results obtained during the study. Therefore, groups can be mixed in gender and age. Participants, indeed, were very satisfied with the process and the professionals of the FVCs applied the protocol without difficulties. FVCs will continue offering it as an intervention tool. This is important in order to scale up the program to new organizations.As mentioned in the introduction, mental health and well-being are part of one of the SDGs established by the United Nations. This objective is even more important in the current era, because due to COVID-19, the number of people affected by symptoms of depression or anxiety is growing and, in people who already had symptoms, these are worsening . In addiThe Egokitzen program, apart from achieving good results in the community context showing a reduction in parental symptomatology and, indirectly, of child symptomatology, is economically efficient. The program achieves a return of EUR 3.10 for each euro invested in it.


Health research frequently requires manual chart reviews to identify patients in a study-specific cohort and examine their clinical outcomes. Manual chart review is a labor-intensive process that requires significant time investment for clinical researchers.This study aims to evaluate the feasibility and accuracy of an assisted chart review program, using an in-house rule-based text-extraction program written in Python, to identify patients who developed radiation pneumonitis (RP) after receiving curative radiotherapy.A retrospective manual chart review was completed for patients who received curative radiotherapy for stage 2-3 lung cancer from January 1, 2013 to December 31, 2015, at British Columbia Cancer, Kelowna Centre. In the manual chart review, RP diagnosis and grading were recorded using the Common Terminology Criteria for Adverse Events version 5.0. From the charts of 50 sample patients, a total of 1413 clinical documents were obtained for review from the electronic medical record system. The text-extraction program was built using the Natural Language Toolkit Python platform . Python version 3.7.2 was used to run the text-extraction program. The output of the text-extraction program was a list of the full sentences containing the key terms, document IDs, and dates from which these sentences were extracted. The results from the manual review were used as the gold standard in this study, with which the results of the text-extraction program were compared.Fifty percent (25/50) of the sample patients developed grade ≥1 RP; the natural language processing program was able to ascertain 92% (23/25) of these patients . Furthermore, the text-extraction program was able to correctly identify all 9 patients with grade ≥2 RP, which are patients with clinically significant symptoms . The program was useful for distinguishing patients with RP from those without RP. The text-extraction program in this study avoided unnecessary manual review of 22% (11/50) of the sample patients, as these patients were identified as grade 0 RP and would not require further manual review in subsequent studies.This feasibility study showed that the text-extraction program was able to assist with the identification of patients who developed RP after curative radiotherapy. The program streamlines the manual chart review further by identifying the key sentences of interest. This work has the potential to improve future clinical research, as the text-extraction program shows promise in performing chart review in a more time-efficient manner, compared with the traditional labor-intensive manual chart review. After the patient cohort is identified, data abstraction begins, and a number of patient variables can be collected ..9].Grade 0No RP presentGrade 1Asymptomatic; clinical or diagnostic observations only; intervention not indicatedGrade 2Symptomatic; medical intervention indicated; limiting instrumental activities of daily livingGrade 3Severe symptoms; limiting self-care activities of daily living; oxygen indicatedGrade 4Life-threatening respiratory compromise; urgent intervention indicated Grade 5DeathThe study population included a sample subset of 50 patients, from those who received curative RT for stage 2-3 non–small cell lung cancer from January 1, 2013, to December 31, 2015, at British Columbia (BC) Cancer Kelowna. The sample subset was designed to represent the proportions of RP grades in the literature -8,10,19.Patients who underwent surgery after radiation treatment were excluded. Patients who received palliative radiation and patients with small cell lung cancer were excluded.A manual chart review was completed by reviewing patient charts from the institutional EMR at BC Cancer Kelowna. The manual chart review results served as the definitive diagnosis, with which the assisted chart review program was compared. In the manual chart review, RP diagnosis and grading were recorded using CTCAE version 5.0  9]..9].pneumonitis, radiation pneumonitis, radiation induced lung injury, and fibrosis were used as key terms for the assisted chart review. These key terms were chosen by the radiation oncologist contributing clinical expertise in this study, and they represent terminology that a physician would use to identify RP in dictated reports. The output of the text-extraction program was a list of full sentences containing the key terms, along with the document IDs and dates from which these sentences were extracted. The text-extraction program was designed to search through all the charts and extract the whole sentence that contained the key terms. If a sentence was extracted from a patient’s chart, the patient was identified as having RP. The text-extraction program organized the extracted information, identified the patients, and indicated the exact documents containing the key terms. The results from the text-extraction program were then compared with those from the manual chart review.The in-house text-extraction program was built using the Natural Language Toolkit Python platform . Patient charts were extracted from the BC Cancer EMR system and were subsequently formatted into the American Standard Code for Information Interchange text files to be compatible with the text-extraction program. From the charts of 50 sample patients, a total of 1413 clinical documents  were obtained for review. The reports from the BC Cancer EMR system were obtained by either direct conversion to text format documents or were printed in PDF and then converted to text format using the open-source Python Tesseract optical character recognition program. This step of obtaining and converting the documents to text format from the BC Cancer EMR system was necessary, as the text-extraction program input requires text format documents. Python version 3.7.2 was used to run the assisted chart review text-extraction program. The terms If the text-extraction program is shown to be feasible and accurate, a more expedited manual chart review can be performed using the results of the text-extraction program in future studies. Patients with no key terms identified in their charts will be designated as grade 0 RP, and no further chart review of these patients will need to be completed. For the patients identified by the text-extraction program to have RP, the sentences containing the key terms can be reviewed manually, first to confirm that these patients are correctly identified as having RP, and then to grade the RP severity in an expedited manner. Thus, there is an opportunity to improve the text-extraction program specificity during this sentence review process by correcting the false-positive cases.The comparison between the manual chart review and text-extraction program output was viewed and analyzed in 2 different ways: the first approach considered the diseased state to be grade ≥1 RP, and the second approach considered the diseased state to be grade ≥2 RP, with grade 1 RP classified as a healthy state as the patients with grade 1 RP had no clinical symptoms. The text-extraction program was designed to look for any grade of RP when searching through the patient charts, so this lends itself to being able to perform well during the first analysis. However, grade 1 RP is only visible radiographically and thus is not clinically relevant to a patient’s further care. Thus, we wanted to look at how well the assisted chart review system was able to identify patients with symptomatic RP. Statistical analyses were performed using SAS software version 9.4.The results of the text-extraction program used to identify patients with RP of any grade are shown in missed were found to truly have grade 0 RP but were incorrectly labeled as patients with RP because of human error in the manual chart review. If we correct for this human error, the sensitivity improves to 1.0 for the text-extraction program’s ability to identify grade ≥1 RP.The text-extraction program missed 2 patients with grade 1 RP. Upon further review, the 2 patients with grade 1 RP that the text-extraction program In our cohort, each patient’s chart consisted of an average of 28 clinical documents that make up their chart, with a range of 15 to 150 documents. The average time spent during the manual chart review of one patient’s chart was 30 minutes. Therefore, the manual chart review of the 50-patient cohort took 25 hours. In comparison, the assisted chart review text-extraction program processed the 1413 clinical documents and exported the results in <5 minutes.The use of the text-extraction program in this study would be to avoid unnecessary manual review of 22% (11/50) of the sample, including their electronic documents , as these patients were identified as not having RP and thus would not require any manual review. It will also streamline the rest of the manual review as key sentences with the key terms are identified, thus further reducing the number of clinical documents necessary for the manual review to confirm that the patient should be included in the cohort.The text-extraction program was able to identify patients with RP with high sensitivity but, unfortunately, low specificity. This can assist in the identification of a patient cohort of interest in a more efficient manner.The text-extraction program correctly identified 2 patients with grade 0 RP that the manual chart review incorrectly identified. Similar findings have been reported in the literature, where one study found that their automated chart review outperformed their manual chart review as the human reviewer missed the correct classification on manual evaluation of the chart . TherefoThe utility of the text-extraction program in this study would be to perform a rapid scan of a larger data set of documents and avoid unnecessary manual review of many of the non-RP patient charts. The program is able to use key terms, such as RP or fibrosis, to return a list of patients with those terms in the patient charts. This will significantly cut down on the number of charts that the manual review will include. This is mainly because of the fact that even if a patient does develop RP, most of their charts do not include any indication of their diagnosis. The computer program organizes the extracted information into which patient and which exact chart, thus further reducing the amount of chart review that is necessary to manually review to confirm that the patient should be included in our cohort.The end goal of using text-extraction programs to perform chart reviews is to save the researcher time and effort of combing through patient charts to form a cohort in which to begin studying a clinical outcome. Our text-extraction program was able to output its results in <5 minutes compared with the 25 hours it took the manual chart review control to create the RP cohort.A limitation to implementing this assisted chart review program is its current high false-positive rate, leading to unnecessary chart review of patients with no RP. The development of automated chart reviews must consider the balance between NLP program accuracy (no diseased cases missed) versus the amount of time saved by confidently eliminating true RP grade 0 patients in the review. Designing the key terms was an important process to balance the accuracy of the text-extraction program versus the time saved using the text-extraction program. Selecting broad key terms is important to capture all patients who may fall into our cohort; however, more specific key terms would better rule out patients not within the study cohort. Our goal was to maximize the sensitivity of the text-extraction program by including broad terms so as to not miss any patients with the diseased state initially, as the sentence output of the text-extraction program allows for a truncated chart review to improve the specificity. This means that the possible time saved in this feasibility study was less as more false-positive RP patients were identified. Future work is underway to improve the specificity of the text-extraction program with a larger sample.Another limitation of our work is the small sample size of 50 patients. This sample group was used as a proof of concept for our in-house developed text-extraction program. This study’s results will guide further refinement of the text-extraction program and validation with a larger sample of patients.The rule-based text-extraction program used in this study still requires human involvement in a number of steps. The clinical documents in the BC Cancer EMR system had to be obtained manually rather than automatically, which continues to pose a barrier in making chart review research as time efficient as possible.In addition, it is important to point out that expert opinions were necessary to identify the key terms to be used in the text-extraction program. This is not only another human involvement requirement but also indicates that the results are dependent on the quality of the expert. In addition, this makes the program less generalizable to other cohorts without a new expert to create the proper key terms for each specific cohort.Other studies have used NLP programs to assist with chart reviews in many scopes of medicine, including respirology, cardiology, and neurosurgery, and now our cancer research to identify patients who developed RP -24. NLP In conclusion, the NLP-based text-extraction program used in this study is a feasible and valuable method for identifying patients who developed RP after curative radiotherapy. First, the text-extraction program helped save chart review time by completely eliminating patient charts identified with grade 0 RP. Second, the text-extraction program extracted key sentences from patient charts and allowed for an efficient review of relevant phrases, should this be needed to grade patients’ RP severity without having to peruse the rest of their charts. For example, in a quick scan, a researcher would be able to read only the sentences with the identified keyword in a patient’s chart instead of sifting through many full documents.The analysis revealed that the text-extraction program was capable of significantly differentiating between diseased and healthy groups. Compared with the manual chart review of the 50-patient cohort that took 25 hours, the text-extraction program was able to process all the charts in <5 minutes and exported the list of patients that had RP mentioned somewhere in their chart.This work has the potential to improve future clinical research as the text-extraction program shows promise in performing chart review in a more time- and effort-efficient manner compared with the traditional manual chart review. The text-extraction program is available by contacting the authors (RR).


Planning for end-of-life (EOL) care in advance can enhance one’s quality of life at EOL. Culturally sensitive educational programs are needed in Chinese populations to enlighten the public and encourage advance planning due to a culture of death-denying and avoidance. This study describes the team’s efforts to develop and formatively evaluate a death education program designed for community-dwelling Chinese older adults. The program was designed based on the Knowledge-Attitude-Behavior Model, as a 2-session 3-hour program spreading over two days with 1.5 hours for each day. The content paid attention to discussing the importance of making plans for EOL in Chinese culture and discussing how to have death-related conversations with the family and health care professionals. In 2020, semi-structured interviews were conducted by phone with 12 health care professionals and researchers, and four Chinese older adults in China to obtain their feedback on program content and delivery. The directed content analysis method was used to analyze the data. Although they confirmed multiple challenges in conducting death education in China, such as family avoidance even if an older adult initiates the death-related conversation, health care professionals not feeling comfortable with such discussions, etc., they also felt the urgency and importance of delivering death education among older adults and in society as a whole. Detailed suggestions were categorized into relationship building, program preparation , multiple ways of recruitment, target population, length, various formats of content delivery, content , and general support from the public.


Improving the care of older adults in our healthcare system involves teams working together. As the geriatrics population rises globally, health science learners need to be prepared to work collaboratively to recognize and treat common conditions in geriatrics. To enable workforce preparation, the Institute of Medicine and the National League for Nursing emphasize the need to implement interprofessional active learning activities for undergraduate healthcare learners at academic medical centers.The Geriatrics Champions Program was a team-based learning activity created to meet this task. It was a 24-month program, repeated twice, that impacted 768 learners and 151 faculty from medicine, occupational therapy, physical therapy, nursing, social welfare, psychology, pharmacy and dietetics. Each class was intentionally divided into 20 interprofessional teams that met four times annually. Each session focused on one geriatrics domain. The objectives were centered around the specific geriatrics competencies for each health profession, divided into the eight domains written in the “American Geriatrics Society IM-FM Residency Competencies”. Evaluation consisted of individual and team Readiness Assessment Tests (iRAT and tRAT). Surveys were also used to collect feedback using a Likert scale. Wilcoxon signed rank tests were used to compare iRAT and tRAT scores. Other analyses identified characteristics associated with tRAT performance group (Unpaired t-tests) and tRAT performance on the raw scale (Pearson correlation). Paired t-tests using a 7-level Likert Scale measured pre-post change in learner knowledge.p < 0.001). Teams were more likely to score 100% on the initial tRAT attempt if more team members attended the current session (p < 0.001), more health professions were represented by team members in attendance (p = 0.053), and the team had a better track record of past attendance (p < 0.01). In the post-program evaluation, learners felt this program was helpful for their career preparation in interprofessional geriatrics care.Student tRAT scores were 30% higher than iRAT scores (Learners understood that teams performed better than individuals in the care of older adults. Feedback from the learners and faculty was consistently positive and learners felt better prepared for geriatrics care. The program’s benefits may extend beyond individual sessions. Preparing the workforce to work collaboratively in the care of older adults is an essential component to our healthcare system. Teaching geriatrics competencies to health profession learners is a necessary, but challenging, task. There is both a limited amount of time dedicated to teaching geriatrics and a limited proportion of clinician educators proficient in teaching geriatrics care. In geriatrics, interprofessional education (IPE) models have been used internationally for geriatric syndromes. Dementia care management, teaching the “5 M” Framework outlined by the American Geriatrics Society, palliative care, falls risk assessments, and many other geriatric syndromes have been presented using an IPE model –4. Most As our population matures, it becomes increasingly important for learners to recognize geriatrics syndromes, understand who their team players are and treat accordingly with optimal outcomes. The Institute of Medicine’s 2003 report and the National League for Nursing’s 2015 report emphasized a need to implement interprofessional active learning activities for undergraduate healthcare learners at academic medical centers . The GerThe Geriatrics Champions Program (GCP) was a five-year program from 2011 to 2016. The aim of the program was to create an active-learning platform to train interprofessional learners synchronously for geriatrics care in the workforce. It was funded by a Geriatric Academic Career Award by the Health Resources Services Administration .The GCP was a five-year grant; the first year was program and curriculum building followed by a 24-month curriculum, which repeated twice. Individual programs determined whether the course was required or elective. Each program was represented by a core group of one to two faculty. Faculty were chosen based on interest in IPE and prior collaborations with the team. The core faculty group of 6–10 met monthly to ensure the curriculum met their learners’ needs, to be the content expert of relevant topics and to ensure the content met current healthcare guidelines. Core faculty also helped recruit the cadre of over 100 faculty to be team facilitators each year. Faculty were assigned a small student group to work with through the year.They were instructed to not lead the group, but guide the group if astray, allow for equal participation and spark discussion during lull moments. Faculty participated in required faculty development sessions annually to review the format, goals and responsibilities of each session. The lead facilitator was the Principal Investigator of the grant. She has been the lead facilitator for the six-profession interprofessional geriatrics clinic and brought her IPE skills to this program. Faculty were given a certificate of appreciation each year.The target audience was learners from Family Medicine residency programs; undergraduate medical clerkships in Family Medicine; Nurse Practitioner tracts for Adult Gerontology and Family Nurse Practitioner programs; Master’s programs in Social Welfare, Occupational Therapy and Psychology; undergraduate Dietetics program; and doctorate programs in Physical Therapy and Pharmacy.GCP content had thirty objectives divided into eight domains: Special Considerations in Geriatrics Care ; Medication Management; Cognitive, Affective and Behavioral Health; Complex or Chronic Illnesses in Older Adults; Palliative and End of Life Care; Hospital Patient Safety, Transitions of care; and Ambulatory Care. The domains were guided by the American Geriatrics Society Internal Medicine-Family Medicine (IM-FM) Residency Competencies written in 2010, and expanded to fit each profession’s geriatrics curricular competencies . In addiIt was expected that program objectives would be better achieved through both dialog with students from other health professions and greater exposure to the GCP curriculum. Accordingly, as seen on Table The GCP format was a large classroom setting led by the lead facilitator. Learners were divided into 20 small groups of 6–8 learners and 1–2 faculty per group. Learners were intentionally divided into 20 interprofessional teams with at least three professions in each group. Four, 2.5-h sessions were held annually, each focused on one domain. Learners named their teams and stayed within their teams throughout the academic year. The members within each team changed each academic year due to graduation or course completion. Due to the size of the course, the online platform Blackboard® was used.Finally, after each session, learners were given an evaluation consisting of ten Likert scale and open-ended questions of the value of the session, achieving the session’s objectives, the perceived relevance of IPE to their education and the perceived benefits of IPE to their future practice. These evaluations were regularly reviewed, discussed by program designers to consolidate lessons learned but no formal qualitative analysis was completed.Learners were given 60 min of online pre-work before each session which included reviewing articles, online resources, and/or patient cases. Upon arrival to each session, learners were given an online individual Readiness Assessment Test (iRAT) valued at 10 points using the Blackboard® platform. The iRAT consisted of ten multiple choice questions based on the prework material. Learners were shown only their total score, not which items they missed. Immediately following the iRAT (10 min), learners then completed a team Readiness Assessment Test (tRAT) using Immediate Feedback Assessment Technique (IFAT) scratch-off forms. The tRAT valued at 40 points, tasked each team with answering the same iRAT questions, but as a team. Each team member led the team to answer at least one question per session. Each faculty facilitator ensured that all professions’ perspectives were shared for each question. The team IFAT forms had a designated key whereby a star was revealed once the correct answer was scratched off. These scratch-off forms allowed the teams to know if their team answer was correct by selecting the option revealing the star. If an answer was incorrect, they chose an alternate answer until the correct answer was selected. Four points were earned for the correct first attempt; two points if the correct answer was chosen on the second attempt, one point if the correct answer was chosen on the third attempt. The iRAT and tRAT scores were converted to percentages for consistency across scales and compared using Wilcoxon signed rank tests.Since all students knew their iRAT score, but not which items they got wrong, they could use their team’s reasoning to understand their errors. When all teams completed and submitted their IFAT forms (15 min), the course leader engaged all learners to review any questions as a large group that they found challenging. Teams voiced their support for or against an answer, which encouraged healthy discussion (15 min).After a short break, the second half of the session consisted of an audio, video or live patient application case relevant to the session’s objectives. Small groups were asked to rotate the roles of a team leader and team speaker to be used for the discussion. Five multiple choice questions were designed to encourage interprofessional collaborative discussion. After ten minutes of discussion, simultaneous reporting was used for each question with teams prepared to defend their answer when others refuted it. This activity allowed learners to apply their pre-work knowledge to the current case, explore their profession’s role in the case, listen to how other professions could help with the patient’s care, and ultimately gather the necessary information to explain their interprofessional care management choice. After the case discussion, five minutes were allocated for online feedback completion.Points were awarded for each activity towards their team’s final total score. Prizes were given at the end of the academic year to the team with the highest number of points. Participants who attended all four sessions and completed the program evaluation received a certificate of completion for their professional CV.Geriatrics Champions Program coursework: all prework, coursework, iRAT and case information is available electronically by contacting the corresponding author directly.Interprofessional Faculty: Interprofessional leaders to meet monthly and discuss profession-specific curricula to be included within each of the eight domains. Although the interprofessional faculty together compiled the pre-work for each session, the profession who most closely matched the question’s competencies led the session’s pre-work development.Certificates and Promotion letters: For learners and faculty partners.Online platform: Blackboard®.IFAT forms: commercially available.Materials required to run the program included the following:While TBL has been associated with better learning outcomes compared to other large group teaching methods, , 10 therr for Wilcoxon signed rank tests and Cohen’s D for unpaired t-tests, reflect the guidelines from Cohen [The iRAT and tRAT scores were converted to percentages for consistency across scales. Because tRat scores, unlike those for the iRAT, demonstrated ceiling effects and were clustered near top of the score distribution, iRAT and tRAT scores were compared using Wilcoxon signed rank tests. Pearson correlation was used to identify characteristics associated with team performance on the raw tRAT scale. Unpaired t-tests were used to identify pre-post knowledge gains and differences between the highest performing teams and all other teams. The highest performing teams were defined as those scoring 100% on the initial tRAT attempt. This performance threshold served to classify teams by performance group while controlling for the number of tRAT attempts. All statistical analyses used two-tailed tests with a = 0.05 for a 95% confidence level. In all cases, the session-year was the unit of analysis. Effect size interpretations, based on om Cohen .n = 217). Tests were repeated on a restricted sample including only those teams with no members that scored 100% on the iRAT (n = 142). Results reported are from the restricted sample, though results are robust across both samples.The analysis plan was adjusted post hoc to exclude both learners who did not make a good faith effort to complete the iRAT  and teams containing another member scoring 100% on the iRAT . Team performance on tRATs was correlated with cumulative team attendance percentage , team attendance percentage , team attendance , average iRAT score among all attending team members , and maximum individual iRAT score among team members . The significance of average and maximum iRAT scores among team members are consistent with recent research finding positive associations between team performance and higher performing individuals on the team [Table the team . All assnt}x¯ = 6% were lop < 0.01), higher cumulative attendance , higher attendance percentage , and a higher attendance . From Table p = 0.053, effect size = 0.28).From Fig. In the post-program evaluation, written comments (summarized in Table The Geriatrics Champions Program (GCP) was created to unite learners and faculty from seven professions for a workforce preparation course in geriatrics care. It was an interactive IPE program introducing geriatrics domains in a TBL format. Based on the feedback, it was a well-received IPE program that impacted 768 learners and 151 faculty over a five-year period. It will be interesting to see how these interventions impact actual practice in geriatrics care.A committed faculty contributed to the development and delivery of active learning methods in this geriatrics delivery course. TBL was an effective modality, allowing for efficient delivery of content to a large group annually. Feedback from the learners and faculty was consistently positive and resulted in multiple scholarly poster and podium presentations for learners and faculty.p < .001) of the Program over the five years. Five-year data demonstrated that scores increased approximately 9–12 points out of 40, or 30%, from the individual to the team assessment. The likelihood of higher performance may have stemmed from interaction and understanding of other professions. The 30% difference remains statistically significant with a large effect size even when the sample is limited to students who both make a good faith effort to complete the pre-work assignments and are not on a team with another participant that scored 100% on the iRAT [Learners understood that teams performed better than individuals in geriatrics care. The mean scores for the individual (iRAT) and team (tRAT) assessments differed significantly for each session  perform better in TBL settings . HoweverIdentify IPE gaps in the current healthcare system and ensure your program assists in narrowing or closing the gap. For example, interprofessional geriatric care was a common element missing in our curriculum amongst our four Schools. The GCP served to narrow the gap by providing a platform to teach geriatric interprofessional competencies to learners from all four Schools;Decide which professions you want to include in your program based on availability and needs, and be open to growth as others are interested;Develop a reliable cadre of faculty from each profession that share your vision. Allow their expertise to build your curriculum. Creating a welcoming environment where suggestions and constructive criticism are invited, is key to building lasting partnerships;Meet with your team regularly to share ideas, new content, benefits and challenges;Learn about the components of TBL and how you can incorporate the iRAT, tRAT and Interactive Activity into your IPE activity;Pick a location amenable to small group TBL learning activity discussion;Make the activity a win-win for all, including your learners and faculty. Be flexible in how each profession can be involved and let the faculty determine what fits with their learners and curricula, ;Collaborate with interprofessional faculty when making cases to ensure learners can identify their professional contributions to the case and pose thought-provoking questions to mimic a realistic healthcare system case;Foster a collaborative environment where scholarship is encouraged for learners and faculty;Create a peer evaluation system;Seek funding to support program development;Provide incentives for the learners to participate;Gain institutional support for long-term sustainability.Multiple lessons can be shared from this interprofessional geriatrics workforce preparation course:The GCP was a successful program that incorporated an active learning tool to allow our future interprofessional workforce to engage in geriatrics content they may not have received otherwise. With more programs like these, we can hope that evidence-based, interprofessional geriatrics care will be provided for our older adults. Necessary handouts and content can be obtained via the corresponding author.The GCP was an effective program to introduce our budding workforce to the value of interprofessional collaboration in geriatric care. In addition, the TBL format helped identify the benefits of team function. The 30% difference in iRAT – tRAT scores demonstrates to the learners, program leaders and the geriatrics community at large that team efforts consistently outperform individual efforts. Both current and past attendance are important to team performance, consistent with an IPE program where later sessions build upon and expand on content presented at earlier ones. We hope the GCP provides a model that can be replicated at other academic institutions to provide workforce training well-needed to care for our ever-expanding geriatric population.


Background: Public health agencies are increasingly concerned with ensuring they are maximizing limited resources by delivering evidence-based programs to enhance population-level chronic disease outcomes. Yet, there is little guidance on how to end ineffective programs that continue in communities. The purpose of this analysis is to identify what strategies public health practitioners perceive to be effective in de-implementing, or reducing the use of, ineffective programs.Methods: From March to July 2019, eight states were selected to participate in qualitative interviews from our previous national survey of US state health department (SHD) chronic disease practitioners on program decision making. This analysis examined responses to a question about “…advice for others who want to end an ineffective program.” Forty-five SHD employees were interviewed via phone. Interviews were audio-recorded, and the conversations were transcribed verbatim. All transcripts were consensus coded, and themes were identified and summarized.Results: Participants were program managers or section directors who had on average worked 11 years at their agency and 15 years in public health. SHD employees provided several strategies they perceived as effective for de-implementation. The major themes were: (1) collect and rely on evaluation data; (2) consider if any of the programs can be saved; (3) transparently communicate and discuss program adjustments; (4) be tactful and respectful of partner relationships; (5) communicate in a way that is meaningful to your audience.Conclusions: This analysis provides insight into how experienced SHD practitioners recommend ending ineffective programs which may be useful for others working at public health agencies. As de-implementation research is limited in public health settings, this work provides a guiding point for future researchers to systematically assess these strategies and their effects on public health programming. Public health agencies are increasingly concerned with ensuring they are maximizing limited resources by delivering effective evidence-based programs to enhance population-level chronic disease outcomes –3. DespiIt is important to better understand de-implementation in public health settings for several reasons. First, public health departments have increasingly limited resources. An analysis of federal, state, and local public health budgets has shown that public health spending has fallen or remained flat over the last several years , 8. MakiHarris et al. offer a framework for evidence-driven decision-making to guide disinvestment of ineffective healthcare practices and ensure effective resource allocation in healthcare –11. The Additionally, researchers studying de-implementation in healthcare can learn from conceptualizations of the policy termination process developed by political scientists and applied to resource allocation considerations in a number of fields –17. PoliDespite these works on de-investment in healthcare and policy termination in government, there remains a gap in the literature focused on de-implementation in public health settings , 18–22. The purpose of this analysis is to identify what strategies public health practitioners perceive to be effective in de-implementing, or reducing the use of, ineffective programs.In Spring 2019, we selected eight states to participate in key informant interviews who previously participated in our national survey of state health department (SHD) chronic disease practitioners on program decision making. This analysis examined responses to the question: “What advice do you have for others who want to end an ineffective program.” Interviews were audio-recorded, and the conversations were transcribed verbatim through a transcription service, Rev.com . All travia email to inform them of the study and that the study team would be contacting their programmatic staff. This communication was also used to invite the chronic disease director to participate and ask for any staff whose participation would be beneficial. One of the SHD chronic disease directors requested that we not contact their employees and the study team selected an alternative state to recruit from instead from the same quartile of perceived mis-implementation frequency.We selected participant states for geographic and population size representation and based on their self-reported frequency of mis-implementation from our 2018 national survey . We selevia email and two times via phone. Participants were offered a $40 amazon gift card or a $40 donation to a public health charity of their choice.After chronic disease directors were informed of the study, we reviewed position responses from the initial national survey to invite SHD chronic disease employees with position types likely to be involved in and have broad knowledge of how decisions were made about continuing and ending programs. We first emailed SHD employees from selected SHDs who responded to our previous national survey. Snowball sampling was then used as we asked each interview participant to recommend others at their SHD who would be a good fit for the interviews. When this did not yield a sufficient number of participants per state, we used the National Association of Chronic Disease Directors (NACDD) membership list to select additional potential participants from the selected SHDs. We contacted over 200 individuals to participate in our interviews; however, as anticipated, some did not respond to our repeated emails and attempts to contact by phone. Those who declined cited lack of time or inadequate knowledge on decision-making. We contacted potential participants to give them the opportunity to consent to schedule an interview or decline up to three times The interview guide focused on organizational, individual, and external factors that relate to decision-making about programs in SHDs. Specifically, the questions in this interview guide sought to elicit more about decision-making processes related to ineffective programs that continue in SHDs. We omitted the other type of mis-implementation, de-implementation of effective programs, because our 2018 national survey found that nearly 90% of SHD practitioner respondents believed the overwhelming reason for this type of mis-implementation was funding, of which SHD practitioners have limited control over . The intWe used a deductive approach for our thematic analysis in that we developed a codebook to guide the process. The codebook was developed from the interview guide and included nine parent nodes with several sub-nodes. For this analysis, all responses to the question about “…advice for others who want to end an ineffective program” were coded into one node. Interviewers or another team member checked transcripts against the recordings and de-identified the transcripts. We coded and analyzed transcripts in NVivo 12 . We rand(1) collect and rely on evaluation data; (2) consider if any of the program can be saved; (3) transparently communicate and discuss program adjustments (4) be tactful and respectful of partner relationships; (5) communicate in a way that is meaningful to your audience.We completed interviews with 45 state health department practitioners  from eigParticipants frequently discussed how having evaluation in place helps end ineffective programs. Three major sub-themes emerged from this theme that included having an evaluation process in place from the beginning of a program, making sure to collect quantitative and qualitative data, and using data to justify why the program needs to end . The mosParticipants discussed how evaluation needs to be in place at the beginning of the program, including the importance of having staff or contractors skilled in evaluation. Examples of interview responses include the following:Have an evaluation process along with the program planning process because if we do the implementation of the program and then expect an evaluation, that will never tell you where exactly that program was ineffective.” [Participant 1]“Set evaluation parameters upfront, to really do your due diligence and do formative evaluation too to know what are contextual factors in the target populations or community in which you plan to serve through this program.” [Participant 2]“Many participants highlighted not only collecting quantitative measures but also the importance of qualitative evaluation and anecdotal stories about how the program is working.Make sure you have your evaluation data and both quantitative and qualitative evaluation data.” [Participant 3]“Looking at the data, maybe gathering some anecdotal stories or qualitative data, but being really mindful of just looking at the outcomes before you make decisions.” [Participant 4]“Participants also highlighted how valuable and important having data from evaluation is when you are making the case for ending an ineffective program. Participants stated that evaluation data can be used to convince decision-makers that are higher up in the SHD and other stakeholders, who may have dissenting opinions of program effectiveness, that a program is ineffective and needs to end.Data. Show them the data. Save the data, use the data. If it's ineffective, you should be able to show that it's not working… lay out a logical argument that shows the money is being spent and the results aren't there.” [Participant 5]“Those impacted… may not like it, and they may not necessarily 100% agree with the decision, but when you have the data and you have the evidence to support the decision and you're open and honest about that, I think it makes the blow a little bit easier.” [Participant 6]“A second theme that arose from our interviews was to think carefully about whether a program must end or if instead it could be adapted. Participants advised others to thoroughly consider what effects ending the program will have throughout the SHD. Three major sub-themes emerged within this theme which were making sure that a program really needs to end, considering the ripple effects of ending the program, and considering if adaptations or alternatives are possible.Participants acknowledged that ending a program is often difficult and advised others wanting to end an ineffective program to take a hard look and confirm the program must end. They highlighted that others should thoroughly examine program inputs and see if the program can bring about the desired outcomes.Review all the inputs that are necessary to get the desired outputs are… if you go through that whole process and you see that there's nothing else that you can put in for inputs or resources you have or now you don't have available to you, if you're having to fudge it to make it just exist, but it's not going to be successful the way that it was planned to be, probably want to consider ending it.” [Participant 7]“Before ending it completely really looking at all of the data and do an evaluation of the program first to make sure that it's really ineffective, and also I mean definitely talking with stakeholders.” [Participant 8]“Be sure that the evidence is very clear as to why it's ineffective and to be sure that it's well documented and communicated.” [Participant 9]“Another sub-theme that arose was participants' advice to thoroughly consider the effects of ending a program. Interview participants stated that practitioners need to consider how the ripple effects of ending a program, even an ineffective one, may create gaps in services that could affect other programs or stakeholders. It was advised that when working to end an ineffective program that practitioners develop a plan for how to address any undesirable effects that they identify.[Consider] are there some ripple effects that can be seen based on the discontinuation of that program? … Even ineffective programs that can change your partnerships, they can affect inadvertently other programs.” [Participant 10]“Make sure that you've considered all things in programs, most importantly who the program is going to impact, and you brought in sort of their thought process as well.” [Participant 11]“Make sure you've talked to the stakeholders involved to assess what impacts would be felt and then mitigate that impact.” [Participant 3]“A third identified sub-theme was to consider if adaptations or alternatives are possible to turn an ineffective program into an effective one. Participants advised that adapting programs where possible to make them effective can help to preserve programs and to prevent gaps in services. If a program really must end, participants advised practitioners to present alternatives to the ineffective program.Decide whether the program really needs to end or if it is something that really just really need some minor tweaks.” [Participant 12]“Find an effective alternative, or suggest an alternative as a solution to the problem that the ineffective solution was originally created for.” [Participant 13]“You've been thinking through if we redirect these resources, what could we do with it and what the benefit could be. It's also helpful strategy to have some thoughts and comments related to that.” [Participant 14]“A major theme from participants was to transparently communicate about programs and their adjustments. This theme related to keeping decision-makers regularly informed about the status of a program, soliciting their feedback, and documenting changes made to programs in attempts to make them effective. Three major sub-themes emerged on how to facilitate transparent communication, they included documenting and keeping track of attempts to make the program work, communicating and being transparent about programs from the beginning, and to listen and be open-minded.Participants advised documenting all efforts throughout the life of the program to make it work in order to show decision-makers how all options have been exhausted to make the program effective. One participant described it as doing “forensics” of how a program got lost along the way.Any time that we can document, ‘Here are the steps that we've taken to remedy the situation,’ that has always been very beneficial in garnering support from leadership to end a program.” [Participant 15]“Really document why this is an ineffective program to you and for at least a good six months or more… there really needs to be a lot of work shown and documentation of why it didn't work. Going forward you can also learn from all that and try to tweak things to make it even better the next time around.” [Participant 16]“Another major piece of advice was communicating with stakeholders and other staff throughout the life of the program so that there were no major surprises when it became clear the program was ineffective or so all can help to adjust the program along the way to make it effective.Always be transparent even from day one, long before you think it's gonna be ineffective. Stay in close communications and so there are no surprises…” [Participant 17]“I think communication, communication, communication. Certainly it is to engage the staff, engage the evaluator… So, I think it's all just really good transparent communication” [Participant 18]“Finally, participants often mentioned listening and being open-minded in their conversations about programs to get a better understanding of a program's objectives and also to get those on board with ending the program who may have a personal attachment to programs.When we go into those types of meetings where a decision has to be made, it's important to have a fine line of being prepared and coming with the data, but also being humble and open-minded that this program or this approach might be near and dear to someone's heart.” [Participant 6]“I would say to listen first. I listened for a while and then went to a few meetings and was going, “This just doesn't feel right to me.” Listen and understand what the goals and objectives are. Listening is first. Then ultimately, I did a bunch of research. [Participant 19]Partnerships and their importance in ending ineffective programs were a major theme in our interviews. Participants highlighted how partners played a huge role in continuing or ending ineffective programs. It was recommended to be very tactful and respectful in how you work with partners when ending an ineffective program because often SHD practitioners want to work with partners again on other programs. Three major sub-themes emerged on how to be tactful and respectful of partner relationships, including to prevent burning bridges with partners, let partners know you value them, and finding ways to support partners who do good work.A major sub-theme in the ending of ineffective programs was to prevent burning bridges with partners that practitioners may want to work with again. Participants frequently spoke about speaking with partners about why a program needs to end in a very delicate way to preserve their working relationship. It was advised to give partners the full reasoning and rationale for the program ending to help them best understand.Most individuals, in my experience, when you can show them the evidence and the data and show them what they're looking at, hear what they have to say, and if that's still the decision to discontinue at the end of the day, they are more likely to still come to the table on other opportunities that you may have… sometimes it has to be [program must end], but if it's not communicated well, you really can burn an opportunity or future partnerships.” [Participant 6]“It may be that you never partner with them again, but I think, depending on how that ends, kind of signifies your relationship not only to that partner but to other partners that then may get worried about what the future holds for them.” [Participant 20]“One way that participants suggested to preserve good relationships with partners was by letting them know their work and input are valued. Partners may be very invested in a program and it is key to remind good partners that their work is appreciated despite the fact the program needs to end.You need to be very, very sensitive in how you communicate your rationale to the partners…It was very important to frame it in the way that, oh, you know, this program has helped many kids at the local level. All those partners that love the program so much, we need all of them to be on board with this shift as well.” [Participant 21]“Find a way to manage the personal relationships, to make it clear that you value the other person's input, you value what they do, and you're invested in their success.” [Participant 13]“Another sub-theme was to preserve partner relationships when ending an ineffective program was to find ways to support partners that do good work even if the current program engagement with them must end. Participants suggested engaging good partners in other SHD programs or offering to support them in other ways.Show how you can help them [partners] be successful even if that particular program is ending.” [Participant 13]“There needs to be a way back in or a way of building and supporting some of our partners that typically do good work.” [Participant 20]“The final major theme we identified in participants' advice was to communicate in a way that is meaningful to the audience. In order to best communicate why a program needs to end, participants advised both understanding the audience of the message and then deciding which way would be most useful to communicate with them. Three major sub-themes emerged on how to best communicate meaningfully which was to tell stories, use written and concise communication, and to present relevant findings in understandable ways.Participants advised using stories to help convey why a program needs to end. They stated that using stories, with data, can be more memorable and persuasive than presenting data alone when convincing a decision maker that a program is ineffective and needs to end.I've always taken away from listening to legislatures is the thing that sways them the most is the personal stories… personal stories really amplify that data.” [Participant 22]“Speak up, use your data, tell your story.” [Participant 23]“Participants also spoke to the value of concise written communication for decision-makers who are often busy and don't have time to read extensive reports. They noted that concise one-pagers can be useful for this purpose. To develop this one-pager, participants advised “putting yourself in their [decision-makers'] shoes” to identify information that is pertinent to share with the decision-maker and what could be left out to honor their very busy schedules.What's the best way to communicate that? Typically something written short. I like to say, you know, decision makers typically only read one page but we try to give them the whole thing. I would say something written and very concise.” [Participant 24]“I'm a big proponent of a one-pager. I think as we go up the chain, people read less and less, so it might have to be a half-a-page, but any sort of document that concisely shows some background and then what sort of data's been collected, what sort of effort the state has made.” [Participant 15]“Finally, participants highlighted the importance of not just collecting evaluation data but also disseminating it in a way that anyone could understand. They stated that the information from evaluations needs to be understandable by those who make decisions about programs and to avoid letting the data collected go to waste.Present the results from your evaluations in a manner that is understandable by anyone… it used to be very common that it would have an evaluation that would just sit on a shelf and just because you were required by CDC to do that evaluation.” [Participant 25]“So going into those conversations prepared, ‘What would I want to know or what would I want to see if I'm going to be told this information?”’ [Participant 6]“Given the limited capacity and funds in public health agencies , it is cFindings from the present study align with conceptualizations of the public policy termination process and the healthcare resource allocation model by Harris et al. . DisinveDe-implementation of ineffective programs in public health departments is an emerging field. To date, research on de-implementation has been primarily focused on the clinical healthcare setting . A notabEvaluation has been found by other researchers to be an important facilitator of ending ineffective programs. In a survey of 376 local health department practitioners by Allen et al., lack of program evaluation was cited by 28% of participants as a top reason for why programs continued that should have ended . As the Other researchers have made a case for why an intervention should be saved where possible. A longitudinal study by Pinto et al. examined 379 HIV providers' experiences in New York City when the Centers for Disease Control and Prevention began to de-implement HIV prevention interventions that had been in place for 20 years in favor of more effective approaches that had better supporting evidence. The study found that the de-implementation of these interventions had some unintended negative consequences, including downsizing of staff due to decreased funding and a reduction in technical resources and assistance that had come from the intervention before it ended –31. A coTransparently communicating about program adjustments has been highlighted in other work about de-implementation of ineffective programs. Niven et al. conducted a review of articles describing the process of de-implementation, predominately in the clinical healthcare sector. The researchers developed a framework based on 109 studies on de-implementation . At the The importance of partnerships in the de-implementation of ineffective programs was also found in other studies. Pinto et al., discussed above, study found that de-implementation had negative impacts on interprofessional collaboration and partnerships as HIV service providers became more reluctant to send referrals to other agencies that do not engage in mutual referral-making –31. The Participants in this study cited the need for using communication strategies that resonate with decision-makers as an important factor in de-implementation of ineffective programs. The organizational behavior literature provides further insights into what types of communication are effective for an employee having their request granted by a superior. A study by Garner et al. examined what types of supervisor-employee communication interactions are most likely to result in the employee “getting what they want out of the conversation” , 34. SigThere are several limitations to this study including a limited sample size, participant self-censure, and participant self-report of ineffective programs and effective strategies for ending ineffective programs. As discussed in the methods, we selected states to invite participants from to be representative of various factors, including representation from each region of the US. It should be noted, however, that this study has limited generalizability, since it only includes representation from eight states. Additionally, due to the political nature of public health and its funding, participants sometimes self-censured their responses. Despite following IRB protocols and informing participants that their responses, state, and name would remain confidential, participants sometimes let us know they were leaving information out or asked us to redact pieces of information. Lastly, we relied on participants to self-report and define what makes an ineffective program and the effectiveness of the advice that they provided to end ineffective programs. We did not assess effectiveness or strategies objectively.To our knowledge, this is the first analysis to examine advice from public health practitioners on how to de-implement ineffective programs in public health departments. Understanding how public health departments effectively de-implement ineffective interventions is vital to make the best use of limited resources in public health. Many of the study findings are in line with the general evidence-based decision-making principles . These fMore research is needed on factors affecting successful de-implementation processes of ineffective programs in the public health setting. Additionally, this paper examined de-implementation in the state health department setting, but future research is needed on local health departments, which may reveal different effective strategies for de-implementing ineffective programs in this setting.This study adds to the limited body of research on de-implementation in public health agencies. This analysis provides insight into how experienced SHD practitioners recommend ending ineffective programs which may be useful for others working at public health agencies. It also provides a guiding point for future researchers to systematically assess these strategies to further the understanding of de-implementation and its effect on public health programming.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.The studies involving human participants were reviewed and approved by Washington University in St. Louis Institutional Review Board (IRB# 201812062). Written informed consent for participation was not required for this study in accordance with the National Legislation and the institutional requirements.ERW conducted interviews, coded interview transcripts, analyzed and interpreted interview themes, and led and contributed to writing the manuscript. PA and SM guided study design and interview guide development, coded interview transcripts, assisted in theme synthesis, and was a contributor in writing the manuscript. LF assisted in theme synthesis and was a contributor in writing the manuscript. MP managed the development of the interview guide, conducted interviews, coded interview transcripts, analyzed and interpreted interview themes, and was a contributor in writing the manuscript. SM-R guided study design and interview guide development, conducted interviews, coded interview transcripts, analyzed and interpreted interview themes, and led and contributed to writing the manuscript. RB guided study design and interview guide development and contributed to writing the manuscript. All authors read and approved the final manuscript.The findings and conclusions in this paper are those of the authors and do not necessarily represent the official positions of the National Institutes of Health or the Centers for Disease Control and Prevention.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.