Unmarried mothers who raise their children alone in South Korea experience various difficulties in maintaining their health. Improving the health of unmarried mothers, who are socioeconomically vulnerable, is important not only for them but also for the healthy growth and development of their children. We aimed to implement a natural wellness group program using an interactive real-time video platform for unmarried mothers and to verify its effects.This quasi-experimental study utilized a sequential explanatory mixed-methods design. The participants were unmarried mothers raising children aged 0–6 years. The quantitative data collection occurred from August to November 2021. For the qualitative data collection, semi-structured interviews were conducted with seven participants from the experimental group. The experimental group received eight 90-minute weekly online sessions, whereas the control group received no intervention. The outcome variables were physical and mental health, depression, anxiety, self-esteem, and parenting stress.A total of 42 unmarried mothers  participated. The Wilcoxon signed-rank test revealed significant post-intervention differences in mental health, depression, and self-esteem in the experimental group. Moreover, a generalized estimating equation analysis revealed significant differences in self-esteem levels between the experimental and control groups. Four themes identified through qualitative analysis revealed that the natural wellness group program using interactive real-time video not only increased mothers’ vitality and relaxation but also improved depression and anxiety, and had a positive effect on parenting stress.Natural wellness group programs using interactive real-time videos can help improve the psychological health of unmarried mothers with young children, a group that tends to be socially and psychologically marginalized. South Korea (hereinafter referred to as Korea) is a family-oriented country with a negative perception of childbirth outside of marriage . Accordi“Unmarried mother” (UM) is a general term for a woman who has conceived or given birth to a child outside of a legal and legitimate marriage . Owing tBased on this background, it is evident that UMs experience multifaceted difficulties –9. PhysiAmong UMs who are concerned about stigma, disclosure of identity is the reason they give up prenatal and postnatal medical support services . This suIn recent years, interest has grown in nature therapy as a noninvasive intervention for both socially vulnerable people and the general population , 12. It Outdoor as well as indoor nature-based activities can promote both physical and mental health . SpecifiInformation technology is well-developed in Korea, and most households can access the Internet using smart devices such as mobile phones . ProvidiBang et al.  proposedA mixed-methods design can help examine the physical and psychological changes in UMs, as it enables a quantitative and qualitative analysis of the changes generated by an intervention . This stThis mixed-methods study implemented a natural wellness real-time video group program for UMs in Korea and used a sequential explanatory design to analyze the program’s effects. We performed a quantitative analysis, and used the qualitative results as a complement. Quantitative data were collected and analyzed first, followed by qualitative data .https://cafe.naver.com/missmammamia, https://cafe.naver.com/imsanbu). Volunteers who wished to participate in the study were instructed to contact us directly.UMs aged 18 years and over who were raising a child aged 0–6 years in Korea and who voluntarily provided informed consent were recruited using convenience sampling. Individuals who were participating in another psychological intervention during the recruitment period or those who had taken antipsychotic medications in the past four weeks were excluded. The sample size was determined using the G*Power 3.1 program; for a repeated measures analysis of variance with an effect size (f) of .25, significance level of .05, power of .80, two groups, two time points, and an inter-time point correlation of 0.5, the minimum sample size was calculated to be 17 per group. Considering that less than 20% of the participants withdrew their participation or submitted careless responses in a previous study on forest therapy [Participants were assigned to either the experimental or control group according to their preference. No intervention was provided for participants assigned to the control group. The control group was informed that they would not participate in other psychological interventions during the study period. All study procedures were conducted online and there was no chance for interaction between the experimental and control groups, thereby reducing the possibility of contamination between groups.Online Health Promotion Program Using Urban Forests for Unmarried Mothers Living in Residential Facilities [The program chosen for this study was the cilities . The procilities .The natural wellness real-time video group program comprised eight sessions. Sessions 1 to 6 aimed to reduce anxiety and depression and improve self-esteem to promote health; session 7 aimed to promote health and lower parenting stress; and session 8 aimed to reduce anxiety and parenting stress . Four reThe SF-12 Health Survey developed by Ware et al.  and adapThe Korean Screening Tool for Depressive Disorders (K-DEP)  providedThe Korean Screening Tool for Anxiety Disorders (K-ANX)  providedThe Korean version of the Rosenberg Self-esteem Scale  was usedThe Korean-Parenting Stress Index 4th Edition Short Form  was usedQuantitative data were collected using a quasi-experimental pretest–posttest research design. Data were collected simultaneously from both groups from August 29 to November 21, 2021 using the same instrument via an online questionnaire (Google Forms). A baseline survey was conducted after obtaining informed consent and before beginning the program, and a post-intervention survey was conducted immediately after participants completed session 8 of the program. In the control group, two out of 21 participants who expressed willingness to participate in the study could not be tracked for follow-up and did not complete the baseline survey. Hence, a total of 19 participants in the control group completed the baseline and post-intervention surveys (0% dropout rate). In the experimental group, 21 UMs participated in the baseline survey, but one each was lost to follow up, preference for an offline program, and time conflicts after a change of schedule. Hence, a total of 18 participants completed the post-intervention survey (14.3% dropout rate) .Individual interviews for qualitative data collection were conducted over a period of 10 days, and each interview, starting from three weeks after the program’s conclusion, took about one hour. Three researchers conducted semi-structured interviews with seven participants via a virtual meeting platform. Individual interviews were added if the data required complementation or verification. The following interview questions were used: “What motivated you to participate in the program?”; “Tell me about your overall experience in the program”; “Did you experience any physical changes after participating in the program? If yes, please elaborate”; “Did you have any changes in your thoughts about yourself after participating in the program?”; and “Tell me about your relationship with your child after participating in the program.” The interviews were audio-recorded with consent from the participants.The study design was reviewed and approved by the Institutional Review Board at Seoul National University . Participants’ general characteristics were analyzed using frequencies, percentages, means, and standard deviations, and the normality of the variables was tested using the Kolmogorov–Smirnov and Shapiro–Wilk tests. Baseline homogeneity between the experimental and control groups was tested using the chi-squared test, Fisher’s exact test, independent t-test, and Mann–Whitney U test. Changes in the variables after the intervention were analyzed using the paired t-test and Wilcoxon signed-rank test. Changes in the outcome variables over time in both groups were analyzed using generalized estimating equation analysis .Individual interviews were transcribed, and participants’ weekly testimonials and program journals were also analyzed. The collected data were analyzed using the following three steps, as per the qualitative content analysis protocol proposed by Elo and Kyngäs . In the To ensure the rigor of qualitative research, the researchers assessed credibility, fit, auditability, and confirmability . To boosp = 0.033), a significant reduction in depression , and a significant improvement in self-esteem  after the intervention as compared to the baseline. The experimental group also showed higher health scores and lower anxiety and parenting stress scores after the intervention; however, the changes were not statistically significant   . However= 0.365) .I really felt comfortable while participating in the program, and that comfort helped me relieve and manage my stress, (which) again invigorated my life. … I now exercise, am more physically fit, and am more embracing of other people. I think it (the program) laid a foundation for me to do this. (Participant 5)2 = 2.26, p = 0.133) and anxiety   . HoweverI had thought it would be really different from going to a forest in person, but I could successfully imagine the forest environment; besides, there were a lot of supplies and hands-on activities. I did all that with my child, so just imagining itself was really… very meaningful. Having the mindset of trying to feel the forest by imagining was meaningful. (Participant 1)2= 4.04, p = 0.044)  . RegardiI thought it was a unique space when I could ignore other people’s thoughts, think only about myself, and spend time only for myself, and that was once a week…. I think it was an encouraging time for me… “You are a really good person. You are doing well.”I think I was reminded that I am a normal person like this. … You know, people who kept telling me to give up my baby for adoption or abort my baby are the ones who are not normal. I used to think that I was not normal. (Participant 4)(While making the terrarium) I wanted to make it pretty, but I couldn’t do that. It was very difficult for me to think that I was all thumbs. But after making it, I can feel accomplishment. I did it because I didn’t give up. So it was hard to make, but it was fun. I felt proud while making it. (Participant 7)2 = 1.73, p = 0.189)  . HoweverI think I was able to learn that by immersing myself in nature, my stress was reduced, and I was able to direct the energy I had spent on dealing with that stress to other parts of my life (laughs). It definitely helped with my parenting stress. … So when I am not stressed out, I would think, “Oh, my baby’s playing so I should go play with them.” And I would follow my baby around. I am more relaxed and have become more generous. (Participant 5)We conducted a natural wellness real-time video group program and used a sequential explanatory design whereby both quantitative and qualitative research methodologies were used for an in-depth assessment of program effectiveness.Quantitative analysis showed that the program significantly improved self-esteem in UMs. These findings are supported by previous findings  . The quaThe improvement of self-esteem through immersion and accomplishment is consistent with previous findings that hands-on gardening craft activities using nature-derived materials offer an opportunity to gratify the five senses. These activities also help participants feel a sense of accomplishment as they create and refine an artistic work, which contributes to improving self-esteem . They loWe created a group chat room on a social network service to allow the participants to share their plant growth progress and encourage one another even outside the program. Program activities were mainly conducted on weekends, but the chat rooms remained active during the week. As such, we used a user-friendly online platform to allow the participants to interact without temporal and spatial constraints, and to cultivate a sense of belonging and intimacy , 37 whilWhile there were no significant differences in health and depression over time between the experimental and control groups, there were significant post-intervention changes in health and depression in the experimental group. In other words, the control group did not show any significant changes in mental health between the baseline and post-test, while the experimental group showed significantly improved mental health and significantly reduced depression after the intervention (9.39 ± 9.35) compared to the baseline (12.33 ± 9.60).These results are supported by previous findings that show how forest therapy programs effectively reduce depression in institutionalized UMs . HoweverThe positive effects of the natural wellness real-time video group program were also observed in the qualitative analysis. The participants experienced positive emotions, such as “hopeful expectation,” “thrill,” “increased activity,” and “imagination” as a result of the program, which in turn reduced their depression and helplessness. The program helped the participants feel like they were immersed in nature, using various visual and auditory stimuli (forest sounds and showing forest screens). Since coronavirus disease 2019 risk is an important factor to consider when providing interventions, the findings of the present study confirm the usefulness of a natural wellness real-time video group program.Furthermore, our qualitative analysis showed that the UMs had been experiencing anxiety and parenting stress as a result of parenting and financial burdens, as well as concerns about the future, as they raised their child without assistance. However, the “anxiety” of many participants was relieved as they developed self-conviction and confidence through the program; by having more psychological stability, they developed “generous” behaviors and attitudes during parenting. This is consistent with a previous report that found that effective parenting requires mothers to demonstrate self-care, as opposed to an extreme form of self-sacrifice , and thaOne key implication of the natural wellness real-time video group program implemented in this study is that it offers an opportunity for UMs to face their lives and contemplate their inner selves, and motivates them to maintain a healthy lifestyle. This study presents valuable data for developing cost-effective interventions to address the health disparity affecting UMs—a socioeconomically disadvantaged population—by designing an online program accessible to the target population and utilizing easily accessible indoor and outdoor natural environments.This study has the following implications for nursing practice. Gardening crafts are one possible means to help UMs develop self-esteem by promoting a sense of accomplishment and utilizing nature-derived materials to reflect on and care for oneself. This could contribute to increasing self-esteem by boosting self-conviction. Moreover, expanding UMs’ social network through online and offline interventions is necessary to establish psychological stability. Social prejudice marginalizes UMs and intensifies their parenting burden, especially during a public health crisis. In turn, this aggravates negative emotions such as anxiety and parenting stress, emphasizing the need for national mental health policies that support UMs who are falling through the cracks.The participants were allocated to the experimental and control groups at their request. Thus, as this is a convenience-assigned study, the findings should be generalized with caution. Future studies should randomize group allocation to overcome this limitation. Additionally, the failure to recruit an adequate sample size might have been a barrier to obtaining statistical significance; therefore, subsequent studies should re-assess the program effectiveness with larger samples. Finally, we examined the effects of the program through a pretest–posttest design. Subsequent studies must conduct follow-up tests at three or six months post intervention to confirm the continuity of the program’s effect.In this study, we quantitatively and qualitatively examined the effects of a natural wellness real-time video group program for UMs. The quantitative analysis showed that the physical activities, metaphoric self-reflection, hands-on activities gratifying the five senses, activities using nature-derived materials that imparted a sense of accomplishment, and interpersonal interactions through group-based activities included in the natural wellness real-time video group program effectively improved UMs’ health, mental state, and self-esteem. The results of the qualitative analysis were largely consistent with those of the quantitative analysis, but statements from some participants revealed that participating in a natural wellness real-time video group program and the process of caring for nature helped reduce parenting stress by increasing UMs’ psychological stability. Using a mixed-methods design enabled us to shed light on the specific changes experienced by the participants, which could not have been examined in a quantitative analysis. We recommend that community health nurses and experts in various fields who deal with UMs actively utilize our findings.S1 Checklist(PDF)Click here for additional data file.S1 File(PDF)Click here for additional data file.S2 File(PDF)Click here for additional data file.S1 Dataset(XLSX)Click here for additional data file.


The Veterans Affairs (VA) Clinical Resource Hub (CRH) program aims to improve patient access to care by implementing time-limited, regionally based primary or mental health staffing support to cover local staffing vacancies. VA’s Office of Primary Care (OPC) designed CRH to support more than 1000 geographically disparate VA outpatient sites, many of which are in rural areas, by providing virtual contingency clinical staffing for sites experiencing primary care and mental health staffing deficits. The subsequently funded CRH evaluation, carried out by the VA Primary Care Analytics Team (PCAT), partnered with CRH program leaders and evaluation stakeholders to develop a protocol for a six-year CRH evaluation.The objectives for developing the CRH evaluation protocol were to prospectively: 1) identify the outcomes CRH aimed to achieve, and the key program elements designed to achieve them; 2) specify evaluation designs and data collection approaches for assessing CRH progress and success; and 3) guide the activities of five geographically dispersed evaluation teams.The protocol documents a multi-method CRH program evaluation design with qualitative and quantitative elements. The evaluation’s overall goal is to assess CRH’s return on investment to the VA and Veterans at six years through synthesis of findings on program effectiveness. The evaluation includes both observational and quasi-experimental elements reflecting impacts at the national, regional, outpatient site, and patient levels. The protocol is based on program evaluation theory, implementation science frameworks, literature on contingency staffing, and iterative review and revision by both research and clinical operations partners.Health systems increasingly seek to use data to guide management and decision-making for newly implemented clinical programs and policies. Approaches for planning evaluations to accomplish this goal, however, are not well-established. By publishing the protocol, we aim to increase the validity and usefulness of subsequent evaluation findings. We also aim to provide an example of a program evaluation protocol developed within a learning health systems partnership.The online version contains supplementary material available at 10.1186/s12913-023-09799-5. Published protocols for large-scale program evaluations have the potential to improve the quality and usefulness of subsequent evaluation results . This paIn brief, CRH aims to improve patient access to local outpatient care by implementing regionally organized contingency primary care and mental health staffing support to outpatient care sites, primarily through telehealth modalities. By contingency staffing, we mean time-limited staffing to cover staffing vacancies. To accomplish this, CRH aims to implement 18 regional VA hubs that employ contingency staff to remediate vacancies across VA’s more than 1000 outpatient sites, using primarily virtual care modalities . While tThe VA healthcare system is one of the largest managed care systems in the United States, providing care nationally to over 9 million military Veterans across all 50 United States and US territories  under thThe CRH program was initially implemented through VA-wide policy change. In 2018 the United States Congress mandated that VA focus on improving access to care through better support for staffing deficits –12 as wePublication of protocols for program evaluations is less common and less standardized than publication of protocols for randomized trials but may be increasingly important as healthcare delivery systems and their funders take steps to become learning health systems , 15. OurThe CRH evaluation is built on the initial CRH program goals, rationale, implementation strategies, and structure, and calls for documenting these over time. The protocol also calls for multi-method evaluation of the types, quality, and effectiveness of CRH services, as well as their cost and acceptability to stakeholders, and includes an initial formative evaluation focus followed by a summative evaluation of program effectiveness and value.The evaluation protocol relies on the RE-AIM  , 17 framThe objectives for developing the CRH evaluation protocol were to prospectively: 1) identify the outcomes CRH aimed to achieve, and the key program elements designed to achieve them; 2) specify evaluation designs and data collection approaches for assessing CRH progress and success, including return on investment relative to program goals; and 3) guide the activities of five geographically dispersed evaluation teams.The CRH program aims to implement centralized units, termed “hubs,” within each of the 18 VA administrative regions (VISNs). Hubs are expected to provide timely short-term staffing support for up to two years to VA outpatient sites within their regions that experience staffing vacancies in front-line primary or mental health care clinical teams. Hub clinicians rely mainly on telehealth care  to provide direct clinical primary and mental health care to enrolled Veterans in geographically dispersed VA outpatient sites within their regions. All outpatient sites in the region may request hub assistance; those sites that receive hub help are termed “spoke” sites. Program developers anticipated that many spoke sites would be in medically underserved rural areas.The evaluation is designated as quality improvement by and under the purview of OPC and its Executive Director and supports the CRH program’s requirements to report findings to its Advisory Board and to Congress. The evaluation is funded on a yearly basis by OPC, with an anticipated duration of six years (October 2020 through September 2026), when CRH stakeholders will make summative decisions about whether or how to maintain or expand the program.The VA system problem that CRH was developed to address was observed geographic variations in access to care as expressed in the US Congressional Maintaining Internal Systems and Strengthening Integrated Outside Networks (MISSION) Act for VA enacted in 2018 . The ratThe CRH program rationale builds on the concept of contingency staffing –23. ContAt baseline and during the initial implementation year, the main strategies used to implement CRH included mandating program elements  of CRH program leaders and evaluation team leaders to prioritize evaluation questions and to solidify scientific roles among the teams.The initial 2020 approved evaluation protocol included a logic model intended to link key program elements with needed evaluation data It also included specific aims built around the RE-AIM framework, , 17 withThe logic model provided a conceptual framework for evaluating the CRH program that shows a progression from intervention design to outputs and outcomes. The model Fig.  was builColumn B (program inputs) focuses on implementation by periodically documenting program structure, resources, services, and changes over time. This data will inform assessment of implementation fidelity in terms of whether program resources and features were implemented as planned both initially and over time. Resources provided to hubs from CRH program sources and from hubs to spoke sites are identified as labor and capital inputs.Column C (program outputs) includes reach and adoption. Reach assesses the extent to which CRH hubs fully and equitably deliver the care they are expected to provide to spoke site patients needing coverage. Adoption focuses on the extent to which the program is accepted by and valued by stakeholders, on the barriers and facilitators to adoption, and on integration of CRH hub staff into spoke site workflow.Column D (program outputs) calls for measuring CRH care delivery, coverage of staffing vacancies of different kinds, and duration of coverage. It also calls for qualitative assessment of CRH challenges and successes.Column E (program outcomes) calls for summative assessment of CRH effectiveness relative to the program’s pre-identified goals using formal comparative designs. It also calls for assessment of factors related to maintenance or sustainment of the CRH program following its early implementation phase.Figure Table Aim 1 will initiate documentation of implementation, including data on readiness and context in relationship to implementation progress. Analysis of implementation variations and their determinants will be important for understanding implementation and will also contribute to spoke site level matching required for Aim 3. Aim 2 will focus on collecting longitudinal, non-comparative data on formative program effects on patient care, workplace outcomes, and program challenges and successes. These findings will support the comparative data analyses and the data syntheses required for Aim 3. Aim 3 will support summative conclusions regarding CRH effectiveness based on outcomes relevant to major program goals, including achievement of value to stakeholders , 37. AimThe evaluation aims to use rigorous but pragmatic approaches including mixed methods. Both qualitative and quantitative designs . For example, the main effects of CRH are expected to derive from the reduction of competing demands on the staff remaining in place after staffing losses, and thus may affect all spoke site providers and teams, including both patients cared for by covering CRH providers and those cared for by usual site providers. We will also consider some analyses that aggregate results to the national level across all 18 regional hubs and their CRH participants. We consider this approach to be secondary however because of the program’s focus on site level outcomes and because of the challenges in controlling for contextual factors that could affect results.CRH is aimed at achieving acceptable levels of access to care, quality of care, and workplace-related outcomes for spoke sites in which these may have been impacted by staffing deficits. CRH thus aims to deliver care that is equivalent to, rather than better than, usual care. Non-inferiority designs will be used for relevant analyses such as care quality, and interpretation of findings will also integrate the CRH non-inferiority goal.Qualitative data designs focus on data from interviews at multiple levels. Interviewees will include hub and spoke site front line providers, spoke site clinical leaders, and hub clinical and administrative leaders. Data will also be abstracted from ongoing CRH program document review. Key results will be tracked longitudinally (2019 – 2024) based on repeated interview questions and investigation of initially identified themes over time.Table Evaluation teams will continuously identify data elements needed across teams for inclusion into a shared electronic CRH data base. The CRH Core Team will develop common variable definitions for these elements through across-team deliberations. The CRH Core Team analysts will also maintain documentation that links the data elements to aims and evaluation questions. Team-specific data sets will also exist outside of the core; these include variables that address evaluation questions for which a team is responsible but are unlikely to be needed across teams.Formative quantitative data analysis will mainly use simple pre-post measures or run charts, emphasizing graphic presentation, and often controlling for key contextual elements. For summative quantitative assessment of CRH effectiveness, teams will use appropriate multivariable regression models within comparative quasi-experimental designs. Analyses will control for nesting at the regional and/or site levels as appropriate.Internal validity threats are intrinsic to quasi-experimental designs and can be reduced but not eliminated. To avoid confounding due to unmeasured factors related to program implementation and outcomes, we will fit a separate model for each outcome measure using robust interrupted time series or difference-in-difference analytic approaches. To reduce bias due to non-randomized implementation of CRH, we will use latent variable approaches, propensity score matching, or related methods and will control for important contextual factors in all analyses. Finally, we will use timeline data on external and internal CRH program and evaluation events to link repeated measures data to these contextual influences.In general, we will follow a previously described rapid qualitative data analysis approach in which qualitative analysts create and cross-validate summaries for each interview, followed by a random 50% audit to check summaries against transcripts for accuracy and completeness, and finally by inductive and deductive summarizing of major themes . AnalyseThe concurrent mixed methods approach we will use focuses on providing a foundation for synthesizing qualitative and quantitative findings to support decision-making as part of summative evaluation. Our approach builds on triangulation , 34. AnaFormative results will be summarized yearly in a report that is submitted to CRH program leadership, to the CRH Advisory Board, and to the Office of Primary Care. Evaluation Team investigators will also continuously present findings on CRH program teleconferences, including those for CRH hub leaders, and at national VA and non-VA sponsored conferences, and will contribute data to the CRH program’s yearly or bi-annual reports to the US Congress. The evaluation team will meet biweekly with CRH program leaders, and present results to CRH Hub leaders and/or Advisory Board members at least yearly. We will use mixed method analysis results as the basis for a summative modified Delphi stakeholder panel that includes Veteran patients .Additional file 3: Appendix 3. CRH Evaluation Design Based on Logic Model Inputs, Outputs, and Formative and Summative Outcomes.


Recent evidence shows that programs targeting the socio-emotional dimensions of entrepreneurship—e.g., resilience, personal initiative, and empathy—are more highly correlated with success along with key business metrics, such as sales and survival, than programs with a narrow, technical bent—e.g., accounting and finance. We argue that programs designed to foster socio-emotional skills are effective in improving entrepreneurship outcomes because they improve the students’ ability to regulate their emotions. They enhance the individuals’ disposition to make more measured, rational decisions. We test this hypothesis studying a randomized controlled trial  of an entrepreneurship program in Chile. We combine administrative data, surveys, and neuro-psychological data from lab-in-the-field measurements. A key methodological contribution of this study is the use of the electroencephalogram (EEG) to quantify the impact of emotional responses. We find that the program has a positive and significant impact on educational outcomes and, in line with the findings of other studies in the literature, we find no impact on self-reported measures of socio-emotional skills  and creativity. Our novel insight comes from the finding that the program has a significant impact on neurophysiological markers, decreasing arousal , valence , and neuro-psychological changes to negative stimuli. Daniel Kahneman’s theory about human thinking holds that individuals possess two main systems: System 1, which is fast, instinctive, and emotional, and System 2, which is slow, deliberative, and logical8. In Kahneman’s view, emotion regulation drives people to behave based more on System 2 than on System 1, thus enabling them to identify and pursue more optimal outcomes for themselves and others. While definitions of the terms emotion regulation, responsiveness, and emotional self-regulation, may vary, we define emotion regulation as a mixture of cognitive and emotional processes that shape a mental state—i.e., a disposition to act by directing oneself to consciously affect one’s own emotional and actual response to given stimuli9.The literature on behavioral economics claims that even minor emotional manipulations have a substantial impact on behaviors, decision-making, and economic outcomes10. However, the literature on entrepreneurship education highlights the lack of well identified studies outlining the mechanisms behind the observed impact and, in particular, the lack of attention paid to the role that socio-emotional skills play in entrepreneurship education14. There is a vast literature in economics supporting the notion that socio-emotional skills—e.g., self-confidence, internal locus of control, and grit—are relevant for success in life along other dimensions such as salaries, occupational choice, etc22.Recent evidences show that programs targeting the socio-emotional dimensions of entrepreneurship are more highly correlated with success along key business metrics such as sales, survival, etc., than programs with a narrow, technical bent25. In this paper we instead test the hypothesis that the enhancement of emotion regulation capacity is the main mechanism through which socio-emotional skills impact outcomes.However, there is not a strong understanding of the mechanisms by which socio-emotional skills contribute to these measures of success. Testing for grit and locus of control, both important socio-emotional skills, has proven inconclusiveTo conduct this research, we leverage a randomized control trial implemented by the non-governmental organization (NGO) Youth Entrepreneurship in the north of Chile, which provides exogenous variation in participation in entrepreneurship programs. This NGO worked with a sample of eight schools, four of which were randomly assigned to an intervention that consisted of semester-long weekly 90-min workshops during which the students participated in different activities designed to improve their creative, entrepreneurial, and socio-emotional skills.To assess emotion regulation of the students in the program we used electroencephalography (EEG) recordings, employing methods borrowed from affective neuroscience. A major contribution of this paper is the use of neuroscientific techniques to quantify emotional regulation in large cohort, shifting the discussion from largely self-reported measures that form the basis of much of the literature on socio-emotional skills toward more objective brain-based measurements.Specifically, we find a significant correlation between emotion regulation biomarkers and educational outcomes, which supports the fact that emotion regulation is a potential mechanism driving these findings. We also find that the program affects not only the mean of emotion biomarkers, but also the variance of their distribution. Overall, this evidence provides support to the claim that changes in students’ emotion regulation capabilities partially explain the observed impact on educational outcomes.26, started a program in Chile to foster ten entrepreneurship and leadership skills through didactic weekly interventions. This model embraces the principles of “learning by failing, gaming, doing and rethinking”. McClelland26 tested more than 500 businesses, sports, religious and political leaders of the world with the intention of understanding which of their competencies meaningfully set them apart from the majority of people. His research postulated the presence of 30 socio-emotional skills that were relevant in this respect, 10 of which might be developed in the short term. These 10 skills are: (1) searching for opportunity and taking the initiative; (2) being persistent (gritty); (3) working in support networks; (4) searching for information; (5) taking calculated risks; (6) complying with work commitments; (7) systematically planning and monitoring; (8) being persuasive; (9) demanding efficiency and quality; (10) being self-confident. The program targeted young students, ages 16–18, in four high schools with a particular focus on self- confidence and creativity. The annual estimated budget of the program was approximately USD$100,000 for all four schools. The fixed cost  represented 64% of this figure, while the variable cost  accounted for the rest. The estimated cost of the book was USD$24.The NGO Youth Entrepreneurship, inspired by the model of David McClellandThe program targeted young students ages 16–18 who were enrolled in the 12th grade of vocational high school (the last year of secondary education). The program was accompanied by the development of course materials, including a high-quality student hardcover textbook and a teacher textbook, the latter of which included class-by-class guidelines and videos for each activity to be performed. The program also included three pre-implementation training seminars and pilot/simulated training workshops that took place before the core curriculum was implemented in the teachers’ schools. Importantly, the program’s coaches worked together with schoolteachers to ensure the intervention was homogeneous across schools, and to generate local capabilities within each school.The intervention was designed as follows. The classroom was transformed into a friendly game room. All school desks were put aside, and the chairs were arranged in a semi-circle to ensure that each student would sit the same distance from the center. The students were then taught to recognize their own strengths and weaknesses by acting through games and reflections/deliberations, working in teams under pressure, addressing complex tasks, and setting their own goals to be achieved through these tasks. Finally, a discussion about the learning objectives was conducted after each session.Together with the NGO that implemented the program, we invited eight Technical-professional (TPE) schools located in the IV Region of Chile. All invited schools wanted to participate in the program (see SI for more details).The final sample of our neurophysiological data comprises an unbalanced panel with 296 valid EEG recordings that we use for the difference-in-difference modeling. In particular, we have 76 (64) and 104 (52) students in the treatment and control groups at baseline (follow-up), respectively. For the model to estimation of net effect of entrepreneurship program on the VA space, more students’ data are used  and 110(68) students in the treatment and control groups at baseline (follow-up)), because we don’t need to eliminate students who miss personal information that is needed for the difference-in-difference model. Considering the time and complexity involved in collecting these neurophysiological data, especially in an out-of-the-lab setting, this was an extraordinary number of measures. Indeed, we collected more than 400 student EEG recordings. Filtering for artifacts and data quality gave us the final sample (see details in the SI). We also obtained administrative data from the Ministry of Education (MINEDUC) and the DEMRE, which is the institution that has the information on the SAT-like test that is a requisite to apply to universities in Chile. For MINEDUC we have 1,888 students and for DEMRE 1,296 students considering both treatment and control group. These samples are considerable larger because comprises all students in all eight schools considered that has public records on attendance and registration to the SAT-like test, respectively.We define treatment as participation in the entrepreneurship program during the March–June academic semester in 2015. The intervention took place in the participants’ schools and consisted of weekly 90-min workshops during that semester. This study was approved by the institutional review board at Columbia University and written informed consent was obtained from all parents of the participants all participants and/or their legal guardians before the main experimental sessions. We also confirmed that all research was performed in accordance with relevant guidelines/regulations and the ethics of the project was binding with local standards in Chile.Of the eight invited schools, we randomly chose four to participate in the program by simple random sampling without replacement. We collected baseline information from both the treatment and control schools in February 2015. All 12th grade high school students took part in the intervention in the treatment schools. We also considered 11th grade students attending both treatment and control schools as an additional comparison group. In the presence of positive spillover effects from the treated, considering this additional comparison group would only make our results just more conservative. All students attending 12th grade in the treated schools had to participate in the program as part of regular school obligations. This limits the concerns about selection bias due to non-random attrition. The mandatory feature also allows us to estimate average treatment effects, instead of intent to treat estimates.The personnel for this study, excluding the administrators who helped with the internal logistics, comprised a small research team consisting of the authors of the study and a small number of research assistants who were trained by the authors in all the relevant areas of methodology, equipment, and theory.We built a portable laboratory containing seven workstations, each of which consisted of a portable EEG headset  paired with a high-capacity laptop (Lenovo ThinkPad). This experimental setting, which had previously been piloted at Columbia University, enabled us to obtain a proxy measure of our students’ emotion regulation capabilities in a non-lab context .In order to measure creativity, we employ an adaptation to the written version of the 1976 Torrance Test of Creative Thinking (TTCT)32 to measure fluid intelligence.We used Progressive Matrices similar to those created by James Raven33, and voice and facial manifestations for valence37. However, many of these measurements can be consciously modified; thus, the signals they produce are not purely objective36. Therefore, we use EEG recordings, a more plausibly objective instrument to proxy emotion regulation abilities, which directly affect an individual’s disposition to act9, and which have been suggested as a relevant mechanism for other educational interventions targeting youths25. A number of authors have stated that EEG recordings can measure brain activity and predict emotional states and physiological responsiveness, improving both accuracy and objectiveness in respect to the aforementioned physiological measurements39.Over the past decades, emotion-detection research has employed a variety of physiological measurements and methods, including pupil dilation, heart rate, and skin conductance for arousal40, which relies on neurophysiological measurements, to accurately capture and codify emotional data.To establish a more rigorous foundation for our research in the domain of objective measurement, we employ the James-Lange theory of valence and arousal.lAs illustrated by Fig. 42 (see more detail in the SI). We estimate emotional arousal and valence indices at resting state using models from emotion-detection theory (see more detail in the Emotion-detection theory section of SI). As the Positive and Negative Affect Schedule (PANAS) is self-reported, it experiences issues such as test setting bias43 and reference bias21. An alternative can be the use of neurophysiological methods drawn from affective neuroscience31. In particular, arousal is constructed as a beta-alpha ratio on the pre-frontal cortex, while the left, relative to right (LFA), frontal cortical activity is used to build valence indices48.The emotional state with no stimuli, also referred as “at resting state” measure, was constructed using EEG scalp recordings taken while the students watched a black cross in the center of a gray screen for a period of 30 s. We transformed the electrical recordings from the scalp into frequencies , and then interpreted the various frequencies, mainly alpha (8–12 Hz) and beta (12–30 Hz), and brain locations to estimate the students’ emotional states. By only extracting the alpha and beta frequencies, the influence of much noise is already significantly reduced, because the influence of eye movement or blinking artifacts is most dominant below 4 Hz, while heart related movements, i.e., Electrocardiogram (ECG), artifacts around 1.2 Hz, and facial muscle movements, i.e., electromyogram (EMG), artifacts above 30 Hz49 but was not a feasible option for us during the time when we conducted our field experiment). These were: (1) No stimuli, which captures student’s emotion responsiveness at resting state; (2) Positive Stimuli, which captures the responsiveness to positive stimuli net of the resting state measure; and (3) Negative Stimuli, which captures the response to negative stimuli net of the resting state measure. We normalize these indices relative to the mean and standard deviation of the control group. We then trim the indices at the 2.5 and 97.5 percentiles to avoid a substantial influence of the outliers of EEG measurement, as is common practice in literature51.After measuring emotion indices at resting state, we applied a psychometric test that includes the Grit scale, the locus of control scale, the raven-like progressive matrices, and the creativity test. We measured the participants’ emotional responsiveness immediately after the students had finished this test regimen. The experiment consisted of showing an alternating series of five positive and negative images in order to elicit emotional responses from each participant, which we illustrate in Fig. (1) Dropout Rates: In order to assess how and to what extent the program impacted the participants’ educational attainments, we assessed administrative data drawn from Ministry of Education (MINEDUC) to determine when and whether any students had withdrawn from their studies at any point in time. This process yielded a dichotomous variable, which took on the value “1” if a student had dropped out, and “0” if he/she had not. This administrative data was provided by the Ministry of Education under strict rules of privacy. We were able to collect information about all students enrolled in the eight schools that participated in the study, which substantially enlarged the usable sample.(2) PSU Test: The PSU is similar to the SAT in the U.S.; it is optional for students who have completed secondary education but is required for those applying to university. When combined with the cumulative average of secondary education grades, the PSU represents the primary selection criterion for Chilean universities. In this regard, the PSU has a relative weight of around 70% in the final university application score. The PSU score is normalized to an average 500 points and a standard deviation of 100 points, with a fixed status ranging from 150 to 850 points. Apart from assessing whether or not the program would make students more or less likely to complete their compulsory secondary studies, we also wanted to gauge whether or not it would impact their desire to pursue further, post-secondary studies after completing high school. The willingness to consider post-secondary education is an indicator of better aspirations for their future. We therefore consider as a binary outcome of interest whether or not a student registered for the PSU. The data for this assessment, which were provided by the Department of Measurements (DEMRE) at the University of Chile, indicated which students had registered for the PSU.s. Treati is a dichotomic variable equal to 1 if the student is enrolled in the program. Gi represent the gender of the student. In the dropout model we were able to include school fixed effects, represented by (1) Educational outcomes: We expected the entrepreneurship program to affect certain educational outcomes, such as registration to the university selection test  and high school dropout. Both outcomes could be thought as proxies of positive aspirations or motivation. Administrative data on educational outcomes are only accessible at the end of each academic year. We estimate a simple Probit model comparing treatment and control groups at the end of the year, which can be written as:40, to measure neurophysiological markers associated with emotions. We use these measurements to examine whether the program induced changes in these measures, which we determine by employing the valence and arousal indices to gauge emotional reactions to stimuli, thereby testing for emotion regulation as a mechanism for observed changes in educational outcomes. First, the indices of resting state valence and arousal were measured as explained in data collection section. Second, we calculated the emotional responsiveness in the 2-D Valence-Arousal (VA) space52  Emotional regulation: A key contribution of this study is the use of a novel methodology, EEG recordings paired with the James-Lange theory of valence and arousalj emotion-related variable of individual i in school s at time t. The emotion-related variables indexed by j include Arousal (no stimuli), Valence (no stimuli), Valence (positive stimuli), and Valence (negative stimuli). The variables Valence (positive stimuli) and Valence (negative stimuli) measure the difference for each individual i at time t between student’s Valence (no stimuli) and the valence measured in student’s positive and negative stimuli scenarios, respectively. These indicators capture the emotional reaction net of their “resting state”, i.e., level of arousal and valence changes compared to no stimuli. We estimate each student “resting state” level of arousal and valence using EEG recordings taken while the students watched a black cross in the center of a gray screen for a period of 30 s. We consider a 5-s window in the middle of that time frame for our estimation. As before, Treati and Postt are dichotomic variables representing treatment by the program and survey time—i.e., baseline or follow up—respectively. Finally, Xi indicates other controls at individual level, such as gender, while αs represents the school level variables .We estimate a difference-in-difference model given by:s (s = 1 is positive or s = 2 is negative stimuli) at time t (t = 0 is baseline or t = 1 is follow-up) for different groups g , t between student’s emotion response measured in student’s different stimuli scenarios s (either positive or negative) compared to the resting state  by measuring the difference of changes of emotion respect to negative/positive stimuli, which can be summarized by:With the difference-in-difference model and measurement of net effect, we can control for all time-invariant characteristics within and between groups. By subtracting the average differences between groups before the program started and only comparing the difference after the program, we can account for any pre-existing differences between the groups. This methodology also helps to control for any other unobserved variables that may affect both groups in a similar way.31. Similar to the previous model, we control for differences before the program started—i.e., at baseline—between treatment and control groups.(3) Socio-emotional Skills, Creativity, and Cognitive Skills: The entrepreneurship program was expected to positively impact the socio-emotional skills of its participants. Using conventional self-reported tests, we measured grit (perseverance) and locus of control. Additionally, the program was expected to impact creative and innovative skills, which are a mixture of both cognitive and socio-emotional skills. Note that the literature demonstrating best practices for measuring creativity remains scarce, particularly in the context of educational program evaluationsj of individual i at time t. The index j represents one of creativity, grit, or internal locus of control. Treati is a dichotomic variable equal to 1 if the school received the program, and Postt is a dichotomic variable equal to 0 at baseline (pre-treatment) or equal to 1 at follow-up (post-treatment). Finally, Xi indicates school-level and 12th grade dichotomic variables.In this specification, N = 61) and 11th grade (N = 95) students in the control group in all specifications. We include grade indicators to control for unobserved differences in cohorts that might be correlated with the entrepreneurship intervention.In order to achieve the highest possible statistical power in these estimations, we used both non-treated 12th grade  schools in Chile continue on to higher education, compared with 66% of the students in Scientific Humanistic High School Education (SHE) programsWe report estimates of educational outcomes (see Eq. ) in Tabl54. Moreover, descriptive analyses of similar programs previously implemented by Youth Entrepreneurship show positive impacts of such programs in terms of lowering dropout rates among participants relative to historical rates55. As this study is concerned with identifying the mechanisms through which these outcomes occur, such as the program’s effects on emotion regulation or socio-emotional skills, any further discussion of the impacts on these educational outcomes lies beyond its scope.In Column (3) of Table 56. We also expect In the difference-in-difference model of emotional regulation (see Eq. ), we hyp58. In other words, when faced with negative stimuli  after treatment, the students’ neurophysiological responsiveness changed significantly less compared with the emotional reactions in the control group. This might be interpreted as an increase in the resilience trait among participants59.Figure entclass1pt{minimat-test: p = 0.1265). No significant difference in arousal indices between groups was found with respect to positive stimuli. In Fig. p = 0.0134), which has similar impact in order of magnitude relative to similar interventions58 and it is consistent with the result reported from Eq.  in Fig.60. Furthermore, the program’s estimated impact on both the creativity indices was not significant. Estimations were made using either a simple average of the three dimensions of creativity—flexibility, fluidity and originality—or a principal component analysis (PCA) index of creativity exhibited similar results in terms of magnitude, sign and significance.Given the results in Fig. We report the results of the entrepreneurship program’s impact on cognitive skills, as measured by Raven’s test of progressive matrices in the SI. As the entrepreneurship program was not designed to affect cognitive skills, this test was only incorporated in order to rule out any change in cognitive skills as a mechanism of observed impact on educational outcomes. We performed this test only in the follow up measurements, which is why we only have 116 observations. The lack of significant result suggests that changes in cognitive skills is the not channel through which the entrepreneurship program affects educational outcomes.15 and reports from the Organisation for Economic Cooperation and Development (OECD)61, who find that cognitive skills explain only a small fraction of the variance found in labor market outcomes. We also find that participation in the entrepreneurship program does not impact self-reported measures of socio-emotional skills that were specifically targeted by the program’s curriculum, such as internal locus of control and grit. This result is also unsurprising, and commonly found in the literature24. These null results were expected and documented before running the RCT62.In our research, we do not find strong evidence for changes in cognitive skills as a result of participation in the entrepreneurship program. This result likely stems from the program’s focus on socio-emotional skills as well as the age of the targeted population. This result is consistent with related work, such as Almlund et al.64.Although the discussion of the role played by socio-emotional skills in the production function of entrepreneurs is still open, numerous researchers have highlighted these skills as a key component of entrepreneurial activity and the need to incorporate emotional dimensions in the analysisWe find that the program had a significant impact on the students’ emotional state, as indicated by a decrease in arousal and valence indices at resting state, based on EEG recordings. We also estimate significant declines in the participants’ emotional responsiveness to negative stimuli that we interpret as an increase in the ability to regulate emotions among participants. Since the focus is on learning from failure or adverse outcomes, it is unsurprising to find no significant response to positive stimuli. Considering the statistical significance of these results, the magnitude of the impact 70. For example, emotions experienced while making a decision—i.e., choice-option-elicited emotions—are at the base of traditional economic interpretations of utility as emotional carriers of value. Positive emotions increase value and elicit approach, whereas negative emotions decrease value and result in avoidance3. Moreover, emotions unrelated to the judgment or decision at hand, referred as incidental emotions, have also been shown to influence choices3.The link between emotions and cognition has been debated and explored by scholars for centuriesFollowing Kahneman’s ideas, we argue that emotion regulation drives people to behave based more on System 2 than on System 1. System 2, given its more deliberate focus, enables individuals to identify and pursue better outcomes for themselves and others. We argue that the program studied here changes emotion regulation capabilities among participants, essentially pushing them toward behavior closer to System 2, which in turn affects their educational outcomes. In this section, we focus on how to interpret the results on neurophysiological measures and how the program affects emotion regulation. We also discuss why the program affects students‚ “disposition to act” and the economic relevance of our findings when compared to similar programs. We argue that the program affects emotion indices via a reappraisal strategy triggered by learning by failure. This reappraisal strategy, in turn, helps individuals develop a sense of resilience.2. Schaffer et al. suggest an “approach/withdrawal” model to investigate left relative to right frontal cortex EEG asymmetry (LFA) regarding emotional states71. Since then, the literature on psychology and neurophysiology has pointed out that frontal EEG asymmetry is associated with different emotional and psychological states in addition to valence72. In fact, the “approach/withdrawal” model invites us to think of an additional interpretation of our results. A decrease in frontal EEG asymmetry  is consistent with a relative increase in withdrawal behaviors. This is mainly due to the fact that students would like to approach positive stimuli and avoid negative stimuli. Since the entrepreneurship program aims at instilling a more resilient, even-tempered attitude toward failure, this is a plausible interpretation of our results.The behavioral economics literature provides evidence showing that even minor mood manipulations have a substantial impact on outcomes and behavior73, the authors referred to Randolph M. Nesse74, who argued that “Emotional states not only motivate action, they are also goals that we seek to achieve. Most human thought, plans, and actions are intended to induce positive emotions or to avoid negative emotions”. From this evolutionary point of view, a successful injunction of negative emotion would be more relevant for participants’ behaviors because negative emotions suggest a situation that should be altered, while positive emotions indicate situations that should be maintained74.The asymmetry in the impacts of positive and negative emotional states is consistent with other findings in the literature. In Querengasser and Schindler’s workAt first glance, the negative impact on emotional disposition would seem counterintuitive given that the program was designed to improve socio-emotional skills. For instance, changing the attitude toward failure would also change education and labor market decisions, such as efforts put into tasks, occupational choices, entrepreneurship, and the pursuit of creative and original work, among many others. This interpretation is consistent with the impact observed on educational outcome.7 review the recent findings for “Solutions that Seek to Minimize the Emotional Response.” The results of these studies point to four dimensions of emotional responses: Time delay, Suppression, Reappraisal, and the “Dual-emotion solution”. Reappraisal consists of re-framing the meaning of those stimuli that lead to an emotional response and has consistently emerged as a superior strategy for dissipating the emotional response75. Specifically, reappraisal includes behaviors such as reminding oneself that “it’s just a test” after receiving a sub-optimal grade in an exam, adopting a mindset similar to that of a nurse or medical professional to minimize the emotional impact of viewing someone’s injury, or viewing a job layoff as an opportunity to pursue long-forgotten dreams75. In contrast to suppression, reappraisal not only reduces self-reported negative feelings in response to negative events, but there is substantial evidence supporting the notion that reappraisal also mitigates the physiological and neural responses to those events77. Additionally, regulating emotion by means of reappraisal-focused strategies that encourage taking a different perspective has been shown to reduce loss aversion in decision-making78. This result has been observed both in choices and in the relative arousal responses to actual loss and gain outcomes. Finally, there is evidence that reappraisal-focused strategies lead to higher measures of resilience among participants, which is consistent with the hypothesis that the program impacts this skill59. In addition, and according to Casey et al., the role played by positive and negative stimuli may be even more salient among adolescents79. In fact, the greater emotional responsiveness and sensitivity typical of this time of life may play a role in the higher incidence of the onset of affective disorders and addictive behaviors that often occurs during these years. Thus, policies aimed at increasing an individual’s ability to regulate his/her emotions may be even more meaningful when targeted toward adolescents.A core aspect of the program’s methodology was to generate situations in which students had to face failure and reappraise their emotions in such a context. Lerner et al.80 reported that handedness does not appear to have any significant effect on the detection of emotion signatures in EEG, we did not explicitly measure the participants’ handedness in this experiment. However, considering the handedness and brain lateralization would be helpful to reduce statistical sensitivity, as some fields of cognitive neuroscience may not exclude left-handers a priori but may nevertheless do so a posteriori81. Furthermore, given the context of our RCT, the best that we can offer here is a comparison in terms of the effect size as a fraction of the standard deviation of the variables with respect to similar evaluations found in the literature for educational settings. In this regard, the impacts of 0.13 58.Finally, it is hard to translate the estimated results into meaningful behavioral explanations. All emotional indices built with the EEG recordings were standardized with respect to the mean and standard deviation of the control group. Since Mathersul et al.82. They find that students with greater resting state valence were willing to expend greater effort in the pursuit of tasks with larger rewards, particularly when reward delivery was less likely.There are a number of papers that consider valence or left relative to right (LFA) cortical activity, to build behavioral indices of emotions or motivation. For instance, Hughes et al. examined the relationship between valence and effort expenditure for reward, a behavioral index of approach motivation56.Our work has analyzed the impact of an entrepreneurship program implemented in vocational schools in Chile. This entrepreneurship program was designed for individuals to learn from their failures. We find that emotion regulation is the main channel through which the program impacts educational outcomes. A novel feature of our study is the use of EEG recordings as a measure of socio-emotional skills, grounding our discussion of emotion regulation in objective measurements rather than self-reported results, as in earlier studies. To be clear, we do not claim that EEG recordings  record or “read” emotions; rather, these tools serve as useful proxies to measure changes in skills associated with emotion regulation. This promising use of EEG recordings complements recent work that uses cortisol as a physiological measure of psychological well-being in impact evaluationsWe argue that training programs designed to foster socio-emotional skills affect the participants‚ capacity for emotion regulation, which is a unique form of socio-emotional skill. We test our hypothesis in two ways. First, we leverage a randomized controlled trial of an entrepreneurship program designed to foster socio-emotional skills among youth to get exogenous variation in program participation. Second, we use emotion-detection methods drawn from affective neuroscience to obtain unbiased measures of emotion regulation. We also document the limitations of self-reported measures of emotional states.83. We do find, however, that the program impacts participants’ emotion regulation ability, as measured by our use of EEG recordings and plotting this data along arousal and valence indices.We find that the program has a significant impact on educational outcomes as measured by dropout rates and registration to the college entrance exam PSU. We do not find a significant impact on self-reported socio-emotional skills measures, which is consistent with the findings of previous related studies7. On the other hand, program participation also significantly reduces individuals’ emotionally related neuro-physiological reactions to negative stimuli compared to students who have not taken the program. We do not find a significant difference between the two groups in response to positive stimuli. This asymmetry in the reactions to emotionally-laden stimuli is consistent with the asymmetric impact on emotional responsiveness found in the literature2. This might be interpreted as an increase in the resilience trait among participants59.On one hand, we find significant impacts on emotional state at resting state—i.e., a no stimuli condition. Specifically, we estimate a decrease in both arousal and emotional valence indices among participants. After analyzing the results, we establish that the impacts on emotion regulation are likely due to the reappraisal-focused strategies espoused by the program’s rethinking and redoing methodology. Indeed, these strategies have been identified as the most effective for avoiding emotional bias on decision-making58. Moreover, these results provide a quantitative support to recent suggestive evidence based on qualitative analyses pointing out that programs targeting socio-emotional skills might affect the ability to young people to slow down and re-think about their response to a stimulus25. Finally, the emotion regulation indices correlate with our main educational outcome, which supports the hypothesis that emotion regulation is the mechanism behind our findings.As mentioned before, the magnitude of our findings are close to the upper bound of similar educational interventionsFrom a methodological perspective, the experimental paradigm adopted for this study provides an alternative to the approaches most frequently employed in the literature: self-reported measures, latent factor models, and the use of revealed behaviors as proxies. The methodology proposed here enabled us to measure emotional disposition and responsiveness by means of EEG recordings. The use of these recordings provides a solution that overcomes the biases related to self-reporting, and the question of whether programs such as the entrepreneurship one studied here impact any unobservable characteristics besides those that were initially targeted. Finally, the use of this methodology is at the cost of having a smaller sample size given all the logistics and technicality involved in the use of EEG devices.Supplementary Information.


The Infectious Diseases Society of America (IDSA) developed and disseminated a Core Antimicrobial Stewardship (AS) Curriculum intended to formalize AS training in infectious diseases (ID) fellowship programs in 2018. This study identified individual program approaches to curriculum implementation and intended to use this information to develop an implementation guide tailored to specific program needs.We distributed surveys to all fellowship program directors (PDs) who had previously implemented the Core AS Curriculum. Questions were designed to identify ID program structure, curriculum participants, curriculum sections and materials utilized, and resources and barriers to implementation. Both structured and qualitative responses were captured. The results were summarized descriptively and organized into a framework connecting barriers to proposed solutions.Table 1). Additional learners often included ID faculty and pharmacy trainees, and teachers were mostly AS program leadership. Most PDs reported limited faculty time as a barrier to implementation, whereas dedicated AS curricular time was a resource available to most programs . Approaches to curriculum implementation based on survey responses relating to each fellowship program feature were suggested, some of which applied to multiple program features . Qualitative feedback was generally positive, and most PDs indicated that they intended to continue to implement the curriculum. Additional materials such as a facilitator guide and demonstrations were proposed as other components which could assist with curriculum implementation.PDs from 34 unique programs who had administered the Core Curriculum to an estimated 405 ID fellows responded to the survey, out of the 159 institutions invited (21.4%). Most represented adult programs which had administered the curriculum for at least 2 years (Roadmap with infectious diseases fellowship program features (rounded green boxes) connected to identified potential approaches to curriculum implementation (rectangular blue boxes). Program features are categorized as relating to conference structure, program resources, or program barriers. Suggested approaches with potential application to multiple program features are highlighted in orange.The IDSA Core AS curriculum provides an effective means of formalizing basic AS education into ID fellowship training. Curriculum implementation can be optimized by tailoring to training program resources and unique features. An implementation roadmap may be a useful tool to assist ID fellowship PDs with this task.Payal K. Patel, MD MPH, qiagen: Honoraria Julie Ann Justo, PharmD, MS, FIDSA, BCPS, Gilead Sciences: Advisor/Consultant|Shionogi: Advisor/Consultant|Vaxart: Stocks/Bonds Erica J. Stohs, MD, MPH, bioMerieux: Grant/Research Support|Merck: Grant/Research Support Zachary Willis, MD, MPH, Merck Sharp & Dohme Corp: Grant/Research Support|Pfizer Inc: Grant/Research Support Trevor C. Van Schooneveld, MD, FSHEA, FACP, AN2 Therapeutics: Grant/Research Support|Biomeriuex: Advisor/Consultant|Biomeriuex: Grant/Research Support|Insmed: Grant/Research Support|Thermo-Fischer: Honoraria Amy Y. Kang, Pharm.D., BCIDP, Paratek: Grant/Research Support Kartik Cherabuddi, MD, FACP, FIDSA, Contrafect Corporation: Grant/Research Support|Labcorp Drug Development: Grant/Research Support|Merck Sharp & Dohme: Grant/Research Support Gary Fong, PharmD, Critical Innovations, LLC: Advisor/Consultant Molly L. Paras, MD, Angiodynamics: Honoraria David Gaston, MD PhD, American Association of Clinical Chemistry: Honoraria|BioMerieux, Inc: Advisor/Consultant|IDbyDNA, Inc.: Grant/Research Support|Illumina, Inc.: Grant/Research Support


Faculty development (FD) programs are critical for providing the knowledge and skills necessary to drive positive change in health professions education, but they take many forms to attain the program goals. The Macy Faculty Scholars Program (MFSP), created by the Josiah Macy Jr. Foundation (JMJF) in 2010, intends to develop participants as leaders, scholars, teachers, and mentors. After a decade of implementation, an external review committee conducted a program evaluation to determine how well the program met its intended goals and defined options for ongoing improvement.The committee selected Stufflebeam’s CIPP  framework to guide the program evaluation. Context and input components were derived from the MFSP description and demographic data, respectively. Process and product components were obtained through a mixed-methods approach, utilizing both quantitative and qualitative data obtained from participant survey responses, and curriculum vitae (CV).The evaluation found participants responded favorably to the program and demonstrated an overall increase in academic productivity, most pronounced during the two years of the program. Mentorship, community of practice, and protected time were cited as major strengths. Areas for improvement included: enhancing the diversity of program participants, program leaders and mentors across multiple sociodemographic domains; leveraging technology to strengthen the MFSP community of practice; and improving flexibility of the program.The program evaluation results provide evidence supporting ongoing investment in faculty educators and summarizes key strengths and areas for improvement to inform future FD efforts for both the MFSP and other FD programs.The online version contains supplementary material available at 10.1186/s12909-023-04155-x. Robust faculty development (FD) programs in the health professions are necessary to provide faculty with the knowledge and skills required to prepare learners for an ever-evolving healthcare landscape. Although FD initially emerged as a form of training to develop teaching skills, it has since expanded to include activities that improve knowledge and skills in other domains such as research, administration and clinical skills –3. In adDespite the importance of FD programs, there exist several barriers to participation, including: a lack of protected time for attendees, competing responsibilities, a perceived lack of recognition and financial reward for teaching, a perceived lack of direction from and connection to the institution, and logistical factors . Lack ofOne such flagship program is the Macy Faculty Scholars Program (MFSP). The MFSP is a FD program created by the Josiah Macy Jr. Foundation (JMJF) in 2010 with the goal of developing health professions educators as leaders, scholars, teachers and mentors . The MFSThe program selects 5 Scholars per year from institutional nominations through a rigorous, multi-level selection process. Accepted Scholars receive base salary support plus fringe benefits to protect at least 50% of the Scholar’s time for two years. Scholars produce academic output, engage in the national MFSP educational network, and participate in the Harvard Macy Institute Program for Educators in Health Professions . ApplicaTo date, the JMJF had not yet engaged in a significant, formal, ongoing evaluation of the MFSP, as the priorities thus far had included building the program, recruiting the mentors, developing selection criteria, partnering with the Harvard Macy Institute for the curriculum, and refining various policies. More generally, rigorous evaluations utilizing both qualitative and quantitative methods of robust national FD programs often remain neglected or considered as an afterthought in the literature, highlighting the need to publish program evaluations that are completed in an academically rigorous manner for transparency of program outcomes –15. TherContext (program objectives and the basis for those objectives), Input , Process  and Product (resulting outcomes and if they met the needs of the target population).[The evaluation team elected to apply Stufflebeam’s CIPP evaluation framework, which contains four major levels of evaluation: ulation)., 20. Theulation)., 21–23. We obtained the context and input components of the evaluation from the MFSP program description and Scholar demographic data, respectively. We applied a mixed-methods approach to evaluate the process and product, utilizing both quantitative and qualitative data obtained from program statistics, survey responses and curriculum vitae (CV). Informed consent was obtained from all subjects and the study was reviewed by the University of Michigan Institutional Review Board, HUM00175014, for ethical approval and ruled to be exempt from ongoing review. All methods and experimental protocols were approved and carried out in accordance with relevant guidelines and regulations.MFSP Scholars with a minimum of one year of post-program activity submitted current CVs for analysis of types and number of academic products. We aggregated and de-identified the CVs. We used three time periods to classify academic output: time period 1 (pre-MFSP) represents years before the program ; time period 2 (intra-MFSP) represents the two years during the program; and time period 3 (post-MFSP) represents years after the program up until the end of 2019, ranging from 1 year, for those who entered in 2016, up to 6 years for those who entered in 2011, with a mean of 3.5 years. Nine of the authors  participated in data extraction and performed exploratory data analyses including a review of data distributions by histogram and Q-Q plots, as well as descriptive statistics such as mean, median, variance, skewness, and kurtosis, and outlier analyses using interquartile range and Cook’s distance . We utilized descriptive statistics and analysis of variance (ANOVA) to examine the results. We log-transformed numerical data to meet assumptions of normality for the statistical tests prior to comparison between groups. We used a one-way, within subjects ANOVA to test within-subject differences in outcomes during time periods 1, 2, and 3 for statistical significance. We used partial eta-square as effect size measures. Post-hoc analyses assessed the effect of protected time on these outcomes by an analysis of covariance (ANCOVA) using protected time as a covariate. All statistical tests were two-sided with a 0.05 significance level. We performed analyses with IBM SPSS version 26.0 .Following the Standards for Reporting Qualitative Research, we utilized a constructivist approach to engage in a qualitative analysis of open-ended survey responses to further understand the participants' experience with the MFSP program . With reinputs in the CIPP model for the MFSP program.Table products of the CIPP model, or rates of academic outputs per year for each category of the CV analysis over each time period and results of the statistical analysis to determine between-group differences and effect size. A total of 31 CVs met criteria for inclusion in the CV analysis.Table Three areas of accomplishment demonstrated statistically significant temporal differences, though with different patterns of change. The rate of education-specific grants  decreased going from time period 1 to 2 as well as from time period 2 to 3. The rate of total grant dollars (either in a PI or co-investigator role) decreased going from time period 1 to 2 but increased going from time period 2 to 3. The rate of education-specific national presentations increased going from time period 1 to 2 but decreased going from time period 2 to 3.Four areas demonstrated a non-statistically significant trend toward temporal differences. The rate of education-specific grants (either in a PI or co-investigator role) decreased going from time period 1 to 2 as well as from time period 2 to 3. The rates of total publications, education-specific presentations , and education-specific international presentations increased going from time period 1 to 2 but decreased going from time period 2 to 3.Table Survey response rate was 94% (34/36). Most respondents “agreed” or “strongly agreed” that the MFSP developed them as a teacher (29/34), mentor (27/34), scholar (33/34), and educational leader (34/34). Most “agreed” or “strongly agreed” that the program was a valuable use of time (33/34), allowed things to be accomplished that otherwise would not have been (34/34), and would recommend the program to others (33/34). Scholars reported a mean and median of 51 ± 12% protected time.process component of the CIPP model and fit under two overarching domains: areas of strength and areas for improvement  Scholars represented double the percentage of URM applicants. Lack of diversity in the health professions remains a critically important issue with evidence that diversity within the care team enhances patient care [The ent care –32. In aent care –37. By cprocess analysis.The MFSP utilized several educational strategies to meet the previously described objectives. First, they utilized their national network of mentors, faculty, and alumni to create a robust CoP that facilitated career growth and academic output for the Scholars. Second, the program accessed the Harvard Macy Institute Program for Educators in Health Professions curriculum which allowed the Scholars to participate in an established program that guided their own professional development as health professions education scholars. The MFSP use of these resources and educational strategies supported the program’s intended goals, and the actual implementation was further explored as part of the process of the CIPP model, or the actual implementation and execution of the planned activities to support the products, we found that the MFSP program was ultimately successful in its implementation and execution of planned activities. Our data demonstrated successful mentorship relationships, robust CoP, significant resources, and growth as educational scholars. While the participants rated the program highly overall, the analysis also revealed specific areas for improvement, including recommendations regarding mentorship, diversity, technology, and a request for continued engagement in the program post-graduation.Regarding the Mentorship and sponsorship in academic medicine represent key drivers of personal and professional development, research productivity, and career satisfaction –40. AlthIn addition, Scholars indicated that the MFSP afforded entry into a robust CoP of educators in the health professions, including other Scholars, alumni, and faculty of the program. Membership in CoPs result in increased social support, which can enhance well-being and engagement, and may mitigate the risk of burnout –43. AlsoFinally, integration of technology represents another potential area of growth for the MFSP and FD programs in general. Specifically, Scholars suggested improving the MFSP’s social media presence and increasing the frequency of contact via videoconferencing and other digital platforms, a theme further highlighted in the context of COVID-19 related gathering restrictions. Furthermore, Scholars requested continuous programming and connection to the MFSP community beyond the two-year fellowship by enhancing the alumni network and promoting communication in between in-person meetings. Engaging past participants of FD programs as future leaders of the program enlarges its CoP, while also providing future sponsorship and leadership opportunities to HPEs. Utilizing these approaches allow for affordable and flexible ways to promote engagement among geographically distant participants and enhance a CoP –49.products of the CIPP model, the resulting outcomes or achievements of the program, demonstrated that the MFSP successfully developed its Scholars academically. Prior studies have evaluated competitive grant awards and publications as indicators of academic success among faculty and we also utilized these metrics to assess that objective [The bjective , 50–54. bjective . Ongoingbjective , 56. IndAs the MFSP represents a flagship FD program for health professions educators, lessons learned through its evaluation provide further evidence for policymakers, organizational and institutional leaders that investing in health professions educators, particularly through providing actualized protected time, provides meaningful return on investment with regard to developing leaders, scholars, teachers, and mentors, with the intent of benefiting society at large. Additionally, the program evaluation generated a list of key themes and resultant strategies to inform future iterations of the MFSP program that are broadly applicable to other faculty developers in health professions education as they create, optimize, and expand their programs Table .Table 5RIn applying the lessons derived from the program evaluation, however, the learning and practice environments of related FD programs must be taken into consideration. The MFSP is a premier, nationally recognized program that draws faculty learners from institutions throughout the country. There are few similar programs. Instead, most longitudinal FD programs draw from, and are supported by, single institutions which seek to build their local educational expertise. The contrast between the national focus of the MFSP and the local focus of most institutional programs is significant in many ways. The financial support for protected time is critical to the MFSP but “support” from local programs may well reflect the influence of other values and priorities to foster the strength and reputation of the institution – above and beyond the finances for “protected time.”Similarly, the MFSP seeks to build a national community of scholars and collaborators, whereas local programs are likely to be more concerned with building their own, local CoP. There is a risk of a local FD program focusing too narrowly on the needs and benefits of the institution but we, the authors, many of whom have participated in local FD programs, believe firmly that all local programs should recognize that medical education is a national, even international, enterprise that extends beyond the narrower interests of local contexts.Finally, we believe that all longitudinal FD programs should rigorously evaluate themselves for both local impact but also national significance. The perspective of evaluation needs to be national in scope and simply describing a program is only a first step towards the needed analysis of the key features of the program, the characteristics of the learners, and the nature of the educational context. Leaders of such FD programs need to build the community’s knowledge and practice, not just a local institution. This is how individual programs contribute to the greater good in health professions education.We acknowledge several limitations of our evaluation. Of note, over the 10-year period examined, the curriculum changed dynamically in response to ongoing feedback, such that Scholars' experiences varied depending on the years they participated. Additionally, CVs vary in their format and content, and different reviewers vary in their CV interpretation. In order to streamline data extraction, the CV analysis subgroup jointly coded a CV in order to develop the extraction form (see Additional file Of note, the large number of statistical comparisons could technically lead to spurious results for the quantitative data. Although conservative measures such as Bonferroni corrections were considered, we withheld use of them, given that the trends were of greater practical concern for the program evaluation than statistical significance. In addition, trends were also supported by themes identified by a separate review group in the qualitative analysis data.Despite these limitations, the evaluation process’ many strengths included: multiple independent and objective perspectives of early career health professions educators who represent potential stakeholders, a detailed analysis of Scholars’ CVs, a relatively high response rate to the survey administered, use of an applicable evaluation framework previously utilized for FD programs, and integration of findings with relevant education theory and literature.Over the 10-year course of its history, the MFSP has received positive responses from participants and has successfully developed Scholars as mentors, teachers, educators, and scholars. The increase in academic productivity observed most prominently during the two-year period of the program highlights the importance of investing in health professions education faculty through actualized, rather than simply promised, protected time. Scholars also commonly cited the mentorship and CoP as the most valuable components of the program for achieving its stated goals. Areas for improvement included enhancing the diversity of program participants, program leaders and mentors across multiple domains including race, gender, geographic location, and health profession type; leveraging technology to strengthen the MFSP CoP through increased social media presence and use of remote teaching strategies; and improving flexibility of the program to meet Scholar needs. Through this program evaluation, we have demonstrated key strategies that can be utilized by other existing FD programs to inform continued development and expansion of their programs and ultimately, support the important work and development of health profession educators.Additional file 1: Appendix 1. CV Data Extraction Form. Appendix 2. MFSP Scholar Survey.