This article was migrated. The article was marked as recommended.IntroductionThere is little evidence on tools that can be used to identify individuals most at risk for remediation and attrition in postgraduate surgical training. We explore the extent to which a situational judgment test (SJT) can predict which trainees are likely to require remediation or leave the program.MethodsPostgraduate trainees in a single general surgery residency program in the United States completed a 50-item SJT. Data regarding remediation actions and attrition were retrieved from the program for the two years following completion of the assessment. United States Medical Licensing Examination (USMLE) scores were also included to examine their ability to predict remediation and attrition.ResultsComplete data was available from fifty-two of 56 (92%) residents in the program. Twenty-one percent (11/52) of residents required at least one remediation intervention within the one (for PGY5s) to two years after completing the SJT. Mann-Whitney U tests revealed a significant difference in SJT performance between those needing remediation versus those who did not require remediation, such that those requiring remediation performed worse on the assessment (p ConclusionsThese data demonstrate that SJTs can be created to effectively identify surgery residents most at risk for remediation across a two-year timeframe. These data provide an additional layer of validity evidence to support the role of SJTs in surgical education and align with other studies failing to find linkages between USMLE scores and residency performance criterion. Modern surgical training is intense, fast-paced, and rigorous. To succeed in this environment, trainees must be able to quickly adapt to new environments, master new surgical techniques and technologies, and successfully work within dynamic teams. As a result of this unique environment, many postgraduate trainees fail to meet program expectations during their training and require additional focused interventions to get back on track. In the United States (U.S.), nearly one third of general surgery trainees require remediation during their five years of post-medical school training, with 25% of those remediated requiring more than one intervention . AttritiMany training programs have performed retrospective reviews of the reasons and types of remediation in their programs, in hopes of identifying early on those trainees at risk. Interestingly, despite the technical demands of surgical training, deficiency in technical skills is an uncommon reason for remediation and a rarely a reason for dismissal . When exGiven the high incidence of non-technical and non-cognitive issues requiring remediation in surgical training, it is not surprising that current methods of evaluating trainees or screening applicants for training fail to identify those at risk. In the U.S., traditional screening of residency applicants includes review of U.S. Medical Licensing Examination (USMLE) scores, medical school grades, letters of recommendation, personal statements, and unstructured interviews. However, these tools have been criticized not only for their inability to predict performance in residency, but also to identify those most at risk for remediation and attrition . Thus, tto do rather than the abilityto know by placing candidates “in the situation” separates SJTs from more traditional, straight-forward assessments such as multiple-choice questions, short answer questions, and the like. As such, we explore the following research questions in this study:In this study, we explore the extent to which a situational judgment test (SJT) can predict which trainees are likely to require remediation or leave the program. SJTs present candidates with a variety of situations they would likely to encounter if they were brought into the organization, and are asked to make judgments about the effectiveness of possible responses. SJTs have become widespread across organizations outside of medicine, likely because of the robust body of validity evidence that exists showing their ability to predict on-the-job performance , reduce 1.Can SJT scores be used to predict remediation and attrition in residency?2.To what extent can SJT scores be used to complement examination scores in predicting resident remediation and attrition?Participants included postgraduate trainees in a single general surgery residency program in the United States. The program was chosen because of its relatively large number of residents and robust assessment database.A pool of 50 situational judgment test items were created by a team of MDs and PhDs. Semi-structured interviews with subject matter experts (SMEs) and review of historical resident performance data were used to identify which competencies were critical for residents to be successful in the program. Data from these interviews informed the development of SJT items, similar to prior studies . FacultyThe final SJT was given to all postgraduate year (PGY) 1 - 5 residents at the beginning of the academic year. Results from the assessment were neither used for any developmental or assessment purpose nor were they shared with program faculty.Remediation data, defined as any documented action taken by program leadership to improve a resident deficit, were collected for the following two years. All remediation events were captured as dichotomous (yes/no) variables and categorized according to the Accreditation Council for Graduate Medical Education (ACGME) core competencies: medical knowledge, patient care, interpersonal and communication skills, professionalism, systems-based practice, and problem-based learning and improvement.Resident attrition data were also collected. Residents who left the program prior to graduation for any reason  were included in the turnover group according to a dichotomous (yes/no) labeling.χ2 were used to compare the SJT score distributions for trainees who were marked positive for either remediation or attrition over the two-year time frame. As USMLE 1 scores are traditionally used as the primary initial screening tool among residency Program Directors in the United States , these data were also included as a potential predictor for remediation and turnover.Mann-Whitney U tests andComplete data were available for 93% (52/56) of residents in the program. Residents consisted of categorical PGY1s (25%), PGY2s (19.2%), PGY3s (25%), PGY4s (17.3%), and PGY5s (13.5%).Twenty-one percent (11/52) of residents required at least one remediation intervention within the one (for PGY5s) to two years after completing the SJT. The majority of these interventions were related to deficits in professionalism (56%), with the remaining related to decision-making (33%) or technical skills (11%). Mann-Whitney U tests revealed a significant difference in SJT performance between those needing remediation versus those who did not require remediation, such that those requiring remediation performed worse on the assessment (p = 0.04). Further exploration of remediation frequency by SJT score quartile reveals that trainees performing below top quartile of the SJT is associated with a 3-5 times higher likelihood of receiving a remediation intervention.Eight percent (4/52) of residents left the program within the one to two years after completing the SJT. Two residents (PGY2 and PGY3) transferred to another general surgery program and the remaining two (both PGY1s) joined another medical specialty. There was not a significant difference in SJT scores between those who remained in the program 1-2 years after completing the assessment (p = 0.41).Finally, we explore the relationship among USMLE1 scores and these variables. SJT and USMLE1 scores were not correlated. Additionally there were no differences in USMLE scores between the two groups in either remediation interventions  or attrition .Identifying residents at risk for performance deficiencies and turnover is an important, but complex, task for training program leaders. Problem residents can put a strain on the program director and clinical faculty, increase the workload of other healthcare providers, and complicate patient care activities . AdditioWe explored the extent to which SJTs could be created and used to accurately identify residents at risk for remediation, and compared its utility to the most common screening tool currently used - licensing examination scores. By following residents in a single institution over time, we found that SJTs can indeed predict those at risk for remediation. Similar to other work , our datThis study failed to find support in this sample for using the SJT to identify who might leave residency prior to graduation. Although the four individuals who did leave the program within the two-year timeframe did have lower scores on the SJT, the differences among this small group were not robust enough to reach significance. Additionally, given that the majority of attrition occurs within the first few years of surgical training , the PGYOur findings also contribute to the burgeoning literature showing the limited value of using USMLE scores for selection . AlthougAs with any study, these findings are not without limitations. First, these data include just one sample of surgery trainees within a single institution. Although the institution examined represents one of the largest general surgery training programs in the U.S., and thus includes the most robust data available from just one program, it is nonetheless a single institution study. However, given that the prevalence of remediation and attrition in this sample was lower than the national rate, these data may actually underestimate the role of the SJTs in predicting these outcomes. Further, the USMLE scores in this sample are well above the national mean. It is unknown if inclusion of a wider array of USMLE scores would have resulted in different outcomes. Finally, although the remediation data included here was retrieved from program records and thus reflect the most serious instances that were addressed, it may not fully capture all instances wherein a trainee may have received coaching from a number of different supervisors or colleagues. As with any exercise in measuring efficacy of interventions or tools, we are at the mercy of the assessment data available.These data demonstrate that SJTs can be created to effectively identify surgery residents most at risk for remediation across a two-year timeframe. These data provide an additional layer of validity evidence to support the role of SJTs in surgical education and align with other studies failing to find linkages between USMLE scores and residency performance criterion.•Tools are lacking to identify surgery residents most at risk for remediation and attrition•SJTs can be created to effectively identify surgery residents most at risk for remediation across a two-year timeframeAimee K. Gardner is Co-Founder of SurgWise Consulting and Assistant Dean of Ressearch & Evaluation at Baylor College of Medicine.Brian J. Dunkin, MD is Co-Founder of SurgWise Consulting and Professor of Surgery at Houston Methodist Hospital. This review has been migrated. The reviewer awarded 5 stars out of 5 Thank you, Aimee and Brian, for this well-written paper reporting what appears to have been a carefully-executed and important study. The positive findings, that your situational judgement tests (SJTs) predict which trainees will require remediation in relation particularly to professionalism issues in the subsequent one to two years, are truly impressive and surprising, given the concerns that have been raised about the susceptibility of SJTs to faking. This effect could be expected to be particularly prevalent among intellectually able candidates, such as those selected for surgical training. Accordingly, it would be really interesting to know more about how the SJTs were developed and how they appear to have avoided this effect – though I certainly take Barbara’s point about the risks of revealing too much detail in an environment that includes predatory commercial coaching companies. On the question of ‘what to do’ prospectively with the trainees for whom SJTs predict a high risk of professionalism breaches (apart from not selecting them!), there is some evidence pointing to the value of small-group educational interventions that promote affective domain learning through reflection on emotionally impactive clinical experiences  and deliberate connection with the professional values they highlight. It would be interesting to see if providing this kind of intervention to all trainees, prospectively, ‘flattens’ the effect of the SJT scores on the rate of requiring professionalism remediation.Reviewer Expertise:NANo decision status is available This review has been migrated. The reviewer awarded 4 stars out of 5 It is good to see research on non -cognitive aspects of trainees. At the end of the day , unprofessional behaviour is what gets trainees into difficulty with regulatory bodies and institutions . As the authors state 1/3 of trainees in surgical training will require remediation. Some of these trainees are likely to abandon training . Any tool which may help to identify these trainees in difficulty , early enough , would be most welcome .Most of medicine is not “black and white “ . It is in the grey area where most trainees have difficulty . The script concordance tests have been popular for this reason . In this interesting study, trainees took part in Situation Judgement Testing and were followed up to assess requirements for remediation; poor test scores correlated with higher rates of remediation. Examples of SJT would have been useful , as were provided in the studies on script concordance testing . The study raises the question when to intervene: should we intervene with a low score or wait till something goes wrong ? This is a single institution study with a small sample size; it has internal validity ,but generalisation may be difficult. It will need a larger sample size and multicentre involvement to address this .Reviewer Expertise:NANo decision status is available This review has been migrated. The reviewer awarded 4 stars out of 5 Given the predatory nature of the commercial coaching industry, I am not surprised that the authors did not share details of their SJT items. However, even though the small number of participants in this study precludes generalisation, I really like the robust process of item development that was described. Using data from historical cases together with SME experience to both develop and review the items may well have contributed to the significant findings. An interesting question would be whether this particular SJT relates in the same way to performance at a different institution - such information would help us know if a bespoke test is needed for each training program's specific context.Reviewer Expertise:NANo decision status is available This review has been migrated. The reviewer awarded 4 stars out of 5 Thank you for inviting me to review this article. Like the other three reviewers, I appreciated the well-written case study that explored the use of situational judgment tests to identify at-risk learners. This article will be of interest to educators working with learners in health professional education at any stage training. For example, at my medical school, we are seeking to enhance the identification of “non-technical and non-cognitive issues” in our students earlier in the programme. We have not discussed the use and benefits of situational judgment tests as a possible solution. As reviewer Ken Masters suggested, sharing the situational judgment test constructed by the authors would have given me something concrete to present to my colleagues for discussion. Nevertheless, this article will be useful to stimulate conversation about how we might approach building an instrument fit for the purpose of our educational programme that will enable us to detect students who would benefit from more support earlier in their training.Reviewer Expertise:NANo decision status is available This review has been migrated. The reviewer awarded 4 stars out of 5 This is an interesting and well done study and shows the potential utility of the situational judgment test for analyzing which residents may be at risk for remediation. Overall I think this makes a positive contribution to the literature regarding assessment. There are a few points which could be clarified/strengthened for future work:(1) The authors describe the creation of the SJT by a “team of MDs and PhDs” and the identification of “which competencies were critical for residents to be successful in the program.” It would be helpful to have a list of the competencies and the questions for other programs  who may wish to create their own SJTs. In addition, having this information accessible would be helpful in terms of the breakdown of the competencies – are they mostly professional, or are they technical/specialty-specific? This may help address how generalizable the findings are to non-surgical specialties(2) The authors found that most of the remediation interventions were related to deficits in professionalism, but that 33% were related to decision making and 11% to technical skills. Again, it would be useful to have lists of the exact type of deficiencies (what types of “decision making” errors – medication choice? Test ordering? Level of severity of illness? Etc) to better determine the generalizability of the study(3) It would be helpful, as another reviewer suggested, for the authors to describe “next steps” – I could see this expanding either (a) to other specialties within the same center or (b) to surgical specialties at other centers – the overall goal of course would be to all specialties at all centers, but it would be helpful to think about the next immediate goal.Reviewer Expertise:NANo decision status is available This review has been migrated. The reviewer awarded 4 stars out of 5 A succinctly and well-written case study, clearly laying out the issues (and their importance) to be addressed, and situating the study within the literature and practical context. In particular, as the authors note, the study highlights the weakness of relying on USMLE scores for selection. I hope that the authors are planning to expand their research into other centres, in order to determine the generalisability of their SJT. On that note, it is a pity that the authors have not shared their SJT; I believe that it would of value to others working in the field.Reviewer Expertise:NANo decision status is available This review has been migrated. The reviewer awarded 4 stars out of 5 Thank you. I was interested in this as there has been much discussion of SJTs on this (Europe) side of the pond.One issue is whether the assessors can agree on the correct answers. Another is what action to take as a result of this kind of study. This paper found an association between the SJT and remediation events; most of which were about professionalism. There was no correlation with USMLE results.I think one could have predicted this; indeed if there was no association with events one would wonder if the SJT was doing anything at all. Neither is it surprising that there was no correlation with the USMLE as this and similar assessments are clearly assessing a different domain to the SJT.The problem is what to do? The authors suggest early intervention with those with a low score which certainly seems reasonable, assuming that remediation works. The wider question is whether SJTs can predict failure. We need a prediction more than a association. How well does a low score on the SJT predict subsequent problems and how good does that prediction need to be before we take action such as terminating the course. This paper suggests that that is still some way off.Reviewer Expertise:NANo decision status is available


Access to, and occupational performance in, out-of-home-life-spaces is linked to health, wellbeing and quality of life for older adults. There is little evidence of how this relates to older adults with limited resources in an African urban context.To describe the out-of-home-life-spaces accessed and valued by older adults with limited resources, living in an urban South African setting.An exploratory concurrent mixed methods study saw 84 rehabilitation clinicians conduct 393 face-to-face interviews with older adults. Clinicians produced reflective field notes and participated in focus groups. Quantitative data were analysed using descriptive statistics with SPSS Version X. Qualitative data were analysed through inductive content analysis.Older adults walked, used mini-bus taxis or private vehicles to get to places of worship, medical facilities, shops, family and friends and special interest gatherings on a weekly or monthly frequency. Lack of funds was the main barrier. Older adults aspired to travel, go on holiday and to visit out-of-town family homes.Exploring the daily lived experience of older, urban South Africans with limited resources brought to light the value they attribute to participation in activities that contribute to the wellbeing of their families and communities. Such activities are found in a variety of life spaces.Results could inform policy makers and service providers in their planning of community mobility, transportation services and health care, for older adults with limited resources. Over the life course, people live their lives in a variety of spaces that could potentially extend from where they sleep, to places across the globe. However, as people age, they may experience life space constriction, which negatively affects their wellbeing and health  were invited to participate as field researchers in the study. A multi-professional group of 125 clinicians volunteered and attended research training workshops at their places of work and received research kits that contained the interview forms, notes, appreciation certificates and stationery needed for the research. Of these, 84 clinicians completed and complied with the research requirement. During these workshops, the ethical and methodological principles of the research were discussed and the interview form was introduced. The principles of semi-structured interviewing, keeping of field notes, professional reflection and debriefing were reviewed. The 84 clinicians were composed of 28 occupational therapists and 17 occupational therapy technicians, 21 physiotherapists and four physiotherapy technicians, 11 speech and hearing therapists and audiologists and three podiatrists. The first author kept regular contact with participating rehabilitation clinicians, coordinating data collection throughout the research process.Convenience sampling was used to recruit 393 community dwelling adults over the age of 65. Field researchers, who as per public healthcare regulation, were in uniform with their names and area of practice visible introduced themselves if there was no prior relationship. They recruited older adult participants, in-person, during participants’ routine visits to public healthcare facilities. Some participants were service users while others accompanied friends or relatives. Field researchers were asked to recruit as many participants as possible; the numbers they recruited ranged from 1 to 12, with a median of five participants. The decision to recruit older adults at primary healthcare clinics was made as these venues are strategically situated in urban areas, townships, and informal settlements and offered proficient access to the research population. All quotes in the article have participation identifiers that van be explained as follows: (1) the district in Gauteng and the number of the interview (2) the participant gender and (3) the participant age. The districts are defined by a letter: (1) T=Twane; (2) M=Johannesburg Metro; (3) E=Ekurhuleni; (4) W=West Rand; and (5) S=Sedibeng.There were three components to data collection: a 27-item semi-structured interview with the participants, field notes capturing the insights and reflections of the field researchers, and focus groups facilitated by the first author with the field researchers. These focus groups captured their thoughts and insights about the interview process. This article presents an aspect from a larger study with these different sources and is focused predominantly on out-of-home life spaces emerging from the interviews with participating older adults.The 27-item English semi-structured interview guide was paper-based and developed for the local context, based on researchers’ expertise in the field of community mobility, and their work with older adults. It was piloted in February 2018 to improve the content validity of the tool. The pilot study comprised a focus group with four older adults similar to the target population, using the speak-aloud technique Charters . Pilot pSemi-structured interviews, comprising 9 open and 18 closed questions, were chosen as a culturally sensitive method of data collection for the multi-ethnic study population . The interview guide also acknowledged the importance of oral traditions in an African context Tuwe  by invitData collection commenced with field researchers conducting one interview per older adult in accordance with the 27-item semi-structured interview guide. On the same day and place they went to visit a public healthcare facility, older adult participants took part in a face-to-face interview for this study. Interviews took place in rehabilitation treatment areas and were conducted by trained volunteering rehabilitation clinicians. Older adult participants came to the healthcare facilities accompanied or unaccompanied. They received a certificate of appreciation for their participation. Due to a lack of funding for the research and logistics of the research context, no refreshment was provided.n = 84) complied with all aspects of the study by conducting interviews, keeping field notes, and taking part in debriefing and discussion groups. Lastly, participating rehabilitation clinicians took part in audio recorded group discussions that were facilitated by the first author and explored final thoughts and insights; these were used to complement and enrich the findings.All interviews took place between 01 June and 14 September 2018. During and directly after the interviews, field researchers kept field notes in handwriting on the interview form, to capture insights and reflections they deemed noteworthy as prompted by the interview guide. Of the trained and equipped clinicians, 67% , ‘Why is it important for you? and Are there places you want to go to or things you want to do but find it difficulty or unable to?’ There were five spaces in which their answers were captured. For each place named, participants were asked why they did not go there, or found it difficult. ‘What places would you go to, and things would you do if you had no transport mobility restrictions?’For the last question, interviewers prompted participants to disregard cost and health and other restrictions. For each place named, participants were asked the meaning of that place, and what mode of transport they would use to get there.Data analysis for the results focussed on: (1) the characteristics of adults aged 65+ who attend state primary health clinics in Gauteng, South Africa, (2) the valued out-of-home places they visit, how do they get there, and why are they meaningful, and (3) where adults aged 65+ with low incomes wish they could go? The interviews, field notes and transcribed audio recordings were captured on Microsoft Excel and Word. Closed questions were captured using the response options in the interview guide. The Excel spreadsheet was uploaded to the SPSS X IBM Corp  and the Data analysis saw descriptive quantitative analysis of biographical data. Inductive content analysis was conducted to identify categories of out-of-home life spaces, modes of transport used and the meanings of places. Statistical analysis include counting the number of places named, the mean number of places named by each participant, the number of places visited in each category, the frequency of visiting places in each category, the number of modes of transport used and transport type for each category of place. Pivot tables and principal component analysis attempted to determine the relationship between categories of out-of-home places, frequency of visits, and modes of transport used. However, the analysis revealed no meaningful or consistent relationships.The face validity of the interview guide was improved by means of the pilot study. Inter-rater reliability was addressed when the first author provided the same training on data collection techniques to all rehabilitation clinicians. The operationalisation of the research was purposefully kept simple and well-documented in the protocol, consent forms and various dissemination formats to improve reproducibility of the study.Stellenbosch University’s Human Research Ethics Committee (HREC) provided ethical approval, Ref No: N18/01/003. Gauteng Healthcare’s Research Committee approved the research, allowing the conducting within Gauteng Public Healthcare, Ref No: DRC Ref 2018-03-008. The South African National Health Research Database (NHRD) registered the study, Ref No: GP201802 022. All work was conducted in accordance with the Declaration of Helsinki  attendance registers, paper-based, electronic and audio interviews, discussion groups and feedback forms. Clinicians received CPD certificates. Other than that, they and participating older adults were not remunerated and incurred no cost as interviews were done at the healthcare facilities where they worked or that they visited for scheduled rehabilitation services. The data collection stage was concluded at participating clinicians’ places of work with a debriefing session and discussion group. In 2019, results were analysed, summarised and disseminated to all participating rehabilitation clinicians and to rehabilitation services management in the head office of Gauteng Health. This was done in the form of emails and oral presentations at staff meetings and stakeholder forums. A report was sent to the Gauteng province’s Head of Government Office.n = 16) of the older adults lived alone. Most reported living in a brick house (91%) with indoor access to electricity and water (92.7%). Interviewed older adults shared their home with up to 18 other people . Although more than half (52%) had no schooling or primary education, the majority spoke three or more languages (60%). Most (92.7%) were financially dependent on the non-contributory government pension and living on less than 4 US dollars a day. This income supported an average of two other people .In total, 393 older adults, aged between 65 and 98 years  were interviewed see . Clinicin = 39), were within the urban area of Gauteng province.The 393 interviewed older adults named 1118 currently visited out-of-home life spaces in total, a mean of 2.8 places per person. Inductive content analysis reduced this list to 23 categories of out-of-home places within six life space themes. These themes, in order of frequency with which they were named, included: (1) community and social spaces, (2) medical facilities, (3) shops and public amenities, (4) family and friends, (5) places for recreation and physical activity, and (6) work. All places, except annual family visits  because ‘We must be part of our community’. Of these, places of worship were most frequently named . Places of worship offered weekly opportunities for spirituality: ‘In church people pray for me. I go for healing prayers’ . Also: ‘I go because I love God too much’ , and ‘So I can go to heaven when I die’  and ‘Because the preacher, who talks to God, tells me God loves me and this makes me feel all right’ . Some participants were elders or deacons or fulfilled other leadership positions. Places of worship also enabled participation in charity and social upliftment activities, such as soup kitchens and support outreaches to orphanages and homeless shelters. Attending funerals, even if you did not know the deceased or the family, was reported as much valued. This involved food preparation and expression of bereavement to show support and respect towards the deceased. Some places of worship provided opportunities for older members to go camping and/or on holiday and organised festivals and events such as soccer leagues. Transport to an annual mega-church gathering where congregations gather in different provinces for a few days at a time, was also arranged. Places of worship were accessed mostly through walking  and minibus taxi  and less frequently by motor vehicle . The latter was reported to take the form of ride-share clubs.Community support groups and special interest groups were mentioned as formal or informal gatherings taking place in private homes, funeral parlours, community halls, all-purpose community centres, healthcare centres and care homes. Seniors’ interest groups provided opportunities to ‘gym’ (exercise), grow food in communal gardens, prepare food for the soup kitchen, ‘to learn about new things’, to cook and eat together, sew and crochet, socialise, dance and sing. As one participant noted, ‘Singing in the choir is too good for my spirit’ . Reportedly, a sewing group always began with a prayer session. Lunch was often prepared and enjoyed collectively after a group gathering, and singing and dancing occurred spontaneously, if or when the occasion allowed it. The value of the gatherings was summed by a participant as, ‘I go there to mix with people when I am lonely or stressed’ .Being useful and involved in helping their community was a prevalent theme throughout the interviews. An older adult reported going to the local primary school to ‘watch the children so the teachers can have tea-time’ . Another reported ‘I visit the sick people in my community and then I pray for them’ . Working in community vegetable gardens was noted as important for socialising and to produce ‘food to feed us’ as they also sold and processed the vegetables from the gardens for personal and community use, or to supply soup kitchens in their vicinities. At one community health centre, a retired nurse taught other older adult volunteers how to bed-wash bedridden individuals. Using designated transport, they then visited homes in their community where families required assistance with caregiving tasks. As one participant noted, ‘You are only somebody if you help somebody’ .Other community occupations included stokvel and burial societies . A stokvel is a savings scheme where by-invitation-only groups of people contribute fixed sums of money to a central fund on a weekly, fortnightly, or monthly basis. The accumulated fund is allocated to a member of the group on a rotational basis at a stokvel meeting. Burial societies, which meet regularly, are another informal financial group formed by people to cover the costs and cultural expectations of a member’s funeral. If a group member dies, all members are expected to contribute to their funeral costs.Although all participants were interviewed at a medical facility, only 69% (271/393) participants included these facilities in their life space. Their visits were mostly monthly . Mobility modes included walking, minibus taxi, private car, bus and other forms of transport. Participants talked about visiting medical facilities, ‘to get my medicine’, ‘because the doctor says I must go’ or because ‘I have to go to know about my sickness’. Six participants also talked about visiting medical facilities for the health needs of family members, ‘I have to take my grandchild (who is disabled) to see the therapists for his exercise’ , and ‘My wife is in a wheelchair so I must push her to see the nurses’ .Most participants  reported visiting shops and public amenities. These places included a variety of shopping centres, income-grant pay points, local government and postal services, and grandchildren’s school. Journeys were mostly made by minibus taxi , but also walking  and private motor vehicle . Most participants reported visiting shops and amenities monthly , with very few reporting daily visits . Shops were visited for a variety of reasons. Shopping centres often had points where participants went to receive the monthly old age grant. At shopping centres, they also purchased groceries and electricity for prepaid metres. They talked about paying accounts and running family errands as their adult children were working. Visits to shopping centres were noted as outings and socialising events, appearing to be a means of staying active and avoiding boredom. Window shopping, ‘buying with my eyes’ was a favoured pastime. One participant noted ‘I enjoy seeing all the people. Being able to talk to everybody and hearing what is happening’ . Another participant reported visiting the shops for self-care ‘I have to do my nails and my hair… it makes me so happy’ .Some visits to family were frequent (weekly or monthly) and to people nearby, while other visits were annual and to ancestral homelands. A participant said:‘I have to go to Swaziland to go and see my cattle. I love my family. We need to keep our family relations. The grandchildren must know who you are’ .Walking to friends and family close by was common, and was associated with daily  and weekly  visits. Minibus taxis and private cars were used to visit families across town, usually monthly . Of all the bus journeys reported, almost half were taken to visit family and friends . Annual trips to family in faraway places  were taken by bus, minibus taxi, or in a rideshare with someone who had a motor vehicle.n = 62 places). Yoga, meditation, tennis, squash, fishing and soccer  were named as sports. Only eight participants named theatre or cinema. Most lived in Laudium, a suburb with a predominantly Indian community. Participants talked about watching ‘our movies’ (Bollywood) in a community centre. Two participants reported visiting a beerhall (Shebeen), while one reported gambling.Places for leisure and physical activity were seldom named  or weekly  basis. They either walked , used a minibus taxi , or a private vehicle .n = 268) reporting 367 places they wanted to go to but found difficult to get to. Community and social spaces that were difficult to reach included political, religious and cultural gatherings. Large specialist teaching hospitals were difficult to reach medical facilities . Shops and public amenities that were difficult to reach included wholesale and trendy shopping outlets . Reaching family and friends was a barrier for many , as were recreational and leisure facilities  such as Carnival City, sports stadiums and the Johannesburg Zoo. Fifty participants reported not being able to go on holidays. Barriers were predominantly related to financial constraints , health restriction  and transport problems . Crime and community unrest  on route to, or at places of value was also a barrier. These are expanded on in Most participants experienced restricted life space mobility, with 68% , with destinations inside South Africa being the most popular. Many participants mentioned never having seen the ocean and wanting to go on a ‘beach holiday’. Popular tourist and historical sites were also mentioned, such as Robben Island; one participant said, ‘I want to see for my own eyes where (Nelson) Mandela and  Sisulu were put in jail’ . Other responses indicated places similar to those mentioned as difficult to go to. Of the 353 places named, frequently mentioned places were homes of family and friends whom they had not seen for many years . Religious and/or spiritual journeys  to places such as Mecca or Jerusalem were mentioned, as were wholesale shopping outlets , day outings to the zoo  and theme parks . Participants who had worked as domestic workers in private family homes mentioned wanting to go and visit previous employers , ‘I worked for that family for 37 years and I miss them and I want to see the children; if they are married and grown up now’ .Interviews were concluded with a question: What places would you go to, and things would you do if there were no restrictions or barriers that kept you from doing so? From this question, holidays emerged as an out-of-home place that was out of reach. Of all the participants who responded to this question , all wanted to travel beyond their urban area or go on holiday  or recruit a generalisable sample of older adults. Instead, this research was situated within an interpretivist paradigm in which the subjective reality of older adults’ life spaces was of primary interest. We wanted to understand each person’s reality based on their experience of that reality. For that reason, we deliberately placed emphasis on the inductive analysis of where older adults in Gauteng go, and interpreted these findings as life spaces that were within reach, difficult to get to, or out of reach. It did not matter how far away from a person’s home a destination was, we wanted to know if they could get there or not. We viewed participants’ quotes as critical to understanding their reality. The quantitative data was complementary to this analysis. It helped us understand how many participants went to places, the frequency with which people went there, and their most common modes of transport. This article can therefore be criticised for not analysing the quantitative data to its full, objective, potential. But can also be criticised for not using in-depth semi-structured interviews with a smaller number of participants. This study and the methodology we used may raise more questions than answers. As only the second study in Africa to address the community mobility needs of older people, the hope is that the questions raised for reviewers and readers will be a spark for future research.Another potential limitation was balanced reporting of quantitative and qualitative findings obtained using mixed methodology to answer our research question. Such complexities associated with mixed methods have been described as a potential  Plastow . This arA report (United Nations The critical importance of spiritual occupations for older South Africans with limited income is shown in this study. It also brought to light the value they attribute to participation in activities that contribute to the wellbeing of their families and communities, and that such activities are found in a variety of life spaces. Policy makers, researchers and service providers should take note of this, and incorporate social participation into their planning of community mobility, transportation services and health care for this cohort. Finally, the findings of this study taught us that older adults with limited income are an underutilised support resource in South Africa, that could be activated and mobilised to the benefit of the communities they live in.


Emotional intelligence (EI) is associated with a range of positive health, wellbeing, and behavioral outcomes. The present article describes the development and validation of an online training program for increasing EI abilities in adults. The training program was based on theoretical models of emotional functioning and empirical literature on successful approaches for training socioemotional skills and resilience.n = 168) or a matched placebo control training program (n = 158). Each program involved 10-12 hours of engaging online content and was completed during either a 1-week (n = 175) or 3-week (n = 151) period.After an initial design, programming, and refinement process, the completed online program was tested for efficacy in a sample of 326 participants  from the general population. Participants were randomly assigned to complete either the EI training program  measures of EI (relative to placebo), indicating self-perceived improvements in recognizing emotions, understanding emotions, and managing the emotions of others. Moreover, those in the EI training also showed increased scores in standard performance-based  EI measures, demonstrating an increased ability to strategically use and manage emotions relative to placebo. Improvements to performance measures also remained significantly higher than baseline when measured six months after completing the training. The training was also well-received and described as helpful and engaging.Following a rigorous iterative development process, we created a comprehensive and empirically based online training program that is well-received and engaging. The program reliably improves both trait and ability EI outcomes and gains are sustained up to six months post-training. This program could provide an easy and scalable method for building emotional intelligence in a variety of settings. Emotions figure prominently in everyday life, yet people differ in their ability to perceive, process, regulate, and utilize emotions in an adaptive manner . This inDespite the relative success of previous intervention work, there still remains a need for strong, empirically based EI interventions, and there are remaining questions about the content, design, and administration of such inventions that need to be addressed . For insA major source of debate and confusion in the EI literature has revolved around how to best conceptualize and measure EI . The funor trait EI measures as outcomes, making it difficult to make systematic comparisons between the different conceptualizations of EI. This is important because some have noted that the two approaches may be distinct and complementary , and the training modules were developed and refined through an extensive multiple-iteration process. Once finalized, the intervention was tested against a closely matched placebo control program using a randomized control group design in a large sample of healthy adults. Our primary hypothesis was that individuals assigned to the EI training program would show improvements to both ability-and trait-based EI scores relative to those completing a similarly engaging active control training program. As a secondary hypothesis, we tested whether administering the program in a distributed manner (over 3 weeks) would increase the learning and retention of the material relative to a compressed administration of the same content (over 1 week).In the subsequent sections, we describe the development and refinement process and the methods for program validation.Conceptual Development. The goal of this phase of the project was to develop a training intervention that was grounded in prior research on EI, emotion theory, and skill development. To that end, a panel of experts on emotion theory and clinical psychology  was assembled to provide input regarding potential ways to improve upon an early pilot version of the program that was described elsewhere  program, included multiple lessons with no emotional content  but were otherwise comparable to the EIT program. The PAT was similarly organized into a three-tier structure: Introduction to the basics of the scientific process (tier 1), learning about scientific practices and how they can be applied to better understand the external world (tier 2), and opportunities to apply knowledge (tier 3).Iterative Refinement. After the modules were fully designed, completed, and implemented, we tested and refined the programs in an iterative manner with small groups of healthy participants. Participants were enrolled in either the EIT or PAT programs and were instructed to record experiences with glitches and errors, issues with comprehension, and thoughts about the appearance and design of the program. Once a sample of n = 40 was achieved, data collection was paused in order to examine and correct any issues that were identified. This process was repeated four times for a total sample of n = 159, ageM = 22.69, 64% women. During program testing, we were also able to ask the participants to rate their perceptions of each module. Using a 7-point Likert scale, participants were asked to rate how helpful , how engaging , and their motivation to improve their emotional intelligence  after completing each module.To test our hypotheses that EI training would result in greater increases in ability-based and self-reported (trait) EI scores relative to individuals in the PAT program, and whether administering the program in a distributed manner (over 3 weeks) would increase the learning and retention of the material relative to a compressed administration (over 1 week), we conducted a large randomized trial comparing the EIT program versus the PAT program.Participants. Participants were recruited from a large southwestern university in the United States and the surrounding metropolitan area. The study was generically advertised as an “awareness” training study to avoid expectancy effects associated with emotional skills training. To ensure that our study was adequately powered, we conducted an a priori power analysis. Using the effect sizes found for the MSCEIT in the initial pilot study for the EIT program (n = 40 would be sufficient to detect a similar effect size for the interaction of interest (f = 0.28). However, for the critical between-group differences between the EIT and PAT training, we would need n = 148 per training condition for high power (1−β = 0.95) to detect similar effect sizes to the ones found in the pilot study (f = 0.21). Given the dynamic and longitudinal nature of the study, we also assumed that we would need to account for levels of attrition in our sample size estimate. Therefore, we aimed to recruit a large sample of n = 450 that would easily allow us to achieve this sample size, even after accounting for possible large levels of attrition . A total of n = 448 participants took part in the baseline assessment session, ageM = 23.72, 72% women, 6.5% Asian, 3.3% Black or African American, 21.0% Latino or Hispanic, 60.7% Caucasian, and 5.8% multiracial. Participants provided written informed consent and were compensated for their participation. The protocol for this study was reviewed and approved by the Institutional Review Board of the University of Arizona and the U.S. Army Human Research Protections Office. program , we founThe following measures were administered:Ability-Based EI. The MSCEIT  was used to assess self-reports of EI abilities. The SREIS is a 19-item scale that was specifically designed to map onto the skills assessed by the MSCEIT  is a 153-item scale that comprehensively assesses domains related to trait EI , how engaging , and how motivated they were to improve their emotional intelligence . In this case, however, participants rated the program as a whole, rather than each individual module.Procedure. Participants first reported to a laboratory where they completed demographics and baseline EI assessments and were randomly assigned to either the PAT or EIT program, and either the compressed (1 week) or the distributed (3 week) training schedule. After completing the program according to the assigned schedule, participants reported back to the laboratory to complete post-training EI assessments. All EI assessments were administered at both time points (pre- and post-training). We also conducted a six-month follow-up assessment to investigate whether the skills developed in the EIT program would demonstrate long-term persistence. It should be noted, however, that the scheduled follow-up assessments happened to align with the onset of the COVID-19 pandemic in March 2020. As the result of a university shutdown and moratorium on in-person research, several modifications were made to the study protocol, including administering all assessments online, adding flexibility in the expected timing for completing the follow-up, and collecting a smaller sample size than desired. A total of n = 91 participants completed the 6-month follow-up.Through the iterative data collection, we were able to identify and address a number of minor issues , major issues , problems with clarity , and negative personal perceptions . We were also able to determine that participants in the EIT program tended to have positive subjective experiences with the training. As shown in Baseline Descriptive Statistics and Comparison. Of the initial 448 participants, 122 did not have usable data from the post-training assessments due to scheduling issues, withdrawing from the study, or lack of compliance with the study protocol (27% attrition rate). In total, we were able to produce a high-quality dataset of n = 326, with n = 168 in the EIT condition , and n = 158 in the PAT condition . Means, standard deviations, and internal reliability of the assessments at baseline, as well as correlations between all measures, are presented in χ2 = 0.23, p = 0.628, nor did they differ in terms of demographic variables, ps > 0.602. Excluded participants did tend to score lower on the MSCEIT, t(440) = −3.58, p < 0.001, and TEIQue, t(445) = −2.98, p = 0.003, and were more likely to have been assigned to the 3-week training schedule, χ2 = 14.33, p < 0.001.Independent sample t-tests showed that participants in the EIT and PAT programs did not differ in terms of age, sex, or EI scores at baseline see . We condProgram Effects.f2 = 0.02 is considered small, f2 = 0.15 is considered medium, f2 = 0.35 is considered large  to “skilled” (>110 to <130) (d = 0.47). Comparatively, the PAT condition improved by 2.11 points (p = 0.013), reflecting a small effect size (d = 0.21). Although there was no difference at baseline, the EIT group demonstrated significantly higher MSCEIT Total scores at the post-training assessment (p = 0.042). Participants also showed improvements in each of the branches of the MSCEIT, but these changes were not statistically significant. Participants in the EIT program showed an increase in their total SREIS scores relative to those in the PAT program (p < 0.001), as well as improvements in the subscales related to perceiving (p < 0.002), understanding (p < 0.001), and managing the emotions of others (p = 0.043). Moreover, the increase in SREIS within the EIT group reflected a medium effect size (d = 0.46), while the PAT group was associated with a slight and very small decrease in self-reported EI (d = −0.05). Finally, the overall TEIQue score also had a marginal trend toward improvement in self-perceived emotional intelligence in the EIT relative to the PAT condition.Overall, participants in the EI training program tended to show small improvements to their EI scores relative to participants in the placebo condition [ed large ]. Particed large , p < 0.0to <130) , reflectr = 0.46, p < 0.001, indicating that participants who showed an increase in their SREIS scores also tended to show an increase in the TEIQue scores. However, the level of improvement for MSCEIT was unrelated to changes in SREIS, r = 0.07, p = 0.349, and the TEIQue, r = 0.08, p = 0.314. Further examination of the parallel MSCEIT and SREIS subscales also did not find significant correlations between performance-based and self-reported improvements in perceiving, r = 0.001, p = 0.993, using, r = 0.12, p = 0.165, understanding, r = 0.09, p = 0.231, and managing emotion, r = 0.12, p = 0.130 (SREIS managing self) and r = 0.01, p = 0.902 (SREIS managing others).For those in the EIT program, we examined relationships among improvements on each of the EI measures (quantified as the within-person difference between post-training and baseline scores for each measure). Changes in the SREIS and TEIQue total scores were correlated, Effects of Training Distribution Schedule. One of the study aims was to investigate whether distributing the training across 3 weeks would result in better learning and retention when compared to a compressed 1-week training. Linear mixed effects models with an added condition × time × distribution term revealed that only the SREIS perceiving subscale showed a significant difference, t = 2.69, p = 0.008, such that the effect of the EIT program was stronger in the distributed  training schedule versus the compressed  schedule only for that outcome variable. No other significant effects were observed, with p-values ranging from p = 0.073–0.915. We therefore conclude that the distribution of training did not affect the post-training outcomes.Program Perceptions. During the post-training assessment session, we asked participants to report on how helpful, engaging, and motivating the programs were. As shown in t(300) = 0.59, p = 0.553. However, participants in the EIT program perceived the training to be more helpful, t(300) = 8.33, p < 0.001, and were more motivated to improve their emotional intelligence, t(300) = 9.69, p < 0.001, relative to participants in the PAT program. These ratings were also highly correlated, such that participants who found their program to be helpful also tended to find it engaging  and were motivated to improve ; and participants who found their program engaging were also motivated to improve .Long-Term Persistence of EI Improvements. As initially designed, the six-month follow-up was intended to be a key component of the main analyses. Unfortunately, the COVID-19 pandemic emerged contemporaneously with the timing of the 6-month follow-ups, potentially affecting the emotional and mental health functioning of the participants and leading to a disruption in data collection. Therefore, due to COVID-related changes to the study protocol and the general climate in which participants completed the assessments, we cannot be fully confident that the following results represent an accurate evaluation of the long-term effectiveness of the EIT program. We therefore present the 6-month follow-up analyses separately and interpret these findings cautiously.n = 91 participants completed the 6-month follow-up with n = 37 in the PAT condition , and n = 54 in the EIT condition . There were no baseline differences found between the 91 participants who completed the 6-month follow-up assessments and the 357 participants who did not in terms of age, gender, training distribution condition, SREIS, and TEIQue total scores . The number of participants in each condition did not differ significantly, χ2 = 2.86, p = 0.099, but participants who completed the 6-month follow-up had higher MSCEIT total scores at baseline (M = 109.88 SD = 16.21) than the participants who did not complete the 6-month follow-up , t = 2.86, p = 0.004.In total, t = 2.57, p = 0.0112, f = 0.03. Post hoc analyses revealed that the MSCEIT scores of participants in the EIT program were significantly higher than they were at baseline, at Time 2, t = 5.99, p < 0.001, d = 0.47, and marginally higher at Time 3, t = 1.76, p = 0.083, d = 0.24, suggesting that the skills trained in the program were modestly retained across an extended period, despite the emergence of the worldwide COVID-19 pandemic. There was also a significant group × time interaction for the SREIS, t = 2.40, p = 0.0172, f = 0.01. However, post hoc analyses revealed that this effect was driven by increases in SREIS scores for EIT participants at Time 2, and their SREIS total scores were no longer significantly different than baseline at Time 3, t = 0.95, p = 0.349, d = 0.13. Finally, there was a marginally significant group × time interaction for the TEIQue, t = 1.95, p = 0.0522, f = 0.01, but these scores on the TEIQue did not significantly differ from baseline at Time 3, t = 1.57, p = 0.123, d = 0.21. There was no effect of training distribution on any of these scores, ps > 0.320.To analyze the long-term effects of training, we ran linear mixed effects models that accounted for differences in age, sex, time, and the interaction between program condition and time. There was a significant group × time interaction for the MSCEIT total score, The goal of this study was to develop an empirically based, online training program and investigate its efficacy in building and sustaining emotional intelligence. We created a roughly 10-h web-based training program based on the MSCEIT model of EI and theoretically related traditions such as emotion regulation, mindfulness, social skills, and clinical practice then investigated the efficacy of the program using a large sample and a randomized control trial design. As hypothesized, participants who completed the EI training program improved their total ability and self-reported EI scores relative to those in the placebo condition. Overall, participants in the EIT program demonstrated over a five-point increase in their ability-based EI, to elevate the mean performance from what the MSCEIT manual describes as the “competent” to the “skilled” range , consistAn important advancement provided by the present study was our inclusion of multiple assessments that measured both ability-based and trait EI to allow for systematic comparisons between the two theoretical approaches to EI. We found that the EI training resulted in overall improvements to both ability-based EI and self-reported EI. Overall, the EIT program was associated with increases in Total EI scores on the MSCEIT, the most well-established and widely used ability metric of EI. The total score represents a composite index of EI abilities that is broadly representative of an individual’s ability to detect emotional information from people and situations, understand how this information relates to contexts and goals, use that information to enhance thought processes, and effectively modulate emotions in oneself and others appropriately. In particular, the EIT program was most effective at improving the ability of participants to manage and control emotional responses based on consideration of the situational context, a key component of social–emotional success . The EITInterestingly, while the EIT program was effective at improving both ability and self-reported EI, within individuals overall improvements on the MSCEIT and SREIS were uncorrelated. This is consistent with prior work suggesting that trait and ability EI are only weakly correlated with one another, if at all . EssentiWe also found differences in how improvements to the different EI metrics were maintained over time . We found that increases in the MSCEIT score were marginally higher than the baseline 6 months following the EI training, whereas scores on the two self-reported EI assessments had returned to baseline levels. Overall, emotional traits should be fairly stable, but trait-related behaviors and self-perceptions can momentarily fluctuate based on context . ParticiWe also tested a second hypothesis related to the distribution schedule for completing the training content. As previous research on learning models raised the possibility that distributing training over longer periods of time could help participants learn and retain the intervention material more effectively than compressing the content into a shorter timeframe , we had Overall, the present study clearly shows that the EIT program produced improvements in several relevant EI metrics. However, we did not find significant results for all subscales and branches of the outcome measures, and the effects were smaller than what has typically been reported in published research using in-person training approaches . This maThe present study was designed as an initial validation of our novel EI intervention to show that teaching EI skills can result in an increase in quantified levels of EI. With evidence to suggest that the program can increase EI, it would be useful to continue investigating the generalizability of the program in relation to specific populations and external outcomes. This initial validation study tested the effectiveness of the EI intervention in a relatively healthy sample who tended to have above-average scores on each of the three EI measures assessed. It would be useful to extend the present research by investigating the intervention’s effectiveness in specialized samples; such as individuals with poor social–emotional functioning, individuals with clinical disorders characterized by emotional difficulties, or individuals in emotionally demanding roles such as the military or first responders . In addiWe have successfully developed and validated a novel online EI training program based on well-established theoretical models of emotional intelligence that incorporates empirically supported training content and methods. Our study design addressed several methodological limitations of prior studies of EI training programs. This large, randomized placebo-controlled validation study demonstrated that the program is well accepted and positively received by participants, significantly improves EI scores on both trait and ability measures, is robust to variations in the training schedule, and shows a trend toward sustaining EI skills up to 6 months post-training, even during the emotional challenges posed by the start of a worldwide pandemic crisis and nationwide lockdown orders. With further validation, this approach could provide a standardized and scalable training method for building critical emotional intelligence skills.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.The studies involving human participants were reviewed and approved by University of Arizona Institutional Review Board. The patients/participants provided their written informed consent to participate in this study.MD carried out the primary data organization and statistical analysis, wrote the initial draft of the manuscript, and contributed to revisions. RS contributed equally to the initial drafting of the manuscript and contributed to the initial study design and development of the EIT materials. SC, RW-L, JS, and SB executed data collection, organized the study database, and also reviewed and edited sections of the manuscript. LH reviewed and edited versions of the manuscript. KW, RL, and JA contributed to the initial conceptualization and redesign of the EIT materials and edited and reviewed drafts of the manuscript. ND, AA, and JV contributed to data collection and statistical analysis, and reviewed drafts of the manuscript. WK was responsible for the initial conceptualization and design of the project, obtaining the research funding, provided oversight of the EIT program design, data collection, statistical analysis and interpretation of the findings, and contributed to initial drafting and revisions of the manuscript. All authors contributed to the article and approved the submitted version.The study was supported by funding from the U.S. Army Medical Research and Development Command (W81XWH-16-1-0062) to WK.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Gender stereotypes and lack of access to relatable role models, mentors, and STEM opportunities have been suggested to deter middle school girls and students from underrepresented backgrounds away from STEM. Seeds of Success, an out-of-school program, is designed to inspire girls to consider STEM careers by countering gender stereotypes through relatable role models, promoting STEM confidence through STEM workshops and hands-on activities, and encouraging alignment between cultural and STEM identities through community-based STEM projects that develop leadership skills. Since 2015, the program has impacted 453 students who in turn have reached more than 42,777 people in Puerto Rico through their STEM Ambassadors projects. A robust mix-method evaluation of the 2020 and 2021 cohorts demonstrates significant improvements in participants’ STEM attitudes and science identity, as well as in their self-perception as a leader, confidence in their ability to succeed in science, knowledge about STEM careers and opportunities, and access to STEM role models. Moreover, 95% of participants intend to continue participating in STEM activities after the program and overall scores for the entire survey were significantly higher after the program than before. We discuss lessons learned for other programs seeking to empower girls from historically underserved backgrounds in STEM. Science, technology, engineering, and math (STEM) careers are among the highest-paying and fastest-growing careers in the U.S. . Yet, woThere are additional challenges with respect to girls from underrepresented or low-resource backgrounds who are more likely to lack access to STEM courses, resources, and supportive environments. For example, Hispanic high school students are interested in STEM subjects and careers at rates similar to those of students from majority groups. But Latinx high school students exhibit significantly lower STEM participation, preparation, college plans, and confidence in their STEM abilities . A lack Three strategies have shown to be successful in countering the factors that result in gender and race disparities: experience and skill development for fostering self-efficacy, exposure to relatable role models to improve STEM attitudes and increase science identity, and connectedness to community to reinforce the ‘fit’ between cultural and STEM identities. Based on this body of evidence, we established a set of activities  for the Seeds of Success Kit with educational materials related to STEM and STEM careers, a Foldscope activity to complete at home  science identity, (2) main program objectives, (3) STEM interest, (4) leadership identity, (5) STEM attitudes, and (6) numerical skills and perception. We also inquired about their satisfaction with program activities.A Pre-Intervention Survey consists of 52 closed-ended questions. Questions focus on participants’: (1) demographic characteristics, (2) science identity, (3), program objectives (4) STEM interest (5) leadership identity (6) STEM attitudes and (7) numerical skills and perception. Most questions are presented as 5-point Likert items. A few negative items are included to avoid acquiescence bias. Questions regarding participants’ perceptions of scientists come from the General Social Survey ; questioA Mid-point Satisfaction Survey consists of 30 Likert questions and four open-ended questions to examine the students’ satisfaction with the program activities. Questions are based on a published survey from “Assessing Women in Engineering” .A Post-Intervention Survey includes the Pre-Intervention non-demographic question set plus questions related to the girl’s satisfaction with the program’s components and overall impression of the program’s impact.All instruments were sent via email to program participants and collected through the Yale Qualtrics XM online survey platform. Surveys were tracked with a randomly generated ID number assigned to each participant at the beginning of the study. The Pre-Intervention Survey was sent after program orientation, the Mid-point Satisfaction Survey 4-months into the program, and the Post- Intervention Survey at the end of the program.A STEM Role Model Survey consists of 12 multiple selection questions, seven Likert questions, and three open-ended questions focused on (1) academic stage or professional work sector, (2) previous mentoring experience, (3) participation satisfaction and impressions, and (4) perceived impact of the program on themselves and on the girls. Surveys were sent via email at the end of the program.Survey descriptive statistics, reliability, and goodness-of-fit were calculated using IBM SPSS Statistics software package, v.29. The survey items and instruments have undergone reliability and validity testing. Cronbach’s alpha was used to estimate the internal consistency of the survey using the following measurement criterion: α ≥0.90 (high), α ≥0.80 (good), α ≥0.70 (adequate) . The KaiContent analysis was used to interpret participants’ experiences expressed in the open survey questions. A deductive approach was used to study program impact. We focused on manifest content to code the text’s visible and surface content. Before beginning analysis, we identified preconceived categories from the literature. Data were coded according to the predetermined categories. The reported quotes were taken from open-ended questions.The program application was disseminated through CienciaPR’s website (~40K visitors per month), social media channels (reach of ~30K per week), and newsletter , as well as through a media campaign that included a press release and radio and TV interviews with past students. School directors and teachers were also alerted about the application through a collaboration with the Department of Education of Puerto Rico (DEPR). In 2020 and 2021 we received 476 and 305 applications, respectively.Participants were selected by a committee of past mentors and based on a rubric that prioritized (a) need , (b) interest , (c) leadership potential , and (d) geographic diversity .A total of 241 girls participated in the program during the reported cycles (121 in 2020 and 120 in 2021). Participating girls came from 73 out of 78 municipalities in Puerto Rico  and inclIn order to receive their completion certificate and invitation to the closing ceremony, girls were required to participate (live if possible or asynchronously if they had a conflict) in the program orientation and leadership workshop, at least one mentoring session, two of the “Juntas Podemos” webinars, and a chemistry of everyday life hands-on workshop focused on concepts such as chemical reactions, pH, acids and bases, indicators and buffers, delivered by students leaders from the University of Puerto Rico, Rio Piedras’ student American Chemical Society (ACS) chapter. Although delivered online, participants were encouraged to interact with each other by commenting on each other’s questions or progress using the chat function or live in breakout groups during the peer mentoring sessions and hands-on workshops. Participants who completed these requirements and completed the community outreach project were recognized as STEM Ambassadors during the closing ceremony.To promote engagement, program communications strategies were based on videos, WhatsApp text messages, and posts that celebrated girls’ efforts in completing the required activities. These strategies were based on a 2020 survey that indicated that parents preferred WhatsApp compared to email. We actively monitor girls’ participation and potential risk of drop-out. When a girl notifies us that she is not continuing in the program, or when we notice a number of absences or incomplete milestones, a meeting is scheduled with the parents to understand the reasons and identify alternative participation plans.All participants both years completed the leadership workshop, 80% completed at least 1 mentoring session with a Latina STEM role model, 71% completed requirements for the final certificate, and 62% completed the final STEM Ambassadors project .As part of the program, girls were encouraged, guided and supported to design and complete a STEM Ambassador outreach activity. To assess the impact of the STEM Ambassadors project, girls had to submit an evidence form that included a brief explanation of their activity and experience, the number of people reached, and videos or pictures of their activity. The STEM Ambassadors projects during the 2020 and 2021 cohorts reached an estimated 27,904 people from 54 municipalities across PR . ProjectA total of 180 women in STEM (120 in 2020 and 60 in 2021) were recruited through the CienciaPR network  to mento“The most gratifying experience was to orient my girl about STEM careers, and through her, reach other girls…”“The most gratifying experience was being able to share and let her know the opportunities in STEM that students can have. To ensure that girls are aware of the opportunities that exist, preventing them from leaving high school without guidance.”In their post-intervention survey, most mentors  indicated that they had participated because they were interested in contributing to the development of girls in STEM disciplines and this was the goal that was most commonly accomplished at the end of the program (86.5%). A majority of mentors also agreed or highly agreed that the Seeds of Success program helped them feel confident being a role model for girls interested in STEM (84.8%), improving their mentoring skills for middle school girls (87.3%), contributing to the development of girls in STEM disciplines (92.4%), helping girls appreciate the importance of STEM disciplines for society (92.4%), inspiring girls to consider STEM careers (88.6%), and feeling motivated to seek other mentoring opportunities (90%). Overall, mentors greatly enjoyed their mentoring experience . Here are some of the mentor’s testimonies:Mentors were essential in guiding participants towards the completion of their STEM Ambassadors projects and providing constructive and helpful feedback in designing their projects, as pointed out by participants with a median value of 5 (strongly agree) for all satisfaction items related to mentoring .Before starting, participants and their parents were invited to take part in a study to assess the impact of the program. A total of 192 participants (79%) enrolled in the study and from this study sample, 127 (66%) completed all surveys. The school level and school type of enrolled study participants was comparable to program participants. Enrolled study participants were in the 7th (15%), 8th (39%) to 9th (46%) grades and came from public schools (63%), private schools (35%), and homeschooling (2%).Based on our Pre-Intervention survey, 25% of study participants had previously taken STEM honors or advanced classes and only 14% had been enrolled in a special science or engineering program. Based on PR census data , most paAccording to the data reported in the Mid-Intervention and Post-Intervention Surveys, study participants were highly satisfied with the activities carried out. All survey items about program activity satisfaction received a median value of 4 or higher in a scale of 1–5, with 5 being the most positive score. Mean score across all activities for the 2020 cycle was 4.37 and 4.5 for the 2021 cycle. A total of 97% of the students “liked” or “really liked” the program. The activities that participants liked the most according to their answers in open-text survey questions were the workshops and getting to know STEM professionals.Mean values for item answers in the Pre/Post-Intervention Surveys ranged from 2.33 to 4.55. Skewness  and Kurtosis values  are acceptable to prove normal univariate distribution. KMO value of 0.744 and Barlett’s test (2330.265) and significance (<0.001) have shown to be adequate. Cronbach’s alpha value at 0.908 estimates a high internal consistency.Exploratory Factor Analysis (EFA) with principal axis extraction method was used with an oblimin rotation method. A 6-factor solution was extracted that explained 47.25% of the variance . For simA Wilcoxon signed ranks test was used to compare the Pre- and Post-Intervention Survey scores for each individual item, for the six factors, (referred to here as categories), and for the survey as a whole. The categories of Science Identity, Program Objectives, STEM Attitudes, and Numerical Skills and Perception showed a significant difference, as seen in Most of the items in the Program Objective category showed a significant difference, revealing an increase in self-perception as a leader, confidence in their ability to succeed in science, knowledge about STEM careers and opportunities, and access to STEM role models after program participation . In addiAccording to open-text answers in the Post-Intervention Survey, the program helped participants understand different branches of science, think about their professional goals, confirm their interests in STEM, and feel closer to achieving their professional goals.“…Now I know more careers in STEM and I already have a clearer idea of what I want to study in the future.”“The Seeds of Success program was an unforgettable experience. Thanks to the program I found a profession that interests me and that I hope to study in the future. I think it’s a very inspiring initiative, and personally seeing how participating in the program paid off is very gratifying.”“I will always remember Seeds of Success as the program that began to open doors for me in order to be closer to fulfilling my dream of becoming a professional.”Girls also expressed how meeting female STEM professionals and other girls with the same interests helped them realize the importance of STEM professions. Some stressed the importance of meeting a variety of STEM professionals.“The thing that I most liked was being able to get to know different types of STEM jobs and realizing the importance of STEM professions.”“I liked that there were many girls like me interested in learning.”“What I liked the most was the interaction with my mentor. It taught me a lot and motivated me to become a scientist in the future. Her research area was very interesting.”“I loved the mentoring. Having a mentor to help me organize my ideas was an experience I had never had. I really thank my mentor for helping me follow my dreams of being able to lead and present issues to my community with pride and confidence in myself.”Interestingly, the “Juntas Podemos” webinars, helped participants see themselves as scientists, dismantle common science stereotypes, and realize that science and engineering are not innate abilities but rather skills you can develop.“What I liked the most about Seeds of Success was the series of talks by the scientists, since it showed me that I too have the possibility of one day being one of them and fulfilling my dreams.”“The thing I most liked about the Seeds of Success program was the “Juntas Podemos” series. I love these types of talks because they show you that scientists and engineers are ordinary people. You don’t have to be a genius to be a scientist. You just have to put your mind to it and work hard smartly.”“What I liked the most were the “Juntas Podemos” talks, because I was able to hear how each of these women achieved what they wanted and never gave up.”Participants also let us know that the STEM Ambassador projects helped them develop leadership skills. Specifically, they talked about how the project helped them develop perseverance, challenged them to implement new STEM initiatives in their communities, and made them feel good about sharing their knowledge with schoolmates.“It helped me a lot to develop as a leader. It helped me feel that if I try hard enough I can be what I want.”“The program helped me grow as a person. I have learned many things from different aspects of science. I even learned how to develop leadership and other important qualities. I am delighted to be part of a program like this.”“What I liked most about Seeds of Success was developing my leadership skills through a science project and being able to share it with my schoolmates.”“What I liked is that it is a different program that challenges you to do new things that you never thought would happen.”“This helped me to be a leader, thanks to my work the school director allowed me to have a club at school. It was simple but the next school year we are designing it to be very motivational and very professional.”This article presents an example of a program designed to promote STEM interest among girls from historically underrepresented backgrounds by leveraging three strategies: (a) exposure to STEM experiences, (b) relatable role models, and (c) empowering place-based STEM outreach projects. Here we discuss details about the program’s implementation, evidence of short-term impact, and key programmatic components.Over the years we have developed a diversified set of strategies to ensure broad and diverse recruitment of girls throughout PR. Our program is benefitted by CienciaPR’s extensive community, popular website , and longstanding and active presence on social media . We have also established a collaboration with the DEPR to disseminate information about the program. Perhaps the factor that has helped most in ensuring broad visibility has been CienciaPR’s healthy relationship with media outlets and the use of a media relations consultant to issue press releases during recruitment periods and assure radio and TV coverage for past program participants and their STEM Ambassadors projects. We have also begun piloting novel social media recruitment strategies suggested and managed by past participants. For example, for the recruitment of the 2020 cohort, we partnered with Latinas in STEAM, an Instagram and Facebook movement established by a Seeds of Success participant as part of her STEM Ambassadors project. Involving past participants is aligned with one of our core beliefs that “if you can see it, you can be it.”Our application evaluation rubric has allowed us to recruit a diverse set of participants with respect to school and geographic diversity and to target girls from underserved backgrounds as determined by past STEM experiences and school type (64% public school). Although we do not have data on girls’ race/ethnicity and family socioeconomic background (we have started collecting data with the 2022 cohort), we assume based on the most recent census that the majority of girls identify as Hispanic  completed all components of the program in a synchronic fashion, including the STEM Ambassadors project. Given the relatively high number of activities, the length of the program, and the challenges of implementing a STEM Ambassadors project, this participation rate is to be expected. Importantly, a majority of girls (80%) participated in the mentoring sessions and remained engaged throughout the program, receiving a certificate of participation at the end (71%).Over the years we have experimented with several strategies to promote participation, retention, and STEM Ambassadors project completion. A common cause for attrition encountered during discussions with the girls and their parents is a high load of classes at school. To counteract this effect, girls who are unable to participate live are given the opportunity to watch recordings of the workshops and “Juntas Podemos” webinars asynchronously so they may complete those requirements. We have also recently implemented an obligatory pre-application information session for girls and parents where we discuss required program activities and dates. Girls can choose to not finish the application process if they don’t think they will be able to participate in scheduled activities. In addition, we have implemented a time management panel featuring past program participants who share how they were able to balance school, personal activities, and the program activities. After implementing these strategies, we have seen an increase in girls’ retention.We have also implemented a number of strategies to improve program communications that have been well-received. In the past we found that solely focusing on website content and email communication through the parents was not effective. Communications improved when we began developing dynamic media content  and complementing emails with “Whatsapp” to send reminders, program communications, and celebrations of activity completion. We also developed a more attractive and defined branding and visual language that leveraged images and stories from past alumni and participating mentors. These were incorporated into program materials, including a printed calendar with important program dates. We believe this communication strategy has been critical for engaging and retaining girls throughout the program.Finally, with respect to the STEM Ambassadors projects, we recognize that this is perhaps the most challenging aspect of our program. To facilitate completion and development of leadership skills, mentors are instructed to discuss and advise the girls about their projects, we provide a guide with examples of different types of projects they may want to undertake, we model the development of an STEM Ambassadors project idea through a design thinking exercise during the initial meeting, we secure collaborations with STEM or other community organizations in Puerto Rico that the girls could work with, we provide ongoing staff support, and we award a modest stipend ($50) to help with incidental expenses. In addition, we have begun to incentivize completion by awarding sponsor-donated prizes to the most creative and impactful projects per region during the closing ceremony when all girls present their final projects, and by highlighting as many projects as possible in our social media and in the media campaigns for the recruitment of the next cohort.Participants from the 2020 and 2021 cohorts indicated high levels of satisfaction with the program in general and with each of the activities in which they participated. A statistical analysis of survey responses before and after program participation demonstrated significant changes in participants’ responses to the survey as a whole, as well as with respect to grouped items related to their attitudes towards STEM, scientific identity, and confidence in numerical skills. A set of questions meant to examine achievement of the stated program objectives was also highly significant, indicating that the program increased girls’ knowledge about STEM careers and STEM opportunities, access to role models and mentors, and self-efficacy in the sciences. We suspect that a lack of change in the grouped items related to STEM Leadership and STEM Interest could be due to a high baseline given the fact that girls were selected based on their high interest in STEM and demonstrated leadership qualities. It could also be due to not having provoked changes in these areas or not having well-elaborated survey items to measure these changes. Interestingly, although the questions for Leadership Identity were not significant at the group level, an item from the program objectives examining girls’ confidence in their capacity to be a leader was highly significant. Similarly, a significant number of girls’ responses to the open-text questions referenced the development of leadership skills and a new self-regard as leaders.Because our evaluation did not include a control group, we are unable to conclude that observed changes were caused by our program. However, studies have found that STEM interest and science identity tend to decrease among girls during middle school , suggestThe results regarding increases in science identity are encouraging. High levels of science identity  have been shown to be related to persistence in STEM academic and career paths . ScienceStereotypes that associate a male gender or majority-group identity with STEM disciplines can discourage girls or students from non-majority groups from considering STEM majors and careers . A lack Quantitative results suggest that we were successful in exposing girls to relatable STEM role models. Similarly, it was encouraging to see mentoring relationships mentioned many times in the girls’ open-text responses regarding the aspects of the program that they most enjoyed. In-group role models can mitigate the negative effects of STEM stereotypes  and boosSeeds of Success provides girls the opportunity to form bridges between their social and scientific identities by guiding and supporting them to develop community-focused STEM Ambassadors projects of their own design. Community-relevant projects are an opportunity to engage students who do not see science as part of their cultural or social identity, or who perceive that STEM careers do not offer an opportunity to connect with others in their community . PresentIn addition to fostering the inclusion of different STEM learners, the girls’ STEM Ambassadors projects are intended to build agency and to demonstrate to girls that they possess the creativity and internal resources to complete a project and be change-makers in their communities. Girls have continually impressed us with their creativity and ambition. Although diverse in terms of topic, target population, and mode of delivery, some trends can be observed among STEM Ambassadors projects. Many of the girls’ projects promote female STEM role models, deliver concepts or skills through workshops, promote respect for the environment, or expose students to the excitement of STEM  may be accessed via mobile devices, which are prevalent in Puerto Rico regardless of income level. In cases of technical difficulties, we approach our network of partners and funders to help provide assistance and/or equipment.All in all, results from our program evaluation lend support to online STEM outreach programs being effective in increasing science identity and improving STEM attitudes. We have come back to in-person activities for our most recent cohort (2022–2023), but we have kept several of the online activities, such as some of the STEM workshops and the group and individual STEM mentoring sessions.Here we present a program aimed at exposing Puerto Rican girls to culturally-relevant STEM female role models, mentors, and activities that inspire them to see themselves in these pathways and careers. Novel components include group and individual online mentoring sessions with Latina STEM experts and the development of identity, confidence, and leadership skills through a STEM Ambassadors program. We observe more positive attitudes towards STEM, higher science identity, and increased knowledge about STEM careers and STEM opportunities among girls who have completed the program. In future iterations, we are planning to assess causality through a matched-control design. We are also currently evaluating alumni’s long-term involvement in STEM, both in and out of school, and ultimately, their trends in college enrollment in STEM majors.The program has been highly popular in PR securing hundreds of applicants each year and assuring participation of girls from underserved schools thanks to our use of traditional and social media, our targeted branding and marketing, and on-the-ground collaborations with schools and education systems. Going forward, we are hoping to secure funding to increase the number of participants in PR, particularly from rural and mountainous regions with high child vulnerability indices . In addiAppendix. Additional Tables1