Since the 1990s, a number of multi-service prevention programs working with women who have substance use, mental health, or trauma and/or related social determinants of health issues have emerged in Canada. These programs use harm reduction approaches and provide outreach and “one-stop” health and social services on-site or through a network of services. While some of these programs have been evaluated, others have not, or their evaluations have not been published. This article presents interim qualitative findings of the Co-Creating Evidence project, a multi-year (2017–2020) national evaluation of holistic programs serving women at high risk of having an infant with prenatal alcohol exposure. The evaluation utilizes a mixed-methods design involving semi-structured interviews, questionnaires, focus groups, and client intake/outcome “snapshot” data. Findings demonstrated that the programs are reaching vulnerable pregnant/parenting women who face a host of complex circumstances including substance use, violence, child welfare involvement, and inadequate housing; moreover, it is typically the intersection of these issues that prompts women to engage with programs. Aligning with these results, key themes in what clients liked best about their program were: staff and their non-judgmental approach; peer support and sense of community; and having multiple services in one location, including help with mandated child protection. For women, problematic substance use is often interconnected to a host of issues including current and/or historic experiences of violence, intimate partner violence, trauma, inadequate or unsafe housing, homelessness, poverty, child welfare involvement, maternal–child separation, and mental health and physical health problems ,2,3,4. IFor some women who use substances, pregnancy is pivotal, a time when they are willing to contemplate making life changes. Indeed, the research literature suggests that women who are pregnant will respond to supports that are aimed at improving their health, including efforts to decrease or stop substance use or to increase their safer use of substances ,10,11. IA number of community-based prevention programs emerged in Canada in the early 1990s to address the health needs of pregnant and early parenting women with substance use issues and other complex challenges. Sheway in Vancouver and Breaking the Cycle in Toronto led the way; similar programs followed, all of them unique yet designed to address women’s holistic needs through outreach and single access drop-in services and by employing relational, trauma-informed approaches .It is worth noting that many women who use substances during pregnancy are polysubstance users . MoreoveEstablishing the effectiveness of such programs has been challenging, in part because they are community-based and/or lack resources, time, knowledge, or expertise to carry out an evaluation of their programs, or they are small programs that have not been the focus of larger studies . As suchExperience better and earlier access to prenatal careReduce their substance useImprove their health and wellness outcomes and those of their childrenRetain and/or regain custody of their childrenDevelop new social/support networks and friendshipsAcquire parenting skillsThat said, there are a few notable exceptions, namely Sheway ,19, BreaMoreover, the research and evaluation literature indicates that services that employ non-judgmental, relationship-based, trauma-informed and harm reduction approaches and that also understand and seek to remove social environmental barriers to participation, such as transportation, child care, meals, stigma, and fear of child removal, have been found to be most effective in reaching vulnerable pregnant and parenting women with substance use issues ,5,8,26. While these approaches have evidence, additional study is needed to understand their implementation in community settings, the relative importance of various program components especially in light of different community and cultural contexts, and the linkage to other levels of health care and social supports for women and children. Indeed, these were the conclusions of a recent national gap analysis related to FASD prevention, which found that evaluation of specialized, holistic FASD prevention programs was a key gap in research, and moreover, that a multi-site evaluation of FASD prevention programs would be a valuable next step . To addrWhat are common elements of these diverse, multi-service programs?How do the programs reflect their community’s context?What program components are most helpful from women’s perspectives?What are the programs’ impacts and outcomes for clients and their children?The Co-Creating Evidence: National Evaluation of Multi-Service Programs Reaching Women at Risk (CCE) project is a multi-year national evaluation of eight different holistic programs serving women at high risk of having an infant with prenatal alcohol exposure. The study is the first of its kind in Canada. The evaluation, which runs from 2017 to 2020 and is funded by the Public Health Agency of Canda, aims to: bring together many of Canada’s community-based multi-service FASD prevention programs to share promising approaches and practices; undertake a prospective, multi-site evaluation on the effectiveness of FASD prevention programming serving women with substance use and complex issues; and identify characteristics that make these programs successful. The project’s primary research questions are:The study is employing a mixed-methods design involving: two rounds of semi-structured interviews and questionnaires (taking place in April–July 2018 and September–December 2019) with clients, focus groups and interviews with programs’ staff and service partners; output/program data; and client intake/outcome “snapshot” data. Both quantitative and qualitative data are being gathered in order to obtain an in-depth description of each of the participating program’s theoretical foundation, administration, activities and services, strengths, challenges, and outcomes from the perspective of clients, staff and service partners. Seven of the eight programs specifically target at-risk pregnant or early parenting women struggling with substance use or who are early in their recovery. One program focuses on serving women who are at risk by virtue of being young and possibly social isolated; while substance use may be an issue, it is not the primary focus, although given the very limited range of services available in the region, it is an issue that comes up with increasing regularity. While these programs are doing FASD prevention work, because they approach the women’s issues holistically and employ a social determinants of health lens, and because of the stigma associated with FASD and alcohol use during pregnancy, the programs typically do not depict themselves as FASD prevention programs. This article shares preliminary findings from the study, based on the first round of data gathered between April and December 2018. To begin, the article presents the project’s Theory of Change, which was collaboratively created by the program sites and the project team as part of the developmental phase of the study. In program evaluation, the Theory of Change is a useful tool for depicting how a desired social change occurs as a result of the service/program/intervention, by showing the interconnections between a program’s philosophy/approaches; activities; and anticipated outcomes at the client/family, community, and systems levels. Then, informed by the study’s first research question, the article briefly lists the services and activities offered across the eight programs; as well, as means to increase understanding of who are the women served by these programs, the article describes women’s situations at intake, emphasizing the multiple, complex issues that women seek to address when accessing these prevention programs. Next, the article highlights how clients use the services. Lastly, guided by the study’s third research question, the article presents qualitative findings based on the client interviews regarding what women like best about their program. Interim qualitative findings in relation to the connection between what women were hoping to gain from their program and what they perceived as the most significant change in their life as a result are presented in a separate article. Findings based on the second round of on-site data collection will be presented in future publications, including a more in-depth analysis of the full 18 months of program output-based and client-based data compiled by program sites. Box 1HerWay Home (HWH) offers drop-in and outreach support, on-site wellness and prenatal/post-natal groups as well as other health/medical services for women and their children, through a combination of program staff and in-kind support from health/medical partners. Women can participate in HWH until their child is approximately three years old.Sheway (SW) offers a broad range of on-site health/medical and social services/supports through its program staff and via partnerships. Housing that is available to Sheway clients is located on the third floor of the building (operated by the YWCA through its Crabtree program). Sheway also has a health clinic. Voluntary child welfare services are provided on-site through a partnership agreement with the provincial Ministry. The length of time that women can participate in Sheway is flexible and not set by the child’s age.Maxxine Wright (MW) offers health and social supports through co-location with Atira Women’s Resource Society, which operates transition housing and second stage housing on-site. Atira offers most of the social programming, with participation from MW. Health/medical care is provided by MW. Child welfare and income assistance services are provided on-site through a partnership agreement with relevant provincial Ministries. Women can participate in MW until their child enters school.Healthy, Empowered, and Resilient (H.E.R.) is located within the Boyle Street Community Services, which provides an array of social, mental health, family, and cultural services in Edmonton’s downtown core. H.E.R. provides outreach to highly street-involved clients; through its staffing and partnership with Boyle McCauley Health Centre, H.E.R. clients have access to prenatal care and post-natal support. Women can participate in H.E.R. until six months post-partum.Raising Hope (RH) is a residence-based program that is located in an 18-unit apartment building (purchased by a non-profit housing society for the program’s exclusive use). A range of health/medical, social/cultural supports and programming, including child care, is offered on-site; residents are required to take part in daily programming. Women and their children can stay for 18 months.The Mothering Project (MP) is a program of Mt Carmel Clinic and is co-located with the Clinic. Through its staffing and partnerships, the MP offers a broad range of drop-in, outreach and on-site supports and health/medical service along with a dedicated space for cultural ceremony. A licensed day care is co-located with MP with spaces set aside for program clients. Women can participate in the MP until their child reaches the age of five.Breaking the Cycle (BTC) is one of the first FASD prevention programs in Canada. The program provides children’s developmental assessment and mental health services with wrap-around services for women. Each woman is connected with a counsellor and each child is connected with a Child Development Worker. Women can stay at BTC until their child is six years old.Baby Basics (BB) is a weekly drop-in parenting program operated by Kids First Family Resource Program, for women under age 25 and their children aged 0–6. Although not specifically directed at women who are using substances, there are very few such options available to women in the region. BB offers a safe place for women to access support and talk about a range of issues. Women can participate in BB until their infant is one year old.As a multi-site evaluation, this mixed-methods study is being guided by collaborative and participatory principles , includiThe evaluation study received ethics approval from the University of British Columbia Office of Research Ethics (H17-02168), the Fraser Health Authority, the Vancouver Costal Health Authority, and the Island Health Authority. All study participants provided informed consent to participate in the study. As well, the client and output data gathered by the program sites are fully de-identified and are being transmitted to the project team via a secure file transfer program, where the data are combined into one SPSS database for data management and analysis. Prior to the first transfer, a Data Transfer Agreement discussing the parameters governing the collection, transmission, storage, security, analysis, re-use, archiving, and destruction of data was signed by each site.In this study, data are being gathered in two different ways: (1) by the program sites, wherein quantitative output data and client-based data are being compiled and sent to the project team quarterly for a total of 18 months (April 2018 to September 2019); the Client Database contains demographic and high-level outcome indicators for each woman attending the program during the quarter, including related to housing, birth outcome, substance use, and child welfare-related outcomes; and(2) by the project team, wherein two researchers visit each program twice to conduct face-to-face semi-structured interviews, focus groups, and questionnaires, supplemented by telephone interviews when necessary. The first round of site visits and in-person data collection took place between April and July 2018; the second round will take place in the fall of 2019. Following that, quantitative and qualitative data analyses based on the full data set will be undertaken. The research team conducted individual interviews and fixed-choice questionnaires with clients in a private and comfortable office at each program site. The qualitative interviews were conducted using a “guided conversation approach” , which eThe Interview Guides were custom-created for this study. The Interview Guide for Clients contained open-ended questions focusing on: how the woman first learned about the program and what she hoped to get out of her involvement with it; her life situation at the point at which she first engaged with the program; her satisfaction with the program ; and what was most important to her about the program. The interview with clients also covered perceived impacts of the program and included a modified version of the Most Significant Change (MSC) technique , which iThe Questionnaire for Clients, utilizing a 5-point Likert-type scale, was included in the interview process. While the questionnaire was created specifically for this study, it also included standardized questionnaire items that have been used in evaluations of trauma-informed and/or harm reduction focused programs . FindingPrior to launching formal data collection, the research team pilot tested the interview guides and interview process at four sites to gain feedback from clients and staff regarding the process and the questions, including whether clients were comfortable with answering potentially difficult questions about their lives. 125 program participants/clients (of whom 123 completed the Questionnaire for Clients);61 program staff; and42 service partnersA total of 228 people took part in the first round of in-person (on-site) data collection for the study, including:For the purposes of this article, only client data are presented and discussed.n = 32 at one of the large sites to n = 8 at two of the sites; there were three sites at which more than 20 clients were interviewed and five sites at which 8–11 women were interviewed. The variation in the number of interviews conducted per site reflected the size and scale of the programs and was roughly proportional to the number of clients per site. Events outside of the program, including crises in the community and/or in clients’ lives also impacted response to the invitation to take part in the study.The number of interviews with clients varied across sites, from Eligibility criteria for client participation in the interview/questionnaire were as follows: (1) women had to be accessing services from the program in the month of data collection; (2) women had to be 16 years or older; and; (3) women had to be English-speaking. Clients were recruited and informed by program staff approximately one month in advance about the research team’s site visit and the opportunity to participate in a face-to-face interview. Posters announcing the interviews were also displayed in program space and announcements were made at meetings and groups. The recruitment material included information about confidentiality and anonymity. For clients, the study employed a voluntary sampling approach, as Program Coordinators and the research team believed it important for all clients who wanted to take part in an interview and who were available during the researchers’ site visit to have the opportunity to do so.A nominated sampling approach was used to create the sample of service partners; program staff at all sites provided the researchers with contact information for each program’s closest service partners. The researchers then conducted face-to-face or phone interviews with service partners, which lasted from 30 to 90 min. Interviews and focus groups with program staff were conducted in person and were from 1 to 2 h in duration.For the quantitative client and output data, a frequency analysis was conducted to describe the program activities and outputs, the client’s pregnancy, substance use, child welfare and housing status at intake into each program. The analysis was conducted in IBM SPSS 26 . As the interviews with clients, staff, and service partners involved open-ended questions, qualitative data analysis techniques were used, and qualitative data analysis software  was utilized to facilitate the analyses. In keeping with these techniques, written transcripts from all interviews were read multiple times by the researchers to begin the process of identifying themes and issues. Initially, each researcher coded the transcripts separately and identified preliminary themes inductively. The team highlighted naturally occurring patterns in the data, which formed the basis of the thematic analysis ,36. As mWith regard to the on-site client-related data collection , we are aware that the voluntary sampling approach could have resulted in biases in that clients with more positive views about their program would have been disproportionately inclined to take part in the evaluation study. Moreover, because there was a circumscribed number of days for each site visit and on-site data collection, we cannot be guaranteed that we achieved “saturation”, nor was the concept of saturation the means by which we determined the number of interviews to conduct at each site. At the same time, we have no reason to believe that clients who held less positive perspectives were disinclined to participate in the study, and the confidential, conversational approach to interviewing facilitated participants sharing their diverse experiences and perspectives. As well, given that there will be a second round of on-site data collection with clients, there will be an opportunity to explore the issue of sampling bias and determine whether saturation was achieved.In program evaluation, a Theory of Change is created to show the interconnections between the program’s philosophy/approaches; activities; and anticipated outcomes at the client/family, community, and systems levels. As part of the developmental phase of this project, the team and the Program Coordinators/Managers of the eight participating sites collaboratively developed a Theory of Change , and four are under the aegis of a community-based agency (CBA). As can be seen in Based on the client data submitted by program sites, in the first two quarters of the study’s data collection period (April–September 2018), a total of 712 women participated in the eight programs. In addition, based on the client data at intake provided by program staff, 57% of clients were reported as actively using substances (21–97% across the eight program sites). Overall, 41% of clients across program sites were engaging in problematic substance use, defined as the use of substances that result in negative consequences in a person’s daily life, including adverse health consequences ,39, and Substance use issues—yet readiness for changeUnsafe or inadequate housingMaternal–child separations and/or child welfare involvementViolence, trauma and/or intimate partner violenceLimited social support and/or isolationMental health issuesIn the qualitative interview, after discussing how they learned about their program and what they hoped to get from participating in it, clients were asked to describe their situation prior to becoming involved. The top themes that emerged included:Numerous women spoke of their problematic substance use and their desire to reduce or quit upon learning of their pregnancy; in addition, nearly all of these women also spoke of inadequate housing, violent relationships, mental health issues, maternal–child separations, and isolation (sometimes self-imposed in order to address their substance use issues). In these women’s words: I was pregnant. I quit using during the summer, but I was struggling with abuse, depression, anxiety and co-dependency.I was pregnant but was determined to stay sober. My first daughter was apprehended at birth and was affected by alcohol, so I wanted to do it differently this time.I was trying to quit doing drugs, hanging out with people using drugs. They were a bad influence. I was also pregnant with my sixth child; the five older children are all in care.As also indicated by I was working on the street and supporting my habit. I was homeless and couch-surfing.I was living in a two-bedroom apartment with my mom, sister, two boys, my ex and sometimes my mom’s friends. There could be 13 people at a time in the apartment.My partner didn’t respect my space—there was partying, and I was uncomfortable; the housing was unsafe. There was domestic violence.Another key theme in women’s stories was their experience of maternal–child separations, and specifically the removal of their infant or older child(ren) by child welfare authorities. For many of these women, becoming pregnant had fueled a desire for change related to their substance use and related circumstances and a hope of having their baby remain in their care. In these informants’ words:I was living in [supportive housing]; pregnant and then gave birth. I was depressed and wanted to address my drug and alcohol use. My son was apprehended at birth.I had started using again. It wasn’t a good time; after I had the baby I dropped her off with (Child and Family Services)—I felt that I wasn’t able to care for my daughter and that it was for the baby’s own safety.I was pregnant. I had just quit using drugs a month before that. I was living with my boyfriend’s parents at the time. I knew that I’d be better off here at [the program] if I wanted to get my kids back.Many clients spoke of experiencing violence, trauma or intimate partner violence, which was inextricably linked to their mental health issues and substance use and also had given rise to child safety concerns and/or their child(ren) going into care. As well, several women reported having unsafe housing, were at high risk of homelessness, or were living in transitional housing because they were in the process of leaving their former partner. My partner assaulted me when I was two months pregnant. I have two older children and am a single parent with social anxiety, mental health issues and a long history of trauma.As noted previously, many women spoke of feeling isolated, disconnected and lacking support. For some, this was because for a long time they had not had a strong social support network, having perhaps grown up in foster care or without supportive family; for others, their sense of isolation was due to mental health and/or substance use issues; for others still, they moved to a new community where they knew no one in order to flee a violent relationship or unsafe living situation; or, lastly, some women realized they needed to isolate themselves or cut their ties with unhealthy people in their lives in the interest of achieving their goals related to reduced substance use and healing.I didn’t have a lot of social support. I have no family and was raised in foster care and by my grandmother who is now deceased.I was living in an SRO, using crystal meth and heroin every day. I almost never left the SRO. I didn’t like to go out. I was very much shut off from other people.Lastly, while mental health issues were another key theme in women’s discussions of their situation prior to engaging with their program, in nearly all instances, the issues of substance use, violence, and mental health issues were fully intertwined.I was fleeing an abusive relationship. My mental health was poor. I was addicted and not taking care of myself.I was starting to fall back into drug use. I was depressed, and I was scared of my behaviour.In the qualitative interviews, clients were asked to provide examples of how they engaged with various programs activities or services. Their comments, presented in Staff and their approach—caring, non-judgmental, supportive and helpfulFriendships, social and peer support, sense of community“One-stop”—multiple services in one placeHomey, safe, healthy environmentHelp with child protection issuesPositive impacts for my child—friendships and socialization skillsAs part of the qualitative interview, clients were asked what they liked best about their program. While many informants identified more than one theme in their response to this—and all other—open-ended questions, the top themes to emerge were:A brief discussion of these findings follows.The strongest theme of what clients liked best about their program, voiced by more than one-third of interview participants, was the staff and staff’s caring, respectful, and supportive approach. Time and again, women emphasized that staff’s non-judgmental and non-stigmatizing approach was extremely important as it both promoted trust and honesty in their relationship and also led women to feel “like a person”. As a related point, several people stated that staff at their program “never gave up” on them, which they appreciated greatly as it made them feel valued and worthwhile. Further, numerous clients praised staff of their program for being very helpful and often going “above and beyond” typical service provision. In clients’ words:The people that work at [the program], they aren’t judgmental. I feel like a person at [the program], not a nobody.The staff—they give you endless support. When you tell them something, like if there’s been a slip, there’s no judgement. They’re very genuine. The right people work here.[I like] that they never gave up on me. I was stubborn and had mood swings when I was coming off drugs. I really appreciate that they never gave up on me.The second strongest theme was that women most liked the friendships they had developed with other program participants and the support they gained through their relationships with women and staff. Women appreciated making connections with others who could relate to their experiences. Along these lines, a number of clients stated that they liked the sense of community or family that they experienced at the program, and one woman expressed that she liked “being able to mentor other women”, which was facilitated by the program’s drop-in and group activities.Definitely, the friends I’ve met here. […]The moms are both in recovery and are new moms. It’s easy to relate to them and to get along; we have things in common and have the same aspirations and common goals.It feels like a family. The staff genuinely care about everyone. They always have time for you. I’ve made good friends at [the program].The sense of community. That’s the biggest thing. When we come here—because we see and hear other people—we get a more in-depth relationship with people here.Another strong theme of what clients liked best was the program offering multiple inter-connected services and resources all “in one spot”; as one client stated: “everything I need is in one place”. This meant that women did not have to travel to get to appointments, which could be a formidable barrier to accessing services. Moreover, as clients pointed out, being able to obtain health and wellness services and basic needs support through a “one-stop” program made it easier to satisfy social workers’ expectations related to child welfare and custody. Clients stated:I don’t have to go far to get to a doctor. There are all kinds of different resources here. There is a welfare worker, a housing worker, the tax lady, as well as food to eat. It’s awesome. There’s also a Walking Group and a Boxing Group.There are lot of services in one place, such as the public health nurse, the nurse practitioner, and a social worker, which makes it easier to meet the Ministry’s goals and expectations. I don’t have to travel far to get to my appointments. It’s very convenient and close to housing.In voicing appreciation for their program’s different services and resources, clients highlighted the program’s health and medical services, easy access to a social worker, as well as the practical and material support, including food, grocery vouchers, clothing, and infant supplies.I like that they are hands on with support. They have medical, counselling, and practical support; everything I need is in one place. And I like that it is a safe place and there is food.I like that you can come here and get any support that you need—for example, I can get help getting welfare or with my addictions.In keeping with the themes of caring and non-judgmental staff, and friendships and sense of community, another top theme of what women liked best about their program was that the environment felt “homey”, comfortable, and safe. Indeed, one woman spoke of feeling as though she could “relax” at the program—which was a contrast to how she felt in other situations or programs—while another deemed the program a “sanctuary” within a community that otherwise, to her, was mainly “dysfunctional”. Importantly, one woman also noted that the program was a place where she could find safety away from an abusive ex-partner. It’s a safe place for me no matter what is going on in my life—I won’t be judged by anyone. I can be myself. It is a place to connect with other women. I can be less guarded at [the program].It’s a place to eat and relax and get away from my abusive ex. He can’t bug me at [the program].The ladies—my peers—and the staff. And that it’s in the North End. There’s so much in the North End that’s dysfunctional, and the program here is the one thing that’s functional. It’s like a sanctuary.Another key theme of what clients liked most about their program was the support they had accessed in relation to child welfare issues, in particular, in getting advocacy and support during meetings with child protection workers and/or in having supervised visits with their child(ren) on-site at the program. Similarly, one client expressed great appreciation for staff’s support when her children were removed. I had a meeting with program social worker who encouraged me to meet with CFS and even inspected my house to give me suggestions for what CFS would be looking for. So, when we met with the CFS worker, I was surprised at how well the meeting went.Plus, they are willing to support parents with kids in care. My two daughters have been removed again. [The program] is a support for me, helping me to get my children back.That they are always available to help, no matter how I am feeling and no matter what is happening in my life. For example, they helped calm me down when the kids were apprehended. They gave me confidence and kept me on the right track. They gave me a lot of support. I had my supervised visits with my kids two to three times a week at [the program].A final key theme of what women like best about their program was that their child(ren) benefited and gained skills and friendships, just as they did themselves. In women’s words:Plus, my daughter gets so much out of it.The best thing about [the program] is the drop-in. You meet other women and learn from them. You make friends. Plus, children get to make friends.I like that it is geared towards children too. At first, I didn’t want my son involved because I didn’t want him around drug users but then I saw that the focus included children and that they were seen and assessed through [the program]—that [the program] was beneficial for children as well.The preliminary findings of the Co-Creating Evidence study confirm that the programs are reaching vulnerable pregnant and parenting women who are facing a host of complex circumstances including substance use. They are doing so in part by addressing the social environmental factors affecting women’s access to services  and the reasons for and correlates of their substance use, while providing low barrier access, all of which have been shown to be effective strategies in service delivery ,5,8,40. The interim findings provide additional confirmation and support previous research demonstrating that substance use, mental health, trauma, violence, child welfare involvement, and lack of adequate housing are most often co-occurring issues ,5,41,42.Along these lines, other studies have concluded that programs that adopt a trauma-informed and harm reduction perspective are more effective in reducing pervasive stigma and other barriers and engaging women in services ,8,10,26.With respect to client characteristics, there was a wide range amongst programs in relation to the percentage of clients experiencing problematic substance use at intake. At some programs, problematic substance use was a primary eligibility criterion, whereas at others, an eligibility criterion was for pregnant or parenting women to be impacted by substance use—a subtle difference. One program did not use problematic substance use as a criterion at all. The three programs with the highest percentage of clients with problematic substance use were also geographically located in areas with high levels of poverty, drug and alcohol use, homelessness, violence, and mental illness. At nearly every program site, the percentage of women with unsafe or inadequate housing at intake was approximately the same as the percentage of women with problematic substance use or who were new to recovery at intake. Perhaps not surprisingly, maternal–child separations and involvement of child welfare/protection authorities represented a strong theme in clients’ lives, lending more weight to the assertion that an integrated, multi-service model is most effective when it comes to addressing the range of unique and varied needs of pregnant and parenting women with problematic substance use ,8,10,14.Despite knowing more about what constitutes effective programming when working with pregnant and parenting women with problematic substance use, little has been written about the actual components of programs or how they are effective. A summary review of four community FASD prevention programs, prepared by Canada FASD Research Network’s Action Team on FASD Prevention from a Women’s Health Determinants Perspective noted that without practical assistance by way of nutritional and transportation supports, women cannot meet other goals such as reducing or stopping their substance use . The Co-Finding affordable, safe housing was a constant quest; the advocacy and proactive support provided by program staff, including connecting them with housing programs, helping to complete housing applications, meet with landlords, practice being interviewed for housing, and apply for income assistance and shelter allowance, helped many clients begin to find stability in their lives. Without housing, they were far less likely to satisfy child welfare authorities’ requirements for a safe environment or to address related custody or safety concerns. The handful of programs that had housing on-site , meant that clients had peace of mind in relation to two significant domains: accommodation along with proximity to a one-stop multi-service program that addressed all of their other social, medical, and health concerns. In terms of health services, in nearly all of the eight programs taking part in the current study, primary and/or prenatal care was an integral component of the program, delivered in-house or through partnership in the community. As such, clients availed themselves of a range of sources of pre-natal/post-natal health support , health services for their own issues apart from pregnancy , and health assessments for their children  as well as referrals for their children . Not surprisingly, the provision of a range of health services enabled clients to stay on top of their own health and wellness needs. For many this was a new experience, having been wary in the past of seeking health care services due to their association of prenatal care with being reported to child welfare authorities. Further, with a more positive and proactive relationship with health care practitioners, their children’s health and development issues were also potentially identified much sooner. Early identification is often key to mitigation of potential long-term negative impacts of development delays, dental deterioration, mental health, and other physical maladies. This aspect of programming—the provision of or linkages to primary health services—appears to distinguish the programs in the Co-Creating Evidence study from integrated treatment services for pregnant and parenting women, as a recent review of integrated services for similar clients in the province of Ontario found that primary and pre-natal care services were rarely named as partners, identifying a potential area for quality improvement of integrated treatment services .The other primary activities that clients substantially valued were child welfare support and advocacy, substance use groups/programming, and peer/social connections. With respect to child welfare or custody, all but one program actively worked with clients to address child safety concerns, either by having their own social worker on-site who helped walk clients through meetings with provincial child welfare authorities, or through an arrangement with provincial child and family services to provide a social worker, who helped clients navigate extant child safety expectations and concerns.Many clients expressed deep relief for the guidance they received, finding that it made it easier for them to ultimately address the safety concerns that were getting in the way of them parenting their infant or older children. Often clients were able, with the program’s support, to prove and improve their capacity as parents, thereby allowing child welfare services to close their files. With the active involvement of program staff, even clients who were unable to parent day-to-day were often able to establish or deepen a relationship with their child; programs often allowed supervised visits to occur on-site as a way of supporting the mother–child relationship. Peer relationships were also very important, as clients sought to build a different life and to develop new, healthier social connections for themselves and their children; for many women, the friendships and positive relationships formed through their program participation continued beyond the program’s hours and walls—e.g., through weekly walking groups and children’s birthday parties. Further, programs’ safe, non-judgmental environment resulted in women experiencing a positive sense of community and belonging. In terms of what makes the programs successful, from clients’ perspectives, there is no doubt that the combination of the right kind of staff, the right kind of approach, and multiple services in one location were significant factors. This too fits with the literature, which suggests that integrated, wraparound services that address the social determinants of health, offer a spectrum of services that meet a woman where she is at, and demonstrate a commitment to her and her child while adhering to obligations to safety, are practices that make a difference . These programs reaching vulnerable pregnant and parenting women who face problems with alcohol and other health and social challenges are a key component of a comprehensive FASD prevention response. Evaluation research that identifies the range of services required and found helpful by women and service providers is critical to advancing the work of FASD prevention, and to health and social care system planning overall.


Sulfur isotope fractionation resulting from microbial sulfate reduction (MSR) provides some of the earliest evidence of life, and secular variations in fractionation values reflect changes in biogeochemical cycles. Here we determine the sulfur isotope effect of the enzyme adenosine phosphosulfate reductase (Apr), which is present in all known organisms conducting MSR and catalyzes the first reductive step in the pathway and reinterpret the sedimentary sulfur isotope record over geological time. Small fractionations may be attributed to low sulfate concentrations and/or high respiration rates, whereas fractionations greater than that of Apr require a low chemical potential at that metabolic step. Since Archean sediments lack fractionation exceeding the Apr value of 20‰, they are indicative of sulfate reducers having had access to ample electron donors to drive their metabolisms. Large fractionations in post-Archean sediments are congruent with a decline of favorable electron donors as aerobic and other high potential metabolic competitors evolved. Changes in S-isotope ratios over time provide clues to understanding the co-evolution of Earth and its biosphere. Here the authors determine the isotope effect of the first reductive enzyme in the sulfate respiration pathway and reinterpret sedimentary S-isotope records based on this biochemical constraint. The sulfide produced by microbial sulfate reduction (MSR) readily reacts with iron to form iron sulfide and in turn pyrite, a redox sink for sulfur and iron4. Thereby, this metabolism provides a linkage between the global sulfur cycle with those of carbon, oxygen, and iron. Given the presence of four stable sulfur isotopes , MSR discriminates against heavy isotopes, resulting in isotope signatures which can be recognized in modern aquatic environments and are also preserved in sedimentary sulfate and sulfide phases over geologic time5. Sulfur isotope fractionation has thus been used both as a diagnostic marker for sulfate reduction and as a tool to reconstruct the early evolution of sulfur metabolisms and ocean redox chemistry10. In particular, small 34S/32S fractionations between sulfate and sulfide before the great oxidation event (GOE) have been considered to reflect low-sulfate concentrations in the Archean oceans12. Such interpretations are grounded in the results of extensive culture experiments over the last half-century, which have revealed a set of relationships between microbial respiration rates, sulfate concentrations, and the kinetic and equilibrium isotope effects controlling net fractionation24. A complete mechanistic understanding of the variables that control sulfur isotope fractionation has yet to be achieved, however, in part due to a lack of information on the individual intracellular processes which cumulatively give rise to net sulfur isotope fractionation between sulfate and sulfide.Sulfate is quantitatively the most abundant water-soluble electron acceptor for biological respiration on the planet26, and fluxes through potential branch-points27. The enzymatic isotope effect is typically a reflection of the transition state formed during the reaction, and flux through a pathway and its branch-points is a complex function of physiological state and environmental conditions. To date, a few experimental constraints on metabolic sulfur fluxes have been imposed either by using the oxygen isotope exchange between sulfite and water24 or by measuring the sulfur isotope compositions of metabolic intermediates in the MSR pathway28. Those studies illustrated the principles underlying physiological variations in sulfur isotope fractionation, but did not give a quantitative measure of the degree of fractionation: it is the enzyme-specific isotope effects which represent the end-members of the kinetic isotope fractionation range. Knowledge of these values will allow for a quantitative interpretation of sulfur isotope fractionation arising from sulfate reduction and a linkage of isotope fractionation to the physiological state of the cell.Kinetic isotope fractionation between sulfate and sulfide is, in theory, dependent on the isotope effects of the enzymes involved, the relative rates of each enzymatic reaction26. Recently, Leavitt et al.29 conducted the first enzyme-specific sulfur isotope fractionation measurements, using DsrAB, a partial enzyme complex of dissimilatory sulfite reductase, and found the produced thiosulfate and trithionate to be depleted in 34S by 15‰ at the reduced sulfur S0 position of those compounds relative to the reactant sulfite. Although this reaction system lacked the DsrC subunit required to achieve full conversion of sulfite to sulfide30, the measured fractionation was much smaller than the previous theoretical value of ca. 50‰26, revealing possible discrepancies between experimental and indirect fractionation estimates and further motivating the need for the experimental assessment of enzymatic isotope effects. Remaining to be determined is the sulfur isotope fractionation by Apr, the first reductive enzyme in the MSR pathway, as well as for the full DsrABC complex. Fractionation by the Apr enzyme is predicted to exert greater control on the overall fractionation of the whole pathway as the Apr step becomes rate limiting at fast respiration rates31. So far, a normal isotope effect of 25‰ has been inferred for Apr26, but this value has never been measured.MSR to sulfide is carried out by three different enzymes: sulfate adenylyltransferase , adenosine phosphosulfate (APS) reductase Apr; EC 1.8.99.2), and dissimilatory sulfite reductase   were calculated to be 20.3 ± 0.5‰ and 20.1 ± 0.8‰ at 32 and 20 °C, respectively. This is the first experimental constraint on sulfur isotope fractionation by Apr, and is about 5‰ smaller than the previously inferred value of 25‰26 that was based on the equilibrium sulfur isotope fractionation between sulfate and sulfite at a physiologically relevant temperature26. Because the equilibrium isotope effect for reversible reactions equals the difference between kinetic isotope effects for forward and backward reactions33, the assumption of a 25‰ fractionation value is equivalent to assuming no kinetic fractionation in the backwards reaction. Our new measured value of 20‰ for the forward (reductive) direction suggests that Apr likely exhibits a small, inverse isotope effect when operating in the reverse (oxidative) direction. We note that an enzymatic preference toward heavy isotopes, while rare, has been previously reported in other reactions, e.g., for nitrite oxidation36. °C Fig. . As the 34S Fig. , consistres Fig. , indicatres Fig. . From th37. While many enzymatic reactions display weak temperature dependence of the kinetic isotope effect at the actual catalytic step, the steps in the reaction which precede or follow the catalytic step might contribute to the temperature dependence of the observed isotope effect40. The similarity of the measured sulfur isotope effects at 20 and 32 °C despite the nearly quintupled rate therefore indicates that the catalytic bond-breaking step is nearly rate-limiting for APS reduction, rather than substrate binding or product release. In contrast, fractionation of sulfur isotopes in whole cell studies has been found to change with temperature41 and also to be inversely correlated to the cell-specific sulfate reduction rate (csSRR)18. These differences between enzyme- and cellular-level fractionations suggest that the relationships between physiology and sulfur isotope fractionation are not rooted in the kinetics of an individual enzyme, but are instead expressed by the overall sulfate reduction pathway. We explore this next with the aid of a recently established model framework31 and our quantitative constraint on the sulfur isotope effect of Apr.The almost identical fractionations at two different temperatures may have implications for whole cell-derived isotope fractionation, as well as for the mechanism of enzymatic catalysis. APS reduction is thought to be a multistep reaction, which begins with nucleophilic attack on the sulfur of APS by the flavin cofactor FAD, followed by loss of AMP during collapse of the covalent APS-flavin adduct31. The less reversible the reaction, the less isotope effects of the subsequent reactions will be expressed. Reversibility can be recognized as a measure of the relative degree of rate limitation at different steps in a pathway, which is ultimately set by the thermodynamic conditions of the reaction43:b and f denote backward and forward rates, R is the gas constant, T the temperature, and ΔG is the free energy change associated with the reaction. As the value of ΔG strongly depends on the concentrations of involved reactants and products, Wing and Halevy31 proposed a model with intracellular metabolite levels as the central determinant of reversibility and thus sulfur isotope fractionation. We updated their model to include the newly determined value for 34εApr and used it to explore varying redox potentials of the electron donating half-reaction at constant sulfide concentration  greater than −5 kJ mol−1, the resulting reversibility is as small as 0.1 . Although such a small difference between the redox potentials of electron carriers and APS  can contribute to maintaining reversibility in APS reduction and thereby produce larger isotope fractionation, such low energy yields make it difficult to account for the energy needed to generate a proton gradient that is presumed to be coupled to the oxidation of membrane-bound electron carriers56. However, if electron flow from physiological carriers to APS or sulfite were to be directly coupled to proton motive force generation, the reaction energies of these processes would be added to the expected energies of the APS or sulfite reduction. Because ion translocation to the outside of a chemiosmotically charged membrane is endergonic, the reversibility of a directly coupled APS or sulfite reduction step would increase as a result of the additional energy requirement , it may0.1  would likely account for a substantial fraction of pyrite formation, like those in some modern euxinic basins63. More importantly, neither scenario explains why the maximum sulfur isotope fractionation during the Archean would be so similar to the sulfur isotope effect of APS reductase without exceeding it.In contrast to modern environments, sulfur isotope fractionations between sulfate and sulfide rarely exceed 20‰ in rocks older than 2.5 Ga Fig. . Such sm2, play a role in controlling the expression of sulfur isotope fractionation. Previous culture studies have demonstrated qualitatively that decreasing availability and reactivity of electron donors leads to larger sulfur isotope fractionation19, and our experimental determination of the Apr kinetic isotope effect provides quantitative support for these qualitative arguments. Based on the results above, we consider the possibility that the availability of more favorable electron donors was responsible for reduced sulfur isotope fractionation during the Archean. In an anoxic ocean harboring other microorganisms competing for reductants, using sulfate as an electron acceptor would have been a major evolutionary innovation. Sulfate-reducing microbes would likely have outcompeted ancient methanogens and acetogens for common electron donors such as H2, because sulfate reduction provides a greater free energy yield than CO2 reduction66. Considering H2 as an electron donor, at 100 µM concentration of sulfate and 10 nM H2, hydrogen oxidation coupled to sulfate reduction still yields −43.5 kJ mol−1, while methanogenesis provides only −13.7 kJ mol−1; acetogenesis is unfavorable at +20.4 kJ mol−1. In addition to the thermodynamic favorability of sulfate reduction when considering common electron donors, sulfate-reducing microorganisms would be able to utilize higher potential (lower energy yielding) reductants for respiration, again by virtue of delivering electrons onto sulfate, although these low-energy electron donors would likely have been utilized only if high-energy ones were substantially consumed. Competition with iron reducing microorganisms could also be considered. However, limited access to iron-bearing detrital phases yields lower rates of iron reduction in the water column67, and sulfate reduction often precedes iron reduction even in the low-sulfate freshwater sediments68. The absence of sulfur isotope fractionations greater than 20‰ is thus consistent with the hypothesis that privileged access to electron donors would be able to drive the Apr reaction irreversibly. A similar scenario could explain, at least in part, why sulfur isotope fractionation larger than 20‰ became widespread after 2.5 Ga, when the GOE, triggered by the rise of oxygenic photosynthesis, made O2 available as an electron acceptor69. With increasing oxygen concentrations, aerobic microorganisms could outcompete sulfate reducers for access to electron donors, relegating sulfate reducers to lower net energy environments. Adaptation to this thermodynamically less favorable lifestyle may have made the MSR reaction more reversible, leading to larger fractionation.Not only sulfate concentration but also electron donors, such as organic matter or H2, also seems plausible in accounting for the rise in sulfur isotope fractionation values after the GOE. Regardless though, it is now clear that the electron donor is a contributing determinant to isotope fractionation during MSR, with the Apr-catalyzed reaction exerting significant control on the observed fractionation. Consequently, differences in δ34S values of sulfate and pyrite cannot be used as a quantitative constraint on sulfate concentration, e.g., in the Archean oceans12, without additional information about the reductant pool.This hypothesis is yet speculative, and is not exclusive of other effects such as those from low-sulfate concentration. Indeed, simultaneous loss of electron donors and increasing sulfate concentration, both as a result of increasing atmospheric ODesulfovibrio vulgaris Miyazaki as previously described70, flash frozen in 75 mM Tris-HCl buffer (pH 7.5), and stored at −80 °C for later assay. All chemicals, except for APS, were used as received from the manufacturer without further purification. The APS sodium salt was obtained from Sigma-Aldrich (CAS No. 102029-95-8), and given the relatively low-purity grade (85%) it often contained >10% sulfate (mol/mol) as impurity. The stock solution of APS (~40 mM) was prepared using degassed deionized water, split into a series of 250 μl aliquots, and stored at −80 °C until needed. Upon thawing, 50 μl of 0.2 M BaCl2 solution was added to each aliquot, and precipitated BaSO4 was removed by centrifugation. Excess Ba2+ was eliminated from solution by adding 100 μl of 0.1 M Na3PO4 solution, and again, the insoluble Ba2(PO4)3 was removed by centrifugation. After dilution, the final APS concentration in the working stock solution was set to 7.5 mM. A few tens of mM phosphate would be present in this working stock, which was not quantitatively significant because more than ten times larger amount of phosphate was added to the reaction mixture to buffer the pH  equipped with an AS11-HC column , using a gradient elution with KOH as mobile phase28. The concentrations determined by IC are subject to an error of ±5%. For isotope analysis, the eluant fractions corresponding to sulfate and APS were collected, and APS was quantitatively converted to sulfate via hydrolysis at 65 °C for 12 h.APS reductase enzyme was purified from 74. Samples were introduced to plasma via an ESI PFA-50 nebulizer and Cetac Aridus II desolvator. Sulfur isotope ratios of the sample and working standard were measured in 50 alternating cycles, and instrumental blank was estimated after each sample block. The mean blank value was subtracted from the measured signal for each mass. The measured 34S/32S ratio was calibrated using a linear interpolation between the two bracketing standard values. Sulfur isotope ratios are reported here using the conventional delta notation:34Rsample and 34RVCDT are the 34S/32S ratios of sample and Vienna-Cañon Diablo Troilite (VCDT), respectively. Our working standard was calibrated against the IAEA S-1 reference material (δ34SVCDT = −0.3‰) and has a δ34S value of −1.55‰ ± 0.16 (2σ) on the VCDT scale. Analytic reproducibility for δ34S has been previously evaluated as ±0.2‰ (2σ)74. Measured concentrations and isotopic compositions of APS and sulfate were summarized in Supplementary Table δ34S(x) is the sulfur isotopic composition of the total, APS, or sulfate sulfur pools, and [APS] and [SO4] are their molar concentrations. Calculated isotopic compositions of total sulfur varied within 10% of that of the beginning, which is a threshold value commonly used in isotope studies75. Closure of isotope mass balance indicates that assumption of a closed system is valid, and no unknown sulfur pool was present in our experiments. As shown in Supplementary Table δ34S value of accumulated sulfate from that of sulfite produced during the assay. Given that no additional sulfate evolved in the control experiments, the contribution from an initial sulfate pool was removed via isotope mass balance:δ34S(SO3) is the isotopic composition of produced sulfite, δ34S(SO4) and δ34S(SO4)0 are sulfur isotope compositions of the accumulated and initial sulfate, and [SO4] and [SO4]0 are sulfate concentrations at the time of sampling and at time zero. Errors from isotope analysis via ICP-MS (0.2‰) and concentration measurement via IC (±5%) were propagated via first-order Taylor series expansion, resulting in an uncertainty estimate for δ34S(SO3) values of better than 0.5‰. Sulfur isotope fractionation factor (34εApr) was calculated using an approximate solution to the Rayleigh distillation equations76:f is the fraction of the remaining APS, δ34S(APS)0 and δ34S(APS) are sulfur isotope compositions of the initial and remaining APS, respectively, and δ34S(SO3) is isotopic composition of produced sulfite. Using linear regression analysis, values of 34εApr were obtained from the slope of -lnf versus δ34S(APS) and (f·lnf)/(1 − f) versus δ34S(SO3). All analytical errors were propagated via Monte Carlo simulation (n = 5000).The samples containing dissolved sulfate were dried on a hot plate and diluted in 5% nitric acid to a sulfate concentration of 20 µM to match the in-house working standard. Isotopic analysis was conducted on a Thermo Fisher Scientific Neptune Plus multi-collector inductively coupled plasma mass spectrometer (MC-ICP-MS), operated in medium resolution following the method previously described31, we ran the model with a constant sulfide concentration but varied redox potential for the electron carriers coupled to the MSR pathway  in the Archean rocks. Since the proliferation of Archean 33S studies has also yielded a record of the ratio between the more abundant isotopes, 34S and 32S, we supplement the existing compilation with the Archean δ34S data collected during the last decade (Supplementary Data Sulfur isotope data from modern environments, shown as a histogram in Fig. Supplementary InformationDescription of Additional Supplementary FilesSupplementary Data 1Supplementary Data 2


Even in anonymous evaluations of a postgraduate medical education (PGME) program, residents may be reluctant to provide an honest evaluation of their PGME program, because they fear embarrassment or repercussions from their supervisors if their anonymity as a respondent is endangered. This study was set up to test the hypothesis that current residents in a PGME program provide more positive evaluations of their PGME program than residents having completed it. We therefore compared PGME learning environment evaluations of current residents in the program to leaving residents having completed it.This observational study used data gathered routinely in the quality cycle of PGME programs at two Dutch teaching hospitals to test our hypothesis. At both hospitals, all current PGME residents are requested to complete the Scan of Postgraduate Education Environment Domains (SPEED) annually. Residents leaving the hospital after completion of the PGME program are also asked to complete the SPEED after an exit interview with the hospital’s independent residency coordinator. All SPEED evaluations are collected and analysed anonymously. We compared the residents’ grades  to 10 (excellent)) on the three SPEED domains  and their mean  between current and leaving residents.p = 0.53, 95% CI for difference − 0.16 to 0.31) nor the department SPEED domain grades  = 0.79, p = 0.51) were significantly different between current and leaving residents.Mean (SD) overall SPEED department grades were 8.00 (0.52) for 287 current residents in 39 PGME programs and 8.07 (0.48) for 170 leaving residents in 39 programs. Neither the overall SPEED department grades (t test, Residents leaving the program did not provide more critical evaluations of their PGME learning environment than current residents in the program. This suggests that current residents’ evaluations of their postgraduate learning environment were not affected by social desirability bias or fear of repercussions from faculty. In postgraduate medical education (PGME), a department’s learning environment is considered to be vital for high-quality postgraduate medical education , 2. A heThe learning environment has been described as the formal and informal context in which learning takes place , comprisResidents generally feel capable of assessing the quality of their PGME program and providing feedback on it to their supervisors . AnonymiIn the Netherlands, PGME programs consist of 4–6 years of workplace learning in teaching hospitals, partly in a general teaching hospital and partly in a university hospital. Competition for enrolment in nationally recognized PGME programs is fierce, which is why the majority of freshly graduated doctors choose to obtain clinical experience for a few years as a junior doctor before applying for a residency position. As a result, almost all PGME departments in Dutch teaching hospitals employ both junior doctors not enrolled in formal PGME training, and residents in the nationally recognized PGME program of that discipline. Both junior doctors and residents are licensed physicians and are involved in patient care, with residents acting increasingly independently with increasing experience and competence throughout the PGME program. Although junior doctors are not formally enrolled in PGME programs, they participate in the department’s educational activities for residents and share clinical duties and on-call shifts with residents.We conducted a cross-sectional observational study of residents and junior doctors in two hospitals in the Netherlands: Isala Hospital in Zwolle (1100 beds) and the Medical Center in Leeuwarden . Isala and MCL employ approximately 120 and 95 residents in formal PGME programs and 100 and 65 junior doctors, respectively. Both hospitals are certified by the Royal Dutch Medical Association as licensed general teaching hospitals in 28 and 23 PGME programs, respectively. Because each PGME program has its own design and timetable, the population of residents in Isala and MCL changes almost every month, with residents moving in and out of PGME programs. Residents spend between 6 and 48 months of their PGME training at the hospital, depending on the program they are enrolled in.As prescribed by the Royal Dutch College of Medicine , both IsEach resident or junior doctor leaving the hospital after completion of the PGME program (resident) or expiration of their contract (junior doctor) is invited for an exit interview, collecting data on the resident’s or junior doctor’s experience in working at the hospital in a semi-structured fashion. The aggregated results of these exit interviews are fed back to faculty, again without disclosing individual respondents’ responses or characteristics. These exit interviews are being conducted by the hospital’s junior staff coordinators, who are the primary contact persons for residents and junior doctors throughout their career at the hospital, and who are independent from the hospital’s faculty providing the PGME programs. These junior staff coordinators are highly valued by residents and junior doctors as their advocates and confidants, and serve the recommended role as an independent “honest broker” to collect anonymous data on PGME program quality . As partResidents and junior doctors currently working at the hospital Residents and junior doctors participating in an exit interview as outlined above We used the SPEED results that were collected routinely as part of the PGME quality cycle in the two hospitals, in two groups of residents and junior doctors:Between January and December 2017, all 220 current residents at Isala were invited to complete the web-based SPEED survey. At the MCL, all 160 current residents were asked to complete the web-based SPEED survey between October 2017 and October 2018.Throughout 2017, exit interviews were conducted with all 95 leaving residents at Isala and with all 75 residents leaving MCL in 2018.general domain grade for each domain on a scale from 1 (poor) to 10 (excellent) [The SPEED comprises 15 items in three domains , scored on a 5-point Likert scale ranging from one (strongly disagree) to five (strongly agree), and a cellent) . This waThe full version of the SPEED is available in its original open access publication .department SPEED domain grades: we calculated three department SPEED domain grades by averaging per department the general domain grades given by respondents for content, atmosphere, and organization of the program;overall department SPEED grade: for each respondent, we calculated the mean SPEED grade by averaging the three general domain grades; subsequently, we calculated the overall department SPEED grade by averaging the mean SPEED grades per department.Because of the anonymity of the data, we were not able to link individual scores of current residents to those of leaving residents. Our analyses were based on the following variables for each department:p > 0.1) we used parametric tests (MANOVA and Student’s t tests) to analyse the data.Because the distributions of the department SPEED domain grades and the overall department SPEED grades were not significantly different from normal distributions  to assess differences in our primary outcome parameter, i.e. the department SPEED domain grades, between current and leaving residents. We also examined differences in the primary outcome parameters between the two study sites, to explore potential systematic differences in perceived learning environment quality between hospitals. We used Student’s > 5 residents) and small departments (< 5 residents), and by calculating the correlation coefficient between the number of residents in a department and the difference in department SPEED grades between current and leaving residents.As secondary outcome parameters, we analysed whether the differences in SPEED domain grades and overall department SPEED grades between current and leaving residents were related to the number of residents in a department, by comparing it between large departments .p = 0.30) or the overall department SPEED grades .Completed SPEED questionnaires were obtained from 193 current residents in the program at 21 departments at Isala (response rate 88%) and from 96 current residents in 18 programs at MCL (response rate 60%). Exit interviews including SPEED domain grades were completed by 95 leaving residents from 21 departments at Isala and by 75 leaving residents from 18 departments at MCL . There were no significant differences between hospitals in the department SPEED domain grades  = 1.25, p = 0.51) or the overall department SPEED grades between current  and leaving residents .Department SPEED domain grades for the content, atmosphere and organization of the PGME program were comparable between current and leaving residents Fig. . There wp = 0.026), with leaving residents in larger departments providing lower grades than current residents. In large departments, the difference in overall SPEED department grades between current and leaving residents was slightly larger than in small departments (95% CI for difference 0.03–0.63, p = 0.03, see Table There was a trend towards higher department SPEED domain grades in residents from smaller teaching departments than in those from larger teaching departments, but these differences only reached marginally statistical significance for the organization domain scores of current residents Table . There wIn this study of residents from two general teaching hospitals in the Netherlands, we found no statistically significant differences in overall department SPEEED grades or department SPEED domain grades between residents leaving the PGME program and residents currently enrolled in the program Fig. . This stWe considered several potential explanations for this finding. First, and most likely, the current residents in both teaching hospitals may have felt safe enough to provide the organization with honest feedback on their PGME learning environment. Second, differences between the cohorts of current and leaving residents may have distorted the outcomes. However, although individual experiences of a PGME program likely differ between residents, there are no clear reasons to expect systematic differences in experiences of the PGME program between current and leaving residents. They followed the same program, with the same supervisors, performed the same clinical work and followed the same formal education sessions outside the clinical workplace. In addition, the cohorts of current and leaving residents overlapped in part. During the study, there were no interventions targeted at improving or changing the learning environment in the two hospitals that may have affected our findings. Moreover, research on data of 7 cohorts of medical students showed that differences between cohorts explained only 0.01% of the variance in multiple choice examination results, compared to 83% for the differences between subjects within cohorts, and 12% for random error . SimilarThird, leaving residents’ SPEED scores could have been affected by social desirability bias, if these residents desired to stay at or return to the same department later in their career. However, it is unlikely that this would apply to all leaving residents. In addition, even if residents who wish to stay provided higher SPEED scores, it is unknown whether this reflects social desirability bias or true satisfaction with the program. Fourth, considering that leaving residents would benefit less from any improvements to the PGME program based on their critical feedback, leaving residents may be subject to the so-called “peak end” effect, i.e. the tendency of people to evaluate experiences based on the best or worst components at the end of the experience rather than comprehensively . We had The study was sufficiently powered to detect a relevant difference in learning environment scores between current and leaving residents, making it unlikely that a larger study would have shown significant differences in SPEED grades between these two groups of residents.To our knowledge, our study is the first to compare evaluations of PGME programs between current and leaving residents. Our findings argue against bias in current residents’ evaluation of the quality of their PGME learning environment. The trend towards higher SPEED grades in small departments, and towards larger differences in SPEED domain grades between current and leaving residents in larger departments was completely explained by the organisation domain, and was in the opposite direction than expected if current residents were concerned of identity disclosure with the associated fear of embarrassment or repercussions from their supervisors , 20. ThiThe strengths of this study include the use of a validated concise tool with a sound theoretical basis to assess the learning environment , 9, the We found comparable evaluations of the PGME learning environment between residents having completed the program and residents in the program. We argued that there was no effect of social desirability bias on these evaluations, and that the outcomes of these evaluations by residents currently enrolled in the program seem trustworthy.


Some people with rare diseases rely on peer-led support groups for disease-specific education and emotional and practical support. Systemic sclerosis (SSc), or scleroderma, is a rare autoimmune connective tissue disease. Many people with SSc cannot access support groups, and, when support groups exist, they may not be sustained due to challenges that could be addressed via leader training. The Scleroderma Patient-centered Intervention Network (SPIN), along with SSc patient organization partners, developed a training program for SSc patient support group leaders, the Scleroderma Support group Leader EDucation (SPIN-SSLED) Program. We recently completed a feasibility trial in which we successfully delivered the program to two groups of support group leaders who reported a high level of satisfaction with the program and its delivery. The primary objective of the full-scale SPIN-SSLED trial is to evaluate the effect of the program on support group leaders’ self-efficacy for carrying out their leadership role. Secondary objectives include evaluating effects on leader burnout, leader satisfaction (participation efficacy), and emotional distress.The SPIN-SSLED trial is a pragmatic randomized controlled trial (RCT) in which 180 support group leaders will be randomly allocated to training groups of 6 participants each or to a waitlist control. We will use a partially nested RCT design to reflect dependence between individuals in training groups, but not in the waitlist control. Participants allocated to the training program will receive the 13-module SPIN-SSLED Program, delivered via webinar over the course of 3 months in weekly 60–90-min sessions. The primary outcome is leader self-efficacy, measured by the Scleroderma Support Group Leader Self-efficacy Scale post-intervention. Secondary outcomes are leader self-efficacy at 3 months post-intervention, and leader burnout, volunteer job satisfaction (participation efficacy), and emotional distress post-intervention and at 3 months post-intervention.The SPIN-SSLED trial will test whether a training program for SSc patient support group leaders increases the self-efficacy of group leaders to carry out leadership tasks. The program has the potential to significantly improve the effectiveness and sustainability of existing SSc support groups, to increase the number of available support groups, and to be adapted for other chronic diseases.NCT03965780. Registered on 29 May 2019.ClinicalTrials.gov,  People with rare diseases face the same challenges as those with more common diseases plus other unique challenges, including limited education on disease and a lack of specialized support options –5. ProfePeer support interventions, including support groups, have been found to increase positive health behaviors, self-efficacy for disease management, and mental health –17. DisePeer-led support groups play an important role for many people with systemic sclerosis (SSc), or scleroderma. SSc is a rare, chronic, autoimmune disease characterized by vasculopathy and excessive collagen production , 21. OnsCurrently, there are over 250 leaders and co-leaders affiliated with Scleroderma Canada and Canadian provincial organizations, the Scleroderma Foundation of the United States, Scleroderma & Raynaud’s UK, Scleroderma Australia and Australian state organization, and Scleroderma New Zealand; almost all of these leaders are people with SSc –32. ManyResearch in SSc and other diseases, including cancer, has established that leading a support group poses significant challenges and a high level of burden for patient leaders, often resulting in burnout , 33, 34.The Scleroderma Patient-centered Intervention Network (SPIN) partnered with SSc patient organization leaders and with a Support Group Leader Advisory Team that was formed by SPIN to develop the Scleroderma Support group Leader EDucation (SPIN-SSLED) Program. The program is a 3-month group videoconference training program, designed to improve skills and self-efficacy, reduce burden, and reduce emotional distress among support group leaders. By providing key knowledge and skills, the SPIN-SSLED Program may improve the ability of SSc peer support group leaders to lead sustainable, effective support groups; reduce burden on leaders; and encourage new leaders to set up support groups where none exist, locally or via the Internet.The program is designed to be delivered by videoconference because in rare diseases, including SSc, support group leaders are widely dispersed geographically. Videoconferencing has been successfully used to train educators, therapists, and other health service providers –40. SystN = 29; website, discussion forum, 2-day face-to-face training) or low-resource  interventions. The RCT did not find evidence that the high-resource program was more effective. However, the trial was substantially underpowered, not enough information was provided to determine intervention content or how it was delivered, and risk of bias was high due to methodological limitations. A recent update of the systematic review did not identify any additional trials [We previously conducted a systematic review of trials that have evaluated the effects of training programs for patient leaders of illness-based support groups on the competency, self-efficacy, burden, and emotional well-being of group leaders . Only onl trials .We recently conducted a feasibility trial of the SPIN-SSLED Program  that involved delivery of the SPIN-SSLED Program to two training groups of five participants each . SclerodIn the feasibility trial, the pre-intervention mean (standard deviation (SD)) total score on a measure of support group leader self-efficacy, the Scleroderma Support Group Leader Self-efficacy Scale (SSGLSS) , the priThe planned full-scale SPIN-SSLED trial will be a parallel-group, partially nested RCT (PN-RCT) with a 1:1 allocation ratio that will test the superiority of the SPIN-SSLED Program to a waitlist control group for improving support group leaders’ self-efficacy (defined as their perceived ability to carry out actions needed to be successful in support group leadership)  and imprThe primary objective of the SPIN-SSLED trial is to evaluate the effect of the SPIN-SSLED Program on support group leaders’ self-efficacy, measured by the SSGLSS  post-intThe planned trial has been registered , and the present protocol follows recommendations for reporting from the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 statement . ResultsThe trial will be a pragmatic RCT that tests whether the SPIN-SSLED Program improves support group leader outcomes compared to leaders assigned to a waitlist control. Pragmatic RCTs are intended to replicate real-world conditions and support a decision on whether an intervention should be provided , 59, 60.Support group leaders randomly assigned to the SPIN-SSLED Program will be clustered into training groups. Members of each training group will interact during videoconference training modules. Support group leaders randomly assigned to the waitlist control will not be clustered; they will only complete trial measures. A standard cluster RCT design is used when interventions are delivered to groups, rather than individuals, in order to account for dependence between individuals within clusters . The SPITrial participants will include current support group leaders and candidate leaders who are affiliated with Scleroderma Canada and Canadian provincial organizations, including Sclérodermie Québec; the Scleroderma Foundation in the USA; Scleroderma & Raynaud’s UK; Scleroderma Australia and Australian state organizations; and Scleroderma New Zealand. To be eligible, support group leaders must be identified by one of our partner organizations as a current or candidate leader, must be able to use the Internet to access training sessions, must indicate that they would be comfortable with participating in sessions offered in English or French, and must be able to complete study questionnaires online in English or French. In addition to these requirements, we will only enroll one support group leader per support group in order to avoid contamination, whereby leaders in the waitlist arm could receive training materials from their support group co-leader. In the case where there are multiple leaders for a single existing support group, the co-leaders must come to a decision together on who they would like to be the primary leader and prioritized for enrollment. The leader(s) designated as secondary will only be enrolled if the primary leader in the group must drop out prior to random selection and allocation or does not have any day and time availabilities that match those offered as part of the trial and, thus, will not be eligible for random selection and allocation. Leaders who are designated as secondary and who do not undergo training will be placed in the waitlist to undergo training post-trial. All participants were free to access any care resources or other interventions made available to them throughout the course of the trial.The SPIN-SSLED Program uses problem-based learning, which is a learner-centered approach that integrates theory and practice by providing necessary knowledge and skills, presenting complex, real-world problems, and working to identify approaches to solving problems , 64. EacThe SPIN-SSLED Program will be offered in English or in French; the English version of the Program was translated into French by a research assistant and then reviewed for consistency by bilingual research team members. The program includes 13 modules that are delivered via videoconference over the course of the 3-month program in weekly 60–90-min sessions. Module topics include (1) The leader’s role; (2) Starting a support group; (3) Structuring a support group meeting; (4) Scleroderma 101; (5) Successful support group culture; (6) Managing support group dynamics; (7) Loss and grief: The support group leader; (8) Loss and grief: supporting group members; (9) Advertising and recruitment for the support group; (10) The continuity of the group; (11) Supporting yourself as a leader; (12) Remote support groups; and (13) Transitions in support groups. See Additional file All English-language SPIN-SSLED training groups will be facilitated by a single instructor who assisted in the development of the SPIN-SSLED training program and delivered the program during the feasibiilty trial. She is a trained social worker with 28 years of total experience and over 10 years of experience working in SSc. The French-language training groups will be facilitated by a single instructor who is a recent graduate of the SPIN-SSLED program and has led a support group in Quebec for the last 5 years. Instructors will guide each session using the SPIN-SSLED instructor manual, which is based on the program manual but includes guidance on introducing material and discussion prompts. Participants will receive a SPIN-SSLED program manual that summarizes didactic material from sessions. Based on our previous experience and consistent with previous trials of videoconference training, six support group leaders will be assigned to each training group to maximize effective interaction and participation , 37. TraAll SPIN-SSLED sessions will be video-recorded and audited for fidelity to the program manual by two members of the research team. We will use standard methods for evaluating intervention fidelity , includiParticipants may choose to discontinue their participation in the training sessions at any time. We do not envision, however, the need to modify the intervention or intervention assignment for any participants or to discontinue their participation in the program.N = 102, N = 55). We found that it had good internal consistency  and hypothesis-consistent convergent validity with a burnout measure, the OLBI [The primary outcome analysis will compare SSGLSS  scores bthe OLBI . In our the OLBI , SSGLSS N = 327) and was found to be reliable and constructually valid [Secondary outcomes include the SSGLSS 3 months post-intervention and other outcome measures post-intervention and 3 months post-intervention. Leader burnout will be measured by the OLBI, which assesses exhaustion and disengagement due to burnout and has been validated in diverse populations  , 49. Thely valid . As in oly valid , 71, we ly valid , 52. PHQly valid , which ily valid , a standly valid  and is aWe identified several meta-analyses that have evaluated self-efficacy in terms of knowledge acquisition and confidence in implementing skills acquired in training programs. A 2016 Cochrane review reported a standardized mean difference effect size of 0.87 for four educational interventions designed to change knowledge of sickle cell disease among patients and caregivers (standardized mean difference = 1.12 with an outlier study removed) . SeveralFor the secondary outcomes, burnout and emotional distress, based on published meta-analyses, a standardized mean difference effect size of 0.50 represents a clinically meaningful effect size for improvement that has been achieved in training programs for managers, caregivers of chronically ill patients, and parents of children with difficult behavior , 81–83. Members of our Support Group Leader Advisory Team and our patient organization partners have emphasized the importance of evaluating the trial’s planned secondary outcomes. Thus, we will attempt to enroll 180 participants total  in order to have sufficient power to adequately evaluate secondary outcomes.At the initiation of the trial and prior to random selection and allocation for specific training and waitlist groups, patient organization partners from Scleroderma Canada and Canadian provincial organizations, the Scleroderma Foundation in the USA, Scleroderma & Raynaud’s UK, Scleroderma Australia and Australian state organizations, and Scleroderma New Zealand will contact group leaders to describe the SPIN-SSLED Program and will provide the SPIN team with a list of eligible support group leaders. This recruitment phase is planned to begin in July 2019.SPIN-SSLED personnel will then send an email invitation with a link to a Qualtrics survey containing the consent form, a demographic questionnaire with information about participants’ support group experience , and questions on the days and times when the interested group leaders could attend training sessions. In addition to describing the study, the consent form will explain (1) that some group leaders who enroll in the study will be randomly selected every 3 months to participate in the SPIN-SSLED Program and that others will be allocated to a waitlist; (2) that participants randomized to participate in the program plus those allocated to the waitlist will complete measures online at the time of randomization, post-intervention, and 3 months post-intervention; (3) that, depending on the number of leaders who enroll, it is possible that some group leaders will not be selected to receive the training nor be asked to complete trial measures as part of the control group; and (4) that enrolled participants who do not receive the training as part of the trial, either because they are selected for the waitlist or because they are not selected for the training group or the waitlist, will be offered the training post-trial per our agreement with our partner patient organizations. At the initiation of the trial, we will send up to three emails, one per week, to leaders who do not respond to the initial email or enroll in the trial. For existing support groups where there are co-leaders, one will be specified as the primary participant and any others as secondary; only one leader per support group will be eligible for inclusion in the trial, and the secondary leaders will only be enrolled if the primary leader day and time availabilities do not match those that are able to be provided in the trial. All interested leaders will be provided contact information for SPIN-SSLED trial personnel, who will answer any questions they may have during the consent process and over the course of the trial.We can feasibly deliver the program to 3 training groups simultaneously: thus, to deliver the 15 planned training groups, the intervention will be delivered in five “waves” with 3 training groups of 6 participants each per wave, plus 18 participants randomized to the waitlist control per wave. Prior to starting a new wave, we will email participants who have not yet been selected for participation in a prior wave to allow them to update their available days and times. We will then determine characteristics  of the training groups that are needed for the new wave and initiate random selection of participants and random allocation.To ensure that we will achieve adequate enrollment to reach our target sample size, we are working closely with partner patient organizations, who will inform potential participants of the trial and will emphasize to their affiliated support group leaders the organizations are partners in the trial. They will inform affiliated leaders that participation will allow them to be certified as trained support group leaders and that the organizations plan to require the training in the future.We will also advertise the trial through SPIN’s active Facebook and Twitter media and on the SPIN website so that potential participants and group leaders may contact their patient organizations. Additionally, a webpage  has beenInterested leaders who provide consent for participation will be entered into different pools based on their availabilities and taking into consideration time zone differences. Figure All 12 leaders  will receive an email invitation including a clickable link to the online survey platform Qualtrics, where they will be asked to complete the study baseline measures. This email will also communicate participants’ assignment to the training program or waitlist control. A second email will be sent to leaders allocated to the training group with the date and time of their first training session, the topic of the first session, the program manual, and information on how to log in to the videoconferencing system and online chatroom. Calls will be made if measures are not completed. Similar procedures will occur post-intervention and 3 months post-intervention.A potential concern is that participants will not be blinded to intervention status. In most pragmatic trials of training, education, or behavioral interventions, as in the SPIN-SSLED trial, participants cannot be blinded. This is understood as part of the response to being offered a treatment, similar to what occurs in clinical practice . Since tA second concern relates to the potential for contamination if participants randomized to the SPIN-SSLED Program share learning material with participants in the waitlist control. It is not likely that material would be shared between leaders from different support groups. Nonetheless, to attempt to minimize the influence of possible contamination, we will explain this concern to participants in the training arm of the trial and ask them not to share their material or discuss the training sessions with other group leaders during the trial.Outcome measures will be completed using the online surveying tool Qualtrics. This method was used in the SPIN-SSLED feasibility trial, and data completion was 100% for all variables at baseline and post-trial. Limits on eligible values that can be entered will be set in Qualtrics to reduce erroneous entries. Once the online survey data are collected, the data will be exported to the statistics software program, IBM SPSS. Members of the study team will check and clean the data using SPSS. All information obtained about the participants during this study will be treated confidentially within the limits of the law. To protect the participants’ privacy, upon inclusion in the SPIN-SSLED trial, a unique participant identification number will automatically be assigned to each participant. An encrypted database will be created for the SPIN-SSLED Program, which includes the participant identification number and name. Data security measures in place at Qualtrics are described in the Qualtrics security statement . InformaAnalyses will be conducted by a statistician blind to trial arm allocation. For the primary outcome analysis (SSGLSS post-intervention), we will use an intent-to-treat analysis that compares all patients randomly allocated to the SPIN-SSLED Program to all patients allocated to the waitlist control. Intervention effect will be estimated using a linear mixed model, adjusted for baseline SSGLSS scores. The model will include a random effect to account for clustering of participants in the training groups, but not for participants in the waitlist control arm, because there is no clustering in the control arm , 62. We Analyses of leader burnout, participation efficacy, and emotional distress outcomes will only include experienced leaders because candidate leaders would not yet have experienced burnout, participation efficacy, or emotional distress due to the burden of leading a group. These outcomes will similarly be analyzed (1) controlling for baseline scores only and (2) controlling for baseline scores, age, sex, and whether or not the leader has SSc. Statistical significance for all analyses will be determined based on two-sided α = 0.05.The trial will be overseen by the SPIN Steering Committee along with the trial investigators and the Support Group Advisory Team. The Steering Committee will provide scientific direction for the RCT and will meet periodically to assess its progress. It will be responsible for RCT protocol execution, routine monitoring of data quality, and will meet semi-annually to discuss recruitment and retention and to assess that the trial is meeting key milestones consistent with the timeline.Participation in the SPIN-SSLED trial will involve weekly online training sessions and completion of online measures. We do not anticipate any safety concerns with the use of the SPIN-SSLED program, although if there are any adverse events, they will be reported to the local research ethics committee. Although it is hypothesized that the SPIN-SSLED Program will improve leaders’ self-efficacy for performing leader tasks, reduce burnout, and reduce emotional distress, it cannot be guaranteed that leaders will receive any benefits from this study. However, information learned from this research may lead to more effective SSc support group leader training programs, which may benefit those living with SSc in the future or people with other diseases. There will be no financial compensation for leaders who participate in the SPIN-SSLED trial.The SPIN-SSLED trial has been approved by the Research Ethics Committee of Centre intégré universitaire de santé et de services sociaux (CIUSSS) du Centre-Ouest-de-l’Île-de-Montréal (#2020–1780). All participants will provide electronic consent prior to taking part in the study. Any modifications to the protocol which may impact on the conduct of the study, including changes of study objectives, study design, patient population, sample sizes, study procedures, or significant administrative aspects will undergo a formal amendment to the protocol. This amendment will be submitted to the Research Ethics Committee for approval and documented in the trial registration.Our trial team has worked closely with patient organization partners from around the world and with a Support Group Advisory Team to design each stage of preliminary research, the SPIN-SSLED Program, and the SPIN-SSLED feasibility and full-scale trials. To the best of our knowledge, once tested, SPIN-SSLED will be the only peer support group leader training program that has been evaluated in a well-conducted RCT in any disease. Our SSc organization partners plan to implement the program post-trial to train and certify peer support group leaders, and the trial team will work with them to do this. Beyond SSc, the SPIN-SSLED Program will be easily adapted for use in other diseases.This is the first version of the protocol, finalized on 15 July 2019. Recruitment will begin 1–2 months prior to the beginning of the trial (July and August 2019), which is currently planned for September 2019.Additional file 1. SPIN-SSLED Program module overview.