The design, implementation, and evaluation of the study were systematically guided by all five dimensions of Glasgow and colleagues’ RE-AIM  framework. Linear mixed models, descriptive analysis and a framework approach were used to analyse the data. Significant improvements were observed in pre-service teachers’ willingness, confidence, and competence to implement physically active pedagogic strategies following the intervention. Pre-service teacher perceived effectiveness of such strategies on student outcomes also significantly increased and perceived barriers decreased. High adherence was consistently reported and the program was maintained after completion of the implementation trial by all lecturers. Four key themes spanning multiple dimensions and participant levels informed recommendations for program scalability: an “inter-systemic approach”, a “co-design” approach, “embedded in professional practice”, and “evidence of impact” on teacher practice. Anchored in real-world settings and tethered by implementation science, Transform-Ed! could have the potential to advance the teaching capability of teachers, and transform the learning experience and physical and academic outcomes of primary school students.The physical activity levels of children in Australia are critically low and correlate with reduced academic achievement and poor health outcomes. Schools provide an ideal setting for physical activity interventions to help children move more. Instead of targeting in-service teachers, this study embedded an evidence-based active pedagogy program called Transform-Ed! into pre-service teacher education. Pre/post surveys and post-program interviews and focus group discussions were conducted with key stakeholders ( It is well known that physical activity is vital for the health and development of children , while hA recent systematic review of classroom-based physical activity interventions highlighted how they can be a low-cost, practical and time-sensitive way to increase physical activity throughout the school day . Howeverhttps://transformus.com.au/).Transform-Us! is one example of a successful school-based physical activity intervention. Based on social cognitive theory , behavioImplementation science examines how evidence-based interventions can be applied in routine practice . The RE-Pre-service or initial teacher education should prepare graduates for classroom teaching with evidence-based content knowledge and a solid understanding of teaching practices proven to optimise student learning . Indeed,In 2018, a pilot study demonstrated the feasibility of Transform-Ed!, an evidence-based active pedagogy program, that was embedded into a core first-year unit of the Bachelor of Education (Primary) degree at Deakin University . The nexInformed by the successful Transform-Us! program, Transform-Ed! aimed to educate and upskill pre-service teachers in (i) planning and delivering active teaching, (ii) creating active environments, and (iii) encouraging active families. Examples of active pedagogic strategies include physically active academic lessons, physically active breaks from prolonged sitting, health-based curriculum content, alterations to the classroom to enable activity, encouraging activity at recess and lunchtime, and physically active homework. The research team and key stakeholders at the university  co-created the Transform-Ed! curriculum to align it with the priorities, structure, objectives and desired outcomes of pre-service teacher education .The present implementation trial draws from multiple evidence-based theories and frameworks, as shown in First and foremost RE-AIM  was the Participatory action research ,41 was aPractice architecture theory  informedIn addition to using practice architecture theory, lecturers used the fundamental elements of embodied pedagogy and transformative education. Embodied pedagogy joins the body and mind in a physical and mental act of knowledge construction , which iPeer support in learning is underpinned by the sociocultural theory of learning , and in Senior academics  were invited to attend the pre-program presentations, re-development discussions and post-program interviews. All lecturers and sessional staff likely to be responsible for the delivery of the core unit in which Transform-Ed! was embedded were invited to participate in initial program dissemination (presentations) and pre-implementation adaptation workshops, curriculum co-creation sessions, Transform-Ed! delivery, a self-report adherence checklist and post-program interviews  enrolled in the Transform-Ed! unit were invited to participate in the study. Further advertisement of the study included word of mouth, direct emails, flyers, the School of Education newsletter, and the unit online platform. A plain language statement was provided to all participants via email, and signed consent to participate was required. The study was approved by University Faculty of Arts and Education–Human Ethics Advisory Group (HAE-17–207).All The 12-week Transform-Ed! program was run in Trimester 1, 2019, embedded in a core curriculum unit of the undergraduate Bachelor of Education (Primary) degree. The unit also serves as an elective available to undergraduates outside of the Bachelor of Education (Primary) degree, for example Bachelor of Arts. The 12-week duration was selected to align with the length and structure of the Trimester and unit. A mixed method approach was used to evaluate the Transform-Ed! implementation trial . The usehttp://www.re-aim.org/wp-content/uploads/2016/09/checklistdimensions.pdf), which indicates ‘exemplar use for each RE-AIM dimension’, and the RE-AIM Criteria and Scoring Instrument (https://rtips.cancer.gov/rtips/reAimCriteria.do) were used to guide the quantitative aspects of the study (see The RE-AIM Model Dimension Items Checklist (tudy see . Each REAs summarised in https://zoom.us/). Stakeholder interviews explored organisational or system level barriers and facilitators to program reach, adoption, implementation, effectiveness, and maintenance  in the School of Education, through Zoom video calls  and the ‘RE-AIM Elements and Qualitative Data Questions and Examples’  as I was gaining knowledge and building a resource bank—this meant I could disseminate the messaging more broadly, more widely and more consistently.”(Lecturer 1)reach was that the program was “evidence-based” and practice informed, enabling transferability to practice.Another facilitator to “The results speak for themselves. This is its selling point and how you will get it out there. It’s based on such sound evidence and then when you take this toolbox [active teaching strategies] and experiment first-hand with them, it is transferred immediately.”(Lecturer 2)Sufficient provision of support and resources, coupled with flexibility to adapt the program, also emerged as a dominant theme.“I was able to include all the key concepts and resources in each session but as it is a very versatile program; I could adapt it as needed to reach the different cohorts.”(Lecturer 2)Lecturers suggested that if the program could be prioritised at an organisational level and prescribed in policy and curriculum documents , it may enhance the reach further.“Exposure to other areas of the curriculum across the education faculties is essential, it needs to be incorporated into all subject areas within the universities.”(Lecturer 3)Three major effectiveness themes emerged from stakeholder data. All three were associated with impact. Stakeholders collectively agreed that evidence of program impact across three separate, but interrelated, levels would be the ultimate measure of effectiveness. This included change in teacher practice, a change in school culture, resulting in the “ultimate indication of program effectiveness”, improved student educational outcomes.“All other measures of effectiveness are really just proxies. What we really need to show is transformed practice at a school level to improve student outcomes.”(Stakeholder 1)“The health, wellbeing and educational outcomes of a student is the ultimate indicator of impact.”(Stakeholder 3)Four major themes regarding effectiveness emerged from the lecturer interviews. The lecturers shared that the “effects of the program were immediate” and that the practical and experiential learning provided the pre-service teachers with “real evidence”, and subsequently significantly enhanced their learning.“The pre-service teachers were not only learning about these strategies but they were experiencing them and experimenting with them. It was this practical and experiential learning that consolidated the theory.”(Lecturer 4)Another major theme positively influencing effectiveness was the “embedded nature of the program”. Key content was embedded in each seminar and lecture across all weeks of the unit, as well as being integrated into curriculum documentation and assessment tasks. In addition, lecturers reported that they “actively modelled”, discussed and educated the pre-service teachers around the key aspects of the program in every session. Further, they provided opportunity for the pre-service teacher to experience receiving the content (as a student) and delivering the content (peer micro-teaching).“I think the pre-service teachers need to see how it works in practice. For this to become a real prospect for them they need to see it, be part of it and practice it. Without that integrated or embedded practice it may be hard for them to translate into their teaching when they are graduates. I think that is one of the best aspects of this program—its embedded across all aspect of the unit, lectures, seminars and assessment.”(Lecturer 4)Lecturers reported that their increased levels of ownership and empowerment as a consequence of the co-design process and resultant growth in program knowledge, confidence, and passion positively influenced program effectiveness.“I am a practising primary school teacher as well as a lecturer, so I was able to contribute a real-life perspective and make this applied. It was like connecting the research to practice and gave me real ownership as I was bringing in real life examples from my teaching.”(Lecturer 5)The lecturers also reported that there was significant growth in the pre-service teachers’ confidence and motivation as the program progressed. Moreover, the lecturers perceived that the pre-service teachers enjoyed, were satisfied, and had a positive experience with the program, which subsequently enhanced effectiveness.“From a lecturer perspective it was observing their [pre-service teachers’] passion and drive grow over the unit; to be active teachers. They realised that there is no other way; this is the only way. For the pre-service teachers, it was the belief in themselves, realising that they can now do it with the skills that they have. They feel equipped with their new ‘teacher toolbox’ to be active classroom teachers.”(Lecturer 2)Lecturers suggested that limited exposure of the program, both across other units in the course, as well as other years of the degree, may be a barrier to effectiveness.“The lack of consistent exposure and dose of active breaks and active teaching across their course will be a challenge. This is the only unit across the entire degree that they are exposed to active teaching the lack of exposure may dilute the message. Also being in first year, they have such a long time before they are out teaching that they may forget the message.”(Lecturer 1)n = 30/30) unanimously reported positive perceptions of the program. Although a few (n = 5/30) shared feelings of apprehension at the beginning, many expressed that the program effectively equipped them with skills, strategies, and knowledge, resulting in increased confidence.Via focus group discussions, pre-service teachers n = 21/30) reflected on the growth in their knowledge and understanding of the importance of integrating physical activity in their teaching.Many of the pre-service teachers n = 22/30) expressed that the program effectively provided a transferable skillset, viewed as a “value-add to their generalist classroom teacher toolkit”.Most of the pre-service teachers Stakeholder interviews revealed three major adoption themes. Firstly, stakeholders suggested that there needed to be a “disruption of perspectives” and an “exposure of contradictions”. Primarily in unpacking the shared purpose, vision and priorities of pre-service teacher education more broadly among academics, and respectfully but critically highlight how current practice contributes to this purpose, or contradicts it.“Our primary purpose, our work here is to teach people, not just to teach content. We need to make the connection between teacher practice and student outcomes, regardless of content. By understanding this sole purpose, we, the collective we, may begin to see the disconnect between what we to do and what we actually need to do. However, we should not only expose this contradiction but provide a solution.”(Stakeholder 4)“We need to understand what holds things [current practice] in place. Once we understand this we can start to shift or transform this.”(Stakeholder 1)adoption regarded utilising an “authentic collaborative process to create a shared vision”.The second theme that emerged as a facilitator to program “It [Transform-Ed!] should be based on shared ideas and true collaboration—both researcher and lecturers need to be willing to share, learn and change.”(Stakeholder 3)The third theme was ‘evidence of impact’. Specifically, stakeholders shared that the program had the capacity and potential to transform initial education, teacher practice and student outcomes simultaneously.“The beauty of this program is that it situates in the middle of the conversation, the high-end measure here which is student learning. If we can show improved student outcomes, you know that you are changing teacher practice. With this evidence you will have more people willing to adopt.”(Stakeholder 1)Three themes emerged from the lecturer interviews. Again, most prevalent was knowledge and education. The lecturers conveyed that the more they learned about and experienced the program, and the more they observed the results, the more willing and passionate they were to adopt it.“The more I learn about this program and the more I see the impacts of it, the more convinced I am to use it in my teaching.”(Lecturer 1)The lecturers reported that the co-design/co-develop aspect of the program enhanced feelings of ownership and empowerment. They reported on the idea of personal “buy-in” and “passion” for the program. In addition, the lecturers stated that the co-design aspect ensured that the program was practice informed. As such, the lecturers used their experience as teachers to feed into the evidence-based program.“Researching additional activities and bringing examples from my own teaching made me feel truly part of this. I could then also advocate for these activities as I knew they really worked in practice. I had ownership and could authentically add value to what was already there.”(Lecturer 4)The lecturers reported that although they felt the support and resources provided were sufficient, an increased amount and diversity of resources, especially around active breaks and active lessons, may have led to greater levels of adoption.The major implementation themes emerging from stakeholder interviews included the simultaneous transformation of practice, being embedded in professional practice and alignment to personal and professional priorities. Regarding the simultaneous transformation of practice, the stakeholders referred to how a change in pre-service teacher practice required concurrent changes at the school level. They made mention of “true implementation impact”, suggesting that the “life” that a practice, tool, resource or artefact takes on in a school is the real indictor of successful program implementation.“Making the tool or resource [available] is not enough, you need to implement it and embed it in practice, to ultimately transform practice. This is when the artefact takes on a life of its own. It is then that you can measure this change in practice. And that is your impact.”(Stakeholder 1)Similarly, implementing the program into professional practice emerged as a prominent enabler. Stakeholders made mention of the need for “clearer links, connections and pathways to professional practice” to enable broader and more effective implementation. Consistent messaging from placement supervisors, coordinators and placement setting staff, to ensure pre-service teachers feel well equipped and supported to embed the practices into their teaching was also perceived as important. Stakeholders again reported on the need for “true collaboration with a broad range of potential deliverers” to understand priorities and practices and, subsequently, create a shared vision for program implementation.“To take this to other areas of the curriculum and bring on board more people to implement the strategies, first you need to understand why they do what they do, why they don’t already teach in this way, and then you need to make clear links and benefits of this program to their practice. You will need to be open to learning and shared understanding, just as they will be—this is true collaboration.”(Stakeholder 2)Three major themes emerged from lecturer interviews. Consistent with previous domains, knowledge and education again emerged as dominant. Lecturers conveyed that the more they were educated about the program the more empowered, confident and motivated they felt to implement the program. Lecturers also reported that their previous exposure and experience in delivering the active breaks and lessons  was invaluable, and again contributed to increased feelings of confidence.“When I delivered it for the second time, I then knew exactly what and where the breaks and activities were. I was much more confident and well versed the second time around. I actually think seeking out my own active breaks gave me more ownership and confidence in the unit. I knew exactly what I had to do and when I needed to do it.”(Lecturer 3)The lecturers also perceived that knowledge and education were powerful “change agents” in regard to pre-service teachers’ attitude, confidence, and willingness to implement the Transform-Ed! key messages in their current (peer micro-teaching) and future (school) teaching.“A lot of pre-service teachers came in saying they were not skilled enough, not confident enough to do the unit. They realised quickly that these skills and strategies were accessible to all. They quickly understood the importance of integrating physical activity into their teaching, and how transferable they are to teach any content in the curriculum.”(Lecturer 1)implementation was “consistent exposure”. Congruent with their adherence checklist data, the lecturers conveyed that as the content was formally embedded and integrated across every aspect of the targeted unit  it allowed for consistent and repeated exposure of program messaging across that unit. Of note was the practical application of the messages, both in regard to lecturer modelling of practices, as well as pre-service teacher experiential learning. However, unanimously, the lecturers reported that exposure of the program in one unit only is not enough to elicit long term change in teacher practice, school culture and student outcomes.A theme that emerged as both a facilitator and barrier to “I think we need more ‘champions’–more people demonstrating these pedagogies–so that the pre-service teachers have constant exposure to it. More units across first year and also the program spread across more years of the course.”(Lecturer 2)As with adoption, another influential implementation related theme was the co-design/co-develop aspect of the program. Lecturers conveyed that they felt ‘empowered’ and shared feelings of ‘program ownership’ which facilitated implementation.“By choice I invested in a range of resources for lecture active breaks. This was not essential and could be avoided, but I was empowered to provide as many examples as I could, so I happily sourced out these extra activities and examples.”(Lecturer 1)n = 30/30) shared that the program had a positive influence on their perceived ability to implement diverse teaching strategies. This particularly regarded increased knowledge gained, the practical and transferable nature of the strategies and the “value now placed on classroom-based activity”. What follows is just an example of the multiple comments pre-service teachers made in relation to the perceived value of the program:Pre-service teachers unanimously maintenance included the need for an inter-systemic approach, for the program to be embedded in professional practice and for the program to provide a “transformation of practice rather than just translation into practice”. Stakeholders also shared the importance of understanding what holds current practice in place, what prevents change.The major themes to emerge from stakeholder interviews regarding “There is a need to have all the key players or levers across multiple systems, sitting around the one table at the one time. This includes people like the timetablers, resource and facility managers. These people are often left out but are actually critical to the logistics of making or preventing change.”(Stakeholder 1)Stakeholders also reinforced the need to “create clear pathways into professional practice” and then measure the impact of the program on teacher practice and student outcomes. The final theme that emerged was the “need for transformation, rather than just translation”.“Numerous programs can be translated from research to practice or from setting to setting. But for a program to really stand the test of time, it needs to be transformational. To enact change in and on the human system.”(Stakeholder 1)Following the implementation trial, the program has been sustained for two additional trimesters, by all lecturers at the three campuses. Further adaptations have been made by the lecturers to enhance contextual fit. This has largely included greater exposure to active strategies via modelling, the provision of more resources and further opportunities for peer teaching and greater emphasis and priority in assessment tasks. Moreover, lecturers were able to adapt the program quite significantly to successfully deliver in an intensive format in Term 3, 2019, and an online format due to the impact of Coronavirus (COVID-19) restrictions in Term 1, 2020.“I was able to include concepts even in the variation in delivery due to COVID-19. For the online lectures we were quite creative by providing videos to demonstrate the active academic sessions which the preservice teachers really enjoyed”.(Lecturer 3)Four major themes related to perceived facilitators of maintenance, emerged from the lecturer interviews. Again, the idea of consistent and repeated exposure of messaging across the unit was prevalent.“They [pre-service teachers] need to see the consistent and repeated exposure of these practices. Without this, by fourth year it will be diluted. If they can see it modelled and have experimentation with teaching across their whole course and in professional practice. This reaffirms what they are learning and will help them embed it into their teaching practice.”(Lecturer 2)maintenance. Lecturers reported that as the content was embedded in lecture and practical curriculum and modelling, in set peer teaching opportunities and experiential learning, “the program just become part of [their own] regular teaching practice”. However, lecturers also shared that if the program were formally embedded into teacher professional practice, it would have a more sustained impact on pre-service teachers.Similarly, the program being “embedded in practice” was a perceived facilitator of program “It [Transform-Ed!] needs to be built into the professional practice experience, and there needs to be more support. The more they experience it in practice, the more these concepts will be consolidated and become part of their regular practice.”(Lecturer 2)program champions’. They reported that early adopters  and undergraduate teachers would be well placed to advocate more broadly for the program.To further enhance program maintenance, lecturers suggested incorporating ‘“We could also rely on the pre-service teachers—we have given them some foundational knowledge and strategies to deliver curricula actively and they have had such a positive experience. If they question other lecturers as to why they are not seeing these strategies in their lectures or seminars—the demand may ignite interest in more lecturers.”(Lecturer 4)The lecturers suggested that there needs to be system or organisational level changes for the program to be expanded, prioritised and sustained. They suggested that the program needs to be embedded across more units and more years of the course, embedded within professional placement experiences in schools, and prioritised in Graduate Teacher Standards.“There needs to be a system level approach to this. Modelled from the top down and bottom up—if we can get it into all aspects of teaching and learning, as research, then I think we will have a chance to really impact teaching.”(Lecturer 2)n = 25/30) reflected that the skills and strategies learned across the unit, would be very much part of their future teaching, and that this program has “shaped” and “informed” the type of teacher they wanted to become.The majority of the pre-service teachers n = 18/30) suggested that they would “advocate for change” or be “champions of active teaching” when out in schools.Not only did the teachers report that they would incorporate these strategies into their own teaching, but many This study is the first to use implementation theory to understand the potential of pre-service teacher training to support school-based physical activity interventions in real-world school systems. The primary purpose of this research was to investigate the reach, effectiveness, adoption, adaption, implementation and maintenance of Transform-Ed! across the first year of an undergraduate teacher education unit, using the RE-AIM framework. Overwhelmingly, the Transform-Ed! program was received positively by key stakeholders, lecturers and pre-service teachers alike. The program improved pre-service teachers’ perceived competence, confidence, and willingness to integrate active pedagogies into current and future teaching practice, and their emerging identities as teachers. Indeed, the impact of the program on pre-service teachers’ perceptions was more substantial than what we found in the pilot study, particularly in regards to their willingness to implement the strategies . The lecProgress in school-based physical activity interventions has been potentially hampered by a focus on individual level, short-term effectiveness trials targeting currently practicing teachers. To address this, the current research targeted pre-service teacher education in an upstream approach which comprehensively used all five RE-AIM framework domains  to desigThe inter-systemic approach refers to organisational level involvement. The broader the scope, or reach, of the system, the more actors would be influencing the ‘human system’, lending a greater likelihood of effect. This is congruent with ‘action theory’, which enables systematic learning of what works and what does not work at the organisational level . Glasgowadoption, implementation, and effectiveness dimensions. In particular, co-design here refers to processes that create a shared vision among participants, which has been viewed as effective in the development of school-based physical activity frameworks [Co-design emerged as an important feature across the ameworks . Co-desiameworks . Key staameworks , but alsameworks . Therefoameworks .With the lecturers, the intention of participatory action research was that they were active contributors to the research, and had increased control over the design and delivery of the Transform-Ed! program . A critiThe third major theme was the importance of embedding Transform-Ed! into professional practice, which emerged across the effectiveness, implementation, and maintenance dimensions. This means that the program needs to be fully rooted in the teaching practice, providing easy transferability and relevance of the physically active pedagogical skills. This is congruent with recommendations by the Teacher Education Ministerial Advisory Group, suggesting that initial teacher education should be a mutually reinforcing experience of higher education and professional learning and practice . All parThe final of the four major themes was the importance of evidence of impact, in regard to both adoption and effectiveness of the program. Evidence of impact refers to both the evidence base that the program is built on , as well as evidence that the Transform-Ed! program works. Key findings from the Teacher Education Ministerial Advisory Group  reveal tRegarding evidence of impact, the main limitation of this trial was that it did not measure the impact of Transform-Ed! on primary school children’s physical activity, sedentary behaviour, or engagement with their learning or academic outcomes. As mentioned, future research is needed to examine the real-world impact of Transform-Ed!. It was readily noted by key stakeholders that the ultimate measure of effectiveness would be improved student outcomes, as this would be simultaneously indicative of changed teacher practice and changed school culture . It has The results of this implementation trial of Transform-Ed! reinforced decisions about its design, and inspired enhancement of these features in future scaled-up version of the program. For instance, heavy involvement at the organisational level and emphasis on a process of co-design should be retained and taken even further. Participants reported that the program needs to be embedded more comprehensively in professional practice, for example, by expanding the program across units and years of the teaching degree and via professional practice teaching placements. Perhaps, most importantly, evidence of impact would need to be scaffolded in future research, primarily by determining the effectiveness of the program on improving primary school student activity levels and engagement in learning and academic outcomes, and also investigating the potential for flow-on effects of the program to other teachers in placement schools. According to the present study, targeting pre-service teacher education appears to be a promising avenue to enhance the capability of Australian teachers and, in time, hopefully transform the learning experience and outcomes of students.


Background: Explored the role of public health centers in the delivery of physical activity programs to older Brazilians.Methods: Total of 114 older adults (81% women) from public health centers across the city of Florianopolis, Brazil, were randomized into three groups: behavior change group (n = 36), traditional exercise group (n = 52), and control group (n = 26). The behavioral change group included 12 weekly meetings (2 h each). The traditional exercise group offered a 12-week exercise class. Individuals in the control group participated only in measurements. Program evaluation included a mixed-methods approach following the RE-AIM framework . Trained interviewers conducted 12 focus groups and 32 interviews with participants in the program, professionals delivering the programs, community health workers, and local and city administrators overseeing public health centers. Participants completed health, quality of life, and fitness assessments at four time points.Results: The study reached 11.5% of the eligible population in the community. Older adults' resistance to change and limited understanding of behavior change science by public health center staff hindered program reach. Physician encouraging patient participation and personal invitations by community health workers were perceived as favorable factors. Results of program effectiveness and maintenance suggest that behavior change strategies may be better suited than traditional exercise classes for decreasing sedentary time and increasing moderate-to-vigorous physical activity, as well as improving participants' quality of life. Only 14% of public health centers in the city adopted the programs. Heavy workload of health educators delivering the programs and limited physical space for program delivery were barriers for adoption. The fidelity of program delivery was high and indicates that the programs are culturally-appropriate for the Brazilian context and feasible for implementation by local health educators.Conclusions: Our findings support the potential for dissemination of behavior change and traditional exercise programs to older adults through public health centers in Brazil.REBEC: RBR-9pkxn2 (retrospectively registered) Register April 20, 2019. Regular physical activity (PA) has been associated with maintenance and improvements in functional capacity and quality of life in older adults . CurrentIn Brazil, the 1990's were marked by an increase in physical activity promotion for older adults, when the Federal Government, States and Municipalities began subsidizing exercise classes to individuals for free or at low cost. The majority of these classes involved structured exercise programs  led by pDespite the governmental efforts supporting the implementation of traditional exercise programs for older adults, PA participation remains disappointingly low across Brazil. For instance, the Brazilian national public health surveillance system (VIGITEL) assessments between 2011 and 2016 found no significant increases in leisure time PA, and reported that only 22% of adults 65 and older meet the recommendation for PA , cities across Brazil support teams of multidisciplinary professionals at local HCs to provide health care at no cost to about 4,000 people living in each neighborhood or community. Each team has a physician, nurse, technical nurse, and, at least, five community health workers with some also including nutritionists and exercise specialists.Public Health Centers (HC) divided in five health districts. The implementation of our project started with an initial meeting where the study was presented to all five health district coordinators. Of those, two coordinators (districts North and East) demonstrated interest in participating in the study. A total of 20 HCs belonged to districts North (n = 11) and East (n = 9), but not all of them had the physical structure and human resources to offer the programs. Thus, a total of six HCs were involved in the study and randomized to one of three groups  stratified by health district .The BCG participated in a behavioral change program that was adapted from “Active Living Every Day,” or ALED . ALED isParticipants in the TEG received a 12-week exercise class conducted at the local HCs. The classes were held three times per week for 60 min duration. The classes included a 5- to 10-min warm-up, 25 min of aerobic exercise at 50–80% of max. aerobic power, resistance training for 20 min, and a 5-min cool-down. Participants had their heart rate and ratings of perceived effort tracked throughout the sessions . ClassesIndividuals in the CG participated only in measurements, without any intervention. They were asked to continue their routine activities before the start of the study. At the completion of the 9-month post-randomization assessment they were offered participation in the TEG classes.We chose to conduct a comprehensive program evaluation including all dimensions of RE-AIM  using quDescribed as the proportion of a target population that participates in an intervention . We asseMeasured at the individual level and reflective of the success of an intervention in improving health outcomes . To asseWeight was measured with the assistance of a medical weight scale. Height was measured with a stadiometer. Body mass index (BMI) scores were calculated and participants were classified as underweight <18.5), normal weight (18.5–24.9) and overweight/obese (25.0 or more)  (−1) and in moderate-to-vigorous physical activity   for 60 min or longer were interpreted as “accelerometer not worn” and excluded from the analysis . Physica−1) , in lighs min−1)  was calcQuality of life was evaluated by WHOQOL BREF and OLD questionnaire , 23. TheDefined as the absolute number, proportion, and representativeness of settings and intervention agents who were willing to initiate a program . AdoptioAt the setting level, implementation refers to staff fidelity to the various elements of an intervention's protocol. This includes consistency of delivery as intended and the time and cost of the intervention . We follDefined as the extent to which a program becomes institutionalized or part of routine organizational practices, maintenance also refers at the individual level to the long-term effects of a program on health outcomes . In our post-hoc test was used to compare difference between groups at different assessment points. We used intention-to-treat (ITT) analysis to keep all participants with non-missing baseline outcome measurements.The demographic characteristics of the sample are presented in Two Portuguese-speaking investigators transcribed the interviews and focus groups and checked them for accuracy. Transcript analysis was conducted in Portuguese. Deductive thematic analysis was conducted to identify themes/quotes within the RE-AIM model. A team of bicultural native Brazilian-Portuguese and English speakers translated the quotations through a translation/back-translation process to ensure semantic equivalence across languages , 25. Thin = 114) characteristics at baseline. Findings show a similarity among demographic and health variables across the three groups . Average age was 69.8 years, the majority were women (80.7%) with high school education or less (88.5%). Approximately, 57% were overweight or obese with an average BMI of 27.9 kg/m2. There was a high prevalence of chronic diseases with 43.9% reporting high blood pressure and 25.4% heart disease. The only significant difference among groups at baseline was regarding occupation and MVPA, where participants in the traditional group were more likely to be working and were found to be engaged in less moderate/vigorous PA.We were having a lot of turbulence in the unit [HC], we almost could not get involved. It was soon after that we were without the doctor. And, after then, the doctor went on vacation, so for me it was a very hard time to work.” (CCS BL). Another barrier deals with older adults' resistance to change. A community health worker described “Older adults struggle with new things…they fear if we invite them for something new, they are scared…” (ACS ST). A final barrier for recruitment was lack of staff familiarity with behavior change programs. Innovative and new to most staff in the HCs, the program was relatively difficult to explain and to understand. A community health worker described this concern when saying “Very often the team could not understand the concept of the program [referring to the behavior change program]” (PEFI).Results are shown in The involvement of the health team was very important” (PNS CAN). In addition, the program materials were perceived to be attractive by participants. An older adult participant said “It was very good [referring to the program recruitment material], because for me it was essential, you know why I do not live without it [laughs] that motivates you there…” (BP3). Distribution of flyers and personal invitations were effective ways to reach participants. This was illustrated by a community health worker who stated “We have a definite demand of people attending the center, we are capable of attending a defined number, but by word of mouth the information gets through and we will always have someone else attending, so I believe there would be greater attendance, a bigger group” (PNS ST).On the other hand, several facilitators for reach were identified. Doctors and other health professionals encouraging patient participation was viewed as favorable in interviews. One community health worker noted “P < 0.05). This result was maintained at 3- and 9-month post-intervention evaluations (P < 0.01). Participants in the BCG increased their MVPA behavior at post program assessment, and maintained their physical activity participation levels at 3-month post-intervention. Neither the TEG, nor the CG, increased their MVPA, experiencing significant declines in MVPA over time (P < 0.05). No difference was observed among groups and no time effect was observed in BMI. Participants in the BCG showed significant improvements in quality of life (WHOQOL-Brief) at 9-month post-intervention relative to baseline (P < 0.031).I needed to be physically active, I thought that I was limited by my illness, but I want to reach program targets and, I started walking more” (12 BLP). Participants also described their appreciation with regard to the program effectiveness. “Oh, I learned a lot with the program, mainly the walking activities, it was also good to know the number of steps we take, this device (pedometer) is great” (P1).Analysis of the interviews and focus groups suggest that many of the participants perceived positive changes in themselves as a result of participating in the study. One participant in the BCG stated “(…) I started to take the program seriously (…) the activities are important, today I feel satisfied with my weight loss, some people lost five, ten and after until twenty kilos (refer the lose weight), a little every day, I am very satisfied (14 BLP)”.The satisfaction with results achieved was noted by a participant with cardiovascular disease and obesity, as follows: “n = 5), HCs with insufficient physical space to offer the programs (n = 5) and, HC professionals simply declining to participate . Three of the five health districts declined to participate due to prior commitments or no interest in the study. Of the 18 health centers located in the two participating HDs, six agreed to participate in the study. Reasons for non-participation included health educators declining to participate for work or health reasons ; “Everybody is busy an d overworked” (ACS ST); “I cannot let the staff dedicate eight hours per week to the program during the training period when the program is focused on only twenty or thirty people and our demand is much greater than that and they must take care of others areas” (G2).These following quotations illustrate the heavy workload reported by the health center teams: “it is expensive, considering what is financially available from the government to each person, what is given to the cities, the fixed minimum wage, which is very low, the program is expensive…” (G1). “We would have to quit a program like “Floripa Ativa” or the walking group to include this new program…” (PEF3).Two researchers with expertise in RE-AIM and the ALED program assessed program implementation twice in each study site. The evaluation for all items under analysis achieved an average of 98% fidelity. The estimated cost per participant for 3 months in the BCG program was R$ 65 (about U$ 30) and the TEG was R$ 50 per month (about U$ 23). Overall, 47% of TEG participants attended at least 75% of the sessions compared to 27.3% of their BCG counterparts. When managers were questioned regarding the cost of the programs, they found it to be expensive, as shown by the following quotes: “…This study evaluated the role of public health centers (HC) in the promotion of physical activity programs among older Brazilians, and compared the impact of two program strategies, behavioral change and traditional exercise. The RE-AIM framework was used to ground the evaluation of our programs. This framework has been previously used to evaluate programs offered in real-world settings , 26, 27.Our analysis investigated the reach, efficacy, adoption, implementation, and maintenance of RE-AIM components by collecting qualitative and quantitative data from program participants and partners at the organizational level. Our findings reveal both strengths and areas for improvement of the program strategies, and identified important factors associated with the utilization of public health centers in such initiatives.reach of the programs offered at public health centers, with participation levels of about 12% of the older adult population. This finding suggests that future studies should seek to understand better older adults' resistance to change, and to develop culturally-sensitive strategies to overcome these barriers to reach. A systematic review published by Franco et al.  is consistent with the study by Baruth et al. . The patients/participants provided their written informed consent to participate in this study.TB designed the research. FA guided on the design and statistical analyses. CR analyzed the data. LK, FB, WC-Z, and AS contributed to data collection and interpretation of findings. TB, CR, LK, FA, FB, WC-Z, and AS wrote the manuscript and TB had primary responsibility for final content. All authors contributed to the manuscript editing, read, and approved the final manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The Internet of Things (IoT) has been implemented to provide solutions for certain event detection because of ease of installation, computing and communication capability, and cost-effectiveness. Seismic event detection, however, is still a challenge due to a lack of high-fidelity sensing and classification efficiency. This study proposes BLESeis, an IoT sensor for smart earthquake detection. BLESeis comprises three main parts: (1) high-fidelity vibration sensing using a MEMS accelerometer and digital filtering; (2) an embedded earthquake detection algorithm; (3) BLE (Bluetooth low energy) beacon for earthquake notification. For high-fidelity vibration sensing, a triggering algorithm and embedded finite impulse response (FIR) low-pass filter are developed. The acquired vibration is then classified by the earthquake detection algorithm developed to identify the earthquake signal from other vibration sources using time and frequency domain analysis. Upon detection of an earthquake, the BLE beacon broadcasts using the proposed data packet for efficient notification and visualization. The performance of the proposed system is evaluated through numerical simulations and a set of experiments using shaking table tests. The experiments show the feasibility of the low-cost earthquake detection and notification system. Earthquakes can threaten the lives of thousands of people in densely populated regions resulting in substantial financial loss. Earthquake early-warning (EEW) systems serve to detect the magnitude of an earthquake rapidly and alert many people to take protective actions such as covering and stopping trains ,2. SeverRecently, the EEW system has been evolved with the rise in the Internet of Things (IoT) system driven by the convergence of technologies such as micro electro-mechanical systems (MEMS), wireless communication, and increased computing power ,21,22,23In helping to resolve these issues, in this paper, a low-cost IoT sensor for earthquake detection, BLESeis was developed. BLESeis, built with BLE (Bluetooth low energy) technology, measures high fidelity vibration, processes seismic event data, and broadcasts results such as PGA (peak ground acceleration), and MMI-scale. Contributions of this paper are enumerated as follows: (1) BLESeis was designed with a MEMS accelerometer and a BLE integrated MCU  as a stand-alone seismic sensor that conducts data measurement, processing, and notification on board; (2) embedded software was developed to establish high-fidelity data measurement, seismic event detection; (3) custom BLE data packet was proposed for seismic event notification. The performance of the developed BLESeis was evaluated through a lab-scale test using a shaking table. The remainder of this paper is organized as follows. BLESeis is designed to deliver a reliable and fast customized alert to nearby Bluetooth devices. The hardware of the BLESeis consists of a high sensitivity MEMS accelerometer and BLE-integrated MCU. The software framework is divided into three main tasks: (1) sensing task, (2) seismic event detection task, and (3) notification task see .® Cortex™-M4 CPU running at 64 MHz, allowing fast and efficient computation of complex functions requiring floating-point math for signal processing. nRf52840 provides extensive memory availability of 1MB and 256kB respectively in flash and RAM, enabling high-frequency sampling and computationally intense data processing required for seismic data acquisition and processing. Another salient feature of nRF52840 is the integration of Bluetooth 5 that enables low-power and long-range communication.The IoT sensor platform used in the BLESeis is the Adafruit nRF52840 Feather Express  that was developed based on nRF52840  with Arduino IDE support see ; the nRFFor high fidelity seismic data acquisition, the BLESeis uses a digital MEMS accelerometer, LIS3DHH, which features a high sampling of 1100 Hz and a low noise density of The sensing task consists of STA (short-term average)/LTA (long-term average) trigger , low-pasThe STA/LTA trigger is computationally efficient and the most widely used method for seismic vibration detection . The STAThe implementation of the STA/LTA algorithm for proposed BLESeis follows the flowchart, as shown in Once the trigger is activated, the acceleration is measured for 10 s at a sampling rate of 1100 Hz. In order to avoid aliasing, FIR (finite impulse response) low-pass filter  is applicf) at 20 Hz, n of 7. The cut-off frequency is determined to remove unnecessary noise above 20 Hz to improve the accuracy of the earthquake detection. The length of the filter is truncated to 275 samples  for efficient filter implementation. Liu et al. have proposed transfer function for the low-pass filter expressed asThe seismic event detection task analyzes the acquired signal in the time and frequency domain to distinguish earthquake from type 1-stationary signal and type 2-non-stationary signal, including a few major frequency components see . Note thN without overlap were taken. Let N is 275 samples, which is a quarter of the total data size. The type 1 signals such as random and harmonic oscillations can be classified using the stationarity index of the signal defined asLet After the time-domain classification, the frequency-domain analysis was conducted to detect the earthquake signal. The PSD  of the measured signal BLE is the term for Bluetooth 4.0+ and is characterized by low power consumption, and it has the ability to exchange data either in connection or advertising mode . In adveThis paper proposes a BLESeis protocol to more effectively advertise seismic information ) to surrounding IoT devices. The proposed protocol was created by modifying iBeacon’s data packets without affecting the standard. Note that iBeacon protocol was selected because of ease of modification for the intended use . The definition of the BLESeis data packets is given as:Company ID (2-Byte): Identifier of the manufacturer of a BLESeis )GPS coordinate: (16-Byte): GPS latitude (8-Byte) and longitude (8-Byte) of the BLESeis PGA (2-Byte): Maximum measured acceleration in mg unit. Level (2-Byte): MMI scale of PGA multiplied by 10TX (1-Byte): TX power level, indicating the signal strength of the BLE device when transmittedThe numerical simulation was conducted to validate the performance of the STA/LTA detector and seismic event detection algorithm. The artificial earthquake, structural vibration, and random signal were prepared; the artificial earthquake was generated by Kanai–Tajimi ,30,31,32Note that the Hanning window  is appliWith a total of 300 vibration datasets consisting of 100 datasets for each case , the detection of the trigger was validated as shown in The STA/LTA algorithm successfully detected the change in vibration and showed a high precision and reliable detection capability by showing the time of trigger at 3000 ms for type 1, 4713 A total of 300 datasets were evaluated using the proposed earthquake detection algorithm. The seismic event detector first classifies the stationary and nonstationary signal using Equation (4) with the The performance of developed BLESeis was evaluated experimentally through two types of tests: ambient vibration tests and shaking table tests. Ambient vibration tests were conducted to demonstrate the low-noise level and performance of the embedded filter of the sensor. The shaking table tests were conducted on a shaking table with a generated random, earthquake, and type 2 vibration in order to demonstrate the performance of the seismic event detection.The ambient vibration tests were conducted to measure the noise floor of developed BLESeis and compared with the amplitudes of the various magnitude of earthquakes measured at the distance of 10 km .To validate the seismic event detection efficiency of the developed BLESeis, three types of vibration were excited by the shaking table: (1) type 1-random, (2) type 2-vibration with different driving frequencies, and (3) type 3 – artificial earthquake. The three types of vibration followed the design made in the numerical validation. The peak magnitude of each vibration generated by the shaking table was adjusted to have an MMI scale of 5 and 7. A total of 60 tests were conducted to validate the performance of developed BLESeis and were compared with the reference accelerometer.The detection capability of the BLESeis was evaluated through a total of 60 experiments using different loadings, and the results are summarized in The proposed beacon packet was captured and visualized through a smartphone see . The UUIIn this article, the BLESeis, which can detect earthquakes using low-cost MEMS sensors, is introduced. The hardware and software framework were designed carefully for the BLESeis to enable STA/LTA trigger, high-fidelity vibration sensing, and earthquake detection using time and frequency domain analysis. Moreover, the beacon packet is designed to efficiently broadcast the sensor’s location, PGA, and MMI-scale. The numerical simulation was conducted to validate the performance of the proposed detection algorithm, and parameters associated with triggering and detection are set accordingly. The experimental validation was carried out on a shaking table excited by random vibration, vibration with two major modes, and earthquake. The experimental results showed that the developed BLESeis detects the occurrence of a vibration and successfully classifies an earthquake with 100% accuracy. Moreover, the BLE beacon packets were shown to broadcast with defined information, including location, PGA, and MMI-scale that can be used for a customized alert system.Note that, even though the proposed sensor showed remarkable performance, it has the following limitations: (1) the notification range is limited to 10 m with BLE, so that long-range communication is difficult. The proposed sensor used BLE for close-range notification with advertising mode, and it cannot transfer the notification to distant devices; (2) the parameters of the proposed algorithm, such as the ratio of STA/LTA, should be re-defined according to the properties of the ground or place where the sensors will be deployed. Numerical simulation considering specific ground properties might have to be conducted.Future research will be conducted to develop a low-power sensor that can be used for long-term monitoring of a seismic event. Furthermore, long-range earthquake notification using ZigBee or LoRa will be researched to address the limitation of the notification range.


Population size estimates (PSEs) for hidden populations at increased risk of HIV, including female sex workers (FSWs), are important to inform public health policy and resource allocation. The service multiplier method (SMM) is commonly used to estimate the sizes of hidden populations. We used this method to obtain PSEs for FSWs at 9 sites in Zimbabwe and explored methods for assessing potential biases that could arise in using this approach.This study aimed to guide the assessment of biases that arise when estimating the population sizes of hidden populations using the SMM combined with respondent-driven sampling (RDS) surveys.We conducted RDS surveys at 9 sites in late 2013, where the Sisters with a Voice program (the program), which collects program visit data of FSWs, was also present. Using the SMM, we obtained PSEs for FSWs at each site by dividing the number of FSWs who attended the program, based on program records, by the RDS-II weighted proportion of FSWs who reported attending this program in the previous 6 months in the RDS surveys. Both the RDS weighting and SMM make a number of assumptions, potentially leading to biases if the assumptions are not met. To test these assumptions, we used convergence and bottleneck plots to assess seed dependence of RDS-II proportion estimates, chi-square tests to assess if there was an association between the characteristics of FSWs and their knowledge of program existence, and logistic regression to compare the characteristics of FSWs attending the program with those recruited to RDS surveys.The PSEs ranged from 194 (95% CI 62-325) to 805 (95% CI 456-1142) across 9 sites from May to November 2013. The 95% CIs for the majority of sites were wide. In some sites, the RDS-II proportion of women who reported program use in the RDS surveys may have been influenced by the characteristics of selected seeds, and we also observed bottlenecks in some sites. There was no evidence of association between characteristics of FSWs and knowledge of program existence, and in the majority of sites, there was no evidence that the characteristics of the populations differed between RDS and program data.We used a series of rigorous methods to explore potential biases in our PSEs. We were able to identify the biases and their potential direction, but we could not determine the ultimate direction of these biases in our PSEs. We have evidence that the PSEs in most sites may be biased and a suggestion that the bias is toward underestimation, and this should be considered if the PSEs are to be used. These tests for bias should be included when undertaking population size estimation using the SMM combined with RDS surveys. The Jh method -7. MethoThe service multiplier method (SMM) is a commonly used method to estimate the size of key populations. The method uses 2 data sources -12, one In recent applications, respondent-driven sampling (RDS) surveys have been used to obtain a probability-based estimate of the proportion of the target population who are service users ,7,11. RDIn addition to the SMM, various approaches for population size estimation have been used, including the enumeration method ,12, the In this paper, we build on existing guidance for implementing the SMM with RDS data  to critiWe first describe the data sources used, our application of the SMM, and then our approach to investigating the degree to which our study met the methodological assumptions and the potential resulting biases.Sisters with a Voice program  run on behalf of Zimbabwe’s National AIDS Council and Ministry of Health and Child Care. The program provides reproductive and sexual health services to women, identifying themselves as sex workers [Service data come from the  workers . During  workers . For eacThe probability-based sample comes from a baseline RDS survey of the Sisters Antiretroviral therapy Program for Prevention of HIV—an Integrated Response (SAPPH-IRe) trial, a cluster randomized controlled trial that was conducted among FSWs at 14 different sites across Zimbabwe in November and December 2013 (PACTR201312000722390) ,28. RDS seeds) from subgroups of the target population at each site, through the mapping of sex work in each community by geography, age, and sex work typology [To initiate RDS recruitment, we purposively sampled 6 to 8 participants  was attM, the number of visits to the program of unique women within the reference period, FSWs were counted only once using their identifier [To determine entifier . We exclentifier . We usedentifier . where N is the estimated population size of FSWs at each site, P is the RDS-adjusted population proportion of FSWs who reported program attendance 6 months before the RDS survey, and M is the total number of FSWs who attended the program within a period of 6 months before the RDS survey [Sisters with a Voice clinic.”We applied the formula for the SMM, S survey ,7,11. ThThe RDS-II estimator was used to estimate P , and theN by combining the variances of P and M using the following formula:  where mμ is the mean of M and pμ is the mean of P [As recommended, we used the delta method to estimate the variance of ean of P ,35.The SMM makes at least four assumptions, including (1) all members of the population being counted should have a chance of being included in both sources ,11, (2) For RDS-II estimates to be considered unbiased, assumptions including reciprocity, sampling with replacement, a completely connected networked population at each site, accurate report of personal network size, final sample independent of the original seeds, and random recruitment have to be satisfied ,34,36-40Reciprocity is an assumption of the Markov process, which states that if individual A recruited individual B, then in principle, B could have recruited A . Given tP include recall bias where women may misremember dates and/or may not have recognized a service they visited as the program service and mobility (including mobility in and out of sex work) as a sampling bias where women who access the program may not be sampled at the time of the survey, and those who are sampled may not have potentially used these services over the past 6 months. A bias in the estimation of M could arise if the program failed to perfectly identify unique women visiting in the reference period.Other potential biases in We, therefore, investigated some of the RDS and SMM assumptions listed in P stabilizes as the sample size increases [In the RDS framework, seeds are selected purposively with the assumption that if recruitment is done with a sufficiently large number of waves, then the final sample would be independent of the seed characteristics . We usedncreases . A convencreases . If the P varied by seed using bottleneck plots. The vertical axis of the bottleneck plot shows the estimate of the RDS proportion and the horizontal axis shows the cumulative RDS sample size, and these are shown separately for each seed . If the individual seed estimates are not all converging toward a shared estimate, it might imply that the population is not really well networked, there is strong segregation into subgroups or that recruitment has got stuck in one branch of the network (a bottleneck).We assessed whether the RDS-II weighted cumulative estimates of P. Recruitment homophily is the tendency for women to recruit others like themselves with respect to reporting program attendance. In this case, it is the ratio of the number of recruits that have the same program attendance status as their recruiter to the number, we would expect by chance. If recruitment homophily on P is approximately 1, then there is little evidence of recruitment homophily, whereas values larger than 1 indicate more homophily.The indication of nonrandom recruitment was investigated by measuring recruitment homophily on iY) 0 + β1X= β*Site where Y is knowledge of the existence of a program and X represents each individual characteristic.The SMM requires that all members of the target population have a nonzero probability of being included in both the RDS survey and the program data ,9, indicWe also assessed the SMM assumption that the 2 data sources to be combined should be drawn from the same population, with the RDS data being representative of this population . Under tUnweighted descriptive analyses of program data and RDS-II weighted descriptive analyses of RDS data as well as comparison of the 2 data sources were performed using Stata version 14.2 (StataCorp LLC), and all the other RDS diagnostics were performed using RDS Analyst version 0.5.1, which is based on the RDS package for R . PSE calWe recruited a total of 1739 FSWs from 8 seeds at site 1 and 6 seeds from each of the other 8 sites. Of these seeds at each site, only 1 seed had attended the program at site 1, 3 at sites 7 and 9, 5 at sites 2, 3, 5, 6, and 8, and all 6 at site 4.The PSEs and 95% CIs calculated using the SMM are shown in The number of women who attended program sites in the previous 6 months before the survey ranged from 28 at a site where the program was relatively new to 372 at a site where the clinic had been established for 2 years. The proportion of FSWs reporting program attendance varied from 11% to 69%. The highest PSE was 805 FSWs (95% CI 456-1142) and the lowest was 194 FSWs (95% CI 62-325). The 95% CIs for the majority of sites were wide .P converged as the sample sizes increased, indicating that the final estimate of P might be independent of the seeds , and for HIV testing, where relatively more women who had ever been tested for HIV had knowledge of program existence compared with those who had not tested  and number of children aged under 18 years (P=.01).There was little evidence of an association between the majority of sociodemographic characteristics and knowledge of program existence. Evidence of association was seen for education, where a higher proportion of women who reported secondary school or higher had heard about the program compared with those who reported primary school or none , with data relating to the program encounters at these same sites over the same recall period (M). Using these data, we estimated the size of the FSW population at each site using the SMM. Estimated population sizes ranged from 194 (95% CI 62-325) to 805 (95% CI 456-1142) across the sites for the period from June to December 2013, reflecting between 1% and 5% of the total female population aged 15 to 49 years in these sites.We combined data on the proportion of FSWs recruited to RDS surveys in 9 Zimbabwean sites and who reported attending the program , with the exception of 1 site. At this same site (site 1), although convergence had been achieved, the bottleneck plot appeared to show that program attendance might have differed substantially by the subnetwork of FSWs.We employed existing RDS diagnostics  alongsidon et al  in theirWe found little evidence that women with particular characteristics were likely excluded from the program, suggesting that the SMM assumption that all members of the target population should have a nonzero probability of being included in both the RDS survey and the program was met. Characteristics of program attendees were similar to RDS participants, suggesting that the data sources were likely from the same population with the RDS surveys representative of the population, therefore partly satisfying the requirements of the SMM.The major strength of the SMM is that it can be implemented using data collected for other purposes ,23,44, uThis study has several strengths. Our simple and straightforward diagnostics were able to identify potential biases and suggest the potential direction of bias in the PSEs. The RDS survey data were carefully collected with an in-house coupon manager software to track coupons, verify them, and check that they were redeemed only once . The defStudy limitations include the inability to investigate all assumptions made by RDS and SMM. The SAPPH-IRe trial baseline was not set up to be used to estimate PSEs, and as such could not investigate all assumptions made by RDS and SMM. We were not able to assess the RDS assumption of accurate reporting of personal network size by participants. We also could not assess the SMM assumption that the 2 data sources should be independent of each other. We do not have data about every sex worker that a woman knows and all of their characteristics to assess whether the ones she recruits for the survey are a random sample or not . The assessment of convergence and bottleneck plots is rather qualitative and relies upon visually assessing graphics, which might result in making subjective conclusions.P, and we are not sure about the implications of this. To get an indication of whether the population of FSWs recruited to RDS surveys and those recruited to the program differed, and to assess whether women who had heard of the program differed to those who had not, we combined the RDS samples. This overcame the difficulty of making these assessments with small sample sizes, but it violates the RDS assumption of a completely networked population to do this [Although we checked the design effect for the primary outcome of the trial for which these data were collected, which confirmed that the target sample sizes of 200 were adequate, we did not check the design effect for  do this .P obtained for use in the multiplier method PSEs, and in assessing further multiplier method assumptions where data sources allow. We recommend that they are included when undertaking population size estimation using SMM combined with RDS surveys. PSEs are increasingly being taken up in policy making and by funders to set program targets, even at subnational levels. If the PSEs are assumed to be unbiased, programs may either be expected to reach people who do not exist or be inadequately funded to meet the needs of key populations who are undercounted.Although there is guidance on RDS sample size calculations ,48 and gWe used a single multiplier for illustrative purposes, but in line with other groups, we recommend the use of more than one as multipliers are prone to unmeasurable bias . PSEs masex worker actually self-identify and are identified by their peers [When incorporating the SMM in RDS surveys for population size estimation, it is important to understand the context in each site, which can be achieved through detailed mapping . Key dynir peers , as wellir peers . Some fuir peers .Double counting of participants in program data needs to be minimized, as this could potentially result in overestimation of the PSEs. The program to be used in population size estimation should be accessible to all members of the target population, and members need to be given unique identifiers coupled with collection of additional information such that if they forget their program unique identifiers, they can easily be reminded. This will reduce the problem of duplication in the counting of individuals who attend the program on several occasions and partly contribute to the accurate calculation of PSEs. When estimating key population sizes, the SMM will ideally be triangulated with other population size estimation methods . The size estimates obtained from each of these methods can be quite variable ,7 such tgold standard method for estimating the population sizes of hard-to-reach populations, the SMM is a recommended method to use [The SMM can be used to incorporate RDS proportion estimates . Withoutd to use ,7. We im


This article uses an evaluation of New Heights, a school-based program for pregnant and parenting teens in the District of Columbia Public Schools, to illustrate how maternal and child health programs can obtain rigorous evaluations at reasonable cost using extant administrative data. The key purpose of the article is to draw out lessons learned about planning and conducting this type of evaluation, including the important role of partnerships between program staff and evaluators.This article summarizes the evaluation’s research design, data sources, and lessons learned about ingredients contributing to the successful implementation of this study. The evaluation employed a difference-in-differences design to estimate program impacts using administrative data merged across agencies.Several features of New Heights and its context facilitated an evaluation. First, New Heights leaders could clearly describe program components and how the program was expected to improve specific student education outcomes. These outcomes were easy to measure for program and comparison groups using administrative data, which agencies were willing to provide. Second, buy-in from program staff facilitated study approval, data agreements, and unanticipated opportunities to learn about program implementation. Finally, time spent by evaluators and program staff in conversation about the program’s components, context, and data resulted in greater understanding and a more useful evaluation.The New Heights evaluation is a concrete example of how a small program with a modest evaluation budget can obtain evidence of impact. Collaborative relationships between researchers and program staff can enable these informative studies to flourish. Programs that seek to improve outcomes for parents and children are increasingly asked to provide rigorous evidence of their effectiveness. However, programs without large budgets for evaluation can be daunted by the apparent challenges of building rigorous evidence about the effectiveness of their approach. Identifying an appropriate comparison group and obtaining data at reasonable cost are two of the biggest hurdles. The good news is that there are alternative evaluation designs that do not involve expensive sample recruitment or primary data collection.Programs that seek to improve outcomes for parents and children are increasingly asked to provide rigorous evidence of their effectiveness. Typically, programs must provide evidence that outcomes for program participants improved relative to a comparison group that did not experience the program. Funders, including government agencies and private philanthropy, may want this evidence to understand the impact of their investment or to justify additional support for a program. Information about program effectiveness, coupled with an analysis of factors that support or impede implementation, is an essential ingredient for an evidence-informed approach to program development and improvement.However, programs without large budgets for evaluation can be daunted by the apparent challenges of building rigorous evidence about the effectiveness of their approach. Identifying an appropriate comparison group and obtaining data at reasonable cost are two of the biggest hurdles. Seeking evidence of effectiveness, some programs focus their limited evaluation budgets on primary data collection, which is typically one of the key cost drivers in an evaluation. When their budgets cannot accommodate much data collection, some programs settle for studies with samples that are too small to detect impacts even if the program were effective. The good news is that there are alternative evaluation designs that do not involve expensive sample recruitment or primary data collection. In some cases, extant data from multiple agencies can be used creatively to design informative, high quality effectiveness evaluations at lower cost. Data from local health departments and school districts, for example, can provide a wealth of information to evaluate program success.This article uses an evaluation of New Heights, a school-based program for pregnant and parenting teens in the District of Columbia Public Schools (DCPS), to illustrate how it can be an option for some programs to conduct a rigorous evaluation using extant administrative data. Although we briefly summarize the evaluation’s design and findings  at the U.S. Department of Health and Human Services launched the Pregnancy Assistance Fund (PAF). The program seeks to improve intermediate outcomes, such as access to health care and education, which in turn are hypothesized to delay a subsequent pregnancy and bolster the long-term well-being of teen parents and their children.In the 2011–2012 school year, DCPS used a PAF grant to expand an existing program, New Heights, to all of its large comprehensive high schools. New Heights is a voluntary, school-based program of supports to help expectant and parenting students—both mothers and fathers—navigate the challenges of pregnancy and parenthood and complete high school. Recognizing that expectant and parenting students can feel overburdened, embarrassed, and discouraged, the program seeks to reorient students toward identifying immediate education goals, making longer-term plans, and identifying clear pathways for achieving them.New Heights’ key feature is a dedicated program coordinator in every school. Coordinators are trained staff employed by the district who deliver the program’s multiple components, tailoring their efforts to the needs of their students. Coordinators are responsible for integrating four main components into the regular school day: (1) advocacy; (2) targeted school-based case management; (3) weekly educational workshops; and (4) small rewards, primarily in the form of baby care products, for program participation. Taken together, these components aim to help expectant and parenting students identify their strengths and use them to overcome challenges to self-sufficiency and educational success.To increase school engagement through improved attendance, New Heights supports students in overcoming the barriers, such as childcare and school policies, that keep them out of the classroom. The program also aims to increase credit accumulation and empower students to advocate for themselves. These short-term outcomes are expected to lead to long-term improvements such as increased graduation rates, postsecondary enrollment, employment, and the delay of subsequent pregnancies.The evaluation used a difference-in-differences (DID) design to estimate the impact of New Heights for all eligible female students in study schools, including both program participants and non-participants. Although the program served both males and females, the study focused on females because they could systematically be identified as parenting and the males could not. The study focused on intermediate outcomes related to education but did not examine birth outcomes because the design required that outcomes be observable for all students, both parenting and non-parenting. Notably, the study used only retrospective, administrative data from DCPS, the District of Columbia Department of Health (DOH), and the District of Columbia Department of Human Services (DHS). Later in this article, we describe our strategies for obtaining and making sense of these data.before New Heights was introduced and outcomes for subsequent cohorts of students attending the same schools after the program was introduced.For each outcome, the study team estimated the pre-post difference in relevant academic outcomes for parenting students. This estimate is the difference between outcomes for cohorts of students attending the schools before the program was introduced and after the program was introduced.To account for factors other than the New Heights program that might affect outcomes, the study team estimated the difference for non-parenting students attending the schools To create a final estimate of program impact, the study team subtracted the non-parenting students’ pre-post difference (step 2) from the parenting females’ pre-post difference (step 1). This estimate is the “difference in differences” from which the method takes its name.To refine the calculation of the impact for students who actually received services from New Heights (75% of those who were eligible), we divided the estimated impact for all eligible students by the proportion of eligible students who participated in New Heights.The DID study of the effectiveness of New Heights followed four main analytic steps:The DID approach was a good choice for this evaluation because the New Heights program operated in all DC comprehensive high schools during the study years. The program’s full coverage across comprehensive high schools meant that the study could not compare outcomes for students in the same cohort at schools with and without the program. Instead, the study compared outcomes for parenting students who had the opportunity to participate in New Heights to outcomes from previous cohorts of parenting students at the same schools  and increased credit accumulation by 1.1 credits (a 24.4% increase). Among students who were 17 years and older, New Heights increased the graduation rate among by three percentage points (an 18.8% increase compared to the rate for parenting students prior to New Heights).Several features of the New Heights program and its context facilitated an impact evaluation, as we describe in depth in the following sections. The program had a well-specified logic model and access to administrative data on births, program participation, and the education outcomes it hoped to improve. Program leaders were deeply confident in their approach and welcomed an independent evaluation, and agency researchers helped the evaluators to access and understand the administrative data. Even so, not all evaluation conditions for New Heights were ideal. The evaluation began four years after the program began to offer services, which ruled out some design options. The encouraging lesson of New Heights, however, is that obtaining rigorous evidence of effectiveness does not necessarily require optimal conditions.Logic models are the cornerstone of a strong evaluation of program effectiveness. A good logic model specifies the key components of a program that are hypothesized to produce one or more specific outcomes. New Heights leaders were able to clearly describe the components of their program; how those components fit together; and how the program was expected to improve specific, meaningful education outcomes . These outcomes were easy to measure for the program and comparison groups using administrative data.Further, New Heights was a mature program that had operated for several years and had honed its vision. It had excellent staff monitoring and supervision to ensure that the shared vision was implemented. All of the work to develop and refine the program made New Heights ready for an impact evaluation.The study relied on merged, longitudinal administrative data from DCPS, DOH, and DHS for 2007–2015. These data were initially collected for purposes other than evaluation (such as basic recordkeeping and compliance reporting) but could be used in new ways by the evaluation team to learn about the impact of the program. Using administrative data—rather than surveys or interviews—also reduced evaluation costs and the data collection burden on program staff and participants.To obtain these data, the evaluators executed separate data agreements with each agency. Because DCPS and DHS were already collaborating on the program’s implementation, both agencies were invested in the success of the evaluation and sought to support the evaluation team through the agencies’ research review processes.The study used data from 2007 through 2015 to:Identify females in the schools with and without children. Data from DOH identified teens who had given birth in the district and the date(s) they gave birth.Form comparison groups. Administrative data from DCPS, DOH, and DHS enabled the study to create a comparison group of parenting students who attended the schools prior to the introduction of New Heights and a second comparison group composed of non-parenting students.Assess outcomes. Key education outcomes—attendance, credit accumulation, and graduation—were standard information kept by the school district for all high school students across multiple years.Assess program participation. DHS records identified parenting students in the study schools who also participated in New Heights.The evaluators merged DCPS and DOH data using students’ first name, last name, and birth date. Although some student data could be matched on exact name and date of birth, other matches could be made only by using an approximate spelling of the first and/or last name. To assess the quality of matches, the team hand-reviewed a randomly selected set of cases for accuracy. This in-depth examination of match quality showed that some matching strategies were acceptable (93% accuracy or more), but other strategies could not be used because they produced too many inaccurate matches.From the start, New Heights program leadership welcomed the evaluation. Because they had been tracking their results through performance data, program staff had a data-informed confidence that their program was making a difference for teen parents. Still, the program staff needed time to develop confidence in the evaluation team and become convinced that the study design was the most rigorous, unbiased, and best possible for demonstrating program effectiveness. Once program staff understood the design and the evidence that it could provide, they were willing to champion the evaluation within DCPS, DHS, and DOH so that evaluators could acquire the necessary data.Although a program implementation study had not been planned initially, the growing trust and regular communication between program staff and evaluators enabled them to put an implementation study in place when the opportunity arose. These qualitative research activities, which were ongoing while administrative data were being collected, cleaned, and analyzed, were invaluable in making sense of the impact findings. The impact and implementation findings enabled telling a story of not only whether the program had an impact but why it might have had that impact.As they sought to develop an evaluation plan, New Heights program staff and the study team held repeated, extensive conversations in person and via email about the program and the district context. Through these conversations, program staff and evaluators developed a design for a retrospective, rigorous evaluation that was feasible within the local context, time, and budget. One member of the evaluation team reported, “It was through the interaction between the evaluators and the program staff that the research design became something that we could imagine. We initially had modest expectations. But as we talked, the ideas began to flow.”When the evaluation got underway, DCPS researchers and program staff spent time making sure that the evaluators understood the changing institutional context (such as changes in DCPS policies) to place findings in context. This work included discussions about changes in how administrative data elements were reported, coded, and calculated by DCPS across the eight years of data.Refine the program’s logic model until it faithfully represents what the program offers and the outcomes it expects to affect. Logic models are a basic building block of evaluation.Take time to learn about the administrative data that is available or might become available. The more that program staff learn about these data—including the data elements, the completeness and quality of the data, and the potential to crosswalk data across agencies—the better prepared they will be to provide this important information to a potential evaluator. This information can help them to assess the feasibility of an evaluation and develop an approach that can work for the program.Consider whether there are elements of the study design used for New Heights that can be copied outright or slightly adapted for the evaluation. There are no copyrights for research designs!Seek inexpensive sources of evaluation assistance, such as graduate students at local universities—or even universities outside of the area. If program staff are able to explain their logic model clearly and provide good detail about available administrative data, they have a good chance of capturing the interest of academic researchers.Although the use of administrative data substantially reduced the cost of the New Heights evaluation, the study did receive funding from the U.S. Department of Health and Human Services. Programs that do not have these resources could take some of the following steps to get ready to evaluate their effectiveness:In recent years, there has been growing interest in how researchers and program staff can work in partnership to conduct more rigorous research to answer questions that are important for policy and practice. The idea behind these partnerships is that when individuals with a variety of expertise and desire to learn are around the table, the resulting research will be stronger and more relevant. The New Heights study is a concrete example of how that kind of partnership produced evidence of impact for a relatively small program with a modest budget for evaluation. The lessons learned from this study are that creative, cost-effective designs are possible—and also that collaborative relationships between researchers and program staff can enable informative studies to flourish.