Stepping On) in the community setting. As the first step for this study, a panel of subject matter experts was convened to suggest which parts of the Stepping On fall prevention program were considered key elements, which could not be modified by implementers.Falls among older adults result in substantial morbidity and mortality. Community-based programs have been shown to decrease the rate of falls. In 2007, the Centers for Disease Control and Prevention funded a research study to determine how to successfully disseminate the evidence-based fall prevention program . Consensus was achieved that these elements were key: 17 of 18 adult learning elements, 11 of 22 programming, 12 of 15 exercise, 7 of 8 upgrading exercises, 2 of 4 peer co-leader’s role, and all of the home visits, booster sessions, group leader’s role, and background and training of group leader elements. The top five key elements were: (1) use plain language, (2) develop trust, (3) engage people in what is meaningful and contextual for them, (4) train participants for cues in self-monitoring quality of exercises, and (5) group leader learns about exercises and understands how to progress them.The Rasch analysis of the 19 panelists using fit statistics shows there was a reasonable and sufficient range of diverse perspectives (Infit MnSQ 1.01, Stepping On program delivery. These elements were considered essential to program effectiveness. Findings from this study laid the foundation for translation of Stepping On for broad US dissemination.The Delphi consensus process suggested key elements related to  However, the manual that is written for the testing phase of a new intervention may not provide sufficient information for community organizations to know what elements can be adapted, and what elements must be retained as essential for program effectiveness. Thus, before translating a program for widespread dissemination, it is essential to determine key elements  from the Centers for Disease Control and Prevention (CDC), we received funding to translate and package the research . In the elements . Subjectelements . Input fStepping On, as suggested by content experts through use of a modified Delphi consensus, prior to packaging the program for widespread dissemination across the US.The purpose of this research study was to identify key elements of Stepping On was the first step in a multi-step process to translate Stepping On for US audiences, guided by the Replicating Effective Programs (REP) framework ; a geriatrician with 15 years clinical and research experience in falls prevention and community-based research (JM); a nurse with 4 years of experience implementing community-based falls and chronic disease self-management interventions (SC); a PT with 10 years clinical and research experience in falls prevention (TS); the CDC’s program officer (KM); and three public health researchers with experience in injury prevention. Two of the study team members had prior experience with use of the modified Delphi technique.The study team who convened the Delphi panel and developed the questions for the panel consisted of: the Stepping On. This included experts in the areas of falls prevention, exercise, and self-efficacy-based interventions, as these were considered by the program developer as primary constructs of Stepping On. A list of subject matter experts from the US, Canada, and Australia was generated by the study team, based on study team members’ knowledge of the literature. The list included public health researchers and health professionals  who were experts in falls prevention. The list also included exercise scientists with expertise in older adult exercise programming, researchers with expertise in self-efficacy-based interventions, and some of the early Australian leaders of Stepping On, based on their experience with implementing the program. We chose to include both independent experts in the field as well as Stepping On leaders, to mitigate potential bias from using either group alone. For example, program leaders could define key elements based on what they enjoyed doing in the workshop, what they found to be helpful, what they were skilled at, etc.The study team identified a list of potential experts to serve on the Delphi panel. The team defined potential experts as individuals who had expertise related to the concepts, activities, and subject matter of Stepping On into a US community program. They were informed that the specific aims of the grant included gathering information from content experts about key elements of Stepping On through the Delphi process, testing and evaluating implementation of Stepping On in community settings, and producing a final package for broad dissemination and use nationwide. All experts were informed that participation as a panelist would involve reviewing the original research article on Stepping On and the leader manual, and then participating in a Delphi panel of up to three rounds. Experts were informed that anonymity would be maintained by use of anonymous SurveyMonkey™, and that informed consent was not required per the University of Wisconsin Institutional Review Board.The letter of invitation informed experts that the Delphi study was being done as part of a larger study funded by the CDC to translate the Australian-based via email in waves to ensure that the panel would consist of equal numbers of PTs, OTs, geriatricians, exercise scientists, public health researchers, and Australian Stepping On leaders. For example, if the first PT refused, then the second PT on the list was invited, and so on. Table Forty-four experts were invited to participate in the Delphi panel by the study team Stepping On Leader manual. The study team then met three times as a group by phone over 2 months to determine the domains into which the Stepping On program content and process should be organized. Some domains were suggested by the Australian program developer and others by other study team members, based on questions that arose from their experience with program implementation in Wisconsin. The final set of 10 domains was determined by consensus among study team members. The program developer suggested the domains of: adult learning, program parts, exercise, home visit, and booster session. Other study team members suggested the domains of: training and background of group leader, role of peer co-leader, and manual, based on questions that had arisen from their experiences with program implementation in Wisconsin. After discussion, the study team added the domain of “role of group leader,” to categorize elements related to group facilitation.The study team developed the beginning set of questions for the modified Delphi panel. To do so, first each study team member reviewed the existing After determining domains, through four meetings by phone over 3 months, the study team generated questions within each domain regarding potential elements of program effectiveness. The final round one questionnaire consisted of 112 items over the 10 domains: adult learning (18 items), program parts (21 items), exercise (23 items), qualifications of invited experts to introduce exercise (4 items), home visit (13 items), booster session (4 items), role of group leader (12 items), training and background of group leader (10 items), role of peer co-leader (4 items), and manual (3 items).Stepping On program, and asked: “How important is this element to the program’s effectiveness?” For example, for the domain of “adult learning,” the survey stated “The manual describes a number of elements that are used in adult learning to help engage in homework and class participation.” One element provided under that section was “Invite feedback,” followed by question “How important is this element to the program’s effectiveness?” The panelists were asked to rate importance on a scale of 1 = “Not important at all”; 2 = “Probably not important”; 3 = “Possibly important”; 4 = “Very Important”; and 5 = “Essential.” “Essential to the program’s effectiveness” was defined in the cover letter as indicating “an element whose absence would reduce the program’s effectiveness” in reducing falls among program participants. The 5-point scale of importance has been used to elucidate key components of yoga interventions for musculoskeletal conditions . We provided no further definitions of the points on the scale. All questions gave a choice of “unable to answer.” At the beginning of the survey panelists were instructed “Please check a response for every item of each question. You are required to answer each item before moving on, so check ‘unable to answer’ if you do not feel you can answer the item.” Panelists were asked to provide any comments they wished about any of their responses, and a space for comments was provided at the end of each domain of questions . The round one survey was piloted with three Wisconsin Stepping On leaders to ensure clarity of questions and response categories.Some questions used a scale of 1 = “Definitely not”; 2 = Probably Not; 3 = Unsure; 4 = Probably; and 5 = Definitely. Examples of these questions included: “Handouts are provided in the via telephone conference to develop round two questions based on round one results, then doing the same for round three, based on round two results. All scores and comments from each round were tabulated and provided verbatim to the study team. The study team examined all items that did not achieve consensus in round one to determine which should be advanced to round two. To reduce respondent burden, the study team only selected items for round two that had significant controversy  in round one. For questions being advanced to round two, the study team repeated or rephrased the round one question in the round two survey, and provided the comments and responses that the panelists had given in round one , Then the panelists were asked to re-score the question, using the same scale as in round one, and provide a rationale for their score. In round three, the same occurred, with the study team giving the range of responses and comments from round two.For computerized versions of the Delphi process, it is essential to have a moderator whose role is to synthesize information from each round and make decisions about what should be provided back to the group . The stulimited to 10–14 participants” was essential. As another example, in round one, the question was asked: “Should the manual have more information about maintaining safety with the community mobility session?” In round two, this was reworded for clarity as “Should the manual have more information about preventing falls and injuries from occurring while conducting the community mobility session?”In some cases, a panelist’s comment from the previous round suggested that the question was unclear. In these cases, the study team clarified the question in round two or three. For example, in round one panelists were asked whether a “group size of 10–14 participants” was essential. One panelist commented “With respect to group size a smaller group of 8–10 members may be more manageable for the facilitators (and productive) when there are more ‘higher need’ individuals in the group.” This led the study team to clarify the wording of the question in round two to ask whether a “group size Stepping On. We added this as a question in round two, asking panelists “Would anyone who has experience with exercise training and effectively lead an exercise session be qualified to introduce exercise concepts?” using the scale 5 = “Definitely” to 1 = “Definitely not.” Panelists were asked to provide a rationale for each of their scores.Some comments suggested ways of viewing elements that the study team had not considered. These comments were provided back verbatim to the panelists in round two or three. For example, one panelist in round one commented that anyone who has experience with exercise training and can effectively lead an exercise session would be qualified to serve as the invited expert to introduce exercise concepts in The Delphi process consisted of three rounds, with 3 months between each round. Round one survey required 30–40 min to complete; round two required 15–30 min; and round three required 5 min. All 19 panelists completed the first round. Two panelists (11%) did not complete rounds two and three. Because all responses were anonymous, we could not identify which panelists did not complete rounds two and three and why. The University of Wisconsin Health Sciences Institutional Review Office reviewed the Delphi Consensus protocol, which was determined to be exempt.Data was exported from SurveyMonkey™ to Excel for analysis. Descriptive statistics were used to provide frequencies of responses for each question. The primary outcome for each item was consensus that it was key element, consensus that it was not a key element, or no consensus.Consensus in round one was defined as 70% of the panelists scoring in one category, with the categories of “Essential” and “Very important” combined. In round two, consensus that an item was a key element was defined as 80% scoring in “Essential,” “Very important,” or “Possibly important,” as long as the “Possibly important” category did not equal or exceed 50%. Consensus that an item was not a key element was defined as >80% scoring in the “Possibly important,” “Probably not important,” or “Not important at all” categories, if the “Possibly important” category did not equal or exceed 50%. In determining the criteria for round two, the study team made the decision to combine the “possibly important” group with the other “important” options for the items where there were a minority of responses in the “Possibly important” category, taking into consideration that there had been little movement in that category compared to round one. The study team chose this approach because it conserved items as essential when there was a majority distribution of responses among the three categories.vice versa  was conducted on round one responses to explore the degree of diversity and/or homogeneity of responses of the panel. Rasch analysis allows responses from individuals to be tested against response patterns predicted by the model. The pattern expected by the model is a probabilistic form of Guttman structure, which is a deterministic model that has a strict hierarchical ordering of items . This alce versa . Rasch ace versa , 26. Meaesponses .Stepping On program. Three panelists had previously implemented Stepping On. Five panelists had expertise with self-efficacy based interventions.Table Stepping On program’s effectiveness in reducing falls, using a rating scale from 1 = not important at all, to 5 = essential. Twenty-four questions asked about items related to program implementation using a rating scale from 1 = definitely not, to 5 = definitely. The number of panelists completing each individual question ranged from 17 to 19 with a mode of 18. At the end of round one, 69 items reached consensus. Table The round one survey consisted of 112 items over 10 domains. For 88 items, panelists were asked to rate the importance of each item for the Z score −0.1, Outfit MnSQ 0.96, Z score −0.2 with a separation of 4.89). Only one panelist was “misfitting”  showing that they assessed items very differently than the other panelists. Another panelist (Point-measure correlation = 0.25) contributed the least, responding to most items as “essential” and not responding to others.The Rasch analysis of the 19 panelists using fit statistics shows that there was a reasonable and sufficient range of diverse perspectives  were (1) use plain language, (2) develop trust, (3) engage people in what is meaningful and contextual for them, (4) train participants for cues in self-monitoring quality of exercises, (5) group leader learns about exercises and understands how to progress them, (6) group leader links exercises to preventing falls, (7) the group leader learns about how to upgrade exercises, (8) the home visit provides assistance with follow-through of falls prevention strategies and activities, (9) reinforces those falls prevention activities that have been accomplished, and (10) the leader facilitates increased sense of ownership by participants.Table Although panelists did not reach consensus in round one, most felt that the number of exercises provided in the manual was sufficient. Most also felt that alternative exercises should be provided. Because of a wide variety of comments, round two further clarified questions related to exercises. There was no consensus about whether alternative exercises should be provided. Comments revealed that there should be ways to modify the exercises to accommodate those with physical limitations, but that the invited expert should be able to make modifications from the exercises provided. Comments included: “Older adults will ‘tell’ you how he/she would need to modify the activity. The lead instructor needs to check that the modification and progression is safe”; “Need to be sure the older adult is performing the activity within their functional capability. Functional capability is more important than doing the activity ‘perfectly’”; “Activities may need to be adapted according to the health status of the older adult”; “Adapt to individual need”; “The use of a physiotherapist to introduce the exercises means that individuals can ask specific questions about each exercise and if they experience problems, what alternatives there are”; “Some people may need alternative exercise”; “Sometimes important to adapt to unique personal or environmental settings”. Thus, panelists espoused modification of exercises for frailer participants.The Delphi panelists provided a number of suggestions regarding the participant handouts. Comments included “Update the home safety assessment form—seems written for a third party; substitute with a form that is easier to use”; “Review handouts for reading level and font size”; “Home hazard screening checklist—add suggestions for outside of the home”; “Check reading comprehension and cultural sensitivity of all handouts.”The home visit was a key element. Each of the six activities of the home visit was considered key. There was no consensus that any of these activities could be accomplished by phone.Stepping On  felt any professional would be acceptable if they met the criteria posed.There was consensus regarding the skill set the leader would need to bring to the role: experience with adult group facilitation, knowledge of falls prevention and exercise, and experience with seniors. However, there was less consensus regarding type of profession of the person who could fulfill the role of leader. Round three asked if a professional such as a social worker, nutritionist, or director of a senior center would be acceptable to fulfill the role of leader, provided that the person had skills in group facilitation, knowledge of adult learning principles, and prior experience working with seniors. It presumed that the professional would take a two and a half day training at the end of which he/she would demonstrate mastery of falls content. Round three showed consensus that a social worker could fulfill the role of leader, but there was no consensus that a senior center director or nutritionist could. A divergence emerged in comments. Three panelists (18%)  said none of the above categories could serve as leader. Among these three, two stated only PTs or OTs could lead; one provided no comment. Eleven panelists stated that any professional expertise would be acceptable, provided they met the criteria posed in the question. All respondents with prior experience leading Panelists who answered that any professional could “probably” or “definitely” serve as leaders commented that the elements for a successful leader were prior experience with older adults, motivation to run the program , skills in group facilitation, knowledge of adult learning principles, mastery of manual content, including falls prevention content, and awareness of basic safety principles. Panelists stated that to ensure above criteria are met, there should be a screening process before training, a test of mastery of content and safety principles after training, a demonstration of group facilitation skills, and a monitoring process after training to ensure the program proceeds with fidelity to key elements.Stepping On. Panelists reached consensus on most items in round one. Key areas of agreement centered on conceptual underpinnings of the program, roles of the leader, exercise elements, and other program elements. Our findings supported that the conceptual underpinning of the program and the group process were integral to learning and uptake of prevention strategies and that leaders required skills in knowledge of falls prevention evidence, group work, and principles of adult learning, decision making, and self-efficacy.In this study, use of the modified Delphi Technique elicited expert consensus suggesting key elements of Stepping On leaders are expected to meet the criteria established by the Delphi consensus. Second, in the program manual, key elements are signified with a “key” icon, and the manual’s toolkit provides a list of all key elements suggested by the Delphi panel. Third, findings led to development of a new 3-day leader training, which focused on didactics and practice to achieve competency in implementing key elements in practice. Before being certified as a Stepping On leader, trainees must pass a knowledge test of key elements and a competency test to show they are able to use the key elements in practice. Fourth, after certification, new leaders are monitored for fidelity as they facilitate one session of their first Stepping On workshop, and are coached afterward to improve performance. Fifth, findings informed the development of the Stepping On Site Implementation Guide that is used nationally to help organizations prepare to adopt the program with fidelity to key elements  provided information about barriers to participation and completion of Stepping On homework, about readability and understandability of the handouts, and what worked and what did not in the workshop. Two focus groups of leaders provided information about barriers to hosting and leading the workshop, and what worked and did not with leading the workshop. In addition, surveys gathered data from leaders who had received training in Stepping On but had not yet led a class ; from former participants who had not completed the program ; and from invited experts . The findings from focus groups and surveys provided essential information on how to adapt the program to meet setting-specific needs. The findings enabled us to provide a menu of options to leaders and sponsoring organizations so they could adapt delivery for their site, while still maintaining fidelity to the suggested key elements. For example, the need to modify exercises for frailer participants was confirmed by the findings of the survey of PT invited experts. As a result, in the draft leader manual we provided exercise modifications that the invited PT could use. Last, findings allowed us to modify handouts to maximize their acceptability and usability for participants.Translation of a program for broad dissemination cannot be based solely on expert consensus of key elements; it also requires input from stakeholders who have implemented the program, in order to ensure broad successful adaptation. We engaged in a number of steps to ensure that the US Stepping On participants, leaders, and invited experts. All handouts were checked with Microsoft Word diagnostics to ensure third grade reading level. We trained a new leader and co-leader, who implemented the draft program package in a senior retirement community. We monitored program implementation for fidelity, identified lapses in fidelity, and further revised the package to ensure fidelity with future implementation  did not reach the threshold of “essential” although the directionality of scoring favored their importance. These elements were added to the Stepping On national package as being “strongly advised.”It was important to identify areas that lacked consensus, or where consensus depended on the context with which the element would be implemented in Stepping On felt any professional would be acceptable if they had experience in group facilitation, knowledge of adult learning principles, and prior experience working with seniors, and if they received a two and half day training at the end of which they demonstrated mastery. It is an important question whether a non-professional (lay) leader, given the appropriate training and background of experience, can successfully conduct Stepping On. The Chronic Disease Self-Management Program, led by lay leaders, demonstrated success in decreasing hospitalizations and emergency room visits in a randomized trial  and the REP framework  who had expertise in self-efficacy based interventions, who could, therefore, also speak to elements related to group facilitation and adult learning. Second, all panelists were asked to review the  returns . Lastly,Stepping On program delivery across 10 domains ranging from leader background and training, to adult learning elements, programmatic aspects, and exercise elements. These elements and domains were seen as essential to program effectiveness in reducing falls. The Delphi panel’s consensus served as the foundation for development of the US program package, with subsequent research demonstrating its effectiveness in reducing falls. The US Stepping On program package is administered now by the Wisconsin Institute for Healthy Aging  which, through agreement with the Australian program’s developer, trains and certifies Leaders and Master Trainers, distributes materials and issues licenses to organizations to implement Stepping On, and provides pre and post training support to leaders and adopting organizations, all with the goal of maintaining fidelity to the key elements suggested by the Delphi panel.In summary, this Delphi consensus suggested key elements related to Stepping On falls prevention program. She is currently working with University of Wisconsin’s Active Aging Research Center to help develop internet-based technologies to help older adults reduce falls and to maintain independence.Jane Mahoney, M.D., is board certified in geriatrics and internal medicine. She is a Professor of Geriatrics in the University of Wisconsin School of Medicine and Public Health. She also serves as Executive Director of Wisconsin Institute for Healthy Aging, a non-profit organization that disseminates evidence-based prevention programs for older adults. She is Principal Investigator of the Community-Academic Aging Research Network, a NIA-funded initiative to support research collaboration between University of Wisconsin researchers and community partners from Wisconsin’s Aging Network. Dr. Mahoney has received funding from the American Physical Therapy Foundation, CDC, the NIA, and the State of Wisconsin for epidemiologic and clinical research on falls. She has studied risk factors for falls after hospitalization, clinical trials of community-based multifactorial falls interventions, and dissemination research on the This manuscript was presented in part at the 2012 Gerontological Society of America Annual Meeting, San Diego, CA, USA.JM was responsible for conceptualizing the theoretical and empirical formulations of each research project, literature review, study protocol and design, and for collecting, analyzing, and interpreting data and manuscript preparation. KM is a co-author and offered substantive intellectual input and expertise during each phase of the research formulation and manuscript preparation, and provided feedback on all drafts. TS was part of the original study group that developed and analyzed the Delphi data, feedback on the presentation at the GSA in 2012, and provided feedback on draft versions of the paper. I have read and have approved the final version.The findings and conclusions in this manuscript are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention/the Agency for Toxic Substances and Disease Registry.Stepping On Leader Manual, Third North American Edition, Freiburg Press, Cedar Falls, IA; 2011. The remaining co-authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.JM and LC are Co-Authors on the


Background: Complementary feeding (CF) contributes to child growth and development, but few CF programs are delivered at scale. Alive & Thrive addressed this in Bangladesh through intensified interpersonal counseling (IPC), mass media (MM), and community mobilization (CM).Objective: The objective was to evaluate the impact of providing IPC + MM + CM (intensive) compared with standard nutrition counseling + less intensive MM + CM (nonintensive) on CF practices and anthropometric measurements.Methods: We used a cluster-randomized, nonblinded evaluation with cross-sectional surveys . We derived difference-in-difference impact estimates (DDEs), adjusting for geographic clustering, infant age, sex, differences in baseline characteristics, and differential change in characteristics over time.Results: Groups were similar at baseline. CF improvements were significantly greater in the intensive than in the nonintensive group . In the intensive group, CF practices were high: 50.4% for minimum acceptable diet, 63.8% for minimum diet diversity, 75.1% for minimum meal frequency, and 78.5% for consumption of iron-rich foods. Timely introduction of foods improved. Significant, nondifferential stunting declines occurred in intensive (6.2 pp) and nonintensive (5.2 pp) groups in children 24–47.9 mo.Conclusions: The intensive program substantially improved CF practices compared with the nonintensive program. Large-scale program delivery was feasible and, with the use of multiple platforms, reached 1.7 million households. Nondifferential impacts on stunting were likely due to rapid positive secular trends in Bangladesh. Accelerating linear growth further could require accompanying interventions. This study establishes proof of concept for large-scale behavior change interventions to improve child feeding. This trial was registered at clinicaltrials.gov as NCT01678716. Bangladesh, a country of 155 million people with an annual birth cohort of 3.1 million, has made dramatic health advances for its population over the last 2 decades, and is hailed as a remarkable health success story. Although Bangladesh has seen rapid improvements in infant, child, and maternal mortality; immunization coverage; infectious disease treatment coverage; fertility rates; life expectancy; and other indicators of societal development, undernutrition remains a substantial challenge despite recent secular declines (Appropriate infant and young child feeding (IYCF) practices, which include exclusive breastfeeding until 6 mo of age and the provision of safe and nutritionally rich foods in sufficient quantity in addition to breastmilk from 6 to 23 mo of age, are a critical component of optimal child growth and development . In Bangz score (HAZ) of 0.23, resulting in a 29% reduction in stunting, as well as a 62% improvement in the uptake of recommended foods  intervention packages on CF practices and anthropometric outcomes. We were able to find few examples of large scale programs to improve CF practices, and no examples of such programs that were rigorously evaluated. Our study makes a substantial contribution to the literature on improving CF practices through a proof-of-concept rigorous evaluation of a set of interventions delivered at scale.BRAC, a large nongovernmental organization, delivered standard and intensified interpersonal counseling (IPC) and community mobilization (CM) in 50 rural subdistricts in Bangladesh through its existing countrywide essential health care program . For staDuring the intervention period, A&T facilitated the training of >75,000 frontline workers and health providers across the country. The program model reached a large scale, with an estimated 1.7 million mothers of children <2 y of age in 50 subdistricts accessed by IPC by mid-2014. The MM intervention operated at a national level and through national television channels.A cluster-randomized, nonblinded impact evaluation design was used to compare the impact of the 2 A&T intervention packages (NCT01678716). A cross-sectional household survey was conducted at baseline (2010) and exactly 4 y later 2014) in the same communities in households with children 0–47.9 mo of age. We previously reported program impacts on exclusive breastfeeding rates in children 0–5.9 mo of age  in the s. For theSample size calculations were carried out to detect differences in the primary outcomes, i.e., CF practices in children 6–23.9 mo of age and stunting in children 24–47.9 mo of age, between the 2 intervention groups at end line, considering an α of 0.05, a power of 0.80, an intraclass correlation of 0.01 , and an estimated baseline prevalence of the primary outcomes. We hypothesized that the intensive intervention would favor the primary impact indicators; therefore, a 1-sided test was used.Assuming a baseline prevalence of 44% for minimum dietary diversity, we estimated that a total sample of 980 infants aged 6–23.9 mo (490/group) was sufficient to detect a ≥10-pp difference in the proportion of children achieving minimum dietary diversity at end line. In addition, a total sample of 2020 children aged 24–47.9 mo (1010/group) was sufficient to detect a ≥8-pp difference in the proportion of stunted children at end line, assuming a baseline prevalence of 50%. Before we conducted the end line survey, we reverified our detectable effect sizes on the basis of the original sample size, the observed baseline prevalence values of our primary impact indicators, and the intraclass correlation from the baseline survey. On the basis of these variables, our detectable effect size increased from 10 to 14 pp for minimum dietary diversity, and increased from 8 to 10 pp for stunting.One hundred subdistricts across 5 of 6 divisions were selected by BRAC as possible A&T intensive areas on the basis of high poverty and stunting levels and not having been included in the government National Nutrition Program. This list was narrowed to 78 on the basis of geographic proximity, size, and other operational aspects to ensure homogeneity across the sample. Within each division, 4 subdistricts were then randomly selected for inclusion in the evaluation sample with the use of a computer program, for a total of 20 subdistricts. Subdistricts within each division were then randomly assigned with the use of a computer program to either the intensive (10 subdistricts) or nonintensive (10 subdistricts) intervention. The randomization process was carried out in the presence of BRAC and A&T staff and the program evaluators at the BRAC headquarters in Dhaka. Households within the intensive and nonintensive areas were not explicitly made aware of the results of the randomization. In addition, there was no blinding of the intervention at the level of service delivery.1) minimum dietary diversity (defined as the consumption by children of foods from ≥4 of 7 food groups in the previous 24 h); 2) minimum meal frequency as appropriate for age and breastfeeding status; 3) minimum acceptable diet ; 4) consumption of iron-rich or iron-fortified food; and 5) timely introduction of solid, semisolid or soft foods (z score (WAZ), and weight-for-height z score (WHZ) according to 2006 WHO child growth standards . Data analysis was performed with the use of STATA 13; a statistical analysis plan was developed before end line data collection and discussed with the funder and program implementers.Baseline differences between the 2 interventions were tested with the use of linear regression models (for continuous variables) or logit regression models  while accounting for geographic clustering . For impApproval for the study was obtained from the institutional review board at the International Food Policy Research Institute and the Bangladesh Medical Research Council. All mothers of study children were provided with detailed information about the study in writing and verbally at recruitment. Verbal informed consent was obtained from mothers in the survey.No evaluation clusters were lost to follow-up  in both the intensive and nonintensive groups .We found no evidence of a social desirability bias for any of the CF practices. In both the intensive and nonintensive groups, as social desirability scores increased, there was no commensurate increase in reported CF practices for any indicator (P = 0.817) or in the fully adjusted model (P = 0.784). A similar pattern was observed for the proportion of children classified as being underweight and wasted. In children 6–23.9 mo of age, nondifferential patterns also were observed .Stunting declined significantly in children 24–47.9 mo of age in both groups between baseline and end line, by 5.2 pp in the nonintensive and 6.3 pp in the intensive group  at scale had a substantial and significant impact on several CF practices in comparison with changes observed with a less intensive behavior change intervention in Bangladesh. Although improvements in child growth were observed in both groups and for all age groups over time, the DDEs for linear growth and stunting at 24–47.9 mo were not statistically significant; hence, we cannot attribute improvements to the A&T intensified interventions. In contrast to previous studies that have examined impacts on feeding behavior and/or growth in relatively controlled settings , this study documents significant improvements in a large-scale program. The program was delivered over 4 y, reaching nearly 2 million Bangladeshi families in 50 subdistricts. The MM intervention was delivered nationwide via national television channels; the IPC and CM interventions in the intensive areas first were implemented in 50 of 493 rural subdistricts, and later integrated with BRAC’s other health service platforms. Furthermore, the program engaged with the national government’s program by forming strategic partnerships with the Institute of Public Health Nutrition under the Ministry of Health and Family Welfare.1) Compared with the nonintensive program, did the A&T intensive program improve CF practices? 2) Were there differential improvements in child anthropometric indicators? 3) What supports scale-up? We discuss each of these 3 questions below.This study addressed 3 key questions. Our findings indicate significant and large impacts on practices related to CF. The plausibility of the impacts on CF practices are supported by our findings on the reach of the interventions, which showed that interventions were delivered with quality  and reac1) the overall trend in improvement in anthropometric outcomes in Bangladesh, 2) constraints on linear growth not addressed by the interventions, 3) the inability to adhere completely to CF recommendations on a routine basis, or 4) illness and/or poorer growth trajectories because of low birth weight and other factors not measured in this evaluation.Our findings indicate that a differential change in anthropometric measurements was not seen between the intervention areas. This lack of a differential change in anthropometric outcomes between intensive and nonintensive areas over time could be explained by First, the trend in improvement in the nutritional status of children in Bangladesh is fairly strong (1.5 pp/y) . A diffeSecond, the intensive interventions tested in this program focused exclusively on behavior change with the use of multiple platforms to delivery CF education and support. No additional interventions were provided, which could have addressed other causes of constrained linear growth . In foodThird, although there were differential improvements in several CF practices, at end line, only one-half of all children consumed what is considered to be a minimally acceptable diet and less than three-quarters consumed a minimally diverse diet. Therefore, the levels achieved may not have been sufficient to translate into improved growth.Fourth, more than one-half of children reported ≥1 common childhood illness at baseline, reflective of an overall high pattern of child morbidity in this population, which might have contributed to limited impacts on linear growth.Lessons learned for programming to improve CF practices at scale are worth noting in the context of this program evaluation. First, the behavior change interventions tested were developed through formative research for program design; their roll-out and scale up were done through monitoring of quality and coverage; the intervention delivery also included several supervision and management approaches that supported implementation , 27. FroSome limitations of our program evaluation include the following. First, the evaluation areas comprised a smaller geographic area than the total coverage of the program; in a midline evaluation, however, we randomly assessed service delivery in a subset of intensive areas not included in the impact evaluation areas and found no evidence of differential intervention coverage, exposure, or household practices, suggesting limited evaluation bias. Second, in this large-scale programmatic setting, for practical considerations, a cross-sectional evaluation design that sampled children on the basis of the potential of having been exposed to, and benefited from, the intervention was used, rather than tracking individual children. This design precluded our ability to fully link individual child-level exposure to program interventions to growth outcomes for the same children. Third, the intervention was not blinded to implementers or community members, and measurement of CF practices was based on maternal recall. These factors could lead to social desirability in reporting. We found no intervention-specific differentials in socially desirable reporting for CF practices, however . ReporteIn conclusion, CF is recognized as an important contributor to poor child nutrition and child development globally, but progress is held back  because As the global momentum for investing in nutrition ramps up, there is an urgent need for demonstrated large-scale solutions for improving the most fundamental of nutrition actions, i.e., home-based behaviors to improve the quality of diets. This study offers compelling evidence that such interventions can be implemented at scale to deliver impact on what remains a substantial global challenge—improving children’s diets.


To help adolescents with greater psychosocial needs, the Tier 2 Program of the Project P.A.T.H.S.  was designed and implemented by school social workers and teachers. Based on subjective outcome evaluation data collected from the program participants  in 49 schools, program implementers were invited to write down five conclusions based on an integration of the evaluation findings. With reference to 245 conclusions included in the 49 evaluation reports, secondary data analyses showed that most of the conclusions concerning perceptions of the Tier 2 Program, instructors, and program effectiveness were positive. In addition, difficulties encountered and recommendations for program improvement were highlighted. In conjunction with previous evaluation findings, the present study suggests that the Tier 2 Program was well received and was perceived to be beneficial to the development of adolescents with greater psychosocial needs.


EuroFIT is a gender-sensitised, health and lifestyle program targeting physical activity, sedentary time and dietary behaviours in men. The delivery of the program in football clubs, led by the clubs’ community coaches, is designed to both attract and engage men in lifestyle change through an interest in football or loyalty to the club they support. The EuroFIT program will be evaluated in a multicentre pragmatic randomised controlled trial (RCT), for which ~1000 overweight men, aged 30–65 years, will be recruited in 15 top professional football clubs in the Netherlands, Norway, Portugal and the UK. The process evaluation is designed to investigate how implementation within the RCT is achieved in the various football clubs and countries and the processes through which EuroFIT affects outcomes.n = 500); attendance sheets and coach logs (n = 360); observations of sessions (n = 30); coach questionnaires (n = 30); usage logs from a novel device for self-monitoring physical activity and non-sedentary behaviour (SitFIT); an app-based game to promote social support for physical activity outside program sessions (MatchFIT); interviews with coaches (n = 15); football club representatives (n = 15); and focus groups with participants (n = 30). Written standard operating procedures are used to ensure quality and consistency in data collection and analysis across the participating countries. Data will be analysed thematically within datasets and overall synthesis of findings will address the processes through which the program is implemented in various countries and clubs and through which it affects outcomes, with careful attention to the context of the football club.This mixed methods evaluation is guided by the Medical Research Council (MRC) guidance for conducting process evaluations of complex interventions. Data will be collected in the intervention arm of the EuroFIT trial through: participant questionnaires (The process evaluation will provide a comprehensive account of what was necessary to implement the EuroFIT program in professional football clubs within a trial setting and how outcomes were affected by the program. This will allow us to re-appraise the program’s conceptual base, optimise the program for post-trial implementation and roll out, and offer suggestions for the development and implementation of future initiatives to promote health and wellbeing through professional sports clubs.ISRCTN81935608. Registered on 16 June 2015. Robust evidence on whether interventions to improve public health are effective is essential. However, as many authors have argued, it is equally important to understand why particular interventions succeed or fail, under which circumstances and whether their effects can be reproduced so that better programs can be developed and implemented more widely –5. A proIn addition, programs which have been shown to be effective in a trial can be hard to implement more widely or can fail to show effects outside of the specific context of the trial, in part because of the way in which they are developed or implemented –12, partIn this paper, we present the protocol for a process evaluation of the implementation of the EuroFIT health and lifestyle program embedded within a pragmatic RCT . The oveEuroFIT is an evidence-based and theory-based, gender-sensitised, health and lifestyle program targeting physical activity, sedentary time and dietary behaviours in men. EuroFIT is informed by the Football Fans in Training (FFIT) program, which was developed and delivered in Scotland , 21. TheThe EuroFIT program shifts focus from weight management (FFIT) to increasing physical activity and reducing sedentary time. To support this change in emphasis, participants are provided with a SitFIT™ device and access to a game called MatchFIT, both of which have been developed as part of the EuroFIT project. The SitFIT™ is a self-monitoring device that allows real-time self-monitoring not only of physical activity (through step counts), but also of sedentary behaviour (sitting time) and non-sedentary behaviour (upright time). MatchFIT is an app-based game designed to encourage social support around physical activity between sessions and after the end of the program. Both EuroFIT and FFIT aim to support participants in their improving diet.The effectiveness of EuroFIT will be evaluated in a multicentre pragmatic trial in 15 top professional football clubs across four European countries  . For thiThe EuroFIT program consists of 12 weekly, 90-min group sessions of 15–20 men delivered at professional football clubs by club community coaching staff. Each of the 15 EuroFIT clubs commit to deliver two EuroFIT groups. EuroFIT also includes one reunion session which is held at the club 6–9 months after the start of the program (the precise timing is determined by the coaches delivering the program). In the outcomes analysis, we will follow an intention-to-treat procedure . For theThe EuroFIT program has multiple interacting components , 6, 28. Context: An examination of how the broader cultural context of the country and the specific cultural context of the football club into which EuroFIT is introduced, influences and interacts with the delivery and functioning of the program components.Implementation of the EuroFIT program: An examination of how EuroFIT delivery is achieved and what is actually delivered. We will describe the structures, resources and processes through which EuroFIT delivery is achieved, the extent to which EuroFIT was delivered as intended, any adaptations made to the program and the sociodemographic characteristics of the participants recruited to the program.Mechanisms of impact: An examination of the processes through which the program affects outcomes through understanding how participants and coaches respond to and interact with EuroFIT and how the intervention supports change (or not).For the purposes of the process evaluation we describe context, implementation and mechanisms of impact as follows :ContextWe have structured the 18 ROs for this process evaluation around these three domains: Table This mixed methods process evaluation is embedded in the main EuroFIT trial, which is being conducted in the Netherlands, Norway, Portugal and the UK . The proEthical approval for the process evaluation has been obtained from the appropriate country-specific ethics committees .Quantitative data will be collected through participant questionnaires in the intervention arm of the trial (n = 500), coach questionnaires (n = 30), attendance sheets and coach logs of sessions delivered (n = 360) and participants’ logs from the SitFIT and the game-based app MatchFIT. To minimise the respondent burden, questions relevant to the process evaluation will be added to participant questionnaires (for intervention group members only) which collect trial outcome data. The Robertson Centre for Biostatistics (RCB) within the University of Glasgow provides data management in support of delivering the trial . All triThe baseline participant questionnaire will gather information on the characteristics of the participants who were recruited to EuroFIT , how they heard about the program and why they joined . The post-program participant questionnaire (intervention group only) will contain questions about attendance at sessions, experiences of the program, perceived competence of the coaches delivering the program and the environment they created, and the extent to which participants used the activities and tools from EuroFIT while taking part in the program (ROs 5 and 13). The 12-month participant questionnaire will contain questions for the intervention group on the extent to which participants remain in contact with the other men and the coaches who took part in the EuroFIT program at their club and on the extent to which  tools from the EuroFIT program are still being used after the program has ended (RO 15).If participants withdraw from the trial (measurements) or drop out of the program, their reason for leaving will be assessed via structured telephone interviews (RO 9).Data are collected remotely from users of the SitFIT and MatchFIT and will include logs of data uploads, error reports, user clicks, logins and logouts (RO 5). Each item of logged data includes a timestamp, information about the web browser and device type being used, and (except for pre-registration and pre-login activities in MatchFIT) a unique SitFIT identifier. As part of the post-processing of the data, the SitFIT identifier will be linked to the participants’ ID, usernames will be replaced with a pseudonym and the text of the personal annotations in MatchFIT will be replaced with an integer representing how much text was written.Club coaches will be asked to keep an online record of attendance at each of the 12 weekly sessions and the reunion session, to be completed as soon after each session as practicable (RO 5). The program will be delivered to two groups of approximately 15–20 men in each of the 15 participating football clubs. During the course of the trial, a total of 360 EuroFIT sessions will take place. In addition, coaches will be asked to complete an online log to assess their overall view of each session, to record the time they spent on preparation and delivery of the session, and to record the extent to which they consider that intervention components were delivered as intended or not (self-report). In addition, coaches will be asked to provide feedback on the session through a set of open ended questions in terms of what worked well and what did not work well .An evaluation questionnaire will be administered to coaches after the two-day intervention training (RO 3). After the last program session has been delivered, all coaches will be asked to complete an online questionnaire . This questionnaire contains an overall evaluation of the program and questions asking how often coaches used the activities and tools that are specific to the program.Observations of sessions, interviews and focus group discussions will be conducted. SOPs have been written for the qualitative data collection and analysis, based on templates provided by the ACT consortium , 30 and Semi-structured interviews will be conducted with club community managers and club community coaches at the participating football clubs . The purpose of the interviews is to find out what coaches and club representatives think of the program, their experiences of what worked or did not work with regard to getting the program up and running, barriers and facilitators for implementation, recruiting men for the program, delivering the program from week to week, whether or not they see a future for the program in the club and why, and what would be needed in the future to ensure that they continue with the program. The interviews will explicitly aim to discuss barriers and facilitators for implementation as well as capture contextual issues (such as availability of facilities) that may have shaped the delivery of the program. Club community managers and coaches will be eligible to participate in an interview if they have been personally involved in the recruitment, delivery or organisation of sessions of the EuroFIT program at their club. We plan to complete 15 interviews with club community managers and 15 interviews with coaches who delivered the program in the participating football clubs . This sampling strategy enables us to obtain views and actual experiences from each participating club across the four countries. Interviews will be conducted by a trained member of the process evaluation team not directly involved in the training of the coaches and who is based in the same country as the football club. A topic guide has been developed, will be piloted in the first few interviews and will be adapted, through discussion with the process evaluation subgroup, where necessary. Interviews will be audio-recorded with permission from the interviewee. The researcher will also take brief notes during the interviews.Researchers will observe a sample of deliveries of the EuroFIT program at each participating club to obtain data on how men interact with one another, how they respond to different elements of the program, how they interact with the coaches that lead the sessions and any features of the interaction that might influence program effects . We have chosen not to observe sessions 1 and 2, in order to allow groups to ‘form’ without adding the anxiety of being observed by researchers . We willWe will conduct focus groups with EuroFIT participants at each football club. The purpose of the focus groups is to find out what men think of the EuroFIT program, any impacts it has had on their lives, which elements of the program they viewed as helpful and unhelpful in supporting them to make changes and any suggested changes to the program . The first focus groups will take place after the 12-week program has ended and after the post-program measurements for the trial have been completed. In these post-program focus groups, we will be particularly interested in participants’ reasons for joining and continuing with the EuroFIT program; their views and experiences of the program and materials; any impacts they feel that taking part has had on their lives (including changes they have made to their behaviours); and which elements of the program were viewed as helpful and unhelpful in supporting them to make changes.The second set of focus groups will be conducted after the 12-month trial measurements have been completed and aim to understand men’s experiences after 12 months of attempting to make and maintain (or not) any changes made as a result of participating in the EuroFIT program; what aspects of the program were helpful/less helpful for supporting long-term change; what barriers and facilitators they have experienced to making/maintaining change; the extent to which the changes have become routinised in their daily lives and integrated with their (reformulated) identities; and the responses of their family, social and work colleagues to their changed behaviours, appearance and wellbeing.The session attendance sheets compiled by the EuroFIT coaches will be used to sample participants for the post-programme focus groups. To be invited for a focus group, men must have attended at least half of the 12 program sessions, because we think it is important that the focus group participants can share their views based on substantial experiences of the program. For each focus group, we aim to recruit 6–8 participants, drawn from both delivery groups, who are of varied ages.Each focus group will take place at the football club and will be conducted by a moderator, who will be fully trained to ensure consistency across countries. Where possible, the moderator will be assisted by a note-taker . An audio-recording will be made, with participants’ permission. Men will receive a €20/£20 voucher in return for their participation.Elements of this process evaluation were tested in one football club in the UK which offered a pilot version of the EuroFIT program in 2015. We conducted observations of this delivery of the 12 EuroFIT program sessions and used a standardised observation proforma. We also piloted the format of the coach log data collection instrument. The participants and coaches who took part all found the methods acceptable. Based on this study, we made small adjustments to the coach logs and the observation proforma.Descriptive statistics  will be used to report participants’ and coaches’ characteristics and results of pre-structured questions from the questionnaires. Data from the attendance sheets and coach logs will be summarised using this method. Descriptive summaries and charts will be produced to present attendance, sessions delivered and key components delivered, to help identify patterns of incomplete delivery. In addition, estimates of the total program hours received by participants will be computed per club using data from the attendance sheets. All reported (suggestions and reasons for) adaptations to the program and any other answers to open ended questions will be listed, analysed and summarised. The results will be translated into English by a local member of the process evaluation team and reported back in English to the process evaluation lead. Descriptive statistics and visualisations will also be used to explore and report the use of the SitFIT and MatchFIT. The key performance metrics are uptake and retention rate (measured in terms of weekly active users). The use of individual features of MatchFIT will also be explored.All note-taking during observations, and all interviews and focus groups, will be conducted in the local language; all data will be transcribed in the local language of the club delivering the program . TranscrA structured thematic, framework approach will be used to analyse the qualitative data. The analysis will consist of five steps , with eaWithin each country, three files  will be read line by line by a member of the process evaluation team with expertise in qualitative research. For each line or subsection that carries meaning in relation to the ROs or unanticipated issues, the text will be highlighted and assigned a code. All the codes will be described in English. In each country, a second independent coder will read and check the codes. Any disagreement will be resolved through discussion; if necessary, a third local researcher will be consulted to aid resolution of any disagreement in the interpretation of the coding.A face-to-face meeting of the process evaluation team will be organised, during which the initial local codes, derived from the three selected files, will be shared and discussed. A researcher from the consortium who is expert in qualitative methods, and not directly involved in data collection and coding, will chair this meeting. During this meeting, we will create an affinity diagram where we will separately record each suggested code on a post-it note, assemble similar suggested codes into groups, discard duplicate codes and assign groups of similar codes an adequate descriptive label . During The coding of the first three files will then be adjusted according to the consolidated codebook. The remaining files will also be coded against the consolidated codebook. Regular conference calls, email and, if necessary, face-to-face meetings will be organised to adjust and refine the codebook. When all the data have been coded, each country will provide the process evaluation lead with a report in English that summarises the key findings for each RO, following a standard template.In dialogue with the process evaluation team, the process evaluation lead will analyse the country specific reports and write overall findings paying attention to any cultural and other differences in sub-groups and countries.A draft set of findings and conclusions will be circulated to the process evaluation team and the consortium for feedback.Once qualitative and quantitate analyses are complete, we will compare and integrate findings from different data sources to answer the 18 ROs. The integration will be guided by the ‘triangulation protocol’, which suggests researchers assess degrees of agreement and dissonance across the datasets and also to identify areas of ‘silence’, i.e. where a given dataset has nothing to contribute , 37. A cFollowing Moore et al.’s model, the finalised convergence matrix will be used to establish the influence of context, implementation and mechanisms of action on the outcomes of EuroFIT. The finalised convergence matrix will also be used to assess the extent to which the logic model, and the causal assumptions it contains, can explain the outcomes the program produced. The analysis will guide us to the key mechanisms that are essential for program success and the parts of the program that are difficult to implement (or even implementation failures). The synthesis of findings from the different data sources is an important step in interpreting program outcomes, understanding the context in which the program is delivered and identifying the mechanisms of impact or implementation difficulties.This paper describes the design of a mixed methods process evaluation embedded within a multicentre pragmatic trial, for which 1000 overweight men, aged 30–65 years, will be recruited in 15 top professional football clubs across the Netherlands, Norway, Portugal and the UK. EuroFIT is an evidence-based and theory-based, gender-sensitised, health and lifestyle program that aims to attract men who wish to make changes to their physical activity, sedentary behaviour and diet, through an interest in football and their personal connections and loyalties to the football club they attend. The process evaluation is designed to investigate how the implementation is achieved in the various football clubs and countries and the processes through which the EuroFIT program affects outcomes. By publishing the protocol for this process evaluation, we make the methodological choices that we have made explicit.Professional sporting venues, and in particular professional football clubs, are increasingly seen as promising sites for the delivery of health promotion programs because of their potential to reach people who might not otherwise consider taking part in a lifestyle program , 38, 39.While we have tried to be comprehensive in addressing all issues utilising different methods for data collection, we have had to make some compromises due to time and resource constraints. The number of observations of deliveries had to be kept to a maximum of two per club (30 sessions out of a possible 360). Observing more deliveries might enable a more nuanced understanding of how each component of the program is delivered and following one delivery group over the full 12-week program could also offer more insight into the process of group formation and group dynamics. The ongoing contact between researchers and the delivery of the intervention, through observations conducted during the trial, will allow researchers to be responsive and to construct topic guides for coach and club interviews and participant focus groups that reflect areas of interest that emerge. As described above, the ongoing contact will be mostly passive, as observers will work in a non-participative manner to limit interference with the program delivery during the trial.Apart from the 30/360 sessions we are able to observe, we rely on the coaches to self-assess the ‘fidelity’ with which they deliver their weekly sessions. Although self-report presents some pragmatic advantages in terms of simplicity, convenience and affordability, we are well aware of the risk of social desirability bias. For instance, coaches may be reluctant to say that they did not deliver certain intervention components or they might not have the skills to rate their own competence . The tesDespite these limitations, a balanced design for the process evaluation of EuroFIT in the context of a RCT has been achieved by following the MRC guidance on the conduct of process evaluations . We ther


Graduate Medical Education (GME) is the portion of the medical education continuum that spans the period following graduation from medical school to independent practice.  In the United States, successful completion of GME is essential for board certification in one of over 140 recognized specialties. The Accreditation Council for Graduate Medical Education (ACGME) is the accreditor for over 10,000 programs. Each program is required to have a single physician program director responsible for the program’s administrative and educational oversight. Virtually all program directors remain clinically and academically productive. Over the last 17 years, the ACGME has transitioned its accreditation emphasis to promote competency based education. Furthermore, the ACGME requires programs to provide faculty development in teaching and assessment for their faculty and to evaluate the program's performance in providing it annually.Furthermore, faculty attrition has been linked to inadequate faculty development.We used situational learning theory and best practices from communities of practice, to develop implement, and evaluate an institutional faculty development program for over 100 GME program directors of both ACGME accredited and internally sponsored programs. The purpose of this Perspective is to present our eight-year experience.The faculty development programWorkshop topics were identified through a literature review, an online survey of program directors soliciting the major gaps in their educational competence and confidence, an analysis of the citations programs had received for deficiencies from the ACGME and a review of the major changes in ACGME requirements.  Each topic was developed into an hour-long workshop.  Six to 8 topics were offered each spring and fall.  These topics were each presented three different times and days during a single week to maximize program director attendance.  Sessions were recorded using the technology available through the institution. Presentation materials, including practical tools and resources, were compiled into a toolkit and distributed at each workshop for program directors to use within their own programs. These materials were also posted to a password protected website.  A certificate of exemplary attendance was sent to the Departmental Chairs of participants who attended 10 or more sessions over the course of an academic year. Evaluations were designed to measure participant satisfaction and elicit feedback for improving future workshops and identify timely topics. We used Kirkpatrick’s four levels of evaluation to evaluate the program.Over the course of eight academic years, from fall 2007 through fall 2014, 97 different workshops were presented to over eleven hundred participants. An archived library was created by uploading recordings of the workshops, presentation slides and resource toolkits. The workshops were categorized by themes which ranged from “ACGME Competencies” to “Technology & Social Media in Teaching.” Workshops were co-taught by internal experts from across the institution.There were eleven hundred and twenty session attendees, consisting of two hundred and four unique individuals who attended at least one session. Forty percent of program directors attended sessions each year and represented twelve of the then thirteen clinical departments. Sixty-eight percent planned to use the electronic tool kits. Participants regularly attended multiple different sessions based on their availability and their specific needs for the content being offered. Session attendance fluctuated with the time of day and year. Attendance was best for sessions held at noon (546 attendees).Four hundred and seventy evaluations (42%) were returned over the course of the series. The aggregate response results demonstrated participants found the logistics, content and networking opportunities provided by the sessions to be “extremely helpful”.  The cost was modest.We demonstrated a high rate of participation and satisfaction with a voluntary institutional faculty development activity for program directors and individuals with major GME roles. Workshops covered critical content identified by participants and through gaps noted by the accrediting body. The format was convenient, practical, and participative. Materials were provided that could be put to immediate use. Because there was no one day and time that accommodated the majority of attendees, we offered both live and enduring formats. Following Kirkpatrick’s four levels of evaluation, the evaluation data demonstrate learner satisfaction (level 1).  While more difficult to measure, the authors believe workshop participants learned (level 2) and changed their behavior (level 3), potentially leading to positive results (level 4).We believe there was increased program director satisfaction, greater networking, enhanced program quality and improved perception of the value added of the institutional GME office beyond administrative functions. The success and feasibility of this model of faculty development helped lead to an institution-wide monthly Medical Education Grand Rounds, a newly implemented Faculty Teaching Academy, and greater collaboration supporting medical education research and scholarship.We believe the networking opportunities provided by these workshops helped create a community of educators committed to continual improvement of themselves, their programs, one another, and the institution. We believe other medical educators can easily adapt our model of faculty development to their own institutions.The authors declare that they have no conflict of interest.


CML‐BC) from de novo BCR‐ABL1‐positive acute myeloid leukemia (AML) is a diagnostic challenge with therapeutic consequences. In our case, a basophilia, the presence of the Philadelphia chromosome in all metaphases and the strict exclusion of molecular hallmarks of AML lead us to retain the diagnosis of CML‐BC rather than BCR‐ABL1+ AML.Differentiating chronic myelogenous leukemia in primary blast crisis ( BCR‐ABL1‐positive AML is a rare disease recently recognized as a provisional entity in the 2016 revision of the World Health Organization (WHO) classification of myeloid neoplasms De novo 9/L), and a leukocytosis (42.2 × 109/L) with a basophilia (8.4 × 109/L), 39% of blasts, and 3% of myelocytes. Bone marrow studies showed a hypercellular marrow with 38% of blasts with deep purple granules in the May–Grunwald–Giemsa stain , a thrombocytopenia 73 × 1009/L, andBCR‐ABL1 transcript without mutations in the ABL1‐kinase domain. The detection of this recurrent genenetic abnormality according WHO 2016 classification excludes the diagnostic of AML with myelodysplasia‐related change. Thus, the diagnostics of de novo BCR‐ABL1‐positive AML or CML in primary blast crisis have to be discussed. The High‐Throughput Sequencing  did not evidence any mutations in genes classically associated with de novo AML or myelodysplastic syndromes . No deletion in IKZF1 or CNKN2A/2B genes was evidenced with Multiplex Ligation Probe Assay . According to the cytological and molecular pattern of the disease, we finally retained the diagnosis of CML in primary blast crisis rather than de novo BCR‐ABL1+ AML.Cytogenetic analysis identified the Philadelphia chromosome in 100% of analyzed metaphases with additional chromosomic abnormalities: 46,XX,add(7)(q35),t[5],47,idem,+8[5]. Molecular analysis revealed a p210 2/day SC, 7 days a month) in combination with dasatinib. After 3 months of treatment, the patient reached a complete cytogenetic response. The BCR‐ABL1 transcript level remained stable at 0.25% IS with a follow‐up of 9 month since the initial presentation. The tolerance was acceptable, and the complete blood count showed a hemoglobin level at 11.8 g/dL, 222 × 109/L platelets, and 2.2 × 109/L neutrophils. The opportunity of allogenic stem cells transplant is actually discussed.The patient received an induction therapy with idarubicin, cytarabine and lomustine in combination with a second‐generation tyrosine kinase inhibitor dasatinib (140 mg/day). Bone marrow evaluation at day 30 evidenced a partial response (9% of blasts) according to recent ELN2017 guidelines BCR‐ABL1+ AML is a diagnostic dilemma, particularly if in the absence of previous history of leukocytosis or splenomegaly. Neuendorff et al. recently reported an algorithm based on cytological and molecular features Blastic crisis remains the main therapeutic challenge in CML, even in the tyrosine kinase inhibitors area. Usually, blast crisis arises in patients with known CML, but it can be the initial presentation of the disease. In this case, the distinction between CML‐BC and de novo BCR‐ABL1 leukemias. In our case, the bone marrow blasts also showed basophilic granules in their cytoplasm. This pattern of acute basophilic leukemia is very rare. It has been reported in association with t, 12p rearrangements or t, the latter consisting in the blastic phase of CML in the literature BCR‐ABL1 AML. On the other hand, myelodysplastic changes are most often associated with de novo BCR‐ABL1 AML Cytological examination can provide a diagnostic orientation and guide the molecular work‐up. Indeed, the increase in basophils is strongly associated with the diagnosis of CML on with t;9, 12p rBCR‐ABL1+ AML can be explained by their distinct physiopathology. Indeed, BCR‐ABL1 is the main driver event in CML, sufficient to initiate the disease. Conversely, applying the two‐hit leukemogenesis model, BCR‐ABL1 consists in a secondary event in AML conferring a proliferative advantage to the cell (class I mutations). It requires an initiating event affecting the myeloid differentiation (class II mutations) to produce the AML phenotype. Consequently, the presence of the Philadelphia chromosome in less than 100% of metaphases during karyotype is the major criterion for the diagnosis of BCR‐ABL1+ AML BCR‐ABL1+ AML displays characteristics of lymphoid malignancies as blasts frequently express lymphoid markers and harbor deletions of IKZF1 or CDKN2A/2B genes BCR‐ABL1+ AML.The main genetic differences between CML‐BC and BCR‐ABL1+ AML was initially proposed Patients with CML in primary blast crisis should be treated with a tyrosine kinase inhibitor with or without chemotherapy, with the goal of obtaining the best response and proceeding to allogenic stem cell transplantation as quickly as possible BCR‐ABL1‐positive AML. In absence of splenomegaly or previous history of leukocytosis, it requires a careful cytological examination and extensive molecular investigations.The diagnosis of CML in primary blast crisis is rare but needs to be systematically excluded before retaining the diagnosis of de novo CP: analyzed the cytological and molecular data and wrote the manuscript and subsequent revisions; RH: manage the patient and critically reviewed the manuscript. All authors interpreted the data and take the responsibility for the accuracy of the case presentation.No conflict of interest to report.


Purpose Improving pregnancy outcomes for women and children is one of the nation’s top priorities. The Healthy Start (HS) program was created to address factors that contribute to high infant mortality rates (IMRs) and persistent disparities in IMRs. The program began in 1991 and was transformed in 2014 to apply lessons from emerging research, past evaluation findings, and expert recommendations. To understand the implementation and impact of the transformed program, there is a need for a robust and comprehensive evaluation. Description The national HS evaluation will include an implementation evaluation, which will describe program components that affect outcomes; a utilization evaluation, which will examine the characteristics of women and infants who did and did not utilize the program; and an outcome evaluation, which will assess the program’s effectiveness with regard to producing expected outcomes among the target population. Data sources include the National HS Program Survey, a HS participant survey, and individual-level program data linked to vital records and the Pregnancy Risk Assessment Monitoring System (PRAMS) survey. Assessment Descriptive analyses will be used to examine differences in risk profiles between participants and non-participants, as well as to calculate penetration rates for high-risk women in respective service areas. Multivariable analyses will be used to determine the impact of the program on key outcomes and will explore variation by dose, type of services received, and grantee characteristics. Conclusion Evaluation findings are expected to inform program decisions and direction, including identification of effective program components that can be spread and scaled. The HS program, which began in 1991, was transformed in 2014. To understand the implementation, utilization and overall impact of the transformed HS program, a national program evaluation is being conducted. This evaluation strives to address the challenges and limitations noted in prior evaluations, most importantly, the lack of an appropriate comparison group, which is essential to determine program impact.Improving pregnancy outcomes for women and children is one of the nation’s top priorities. The infant mortality rate (IMR) is a widely used indicator of the nation’s health. In 2013, the U.S. IMR was 5.96 infant deaths per 1000 live births. However, racial-ethnic disparities persist and in the same year, the IMR for infants born to non-Hispanic black mothers was 11.11, more than double the non-Hispanic white IMR of 5.06  improve women’s health; (2) promote quality service; (3) strengthen family resilience; (4) achieve collective impact; and (5) increase accountability through quality improvement, performance monitoring, and evaluation.Each HS grantee is required to address the five approaches, although they may engage in a diversity of activities within the five approaches. The evaluation has been designed to account for the diversity of HS grantees and activities and will involve documentation of the implementation of the transformed HS program components  and their alignment with the five HS approaches. This information will be used to examine factors that help explain effective implementation of the transformed HS program.The five approaches set the framework for the transformed HS program, which is depicted in the program logic model Fig. . The traWith the program’s transformation, there is a need to assess its implementation and understand the overall impact of the program using a robust and comprehensive evaluation design. Prior evaluations of HS  sought the input of an external committee to guide the design and implementation of the national HS evaluation. In October 2014, a Technical Expert Panel (TEP) of maternal and child health researchers, practitioners, project directors and policy stakeholders was convened to discuss and recommend an evaluation design for the transformed HS program. The TEP will serve as an external consultative committee and provide direction on the design and implementation of the evaluation. The evaluation management team will meet quarterly with the TEP to continue to obtain their input and recommendations for the evaluation design, progress and findings, and the final report.The overarching goal of the national HS evaluation is to determine the effect of the transformed program on participant-level characteristics . It includes three components: (1) implementation; (2) utilization; and (3) outcome. The purpose of the implementation evaluation is to describe HS programs and strategies and to identify program factors that are associated with effective implementation. The purpose of the utilization evaluation is to examine the characteristics of participants and non-participants and factors that help explain differential penetration, or service rates. The purpose of the outcome evaluation is to assess the overall effectiveness of the program with regard to producing expected outcomes among the target population and factors that help explain variation in the program’s impact on individual level outcomes. The outcome evaluation will employ propensity-score methods, which will include two types of comparisons:A matched individual comparison analysis of linked vital records for HS participants and non-participants in the same general geographic service area for all 100 HS grantees, which maximizes generalizability and will allow for assessment of the key outcome of interest, infant mortality, with adequate statistical power.A matched individual comparison analysis of HS participants and non-participants by oversampling of the Pregnancy Risk Assessment Monitoring System (PRAMS) survey for a random sample of 15 HS grantees. This component of the evaluation data collection strategy will maximize internal validity with a broader set of outcomes and control for matching characteristics that can influence selection into the program.The primary data source for the implementation evaluation is the National Healthy Start Program Survey (NHSPS). The utilization and outcome evaluations will link state/jurisdiction vital records , the Centers for Disease Control and Prevention (CDC) PRAMS survey, and participant-level program data to compare HS participant and non-participant characteristics and outcomes Fig. . HS partBenchmarking methods will be used to contextualize HS outcomes by comparing individual level outcomes and personal, clinical and socio-demographic risk factors among HS participants to data available from other sources or benchmarks. An attempt will be made to use a low-income reference group for national sources. HS communities are selected based on demonstrated need and thus, will likely demonstrate poorer outcomes than the average community in the United States. HS programs demonstrating progress toward or in reaching the national average will represent an accomplishment but the benchmarking method is unlikely to allow for attribution of any differences to HS program effects.To link HS participants to vital records data, all 100 HS grantees will collect individual identifiers from eligible program participants Table . GranteeThe National Association of Public Health Statistics and Information Systems (NAPHSIS) will develop a model data sharing/transfer agreement to be adapted and signed for each HS grantee, VRO, PRAMS program and MCHB. This evaluation will be conducted in accord with prevailing ethical principles and program evaluation standards, and is being reviewed by an Institutional Review Board. All participants will provide informed consent prior to their inclusion in the evaluation study.In addition to linking HS participants to vital records, the evaluation will also link participants to PRAMS survey data. The HS program currently has 86 grantees located in states that conduct the PRAMS survey. To improve the chances of evaluating an operational HS program early in the grant cycle, the PRAMS oversampling was restricted to continuing grantees . Similarly, CDC recommended restricting the sample to grantees in states which currently field the PRAMS survey (n = 40) given the potential lack of capacity in new PRAMS Phase 8 states (up to 61 states/jurisdictions/tribes). Therefore, the HS Sampling Frame for the PRAMS oversampling included 63 of 75 continuing grantees that are located in current PRAMS states.Based on available funding and CDC support services, it was determined that 15 HS grantees could be selected for PRAMS oversampling. To ensure scientific integrity, the 15 HS grantees were randomly selected within strata determined to be of importance to the program. The strata include cells categorized by Grantee Level 1, 2, 3),,3.Beginning in 2017, 15 randomly selected HS grantees will send individual identifiers , and the evaluation produces estimates of the effects of HS on individual-level outcomes. A propensity score matching approach will be used to match participants and non-participants. The propensity score method uses the set of variables to compute the probability of being served by HS for each HS participant and nonparticipant using a logistic regression model. In other words, the demographic and risk factors (independent variables) are used to predict whether individuals are HS participants (dependent variable). The resulting propensity scores are the chances that each individual is a HS participant, or the predicted propensity to be a HS participant.Given the general PRAMS sample size, however, non-participants will not be restricted to the same “community” in the PRAMS matched comparison. Thus, census tract or zip code-based poverty will likely be used to control for community characteristics. In the vital records comparison, however, participants will be able to be matched to non-participants in the same general geographic area (city/county) that is served by HS . This will enhance MCHB/HRSA’s ability to draw conclusions about the effectiveness of the program in influencing the key health outcomes of the transformed program. Given that there are likely to be many more non-HS participants in vital records than HS participants, the analysis could be statistically strengthened by a 1:N  match.The implementation evaluation will use program and participant survey data to develop metrics to assess more (versus less) effective implementation of HS services. Program goals and fidelity to implementing the five HS approaches will be analyzed by assessing, for example, number of participants enrolled in health coverage and methods used for enrollment, use of standardized curricula and interventions across grantee sites, and types of prevention education models used. We aim to identify program approaches/models considered to be key to effective implementation and identify metrics for assessing performance.The analysis will also test the statistical significance of bivariate and multivariable associations between program- and organization-level factors and indicator(s) of effective implementation. Program-level factors may include the outreach strategies employed; number and types of referrals provided; the number and types of screenings provided; case management models utilized; caseloads maintained; and promotion of male involvement, among others. Organization-level factors will likely include the type of program ; the HS program level ; the lead agency type; age of the program; staffing characteristics; and the type of approaches and services provided, among others.Analysis of the utilization evaluation will include descriptive analyses of HS participants in terms of a number of individual characteristics, including socio-demographic indicators, health behaviors, utilization of non-HS health services and health outcomes. Bivariate analyses will test for statistically significant differences in health behaviors, health service utilization patterns, and health outcomes between HS and non-HS participants and among HS participants, by level of utilization of HS services. Descriptive analyses will also examine service or penetration rates by intended target characteristics  and summarize utilization levels among participants at the grantee level.The outcome evaluation analysis will estimate the effect of program participation by comparing outcomes of HS participants and non-participants using multivariable techniques. Individual-level propensity score matching  seeks to assess the transformed HS program, which was designed to apply lessons from emerging research, past evaluation findings, and the Secretary’s Advisory Committee on Infant Mortality; (2) addresses the principal limitations of previous HS evaluations by employing two types of matched individual comparison analysis and the use of standardized datasets; and (3) will help further our understanding of the “evaluability” of complex public health interventions. Further, evaluation findings are expected to inform program decisions and future program direction and to enable not only a determination of whether HS is effective in impacting participant outcomes, but why and how, so that effective program components can be spread and scaled.