Science Teachers Learning from Lesson Analysis (STeLLA) PD program, in contrast, was studied in a cluster randomized experimental design that examined impact on teaching practice and student learning. The STeLLA video-based PD (VbPD) program demonstrated significant impact, with high effect sizes, on elementary teachers’ science teaching practice and their students’ learning. Previously published reports provide details about research methods and findings but only broad sketches of the STeLLA program design and implementation. Deeper explorations of the STeLLA design principles can contribute evidence-based knowledge about the features of effective PD and enrich the existing but limited consensus model of effective PD. This article addresses the following questions:What design principles guided the development, implementation, leadership, and scaling up of a video-based PD program that had significant impact on student learning?What do the STeLLA design principles contribute to the existing knowledge base about effective video-based PD?Most studies of teacher professional development (PD) do not rigorously test impact on teaching practice and student learning. This makes it difficult to define what is truly “effective.” The Results from rigorous studies of the STeLLA program are summarized in this paper; details are reported elsewhere and included here as supplementary materials. This article is not a standard research results paper but instead describes the design principles guiding the development, implementation, leadership, and scaling up of the STeLLA VbPD program.The authors argue that this set of design principles is powerful for four reasons: 1) its demonstrated impact on teaching practice and student learning, 2) its strong theoretical and research foundations, 3) the stability and usefulness of the design principles as implemented in changing contexts over a 10-year period, and 4) the coherence and interconnectedness of the principles. The STeLLA VbPD design principles contribute to the field by empirically supporting and advancing the existing consensus model of effective PD. Further study can build on this effort to strengthen our understanding of effective PD based on evidence of impact on teaching practice and student learning. What are the key features of effective professional development? Researchers have been writing about this question for decades .The Science Teachers Learning from Lesson Analysis (STeLLA) PD program is one of a handful of programs that has been studied in a cluster randomized experimental study where the program was compared with another PD program of equal duration and where analysis included impact on teaching practice and student learning. This yearlong, video-based, analysis-of-practice PD program demonstrated significant impact, with high effect sizes, on upper elementary teachers’ science content knowledge, teachers’ pedagogical content knowledge (PCK), teachers’ science teaching practice, and, most importantly, on student learning Fig. . As a reWhat design principles guided the development, implementation, leadership, and scaling up of a video-based professional development program that had significant impacts on teaching practice and student learning?What do the STeLLA design principles contribute to the existing knowledge base about effective video-based, analysis-of-practice PD?Research reports about this STeLLA line of research provide details about research methods that explain how we know that the STeLLA video-based, analysis-of-practice program is effective, but they provide only a surface-level sketch of the program design and implementation  the substance and form of the STeLLA approach to video-based analysis of practice, (b) the implementation of video analysis in the STeLLA program, (c) the leadership of the program, and (d) the scaling of this VbPD program through the development of teacher PD leaders in a school district-university partnership. To address the second question, we begin the paper by placing this work in the context of the larger body of research on teacher professional development, focusing on the limitations of the existing “consensus model of effective PD” and the need to better understand what it takes for PD to make a difference in terms of student learning. In the discussion at the end of the paper, we consider the power of the STeLLA approach to video-based analysis of practice and ways in which the STeLLA design principles advance and deepen the field’s current understanding of the consensus model of effective PD. In the end, we hope readers will find their understanding of the STeLLA VbPD design principles to be useful in their own work in designing, implementing, sustaining, and rigorously studying the impact of VbPD approaches. In this way, the science education community can move forward toward a more refined consensus model of PD that is supported by research showing changes in teaching practice that are linked to improved student learning.(i)Focuses on specific subject matter content,(ii)Engages teachers in active learning,(iii)Is consistent with reform documents and school, district, and state policies and practice,(iv)Is of sufficient duration (both in intensity and contact hours), and(v)Involves the collective participation of teachers .While there is widespread agreement that teacher professional development is key to improving teaching and learning, there is little research that examines the impact of various PD approaches on student learning—in science or in other subject matter areas. Despite this lack of student learning evidence, there is a widely accepted consensus model of effective professional development Desimone . In thisThis consensus model was generated from studies whose research designs are weakened by heavy reliance on teacher self-report data, small sample sizes, and lack of control or comparison groups Desimone . For exaOver the past two decades, various authors have made conceptual arguments to characterize this consensus view and have written syntheses of the existing research on PD (across subject matters) . What can our experiences from this program contribute to the field that goes beyond the features identified in the consensus model of effective professional development? Since the STeLLA program demonstrated evidence of student learning, this is a good place to dig deeper—to go beyond the surface, to examine: What is going on in the STeLLA approach to video-based PD that makes it have such dramatic impact?In this article, we share the STeLLA design principles and consider ways that they might be a starting place for developing a more specific and useful model of effective VbPD. Before describing the design principles, we provide a brief summary of the research that supports our assertions of the impact of the STeLLA program.Reinvigorating Elementary Science through a Partnership with California Teachers (RESPeCT) program. Additional studies are currently studying impact of the STeLLA VbPD approach on preservice/first year elementary teachers and their students and on high school teachers and their students. In this section, we summarize findings from the two completed studies (STeLLA-I and STeLLA-II) and provide preliminary findings from the RESPeCT study which is testing the scalability and sustainability of this program. Details of the STeLLA-I and STeLLA-II studies can be found in previously published reports which are available in the supplementary materials  based at the California State Polytechnic University at Pomona (CPP) which was supported by the California Subject Matter Projects network. The goal of this network was to engage university faculty and other educators in providing high-quality, content-specific professional development for teachers across the state of California in nine subject matter areas. The CPP math and science content deepening programs were popular with regional elementary teachers and had been in existence for years.n = 16), teachers experiencing the 1-year STeLLA VbPD lesson analysis program (n = 32) developed deeper science content knowledge (p < .001) and stronger abilities to use PCK to analyze science-teaching practice (p < .001). In addition, teachers in the STeLLA VbPD program increased their use of teaching strategies that made student thinking visible and contributed to the coherence of the science lesson (p < .01). Most importantly, their students’ learning showed significant improvement .The STeLLA-I study examined teachers’ and students’ science content knowledge via written tests (pre-mid-post), teachers’ ability to use PCK in their analysis of video clips of science teaching (pre-mid-post), and teachers’ science teaching practice as captured in lesson videos (pre-post). Research findings , the STeLLA-II study tested the efficacy of the VbPD program using a randomized design (assignment to treatment group at the school level) to examine whether the program, when delivered in a new geographic area  and by PD leaders outside the original program development team, is as effective as a science content deepening program of equal duration. The content deepening program used in STeLLA-I was extended to become a program that supported teachers during the school year in addition to the summer institute. The strength of the comparison group is supported by teachers’ initial interest in being assigned to this program, teachers’ persistence when assigned to this program, and teachers’ feedback about the high quality of this experience in surveys and focus groups conducted by the external evaluation team. The experimental sample included 77 schools, 137 teachers, and 2823 students.Analyses showed that the STeLLA video-based, analysis-of-practice intervention was more effective for both students and teachers than the content deepening program of equal duration .A major challenge for school districts and the science education research community is how to scale up rigorously tested video-based professional development experiences, such as STeLLA, so that they can reach larger numbers of teachers (and their students) in a manner that is practical and sustainable. The RESPeCT project is an NSF-funded partnership project that is studying whether the STeLLA VbPD approach can be scaled to eventually reach most K-6 teachers in an urban, high-needs, Title I school district. The partners are California State Polytechnic University at Pomona, the Pomona Unified School District (PUSD), and the nonprofit science education research organization, Biological Sciences Curriculum Study (BSCS). The goal is that by the end of the 5-year project, PUSD will have the capacity to continue to implement the STeLLA VbPD program with limited input from CPP and have a direct, lasting, and positive impact on their students’ learning. The key to this scale-up effort is the development of a cadre of K-6 grade level teacher leaders within the district.p < .0001). An effect size of 1.6 shows that the treatment teachers made very large gains in their content knowledge versus comparison teachers.The RESPeCT project is using a quasi-experimental research design with a comparison group of “business-as-usual” teachers from schools matched to treatment school demographics. The research phase of the project is still underway but preliminary analyses of teachers' science content learning and students' science learning in grades 1–6  demonstrate significant growth in all content areas as compared with teacher and student learning in business-as-usual classrooms. For example, initial analyses of student content learning gains show a mean effect size of 0.97 across 11 content modules . For teacher content knowledge, there is a strong interaction between the pre- and posttest and group assignment (treatment versus comparison). The null hypothesis that gains by STeLLA treatment teachers and comparison teachers are equal can be rejected with a high degree of confidence  Throughout all of this, the teacher probes student thinking to find out how students are making sense of new data or ideas, challenges students to stretch their thinking and to make new connections, and teaches students to communicate in scientific ways .Looking Inside the Classroom observational study of science teaching in the USA both identified that US science lessons frequently fail to support students in making connections between science classroom activities and the development of science ideas and explanations  video analysis of science teaching and Horizon’s Grounded in research, the conceptual framework serves as the foundation and the centerpiece of the PD program, with all summer and school-year sessions organized around the STeLLA lenses and strategies. It anchors the program and assures program coherence by keeping the substance and activities in the program tightly focused on two analytical lenses and a few key ideas and teaching strategies that can be explored in depth and developed into teachers’ practice.The STeLLA VbPD program is guided by clearly specified teacher and student learning goals that are closely tied to the conceptual framework. Identifying and articulating these learning goals from the beginning is essential for designing and implementing both science lessons and PD sessions that are coherent, focused, and supportive of student/teacher learning. In our work, learning goals for students focus on understanding and using core science ideas. Learning goals for teachers include science content knowledge, pedagogical content knowledge related to classroom science teaching, and abilities to use both these types of knowledge to analyze and teach science.The goal in STeLLA work is for teachers and students to develop understandings of key science ideas and related crosscutting concepts through the use of science practices such as analyzing and interpreting data and constructing explanations . For each grade level in STeLLA-I and STeLLA-II, we identified student and teacher sets of science learning goals in two content areas that were in the state standards and school curricula of participating teachers. In the RESPeCT project, which started after the release of the NGSS, these content learning goals focused largely on NGSS-defined disciplinary core ideas and crosscutting concepts.Several criteria were used to select the science learning goals. Priority was given to ideas that are (a) challenging for teachers and students, (b) that can be used to explain a variety of phenomena in teachers’ and students’ experience, and (c) that are supported by a research base that identifies common student difficulties and naïve theories related to learning this content. In addition, we wanted the subset of approximately 5–10 science ideas in each content area to be interconnected so that they could all be used together in culminating “use and apply” problem solving tasks. All lesson plans, content deepening work, and video analysis work remained focused on this set of interconnected ideas.teacher science content learning goals include the student science learning goals but go beyond them so that teachers have a deeper understanding of the science ideas they will be teaching and a better sense of the future trajectory of their students’ learning in this content area. The teacher science learning goals are addressed in the program in two main contexts: (a) through content deepening activities for teachers in the summer institute that are similar to those in the lesson plans for students, and (b) through opportunities that arise in the context of lesson video analysis throughout the program.The In addition to science content learning goals, STeLLA teachers are also expected to develop pedagogical content knowledge related to the Student Thinking and Science Content Storyline Lenses and teaching strategies. Furthermore, it is a goal for teachers to be able to use their content knowledge and their pedagogical content knowledge to analyze science teaching and learning. A final broad learning goal is for teachers to use their content and pedagogical content knowledge about the STeLLA lenses and strategies in their science teaching practice. The conceptual framework outlines the specific teaching strategies that teachers learn to understand and use Fig. .These broad teacher learning goals related to PCK, analysis abilities, and teaching practice were useful in creating the overall design of the STeLLA program and the development of research assessments. However, these overall program goals were insufficient for mapping out the daily PD sessions during the summer institutes and school year study group sessions. Over time, especially as we began to develop new PD leaders and to write detailed PD leader guides, we recognized the need to state more specifically the teacher learning goals for each professional development session and for each segment of a PD session. We recognized that just as students’ science lessons are more coherent and powerful if they remain clearly focused on one main learning goal throughout a lesson . In this section, we describe how the video analysis work is supported and linked to other design principles, highlighting ways in which the design principles work together to support meaningful video analysis.Although we are highlighting the role of video analysis in the STeLLA program, it is important to keep in mind that teachers in this program are doing more than video analysis. Their analysis-of-practice work (design principle 5) includes analysis of student work, especially the comparison of students’ pre- and posttest writing, and the analysis of the STeLLA lesson plans. In addition, teachers are engaged in teaching STeLLA exemplar lesson plans in the fall and in planning and teaching lessons in a second content area in the spring (design principle 7). Teachers have opportunities to learn from this planning and teaching work that go beyond what they are learning from video analysis work.What is being analyzed in STeLLA video analysis? Before describing what teachers and PD leaders do in STeLLA video analysis, we start by linking back to the foundational design principles 1–3 and the importance of clearly defining the substance of the PD program; that is, what are teachers going to learn from this video analysis work? What are they analyzing and why?While most descriptions of PD programs in research reports provide only a broad, general view of the program substance, we attempted to be more specific in defining what teachers would learn from video analysis while at the same time resisting the temptation to “cover” everything that we know is important in achieving the best practice in science teaching. For the science content, we identified science ideas in the teachers’ curriculum that we knew to be challenging for students and teachers, to be useful in explaining a variety of phenomena in teachers’ and students’ experiences, and to be linked to important crosscutting concepts .To identify the STeLLA teaching strategies, we reviewed the research on science teaching and learning to identify what others now refer to as “high-leverage teaching practices” . This is especially evident in the way different kinds of activities are sequenced over time.In the summer institute, teachers begin the analysis process using videocases from more experienced teachers. In addition to the modeling provided by these teachers in the videos, PD leaders model how to productively analyze these videos and they coach teachers in their analytical efforts. Thus, in line with our theoretical stance, teacher participants have access at the beginning of the program to models of expertise.During the fall of the school year, they transition to analyzing video their own and their peers’ efforts to teach STeLLA model lesson plans, receiving support, feedback, and guidance in this work through their collaborative analyses of videos and student work in study group interactions. This is consistent with the coaching phase of the cognitive apprenticeship model.In the second half of the school year, scaffolding supports such as model lesson plans are removed (fading) and teachers are challenged to use STeLLA strategies and tools to plan and teach lessons in a new content area. This intentional use of a theory of teacher learning required us to consider what types of activities would be most beneficial for teacher learning across time, rather than settling on one type of video analysis and using it throughout the program.What features contribute to productive video analysis? While the foundational design principles drive the substance and form of STeLLA video analysis, there are a number of other features that work together to make important contributions to the quality of video analysis and to teacher learning:In line with our situated cognition theory of teacher learning, it is essential that teachers work on science content that is in their grade-level curriculum. This assures that each teacher is working in a meaningful, authentic context. In addition, when teachers are all using the same content learning goals, it enables shared experiences that allow for deeper analysis of practice and for development of common understandings of the content and teaching strategies.In the fall, teachers are supported in teaching exemplar lesson plans that are designed to highlight anticipated student thinking, the science content storyline, and the STeLLA strategies. These educative curriculum materials are intended to scaffold teachers’ developing understanding of the science content and their beginning use of the STeLLA teaching strategies. For each content area, there is a set of 6–12 lesson plans and a pre-post assessment. As with design principle 10, these shared curriculum materials provide participants with a common teaching experience that enables deeper and more meaningful analyses of that practice.In the STeLLA program, teachers begin the video analysis process in the summer by analyzing the practice of other teachers shown in grade-level and content-specific videocases prepared by the STeLLA program development team. These videocases include classroom video of science teaching that shows the STeLLA strategies in action, video of student interviews about the science content before and after an instructional sequence of lessons, and written student work on pre- and posttests. The STeLLA researchers review these videocases and select clips that will be used during the summer institute to engage teachers in developing their initial understandings of (a) the science content, (b) the STeLLA teaching strategies, and (c) the video analysis process. During the school year, teachers use the STeLLA lesson plans and are each videotaped during one lesson. During study group sessions, teachers analyze video clips of the implementation of these lessons from their own and their colleagues’ classrooms.Analytical tools and processes support teachers in deepening their analysis of science teaching and learning. These include (a) video clip transcripts which are essential for evidence-based reasoning, (b) analysis guides to support teachers in assessing the quality of implementation of each Science Content Storyline teaching strategy, (c) a STeLLA Lesson Analysis Protocol which guides video analysis, (d) a features analysis chart that reveals patterns in student written work, and (e) lesson planning tools that support teachers in developing student-focused lessons with coherent science content storylines.The Lesson Analysis Protocol (LAP) is the core analytical tool that is used throughout the program. The LAP process involves three main steps: (1) identify, (2) analyze, and 3) reflect. As shown in the example of an LAP in Table  reflect.STeLLA Strategies booklet. During the school year, this booklet is frequently used as a resource to support teachers’ claims, evidence, reasoning, and alternatives in the lesson analysis process.There are two key reference documents that are used extensively at the beginning of the program and are then referred back to throughout the program to support teachers in developing and maintaining common understandings of the STeLLA conceptual framework and strategies and of the science content. In the summer institute, teachers first learn about each of the STeLLA strategies by reading about them in the Teachers also refer often to a topic-specific content and pedagogical content background document. On the one hand, this is a content background document that is matched to the student and teacher science learning goals in the STeLLA program. But it is also a pedagogical content knowledge (PCK) document. In this regard, it describes the science content in the context of teaching situations, analyzes common student misunderstandings and difficulties, and describes strengths and weaknesses of common teaching activities, analogies, and content representations.learning community, and they understand the kinds of science knowledge they need to develop when they approach new science content areas—knowledge that enables them to explain a variety of phenomena using key science concepts and practices.In the STeLLA design, leadership of analysis-of-practice work plays a critical role in deepening teacher learning. Our choice of the title, “PD leaders,” reflects our recognition that leadership is necessary for the kinds of transformative change we seek. Some might view this as too “top down” and argue for an approach where teachers and PD leaders are co-constructing knowledge together (this is what some refer to as “facilitating” rather than “leading”). But our experience early on in the development of the STeLLA program convinced us that elementary teachers—who are typically reluctant to teach science—initially need both access to expertise from the science education community and support in learning to incorporate this knowledge into their science teaching practice. With fading supports from PD leaders over time, study groups gradually begin to function in a more bottom up way. In fact, we can imagine experienced STeLLA study groups functioning productively without a PD leader as they continue beyond the 1-year STeLLA program because they know how to analyze science teaching and learning productively to improve their science teaching practice, they know when and where to find outside expertise, they understand how to function as a true It is not enough to say that “PD leadership is important.” In thinking toward the scale up of the STeLLA approach, we recognized the need to understand what it takes to develop new PD leaders: What kinds of knowledge and abilities are needed? Analysis of the work of effective STeLLA leaders enabled us to answer this question with the knowledge and decision-making model of STeLLA PD leadership shown in Fig. The model highlights that effective leadership of STeLLA lesson video analysis depends on PD providers who are able to draw from a rich knowledge base to make decisions and take actions—both in planning and while leading video analysis sessions—that support teacher learning through deep and meaningful analyses. The model specifies that well-prepared STeLLA PD leaders make decisions and take actions by drawing from three types of knowledge: (a) knowledge of the STeLLA PD curriculum , (b) science content knowledge (including how science practices can be used to explain phenomena and to understand disciplinary core ideas and cross cutting concepts), and (c) research-based knowledge about how teachers learn in communities of learners. In addition, PD leaders draw from knowledge of the particular videos that are being analyzed. PD leaders know how to use all of this knowledge to make planning and in-the-moment, program-aligned decisions and adaptations.A key planning decision made by PD leaders is the selection and sequencing of video clips so that they best support targeted teacher learning goals for a given study group session. PD leaders watch the full lesson videos from the participating teachers to select a 3–8-min video clip from each teacher whose video will be analyzed during a given study group session. In making these selections, the PD leaders consider ways in which the video clips can be used (a) to deepen teachers’ understanding of the STeLLA strategies, (b) to clarify and deepen teachers’ science content understandings, and (c) to analyze interesting examples of student thinking. PD leaders also consider how the set of 3–4 video clips for a given study group session can hang together to focus on specific teacher learning goals for the overall meeting. From watching the full lesson videos, the PD leader learns a great deal about the strengths and weaknesses of the lessons and of teacher implementation of the lessons and uses this knowledge to identify key issues that could be productively examined through video analysis. Last but not least, PD leaders consider how a video clip might support and encourage the teacher featured in the clip while also challenging that teacher to reflect on her/his own teaching in ways that lead to growth; this requires assessing that teacher’s readiness to be challenged. All of this is considered as PD leaders select the video clips, identify specific teacher learning goals for the session, and create specific identify and analysis questions to focus the discussion and analysis for each clip.Asking elicit and probe questions to encourage teachers to share and elaborate their thinkingAsking challenge questions to scaffold and deepen teachers’ analyses and understandings of both science content and pedagogical content Modeling the video analysis process by sharing her/his own claim, evidence, reasoning, and alternative about a video clipModeling and reflecting metacognitively on the PD leader’s use of STeLLA teaching strategies Encouraging participation by all Interrupting the video analysis process to engage in some needed science content deepening workHighlighting and summarizing key points Asking teachers to summarize or paraphrase what they have heardProviding feedback about the quality and accuracy of teacher ideas Remaining silent During video analysis discussions, PD leaders make in-the-moment decisions about when and how to take the following kinds of actions:A key decision PD leaders make is about when and how to address gaps in teachers’ science understandings that arise during video analysis. Taking the time in video analysis sessions to wrestle with science content confusions is a critical aspect of the STeLLA program. Our research results show that teachers’ science understandings were better developed in the context of lesson video analysis (in the STeLLA program) than in sessions that focused specifically on deepening teachers’ understandings of the science content (in the content deepening comparison program). This is consistent with our situated theory of teacher learning—understanding the science content is more meaningful to teachers when it occurs in relationship to their teaching context.Without the PD leaders’ active role in scaffolding the video analysis process, especially at the beginning of the program, teachers tend toward a surface level discussion of video clips, favoring positive comments about the teaching or teacher featured in the clip. With PD leader guidance and use of the Lesson Analysis Protocol, teachers’ attention is focused on student thinking, the science content, the storyline, and the use of STeLLA strategies, minimizing the focus on an individual teacher while maximizing the analysis of more consequential issues that impact student learning.The effectiveness of the STeLLA VbPD program has been demonstrated through careful research—effective not only for teacher learning and practice but more importantly for student learning. But video-based analysis of practice is intense, requiring significant human and other resources to support its success. It is definitely not a quick fix. STeLLA-I and STeLLA-II VbPD programs were designed to be workable and sustainable as long as there was significant project funding from the National Science Foundation. The video focus, the need for transcripts to support evidence-based video analysis, the need for science hands-on materials to support the teaching of the STeLLA lessons, and the facilitated, small study group face-to-face structure of this PD approach pose challenges for scalability and sustainability. The RESPeCT project is addressing the challenge of scaling and sustaining this VbPD approach by (a) developing a cadre of teacher leaders at both the district level  and school levels (classroom teachers), (b) developing university science and mathematics faculty who can support this work beyond the life of the grant in PUSD or in new school districts, and (c) implementing changes within the district and university systems that will enable teachers and professors to participate effectively in this program.In this section, we describe more specifically what we are doing in the RESPeCT project to support the scaling and sustainability of this VbPD program. Three design principles guide this work: principle 17 pertains to the development of new VbPD leaders while principles 18 and 19 focus on development and sustainability of systems in which VbPD PD leaders can effectively work.The development of STeLLA VbPD leaders who have the knowledge and abilities described in Fig. Experienced science education PD leaders who are unfamiliar with the STeLLA PD programRESPeCT project elementary teachers who have experienced the STeLLA program as learners but have limited knowledge and experience in leading adult learningDistrict level PD specialists who have experienced the STeLLA program as learnersUniversity science and mathematics faculty unfamiliar with the STeLLA approach and with varying levels of experience in working with teachers.The content of the leadership program and the process of developing VbPD leaders changes based on the needs of the audience. We have worked with a variety of audiences, including (but not limited to):When the audience is experienced science PD leaders with little experience with the STeLLA VbPD program, the balance of leadership development tips toward the content of the STeLLA VbPD curriculum  analysis of their own experiences observing PD leaders in the leadership development program, (2) video analysis of other PD leaders working with study groups, and (3) video analysis of their own PD leadership practice after they begin to lead groups of teachers in this VbPD.PD leaders need to find the right balance between “supporting” and “challenging” teachers. By supporting, we mean using PD leader moves that encourage teachers by signaling that their ideas are valued and that they are making progress toward program goals. By challenging, we mean PD leader moves that press teachers to reconsider, revise, or better support their ideas. This often comes with some level of discomfort. PD leaders learn to make decisions about when and how to support or challenge teachers by analyzing video of other PD leaders and later by analyzing video of their own leadership practice, focusing on PD leader moves that support and/or challenge teachers. In working toward this balance, PD leaders learn to be intentional when they pose questions that elicit or probe teachers’ ideas, when they ask questions that challenge teacher thinking, or when they use any of the other STeLLA strategies in the context of teaching professionals.The leadership development program explicitly addresses how to develop communities of learners. For example, PD leaders consider their own experiences in light of readings about characteristics of professional learning communities (PLCs) . The second clip may show a later lesson where students negotiate this idea through an activity matched to the learning goal (STeLLA teaching strategy C) in which students analyze and interpret data from their observations of a tea kettle (STeLLA teaching strategy 4). A third video clip may come from later yet in the lesson sequence and show students attempting to use and apply science ideas (STeLLA teaching strategy 6) about evaporation and condensation as they compare/contrast cloud formation and steam from a tea kettle.To effectively facilitate science content deepening, PD leaders must achieve a level of confidence with the content that enables them to identify gaps in participants’ content knowledge and to decide “in-the-moment” how to address them. In the leadership program, content deepening is embedded in two contexts. First, as in the STeLLA program itself, content deepening in the leadership program is addressed by engaging participants (teacher leaders as learners) in video analysis. For example, teacher leaders watch a science lesson clip followed by a study group clip in which teachers are discussing the lesson clip. In the clips, both student and teacher misunderstandings of science content are revealed. This provides an opportunity for teacher leaders in the leadership program to clarify their own content understandings as they also think about how to lead sessions where such misunderstandings arise. Content deepening also occurs as teacher leaders are supported by science and mathematics faculty in practicing their implementation of content deepening segments of the VbPD program.Video analysis is also used to deepen leaders’ pedagogical content knowledge regarding the use of STeLLA lenses and strategies. As participants in the leadership program examine video of study groups, they analyze PD leader knowledge, moves, and decision-making. In this context, opportunities also arise to clarify teacher leaders’ understanding of common student ideas and effective use of the STeLLA strategies related to the target science content learning goals.The leadership development program engages PD leaders in practice facilitation of both lesson analysis experiences and content deepening activities. To create safe learning contexts, the practice facilitation involves participants in leading content deepening experiences within their own study group and in leading analysis of video to RESPeCT teacher leaders from other grade level study groups. These practice opportunities are scaffolded to include planning, doing, and reflecting stages. PD leader guides support the practice facilitation, providing details about how to implement each summer institute and school year study group session. These guides are linked to associated PowerPoints, handouts, and video clips.With the development of new PD leaders across the STeLLA-II and RESPeCT implementations, the STeLLA VbPD program has expanded its reach by developing 50 new PD leaders who in turn have worked with 200 teachers and their 7019 students. Through this multi-stage process, we learned that you cannot shortchange any component of the STeLLA model of PD leader knowledge and decision-making. An effective STeLLA PD leader understands the science content, the many facets of the STeLLA PD and classroom curricula, how teachers learn and change their practice in a community of learners, and how to use this knowledge to lead productive video analysis.While the STeLLA VbPD approach has demonstrated strong results in terms of teacher and student learning, it is an ambitious program that faces challenges in expanding the reach of the program. One challenge is the development of new, grade-level specific PD leaders who have the knowledge and abilities described in the previous section. Other requirements that might be impediments to long-term sustainability are the need for funding and other benefits to motivate teacher PD leaders in continuing to offer the VbPD to peer teachers; funding and other rewards to encourage teachers to participate in the VbPD; support and strategies for filming classrooms and making, transcribing, and disseminating video clips on a tight timeline; time for teacher PD leaders to review lesson videos and plan study group sessions; funding for and distribution of PD and science lesson plan materials; stable student science learning goals at each grade level so that the videocases remain relevant; stable grade-level teaching assignments so that teachers are teaching the content addressed in the VbPD program; supports to motivate and sustain university faculty commitment to providing ongoing science content advice to teacher PD leaders beyond the life of the grant; refilling the pipeline of teacher leaders and university support faculty; and strong district and university leaders who can lead successful responses to changes in staffing, curriculum, and institutional priorities that impact the VbPD program.To preserve fidelity to the STeLLA design features and sustain the VbPD program, we must do more than prepare new teacher PD leaders. Educational systems must also be readied. In the RESPeCT project, the systems involved in sustaining the work are the Pomona Unified School District and the California State Polytechnic University at Pomona.Together, but not individually, CPP and PUSD have the needed expertise identified in design principle 18: science education experts and researchers, scientists and mathematicians, teachers, and school personnel at multiple levels. This combined expertise is necessary to modify the VbPD program in ways that preserve the STeLLA design features while incorporating features that will contribute to the program’s scalability and sustainability. Thus, neither organization can scale and sustain the program independently of each other. However, the components of readiness required by both partner systems include (a) a shared priority for improving student science learning, (b) shared values and commitments to VbPD; (c) ongoing communication and shared decision making to support, sustain, and study the collaboration; (d) investments in building capacity for VbPD; and (e) inclusion of multiple stakeholders in shaping and implementing the program.Both CPP and PUSD partners have worked on developing these components of system readiness, as described in the following paragraphs.In response to national and local testing and funding policies, many districts and schools have raised instructional time requirements for elementary English language arts and mathematics which have had the effect of largely squeezing science out of the curriculum , the evaluation team, the research team, the university science and math faculty team, the supporting partner team at BSCS, the treatment schools principal team, the district support staff team, the university support staff team, and the university student support team. To assure these groups are working in concert with each other, there are regular cross-team meetings such as monthly PI/evaluation team meetings. Meeting time provides space for updates and problem solving, while also creating space for the various partners to contribute to decisions and to co-construct the ongoing collaborative work.While we have strong norms for communication and shared decision-making, the process of developing these has not been perfect, in part because of different cultures around decision-making and authority. For example, university faculty are accustomed to having a great deal of freedom in making their own decisions about their professional activities and outside-of-class schedules. For some, the RESPeCT research project involves more people and more collaboration than their science research projects and provides them with less authority to make decisions about their roles and timelines for project work. It took careful two-way communications, mentoring, reasonable modifications of expectations and timelines, and writing templates and leadership guides to keep faculty involved and supportive of project goals and activities.To sustain the PD leaders and the partnership, thoughtful investments in capacity are critical. Collaborative work requires human capital, funding, and time. The RESPeCT partners work together to ensure that funds support the recruitment of and dedicated time for staff, faculty, and PD leaders with the expertise needed to implement the VbPD program; the recruitment of and support for teachers and district teacher specialists to participate in the PD; and the classroom materials, technology, and physical space needed to support the PD program and the teaching of the science lessons. An additional element of human capital we cannot overemphasize is the importance of having a strong support staff who can make sure that all parts of the program are coordinated, communicated, and scheduled appropriately. In addition to the usual roles played by support staff, the following needs of a video-based program that is also part of a research study are addressed: scheduling the videotaping of science lessons; training student videographers; videotaping science lessons and study group sessions; making video clips; producing transcripts of video clips; disseminating videos and video clips; and organizing systems for filing and sharing videos, video clips, and supporting materials.From the beginning of the RESPeCT project, the leadership team engaged in strategic planning and capacity building to ensure that the VbPD program can be supported to reach all K-6 teachers in the district after grant funds expire. The post-grant phase will necessitate difficult conversations and compromises, largely related to funding, but we have laid a strong foundation for a collaborative and reflective professional community within and across institutions that includes patience, compromise, and flexibility in problem solving.In order to change the PUSD and CPP systems in ways that will sustain the RESPeCT program, the partnership institutions must move beyond the interactions of the leadership team to engage administrators, faculty, staff, and students at both the university and the district. Toward this goal, the PI team creates opportunities for a wide range of stakeholders to take authentic roles in collaborative activities. The RESPeCT program directly includes collaborative roles in the PUSD district for 35 teacher PD leaders, six treatment school principals, four teacher PD specialists , district level support staff, the Director of District Professional Development, the Director of Principal Support, and the Deputy Superintendent. At the university, science and mathematics content faculty, education faculty, a science content department chair, the Educational Outreach Center department chair, and CPP undergraduate students all play collaborative roles in the project. Engaging a diverse set of stakeholders is imperative for ensuring that the program meets the needs and interests of those who stand to benefit, namely the elementary teachers and students, but also to sustain the program despite turnover among critical individuals.In addition to casting a broad net for stakeholder involvement, key individual leaders at both the district, school, and university levels play critical roles in maintaining everyone’s confidence in the program’s organization and in keeping teachers fully participating in the program. These key champions play a “boundary-spanning” or “bridging” role  is dependent on working in a context where teachers feel safe to work with their colleagues as they to dig deeply and challenge themselves and each other (principle 9). Such challenges and deeper analyses are more likely when tackling shared content- and grade-level specific curriculum  and using the same curriculum materials (principle 11). Feeling safe in a learning community is not productive if you are not challenged to stretch and change your thinking. In the STeLLA line of research, we have found that such challenges can come from knowledgeable and skilled PD leaders , as well as from models by more experienced STeLLA teachers (principle 12). But the ultimate goal is to create learning communities where such challenge can come from teacher participants themselves (principle 9) as they learn from videos from their own classrooms (principle 12). And all of this takes time (principle 8).At the beginning of this paper, we made the case that the long-accepted and widely used consensus model of effective PD is of limited value in guiding the development, implementation, leadership, and scaling up of VbPD programs that positively impact teaching and student learning. We argued that this model lacks research support showing that it is effective in improving teaching and student learning. In addition, it provides only broad, surface level recommendations about how to design and implement PD experiences: Effective PD is content-focused, engages teachers in active learning, is matched to local, state, national policies, is of sufficient duration, and involves collective participation of teachers at a school or grade level.A clearly articulated conceptual framework that guides the substance of the program (principle 1)Clearly specified learning goals for teachers  and students  (principle 2)Program substance that prioritizes depth over breadth (principle 3)A theory of teacher learning to guide the structure, organization, and form of the program (principle 4)Video-based analysis-of-practice guided by an explicit analysis process Shared content, curriculum, and curriculum materials  Educative curriculum materials (principle 11)Science content learning experiences that are intertwined with and grow out of analysis-of-practice work (principle 6)Scaffolded teaching practice (principle 7)Expertise and leadership support from PD Leaders and clear specification of how their roles change over time Support for PD leader development (principle 17)The STeLLA VbPD program supports the consensus model with strong empirical evidence of student learning. But the STeLLA model also advances and deepens our understandings of effective PD by digging beneath the surface of the consensus model to articulate design principles that provide more specific guidance about features of effective PD. And, importantly, this model is supported by strong research evidence of impact on teaching practice and student learning. Based on our experiences with STeLLA, we nominate the following design principles as candidates that might be added to the consensus model and further tested for their potential as design principles to guide all VbPD:Of course, a refined consensus cannot be built based on findings from this one study. Additional research is needed before this list of nominated features can change the field’s consensus about effective PD . There is some existing research that examines the impact of science PD on both teacher and student learning and supports some of these STeLLA nominations. For example, results from the Heller et al. study  support We hope that this analysis of lessons learned from the STeLLA design principles can be useful for the design, implementation, and study of other professional development experiences for teachers, especially those that focus on video-based analysis of practice. We believe that an evidence-based articulation of VbPD design principles for the field would represent a major step forward in our knowledge about effective PD. This could help us move from the current surface level recommendations about “effective” PD that are today so widely used to guide the development of science teacher professional development programs to a set of well-specified, coherent design principles that are linked to evidence of student learning.


Globally, 52.9 million children under the age of 5 experience a developmental disability, such as sensory impairment, intellectual disability, and autism spectrum disorders. Of these 95% live in low-and-middle-income countries. Most of these children lack access to care. In light of the growing evidence that caregivers can learn skills to support their children’s social communication and adaptive behavior and to reduce their challenging behavior, the World Health Organization developed a novel Caregiver Skills Training Program (CST) for families of children with developmental disorders or delay to address such treatment gap. This report outlines the development process, content, and global field-testing strategy of the WHO CST program. The CST program is designed to be feasible, scalable, and adaptable and appropriate for implementation in low-resource settings by nonspecialists. The program was informed by an evidence review utilizing a common elements approach and was developed through extensive stakeholder consultation and an iterative revision process. The program is intended for a global audience and was designed to be adapted to the cultural, socioeconomic, geographic, and resource context in which it is used to ensure that it is comprehensible, acceptable, feasible, and relevant to target users. It is currently undergoing field-testing in more than 30 countries across all world regions. Around 250 million children, or 43% of all children younger than 5 years, in low- and middle-income countries (LMICs) are at higher risk of not reaching their developmental potential due to stunting, poverty, and disadvantage . In 2016The WHO is seeking to address this treatment gap and strengthen access and quality of health services and supports to families . In thisCentral to the development of caregiver skills training (CST) was the requirement that the target beneficiaries of the intervention were children with heterogeneous developmental difficulties and there was a need to promote scale-up in LMICs with the use of briefly trained nonspecialists. The program was designed to adopt a family-centered approach that fit within a stepped-care model, where CST is to be integrated into existing maternal, child, and family health and social services. In terms of the estimated economic impact of implementing the program, it is notable to mention that the requirements for training the intervention providers, who are nonspecialists, are much lower than in many other comparable fee-based intervention programs requiring specialist providers; thus, the implementation costs are expected to be lower. The impetus for the development of the CST program lies within the WHO’s mhGAP Intervention Guide for mental, neurological, and substance use disorders in nonspecialized health settings, which since 2010 recommends parent skills training for developmental disorders or delays, if available . HoweverThe formative process to develop CST consisted of a systematic review and meta-analysis and an extensive expert consultation. The systematic review included the analysis of implementation processes and component analysis using metaregression techniques with the aim of identifying key components and characteristics of efficacious interventions , 8. The As starting point for the development of CST, the results of the systematic review and meta-analysis were examined and discussed by global leaders with experience in parent-mediated interventions from diverse professional, geographic, and cultural backgrounds at a meeting at WHO Headquarters. The meeting included representatives from 21 countries across all six populated continents with a majority of representatives from LMICs, according to World Bank Classification. Representatives included academic leaders, clinicians, foundation leadership, practitioners, and caregivers (beneficiaries). These representatives were asked to advise on content and structure of the intervention, help address acceptability and feasibility concerns, and identify capacity building strategies.A few aspects of the program were carefully considered at the intervention design stage: (a) engagement of other caregivers and family members; (b) promotion of caregivers’ well-being and acceptance of the child’s difficulty as prerequisites to learning skills; (c) potential demand-side barriers to accessing the program, including stigma; and (d) heterogeneity of children’s developmental and health profiles, with particular regard to the high prevalenceA modular approach was suggested with “core” individual and group sessions followed by additional optional sessions, to address specific needs and considering availability of resources. The content of the core sessions was recommended to include promoting joint engagement, promoting spoken and nonverbal communication to request and share attention, reducing challenging behavior, teaching skills for daily living, and promoting caregiving well-being. Additional topics that were suggested as important included strategies tailored for children who have minimal spontaneous spoken language and additional materials on caregiver well-being and comorbid conditions. Alternative scripts for stories and role plays are provided to enable further tailoring of the content to the needs and priorities of families.The program was designed to include a combination of group sessions in community centers, health centers or schools and individual sessions in caregivers’ homes. The group format for most of the intervention keeps implementation costs  to a minimum and promotes peer support among caregivers, while the individual home visits allow the intervention to be responsive to each family’s needs by tailoring guidance to the caregiver and child characteristics. The home visits serve in fact the purpose of setting goals for the child’s development based on the assessment of skills in a naturalistic setting and the identification of family priorities; tailored coaching on the intervention strategies is then provided, and goals are periodically reassessed. Home visits are also critical for rapport building with the whole family and identification of additional family needs requiring referral to other services or professionals.A cascade training and supervision model was decided on as a critical element for effective and cost-effective implementation of CST by nonspecialist providers  at the health facility or community level, or in schools, allowing for program scale-up. To effectively support the implementation of CST by nonspecialist providers, especially in lower-resource settings, a continuous support and supervision model was incorporated.Given the global lack of service providers and long wait times, which would create an unnecessary delay in access to needed treatment, the inclusion and exclusion criteria for the target beneficiaries of the intervention were chosen to be as inclusive as possible, and it was decided that a diagnosis should not be required. The target group for the program was therefore identified as that of caregivers of children aged 2 to 9 years presenting with a developmental disorder or delay. A further consideration was that creating a program exclusively for families of children with a formal diagnosis of developmental disorder could be a barrier to families’ engagement due to lack of diagnostic services, delay in diagnosis, and stigma. The wide age range was decided on because children are often identified late, particularly in low-resource settings, and a program for caregivers of younger children would need to be offered in partnership with early intervention programs, which may not be yet in place. Other principles agreed on included the need to carefully consider the optimal intensity of the program in terms of number and duration of sessions. Considerable attention was paid to retention strategies designed to decrease the risk of dropout and increase engagement of families, such as regular phone calls, text messages, and provision of refreshments.With regard to the selection of intervention goals, the program was designed to target (a) the child’s development, specifically in regard to promoting social communication and adaptive skills and reducing disruptive and challenging behavior; (b) the caregiver–child relationship; (c) the child’s participation and inclusion in daily home and community activities; and (d) the caregiver’s role and functioning, by promoting self-confidence, parenting skills and knowledge, and coping skills and psychological well-being. The program was designed to include nine core modules for group sessions aimed at training caregivers in the use of strategies that target key domains of child and caregiver functioning Table 1,A major recommendation from the expert consultation was the requirement to appropriately address the heterogeneous needs of children and families, given the broad target group. To allow the intervention to be tailored to children and families’ needs, a threefold strategy was devised: first, goal-setting activities were embedded throughout the program to ensure that the intervention could be tailored to the child’s developmental level and the family’s priorities and flexibly adjusted as the child progresses. To this end, during the first home visit, the nonspecialist provider works jointly with each family to identify two “target routines”  that match the interventionist’s observations about the child’s needs and developmental level, while meeting the family’s priorities and daily activities. These target routines are regularly revised during the program to ensure that they continue to be appropriate. The second individualized component of the program consists of one-to-one coaching provided to caregivers both during group sessions, through role-play activities, and at the home visits, during live interaction with the child. The coaching component facilitates learning of the intervention strategies taught during the group sessions, ensuring that the caregiver is given clinically sensitive, individualized feedback to develop those skills and competencies that particularly suit the child’s needs. Lastly, optional modules with a focus on (a) children who have minimal spontaneous spoken language, (b) those with other comorbid conditions, and (c) caregiver well-being were made available to ensure that comorbidities and other co-occurring needs could be addressed.A suite of materials for field testing was developed, including (a) intervention manuals and user-facing documents ; (b) materials developed to assist countries in the planning and adaptation phases ; (c) materials for training master trainers and facilitators (training of trainer [ToT] course and supervision models); and (d) materials to record, monitor, and assess processes and outcomes in the prepilot and pilot-testing phases .The program’s theoretical foundation and methodology are informed by principles of applied behavior analysis, developmental science, social communication interventions, positive parenting, and self-care methods. The primary program targets are defined as increased spontaneous nonverbal and verbal communication and increased time in shared engagement, and secondary targets include reduced child challenging behavior, improved caregiver coping skills and psychological well-being, and improved family functioning. The program was developed with an additional aim of facilitating stigma reduction against persons with developmental disorders and promoting increased inclusion and community participation of these children.The common thread to accomplish the program’s goals is the shaping of common activities into regular shared caregiver and child routines that become opportunities for learning and development. Caregivers are encouraged to practice interacting with their child within both “home routines”  and “play routines” with toys and recycled materials  that are easily available to the family. Over the course of the program, caregivers are taught intervention strategies derived from principles of Naturalistic Developmental Behavioral Interventions and principles of applied behavior analysis for neurodevelopmental disorders. Naturalistic Developmental Behavioral Interventions, which include JASPER , 10 and Such teaching techniques are used within a naturalistic framework, with the use of natural rewards , use of An additional desirable characteristic of this approach is that it facilitates caregivers’ involvement because these naturalistic teaching strategies can be easily implemented within the family context, by transforming everyday activities  into child-led learning opportunities with a clear structure (routine building) that work toward prespecified developmentally appropriate targets . Exposing the child to multiple learning experiences targeting the same skills in different real-life contexts  is shown to predict greater generalization of skills, tolerance of real-world distractions, and reduced dependence on prompting .Furthermore, the approach favors and reinforces the parent’s natural role, since caregivers are encouraged to commit to daily moments of interaction with the child within regular activities that would be carried out regardless, instead of being asked to take up the role of a “therapist” carrying out additional structured tasks, resulting in happier, less stressful interactions and more positive communication styles . CaregivTo provide a practical example of how within CST caregivers are taught routine building, we illustrate below how to establish a routine in the context of a caregiver and child dyad taking items out of a shopping bag. First, the adult would need to identify which specific actions constitute, on that occasion, the steps that are appropriate for them and their child to follow . Then, the adult would make sure to assume an active but balanced role, rather than solely asking the child to complete actions or, conversely, dominating the interaction completely by leading all the steps. A balanced active role would therefore entail taking turns with the child in completing each step; these may mean alternating on doing the same action (direct imitation), or steps may be different for the child and the adult, such as the child taking items out and the adult putting them into a cupboard or container. A strong focus would be given on promoting child engagement by making the routine affectively salient . The underlying goal, in line with the programs’ primary targets, would be to promote the child’s communication by giving the child the opportunity to communicate (staying silent and looking expectantly at the child on the child’s turn) and responding and expanding the child’s communicative signals  by combining a gesture and language at the child’s level . Positive attention, imitation, and social rewards (praise) would be provided contingently to the child’s actions or approximations, in order to reinforce the performance.key messages  and tips (hands-on strategies and skills for interacting with the child). The latter are shown during the group sessions through adult-learning techniques such as group discussions, modeling, and guided role playing  scale  and a beAs a global program, WHO CST was developed to be adapted to the cultural, socioeconomic, geographic, and resource context in which it is used. Adaptation refers to the systematic modification of an intervention to ensure that it is comprehensible, acceptable, feasible, and relevant to target users . There iAn iterative process of revisions that incorporated inputs from the first stakeholder workshop and a second external expert review resulted in the finalization of program materials (WHO CST Test-Run Version). This version of the program was prepiloted for the first time with a group of caregivers of preschoolers with autism spectrum disorder and co-occurring intellectual disability in Northern Italy. The objective of the test run was the preliminary assessment of feasibility and acceptability of key delivery components and methods of the program prior to making available the materials for global field testing. Group sessions and home visits were led by a WHO CST Team member who contributed to the development of the program and translated the materials in Italian (ES), assisted by a local clinical psychologist with expertise in disability and parenting programs. The choice of specialist, rather than nonspecialist, providers was deemed necessary to allow for live troubleshooting, even though the program had been ultimately designed to be delivered by nonspecialists. The post-program qualitative–quantitative evaluations with caregivers and nonspecialist providers informed a light-touch revision of materials aimed at improving the provider’s manuals by (1) streamlining redundant content, (2) editing the instructions for activities  that had been reported as lacking clarity, and (3) formatting the text to improve readability. The revised program materials were then made available for field testing in 2016 (WHO CST Field Test version 1.0). Data derived from the evaluation of feasibility and acceptability of the test run implementation and those collected from consultation meetings, master training courses, and prepilot testing of the WHO CST Original Version in the first countries involved in the global CST field-testing initiative, such as Ethiopia , were coTo date, there are 30 active field-testing sites, representing all WHO world regions, African (n = 4), Americas (n = 9), Eastern Mediterranean (n = 6), European (n = 4), Southeast Asia (n = 2), and Western Pacific (n = 5). Official field-test versions of the package are now also available in Spanish, and translations are in process in multiple other languages. Participating sites progress through four phases outlined in a monitoring and evaluation framework for field testing, consisting of (1) planning and adaption, (2) ToT and post-ToT practice, (3) prepilot field testing, and (4) pilot testing. Input from field testing is being collected in order to contribute to the development of the final version of the CST package, which will be made available on the WHO website.A survey of adaptation processes and contents among sites participating in the field testing is underway. Preliminary data from 28 sites indicate that the majority of sites (n = 26) reported having adapted the program to the local context, mostly with minor measures: of these, 76% were adaptations of content , 17% were adaptations to improve feasibility , and 7% were adaptations of processes . Minor changes to content included changes to names of characters, idioms, language use, aspects of stories, objects, style of character dialogues in the stories, the addition of psychoeducational messages particularly relevant to the context , and modification of illustrations for ethnic, cultural, and contextual reasons. Changes to program process to support attendance included providing child care for group sessions, weekly peer support phone calls, refreshments, small gifts, post-program celebration, and additional outdoor self-care activities for caregivers. An adaptation for a low-resource, low-literacy setting was conducted in Ethiopia in consultation with community stakeholders. Adaptations included modification of activities that required writing, simplification of provider demonstrations and participant booklets, additional information on addressing expectations of a cure and discouraging physical punishment, removal of the picture schedule component, and increased emphasis on use of gestures. In pilot testing, the locally adapted WHO CST program was found to be acceptable and feasible for caregivers . The proThe field-testing phase and, in particular, data derived from the planning and engagement workshops with stakeholders will provide additional insight into opportunities for integrating the support to caregivers into existing community-based services or programs. Additional studies, evaluating cost-effectiveness, component analysis, alternative delivery methods, and dosage, will also be beneficial. Findings from the field-testing and future research studies can be used to inform future implementation and research on scalable, sustainable interventions for children with developmental disorders or delays.The datasets generated for this study are available on request to the corresponding author.Ethical review and approval was not required for the development of the caregiver skills training program described in this article, in accordance with the local legislation and institutional requirements. However, local ethics approval was obtained at field-testing sites in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.CS conceived of and led the CST initiative. BR led the meta-analysis and systematic reviews, with support of CS, which helped establish the empirical evidence and theoretical framework of the CST program. LP, ES, SS, and FB contributed to the drafting of the CST program materials and field-testing materials; ES and LP supported field-testing sites with supervision by CS. AS provided institutional support. ES and LP led the writing of the paper with input from all authors.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The cohort approach highlighted important outcome successes and identified workload challenges affecting recruitment and retention.The objective of this study was to describe how a cohort review approach was applied as an evaluation framework for a community health worker intervention among adult residents in 5 public housing developments in New York City in 2015–2017. The cohort review approach involved systematically monitoring participants engaged in the Harlem Health Advocacy Partners program during a given time period (“cohort”) to assess individual outcomes and program performance. We monitored participation status  and health outcomes. In this example of a cohort review, levels of enrollment and program disengagement were higher in cohort 1 than in cohort 2. For 6-month health outcomes, the percentage of participants with hypertension who had controlled blood pressure was static in cohort 1 and improved significantly in cohort 2. The percentage of participants with diabetes who self-reported controlled hemoglobin A Findings from community health worker (CHW) interventions targeting chronic disease prevention and management demonstrate inconsistent results, which may be attributable to funding mechanisms. Monitoring tools developed to address resource constraints, such as the cohort review, have not been used previously to evaluate CHW programs.We applied a cohort review approach as an evaluation framework for a community-focused CHW intervention in New York City. We assessed program implementation and outcomes during the first 2 years of the program. The cohort approach highlighted 6-month outcome successes related to hypertension and diabetes control and identified workload challenges affecting recruitment and retention.Adapting a cohort monitoring approach can be useful for evaluating the implementation of CHW programs. Such an approach also addresses issues associated with resource constraints and limited program duration.Although evidence for the effectiveness of community health workers (CHWs) is mounting, reviews of interventions related to chronic disease prevention and management demonstrate inconsistent results –3. One kThese challenges have affected other public health interventions focused on sustained patient interactions, and monitoring tools developed in response to these challenges can be adapted for CHW program evaluation. For example, the introduction of an annual review process known as a “cohort review” was an important innovation in the monitoring and evaluation of tuberculosis control efforts ; it invoWe adapted cohort review methods to the evaluation of a CHW program. By standardizing participant status definitions and tracking outcome milestones, CHWs and evaluators can develop an analytic framework to better monitor participation status, participant characteristics, and health outcomes. The cohort process also allows for the assessment of trends of program performance indicators that are actionable for decision makers, particularly when comparison groups are unavailable or are no longer supported by funding sources.The objective of this study was to describe how the cohort review approach was applied as an evaluation framework for a community-focused CHW intervention, the Harlem Health Advocacy Partners (HHAP) program, in New York City. HHAP is an ongoing municipal project that aims to improve the health of adults residing in 5 public housing developments in East/Central Harlem. Despite rich histories of community organizing, East/Central Harlem has been subject to policies and processes such as redlining, broken windows policing, and “benign neglect” that have contributed to high levels of poverty and poor health outcomes. HHAP was launched to address health and social conditions in the neighborhood, with the aim of closing racial/ethnic gaps in health and social outcomes between public housing residents in East/Central Harlem and other New Yorkers ,7. We deDuring the first year of HHAP, 224 participants were enrolled from February through August 2015 (cohort 1), and subsequent cohorts followed an annual enrollment cycle. Cohort 2 enrolled 348 participants from September 2015 through August 2016. Concurrent to cohort 1 enrollment, we recruited a 1-year comparison sample of 176 residents from 5 nearby developments, selected on the basis of frequency-matched sociodemographic characteristics and proximity to the intervention developments . Participants who reported ever having received a physician diagnosis of asthma, hypertension, or diabetes were defined as adults with these conditions, on the basis of the following question: “Have you ever been told by a doctor, nurse, or other health professional that you have . . . ?” Both cohort 1 and cohort 2 participants were recruited primarily via community outreach conducted by CHWs, who canvassed the grounds of the selected public housing developments and collaborated with community and senior centers in each development to promote the program. The comparison group was recruited from a random sample telephone survey . CHWs atThe HHAP intervention includes 4 components: 1) health coaching, 2) navigation of the health care system, 3) wellness activities, including peer support and walking groups, and 4) advocacy to build leadership among residents to address community health needs and improve systems and conditions that influence neighborhood health. The health coaching provided by CHWs also included referrals, emergency interventions during acute-risk situations , and the setting of one or more SMART  goals. Additional health care navigation support was available through referrals to a partner organization that assists residents in obtaining medical services and ensures they receive the care to which they are entitled. A full description of the HHAP model is available elsewhere .For cohort 1, CHWs conducted intake assessments as part of participant enrollment (baseline), and an academic research team from the NYU–CUNY Prevention Research Center (PRC) conducted follow-up assessments. The academic research team conducted all comparison group assessments. Cohort 1 and comparison participants received a $20 cash incentive for completing surveys. For cohort 2, CHWs conducted baseline and follow-up assessments. Surveys were conducted at 3 months, 6 months, 9 months, or 12 months after enrollment. Among participants enrolled in cohort 1 and cohort 2, 209 of 224 (93.3%) in cohort 1 and 233 of 348 (67.0%) in cohort 2 completed any follow-up assessment survey. For this analysis, we tabulated data on 6-month follow-up from both years; the response rate was 85.7% (192 of 224) for cohort 1, 92.6% (163 of 176) for the comparison group, and 41.7% (145 of 348) for cohort 2.1c (HbA1c) control among participants with diabetes. Blood pressure was the average of 3 measurements taken at each survey point, and we defined control as systolic blood pressure under 140 mm Hg or diastolic blood pressure under 90 mm Hg , we compared the baseline characteristics of cohort 1 with the baseline characteristics of cohort 2 and the comparison group with A greater percentage of residents participating in HHAP health coaching than in the comparison group were aged 65 or older and self-reported hypertension . Most paEnrollment increased 55.4% from cohort 1 to cohort 2, from 224 to 348 participants. Of the 224 cohort 1 participants, 216 (96.4%) participants were still active in the program after 6 months, 5 (2.2%) had disengaged, 1 (0.4%) was on leave, and 2 (0.9%) had died. Of the 348 participants enrolled in cohort 2, 303 (87.1%) were still active after 6 months, 39 (11.2%) had disengaged, 2 (0.6%) were on leave, and 2 (0.6%) had died.P = .79) (P = .16). In cohort 2, the percentage of participants with diagnosed hypertension and controlled blood pressure increased significantly, from 57.7% to 73.9% (P = .002). The percentage of participants with self-reported diabetes who reported their HbA1c as controlled increased significantly in cohort 1 , whereas self-reported HbA1c control did not improve among comparison group participants . Although the change was not significant, we found improvements in HbA1c control among cohort 2 participants .The percentage of participants with self-reported hypertension in cohort 1 and controlled blood pressure did not change from baseline to 6-month follow-up  . Blood pOur findings from the first 2 years of HHAP’s health coaching component demonstrate the utility of the cohort review approach in providing a structure for evaluating a multiyear program, particularly when an ongoing comparison group is not available. The approach highlighted successes in health outcomes among participants retained in the program and challenges in program retention.The assessment showed that more participants in cohort 2 than in cohort 1 disengaged from the program after 6 months. One reason for the higher level of disengagement in cohort 2 could be the challenge of maintaining health-coaching participants carried over from cohort 1 while recruiting for cohort 2, since CHWs in cohort 2 were also responsible for managing participants from the previous year. In addition, in the beginning of cohort 2, programmatic operations were transferred from an external organization to the New York City Department of Health and Mental Hygiene, which may have resulted in a disruption for some participants. Finally, the incentive offered in cohort 1 may have positively influenced program retention and the number of follow-up interviews. Because the cohort process cycle emphasizes continuous monitoring and improvement, the HHAP program addressed retention and workload issues in cohort 3.Our cohort assessment quantified improvements in key health outcomes shown in previous studies, namely in blood pressure  and glycIn planning for evaluating CHW programs using a cohort approach, metrics for the implementation process should be developed a priori and aligned with program objectives. Our analysis underscored the challenge of defining the participation status of a participant as complete. The definition was challenging because the criteria for program completion changed over 2 cohort years; awareness of this challenge helped formalize the definition of completion. Moreover, the program further disaggregated the disengaged group into 4 new categories: withdrew, lost to follow-up, transferred out of health coaching, and unavailable . It will be important to monitor these categories to assess whether participants are not interested or able to participate in the program, which would reflect a poor fit between the program and a participant’s needs (withdrew), or the program is unable to maintain contact with participants because of other factors (lost to follow-up).We found the cohort review approach adaptable to new program goals. For example, to better HHAP’s efforts to address the social determinants of health in addition to disease management, we developed outcome metrics for social determinants of health for cohort 3, and the program will continue to monitor other variables that may contribute to health outcomes.Our study has several limitations. Our findings in part reflect differences in HHAP programmatic operations between cohort 1 and 2. Cohort 1 data were collected by both CHWs and an academic research team and participants in cohort 1 received a cash incentive for completing surveys, whereas cohort 2 data were collected by CHWs only, often with fewer follow-up assessments, and cohort 2 participants did not receive an incentive. Differences in data collection may have biased comparisons between cohort 1 and cohort 2. Some health outcome data were self-reported; however, any bias introduced by self-report is unlikely to be differential across cohorts, except if selection bias was introduced because of higher loss to follow-up in cohort 2. Finally, given the large number of disengaged participants in cohort 2, we are not fully confident that improved outcomes were solely a function of programmatic improvements. Improved outcomes may reflect differential disengagement of participants who would have been less likely to improve.Although previous CHW evaluations focused on individual-level outcomes, we found the cohort monitoring approach to be an effective method for evaluating the implementation process of CHW programs while also addressing issues associated with resource constraints and limited program duration . Adaptin


People living with dementia can experience significant barriers to meaningful participation in their communities, particularly in underserviced rural and small-town settings. Drawing on a multi-method pilot study employing observations, diaries, focus groups and interviews in rural Canada, we examine the potential of an innovative dance program developed by Baycrest Health Sciences and Canada’s National Ballet School, to transform the experiences of people living with dementia and the rural places in which they live. Our findings identify moments, processes, and places of transformation throughout the program including moments of individual self-expression; changing interactions with staff, volunteers, and carers; and changing relationships with home and community. We argue that art-based programs can challenge dominant assumptions about people living with dementia and contribute to the creation of more just health and social care in rural places. In doing so, we illustrate the value of critical arts-based approaches to aging in rural places.


Purpose Hemophilia Treatment Centers (HTCs) provide integrated and comprehensive services to individuals affected with rare bleeding disorders, such as hemophilia and Von Willebrand disease. Through the 340 Drug Pricing Program, HTCs may use pharmacy income to support clinical staff and patient services. The objective of this study was to describe the impact of the 340B program funding on services and support provided by HTCs to persons affected by rare bleeding disorders. Description Federally designated comprehensive HTCs with established 340B programs were invited to participate in a mailed survey in 2014. Participants were requested to report on 340B program-funded staff and services in the calendar year 2013. Assessment The 31 of 37 HTCs responding served over 10,000 individuals, or one-third of the national HTC patient population. The majority of responding HTCs reported that 340B program income supported over 90% of staff such as nurses, social workers, and physical therapists. Conclusion The results from this survey of 31 centers with established programs demonstrates the HTCs’ reliance on 340B program support for vital comprehensive services, that are otherwise non-reimbursable, and highlights the importance of the 340B program in sustaining the high quality of care and in increasing access for a geographically dispersed, medically vulnerable population. What is already known on this subject? Hemophilia Treatment Centers (HTCs) provide comprehensive services to a medically vulnerable population affected by rare bleeding disorders. HTCs are eligible as covered entities to use pharmacy income from the 340B Drug Pricing Program to support clinical staff and patient services.What this study adds? This study provides new information on the impact of the 340B Drug Pricing Program in supporting the comprehensive care model, the evidence-based standard of care for persons affected by rare bleeding disorders, within the HTCs. Income generated through the 340B Drug Pricing Program supports most of the multidisciplinary staff as well as care coordination, community education, and other support activities provided by the HTCs.Since 1975, the Health Resources and Services Administration (HRSA) has funded a national network of 135 Hemophilia Treatment Centers (HTCs) to provide integrated services to and increased access to care for children and adults with rare, inherited bleeding disorders  as part of the Public Health Service Act which allows designated covered entities, including HTCs, to purchase pharmaceutical products at a discounted rate  reported a patient population size of greater than 250 patients, with 42% reporting greater than 400 patients. As only 22% of the 103 HTCS with 340B programs had greater than 350 patients in 2013, larger HTCs are over-represented in the sample. Nineteen of the 31 respondents were described as a university academic institution, 4 were within a children’s hospital and 8 were free standing treatment centers.Applying the mean number of patients in each HTC size category (500 was used for the > 400 category) multiplied by the number of HTCS in that category, the total estimated number of patients served by the centers responding to this survey is over 10,000 individuals. All responding HTCs reported serving patients diagnosed with hemophilia, VWD, other factor deficiencies, and platelet disorders. Not all of the HTCs responded to every question, but all HTCs responded to the majority of questions.Twenty-nine out of 31 respondents (94%) reported they provided the four core comprehensive care services  at clinic visits in 2013. All 31 respondents indicated they utilized 340B program income to support the salaries of their core staff and other staff. As shown in Fig. In 2013, the approximate numbers of annual encounters with core members of the care team funded by 340B during comprehensive care visits were 10,092 encounters by nurses, 9048 encounters by hematologists, 9048 encounters by social workers, and 7656 encounters by physical therapists within participating HTCs. The 340B program income support for clinical staff salaries at outpatient and follow-up visits is similar to that for the comprehensive care visits. Most centers fund > 50% of the salary costs of non-physician staff utilizing 340B program income.Twenty-nine centers reported providing 57,072 urgent or emergent telephone triage encounters and 62,640 medical care coordination encounters, almost all non-billable services. Thirty centers reported 28,880 encounters for psychosocial and vocational services, almost all non-billable services. Figure HTCs provide education for patients and community providers; peer support groups for men, women, young families, peers, parent groups, and patients with thrombosis; and financial assistance with transportation Fig. . The majIn 2013, 20 of the 31 HTC respondents offered 26 overnight camp programs serving 1170 children and adolescents. Nine centers sponsored 16 family camps with 624 individuals attending. Providers most frequently attending the overnight camps included nurses (n = 19), hematologists (n = 15), social workers (n = 14), physical therapists (n = 13), and nurse practitioners (n = 11). Staff attendance at camps is rarely a billable service. Approximately half of the centers support salaries for staff attending camps at the > 90% level and half support camps at ≤ 50% level.The standard of care for children with special health care needs is to provide a multidisciplinary care team , a national registry of HTC patients, reports that the median size of HTC patient populations is 200 patients Forsberg . This suAnother limitation was the inability to collect patient specific data from HTCs. This data is proprietary and participants would only submit data on the ranges of the numbers of patients served. Furthermore, the survey collected information on ranges of funding support instead of exact amounts. The staff effort required to collect the actual dollar amount of 340B program income funding for each service was beyond the scope of this survey.The results from this survey of 31 centers with established programs demonstrates the HTCs’ reliance on 340B program support for vital comprehensive services, that are otherwise non-reimbursable, and highlights the importance of the 340B program in sustaining the high quality of care and in increasing access for a geographically dispersed, medically vulnerable population. The 31 centers represent over 10,000 patients which is approximately one-third of the overall population of the US HTCs. The 340B program income in the calendar year 2013 supported over 200,000 patient and family encounters at 31 HTCs across the US including comprehensive care visits; outpatient and other follow-up visits; coordination of care; patient and family education and support; camp services; and outreach clinics and telemedicine. This survey demonstrates that many of the HTC services that are critical to providing optimal care and preventing complications are not reimbursed by third-party payers. HTCs are dependent on 340B program income to sustain high quality, accessible, comprehensive care for a medically vulnerable and complex patient population. It is unlikely this evidence-based model could be sustainable without 340B program income. Future data collection efforts should focus on an assessment of the utilization of 340B program income at smaller centers with new 340B programs and trends in the level of services supported by 340B program income over time.