Monitoring and evaluating large-scale global health program transitions can strengthen accountability, facilitate stakeholder engagement, and promote learning about the transition process and how best to manage it. We propose a conceptual framework with 4 main domains relevant to transitions—leadership, financing, programming, and service delivery—along with guiding questions and illustrative indicators to guide users through key aspects of monitoring and evaluating transition. We argue that monitoring and evaluating transitions can bring conceptual clarity to the transition process, provide a mechanism for accountability, facilitate engagement with local stakeholders, and inform the management of transition through learning. Monitoring and evaluating large-scale global health program transitions can strengthen accountability, facilitate stakeholder engagement, and promote learning about the transition process and how best to manage it. We propose a conceptual framework with 4 main domains relevant to transitions—leadership, financing, programming, and service delivery—along with guiding questions and illustrative indicators to guide users through key aspects of monitoring and evaluating transition. We argue that monitoring and evaluating transitions can bring conceptual clarity to the transition process, provide a mechanism for accountability, facilitate engagement with local stakeholders, and inform the management of transition through learning. Donors are increasingly interested in the transition and sustainability of global health programs as priorities shift and external funding declines. Systematic and high-quality monitoring and evaluation (M&E) of such processes is rare. We propose a framework and related guiding questions to systematize the M&E of global health program transitions.We conducted stakeholder interviews, searched the peer-reviewed and gray literature, gathered feedback from key informants, and reflected on author experiences to build a framework on M&E of transition and to develop guiding questions.The conceptual framework models transition as a process spanning pre-transition and transition itself and extending into sustained services and outcomes. Key transition domains include leadership, financing, programming, and service delivery, and relevant activities that drive the transition in these domains forward include sustaining a supportive policy environment, creating financial sustainability, developing local stakeholder capacity, communicating to all stakeholders, and aligning programs. Ideally transition monitoring would begin prior to transition processes being implemented and continue for some time after transition has been completed. As no set of indicators will be applicable across all types of health program transitions, we instead propose guiding questions and illustrative quantitative and qualitative indicators to be considered and adapted based on the transition domains identified as most important to the particular health program transition. The M&E of transition faces new and unique challenges, requiring measuring constructs to which evaluators may not be accustomed. Many domains hinge on measuring “intangibles” such as the management of relationships. Monitoring these constructs may require a compromise between rigorous data collection and the involvement of key stakeholders.Monitoring and evaluating transitions in global health programs can bring conceptual clarity to the transition process, provide a mechanism for accountability, facilitate engagement with local stakeholders, and inform the management of transition through learning. Further investment and stronger methodological work are needed. Among programs funded by the United States Government (USG), the transitions of large-scale health programs have been called “graduations” and have been occurring at least since the 1980s with the graduation of family planning assistance programs in Latin America and the Caribbean.USG agencies are not the only ones to engage in such discussions. The Bill and Melinda Gates Foundation, The Global Fund to Fight AIDS, Tuberculosis and Malaria, and Gavi, the Vaccine Alliance, are among several other global organizations with growing interest in the transition from donor assistance toward long-term sustainability. Gavi, for example, has recently revised its graduation policy, which outlines a process for phasing out Gavi support with time-limited catalytic investments to support graduation plans.,,,The discussion of phasing out donor support often comes after years, if not decades, of investments in strengthening service delivery and health systems and significant efforts to reduce the burden of disease. Poorly executed transitions risk reversing health achievements, negatively affecting services and outcomes for the beneficiary population. As ownership is transitioned from donors to local counterparts, clear accountability is needed to ensure transition is successful. However, with a few exceptions,Poorly executed transitions of large-scale global health programs from donors to local governments risk reversing health achievements.,,,,-Despite the high stakes involved in transition processes, there are relatively few documented examples of how the transition process was managed, of the effects of transition on the health outcomes of interests, or of monitoring and evaluation (M&E) of how the transition process itself was executed or managed. The US Agency for International Development (USAID) commissioned midtermThere have been no systematic efforts to develop an approach to monitor and evaluate transitions of global health programs.In this paper, we draw on our team’s experiences to develop an initial description of the key dimensions of health program transitions from donors to local counterparts. Furthermore, we make the case for why M&E of the transition process is important and propose a framework to facilitate the identification of different domains and dimensions to monitor and evaluate before, during, and after transition. Finally, we propose guiding questions linked to these domains, which can stimulate thinking around the potential indicators for the M&E of transition. We argue that the M&E of transition can help us obtain a deeper appreciation of how transition unfolds, as well as of the evidence necessary to refine our understanding of how transitions should be planned and managed. We focus on health programs whose goals are to be continued after the transition, unlike, for example, the transition of polio programming after polio eradication.-Our paper takes a reflective “thought exercise” approach similar to that of Gilson et al.,Overall, to create the conceptual framework and guidance presented here, we:Based our initial thinking on the conceptual framework described in Bennett et al.Searched the literature including peer-reviewed literature, gray literature, personal author collections, and papers recommended by experts to identify additional relevant sources that would inform the adaptation of the framework and our thinking about the M&E of transition.Conducted semi-structured interviews with individuals with experience in health program transitions (N = 6); 5 respondents had firsthand involvement in USAID’s family planning graduation in Latin America, while based either at USG or within implementing partners; 1 respondent was engaged, at the time of the interview, in a CDC-funded activity to measure and monitor the transition of HIV care and treatment from international NGOs to local NGOs, and eventually to the country government.Adapted the Avahan frameworkObtained two rounds of feedback  about the draft framework and guidance from a group of key informants  with experience in health program transition; these informants were selected based on their recent engagement in the development of the Gavi graduation policy (n = 1), their knowledge of the scaling down of PEPFAR activities in the Caribbean (n = 2) and in Namibia (n = 1), and their knowledge of M&E and possible sources of indicators that could be used to monitor or evaluate transition processes (n = 1); all informants were associated with implementing, rather than funding, agencies. During both meetings, feedback was sought on how to package the available evidence around M&E of transition, the components of the proposed framework, and possible indicators and sources of indicators for each of the framework domains.Finalized the conceptual framework and guiding questions for this paper.Ultimately, the final framework and approach presented here is meant to build on previous experiences in implementing and managing M&E of transitions, so as to present a guide for how those engaged in transition can think about the process, as well as for program evaluators and planners on possible approaches to the M&E of future large-scale health program transitions.Our findings are organized as follows. First, we present a conceptual framework highlighting potential transition domains, activities, and outcomes to be monitored or evaluated. Next, based on this framework, we describe possible approaches to the M&E of transition including reflections on timing and organization of M&E approaches. Finally, we suggest guiding questions and potential indicators that might be used to monitor and evaluate health program transition.The proposed conceptual framework  seeks toWe conceive of transition as a process. Pre-transition activities set the parameters for transition including key factors such as the timeline, budget, and partners involved. Subsequent to these pre-transition activities, the transition entails a set of ongoing iterative processes that gradually shift program responsibility from the donor to the program recipient. Transitions may occur across one or more domains: leadership, financing, programming, and service delivery. Each of these domains is explained more fully in Transitions can occur across one or more of 4 domains: leadership, financing, programming, and service delivery.For each transition domain, we identified a set of relevant activities, which help to anchor the necessary responsibilities, rules, norms, and structures into the program recipient environment. Ultimately, these activities facilitate the delivery of program services by the recipient, at a level defined by the transition goals. These activities include:Sustaining a supportive policy environment: Ideally transition plans are developed and executed in an environment where existing policies and involved stakeholders are committed and supportive of achieving the overall objective of the transition and of the health program. In practice, transitions can be politically motivated and implemented abruptly and are often met with resistance and/or disbelief. Therefore, it is crucial for transition planning to conduct activities strategically to build political commitment and support.Creating financial sustainability: The existence of secure and diversified funding is central to the sustainability of a health program.,Developing local stakeholder capacity: Shifting health program responsibility from donor to program recipient means that the capacity previously supplied by donors must be replaced or adapted according to the priorities and capacities of local actors. Ideally, developing local capacity and building ownership is a process initiated long before transition begins; however, in practice, this is not always the case. Organizational capacity assessments may diagnose existing competencies and identify areas in need of investment to reach sufficient capacity for sustaining the health program.Developing local capacity to ensure effective transitions goes beyond strengthening the skills and tools to deliver program activities.Communicating to all stakeholders: As the transition process inherently involves shifts in power and authority, it often faces considerable stakeholder resistance. Timely, transparent, and appropriately disseminated communication plays a key role in persuading stakeholders and forming a group of proponents to support transition. Communication helps to align expectations, forge common goals, and facilitate building positive relationships among key audiences. These activities can help to overcome common risks of transition, such as being overwhelmed by tension and confusion from misinformed audiences, as well as an overall resistance to transition.Aligning programs: Programs may need to undergo a process of harmonization with existing services as they transition. This harmonization can include the adaptation of program services as well as the implementation of common arrangements for planning, management, financial reporting, and M&E, so as to integrate with the national program or host environment. Programs activities can be adapted, completely removed, or remain unchanged in accordance with the program recipient and health system context.The 5 activity areas above are closely interconnected and reflect complex adaptive relationships that influence one another.Together, these activities drive the transition of leadership, financing, programming, and service delivery forward so that program recipients can take full responsibility over these domains and over the health program as a whole. The intermediate result is the institutionalization of the program, by which we mean the development of policies, norms, and structures to sustain the program within the recipient organization’s health system. In total, a well-managed transition process enables the sustained delivery of program services and, hence, sustained health outcomes, even if there may be changes in who delivers services or the mechanisms through which they are provided. Ideally, a transition plan including a context-specific set of activities would be developed jointly in advance of transition and agreed upon between donors and recipient countries. Such a plan would clearly facilitate the M&E of transition, but historically such a document has been rare.The conceptual framework described in this article can be used to:Identify the most relevant domains being transitionedAssess which activities might be prioritized for monitoringFormulate appropriate indicators for monitoring and evaluating transition using the proposed guiding questions explained belowThe framework may also help program planners and evaluators to reflect on the question of when in the transition cycle it will be important to have measures of transition. As for many evaluations, determining the purpose of M&E is likely to be a critical first step.Not all health program transitions will reflect all of the domains described in the conceptual framework. In an ideal situation, the monitoring of transition would begin prior to transition processes being implemented and continue for some time after transition has been completed. The pre-transition period would be used to engage key stakeholders and reach consensus on plans for transition and M&E of transition. This period may also be used for reflection on what type of transition will be taking place and over what timeline. During this period, donors and potential program recipients should engage in open and transparent discussions to develop consensus around the program’s transition goals and M&E plans.The monitoring of transition should ideally begin prior to the implementation of transition and should continue beyond the completion of transition.Establish clear end goals. Clear end goals are critical to guiding M&E plans. Stakeholders should clarify their vision for the future of the program and of its end goals with respect to service coverage and health outcomes.Plan early. Early planning allows for regular and consistent monitoring of the transition process, evaluation of transition preparation activities, and collection of baseline data to determine impacts post-transition.Ensure program recipients are vested in M&E. It is critical to engage all stakeholders and secure commitments from key actors regarding the use of M&E evidence. It is especially important for the program recipient to engage in the process to ensure access to data post-transition and also given their central role in acting upon M&E evidence.Earmark funding for transition M&E. Programmatic transition is distinct from programmatic service delivery and, as such, needs specific earmarked funding, as does the M&E of transition.Protect the neutrality and independence of evaluators. To promote acceptance of, and action on, M&E findings, all stakeholders need to view M&E results as unbiased and independent. External evaluators may help achieve this perception, but evaluation teams composed of donor and program recipient representatives may also be appropriate.Often the triggers for transition will influence the nature of the entire process and the time frame within which it is implemented. For example, donors may signal the need for transition on the basis of target indicators being met, or for political reasons. In other cases, program recipients may initiate transition planning as part of developing a sustainable program. Longer time frames facilitate better planning for transition and stronger transition M&E that, in turn, allow for deeper learning. However, not all transitions will take place under ideal circumstances, and sometimes they may be hurried, responding to political imperatives rather than to carefully determined and mutually agreed conditions. While all transition circumstances offer opportunities for monitoring and for learning, rapid transitions typically limit the scope for rigorous evaluation and may be associated with antipathy toward learning. The costs associated with the M&E of transition will vary according to the scope and scale of the exercise, varying from modest, highly focused efforts that address just one phase of the transition cycle (such as transition preparedness) through to more comprehensive M&E processes, spanning into the post-transition period.No set of indicators will be applicable across all types of health program transitions. Accordingly, rather than proposing a short list of indicators, we present guiding questions and selected illustrative indicators to be considered and adapted. We encourage the use of both quantitative indicators and qualitative investigation as complementary approaches necessary to fully explore transition. Quantitative indicators identify changes that have occurred due to transition, demonstrate trends, and track whether transition goals, objectives, and milestones are being met. Qualitative methods describe transition experiences, explain why changes have occurred and their repercussions, and indicate what feedback and adaptation are taking place. Such a qualitative investigation can be critical to understanding why unexpected effects are occurring or to identifying the underlying causes of poor transition performance.Both quantitative and qualitative data are needed to fully understand the transition process and its effect on the wider health system.The selection of indicators for monitoring should be driven primarily by the importance of what is being measured; the scientific soundness of the measure; and the feasibility of obtaining data on the measure.,-Health program evaluators are likely accustomed to measuring indicators related to health services and outcomes, and such data are often routinely collected as part of program M&E processes. However, the M&E of transition requires understanding of quantitative indicators that may lie beyond typical health program indicators and are likely to be scattered among different stakeholders and data sources. Many of the guiding questions for quantitative indicators could also be explored through qualitative methods. We have included in Based on our collective experiences of transition, including interviews and key informant discussions combined with a literature review, we developed a conceptual framework that broadly details how the transition process for large-scale health programs occurs and offers an approach to identifying what, beyond traditional M&E impact indicators, can be monitored throughout the transition process. The 4 domains of transition of leadership, financing, programming, and service delivery involve sets of activities to be considered during a transition process—from sustaining a supportive policy environment and creating financial sustainability to developing local stakeholder capacity, communicating the transition plan to all stakeholders, and aligning programs—which will require active and evidence-informed management. By monitoring and evaluating these activities during transition and using the resulting evidence for decision making, planners may gain a better sense of where attention needs to be focused or shifted during the transition process, making way for any necessary course correction.While the ultimate goal of transition, namely, sustaining or enhancing services and outcomes, reflects constructs that health program evaluators are accustomed to measuring, other dimensions of transition may present challenges for evaluators. M&E of transition requires consideration of measures of factors that are much less commonly assessed in health program evaluations such as program alignment, the presence of a supportive policy environment, organizational capacity, and effective communication. Further, likely measures of these factors may be embedded in various data sources that are scattered across different stakeholders and often cannot easily be identified through routine M&E channels. Indeed, M&E of transition may require the development and implementation of special data collection tools.M&E of transition requires measurement of factors that are not as commonly assessed in traditional M&E of health programs.Health programs typically emphasize rigorous and scientific M&E methodologies to ensure high data quality and accuracy. However, transition processes may differ in the sense that they rely heavily on the effective management of relationships between different stakeholders, most notably between donors and the recipient organization. Thus, when designing an approach for M&E, it may be beneficial and necessary to allow for a trade-off between scientifically rigorous data collection and quality metrics on the one hand and involvement of the right stakeholders on the other hand. Poor data collection processes or quality can gradually be improved, but there is no substitute for involving the right stakeholders from the outset.The transition process is rarely easy as organizational change is occurring at various levels in a health program. Thus, introducing M&E as an additional component in the process of transition holds several challenges unique to the transition context. First, a lack of buy-in from program recipients can hinder M&E of transition as evaluators may be unable to access post-transition data to assess effects on service delivery and impact after transition. Second, breakdowns in the relationship between donor and program recipients may hinder the accessibility of those conducting M&E as the program is transitioning to the program recipient. Further, and similar to regular evaluations, it will be critical to have reliable baseline data for transition M&E, where it may be particularly important not to assume that performance of the program prior to transition was optimal. Third, although the resources for managing the transition process itself may be easier to obtain, dedicated financial and human resources for transition M&E may be more difficult to secure during a time of financial constraints. M&E of the post-transition period would be most difficult to manage. Donors might have limited resources and leverage to request support for and engagement in the M&E of the post-transition period, given the withdrawal of funding. Arrangements for access to M&E data post-transition should be ideally negotiated and agreed to during the pre-transition period, while the donor maintains influence on programming. Fourth, different dimensions that are monitored during transition change at different speeds. Dimensions such as political commitment can change rapidly while others such as local stakeholder capacity change more predictably. Those involved in M&E need to consider what timelines are practical for data collection and analysis, as well as how often recommendations and feedback should be provided to stakeholders. Finally, there may be resistance to transition from within the donor organization, its implementing partners, and/or the program recipient organization. Transition shifts power and resources, creates additional work, and may bring about unwelcome shifts in organizational priorities. All of these factors may create resentment and negativity toward the transition process, and, by extension, to the M&E of transition.Our study faces several limitations. Our paper is based in part on author experience, creating the potential for biased interpretations and presentations. However, given the limited literature on the M&E of transition, we felt that integrating stakeholder interviews and consultations with our own experience in the field was critical in presenting a balanced final product. Also, although our framework is based on collective practical experiences, it has not been used prospectively in any program transition. Without having piloted the framework in the field, we are uncertain of the final utility of our thinking, and thus, we seek feedback on experience with applying this framework to monitor and evaluate health program transition. Finally, in this paper we focused our discussion on the transition of large-scale programs  and did not explore the transition of small-scale projects.Transition M&E can offer important benefits. The discipline of thinking through what transition entails and how best to describe and measure it can provide greater conceptual clarity to the whole transition process. M&E of transition can also help inform countries undergoing transition about how best to manage it—in terms of learning not only from other countries that have undergone the process but also from their own transitions over time. Such real-time learning through M&E can help identify potential problem areas before they manifest into more serious issues. M&E of transition may also provide an element of accountability for donors, allowing them to be assured that the transition process was executed with attention to detail and with overall sustainability of the program in mind. Finally, M&E of transition allows an opportunity to engage extensively with local stakeholders in the process of transition, ensuring that concerns and needs are appropriately shared as the program is being transitioned to local ownership.Given the major shifts currently taking place in the development assistance landscape, ensuring effective health program transitions that sustain key health outcomes is likely to be a high priority for years to come. To date, M&E of transition processes has been relatively neglected, thus constraining the ability to learn from transition. Greater investments and stronger methodological work on the M&E of health program transition are needed. Piloting the proposed framework and other approaches for M&E transition would be one of the important first steps assisting our collective thinking about how to ensure that the accomplishments and health gains to date are not compromised during upcoming transitions.Greater investments and strong methodological work on M&E of transition are needed.


Stepping On is a small-group, self-efficacy based, 7-week community workshop designed to reduce falls. It addresses four major areas: strength and balance exercises, medication review, home modification, and vision. Sessions are facilitated by a trained leader and a peer co-leader. Physical therapists teach participants to perform and advance balance and strength exercises during three sessions and a pharmacist, low vision expert, and community safety expert attend one session each. A randomized controlled trial, published in 2004, showed Stepping On participants had a 31% reduction in falls compared to controls  background as a health professional, allied health professional, or fitness expert; (2) experience facilitating an adult self-management program; and (3) professional experience working with older adults. Second, we better defined the target population for the program. Individuals who use a standard walker for indoor ambulation may be too frail to benefit from Stepping On, and may require a more individualized approach. In addition, older adults with impaired cognition may not be able to participate fully. Third, we learned that sponsoring organizations need to clearly understand what is involved in implementing Stepping On before committing to its success. With the CDC, we developed an implementation guide  to help Once we had refined the program for national dissemination, we needed a structure to house it. We created the Wisconsin Institute for Healthy Aging (WIHA) to foster successful dissemination of evidence-based health promotion programs in Wisconsin, and national dissemination of Stepping On. WIHA now trains Stepping On leaders and master trainers, licenses organizations to deliver the program, and provides technical assistance and updates. Master trainers observe one workshop session for each new leader they have trained, providing coaching after the session to ensure fidelity. Once a leader has successfully delivered two workshops and passed a fidelity check, he/she may become trained as a master trainer.Stepping On has been implemented in Wisconsin and 19 other states with over 7,000 older adults participating to date. Community-based organizations value the program, and WIHA’s training and coaching results in successful adoption and high-fidelity implementation. Older adults enjoy Stepping On and recruitment is relatively easy. Invited experts, once having participated, want to continue. Since we reconfigured the implementation package based on root cause analysis, the program has been highly effective. Evaluation of 2,018 participants from 2008 to 2011 showed a significant 50% reduction in falls from 6 months before to 6 months after the program.p = 0.056). Currently, to help organizations with start-up, WIHA provides coaching, a pre-leader training webinar, and a wide array of materials through its website . Additionally, WIHA hosts a leader listserv, quarterly newsletter, and an annual Healthy Aging Summit where leaders learn and exchange ideas with researchers, community partners, and health care providers.A number of challenges hamper implementation and sustainability. For example, some organizations struggle to identify leaders and guest experts for the workshop. To overcome barriers to adoption, WIHA piloted a coaching intervention to help organizations implement Stepping On. The intervention, based on a process improvement methodology called NIATx , was effProgram reach is also a challenge. Implementation is limited among African-American, Hispanic, tribal, and other minority cultures. In response to this need, we are working on an adaptation, “Pisando Fuerte,” for Spanish-speaking seniors. Such adaptations are urgently needed to extend benefits of this evidence-based program. Increased funding will help expand Stepping On’s reach. Title III-D of the Older Americans Act provides minimal funds for the aging network to implement evidence-based health promotion programs. There is no reimbursement (yet) through Medicare or Medicaid, and little investment from insurance or health maintenance organizations. While increasing participant fees would help fund program implementation, it would hinder participation by low-income older adults. We need policy changes that enable all at-risk older Americans to benefit from this effective program.We have successfully translated Stepping On from research to practice. This translation has been possible only through united efforts of researchers, policy-makers, and community agencies. Such a combination of stakeholders, dubbed the “triangle that moves the mountain” , createsDr. Jane E. Mahoney is a Co-Author of Stepping On: building confidence and reducing falls in older adults. Leader Manual, 3rd North American Edition, Freiberg Press, Cedar Falls, IA, USA.This paper is included in the Research Topic, “Evidence-Based Programming for Older Adults.” This Research Topic received partial funding from multiple government and private organizations/agencies; however, the views, findings, and conclusions in these articles are those of the authors and do not necessarily represent the official position of these organizations/agencies. All papers published in the Research Topic received peer review from members of the Frontiers in Public Health  panel of Review Editors. Because this Research Topic represents work closely associated with a nationwide evidence-based movement in the US, many of the authors and/or Review Editors may have worked together previously in some fashion. Review Editors were purposively selected based on their expertise with evaluation and/or evidence-based programming for older adults. Review Editors were independent of named authors on any given article published in this volume.


Increasing evidence has suggested that well-resourced programs are more likely than low-resourced programs to deliver comprehensive care to individuals with co-occurring conditions and achieve greater client access and retention. But there is limited understanding of the service delivery process that allows high-capacity programs to reduce their wait times for treatment entry and their rates of dropout.To examine the indirect association of program capacity factors with wait time and retention via mental health and HIV prevention services in publicly funded addiction health services (AHS).We conducted multilevel cross-sectional analysis of 108 publicly funded program data merged with Los Angeles County Participant Reporting System data from 2010 to 2011 for 13,478 adults entering AHS. Multilevel negative binomial regression models were used to test direct and indirect relationships between program capacity and days to enter treatment (wait time) and days in treatment (retention).Findings show that compared to Whites, Latinos and African Americans served in high-capacity programs reported shorter wait times and higher retention rates. As hypothesized, the role of HIV testing and mental health service coordination played an indirect role in the relationship between program capacity and shorter wait times.Findings suggest that coordinated comprehensive services in AHS may contribute to reduction of outcome disparities in access to care. Delivery of comprehensive care may allow programs to develop effective networks to reduce wait time. Implications for health-care reform implementation and program planning are discussed.


This study aimed to assess current education and practices of emergency medicine (EM) residents as perceived by EM program directors to determine if there are deficits in resident discharge handoff training. This survey study was guided by the Kern model for medical curriculum development. A six-member Council of EM Residency Directors (CORD) Transitions of Care task force of EM physicians performed these steps and constructed a survey. The survey was distributed to program residency directors via the CORD listserve and/or direct contact. There were 119 responses to the survey, which were collected using an online survey tool. Over 71% of the 167 American College of Graduate Medical Education (ACGME) accredited EM residency programs were represented. Of those responding, 42.9% of programs reported formal training regarding discharges during initial orientation and 5.9% reported structured curriculum outside of orientation. A majority (73.9%) of programs reported that EM residents were not routinely evaluated on their discharge proficiency. Despite the ACGME requirements requiring formal handoff curriculum and evaluation, many programs do not provide formal curriculum on the discharge transition of care or evaluate EM residents on their discharge proficiency. Millions of patients are seen in the emergency department (ED) with approximately 86% rate of discharge.8Evidence on how to ensure ideal handoffs has been limited, but multiple sources have identified process standardization as an opportunity for quality improvement.12Although there are clear mandates to ensure that handoffs are standardized, and formal handoff curriculum and evaluation are provided to emergency medicine (EM) residents, there is no information available to identify the current practices of EM training programs on the discharge transition of care.The objectives in this study were to (1) assess the current scope of discharge training among EM residency programs by surveying their residency leadership, (2) assess current educational and evaluation practices related to discharge training, and (3) identify whether additional training is necessary based on the current practices and perceived competencies.This survey study was guided by the six-step Kern model for medical curriculum development.The institutional review board at Alameda County Medical Center  granted exempt approval for this study. Members of CORD were invited to complete the survey electronically. The CORD e-mail listserve is exclusive to educators in EM residency programs and includes associate, assistant and primary residency program directors from the 167 ACGME-accredited EM residency programs. Program leaders were recruited via the CORD listserve from March to April 2014. The survey was opened for six weeks in which 87 identified programs responded. Duplicate responses were reviewed and clarified with program directors directly in April 2015 (22 programs). Two programs requested their duplicate responses be deleted and completed new surveys to accurately represent their current practice. All other programs selected their most accurate survey responses. Direct emails were then sent to program directors with links to the survey in April 2015; 32 additional programs completed the survey and the survey was closed on April 23, 2015. Only programs that identified their program name were included in the study to ensure that all surveyed programs represented ACGME-accredited programs and to avoid potential duplicative responses.A total of 119 programs were surveyed, making the overall response rate 71.2% of the 167 currently accredited ACGME EM residency training programs. A majority of programs indicated that residents are given informal education regarding discharge training by senior residents and faculty (87.4%); just under half provide formal curriculum at orientation (42.9%) and/or outside of orientation (5.9%). A small percentage of programs offer no training (6.7%) . Over haMost residency programs reported using a structured discharge system in the ED , while a small minority report using none  or being unsure if this is provided in their main ED . For those with a structured process of discharge most reported that this is being performed most of the time  or always . A majority of programs reported providing structured written modifiable written discharge instructions  and bidirectional conversations with patients  . A majorAll programs reported a variety of tools to assist in the discharge process with only 34, or 43%, of the respondents being satisfied or extremely satisfied with these tools . Over twOver three-quarters  of program leadership reported that junior level residents  are “somewhat competent” in their discharge competency. Almost a quarter felt that their junior residents were “competent” . One program (0.8%) reported their junior residents were extremely competent . For senior level residents, described as residents with over eight months experience, their program leadership identified them as competent  or extremely competent . A minority of programs reported their senior level residents as “somewhat competent” in their discharge skills . None of the respondents reported their junior or senior level residents to be incompetent in their discharge abilities.These results provide insight into the discharge educational practices and clinical training experience surrounding discharge of EM residents as reported by their program leadership. Standardized formal training and evaluation is not the current norm at most programs. Most formal training provided to EM residents is at their orientation, and few programs offer formal educational opportunities beyond the first few weeks of training. Since a majority of program leaders indicated that ideal educational practices would include formal training at orientation and/or workshops or classes outside of orientation, programs may value structured education of discharge competencies but may be constrained by other limitations such as faculty time, didactic scheduling or curriculum availability. The same gap was seen between current training and ideal training for evaluation processes. While most programs do not perform formal evaluations on their residents’ discharge competency, a majority of programs identify that this would be an ideal practice. This implies that EM program leadership values formal evaluation of their residents’ discharge competency but they may be constrained by limitations such as a recognized evaluation tool and/or faculty time. These data also suggest that while program leadership values discharge competency training and evaluations, this education may not be valued as a high priority since most programs perceive their senior level residents to be competent. Although it is difficult to fully endorse competency without a standardized evaluation process, there is support that informal evaluation may be valid in identifying residents’ clinical competencies.16The fact that this study relies on perceptions of program leadership is a major limitation as there is no gold standard to formally measure discharge competency even for programs providing formal evaluation of their residents’ discharge competency. While most program leadership feel that their current process of discharge is “safe and effective,” this perception may be limited. Given that each respondent based their program results on his or her perceptions of the discharge education and performance of residents within the clinical environment, construct underrepresentation and irrelevant variance represent threats to the validity of clinical performance ratings in this study.Program leadership relies on general gestalt that their residents are competent in their discharge proficiency since most are not evaluating this competency. Program leadership may not identify any limitations in senior resident discharge competency because there may be larger, more systematic failures of discharge communication with patients and/or caregivers. In this scenario, resident performance of discharge may be at an acceptable level at the departmental level but departmental expectations of patient discharge competency may not be meeting the patient needs to create safe and effective discharges. Standardizing the process of handoffs between providers in the hospital environment has demonstrated improvement in the quality of communication and increased patient safety in the clinical arena.The conflation of the concept of “safety” and “effectiveness” may be another limitation of this survey. Programs were asked about safe and effective discharges in a single question. These two concepts may be inappropriately linked together and it may be that safety may exist without being effective and vice versa. This may represent a construct error in the survey design and affected results of this survey.Lastly, a major limitation of this study was the gap of approximately one year between the surveys of the first cohort of surveyed programs (87) and the second (32). Although it is unlikely that most programs changed their educational practices dramatically within that time period, it is possible.Further work should focus on the more structured assessment of resident discharge competency through direct observation and evaluation of resident performance to corroborate program leaders’ assessment of resident competence. These evaluation tools should then be validated and studied in a clinical setting with specific process measures and patient outcomes. Following Kern’s six-step model for curriculum development, the next step would be to create specific goals and objectives of the discharge curriculum and develop educational interventions aligned with these goals. Curricular tool suggestions that have been made to structure education and evaluation for provider-to-provider handoffs might be used directly or modified to educate residents in the discharge transitions of care.18The results of this targeted needs assessment indicate a lack of structured training and assessment of resident discharge competency despite current guidelines for formalized training in all handoffs. Although most programs reported senior residents are competent in discharge proficiency, the residents’ training is primarily informal which may lead to significant variability in resident experience and performance. Further research should be aimed at assessing proficiency of resident discharge performance through objective observation with validated evaluation tools. Structured training and assessment recommendations should follow from this research with increased attention to implementing a standard curricular model or toolbox, objective, valid evaluation methods, and identification and management of high-risk discharges.


Practice accreditation is a widely used method to assess and improve the quality of healthcare services. In the Netherlands, a practice accreditation program was implemented in primary medical care. We aimed to identify determinants of impact of a practice accreditation program, building on the experiences of primary care professionals who had participated in this program.An interview study was done to document the experiences of 33 participating primary care professionals and used to identify determinants of outcomes. The Consolidated Framework for Implementation Research (CFIR) was used as framework for the qualitative analysis.After analyzing 23 interviews saturation was reached. The practice accreditation program is based on structured quality improvement, but only some of its elements were identified as determinants of impact. Factors that were perceived to facilitate implementation of the program were: designating one person responsible for the program, ensuring clear lines of communication within the whole practice team and having affinity with or stimulate enthusiasm for improving quality of care. Contextual factors such as participation in a care group and being connected to the GP educational institute were important for actual change. The accreditation program was perceived to have positive effects on team climate and commitment to quality of care in the practice team. The perception was that patient care was not directly influenced by the accreditation program. Receiving a certificate for completing the accreditation program seemed to have little added value to participants.Practice accreditation may have positive outcomes on quality of care, but not all planned elements may contribute to its outcomes. Both factors in the accreditation process and in the context were perceived as determinants of quality improvement. The challenge is to build on facilitating factors, while reducing the elements of accreditation that do not contribute to its impact.The online version of this article (doi:10.1186/s12875-015-0294-x) contains supplementary material, which is available to authorized users. Accreditation and certification are widely used methods to assess and improve healthcare services. These are complex interventions, which typically comprise an audit of a healthcare provider, an assessment of performance, followed by formal allowance of accreditation or certification. Accreditation programmes can have positive effects on quality and safety of clinical care and organizational performance . WorldwiIn the Netherlands, primary care practice accreditation is a voluntary activity comprising a comprehensive audit, which covers clinical and organizational domains, followed by structured planning of improvements and formal review by an external assessor . The proThe aim of this study was to identify determinants of impact of the practice accreditation program, building on the experiences of primary care professionals who had participated in this program.A qualitative study was conducted, which was linked to a cluster randomized trial of the practice accreditation program in the Netherlands . All parThe study was undertaken to identify determinants of impact of the practice accreditation program, building on the experiences of primary care professionals who had participated in accreditation program in primary care; therefore a qualitative method was appropriate.The primary care practice accreditation program in the Netherlands has been offered on a voluntary basis since 2005. Practices have to comply to few minimum standards in order to be eligible for participation . The praParticipants in the study were team members of the primary care practice with a coordinating role in the implementation of the practice accreditation program in the primary care practice.One semi-structured interview with one team member per practice was conducted. All interviews were held by one person (EN), a health scientist and physiotherapist. All 41 practices included in the cluster randomized trial  were appThe interview guides were developed on the basis of literature  and duriAll interviews were audio-taped and transcribed verbatim. Interviews were repeatedly read and analyzed in an iterative approach by three researchers, who independently coded the transcripts of the interviews, followed by collaborative interpretation and consensus. We came to agreement after discussion.Interview data were analyzed until saturation was reached, that is, no new codes were generated. A stepwise analytical approach was used . We provThe Medical Ethics committee Arnhem-Nijmegen waived approval for this trial after assessing the study protocol (file number 2008/258). Participants all consented to recording of the interviews. All data was coded and processed anonymously.All 41 participating practices were invited for the interview study. Eight practices declined to participate in the study due to lack of time or sickness among staff. Therefore, interviews were held with 33 individuals in the year 2012. Interviews lasted from 17 min until 46 min. Table The practice accreditation program was externally developed by the DCGP . Element‘What we disliked was that we were obliged to make a plan on the four most common chronic diseases, you have to do this, you have to do that. When you indicate you have other priorities for the improvement plans, you still have to implement a plan on topics they have made mandatory. That felt annoying sometimes.’ (respondent 13)The adaptability of an intervention is the degree to which the intervention can be tailored to the needs of the organization. A core component in the practice accreditation program is developing improvement plans on four chronic conditions mostly treated in general practice. These plans are tailor-made, using the feedback reports to guide their focus, and therefore be consistent with the needs of the practice. The program did provide additional support for developing and implementing improvement plans, which is important in applying elements of quality improvement. However, some participants experienced there was no possibility to implement their own priorities using improvement plans.‘General practitioners prefer to do the things they enjoy, this (the practice accreditation program) takes a lot of extra time and workload.’(respondent 13)The first step in the practice accreditation program is collecting patient related data on four chronic diseases. Many participants experienced this as the most time-consuming and difficult step of the program. Furthermore, other elements of the program such as developing improvement plans and implementing these plans were experienced as a heavy burden as these are supplementary tasks in addition to daily practice.The intervention consisted of a workbook, a supporting website and the obligation to contract a trained consultant to assist the practice through all steps of the program. Another component is the practice visit of an assessor to assess improvement plans and minimum standards. Practices in this study were in general not content with the supporting website which was found unclear and slow. Experiences with the assistance of the consultant varied. Furthermore there was a lack of consistency in assessment methods of assessors which caused confusion on how to interpret and execute the program.‘We think it is a lot of work that has to be done, making improvement plans, evaluating the plans, it takes a disproportionate amount of time actually.’ (respondent 26)‘I have to pay a considerable amount of money to do all kinds of things that I also would do without paying the money.’ (respondent 13)All respondents expressed their dissatisfaction with the high costs of the intervention. Furthermore they questioned the benefits of the program, in particular in relation to the costs.‘We used results from the patient survey to inspire us in choosing a topic for the improvement plans for this year. There were especially complaints regarding privacy.’ (respondent 26)In the practice accreditation program participants are obliged to conduct a patient satisfaction survey. Based on these outcomes several participants defined the aims of their improvement plans so that the needs of patients could be met.However, participants perceived that patient care was not directly influenced by the accreditation program as the program had no direct influence on patient-caregiver interactions.All practices in the study were affiliated with a chronic care group. In addition to the practice accreditation program, participants in the study mentioned participation in a chronic care group as a contextual factor that positively influenced the quality of the care they provided. Some of the practices were, as training practices, connected to an institute for vocational training of GPs. Peer review meetings for GPs working in training practices appeared to be a positive influence on their attitude towards quality improvement.‘One of our staff members was appointed for several hours a week to focus specifically on quality work. Not only for the practice accreditation program but also for the other certification program we participate in. These are preparations to ensure the whole team is involved in the process.’ (respondent 44)The most important extrinsic reasons to participate in the program were a financial stimulus for GP training practices and the requirement of insurance companies to demonstrate how quality is managed within the practice. Participation in research projects, nationally organized projects (on registration behavior) and participation in other certification programs all provided a positive influence on implementation of the program.‘Well, when you lose staff members because of resignation or illness, it causes major problems. First priority is to keep the practice running and then there is no time left to spend on tasks concerning the practice accreditation program.’ (respondent 46)In small practice organizations, lines of communication were clear which is beneficial for the implementation of the practice accreditation program. However in solo-practices all tasks concerning the program had to be performed by one person. Furthermore, when a practice lost staff members due to illness or resignation, there was no priority for the program and it also implied the loss of knowledge about the program.I think it’s a generational thing. I have the feeling older GPs consider it more difficult to work according to the practice accreditation program than younger GPs.’ (respondent 15)‘The age of general practitioners was mentioned as a factor associated with the enthusiasm with which the program was accepted for implementation.The practice accreditation program requires the involvement of the whole team. Therefore it is advised to organize structural team meetings to evaluate the progress of improvement plans. Participants experienced implementation of the program as more effective when indeed all members of the team were involved and processes were structurally evaluated in team meetings.‘I think we have team members with a critical attitude. All our assistants have a bachelors degree, which is uncommon.’ (respondent 34)The majority of participants in this study had affinity with improving the quality of care they provided, prior to participating in the program, which benefits implementation. Furthermore participants mentioned that the motivation and education level of team members was of influence on the implementation of the program.The degree of motivation regarding implementation of the program may be dependent on the function of the staff members. Some GP assistants experienced the program as a burden while practice nurses had no difficulties implementing the program. Furthermore, in some practices not all GPs believed the program is beneficial and were therefore less motivated to implement the program.‘We wanted to be more conscious of the quality of care we provide and we wanted to reveal our blind spots. The most important reason to participate in the program was to improve the quality of care we provide.’ (respondent 24)Participants started with the program while it provides support when improving quality of care in the practice:‘I often started new things (new procedures) without completing them. The advantage of the accreditation program is that it forces me to complement the circles to implement new approaches in a constructive manner.’ (respondent 22)The program provided tools to work systematically:In the initial stages of the program participants required more assistance from the consultant than in later stages of the program. They then became more accustomed to working according to a quality cycle and the program was more integrated in daily practice.Some participants were motivated to participate in the practice accreditation program because they were also employed in another function relating to quality of care.‘I’m no computer expert, I need help with that and I think that also applies to some of my colleagues’. (Respondent 38)Most participants made no preparations before they volunteered to participate in the program. The practice accreditation program consists of various elements Fig. . Practic‘It (the feedback report) shows the benefits of my efforts and indicates in what areas I should plan improvements. It is very difficult to reflect on the feedback reports sufficiently. I have to spend time to study it, to think about it and reflect on it. You should be able to discuss it with your team. The rush of daily practice leaves no time for this and that is very unfortunate.’ (Respondent 44)Based on the data on four chronic diseases practices receive a feedback report with benchmarks which provides insight into their medical practice. This information was considered to be important however it had little influence on improvement plans. A possible reason is it is difficult to adequately reflect on the outcomes.‘No, we do not look into it that much. This is our practice and we manage it in our own way.’ (Respondent 29)However, some participants considered the feedback report of minor importance.I have to come up with three new plans for this year. You have to be careful you don’t make up things only because the auditor is coming.’ (Respondent 45)‘Another element of the practice accreditation program was the formulation of improvement plans. These plans were in general drafted and implemented by all team members. The practice consultant provided useful feedback on the plans in the first year of the program. However, the subject of improvement plans can be determined to a certain extent only.‘I have noticed over time that the assessors all have different backgrounds. They assess the practice in a non-similar manner. The things that are important differ for various assessors.’ (Respondent 34)Visitation of an assessor is the next element of the practice accreditation program. During this visit the assessor audits the practice. Results of the audit seem to depend on which assessor visits the practice.‘As the manager I have the time to perform accreditation-related tasks. So I took the initiative, otherwise it would not have been a success. We talk about it over dinner, so to speak, so the reflection process is already started. And then at one point I nag that he really has to write those plans, and then he picks up the voice recorder and begins.’ (Respondent 29)Some of the partners of the GPs were team member of the practice team. This appeared to be a highly stimulating factor in implementation of the program.‘One of our assistants had just finished a management training, that was our benefit. We appointed her as coordinator of the practice accreditation program.’ (Respondent 40)When starting with the program some participants expanded responsibilities of other team members for the purpose of guiding the implementation of the program.Every year the practice is audited by an auditor who assesses the objectives of implemented improvement plans and approves new improvement plans. This annual visit is for most participants an important motivator for continuous quality improvement and to keep implementation of improvement plans on the practice agenda.Quantitative feedback about the progress of implementation of the program was provided by feedback reports at the start of the program. It is required for practices to define their improvement plans with a measurable goal. After a year they have to provide evidence that goals have been achieved. Furthermore team meetings were regularly held, as required by the practice accreditation program, to monitor progress of implementation.‘Very often issues are not mentioned because it is difficult to give one another feedback. Now we succeeded in establishing a safe work environment where we can provide each other feedback in a constructive manner.’ (respondent 40)As a result of participation in the program, team members were more motivated in performing their work and their responsibility increased as a consequence of participation in the program. Overall a better team spirit emerged.The aim of this study was to identify determinants of impact of the practice accreditation program, building on the experiences of primary care professionals who had participated in a practice accreditation program in primary care. The presence of a team member who has the specific responsibility for the program, appeared to be a stimulating factor. The practice accreditation program had positive effects on team climate and caused more sense of responsibility for quality of care among all team members. Health care professionals perceived no direct influence by the accreditation program on patient care. Audit and feedback is a crucial element of the accreditation program, however choices for improvement plans were rarely based on feedback reports. Receiving a certificate for completing the accreditation program seemed to have little value to participants.As shown in a Cochrane review  audit anIn this study staff responsibility for quality was identified as an important implementation facilitator, which was also demonstrated in a previous study . SimilarAs a response to this and other evaluations, the Netherlands Institute for Accreditation in Healthcare has adjusted the practice accreditation program to make implementation more feasible and flexible. Data collection has been spread over different years, improvement plans can be documented in a more flexible way, and the use of external advisors is optional. New evaluation is required to examine the impact of these changes on feasibility and outcomes.The strength of this study lies in the qualitative approach which gives us more information on the working elements of the practice accreditation program. The CFIR framework was used to organize the data in this study. All domains described in the CFIR were represented in the results. As the CFIR framework was only used in the last stage of coding, the risk of overlooking material that does not fit in the constructs, was small. General practices in this study all voluntarily applied for the practice accreditation program and subsequently they voluntarily participated in this study. This could imply that participants included in this study had a more than average affinity with quality of care and were motivated to change. Therefore it is recommended to conduct further research in the late majority population. In this study only the quality coordinators of the practices were approached for participation. A focus group study with all team members of the practice could have resulted in additional outcomes to provide more understanding of mechanisms of action regarding implementation of the program. Furthermore, the data collection method we choose might have been inadequate as with face-to-face interviews more in-depth and nuanced data can be collected.This study contributes to the body of knowledge on determinants of impact of a practice accreditation program implemented in general practice. Determinants perceived of importance in this study have to be further explored in future research. However, the results of this study provide feasible, ready to use suggestions, and therefore can be relevant for general practice teams, practice managers and policymakers. An example of such a ready to use suggestion is: designate one responsible team member, to facilitate the implementation of a practice accreditation program.Consistent with results from this study, previous research has shown that accreditation results in improved teamwork, improved access to care, increased awareness of patient safety, improved practice systems and care processes and improved quality of care . NeverthAcross the world, practice accreditation is an established strategy for assessing and improving healthcare practices. Nevertheless, there remains a need for better insight into the factors and processes related to its impact in order to optimize existing accreditation programs.


The universal school-based prevention program has shown its efficacy in several studies when implemented by psychologists. The current study compares the effects of the program as implemented by teachers versus that implemented by psychologists under real-life conditions. A total of 646 vocational track 8th grade students from Germany participated either in a universal prevention program, led by teachers (n = 207) or psychologists (n = 213), or a teaching-as-usual control condition (n = 226). The design includes baseline, post-intervention, and follow-up (at 6 and 12 months post-intervention). The cognitive-behavioral program includes 10 sessions held in a regular school setting in same-gender groups and is based on the social information-processing model of social competence. Positive intervention effects were found on the change in girls’ depressive symptoms up to 12 months after program delivery when the program was implemented by psychologists. No such effects were found on boys or when program was delivered by teachers. The prevention program can successfully be implemented for girls by psychologists. Further research is needed for explanations of these effects.The current study evaluates a depression prevention program for adolescents led by psychologists  Onset of major depression often occurs during adolescence  and is aPrevention is considered one of the most effective strategies to reduce the burden associated with psychological diseases by the World Health Organisation . HoweverThe Institute of Medicine’s  report oTo overcome the burden associated with depressive disorders these results of effective prevention strategies are promising. Nevertheless the question remains how a widespread dissemination of these effective prevention programs can be organized. In addition, we do not know much about possible moderators  of successful outcome of such a school-based, universal program.i.e., universal, selective, indicated), participant attributes , characteristics of the intervention , interventionists , and timing of assessments . The timing of assessments is relevant as effects of prevention in general and universal prevention in particular depends on the increase of depressive symptoms in the control group. Thus, it is not to be expected that prevention programs show effects on depressive symptoms in the short term . Consistent with this expectation, a review found an increase in the magnitude of effect sizes of prevention programs on depressive symptoms at 6-month follow-up compared to baseline-post-intervention comparisons  but did find statistically significant effects at follow-up . Therefore we would not expect any effects to manifest immediately after the intervention took place. ES’s also differed depending on qualification of group leaders.Meta-analyses of studies aimed at preventing depression in children and adolescents have concluded that some efficacious interventions for the prevention or reduction of depressive symptoms in youth exist e.g., ,13,14). ,14. 13,1iew, see ). This hIn school-based prevention programs of depression either psychologists or teachers are often used as group leaders. Both groups have their own advantages: psychologists are experts on psychological problems and their prevention and intervention  and havevs. school, schools with different socioeconomic background), with different participants , and often with slightly different ways of adoption of the program , it is difficult to compare the results, and therefore, no conclusion can be drawn in regard to the relative effect of teachers and psychologist as group leaders. For instance, in a meta-analysis of the effects of the RPR, Brunwasser, Gillham, and Kim )We did not find positive intervention effects for girls or boys on depressive symptoms in the teacher-led condition. This result is in line with a meta-analysis that found long-term positive outcome only when healthcare professionals delivered the prevention program, in contrast to classroom teachers . It is aBesides these moderating group leader variables, other factors could have influenced the outcome, including: group leaders’ social competence , motivation, attitude toward and acceptance of the program, and more intensive and individualised supervision ,25,64,65et al. [The observed gender effect cannot be explained by regression to the mean as suggested by Stice et al.  because et al. , did notet al. tested whether conduct behavior could explain the gender effect they found and found that after controlling for conduct behavior, the gender effect in which girls but not boys benefited from participating in a prevention program remained [Because we found a gender effect only in the psychologist-led condition, the most likely explanation could be disciplinary problems within boy groups. Psychologist are less experienced in handling difficult, dominating, and demanding boys with divergent cultural background, as we find them in the German vocational track school system. This type of school is the track of the tripartite school system in Germany, with a large portion of challenging students . Espremained .et al. [The result that boys in the psychologist-lead group not only did not exhibit an improvement in depressive symptoms but even deteriorated could be an iatrogenic effect. In the study of Kindt, Kleinjan, Janssen und Scholte  both genet al.  tested tet al.’s [For the interpretation of the present results, it is important to consider several limitations. First, missing data and absenteeism of students are a constant problem. Some students do not take part at all in the assessment and others missed school and therefore the assessment at one point or another. However, in our study the rate of students who took part at all assessments was very high. This high rate can be explained by the German school cohort system where all students of one cohort take classes together and therefore are motivated to participate in same activities. This is consistent with Masia-Warner et al.’s  recommenversus intervention effects. Even if in most variables the students of the control group did not differ statistically from those from the experimental group, the smaller ratio of students with migration background could mean that they were the “easier classes”. If that were the case, it could have a positive influence on the expected development of depressive symptoms, so a bias against the positive effects of the program is to be expected. In addition the two groups might differ in other influential variables which were not surveyed. Although the lack of school-level randomization lessens the statistical validity, it does contribute to the ecological validity, as in real life schools and teachers decide themselves if they want to implement a program and this motivation will influence the results of their endeavors [A further limitation is the missing randomization on school level. This is a problem often seen in implementation and efficacy studies, and it could lead to a selection bias of schools and a conflation of school ndeavors .Another limitation is that the students in the control condition took part in their regular school curriculum, which is, in fact, a common procedure in most prevention studies. Including a more active control condition in future research is necessary to separate program specific effects from non-specific factors like attention and support by an “enthusiastic and charismatic” group leader . HoweverThe exclusive use of self-report measures is another limitation of our study. Although previous studies have shown that adolescents are a reliable and valid source of information for depressive symptoms ,73, sociFinally, we did not measure whether the skills trained in LARS & LISA were implemented by the students in their daily life. Addressing these outcomes by behavioral observation measures and modern electronic devices should be part of future research.Summarized, positive intervention effects were found on the change in girls’ depressive symptoms up to 12 months after program delivery when the program was implemented by psychologists. No such effects were found on boys or when program was delivered by teachers. The prevention program can successfully be implemented for girls by psychologists. Future research needs to address the question of what makes psychologists compared to teachers more successful in preventing depression in adolescents and what possible mechanisms may explain why girls benefit more from such a prevention program than boys. Research should focus on factors concerning group leaders’ social competence , motivation, attitude toward and acceptance of the program, and intensity and individualization of supervision.


The pharmacology PT program has sent samples to eight laboratories in four rounds of testing. WWARN technical experts have provided advice for correcting identified problems to improve performance of subsequent analysis and ultimately improved the quality of data. Many participants have demonstrated substantial improvements over subsequent rounds of PT. The WWARN QA/QC program has improved the quality and value of antimalarial drug measurement in laboratories globally. It is a model that has potential to be applied to strengthening laboratories more widely and improving the therapeutics of other infectious diseases.Comprehensive assessment of antimalarial drug resistance should include measurements of antimalarial blood or plasma concentrations in clinical trials and in individual assessments of treatment failure so that true resistance can be differentiated from inadequate drug exposure. Pharmacometric modeling is necessary to assess pharmacokinetic-pharmacodynamic relationships in different populations to optimize dosing. To accomplish both effectively and to allow comparison of data from different laboratories, it is essential that drug concentration measurement is accurate. Proficiency testing (PT) of laboratory procedures is necessary for verification of assay results. Within the Worldwide Antimalarial Resistance Network (WWARN), the goal of the quality assurance/quality control (QA/QC) program is to facilitate and sustain high-quality antimalarial assays. The QA/QC program consists of an international PT program for pharmacology laboratories and a reference material (RM) program for the provision of antimalarial drug standards, metabolites, and internal standards for laboratory use. The RM program currently distributes accurately weighed quantities of antimalarial drug standards, metabolites, and internal standards to 44 pharmacology,  Accurate blood or plasma drug concentration measurement is an essential component of the assessment of therapeutic failure so that true resistance can be differentiated from inadequate drug exposure. When pharmacokinetic data are assessed, it is necessary to differentiate between true patient population differences and differences in accuracy between assay methods or laboratories.In Vitro, Molecular, Clinical, Antimalarial Quality, and Informatics) located in centers of excellence around the world, offer a platform for collaborative research into the understanding, identification, and spread of antimalarial drug resistance. Each group specializes in different aspects of antimalarial drug resistance and works collaboratively to develop customized tools and services designed to facilitate quality-assured data collection, analysis, and reporting. The WWARN laboratory quality assurance/quality control (QA/QC) program  scientific groups,  relationships have not been well characterized for most of the antimalarial drugs in current use, which limits the evidence base needed for prolonging their useful therapeutic life . Most PKThe WWARN QA/QC proficiency testing program for pharmacology laboratories assesses the ability of pharmacology laboratories to assay blood or plasma samples for concentrations of antimalarial compounds and their metabolites. Participation in the proficiency testing program is open to all laboratories doing either therapeutic efficacy studies or other research on antimalarial drug exposure. The program currently offers plasma-based samples for eight antimalarial drug compounds and metabolites: chloroquine/desethylchloroquine, mefloquine/carboxymefloquine, primaquine/carboxyprimaquine, amodiaquine/desethylamodiaquine, piperaquine, lumefantrine/desbutyl-lumefantrine, dihydroartemisinin, and artesunate. Commercially obtained and controlled plasma is spiked with accurately weighed certified reference materials. All active ingredients and the plasma are controlled by the manufacturer and reflected in certificates of analysis. Each analyte is sent in a range of concentrations, including the highest and lowest concentrations expected to be found in clinical samples , which aZ-scores (Z = (x − ax)/SDPA, where x is the participant result, ax is the nominal value, and SDPA is the standard deviation for proficiency assessment. The revised Harmonized Protocol (Z-scores (|Z|) are evaluated according to the following criteria: |Z| ≤ 2, the result is considered satisfactory; 2 < |Z| ≤ 3, the result is considered questionable; |Z| ≥ 3, the result is considered unsatisfactory.Each round of proficiency tests consisted of three cycles per year, starting approximately in July and ending in June of the following year. At the end of every cycle of 4 months, the anonymized results from participating laboratories were evaluated against the nominal concentration  for each antimalarial drug/metabolite. The performance of the pharmacology laboratories that are currently enrolled in the program was assessed and evaluated in terms of their ability to assay the supplied spiked antimalarial plasma samples during four rounds of proficiency testing between 2010 and 2013. Performance is reported using Protocol  describehttp://www.wwarn.org/toolkit/qaqc/reference-material-scheme) to research groups that are members of the network at no cost. The WWARN QA/QC reference material program applies principles of good weighing practice  or LC-UV detection. Laboratories were free to select their preferred assay method in order to have their performance assessed in the proficiency testing program. When performance scores over several rounds of proficiency testing for all laboratories and a particular antimalarial compound  were comZ-scores . A lineaRandom errors arise as a result of chance variations in factors that influence the value of the drug and metabolite concentration being measured. For example, electrical noise, thermal effects, or chance contaminations that are normally outside the control of an analyst may cause results to vary in an unpredictable way . These ein vitro laboratories across six continents  for systematic errors, and when corrective measures were applied, more accurate results were observed for subsequent analyses. An additional advantage for laboratories participating in the proficiency testing program is that participants may communicate their own results, including the regular program reports, privately to a laboratory accreditation or other assessment body, when required for the purpose of assessment, or to clients for the purpose of demonstrating analytical capability.A substantial improvement in antimalarial drug measurement performance was shown over the course of testing for nearly all participating laboratories, highlighting the advantage of an external quality assessment program. While two participating laboratories performed to a satisfactory degree from the start, the rest have demonstrated substantial improvements over subsequent rounds of proficiency testing. Results presented here showed a 41% reduction in overall Even in good laboratory practice (GLP)/accredited industry laboratories, it is not uncommon for internal QC samples to fail to mimic clinical study samples or externally prepared QC samples . Once thExternal QC samples will have a different background composition  than the in-house blank matrix used for preparation of calibration standards and QC samples. It is crucial to test a variety of different matrix sources during validation to evaluate these matrix effects .The ideal method with mass spectrometric detection is to use a stable-isotope-labeled internal standard. Identical structures means that this should theoretically compensate for any variation during sample preparation, e.g., increased salt content in study samples that could decrease recovery on a weak ion exchanger solid-phase extraction column. A stable-isotope-labeled internal standard has the same affinity for the active sites as the analyte and would therefore compensate for this decreased recovery. An analogue internal standard might fail to compensate because it might have higher/lower affinity for active sites than the drug of interest. An isotope-labeled internal standard should theoretically also compensate for matrix effects (see paragraph i above) since it has identical retention times on the liquid chromatography column as the analyte. However, small differences in retention times might be seen between an isotope-labeled internal standard and an analyte which in rare cases fail to compensate for matrix differences .Calibration standards and QC samples are often analyzed in order of increasing analyte concentration. The drug and metabolite concentrations of study samples or external QC samples are not known, and carryover may become a problem.Incorrect assumptions about the formulation of standards used  or dilution schemes can create a systematic bias in the calibration samples and QC samples that might be difficult to identify.Not all studies utilize the same units of concentration when reporting results, and it is the responsibility of the proficiency testing provider to specify what units are to be used to evaluate submitted results.To evaluate the potential causes for differences between actual drug and/or metabolite concentrations measured and the nominal values, the following steps can be performed fairly quickly.(i) A simple way to evaluate for systematic differences in the background matrix or an improper/suboptimal internal standard is to rerun external QC samples, both naive and also with 1:5 and 1:10 dilutions using the laboratory internal blank source. Matrix effects and/or recovery differences are minimized. If these dilutions give a result significantly different from the nondiluted sample, then the method suffers from a problem of matrix and/or recovery effects, and the internal standard fails to compensate effectively for these.(ii) Carryover can also be easily excluded by analyzing a batch of extracted blank samples and zero samples randomly allocated within a batch. Carryover does not always arise in the autosampler but could also come from steps during sample preparation. This experiment evaluates the whole method procedure.(iii) Systematic bias in preparation of stock and working solutions may be identified through careful review of preparation schemes and proper traceability of performed preparations, assessing whether all calculations and the factors for formulation and purity are correct. Some laboratories implement two different stock/working solution sets, one for standards and one for QC samples, which helps to detect if one preparation should fail.(iv) Inaccuracies with the reported units of concentrations are avoided by issuing a standardized result form to all participants. Test results should be converted to comply with the desired units of the result form.in vitro laboratories to assess whether interlaboratory variability of in vitro drug susceptibility testing could be minimized by introducing simple standardization measures. The results of this study will be used to design a proficiency testing program to improve standardization of in vitro assessment across the malaria community. The WWARN QA/QC reference material program in collaboration with the WWARN Antimalarial Quality scientific group will provide antimalarial medicine standards, metabolites, and internal standards to bioanalytical laboratories testing for drug quality. The QA/QC unit may also develop a proficiency test program for these laboratories specifically.The value of the WWARN QA/QC reference material program is evident when the program is evaluated against the cost and availability of antimalarial reference compounds, metabolites, and internal standards. Purchased individually by independent laboratories, the cost of small quantities of these compounds might be prohibitive, and the quality from small manufacturers might be questionable. The WWARN QA/QC unit negotiated a very favorable price by acquiring larger quantities, with the added advantage of a memorandum of understanding between WWARN and the manufacturer to retest the compounds close to expiry dates for recertification. This recertification process was included in the original price for the compounds. Any compounds not readily available from the manufacturer are custom synthesized and are provided with a comprehensive certificate of analysis. The certificates are distributed and updated to all participants of the WWARN QA/QC program. The potential for standardization across laboratories is now also possible since all participating laboratories have the opportunity to make use of the same reference standards for calculating a standard curve and internal QC samples. The reference material program is highly regarded by pharmacology laboratories, and the program has also distributed reference materials to in vitro assays. It is evident that similar programs might benefit researchers seeking to improve the assays used in evaluating other antimicrobials (The WWARN QA/QC program has demonstrated its potential for facilitating quality-assured antimalarial drug and metabolite pharmacokinetics and crobials .


Aim. The purpose of this study was to develop and evaluate a computer-based, dietary, and physical activity self-management program for people recently diagnosed with type 2 diabetes. Methods. The computer-based program was developed in conjunction with the target group and evaluated in a 12-week randomised controlled trial (RCT). Participants were randomised to the intervention (computer-program) or control group . Primary outcomes were diabetes knowledge and goal setting ) measured at baseline and week 12. User feedback on the program was obtained via a questionnaire and focus groups. Results. Seventy participants completed the 12-week RCT  years). After completion there was a significant between-group difference in the “knowledge and beliefs scale” of the DOQ. Two-thirds of the intervention group rated the program as either good or very good, 92% would recommend the program to others, and 96% agreed that the information within the program was clear and easy to understand. Conclusions. The computer-program resulted in a small but statistically significant improvement in diet-related knowledge and user satisfaction was high. With some further development, this computer-based educational tool may be a useful adjunct to diabetes self-management. This trial is registered with clinicaltrials.gov NCT number NCT00877851. In 2012, 21 million people in the USA . Diabetesic] their condition.” National guidance recommends that structured education should be an integral part of care planning; however its provision, particularly provision of tailored education, may be limited by access to trained educators and competing demands on healthcare staff. The UK Healthcare Commission (2007) reported that only 11% of people with type 2 diabetes had participated in any form of structured education, with attendance varying between 1% and 53% . Waist and hip circumference were measured over light clothing using an inelastic but flexible tape measure. Waist was measured at the midway point between the iliac crest and lower rib. Hip circumference was measured at the widest point around the gluteal protrusion. Blood pressure was measured twice from the right arm, using an automated Omron sphygmomanometer with the participant sitting quietly for at least five minutes. A 20 mL fasting blood sample was drawn from the antecubital vein and was processed within 2 hours. All samples were stored at −70°C for batch analysis at the end of the study. Dietary intake was assessed by 4-day food diary. Physical activity was assessed using the International Physical Activity Questionnaire . It's probably another reason for avoidance.”Diabetes education delivered by a team of educators, with some degree of reinforcement of that education at intervals, may provide the best opportunity for improvements in patient outcomes . HoweverThe evaluation of this newly developed tool to aid self-management of diet and physical activity in people with type 2 diabetes indicated that it was highly valued by users and, alongside this, its short-term use was found to significantly improve barriers related to “knowledge and beliefs” about diabetes. There was also a trend towards a statistically significant improvement in overall health status as measured by SF-36 instrument. A secondary analysis of “high” users compared to the control group demonstrated small but significant increases in the “knowledge and beliefs scale” and the “advice and support scale” of the DOQ and a trend towards significance for ADKnowl “diet and food score.” Although there was no significant change between the intervention and control group in self-efficacy, depression, diabetes-specific quality of life, body weight, or metabolic markers, there were indications of positive changes in dietary intake and physical activity between groups, albeit these were not statistically significant. The results, therefore, indicated that this tool did help patients to improve their knowledge, which is one of the key barriers to dietary self-management among people with type 2 diabetes –37. PrevThe LWD program was highly rated by participants in feedback obtained from both questionnaires and focus group discussions. This is reassuring given the level of involvement the target group had in terms of the development of the program itself and the extensive usability testing that was undertaken. Including members of the target group in the development process is a robust form of program development and has been shown to improve participation and program success , 43. TheWith regard to dietary intake, based on initial focus group discussions , the LWDA challenge with all such tools is likely to be the digital divide. A lack of desire to engage with technology was one of the main factors that influenced recruitment to the study. However, on a positive note, most of the people who took part in this study had never used a computer before and yet still managed to navigate their way through the program with ease which was reassuring as ease of use was a key priority when the tool was being designed. Furthermore, the average age diagnosis for type 2 diabetes is decreasing and computer literacy is increasing , 47. Sin all participants who used the program.The study did have some limitations. This was a trial of effectiveness and under these “real-life” circumstances compliance was not optimal; based on usage logs, only approximately half of the intervention group participants used the tool on at least a weekly basis as was recommended at the study outset. The “high user” group may represent a more highly motivated population and hence may not be as generalisable to the general population. However, despite this, some positive changes in knowledge-related parameters perceived health status, diet, and activity were observed and the trial has highlighted the need to focus attention on strategies that could be used to enhance usage, such as exploring alternative modes of delivery, such as app technology, and how this tool could be incorporated into existing diabetes care packages, as discussed above. With regard to the user feedback, only a quarter of the intervention group participants opted to take part in the focus group discussion at the end of the study. Feedback from these sessions was positive; however, it is possible that only the more motivated or interested participants opted to take part and therefore these discussions may not be representative of everyone who used the program. It was encouraging, however, that the positive attitude towards the program displayed in the focus group discussions was, for the most part, consistent with the findings from the evaluation questionnaire that was completed byThis short-term evaluation indicated that this newly developed computer-program, focusing on diet and physical activity self-management, was well received by participants and its usage resulted in small but statistically significant improvements in diet-related knowledge, as well as nonsignificant improvements in perceived health status, dietary intake, and physical activity. Based on these findings, further development of this tool is warranted. Consideration needs to be given to how the delivery platform can be widened to maximise impact, how this tool could be incorporated into existing diabetes care packages, and what level of health professional support is required to encourage usage and help maximise effectiveness of the tool. This tool may be a viable adjunct to diabetes self-management and could help to fill an important gap in patient care.Health professionals are charged with delivering a large volume of information to patients, often within a limited time slot, and human resource issues can also mean there is limited capacity for patient follow-up. Self-management tools such as this can be a useful adjunct to HCP patient education and support and can help fill important gaps in the care pathway, such as the period between diagnosis and receiving structured education, or extended time periods between patient follow-up appointments. For HCPs to have confidence in the effectiveness of any such tool, however, they should be developed in close consultation with the end users and with HCPs and should undergo rigorous usability testing, as was undertaken during this study, before wider dissemination.