Background: Longitudinal clinical experiences are a common component of undergraduate medical curricula, yet these programs have not been systematically characterized in US medical schools.Objective: Our study sought to identify and characterize longitudinal clinical programs (LCPs) in US medical schools and measure associations between programs’ structures and goals.Design: Using a mixed-methods approach, we conducted a secondary analysis of data from publicly available websites. We conducted a systematic keyword search of the websites of 137 LCME-accredited US medical schools to identify LCPs. We included programs with student–patient interactions of at least six months. We categorized programs using qualitative thematic analysis and compared associations between program structures and goals.Results: We identified 98 LCPs in 69 schools. Half (52.0%) of LCPs occurred during the core clinical year. Program structures included ‘clinic attachments’ (50.0%), ‘longitudinal integrated clerkships’ (26.5%), and ‘patient attachments’ (20.4%). We identified goals in 89 programs, including ‘exposing students to specific topics, patient demographics, or practice settings’ (78.7%); ‘clinical or professional skill development’ (65.2%); and ‘understanding the patient experience’ (19.1%). Patient attachments were associated with ‘exposure to specific patient demographics’ (P = .04) and ‘understanding the patient experience’ (P = .03). Pre-clinical programs were associated with clinical skills development (P = .01).Conclusions: Our study identifies the scope and nature of LCPs in US medical schools. Understanding connections between educational structures and goals may guide program design and research investigations of educational processes and outcomes. Academic leaders in medicine have called for the transformation of health professions education –4. The LIn the LIC model, students care for a diverse panel of patients whom they follow across multiple clinical settings over extended periods of time, in lieu of traditional block rotations within discrete specialties ,14,17. TWorley et al.  describeOur study aimed to identify and characterize the full spectrum of LCPs in United States medical schools. We conducted a systematic search of all LCME-accredited US medical schools to generate a list of longitudinal student–patient programs. We used qualitative thematic analysis to determine the nature of the programs’ structures and goals, and we measured associations between them. To the best of our knowledge, this is the first study assessing the landscape of LCPs in the US and associations between LCP structures and goals.This mixed-methods study was a secondary analysis of existing data from publically available medical-school websites and other media.n = 137). Additionally, we performed a search of the schools’ names in conjunction with each of the key phrases using an independent search engine . All searches were conducted between 24 August 2014 and 24 October 2014. We derived and documented details on the programs from the search results and sent a follow-up email to a contact person from each program or school to confirm program details.We performed an online search using the search function on each medical school’s website and seven key phrases: longitudinal patient, longitudinal experience, longitudinal clinical, longitudinal clerkship, longitudinal rotation, longitudinal relationship, and integrated clerkship. The search included all LCME-accredited US medical schools as of August 2014  designed for undergraduate medical students, (2) were at least six months in length, (3) had a stated emphasis on longitudinality, and (4) offered continuity with patients. We limited programs to those at least six months in length based on the determination of ‘longitudinal’ in a narrative review . Rather ectivity . We defiWe extracted program characteristics and the explicitly-stated program goals from program websites or from other online sources describing the program, such as publically available news articles. Based on domains examined in an earlier review of longitudinal community and hospital placements , we extrnot offering ‘the majority of the academic year’s core clinical competencies across multiple disciplines simultaneously’) [When programs met criteria for patient attachment and clinic attachment but not the full LIC definition  ,16,19, wSix investigators  coded data using thematic analysis . The prohttp://vassarstats.net/fisher2x3.html). The significance threshold was set at .05.We compared the characteristics of programs’ structures to program goals arising from the qualitative analysis. We evaluated relationships by using the Chi-square test for independence . For any category with five or fewer programs, we used the Fisher exact test for small numbers  to seven selected program goals to determine associations. We did not use the program characteristic ‘curricular vs. extracurricular’ for comparison as only three programs were extracurricular. We did not compare goals by ‘date of establishment’ because a specific date could not be determined in nearly a third of the programs. We selected the seven program goals for the quantitative association analysis from among codes derived emergently in the qualitative analysis. We chose these goals for the association analysis without prior review of the quantitative data. We selected the seven program goals by considering themes currently prominent in the clinical medical education design literature, including educational continuity –8,21,26,The search of the 137 LCME-accredited US medical schools yielded 98 LCPs meeting our inclusion criteria in 69 medical schools. Our follow-up emails to program administrators yielded responses from 53% of programs.Characteristics of the structure of the 98 LCPs are summarized in Of 98 LCPs identified, 89 programs  were found to have explicitly stated goals. The number of goals per program ranged from 1 to 16 . We derived 39 codes from these goals and organized the codes into major, minor, and sub-categories. Providing students with exposure to specific topics or practice settings was the most common major programmatic goal . For example, 40.4% of programs (36/89) emphasized exposure to social medicine including the social determinants of health, psychosocial aspects of health, and engaging with the healthcare system. Exposing students to specific patient demographics, such as geriatric patients, rural patients, and individuals with chronic disease, was a stated goal for 37.1% of programs (33/89).Fifty-eight programs  had programmatic goals highlighting skill development. The minor categories within this major category were ‘developing clinical skills’ and ‘developing professional skills’. These domains overlapped; 20 programs  listed both clinical and professional skills as goals, 19 programs  listed only clinical skills, and 19  listed only professional skills.Twenty-nine programs  stated goals related to fostering longitudinal relationships. Twenty-four of these programs  explicitly listed student–patient relationships as a goal, and seven programs  emphasized the relationship between student and supervisor. Other programmatic goals included understanding the patient experience , integrating pre-clinical and clinical curricula , and recruiting graduates to a specific geographical location or specialty .P = .04), ‘understanding the patient experience’ (P = .03), and ‘exposure to primary care’ (P = .04)  was associated with the goals ‘exposure to specific patient demographics’ (P = .04) .10.1080/P = .02) or clinic attachments (P = .01). Patient attachment programs were also more likely to emphasize understanding the patient experience than clinic attachments (P = .02), although there was no significant difference between patient attachments and LICs in this regard (P = .07). Patient attachments were less likely than either LICs (P = .02) or clinic attachments (P = .05) to emphasize exposure to primary care.When we compared program types directly, we found patient attachment programs were more likely to have goals related to exposing students to specific patient demographics than either LICs (P = .01) (P = .01) or programs spanning both phases of training (P = .01) to emphasize clinical skills as programmatic goals.Phase of training  was associated with the goal ‘developing clinical skills’ (P = .01) . When coWe found no relationships between stated goals and program length or whether the program was mandatory vs. elective.Our online search of all LCME-accredited US medical schools in 2014 identified 98 LCPs meeting our inclusion criteria at 69 medical schools and demonstrated associations between program structures and programmatic goals. This study examined LCPs that were at least six months in length, were designed for undergraduate medical students, had an emphasis on longitudinality, and offered continuity with patients. At least half of US medical schools offered LCPs meeting our criteria. Twenty schools in the US offer more than one distinct LCP. Programs differed in structure, including date established, curricular vs. extracurricular, mandatory vs. elective, phase of training, program length, and program type .P = .01). Dornan et al. (2006) systematically reviewed medical students’ experiences in the ‘pre-clinical phase’ and highlighted multiple goals we found ; however, other than ‘clinical skills,’ our data did not significantly associate these other goals with the pre-clinical phase. We speculate that pre-clinical LCPs may recruit students by offering opportunities to practice clinical skills prior to beginning the core clinical year [We found associations between characteristics of program structure and goals that imply a degree of coherence across LCPs. We identified a significant association between LCPs in the preclinical years and the stated goal of developing clinical skills  and support the goal of ‘understanding the patient experience’ (P = .03). This association is perhaps unsurprising, as ‘patient attachments’ by definition emphasize pairing students with individuals from particular patient populations, independent of the clinic site or mentor. This finding is consistent with studies of senior mentor programs—a type of patient attachment—which found such programs could improve students’ understanding of and attitudes towards older patients [We found other structures that connected to program goals. ‘Patient attachment’ programs were more likely than other program types to emphasize specific patient demographics  to have goals related to exposing students to specific patient demographics may mark a difference between US LICs and Canadian and Australian LICs. The literature documents LICs in Canada [On the other hand, our finding that patient attachment programs were more likely than LICs . These findings connect to outcomes in the literature demonstrating that LICs effectively support the primary care workforce [In our study, 73.5% of programs offered longitudinal structures with LIC-like elements but did not meet the criterion of the international research definition of an LIC that states LICs must ‘meet the majority of the academic year’s core clinical competencies across multiple disciplines simultaneously’ ,16,18,19orkforce ,19,40,41orkforce , p. 324.orkforce ,13,23.Our work builds upon earlier studies that review LCPs and LCP-like programs. Thistlethwaite et al. 2013) systematically reviewed longitudinal placements and highlight program outcomes that connected to program goals featured in our data, notably ‘the development of a patient-centred approach to clinical care, presentations of illness in primary care settings and the development of insights into community-based medicine’  systemat,15,16, aThe differences in the lengths of the programs these studies included, and variations in length among US programs in our study, raise the question of how much time is needed to define a program as ‘longitudinal.’ Two papers begin to address the question of defining longitudinality by suggesting typologies of longitudinal programs. Worley et al. (2016) identified 54 LCPs across seven countries and categorized them according to program length and discipline coverage . Worley Ellaway et al. 2016) characterized LCPs in Canada by undertaking a study with theoretical and empirical dimensions . Based o characteAlthough our study included all LCME-accredited US medical schools, our search strategy relied upon publicly available information from schools’ websites; therefore, our search may be out of date, may misrepresent the true prevalence of LCPs, or may not fully depict schools’ current LCPs’ structures and goals. We used seven specific keywords in our systematic search; it is possible that other search terms may have yielded more LCPs meeting our criteria. To improve accuracy, we supplemented our search by contacting program administrators, and 53% of programs responded; yet, missing data or errors may persist among the non-respondents. In addition, eligible programs may have been excluded because schools may not post information about all current programs on their websites. Although we followed standard qualitative research methods for coding, our coding was limited by the quality of the information available. We limited our quantitative analysis to seven program goals (related to themes prominent in the literature) in order to identify significant associations while limiting false positives that may occur with multiple comparisons. Our approach may have missed other significant associations between program structures and goals that we did not examine. Finally, this study only included allopathic medical schools; we do not know the extent of LCPs in osteopathic medical schools or their structures and goals.We have identified LCPs in US medical schools and described associations between LCP structures and programmatic goals. To our knowledge, our study is the first to characterize LCPs (both LICs and non-LICs) in the US. In addition, our literature review found no other studies that characterized LCPs of at least six months’ duration unlimited by the year in the curriculum.Future research is needed to determine how program structures can support educational, institutional, and workforce outcomes. Ultimately, clinical education leaders should continue to innovate and study programmatic structures to create the outcomes that learners, patients, and society need most.


Background. Improving performance of primary care systems in low- and middle-income countries (LMICs) may be a necessary condition for achievement of universal health coverage in the age of Sustainable Development Goals. The Salud Mesoamerica Initiative (SMI), a large-scale, multi-country program that uses supply-side financial incentives directed at the central-level of governments, and continuous, external evaluation of public, health sector performance to induce improvements in primary care performance in eight LMICs. This study protocol seeks to explain whether and how these interventions generate program effects in El Salvador and Honduras.Methods. This study presents the protocol for a study that uses a realist evaluation approach to develop a preliminary program theory that hypothesizes the interactions between context, interventions and the mechanisms that trigger outcomes. The program theory was completed through a scoping review of relevant empirical, peer-reviewed and grey literature; a sense-making workshop with program stakeholders; and content analysis of key SMI documents. The study will use a multiple case-study design with embedded units with contrasting cases. We define as a case the two primary care systems of Honduras and El Salvador, each with different context characteristics. Data will be collected through in-depth interviews with program actors and stakeholders, documentary review, and non-participatory observation. Data analysis will use inductive and deductive approaches to identify causal patterns organized as ‘context, mechanism, outcome’ configurations. The findings will be triangulated with existing secondary, qualitative and quantitative data sources, and contrasted against relevant theoretical literature. The study will end with a refined program theory. Findings will be published following the guidelines generated by the Realist and Meta-narrative Evidence Syntheses study (RAMESES II). This study will be performed contemporaneously with SMI’s mid-term stage of implementation. Of the methods described, the preliminary program theory has been completed. Data collection, analysis and synthesis remain to be completed. High-performing primary care systems not only are the first point of contact for continuous, coordinated, comprehensive and people-centered health services, but also provide critical preparedness and response to public health threats3.Improving performance of primary care systems in low- and middle-income countries (LMICs) has been suggested as a necessary condition for the achievement of universal health coverage in the age of the Sustainable Development GoalsThe Salud Mesoamerica Initiative (SMI) is a multi-country, large-scale primary care performance improvement program. It is the result of a partnership between the governments of the eight Mesoamerican nation-states, the Bill and Melinda Gates Foundation, the Carlos Slim Foundation, the Government of Canada, the Inter-American Development Bank (IADB) and, during earlier stages, the Government of Spain. The program is aimed at improving reproductive, maternal, neonatal and child health outcomes among the poorest, rural populations in participating countries. Intended outcomes include increased availability, utilization, and effective coverage of primary care services and a reduction in preventable health inequities. The program’s approach to performance improvement combines the use of high-powered financial incentives at the government-level and the external verification of public sector, primary care system performance.7; subsequent waves of reform targeted public sector accountability and organizational best-practice9. Governments in LMICs adopted and replicated these reforms, oftentimes with the support of multilateral finance institutions and agencies in the official development assistance space.Programs and policies aimed at improving the performance of health systems have been at the forefront of many public-sector reforms. Initial waves of reforms in the public sectors of high-income countries were focused on learning and improvement10, including (1) political interventions as expressed in policies and regulations; (2) reforms by laws; (3) intervention by audit and inspection, based on continuous evaluation of results and conformity to predefined norms; (4) intervention by management, based on organizational science and management practice, such as continuous quality improvement or change management methods, among others; and, (5) intervention by rationalizing professional behaviors such as the introduction of evidence-based practices and the standard comparison of outcomes by public sector providers.Several generic types of reforms follow the logic of public-sector interventionsIn LMIC, accountability-driven reforms have flourished under the rubric of Results-based Financing (RBF); the health sector has regularly been at the center of such reforms. The Multilateral Finance Institutions, the Global Fund to Fight HIV/AIDS, tuberculosis and malaria, and the Global Alliance for Immunizations have financed results-oriented, global health programs, some of which have targeted health system improvements.11; incentives can target health care providers (supply side), households (demand side), or both. Performance-based Financing (PBF), is a prevalent type of RBF in which the incentives are exclusively financial; rewards are only aimed at providers; and the payments are usually adjusted for quality. PBF assumes many forms but, in essence, serves the purpose of reforming the ways in which governments pay health care providers  for the provision of services.There are ambiguities in the definitions as well as in the scope and content of RBF programs and policies. In this study protocol, RBF is understood as “any program that rewards the delivery of one or more outputs or outcomes by one or more incentives, financial or otherwise, upon verification that the agreed-upon result has actually been delivered”13. Such reforms usually assume that incentives and rewards serve as powerful motivators for the achievement of desirable behaviors among utility-maximizing, rational individuals16. These assumptions have conventionally been based on principal-agent theory15, positive agency theory16, and/or rational choice theory18. In recent years, there have been calls for using a more expansive view of human agency when discussing motivation and decision-making. Under such views, humans are not exclusively motivated by rewards and incentives, but can also be driven to action by intrinsic motivators19. This perspective has also influenced contemporary research on PBF in LMIC21.Accountability-driven interventions in public sector reforms are designed to reduce the misalignment in incentives between principals  and their agents 31. However, the evidence on the effects of RBF on large-system reforms targeting government-level improvements is scarce; and studies in LMIC settings are largely absent34.Most of the primary studies that have assessed RBF and PBF programs in LMIC have characterized the effects of financial incentives on provider-level motivation and behaviors31. Most PBF evaluations to date have used a narrow focus such as characterizing changes in health care outputs, while neglecting most other domains of primary care performance improvement35. Furthermore, the empirical evidence about how PBF leads to changes in attitudes and behaviors among public sector actors is scarce35. Domains that have been under-studied include, among others, whether and how do extrinsic and/or intrinsic motivators affect the behaviors, autonomy and responsiveness of providers and managers of primary care delivery systems; the influence that performance measurement data can have on the behaviors of primary care system actors and stakeholders; and the negative effects of RBF reforms, such as gaming, shirking and cream-skimming.The most-recent systematic review of performance-based financing programs in LMIC concluded that PBF is not a single type of intervention and that its effects are dependent on the interactions among multiple variables31.Public-sector reforms tend to incorporate multiple interventions that generate effects at different levels within organizational hierarchies, and among different actors and stakeholders. Those actors and the environments in which they are embedded interact with each other through time, generating inter-dependencies and, oftentimes, leading to counter-intuitive, emergent, and unintended effects. Furthermore, the implementation strategies and ancillary components of reform programs themselves, such as the provision of technical assistance or change management support, can also trigger system changes that need to be better studied36 found that the most influential predictors of organizational improvement and learning were the use of work-groups as learning forums; the availability of performance information systems that collect, store and disseminate performance data; the existence of a mission orientation that builds a sense of shared vision for success and common purpose; and the existence of organizational slack, such as time and resources that allow people to think and learn.Beyond accountability reforms, studies on performance management and performance assessment have empirically studied improvement-driven public-sector reforms. Studies of such reforms in the public sector of the United States have identified factors that can drive organizational learning and improvement. For instance, Moynihan and Landuyt7. However, despite widespread calls for using performance data and information to improve decision-making, the utilization of such data and information can rarely be guaranteed37. Also, little is known about the conditions under which performance measurement work or the mechanisms that lead to system improvement. Studies in evaluation science have addressed these issues39. In this literature, the availability and dissemination of performance evaluation can influence system improvement through multi-level changes on individual, interpersonal and/or collective motivation. Considerable research in evaluation science has been informed by this evidence44. We did not find, however, any study assessing the effects of evaluation results on health system performance improvement in LMICs.Improvement reforms are predicated on the assumption that the continuous collection, availability and analysis of performance data and information would lead to organizational improvement and learninga-priori to attain the agreed-upon targets; once domestic funds are made available and targets are agreed, SMI matches the domestic contribution with grant financing on a 1:1 ratio. Afterwards, the IADB enters into formal performance contracts with each government. In the contract, SMI commits to reimbursing half of the initially invested domestic funds contingent on the achievement of the agreed-upon performance targets45.In SMI, governments agree with the IADB to the implementation of up to three consecutive, 18–24 month programs, aimed at achieving a series of progressively complex health targets  that are externally verified by the University of Washington’s Institute for Health Metrics and Evaluation (IHME). Participating governments contribute domestic fundsCountry-specific performance frameworks with geographical targeting of the poorest, rural populations, were negotiated with each government at the start of the program and have remained stable through time. A pass-or-fail policy was agreed, according to which a government has to achieve 80% or more of the approximately ten (10) targets that make-up any given performance framework to be eligible for the reimbursement of half of the initial domestic contribution.46. After each round of performance evaluation, results are aggregated and disseminated in each country through policy dialogue workshops convened by the government and involving the IADB and IHME.As the agency in charge of external verification of government performance, IHME conducts a full-scale quantitative measurement that follows SMI’s sequential process of implementation. Before each country program starts, a baseline is collected and its results disseminated. After that, at the end of each 18 to 24-month implementation projects, IHME collects household and facility-based data to evaluate the achievement of agreed-upon results. Phase 1 programs started in a staggered fashion in 2011; phase-2 programs will finish during 2017; a third and final phase will start in 2018 and go into 2020. Program targets during phase 1 were focused on adherence to protocols, availability of resources and, in general, structure and process performance. During phase 2, targets were focused on outputs, and phase 3 will be centered on health outcomes, including but not limited to, coverage of exclusive breastfeeding, increased modern contraceptive prevalence rates, effective coverage of antenatal care and institutional deliveries, post-partum and post-natal care coverage, and in some cases, reductions in the prevalence of anemia and effective coverage of measles vaccination, measured in bloodSMI’s original theory of change  hypotheshow its main interventions would trigger outcomes, nor did it provide, either,a-priori explanations about the role that each country’s policy context could play in moderating the effects of program interventions. The SMI partnership appears to have embraced a high degree of flexibility in implementation to facilitate governmental buy-in. In 2011, the partners agreed on a set of common principles including a focus on external and independent measurement of results, accountability and transparency, and country ownership. These principles established the institutional boundaries that, in turn, allowed the IADB to negotiate country-specific performance contracts, results frameworks and evaluation plans with each participating government. They also granted implementing partners a high degree of flexibility in the design of each country’s multi-phased implementation plans and, also, led to performance contracts based on a few high-order principles  that were originally agreed among the funders and the IADB, and reflected inthe program’s operating model.While the program’s original theory of change identified several causal pathways, it did not explain47. Honduras, in turn, had started in the late 2000s a large-scale pilot of contracting-out and pay-for-performance in the delivery of primary care services48. The government thus decided to leverage its own performance-driven policies and programs and has thus implemented SMI in municipalities that had already acquired experience with RBF.In the two countries under study, El Salvador and Honduras, the program’s focus on country ownership led to each government deciding how to best deploy SMI’s non-reimbursable resources and their own domestic financing to increase the likelihood of achieving programmatic success. For instance, El Salvador had undergone a health system reform in the early 2010s, which coincided with the beginning of SMI implementation. The government decided to focus its targets on results that leveraged one of the reform’s central tenets, the provision of universal primary care services through community-centered, Family Health Teams50. It has been argued that realist evaluation provides a valuable and relevant approach for assessing interventions that involve changing human decisions and actions52. Realist evaluation is a form of theory-driven inquiry based on the premise an evaluation needs to answer “what worked, how, in what circumstances and for whom”, rather than the conventional question “Did the program work?”52. The appeal of this approach, compared to other theory-driven methods, lies in its explicit foundations in critical realism – an epistemology located between positivism and relativism52. Such perspective contends that program interventions bring about social change through underlying, usually hidden causal mechanisms, and considers the role of context as indispensable in explaining causality.The evidence gaps identified in our literature review have led to recent calls for new approaches in the evaluation of complex public-sector reforms, such as PBFThis study addresses two research questions: (1) What are the effects of using supply-side financial incentives on the performance of the primary care systems in Honduras and El Salvador? How are those effects produced? Under what contextual factors are these effects produced in each country? And, (2) What are the effects of continuous external verification of performance in the two countries under study? How are those effects produced? Under what contextual factors are these effects produced in each country?57.While there is no single way to implement a realist evaluation study, as the experience with its use and applications grow, various authors have adopted and adapted Pawson and Tilley’s approach and identified a series of steps that are described belowDeveloping a preliminary program theory. Realist evaluation starts with the development of a program theory that serves as a hypothesis about the ways in which outcomes are produced through the interaction between interventions and context conditions, and mediated by hidden, not-observable mechanisms. The latter have been defined as the ideas and opportunities triggered among program actors and stakeholders in response to program interventions57. The process of testing and refining program theories usually relies upon quantitative and qualitative methods and culminates with a refined program theory53.This stage in the study was completed through complementary approaches, including (1) review of program design documents; (2) discussions with program designers to gain in-depth understanding of the original causal links between program interventions and expected outcomes; (3) scoping review of the literature, focused on identifying theories and empirical evidence addressing similar processes of primary care system change; and, (4) facilitation of a workshop with IADB stakeholders, which helped understand their assumptions about how program interventions effects could be produced in the two countries under study. This process of making explicit the assumptions held by program stakeholders before data is collected is an essential aspect in realist evaluation. As a result of these various activities, the research team formulated a preliminary program theory.In a separate, ongoing study we will perform a realist synthesis of performance improvement; performance measurement/evaluation; and, results- and performance-based financing. The process started with a scoping review of social science theories related to the two research questions and led to the mapping and synthesis of theories explaining the contextual factors and causal mechanisms of relevance to this study protocol. This will be followed by a search for primary studies, systematic reviews, realist evaluations and realist syntheses on the themes above and as required by our research questions. The search for the scoping review was done on Science Direct, JSTOR, and Goodle Scholar using a snowballing technique. The theories that were mapped are summarized inBased on a synthesis of the results from the scoping review, and informed by the knowledge acquired from the stakeholder workshop and the document review, a preliminary program theory was developed . It is sThe use of (1) high-powered, supply-side financial incentives aimed at central-level government actors and stakeholders (intervention 1) and the implementation of continuous, external evaluation and verification of primary care service performance (intervention 2) supports country priorities through continuous policy dialogue, technical support, and purposive dissemination of performance results (implementation strategy);Leading to the adoption of innovations in supplies, information, and workforce management (outcome 1); the adoption of performance management reforms such as continuous process and quality improvement (outcome 2); the introduction of policies and regulations that promote primary care improvement and/or reductions in preventable inequities (outcome 3); and, improved, population-level health outputs and outcomes (outcome 4).The behavioral changes listed above occur at various levels within the primary care system, as follows;63;At the individual level, they satisfy psychological needs such as autonomy, competence and relatedness and/or the need to upgrade or improve personal goals and self-efficacy 39.At the interpersonal level, because of the aggregate internalization by multiple individual actors and stakeholders, of changes in ideas and opportunities; and/or through a growing sense of public service and/or community service 65, leading to the emergence of new organizational culture and collective behavior (outcome);Collective level changes could also be triggered whereby the ideas and opportunities of a sufficiently large number of individual actors internalize or assimilate new norms, routines and behaviors which, in turn, spread across inter-organizational and social networks33 thus, increasing the likelihood of the production of population-level health effects (outcome) and, potentially, transforming the primary care system in a sustained fashion (outcome)33.Collective inter-organizational-level changes may further lead to the institutionalization and collective assimilation of aggregate individual- and interpersonal-level behaviors through imitation and/or the adoption of new professional and cultural norms, and/or innovative, pro-performance policies (outcome)69. At the country-level, the availability of solid institutional environments  can create windows of opportunity for the introduction of policy innovations and, also, facilitate convergence between domestic policies and programs, and the externaly-funded interventions. Finally, pre-existing environmental conditions, such as the organizational capacity to absorve new knowledge or the presence of climates that support and enable change, have also been associated with increased assimilation of service innovations71 and need to be considered in the characterization of context.Global, institutional, and organizational contextual conditions are also needed for the attainment of program outcomes and for the triggering of the above mechanisms. They include, at the global and sub-regional levels, the existence of favorable conditions such as influential issue-specific global agendas that match existing governmental priorities or a history of interactions between national health agencies and their agendas, and between those and official development aid agencies and their agendasStudy design. In this step the preliminary program theory will be tested, further developed, and validated or rejected. A multiple case-study design with embedded units with contrasting cases was selected72. The contrasting case approach aligns well with the proposition that the two different country contexts in Honduras and El Salvador can trigger to-be-identified mechanisms that generate program results. Given the system-wide and reinforcing effects of the two interventions under study, we define each country’s primary care system as the unit of analysis. Within each country, at least two high-performing municipal-level primary care delivery systems will be analyzed as embedded sub-cases, each with its unique contextual and service delivery structure.This evaluation is an 18-month study running from May 2017 to December 2018, and executed contemporaneously with SMI’s mid-term stage of implementation. The study seeks to maximize diversity in institutional and policy context to increase the likelihood of identifying variations in policy and program conditions and characterizing the process of change generated to date by the program in one low- and one middle-income country, respectively Honduras and El Salvador. Both countries have to date been exemplars of high-performance, which in SMI is defined as the continuous achievement of 80% or more of the targets agreed between each government and the IADB, and externally verified by IHME.In each country, the study will assess the context, interventions, implementation approaches, and program effects, intended and otherwise. At the central level of government, the study will characterize program antecedents, policy and organizational context, primary care system’s stewardship and policy-setting, and overall program management and implementation. At the local, municipal level, it will explore primary care delivery through Family Health Teams in El Salvador, and on public as well as non-profit, pay-for-performance providers. Primary data collection will include the methods described below.Data collection methods. Realist evaluation is method neutral, and the nature of the research, the evaluation questions and the preliminary program theory determine the choice of study design and methodology57. The primary data collection methods to be used in this study include in-depth interviews, non-participatory obervation, and document review.73. Subsequently, the sample will be snowballed from the initial set of informants.In-depth interviews with key informants in each country will be conducted to identify individual, inter-personal and collective or organizational factors that may affect primary care system performance in each country under study. These interviews will also be used to elicit contextual elements that could act as barriers or facilitators for the delivery of SMI’s interventions. In this study, we aim to gain a high-level understanding of the causal mechanisms and pathways of performance improvement, as reflected in the preliminary program theory. SMI intervenes at the central as well as local levels of the primary care system, generating hypothesized feedback effects between both. The evaluation aims require the characterization of the interactions and inter-dependencies that occur among multiple actors in the primary care system; this would allow resulting data to help explain the complex nature of the process of performance improvement, and ultimately, help the team validate or revise the preliminary program theory. Accordingly, in-depth key informant interviews will be conducted with four sets of actors: (1) Country policy- and program implementation actors in Honduras and El Salvador; (2) Health care providers at primary care facilities in Honduras and El Salvador; (3) Performance verification and evaluation stakeholders at IHME; and, (4) Program designers at the IADB. Key informants will be recruited using a purposeful sampling approacha-priori, but we expect to conduct approximately 80 key informant interviews, which will ultimately be determined based on theoretical saturation74. Respondents will be invited to participate voluntarily in the study; no compensation will be provided for participation. Interview guides will be used to conduct in-depth interviews; a series of probes will also be developeda-priori  configurations that will then be gauged to determine which patterns plausibly explain how each intervention generated the observed effects, expected and otherwise. The final thematic structure will be used to refine the preliminary program theory. Data analysis will be done with nVivo Version 11 for Mac.The coded data will be appraised using two complementary analytic approaches. The research team will use iterative conceptual and pattern coding to identify major emergent inductive themes. At the conclusion of the process, the codes will be arranged into the four major categories of context, intervention, mechanism, and outcomes. The team will scanEvaluation results will be completed by integrating findings from the different data collection methods  to confirm, reject or further develop the preliminary program thyeory and the causal patterns identified. The findings will also be contrasted with secondary quantitative and qualitative data sets collected by IHME and others and with social science literature in search for mechanism-oriented theory that may provide explanation for the emerging causal patterns. The results will be a series of CMO configurations that are backed up by the empirical data that provide plausible causal explanations for the observed findings.Synthesis and refined program theory. In this step the research team will link the emergent CMO conﬁgurations to the preliminary program theory, leading to the adoption, modification, or rejection of the preliminary program theory and will, then, formulate plausible explanations of how and why high-powered, supply side incentives and external verification of performance generate the observed results. The resulting explanations for the observed program effects will then be compiled in the form of narrative summaries, tables, and/or causal loop diagrams. The end of the study product is a final, refined program theory.77.Study findings will be published in peer-reviewed periodicals and disseminated locally among policy-makers in the two countries to be studied. The presentation of findings will be made following the Realist And Meta-narrative Evidence Syntheses study (RAMESES II) that was designed to provide guidance on quality assurance and uniform reporting and improve quality and consistency in the reporting of realist evaluations78. All researchers will engage in the introspective practice of maintaining ‘personal biases memos’ to make explicit all self-identified biases and pre-conceptions that may effect the research process78. All analytic decision notes and memos, biases memos, document analysis syntheses, interview guides, research team meeting agendas and minutes, and analysis outputs including coded transcripts, conceptual frameworks, tables, etc. will be preserved to provide a verifiable audit trail.A set of measures will be taken to increase the validity of the study in terms of reflexivity, credibility and confirmability, and enhance the trustworthiness, transparency, and accountability of the researchThe refined program theory and CMO configurations resulting from this study have several anticipated uses and applications. Program implementers like the IADB and Salvadorian and Honduran government actors, for instance, can use the findings to consider introducing adjustments in SMI’s implementation during its third and final phase (2018–2020). Also, given the study’s focus on exploring the linkages between ongoing, pre-existing policy mandates and priorities, it is plausible to expect study findings to be of relevance to further improve the evaluation and subsequent re-design of domestic health policies in El Salvador and Honduras. Program evaluators like IHME and other research groups, can use program findings to enhance ongoing evaluation activities or to inform the design of new evaluations that deepen one or more of the various casual patterns identified. For example, our research team intends to use the emerging CMO configurations in the area of performance management to inform the design of a new study to explore whether and how SMI quality improvement interventions produce gains in primary care performance. We also expect program findings to set the stage for further realist evaluations in other large-scale primary care reform in contexts other than Mesoamerica.80. Theory-based program evaluations of primary care performance improvement would also benefit from the publication of realist syntheses that rigorously appraise the literature in search of context-mechanism-outcomes and program theory82. Such studies would not only facilitate the work of research teams currently addressing primary care performance improvement research, but would also strategically shape future health system research agendas, particularly in LMIC.Another source of complexity in this study arises from the significant evidence gaps that we identified and from the multiple fields that would need to be rigorously studied to properly address the various outcomes generated by accountability and performance management reforms. As discussed before, such outcomes can occur at different levels of analysis  and in different contexts (high- as well as LMIC). Not only does this type of research demand inter- and multi-disciplinary capabilities within research teams, but it also calls for rigorous, systematic assessment and mapping of the evidence gaps83. One such challenge pertains to settling on an unambiguous and precise definition for what constitutes a mechanism. Several definitions in the literature are of a descriptive nature and focus on well-known features of mechanisms such as them being unobservable, context-specific and being able to generate effects. We settled on a definition of mechanisms as the ideas and opportunities triggered among program actors and stakeholders in response to program interventions57. Such an approach is consistent with a view of social change according to which the beliefs, choices and opportunities of individual actors and the interactions among them (micro-level) are the main drivers of social change. This approach also recognizes that the “macro,” social and cultural environment in which these individual actors are embedded can shape social change by means of the internalization of collective values, norms and institutions among individual actors85.The research team has faced several challenges in shaping this study’s hypotheses, or preliminary program theory. Many of these challenges are common to other realist evaluations and have been discussed elsewhere84. Three specific types of mechanisms that explain social change are thus of interest to this study86.Based on these considerations, this study aims to, first, explore plausible causal explanations based on individual or group-based ideas and opportunities among program actors and stakeholders and, second, to ground those observations on an understanding of the policy and program context in which those actors and stakeholders are embedded. Therefore, we expect that any explanation of primary care system performance improvement needs to address both individual, micro-level, as well as collective, macro-level properties that “are not meaningfully attributed to individuals”Situational mechanisms refer to the macro, organizational-level environment in which SMI actors and their social interactions or linkages occur, including domestic policy-makers, ministry of health managers and primary care providers, among others. It also includes SMI stakeholders such as the implementation agency (the IADB) and the external evaluators of performance (IHME). This type of mechanism operates in the direction from macro environment to individual actors (macro-to-micro change).Action-formation mechanisms are those that explain how actors’ ideas and opportunities influence individual behaviors across the primary care system. In this type of mechanism, the interaction between program interventions and context, trigger changes in individuals’ ideas and opportunities that further influence others in the same social system. This type of mechanism can generate effects that spread from an individual actor to additional actors (micro to micro change). Finally,transformational mechanisms provide explanations of how the sum of new behaviors of multiple individual actors in the primary care system bring about change across the entities that conform the primary care system’s macro environment such as norms and institutions .Due to limitations in scope, this study can only plausibly characterize some of the “macro” and “micro” mechanisms triggered during SMI’s initial stages up until its current, mid-term stage (2011–2017). It is also plausible to characterize intended and unintended effects generated during this same period. Findings can also be used identify propositions about downstream effects that could occur or not during the final implementation phase (2018–2020) and about the mechanisms that could help sustain desirable effects after implementation ends. Longer-term, transformational mechanisms, their anticipated effects and the underlying context-mechanism-outcome configurations will, however, remain outside our scope of work.11, but we remain cognizant of the fact that PBF is not a single intervention and that its “ancillary” components can themselves generate system effects31. In this respect, this study frames SMI interventions as generic types of public-sector reforms aimed at inducing accountability and organizational improvement and learning. Given that the challenges in defining what these large-scale reforms contain in specific contexts –both in high and well as less-developed nations- the use of a realist evaluation approach will likely contribute to the theorizing of how and why specific contexts generate health and non-health effects in primary care performance management reforms.Another challenge in this study refers to the contested nature of the current definitions used to characterize the interventions that conform RBF, PBF or any of the various reforms that use supply-side incentives to drive accountability in public sector actors and, as is the case in SMI, across the entire primary care system. Like others before us, we settled on the definitions provided by Musgrove57, but also relied on the Realist and Meta-narrative Evidence Syntheses study that provides guidance on how to improve quality and consistency in the reporting of realist evaluations77. By aligning protocol design to these guidelines, we expect that the furture publication of this study’s findings will adhere to current best practice.Finally, given the method-neutrality that is central to realist evaluations, this study also faced the challenge of settling on a final sequence and content of research methods and activities. We decided to, first, follow the steps described by Vareilles and Marchal in relatively similar realist evaluations and studiesThe study’s protocol was reviewed and declared exempt by the George Washington University’s Institutional Review Board (study number 041733). The Ministries of Health of El Salvador and Honduras were informed of the proposed research by the IADB and provided written approval for the research activities.Ethical approval documentation will be made available on request. The study will employ scrupulous adherence to the highest ethical standards, and current international and local legislation pertaining to research governance. The data collection will operate under explicit informed consent, which will be preserved in study records. Respondents will be given the choice to provide consent verbally on tape before the interviews, or in writing. To maintain anonymity, respondents will reserve the right to review the study outputs and withdraw consent if necessary. All identifying information will be removed from transcripts and stored separately with access restricted to the research team. All transcripts will be stored electronically in password protected cloud services, and physical documents will be securely stored at George Washington University, Milken Institute School of Public Health. The authors have developed an in depth and well written description of the rationale behind and approaches to the protocol for a study using a realist evaluation approach to do an intern evaluation of AMI, a large multicounty accountability-driven intervention. The approach will complement the plan program evaluation being completed by IHME which is largely looking at explicit program defined outcomes. The description off SMI particularly for readers not as familiar with the structure is very helpful. The authors are clearly fluent in Realist Evaluation and familiar with many of the underlying theories which they use. However there is a lack of clarity of what the main focus is of this manuscript is describing and the use of the term “study” is often confusing as referring to different scopes of work. For example: Abstract:study presents the protocol for a study that uses a realist evaluation approach to develop a preliminary program theory that hypothesizes the interactions between context, interventions and the mechanisms that trigger outcomes. The program theory was completed through a scoping review of relevant empirical, peer-reviewed and grey literature; a sense-making workshop with program stakeholders; and content analysis of key SMI documents.” And then goes onto to say “Thisstudy”. The initial sentence which reads “ This In the text, the reviewer was still confused which study was being described  and clarity would be helpful, including that the protocol describes work already done (development of the preliminary program theory) as well as how it will be applied in the future. In framing the manuscript in the text, later they then state “ This study addresses two research questions:  “(1) What are the effects of using supply-side financial incentives on the performance of the primary care systems in Honduras and El Salvador? How are those effects produced? Under what contextual factors are these effects produced in each country? And, (2) What are the effects of continuous external verification of performance in the two countries under study? How are those effects produced? Under what contextual factors are these effects produced in each country? While I assume that this use of the term: study” refers to the realist evaluation rather than the development of the program theory. For example in the section “ Study design” it states “. In this step the preliminary program theory will be tested, further developed, and validated or rejected.” Given the critical importance of the qualitative data to be collected through interviews, a bit more detail in how the interviewees will be sampled  In their challenges part, it would be helpful to understand a bit more the limitations imposed by the 2 countries chosen from SMI for this study, and what characteristics differ from other SMI countries not chosen for this evaluation Minor: On page 8 in describing the program theory, I am curious that inputs are not explicitly called out as needed (and related to context) and that equity and effectiveness are also not explicit in the theory. Given the design of SMI and the underlying approach of Realist Evaluation, I was curious if the researchers had considered including community interviewer and or patients as critical to the success (and acceptability) of the intervention. Are they also planning to assess fidelity to the planned implementation  which could change the outcomes and be related to or change the mechanisms  " like here . Authors may also broaden the CMO configuration to consider the ICAMO configuration that may improve quality in the analysis and make a better and more explicit use of the role of actors in the analysis. The following comments and questions may help in improving some minor points.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Dr. Dossou: Les auteurs apprécient vos commentaires. Merci beaucoup.We will consider them all, while editing the paper. I recommend publication with minor revisions.  As this is a study protocol as opposed to the actual study, there are no datasets at this time so the answer to the question, "Are the datasets clearly presented in a useable and accessible format" can only be partly or actually NA since the data will be collected using the study protocol that is proposed for publication and eventually that will be carried out to assess the impact of SMI.  Overall, I find this to be a thorough and carefully thought out study protocol that will provide important insights into how the results produced by SMI were actually created.  As such, this study protocol will shed important insights into how a large, complex intervention across multiple countries and over time produced the quite astounding results that marked the success of SMI. Even as we have seen the positive results from the regular evaluations and can easily see the quite significant improvements countries that are part of this initiative have registered, important questions as to what factors actually drove the impact seen remain only partially answered. This study will shed important light on these questions.The authors use PBF and RBF almost interchangeably and sometimes use both terms. I think it might be less confusing to the reader to define terms up front and then use one term.Page 3, paragraph 8, says that studies on the effects of RBF on large scale system reforms are largely absent. Later on the authors cite a systematic review. In fact, there have been a number of systematic reviews of RBF programs beyond the one cited. For example, Andy Oxman has several papers that review  the experience with RBF. Miller and Singer (2013) is another.I also think that in the area of RBF, it's important to not focus only on LMIC experience as RBF is an instrument that has been used and is being used extensively. The Quality and Outcomes Framework (QOF) in the UK NHS is an example. Peter Smith has a number of papers that reviews that experience and Cheryl Cashin and Peter Smith have a paper on how RBF links to the larger issue of Strategic Purchasing.Perhaps my strongest comment is on page 7, paragraph 6, regarding the program theory section. I think it's quite possible to formulate a hypothesis that SMI was not primarily a classic extrinsic financial incentive program but possibly much more an extrinsic non pecuniary program where the rewards were doing well amongst your peers. When you look at the incentive rewards, its difficult to see how such relatively small financial rewards could incent behavior. The counterpoint to this argument might be that the funding provided by the SMI donors was flexible and in these heath systems flexible funding is often rare and highly prized but that too is an issue deserving of further investigation. However, if the funding is small and relatively insignificant, the question is then what drove the behavior and actions taken. A factor worthy of investigation is the SMI approach of engaging multiple countries in a form of joint competition. Ministers of Health were all engaged on SMI and there is some anecdotal evidence that the approach of having them compete together, each trying to attain the targets they set for their own country, created a form of competition or at least a common forum where not performing well would be seen as a distinct negative outcome, thereby conferring strong incentives for them to perform well or endeavor to make sure their health system performs strongly. This kinds of peer effects are known to be powerful in behavioral economics and so we should look for them in this study as well.   I only have a few minor quibbles regarding the article.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Dear Dan,The entire team read your comments. We appreciate them enormously and will tackle them in our upcoming edited and final version.Thanks a lot,Wolfgang on behalf of the team.


This investigation of the implementation of a positive youth development program (Project P.A.T.H.S.) was part of a large study undertaken comprehensively to explore how effective the Tier 1 Program was in practice and how the results can shed light on future developments. Case studies on randomly selected schools were conducted in order to examine the factors that influence the process and quality of implementation of the Tier 1 Program of the Project P.A.T.H.S. Through interviews with the school contact person and focus group interviews with the teachers, an integration of the findings of these studies showed that five factors related to the program, people, process, policy, and place (5 “P”s) facilitated the implementation process of the Tier 1 Program in the school. Based on the integrated findings of a randomly selected school, it was found that the school made use of the changes in the educational policy environment to facilitate school policy and structural changes, to pave the way for the success of the implementation of a new and “unfamiliar” curriculum. Overall, the quality of program implementation in the school was good and the program was well received by the program implementers. Implications of the present findings for future program implementation with reference to school administrative arrangements and implementation issues are also discussed.


What is the effectiveness of a timing and coordination group exercise program (On the Move) compared with a seated strength, endurance, and flexibility exercise program  for improving mobility in community-dwelling older adults?In this single-blind cluster-randomized trial that included 298 older adults, participants in the On the Move group had greater improvements in mobility than those in the usual-care group.Group exercise programs to improve mobility in older adults should include timing and coordination exercises that are important for walking. This cluster-randomized trial compares the effectiveness of a group exercise program that focuses on the timing and coordination of movement with a seated strength, endurance, and flexibility program  on function, disability, and walking ability of older adults. Timing and coordination exercises may be an important addition to community-based health promotion exercise programs to improve walking in older adults.To compare the effectiveness of the On the Move group exercise program, which focuses on the timing and coordination of movement, with a seated strength, endurance, and flexibility program  at improving function, disability, and walking ability of older adults.Cluster-randomized, single-blind intervention trial. Thirty-two independent living facilities, senior apartment buildings, and senior community centers were randomized to On the Move  or usual care . Participants were 65 years or older, able to ambulate independently with a gait speed of at least 0.60 m/s, able to follow 2-step commands, and were medically stable.Exercise classes were 50 minutes, twice a week for 12 weeks and had 10 or fewer participants per class. On the Move consisted of warm-up, timing and coordination , strengthening, and stretching exercises. The usual-care program consisted of warm-up, strength, endurance, and stretching exercises.The primary outcomes were self-report of function and disability (Late Life Function and Disability Instrument) and mobility  assessed by blinded individuals.P = .002) and 6-minute walk distance . Attendance was greater in the usual-care program compared with On the Move . There were no significant differences in any of the other primary or secondary outcomes.Participants  were mostly female (251 [84.2%]) and white (249 [83.6%]) and had a mean (SD) of 2.8 (1.4) chronic conditions. Intervention groups were similar on baseline characteristics. Postintervention, 142 (93.4%) participants in On the Move and 139 (95.2%) participants in usual care completed testing. On the Move had greater mean (SD) improvements than the usual-care group in gait speed (0.05 [0.13] vs −0.01 [0.11] m/s; adjusted difference = 0.05 [0.02] m/s; The On the Move group exercise program was more effective at improving mobility than a usual-care exercise program, despite lower attendance. Additional research examining the impact of the intervention on long-term disability outcomes is needed before recommending routine implementation into clinical practice.NCT01986647clinicaltrials.gov Identifier:  Walking difficulty is a common, costly condition in older adults.Studies examining the impact of group exercise programs on walking have conflicting findings and limitations.Based on previous research,Detailed methods are published elsewhere.Participants were recruited between April 2014 and January 2016, from those individuals residing in independent living facilities and senior housing, and attending community centers in the greater Pittsburgh, Pennsylvania, area. Inclusion criteria were age 65 years or older, a resident or member of the participating facility, and ability to ambulate independently (with or without a straight cane) with a gait speed of at least 0.60 m/s. Individuals with a gait speed less than 0.60 m/s have difficulty participating in the program and raise safety concerns in the group exercise setting. Those non-English speaking; unable to follow a 2-step command or understand the informed consent process; planning to leave the area for an extended period of time; with a progressive neuromuscular disorder; with any medical condition or illness that was not stable ; or with a post–6-minute walk test heart rate of at least 120 beats per minute, systolic blood pressure of at least 220 mm Hg, a drop in systolic blood pressure of greater than 10 mm Hg, or diastolic blood pressure of at least 110 mm Hg were excluded.Both exercise programs were group based, had identical frequency and duration , had 10 or fewer participants per class, and were delivered by trained exercise leaders . All exercise sessions were held on site at the facilities and were initiated within 2 weeks of baseline testing. Exercise program fidelity was determined through site visits by the program developers. A detailed description of both interventions can be found in the eMethods in (pp181-194)On the Move was based on principles of motor learning that enhance “skill,” or smooth and automatic movement control.The usual-care group exercise comparator was a seated strength, endurance, and flexibility program based on programs being conducted in the community-based facilities. The usual-care exercise program contained a warm-up, upper and lower extremity strength exercises, aerobic activities, and a cool-down. The warm-up and cool-down contained gentle range-of-motion exercises and stretches. Strengthening exercises focused on upper and lower extremity muscle groups and used playground balls, the opposite extremity, and body weight to provide resistance to the movements. Aerobic activities included repeated movements of the lower extremities  at various speeds. Upper extremity movements were added to increase the intensity of the activity. The entire program was conducted while sitting (50 minutes).We randomized facilities equally to the 2 interventions stratified by facility type. Participating independent-living facilities were known ahead of time and further stratified by socioeconomic status, academic medical center affiliation, and location in adjacent county before randomization to achieve a balance by design. Each of the other 2 facility types was randomized as they agreed to participate, using random block sizes of 2 and 4. Facility assignment was revealed to the coordinator only after baseline testing. We then randomized participants within each facility to a class run by a study exercise leader (primary aim) or a subsequent one by a facility staff activity person (exploratory sustainability aim). Per a midtrial protocol modification approved by the sponsor for potential safety concerns, if a suitable facility staff person was not available, their class was also taught by an exercise leader, and participants were analyzed accordingly.All measures were collected at baseline prior to randomization, immediately prior to intervention in those randomized to a subsequent class, and immediately following the 12-week intervention by blinded personnel. All testing was conducted on site at the facilities.The primary outcomes were Late Life Function and Disability Instrument (LLFDI) overall function and disability frequency domains, 6-minute walk distance (6MWD), and gait speed. The LLFDI is a pair of self-report instruments targeted for assessing physical function and disability in older adults with acute or chronic problems, and designed to be more sensitive to change than similar measures. The LLFDI has established known-groups validity and test-retest reliability (0.68-0.98).Participant satisfaction was assessed by exit surveys. The items included degree of satisfaction with components of the exercise program , participants’ expectations, perception of benefit from the program, and likelihood of recommending the program to others, and included open-ended questions.Adherence was measured by class attendance rosters. Reasons for missed classes were recorded when available. Adverse events during testing or intervention were recorded for review by the study physician for adjudication and direction.d = 0.5).We based sample size on pilot studies,t test to simultaneously compare the baseline to follow-up change in the primary outcomes between arms to protect the type I error rate from multiplicity. On observing significance, subsequent analyses were to be performed without further multiplicity adjustment. Third, we fit linear mixed models(pp275-283) with baseline to follow-up change in continuous outcomes as the dependent variable; intervention arm as the fixed effect of interest; baseline value of outcome as a covariate; and a facility random effect for site clustering. We used multiple imputation to account for missing data.(p15-18) Fourth, dichotomous secondary outcomes were analyzed similarly, but using a generalized estimating equations model(pp146-147) with a binomial distribution, logit link, and an exchangeable correlation structure for clustering. Fifth, we explored with a series of unplanned subgroup analyses stratified by adherence, setting, and baseline performance and/or function. Finally, we performed sensitivity analyses of results to additionally adjust for baseline characteristics different between groups, using immediate preintervention measurement instead of prerandomization baseline for those in a subsequent class, and ignoring missing data. SAS, version 9.3, software  was used.We performed a prespecified intention-to-treat analysis. Participant flow was summarized using a CONSORT diagram.See the Participants had a mean (SD) age of 80.0 (8.1) years, were mostly female (84.2%) and white (83.6%), and had a mean (SD) of 2.8 (1.4) chronic conditions. Participants walked slowly , and 170 (57.0%) had a 6MWD less than community ambulation distance (300 m). The intervention groups were similar except minor differences in facility type .P = .02). The OTM group had significantly greater mean (SD) improvements than the usual-care group in gait speed  and 6MWD . A meaningful improvement in gait speed (0.05 m/s) was achieved by 64 of 152 (42.1%) and 36 of 146 (24.7%), respectively, in the 2 groups, resulting in a number needed to treat of 5.7. There were no significant differences between groups in other outcomes   . The majP = .01) and facility type (P = .04) interactions with respect to 6MWD. We hypothesize that those attending 20 or more classes  or from community centers  may derive greater benefits from OTM than the usual-care program. There were 4 adverse events , and all occurred during the OTM classes. Sensitivity analyses did not materially change the results (data not shown).Unplanned exploratory subgroup analyses showed significant treatment by class attendance , it is important that all the events occurred during the OTM program. They were expected adverse events of exercise and did not prohibit anyone from finishing the program. All who experienced an event were willing and able to return to exercise. Previously, we had found that older adults recognize the risks of more challenging exercises yet believe that the risks are worth the benefit.Attendance in OTM was lower than in the usual-care exercise program. Participants were equally satisfied with either program, and they felt safe in both programs. On the Move is conducted primarily while standing and was designed to be a more challenging program than the usual-care class. One potential explanation for the lower adherence is that an individual who had a health status change during the 12-week program may have been more likely to continue in the usual-care program, which was completed while seated, than in the OTM program, which was completed primarily while standing. Another potential explanation is that participants need a certain level of baseline reserve to elicit greater gains from OTM than the usual-care program, as suggested by weak evidence of greater between-intervention differences among faster walkers than slower walkers from the post hoc subgroup analyses.Our findings have important public health relevance. Many community sites are looking for evidence-based exercise programs for their facilities. By examining the OTM intervention in a rigorous comparative effectiveness trial, OTM can be submitted to the Administration for Community Living to be evaluated for Evidence-Based Program status. The purpose of this process is to improve access to information on evidence-based interventions with the ultimate goal of reducing the lag time between the creation of scientific knowledge and its practical application in the field. Interventions that have the Evidence-Based Program designation have met the highest-level criteria for Title IIID funding of the Older Americans Act. Obtaining the Evidence-Based Program designation is an important first step in the translation of OTM into widespread use in the community. Our study has several strengths. First, ours was a real-world community-based comparative effectiveness studySome limitations should be considered. In some sites, all individuals were not taught simultaneously, but in classes conducted in series. The study was so designed that during the first session taught by an exercise leader, we could train the staff activity personnel to lead the second session. Consequently, participants randomized to the staff activity personnel class had to wait 12 weeks before starting their exercise sessions. The additional waiting time could have exposed the participants to additional health events, fatigue of waiting, and disappointment of not being taught by a professional. Second, outcomes were examined immediately following the intervention; thus, the long-term intervention effects on mobility, function, and disability and whether the mobility improvements persist over time is unknown. Last, our heterogeneity of treatment examination was exploratory, and we have interpreted cautiously contingent on significant interactions. Interaction testsFrom a community-based health promotion and wellness exercise programming perspective, the On the Move group exercise program was more effective at improving mobility than a usual-care group exercise program, despite lower attendance. Additional research examining the impact of the intervention on long-term disability outcomes is needed before routine implementation into clinical practice can be recommended.


In the era of endobronchial/esophageal ultrasound (EBUS-TBNA/EUS-FNA), many centers forgo conventional transbronchial needle aspiration (C-TBNA) in favour of EBUS-TBNA/EUS-FNA despite no conclusive evidence showing better yields with EBUS-TBNA/EUS-FNA. Assess the feasibility of an algorithmic approach for mediastinal sampling beginning with C-TBNA utilizing rapid onsite cytologic evaluation. Descriptive analysis of 92 consecutive patients referred for adenopathy that underwent C-TBNA and subsequent EBUS-TBNA/EUS-FNA if C-TBNA was negative or nondiagnostic.P = 0.51), the relationship between yield and LN size (P = 0.37), or time difference in procedures following the algorithm compared to EBUS/EUS only procedures . 92 procedures were analyzed. In 50 (54.3%) of cases, C-TBNA alone was sufficient. EBUS-TBNA was performed after C-TBNA in 27 (29.3%) of cases and EUS-FNA in 33 (35.9%) of cases. The yield was 92.9% for C-TBNA, 92.5% for EBUS-TBNA, and 89.7% for EUS-FNA. There were no statistically significant differences in yields by LN station ( An algorithmic approach to assess the mediastinum using C-TBNA initially is feasible without sacrificing yield or procedure times. C-TBNA was sufficient for diagnosis in 54.3% of cases and can be efficiently taught in an IP training program. Tissue sampling of mediastinal lymph nodes (LNs) is often required during assessment of patients with mediastinal lesions or for accurate staging of patients with lung cancer. Endoscopic modalities are safe and minimally invasive tools with excellent diagnostic yield , 2. ConvDespite its reported safety and high impact on patient management , surveysEndobronchial ultrasound-guided TBNA (EBUS-TBNA) provides real-time visualization and access to paratracheal, subcarinal, and hilar LN . TransesThe current guidelines now specifically recommend EBUS or EUS as the initial diagnostic modality for minimal invasive assessment of the mediastinum  but, priThe Ottawa Hospital Research Ethics Board provided approval for this study (20130521-01H). A retrospective analysis of 100 consecutive patients who were 18 years or older and referred for assessment of mediastinal adenopathy from October 2012 to September 2013 was performed. These patients had been examined with a stepwise approach as follows: patients with a computed tomography (CT) scan in the last 30 days showing any visible adenopathy  were first assessed by regular bronchoscopy. C-TBNA biopsies of the highest abnormal node that, if positive, would upstage any diagnosis of malignancy were attempted  and reviewed by ROSE. If this resulted in a diagnosis, further LN passes were done to obtain tissue for cell block analysis and the procedure was then aborted. If no diagnosis was made, C-TBNA was attempted on the next highest staged LN group that appeared abnormal  and this process would repeat until there were no further abnormal LN groups to assess. If all LN groups were negative and there was an endobronchial lesion or mass on CT, endobronchial biopsies, bronchoalveolar lavage, or transbronchial biopsies were performed as indicated. After assessment with regular bronchoscopy, EBUS was performed if there remained LNs that were either negative for malignancy by C-TBNA or not sampled with C-TBNA due to anatomical location that, if positive for malignancy, would impact staging. If further nodes existed that were not reachable by EBUS or the endoscopist felt were easier to reach with EUS, EUS was finally performed. Patients in this time frame who did not have a CT scan in the last 30 days were assessed only using EBUS/EUS and were excluded from the statistical analysis for yield and used as a convenience sample for procedure time comparison .Data were collected on the patient's basic demographics and reason for referral. During the procedure, the endoscopy nurse recorded the start and end time of the total procedure, the start and end time of the regular bronchoscopy, the start and end time of the EBUS bronchoscopy, whether an Interventional Pulmonology (IP) fellow or IP staff physician was doing the case, and number of passes for each LN group. Data were also collected on size of LN on axial CT images, final pathological diagnosis for each biopsy specimen, and number of cells in the cell block. For each biopsy specimen, adequacy of sample size to perform ancillary studies was documented on pathology review.Samples from C-TBNA, EBUS-TBNA, or EUS-FNA were considered “diagnostic” if specific cytomorphological abnormalities were identified by a pathologist, “indeterminate” if the pathologist identified abnormal cells but could not make a diagnosis, “negative” if normal lymphocytes were obtained, and “nonrepresentative” if no lymphocytes or abnormal cells could be identified. Diagnostic, indeterminate, or negative samples were considered positive yield while nonrepresentative samples were considered negative yield. Benign samples were followed by either surgical staging or minimum of 1 year radiological follow-up.Biopsy specimen adequacy was only analyzed if a sample was positive for malignancy. This was conducted by two independent pathologists and any discordance in cell count was reviewed until they reached an agreement. Samples were deemed “inadequate” if they had less than 10 cells, “borderline” if they had 10–49 cells, “acceptable” if they had 50–199 cells, or “ideal” if they had 200 or more cells in the cell block [All procedures were performed by an attending IP physician or by an IP fellow under direct supervision by their IP attending. There were 2 IP fellows performing endoscopy procedures during that period, each in their 6th year of clinical training and both with over 2 years of experience in performing C-TBNA and EBUS/EUS.C-TBNA was performed using an Olympus T-190 bronchoscope with a 2.8 mm working channel. 22-gauge needles were used for all samples. The “jabbing method” was used for all biopsies.EBUS-TBNA was performed through an Olympus UC180F bronchoscope with a 2.8 mm working channel using an EMU1 ultrasound processor. Target LNs were identified and samples were taken from LN using standard methodology [Cytology specimens were air-dried on site and stained by a ROSE cytotechnologist who then gave an opinion on whether a representative sample was obtained. After this, 2–5 more passes in the same location were done to obtain tissue for cell block analysis.Our outcomes were the diagnostic yields of C-TBNA, EBUS-TBNA, and EUS-FNA, diagnostic yield as a function of LN station, diagnostic yield as a function of LN size, the percentage of cases where additional ultrasonographic procedures were required after C-TBNA, average number of cells per LN aspirate, and proportion of samples with sufficient cells to perform ancillary studies if needed. t-tests were used for continuous variables and Pearson's chi-square test or Fisher's exact test was used for counts as appropriate. Binary outcomes were analyzed with the use of logistic regression models. Two-sided P values of less than 0.05 were considered to indicate statistical significance. The analysis was performed using R Statistical Software, v2.15.1.Continuous variables are presented as means and standard deviations and binary variables as numbers and percentages. Two sampleThe first 100 patients referred for assessment for adenopathy were screened. Eight patients did not have a CT in the preceding month and so went directly to EBUS/EUS and were excluded from yield calculations. The remaining 92 all met the inclusion criteria and therefore were enrolled. All 92 patients were included in the final analysis and 206 LN stations were sampled. 92 procedures were analyzed and summarized in In 27 cases, C-TBNA was followed by EBUS. EBUS results potentially changed management by providing a diagnosis in 5 (18.5%) cases and upstaged a previously made diagnosis in 1 (3.7%) case. Overall, utilizing EBUS after C-TBNA was potentially beneficial in 6 (22.2%) of cases.In 33 cases, EUS was performed after either C-TBNA or EBUS. EUS potentially changed management by providing a diagnosis in 1 (3.0%) case and upstaged a previously made diagnosis in 1 (3.0%) case. In a further 3 (9.1%) of cases, EUS upstaged 3A to multinodal 3A disease which may affect treatment decisions. Overall, utilizing EUS after C-TBNA or EBUS was potentially beneficial in 5 (15.1%) of cases.C-TBNA was done on 127 LN groups; 72 of the samples were diagnostic (56.7%), 12 (9.4%) were indeterminate, 34 were negative (26.8%), and 9 (7.1%) were nonrepresentative demonstrating a 92.9% yield for C-TBNA overall. EBUS-TBNA was performed on 40 LN groups; 12 of these samples were diagnostic (30.0%), 1 (2.5%) was indeterminate, 22 were negative (55.0%), and 2 (5.0%) were nonrepresentative demonstrating a 92.5% yield for EBUS-TBNA overall. EUS-FNA was performed on 39 LN groups; 15 of these samples were diagnostic (38.5%), 2 (5.1%) were indeterminate, 23 were negative (59.0%), and 2 (5.1%) were nonrepresentative demonstrating an 89.7% yield for EUS-FNA overall.P = 1.00).The difference in yield between C-TBNA biopsies performed by an IP fellow compared to a staff physician was not statistically significant , C-TBNA alone (P = 0.19), EBUS-TBNA alone (P = 0.42), or EUS-FNA alone (P = 0.73).The difference in yields by LN station overall and in C-TBNA, EBUS-TBNA, and EUS-FNA is shown in P = 0.37). Furthermore, the relationship between yield and LN size was not affected by biopsy modality (P = 0.88).Logistic regression models did not show a statistically significant relationship between yield and LN size overall (P = 0.80). There was no statistically significant difference in procedure times if a fellow did the procedure compared to a staff physician .There was no significant difference in procedure times between cases starting with C-TBNA and following the algorithm and those using EBUS/EUS-TBNA only . 57.1% of samples obtained using C-TBNA had an ideal number of cells to perform ancillary studies if needed compared to 52.4% of EBUS-TBNA and EUS-FNA samples (P = 0.70).The average number of cells per LN sampled was 458 cells using C-TBNA (147 cells per pass), 693 cells using EBUS-TBNA (205 cells per pass), and 631 cells using EUS-FNA (217 cells per pass). 87.3% of samples obtained using C-TBNA had an adequate number of cells to perform ancillary studies if needed compared to 85.7% of EBUS-TBNA and EUS-FNA samples , this may be an unfair comparison to the EBUS group as they could represent a group with a higher proportion of benign disease. However, it is reassuring to note that the algorithmic approach does not appear to increase procedure time.Many graduating pulmonologists will practice in centers where EBUS is not readily available and those pulmonologists should feel comfortable safely and accurately staging the mediastinum with C-TBNA. This study also demonstrates the feasibility of safely teaching C-TBNA as part of an algorithmic approach in a teaching center without lengthening procedure times or sacrificing yield and adds to recent reports that yield is maintained with C-TBNA compared to EBUS when C-TBNA is performed by experienced bronchoscopists .Interestingly, although the mean number of cells obtained per pass using EBUS was greater than that obtained per pass using C-TBNA, a large amount of tissue was still obtained by C-TBNA. In addition, there were no significant differences in rates of obtaining samples with either an adequate or ideal number of cells between the C-TBNA and EBUS groups. This is reassuring for pulmonologists who are currently using C-TBNA.This study was descriptive in nature and it lacks a control group of cases evaluated without use of the algorithm for comparisons sake. Despite this limitation, we believe that following an algorithmic approach enhances the ability to teach IP fellows C-TBNA techniques without compromising yield. Now that current guidelines specifically recommend EBUS or EUS as the first step for minimally invasive mediastinal staging , furtherIn conclusion, an algorithmic approach to assessing the mediastinum using C-TBNA initially is feasible without sacrificing yield or procedure times when utilizing ROSE. C-TBNA was sufficient for diagnosis of mediastinal adenopathy in 54.3% of cases and can be safely taught in an IP training program. EBUS-TBNA or EUS-FNA should be reserved and used in a complimentary role to assess nondiagnostic results obtained using C-TBNA.