TeleMOVE, a home-based, 82-day curriculum that utilizes in-home messaging devices to promote weight loss in VHA patients facing barriers to accessing facility-based services.The burden of obesity is high among US veterans, yet many face barriers to engaging in in-person, facility-based treatment programs. To improve access to weight-management services, the Veterans Health Administration (VHA) developed TeleMOVE with standard, facility-based MOVE weight-management services  over the evaluation period based on the number of patients enrolled per site and the program’s clinical effectiveness, as demonstrated by average weight lost per patient. The secondary aim was to understand factors influencing TeleMOVE implementation variability across demonstration sites to develop recommendations to improve national program dissemination.The primary aim was to establish preliminary evidence for the program by comparing outcomes for TeleMOVE at 9 demonstration sites and compare patient- and site-level measures of program uptake. Data were collected between October 1, 2009 and September 30, 2011. Patient-level program outcomes were extracted from VHA patient care databases to evaluate program enrollment rates and clinical outcomes. To assess preliminary clinical effectiveness, weight loss outcomes for veterans who enrolled in TeleMOVE were compared with outcomes for veterans enrolled in standard MOVE! at each demonstration site, as well as with national averages during the first 2 years of program implementation. For the secondary aim, we invited program stakeholders to participate in 2 rounds of semistructured interviews about aspects of TeleMOVE implementation processes, site-level contextual factors, and program delivery. Twenty-eight stakeholders participated in audio-recorded interviews.We employed a formative mixed-methods design to evaluate the phased implementation of TeleMOVE (n=417) and MOVE! (n=1543) participants at −5.2 lb (SD 14.4) and −5.1 lb (SD 12.2), respectively (P=.91). All sites reported high program complexity because TeleMOVE required more staff time per participant than MOVE! due to logistical and technical assistance issues related to the devices. High-uptake sites overcame implementation challenges by leveraging communication networks with stakeholders, adapting the program to patient needs whenever possible, setting programmatic goals and monitoring feedback of results, and taking time to reflect and evaluate on delivery to foster incremental delivery improvements, whereas low-uptake sites reported less leadership support and effective communication among stakeholders.Although stakeholders at 3 sites declined to be interviewed, objective program uptake was high at 2 sites, delayed-high at 2 sites, and low at 5 sites. At 6 months post enrollment, the mean weight loss was comparable for This implementation evaluation of a clinical telehealth program demonstrated the value of partnership-based research in which researchers not only provided operational leaders with feedback regarding the effectiveness of a new program but also relevant feedback into contextual factors related to program implementation to enable adaptations for national deployment efforts. In 2016, 42% of the patients receiving care in the Veterans Health Administration (VHA) were obese and 37% were overweight, putting these individuals at risk for obesity-related comorbidity, functional impairment, and diminished quality of life . HoweverTeleMOVE. The goal of this joint effort was to combine MOVE! content with HT’s strengths in deploying innovative health informatics, disease management, and telehealth technologies to overcome barriers to care by delivering coordinated and supportive care management through automated communication protocols [TeleMOVE was created for delivery via asynchronous in-home messaging devices, which collect and transmit (store and forward) patient data from the patient to a care coordinator at the facility overseeing care. These devices enable clinicians to prospectively monitor and support patient self-management activities more efficiently and allow for a greater number of patients to engage in programming than relying on conventional face-to-face or phone-counseling protocols.To address these access barriers, the VHA National Center for Health Prevention and Disease Prevention (NCP), which oversees MOVE!, collaborated with the VHA Telehealth Services Home Telehealth (HT) Program to develop a telehealth program called rotocols ,10. TeleTeleMOVE implementation was driven by the need for the VHA to increase access to care services for veterans, particularly in rural regions, and less by the evidence for the application of telehealth technology to obesity treatment. The adoption of in-home messaging devices was informed by clinical evidence supporting the benefit of intensive monitoring for weight management through regular engagement in self-weighing, self-guided psychoeducational materials, and helping patients feel accountable to their health care team, particularly through regular brief motivational counseling phone calls with an interventionist such as an HT clinician [The impetus for rapid linician -13.TeleMOVE implementation. VHA policy leaders recognized that the implementation of this untested innovation would benefit from a systematic phased implementation in which VHA researchers employed pragmatic research methods to rapidly and rigorously evaluate the program to identify implementation barriers, assess clinical impact, and develop recommendations to inform national program dissemination efforts [Although there was evidence for the efficacy of individual voice recognition (IVR) and phone coaching for weight management, studies of in-home messaging devices had not been rigorously evaluated for health promotion application before  efforts -16. Nati efforts .TeleMOVE by measuring patient engagement (enrollment numbers), as well as to assess the program’s clinical effectiveness to yield weight loss by comparing weight loss outcomes with similar patients enrolled in existing facility-based MOVE! weight management services at the participating demonstration sites. Second, we examined variability in TeleMOVE implementation across demonstration sites using qualitative methods to identify contextual factors that distinguished facilities with high program implementation compared with those with poor indicators of implementation. We hypothesized that sites with higher uptake of TeleMOVE would demonstrate greater levels of program enrollment, average weight loss per participant, and theory-based constructs of program implementation relative to low-uptake sites using the Consolidated Framework for Implementation Research (CFIR) [This implementation evaluation had 2 aims. First, we sought to establish preliminary evidence for the impact of h (CFIR) ,19 to asTeleMOVE across 9 Veterans Health Administration medical centers (VAMC) over a 2-year period using a parallel in-person MOVE! cohort as a nonrandomized comparison group for quantitative analyses and qualitative interview methods to understand how contextual organizational factors influenced variability in TeleMOVE implementation uptake across sites.We used a formative mixed-methods design to evaluate implementation of TeleMOVE! was implemented in a systematic, multi-phased deployment that is described in To facilitate organizational learning, TeleMOVE. Patient enrollment began in the first week of November 2009 and continued up to February 2010. Biweekly calls were held between national program leaders and demonstration site personnel to share problems experienced in implementing TeleMOVE as well as solution strategies to overcome these challenges. The research team attended these calls to record process notes. All demonstration sites were asked to recruit at least 30 patients during phase 2 and to only use a single model of telemessaging device. In phase 3, the national rollout of the TeleMOVE program began in April 2010 and continues to the present.Phase 2 began in October 2009, with an Web-based training conference about TeleMOVE as a MOVE! treatment modality that helped patients overcome the barriers to participating in time-specific, facility-based programming via automated, asynchronous telehealth devices that offered patients a flexible and convenient approach to weight management. Notably, TeleMOVE excluded services delivered via clinical video telehealth technologies, such as live MOVE! group sessions, simultaneously broadcast to patients at a remote community-based outpatient clinic (CBOC).Standard MOVE! treatment services were implemented throughout VHA in 2006 as a comprehensive, evidence-based lifestyle approach to weight management for veterans ,3. In 20TeleMOVE was implemented with the Health Buddy, an automated messaging device developed by the Bosch-Health Hero Network , which enabled daily communication between a TeleMOVE coordinator and the participant. The messaging device was the size of a clock radio and featured 4 buttons, a liquid crystal display screen, a speaker, and a connection to a landline phone. TeleMOVE utilized a disease management protocol (DMP) that featured daily communications based on a series of algorithmic interview questions delivered to patients in their homes via a display on the device that assessed patients’ symptoms, health factors, educational needs, and self-management status. In the course of these interactive dialogues with the Health Buddy, patients entered weight information and any responses to daily prompts to be forwarded via landline phone each night to a vendor server from where it was then forwarded to a TeleMOVE coordinator for review.TeleMOVE programming modality received a Health Buddy, MOVE! handout booklet, a pedometer to track daily ambulatory activity (steps), and a digital scale to use at home. The participant and a TeleMOVE coordinator then used the MOVE! booklet to develop a patient-centered treatment plan with specific weight and behavioral change goals to monitor progress. Once this plan was agreed upon, it was recorded in the patient’s medical record. Following the installment of their device, a patient would commence participation in 82 daily communications or sessions at an agreed upon time. The DMP engaged the participant at their home, in a 5-min interactive educational module that was displayed on the device screen. These modules covered topics pertinent to weight management, such as nutrition, exercise, behavior modification, self-monitoring, and goal setting, adapted from the content used in the standard MOVE! booklet for in-person programming for individuals and groups. At the end of each module, the participant was prompted to answer a series of multiple-choice questions to evaluate user understanding. Correct answers were reinforced with positive affirmations and for incorrect responses, participants were encouraged to reference modules from the accompanying MOVE! booklet.Upon completing MOVE! enrollment activities, patients who elected to choose the TeleMOVE coordinator every 30 days that had the purpose of re-evaluating patient goals while providing motivational and problem-solving support. Coordinators also called participants for affirmative responses to red alert questions about increased pain or emotional distress. Finally, the 82 modules had to be completed within 90 days, at which point the participant would decide if they would like to repeat the program for a second cycle. Participants could pause the program for breaks up to a total of 7 days in case of acute illnesses or vacations. Although program duration spanned nearly 3 months, the total dose of patient participation was designed to be roughly equivalent to traditional clinician-delivered MOVE! modalities.The Health Buddy prompted participants to provide daily weight readings from their digital scale to encourage tracking of weight management progress. If a participant went 30 days or longer without losing half to 2 lb per week or lost weight too quickly, a trigger alert for re-evaluation would occur. Participants received 10- to 20-min telephone calls from a TeleMOVE implementation guide permitted the program to be administered flexibly either by the MOVE! or HT programs at each facility or a combination of both services. While most VHA telehealth programs were typically staffed by registered nurses, nurse practitioners, or social workers, the addition of the weight management DMP permitted other disciplines  to deliver TeleMOVE to veterans, provided they agreed to complete local, regional network, and national training related to MOVE! and HT competencies. TeleMOVE participant materials were often distributed to patients through each hospital’s prosthetics service. Given the potential cross-disciplinary staffing complexities of TeleMOVE, the implementation guide provided detailed guidance regarding the methods to track workload, coordinate the distribution of program materials, and estimate staffing resources to meet projected needs of the facility’s patient population.The TeleMOVE or standard in-person MOVE! programming during fiscal years (FYs) 2010 and 2011  at each of the 9 demonstration sites. The in-person MOVE! cohort served as a parallel, nonrandomized comparison group to evaluate preliminary clinical outcomes for TeleMOVE. All VHA weight management program enrollment occurred through each site’s MOVE! program, where each patient was required to complete a 23-item questionnaire called the MOVE!23 about their weight history [TeleMOVE. To enroll in TeleMOVE, patients had to meet additional criteria including not being enrolled in another HT program ; having a working landline telephone; and having no plans to relocate during the 6 months of the initial enrollment in TeleMOVE. Patients enrolled in these program modalities were identified retrospectively using VHA Decision Support System identifier/stop codes to capture workload credit from administrative databases.The quantitative aspect of this mixed-methods study evaluated 2 cohorts of VHA patients who enrolled in either  history  and to cenrollment versus engaged participation were based on operational definitions developed by NCP [engaged participation was defined as patients having more than 2 visits within 180 days of enrollment. The primary quantitative outcomes for this evaluation were (1) cumulative number of patients engaged in TeleMOVE and MOVE! in FY 2010 and FY 2011 at each demonstration site defined by having greater than 2 visits over 180 days; (2) mean weight loss per patient achieved after 6 months of program participation; and (3) the percentage of participants with clinically meaningful weight loss (≥5% body weight from enrollment to 6-month follow-up) [TeleMOVE effectiveness, we provided MOVE! statistics for FY 2010 to serve as nonstatistical comparator references for key weight loss outcomes, including average national mean values for weight loss, percentage of weight loss, change in BMI, and the proportion of participants achieving clinically significant weight loss (≥5% of pretreatment weight) [Quantitative data was extracted from VHA patient care databases to describe patient characteristics, program use, and weight changes associated with program participation at each of the 9 demonstration sites. Participant demographic characteristics and program utilization data were extracted from the VHA Service Support Center-hosted visits ProClarity cube. Data pertaining to medical comorbidities and change in weight were extracted from the VHA Corporate Data Warehouse (CDW). Program use was characterized by 2 indicators: program enrollment and program engagement. Distinctions in program d by NCP . Enrollellow-up) . Baselin weight) .and attaining average weight loss of at least one pound for program participants. For phase 2 pilot implementation, we assessed whether each demonstration site could enroll at least 30 patients over 4 months as an indicator of implementation effectiveness. As indicators of sustained implementation effectiveness, we evaluated whether each demonstration site could accumulate patient panels of at least 100 patients and attain average weight loss per participant by the end of FYs 2010 to 2011. Classification as a high-uptake site was based on attaining NCP/HT enrollment targets while also attaining an average weight loss per participant. Low-uptake sites were categorized by attaining only 1 or no indicators of program effectiveness . These program indicators were tracked over 2 years to assess the sustainability of early program adoption as well as to assess which contextual factors  were correlated with program effectiveness to identify possible determinants of successful implementation. Descriptive statistics were used to summarize patient characteristics and program utilization patterns. Paired t tests were used to compare patient characteristics and change in weight status from baseline, adjusting for clustering by site. Quantitative data analysis was performed using SAS Version 9.2 .Indicators of site implementation effectiveness were rated based on attaining targets for program enrollment TeleMOVE implementation were invited by email to participate in the interviews, including TeleMOVE care coordinators, HT directors, MOVE! coordinators, MOVE! dietitians, physician program champions, program support assistants, regional data analysts, and regional network program coordinators. Participation in interviews was voluntary, and we asked for additional names to ensure that we invited all individuals involved in TeleMOVE.The evaluation plan called for conducting 2 rounds of semistructured interviews at each of the 9 phase 2 demonstration sites. The first round of interviews was conducted by phone, 3 to 6 months after phase 2 (June-August 2010) and the second round of in-person interviews were conducted 6 months after the start of phase 3 (November 2010-April 2011) to capture the dynamic nature of the implementation process. Key facility- and regional-level managers and program staff involved in Verbal consent and permission to digitally audio-record interviews were obtained from participants at the start of their first interview. Staff at 3 sites declined to participate in both rounds of interviews. A total of 42 VHA stakeholders were invited to participate in an interview, and 66% (28/42) agreed to participate in at least one interview; 22 participated in the phone interviews, and 21 participated in on-site interviews. Interview ranged from 18 to 86 min in duration and was digitally audio-recorded and transcribed verbatim into Microsoft Word documents. Additionally, call minutes from biweekly conference calls held among the demonstration sites during the evaluation were analyzed to understand contextual factors affecting implementation effectiveness.The CFIR  was usedThis research study was approved by the VA Ann Arbor Healthcare System Institutional Review Board (2010-010042) with a waiver of signed informed consent for staff interviews and for secondary data analysis of deidentified patient-level outcome data.TeleMOVE and standard in-person MOVE! at the 9 demonstration sites during the first year of program implementation. Notably, among patients who enrolled and engaged in TeleMOVE, 93.9% (467/497) engaged in 2 or more visits over 6 months compared with 71.97% (1189/1652) for MOVE!.TeleMOVE during the first year of implementation. Moreover, 3 sites attained the phase 2 goal of enrolling 30 or more patients, 3 sites enrolled less than 30 (mean=25), and 3 sites recruited between 0 and 6 patients. Following phase 2, 2 of the 9 sites attained cumulative enrollment levels of at least 100 patients by the end of FY 2010, and 4 sites attained this target in FY 2011.. Both Hispanics and African Americans were more likely to choose in-person MOVE! than TeleMOVE, whereas there were no differences in enrollment based on sex. TeleMOVE enrollees were also older than those who chose in-person MOVE!, whereas rural veterans were more likely to choose TeleMOVE than urban veterans. Finally, TeleMOVE participants were significantly heavier than in-person MOVE! participants. However, there were no differences between program participants with respect to the burden of medical comorbidities as measured by the Charlson Comorbidity Index [There were significant differences in demographic characteristics between those who chose to enroll in each program modality see . Both Hity Index .TeleMOVE! and standard MOVE! was at −5.2 lb (SD 14.4) and −5.1 lb (SD 12.2), respectively, with no statistically significant differences between the 2 delivery modalities (P=.91). At the demonstration sites, both program modalities slightly outperformed average national in-person MOVE! results with respect to weight loss. These preliminary weight loss outcomes suggest that the number of patients needed to treat by TeleMOVE to achieve clinically meaningful weight loss (≥5%) is 5, which is indicative of a highly effective treatment.Preliminary indicators of clinical effectiveness are summarized in TeleMOVE implementation. Although there was significant variability in baseline characteristics of demonstration sites with respect to patients served, rurality, and proportion of patients served older than 55 years, no discernible association could be made with these facility characteristics and indicators of enrollment and clinical effectiveness below. The table combines enrollment and weight loss outcomes to generate a combined indicator of implementation effectiveness. High-uptake sites 4 and 5 not only achieved phase 2 enrollment targets of ≥30 patients but also demonstrated the ability to sustain higher patient panel sizes during national rollout while providing clinical benefit. Sites 6 and 7 exemplified cases of delayed high-uptake in which effectiveness not was manifested until year 2, when enrollment numbers increased.Low-uptake sites were broadly characterized by low enrollment at sites 1, 3, 8, and 9. However, site 2 was a low-uptake site that had high enrollment numbers without achieving meaningful weight loss outcomes. Notably, both sites 1 and 9 had low enrollment rates and displayed declines in weight loss outcomes.Among the 6 sites that participated in qualitative interviews, we identified 5 CFIR constructs that illustrated key contextual factors that distinguished high from low implementation sites: complexity, patient needs and resources, networks and communications, leadership engagement, and reflecting and evaluating. TeleMOVE was viewed as complex across all sites because it required (1) several time-consuming and intricate logistical steps simply to enroll a patient, check devices, and mail the messaging device and program materials; (2) troubleshooting installation and resolving ongoing device issues with patients; and (3) resolving issues related to the messaging devices not being user-friendly for older patients or accessible for veterans without a landline phone.Complexity refers to the perceived difficulty of implementing an intervention especially with respect to the duration, scope, disruptiveness, and intricacy of the steps involved. TeleMOVE was a progress toward helping veterans access weight management services in their homes, the technical compatibility of the telehealth device with patients’ home connectivity was an increasing barrier as telecommunications companies upgraded veterans’ landline homes to digital technology. Many patients who were interested in TeleMOVE discovered that they no longer had a home landline phone to hook up the telehealth device because they relied primarily on either cellular or digital cable phone services.Across all 6 sites, respondents felt that although Although many patients were pleased with the telehealth devices, some patients were frustrated by connectivity issues and others were disappointed with the device’s simple interface. Often, these dissatisfied patients chose not to use their device, thereby reducing program productivity due to the time needed for program coordinators to contact these patients, return the devices back to the VA hospital, and to repurpose the device for another patient.Coordinators across sites spent considerable time in calling patients to provide technical assistance, particularly for older veterans who were less confident in using the telehealth devices despite the relatively simple device interface. Coordinators also noted the need to address the ongoing issue of false alerts for issues such as low self-reported mood or errors in the transmission of weight data due to issues with the interface of the digital scales with the Health Buddy. Many of these complexity issues stemmed from the fact that most sites initially attempted to conduct screening and enrollment over the phone rather than conducting face-to-face orientations. With thoughtful experimentation, high-uptake sites found better ways during the phased implementation to mitigate complexity barriers by proactively preparing patients for the device use, whereas low-uptake sites continued to struggle with these issues without re-examining their workflow to identify areas to mitigate problems.TeleMOVE implementation complex due to unanticipated issues with the device platform and adopting new administrative procedures. Meeting minutes from phase 2 implementation conference calls among demonstration sites revealed that the requirement to administer TeleMOVE using Health Buddy devices was particularly disruptive to sites 7, 8, and 9 that relied primarily on Viterion 100 messaging devices for HT programming . The subsequent process to adopt another HT platform to implement TeleMOVE resulted in long delays. Additionally, low-uptake sites reported the process to adopt a new interdisciplinary method to assign workload credits among HT and MOVE! staff and have these changes approved by regional leadership slowed program uptake.Low-uptake sites also found the TeleMOVE moved to more flexible platforms such as mobile phones, individual voice recognition, or the internet. Staff at high-uptake sites utilized these in-person orientations to explain how to use the messaging devices, set initial personalized program goals, and to anticipate requests for technical assistance or follow-up from patients. In comparison, less effective enrollment procedures at low-uptake sites caused the staff to focus much of their attention on enrollment and reacting to technical assistance requests.This construct reflects the extent to which patients’ needs, as well as barriers and facilitators to these needs are accurately known and prioritized by program staff. Accurately assessing patients’ home connectivity was a critical component of the recruitment and enrollment process. Over time, higher performing coordinators developed in-person protocols that carefully assessed this issue during screening and enrollment, recording declined patients’ names in a file for contact when TeleMOVE with coordinators who possessed health-coaching expertise, which they utilized on both incidental and planned monthly action-planning calls to address patient problems, set goals, coordinate reported health issues with the patient’s provider, and maintain patient engagement and motivation. In contrast, lower-uptake site respondents were less likely to mention personalized contacts to help patients feel engaged, accountable, and motivated. Coordinators across high and low uptake sites also noted that the criterion that veterans only participate in one HT program at a time was counterproductive, since many eligible TeleMOVE patients had obesity-related comorbidities such as diabetes, high blood pressure, or high cholesterol and could have benefited from concurrent participation in other HT programs addressing these risk factors in an integrated fashion. Finally, respondents across both high and low uptake sites observed that although patients were prompted regularly by their telehealth devices to provide program satisfaction feedback, vendors refused to share this information with VHA. This contractual dispute denied providers with regular and consistent ratings of program satisfaction that could help inform efforts to make the program more patient centered.High-uptake sites were also more likely to report to patient needs by resourcing TeleMOVE. Failure to communicate effectively across services was more likely in low-uptake sites. In contrast, high-performing sites reported a high degree of cooperation between providers across programs and the ability to reach out to other services, providers, and leaders to gain support in planning and delivering the TeleMOVE program. One high-uptake site exhibited uniquely strong care management and coordination networks by consulting relevant providers to adjust patients’ medications for blood pressure, cholesterol, diabetes, and psychiatric self-management in response to desirable reductions in weight and lifestyle improvements.Strong formal and informal social networks of program staff and leaders were essential to effectively implement TeleMOVE was viewed as a long-term addition to HT services. At low-performing sites, there was less shared consensus about the benefits of TeleMOVE across stakeholders, and facility-level leadership was frequently unaware of the existence of the program.The presence of strong commitment and support by leaders is an indicator of an organization’s commitment to implementing an intervention. In contrast to low-uptake sites, among high-uptake sites, there was strong support by hospital leaders as well as by frontline supervisors and midlevel managers overseeing HT and MOVE! programming. This support was manifested by quick approvals for changes in workflows, staffing credit, and resources, with a shared consensus that Effective implementations require the ability to regularly reflect and evaluate both quantitative and qualitative feedback regarding the progress and quality of an intervention implementation. This CFIR construct reflects a quality improvement mindset that was present to some degree in all high-uptake sites but absent at the low-uptake sites. High-uptake sites provided specific examples of monitoring various aspects of program implementation and then using these data to identify opportunities to improve care delivery and patient outcomes. Conversely, examples of reflecting and evaluating were largely absent at lower performing sites where program staff were more reactive and less innovative in identifying solutions to issues in implementing the program.TeleMOVE weight-management program for helping veterans lose weight while identifying key characteristics of program users and contextual factors for consideration in scale-up of the intervention at other VHA facilities. This evaluation study is important because it addresses 2 major reasons cited for the low use of mHealth and telemedicine programs in medical settings on a population level: (1) insufficient evidence for the efficacy of the telemedicine innovation and (2) poor insights into how organizational and social contextual factors influence the adoption and routine use of such new technologies [Our mixed-methods findings provide preliminary evidence for the clinical effectiveness of the nologies ,28. ThisTeleMOVE throughout VHA by demonstrating the clinical benefit of the program and by providing VA leaders with insights into likely program users. New telehealth programs are often given low relative priority for adoption [TeleMOVE simultaneously addressed the burden of obesity among veterans and access to weight management treatment services by leveraging existing HT infrastructure to provide a solution to these clinical priorities. Across the 9 demonstration sites, veterans averaged a 5.2 lb (2.4 kg) weight loss over 6 months using TeleMOVE, which was comparable with results achieved in standard in-person, facility-based MOVE! programming. These results have since been replicated in subsequent observational studies [TeleMOVE users were more likely to be white, older, heavier, and living in rural areas than traditional MOVE! program participants. These characteristics are relevant to VHA clinical leaders because it is national VHA priority to reach the 25% of veterans who live in rural areas [TeleMOVE participants indicate the program reaches patients likely to benefit from weight loss support. Enrollment figures also suggest that it was feasible for the majority of demonstration sites to achieve patient panel sizes in the 80-120 patient range.Our quantitative findings helped build a case for further adoption of adoption ,29 by headoption . However studies ,31. We aal areas  with preTeleMOVE! to identify contextual factors facilitating or impeding program uptake so that refinements to the implementation plan could be made in a timely manner. Notably, stakeholders observed that TeleMOVE was a logistically and technically complex intervention that did not result in decreased staff time per patient. Reflecting evaluations of similar telehealth programs [TeleMOVE DMP interoperable across the different vendor devices to avoid delays in switching platforms.This evaluation benefited from the systematic assessment of stakeholders’ perspectives regarding programs ,33-35, pTeleMOVE implementation, including understanding the needs, resources, and circumstances of patients using the program. TeleMOVE was implemented during a period when home connectivity was rapidly changing from landline phones to digital and cellular forms of connectivity [Stakeholders told us that it is important to consider what the CFIR identifies as outer setting factors interacting with the ectivity ,37. Manyectivity ,38. As mTeleMOVE over time to tailor the program to specific patient groups. Studies at this time reflect our results that those who were less likely to engage in the program were women, younger veterans, and those living in urban areas who may have preferred other telehealth platforms delivered by IVR or mobile phone apps [TeleMOVE also reduce some of the logistical barriers that diminish the efficiency of relying on the in-home messaging devices and could allow clinicians at CBOCs in rural areas to enroll veterans rather than having enrollment controlled by program staff at a distant VA facility where HT services are centralized. However, high-uptake sites adopted best telehealth practices to reduce logistical issues and increase patient engagement for older patients by using individual or group enrollment sessions to ensure users understood the DMP protocol, how to use and connect the telehealth device, and solve issues [Prior telehealth studies point toward leverage points to improve one apps ,34,36. Te issues  and theoe issues -44.TeleMOVE implementation interacted at multiple levels within demonstration sites’ inner setting. Notably, while demonstration sites already had robust HT programs and saw a value to volunteering for the phase 2 implementation, TeleMOVE caused stakeholders to interact with other services in new ways that were disruptive and challenging in some cases. For example, the process of staffing the program and adopting new billing codes to capture workload credits for low-uptake sites was slow and frustrating at sites with poor communication networks between services. Coordination of programming was further complicated by the fact that TeleMOVE was a stand-alone platform in which program data was maintained on a separate vendor database. Accordingly, data was not readily accessible to stakeholders (patients or providers) or integrated into the electronic medical record in a usable form [TeleMOVE for veteran care.Interviews also revealed how ble form . This laTeleMOVE at their site despite their initial enthusiasm to participate in the early phases of the implementation. Hence, results may not acknowledge significant organizational and contextual factors that may impede effective implementation at a health care facility despite enthusiastic support among some stakeholders. Quantitative analyses of program effectiveness were also limited in that the original design of the study did not call for cost-effectiveness analyses of TeleMOVE relative to standard MOVE! programming. Such analyses would be difficult because a number of indicators to program effectiveness are not readily accessible from vendor databases such as types of contacts between patients and TeleMOVE coordinators, frequency of patient use, reliability indicators , and patient satisfaction data [TeleMOVE data from VHA administrative databases to perform measurement-based care, monitor program process over time, and allow policy makers to make strategic funding decisions. Finally, results from this study may not generalize to other community-based settings that lack the integration of the VHA health care setting and the significant investment in telemedicine platforms.This study is not without limitations. Notably, the study was a nonrandomized program evaluation of volunteer facilities to the implementation of clinical care program for an older generation of telehealth intervention platform. Generalizability was limited to a small number of sites and clinical stakeholders over a short period, and stakeholders at 3 sites with implementation challenges were unwilling to be interviewed regarding the specific barriers to implementing ion data . Teleheaion data . CurrentThe formative nature of this phased implementation program evaluation enabled operational decision makers to obtain real-time feedback from VHA implementation researchers to make several significant modifications to the program implementation guide and toolkit. Below are the recommendations made to operational leaders to inform national implementation efforts that were derived from our mixed-methods evaluation:Conduct initial face-to-face screenings and enrollment sessions to assess patient ability and motivation, verify home connectivity status, and proactively address technical questions related to device use and installation.TeleMOVE DMP.Ensure telehealth devices from multiple vendors could all work from the same basic TeleMOVE to concurrently address weight-related comorbidities .Allow patients to enroll in another HT DMP while in Revise implementation guides to emphasize the need for interservice care agreements between facilities MOVE! and HT services to answer and address specific implementation decisions regarding staffing, referrals, panel sizes, workload credit, staffing and funding needs, and procuring and mailing the telehealth and peripheral devices .Advocate local coordinators to assess staff competencies and encourage staff to undergo recommended standardized trainings in motivational interviewing.TeleMOVE rollout. Some recommendations could not be easily addressed. For example, staff recommended providing advanced training to HT staff in motivational counseling and in behavioral weight loss strategies. This increased level of training was not incorporated into the national implementation plan, but staff were encouraged to consult with local health behavior coordinators for assistance in these areas. There were also strong recommendations for the development of IVR software version of TeleMOVE to address the high number of interested patients without landline phones, but development of such a program was beyond the scope of the project. Finally, it has proven difficult to seamlessly update participants’ medical records with progress data from online vendor monitoring databases.These program modifications were incorporated into the implementation plan used in the subsequent national TeleMOVE and helped clinicians adopt best practices from demonstration sites to help support the existing platform until a newer platform could replace the less efficient telemessaging devices. Consequently, TeleMOVE has evolved to meet patient and clinician needs by recently transitioning to an IVR technology to allow patients to use either a landline or cell phone or use a Web browser–based technology that enables patients to use their personal computer or mobile phone . Although large health systems such as VHA lack the agility to rapidly change technology platforms, TeleMOVE exemplifies a case study where VHA’s commitment to systematic, partner-based evaluations of technology implementation efforts has allowed such interventions to spread, evolve, and sustain in the face of dynamic technological environment.Although the telehealth technology highlighted by this mixed-methods program evaluation may seem dated by today’s standards, the barriers to implementation of new generations of eHealth and mHealth technologies largely remain the same . SpecifiWe showed that an adaptation of telehealth technology could be adapted to promote clinically meaningful weight loss for veterans served by VHA, and formative qualitative data from program stakeholders could help guide national program implementation efforts when summarized by an implementation science framework. Our program evaluation highlights the benefit of implementation researchers partnering with operational initiatives to provide rigorous and rapid evaluation of the systematic deployment of promising innovation. This approach has direct application to the rapid scale-up of promising modes of telemedicine—mHealth and eHealth interventions that have the potential to help provide solutions to gaps in patient care and quality in a dynamic health environment.


IntroductionOne in two physicians experiences professional burnout. Resident coaching is a novel method to provide emotional support and professional development to residents. The feasibility of implementing coaching at a community hospital has not been reported. This curricular innovation examined the feasibility and impact of integrating positive psychology coaching at a community hospital.MethodsThe Massachusetts General Hospital (MGH) Professional Development Coaching Program (PDCP) curriculum was used to train faculty coaches and for the informal pre-coaching session refreshers. Participants were paired and expected to participate in four 1:1 coaching sessions. The impact of the PDCP was assessed through pre- and post-PDCP online surveys.ResultsTwelve interns and nine faculty coaches were included in the program and surveyed. Survey completion was 10/12 (83%) and 6/9 (67%) at baseline and 9/12 (75%) and 7/9 (78%) at end of year (EOY) for interns and coaches, respectively. For interns, Emotional Exhaustion (EE) using the Maslach Burnout Inventory (MBI) was high or medium for 60% of respondents at baseline, and 56% at EOY. Fifty percent of coaches scored medium on EE at baseline, compared to only 14.3% at EOY. Seventy-five percent of respondents rated their PDCP experience as excellent or good. Nearly all interns rated the quality of communication with their coaches highly on a five-point Likert Scale.ConclusionsImplementation of a coaching program in a community hospital residency program is feasible. Burnout using the MBI was stable from beginning to EOY for interns, but improved for coaches. Interns and coaches rated their professional development coaching program experience highly, and would recommend it to others. One in two physicians experience burnout during their career, across all specialities . BurnoutPhysician coaching using a positive psychology approach is a novel method to help prevent and address physician burnout and enhance engagement ,7. The MThe community-based categorical internal medicine program of McLaren Flint hospital is composed of a diverse group of mostly foreign medical graduate faculty and residents. There are 12 residents in each year’s cohort. The program also retains nine full-time and one part-time faculty. The previous approach to mentoring in the residency program utilized a combination of academic advisors, peer and faculty mentoring. Following the MGH model, we implemented the PDCP as a pilot, initially targeting interns. All of the 12 interns were non-US international medical graduates; there were four females. They were informed of the additional support that they would receive from a faculty coach and how that role differed from their resident mentor’s role.Our residency program supported and assisted a faculty member in becoming a certified Health and Wellness Coach and a Coaching Program Director (CPD). The CPD attended a national precourse taught by MGH’s CPD, where they described their model and provided tools for participants to replicate this program at other institutions. Our CPD then collaborated with the MGH CPD to implement the model within this residency program.Coach training and assignmentNine core faculty were invited to participate as a coach and underwent two hours of training facilitated by the CPD. Coaches were provided MGH PDCP guides for the four coaching sessions. The training combined theory and real-life practice utilizing positive psychology, mindful listening, self-reflection, exploration of possible solutions, and validation of the coachee’s self-identified strategies to reach goals. The principles embedded in our positive psychology approach are shown in Table Interns and faculty were each matched sequentially by pseudorandomization . Due to the importance of the resident being able to share weaknesses and vulnerabilities with their coaches we were concerned about coach-coachee relationships in which the coach might, at some time, evaluate the trainee. The fear was that residents would be unwilling to identify their weaknesses and vulnerabilities to someone who might later evaluate them. However, because of the small size of our faculty and their multiple teaching and supervising roles, we were unable to avoid potential coach-coachee and evaluator-resident relationships.AssessmentInterns and coaches were surveyed at baseline before implementation of the program in September 2015 and at end of year (EOY) in May 2016. The outcome measures collected in this evaluation included: 1) experience with the program; 2) professional development goals and activities; 3) professional interactions and working relationships with colleagues; 4) assessment of personal accomplishment (PA) and emotional exhaustion (EE) as measured by the Maslach Burnout Inventory (MBI). MBI is widely used to measure burnout [Our institutional review board exempted this study.At baseline, survey response rate was 83% 10/12) for interns and 67% (6/9) for coaches. At EOY, survey response rate was 75% (9/12) for interns and 78% (7/9) for coaches. As shown in Table  for inteWe asked interns how they were using skills they learned in the coaching program. Interns reported using coaching skills in their interactions with colleagues in medicine (77.8%) and nursing (66.7%), with patients (77.8%), with family and friends (55.5%) and with mentors and advisors (66.6%). When asked, “What is one aspect of the program that they would keep,” interns indicated they would like to retain meetings with both academic advisors and coaches and receive coaching on work-life balance.This investigation explored how a coaching program validated at a university academic hospital could be piloted and integrated into a community hospital residency program, primarily containing international medical graduates. Nearly all interns rated the quality of communication with their coaches highly. Their burnout was slightly below national averages. Despite the differences in program size and composition, interns reported comparable satisfaction as that of the MGH interns with the PDCP and were willing to recommend it to other residencies, indicating that the positive psychology-based coaching was helpful to their progress.We included only core program faculty members as coaches for ease of coordinating coach training and their accessibility to interns. The use of the MGH PDCP curricular material , and accIt has been shown that the learning environment strongly depends on faculty well-being , and thiLimitations of our study are that it was conducted at a single community internal medicine residency program with a small sample size limited to interns. In addition, we were unable to exclude faculty who would serve in an evaluative role. However, our successful implementation and resident satisfaction with their coaches in a small, community-based program would suggest that other programs might be able to implement this PDCP regardless of their size and faculty roles.We found that it is feasible to implement a professional development coaching program used at a university hospital in a community hospital residency program. Both the interns as well as the coaches involved in this found it to be beneficial. Future studies evaluating the impact of positive psychology-based PDCPs with a focus on residents’ personal well-being and its impact on their emotional exhaustion will be helpful.


At a time in education when it is important for programs at colleges and universities to be productive in teaching, student recruitment, and income generation, the AGHE Program of Merit review process can aid in advancing gerontology/geriatrics programs. The Program of Merit has expanded its review now Colleges of Osteopathic Medicine and any Health Professions program may apply for Program of Merit status – your institution need not be an AGHE member to apply. The POM designation provides gerontology programs and those health disciplines that include geriatrics/gerontology content in the curriculum with an AGHE “stamp of approval” which can be used to verify program quality to administrators, to lobby for additional resources, to maintain a quality program, to market the program, and to recruit prospective students into the program. This session will begin with the “Why,” “What” and “How” of applying for Program of Merit and then provide time for small group consultation regarding the Program of Merit application. Preparation of the self-study document will be detailed by the experienced faculty members followed by the review process and timelines associated with application process. Free consultation available!


Background: The heartache from the devastating 8.0 magnitude Wenchuan earthquake, which killed nearly 90,000 people in western China, is still felt despite the large-scale recovery and reconstruction of the affected areas. This study investigated the relationships of earthquake-trauma exposures and personality with posttraumatic stress disorder (PTSD), to identify the long-term consequences of the Wenchuan earthquake on the survivors and the risk factors related to chronic PTSD. We hope the findings can contribute to developing new health care prevention and interventions for the survivors.Methods: We collected a sample of 490 people over 3 years after the Wenchuan earthquake, using questionnaires about demographic information and the traumatic experience in earthquake, the Impact of Event Scale-Revised (IES-R), and the Eysenck Personality Questionnaires (EPQ), to find the consequences of the Wenchuan earthquake on the survivors and the potential factors related to the long-term morbidity of PTSD.Result: Traumatic experiences, such as witnessing someone being seriously injured, having your house seriously damaged, and having close relatives severely injured, were associated with developing PTSD. Personality measured by EPQ was also closely related to PTSD. Regression analyses indicated that a potential linear model characterized the relationship between PTSD, neuroticism and psychoticism, yet extraversion/introversion were not significant factors. In the multivariate logistic regression, neuroticism (a continuous variable measured by EPQ) was of more significance in predicting the morbidity of long-term PTSD, compared with other variables . The final path diagram built a model indicated the moderating role of personality in the relationship between traumatic experiences and PTSD .Conclusion: This study demonstrated the role of personality traits and subjective exposure experiences regarding the vulnerability associated with PTSD after earthquake. Among all personality traits, neuroticism is considered a vulnerability factor of PTSD, and other personality traits also moderate the effects of traumatic experiences associated with PTSD. These findings might be useful for psychologists to develop intervention strategies for people suffered natural disasters, and to help individuals with PTSD to heal fully. Exposure to a disaster can trigger a wide range of psychological consequences including severe psychological distress, immense physical devastation and emotional suffering coupled with a terrible feeling of loss, which could severely affect individuals’ psychosocial functioning and quality of life . On May Posttraumatic stress disorder (PTSD), a trauma-related mental disorder, has been found to be the most common psychiatric disorder among earthquake survivors , 6. PTSDStudies from the past decade have focused on PTSD and potential predictors of PTSD symptoms after disaster. Liang et al.  concludeGiven that long-term PTSD is the most common psychological disorder after an earthquake, the current study aimed to investigate the predictive factors of PTSD about 3 years after the earthquake, and to examine the interplay among the risk factors relating to the genesis of PTSD. We conducted a comparative analysis among a large sample of the Wenchuan earthquake survivors to examine the relationship between personality traits and traumatic experience on PTSD while controlling for other risk factors. According to the estimation, these survivors of the Wenchuan earthquake included a large amount of adolescents, who are generally more vulnerable when they are exposed to trauma and, thus, they often exhibit traumatic reactions following disasters and suffer from longer mental injuries . Therefovia a simple random sampling method. As the disaster area is relatively underdeveloped, schools in each city are mainly concentrated in one area, where the sampling collection is suited to be conducted. Hence, with a stratified sampling method, participants were recruited from junior and senior high schools at the unit of class. In each grade, we randomly selected half classes of this grade in the school. Background information included traumatic history about the participants which was collected to evaluate if they were eligible for the following survey according the following criteria.A cross-sectional survey was conducted in earthquake affected areas of Sichuan province through the multistage cluster sampling method. The data were obtained from two sample surveys between July 2010 and May 2011. First, we randomly selected two cities (Sichuan-Aba Tibetan & Qiang autonomous prefecture and Mianyan Qiang autonomous prefecture) from the hardest-hit areas. In these two prefectures, Maoxian and Beichuan were selected The Wenchuan earthquake experience is the only traumatic experience to them; no experience of other types of trauma such as losing a family member, divorcing, or car accident.Can read and write normally.Finally, we distributed a total of 490 questionnaires and excluded 33 questionnaires that were not completed by participants from the earthquake-affected area, resulting in 457 valid questionnaires (93.3%). The error and missing rate of the data was lower than 10% and replaced by the mean value. Participants were equally capable to complete the questionnaires since they were middle school and high school students. Participants’ age ranged from 12 to 20, which is higher than the average age of the usual middle and high school students. This is due to late attendance at school among minorities because of financial or cultural reasons. The criteria for the target group are listed below:The surveys include (1) demographic measures such as gender, age, and education level (grade); (2) traumatic experience in earthquake; (3) the Impact of Event Scale-Revised (IES-R) and (4) the Eysenck Personality Questionnaires (EPQ).The questionnaire includes 10 questions:E1- Having been in serious danger;E2- Having witnessed someone being seriously injured;E3- Having witnessed people dying;E4-Having close relatives  injury severely;E5-Having close relatives  dead in the earthquake;E6- Having good friends injured severely or dead;E7- Having been your body hurt;E8- Having your house damaged seriously;E9- Having witnessed a tragic scene after the earthquake;E10- Having felt scared about the earthquake.th question was measured on a Likert scale, ranging from 1 (slightly) to 3 (intense).The first 9 questions were coded into yes/no items, while the 10The Impact of Event Scale-Revised (IES-R) was a 22-item commonly used, psychometrically sound self-report questionnaire to measure PTSD symptomatology –33. EachThe Eysenck Personality Questionnaire measures traits of psychoticism, extroversion, neuroticism, and lie scale. It has the needed quality of psychological scale and has been employed at home and aboard for at least 40 years . The ChiThis study was approved by the ethics committees of the Second Military Medical University in Shanghai. After obtaining the approval from the dean and teachers in the schools, informed written consent was obtained from adolescents’ guardians, together with oral approvals from participants. According to the Declaration of Helsinki, a complete survey description was first presented to the participants. Five experienced researchers with training in epidemiology and unified training went from class to class to distribute and explain the purpose of the survey. Each student received 10 CNY in exchange of taking the survey, and each researcher was paid 200 CNY. After surveys were collected, researchers would do a site assessment.2 test. Linear regression analysis and t-tests were conducted to examine the relationship between personality and PTSD, both were entered as continual variables. Multivariable logistic regression was used to analyze the associated variables and their prediction to the morbidity of PTSD. The entry and removal criteria for the variables in the forward stepwise analysis were 0.05 and 0.10, respectively. Odds ratios (ORs) and their 95% confidence intervals (CIs) were presented to show the association. P < .05 was considered statistically significant and the value of Wald implied the predictor dependability. Data were analyzed in SPSS version 17.0 . Structural equation model was built and analyzed in AMOs 21.0 in the way purposed by Ping’s studies  and 191 females (41.8%) with a mean age of 15.47 ± 2.28. The subjects comprised 252 junior high school students (55.1%) and 205 senior high school students (44.9%). They are from different ethnic groups. 382 (83.2%) Qiang, 34 (7.4%) Hui, 24 (5.3%) Han, and other took up 3.7% (n = 17). We divided the participants into a PTSD group (n = 104) and a non-PTSD group (n = 353) with the measure of IES-R and the score 40 as a cut-off, for studying effects of the traumatic experience and then performed the chi-square test in the PTSD group on the demographic data and traumatic experience.2 = 0.11, p = 0.916; age:χ2 = 2.131, p = 0.546, the educational level:χ2 = 7.673, p = 0.06, at the standard of α = 0.05). As for the traumatic experience, the answer for E2, E3, E4, E6, E8, E9, and E10  had closely relationship with PTSD, while no significant relation with other exposures (P > 0.05). Hence, a significantly higher prevalence of PTSD symptoms was found among people who had witnessed someone, either relatives or good friends, who were seriously injured or dying, who had house damaged seriously, who had witnessed a tragic scene after the earthquake, and who had felt scared.The results illustrated in the 2 = 0.208) but slightly positive linear fit of psychoticism and PTSD (R2 = 0.016), whereas a3 shows no significant liner relation between extraversion/introversion and PTSD (R2 = 0.009). The results of bivariate logistic regression analysis were elaborated in The structural model diagrammed in The current study investigated 457 survivors between one and three years after the Wenchuan earthquake. We concluded that the answer for E2, E3, E4, E6, E8, E9, and E10 were closely related to the PTSD. It could imply that the bereavement and interest loss, which hardly compensated for the loss of cherished memories, had the worst influence on someone’s mental health. The result of multivariable logistic regression using backward stepwise elimination showed that E2, E4, and E8 were predictive to PTSD with positive relations, all of which were mental injury caused by a sense of losing the loved, instead of from direct bodily injury. Moreover, personality had a significant relationship with PTSD, which was in line with expectations and previous research. Furthermore, EPQ-N is a significant predictor of PTSD after earthquake. Neuroticism is positively related to PTSD. This finding of personality in the context of earthquake is in line with previous findings in various contexts such as fighters, transportation accidents, industrial and domestic accidents, terrorism, violent crimes, and explosions , 41, 42.Although traumatic experience could be an independent predictor of PTSD , the relAlthough preexisting studies found many factors had a relation to PTSD from the earthquake , there wSome clinical implications could be attained from our study. On the one hand, our results confirmed that personality could be a common factor for mental diseases and neuroticism, especially, was the strongest predictor to PTSD. The defect of personality like high sensitivity and instability should be remedied. In our cohort, these students with PTSD should receive help for the development of personality from the psychological treatment as consummating the personality would be benefit for treating and terminating the chronic PTSD. On the other hand, psychological workers should increase attention and intervention promptly to the neurotic people who experienced trauma in case of the mental injury. Meanwhile, in a sense, self-improvement in personality could be not only praised for self-achievement, but is also good for psychological immunity.Several limitations should be noted. First, its cross-sectional design precludes inferences about causality. Second, this study relied entirely on self-report measures. Some researchers suggested that personality could be influenced by traumatic experience, and victims may subsequently perceive themselves, their identity, and reality differently after the experience of earthquake . Thus faDespite these limitations, we provide stronger evidence to date that personality is correlated to the possibility of PTSD in the survivors experiencing earthquake trauma. Specifically, we identify neuroticism, which implies emotional instability, as a major predictor of the prevalence of PTSD after the earthquake. As post-disaster stressors, traumatic experiences like bereavement and deprivation are also risk factors of PTSD. For the first time, we found that personality has moderating effects on the relationship between earthquake-trauma experience with PTSD in a long term. Intrigued by the Wenchuan earthquake, advanced studies should conduct a systematic investigation for predicting the long-term PTSD and form a more comprehensive understanding of the predictors. These works aim at providing valuable guidance for health care workers to develop efficient early interventions for the trauma population and reduce the prevalence of PTSD. Each year, nature diseases can affect an average of about 200 million Chinese population and kill several thousand people. In the future, this work can be applied to the intervention of PTSD in trauma populations from other nature diseases.As one of the largest earthquakes in human history, the devastating Wenchuan earthquake caused tremendous damage in western China. Over the past ten years, people have successfully rebuilt their hometown and secure living environment with the worldwide help. However, many survivors still live in the shadow of mental illness and psychological distress, so the rebuilding of a spiritual home deserves continuing efforts. Our study hopes to commemorate the victims to contribute to the recovery of mental health among the injured survivors. We found a close relationship between personality and PTSD after earthquake, and personality moderate the effects of traumatic experience on PTSD. Specifically, neuroticism might be a stable vulnerability to the PTSD in the long term. Therefore, neurotic people who are exposed to traumatic experiences of bereavement or deprivation should receive more attention and early health interventions.QY was the first author responsible for the writing and data analysis. WL was the equal first author for the review and data analysis. XY contributed to the edition and review. LW was the director of the research and the corresponding author.This research was supported in part by a grant from Shanghai Pujiang Program (13PJC003) and Innovation Program of Shanghai Municipal Education Commission (14ZS084).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


There is a need for evidence-based substance use prevention efforts that target high school-aged youth that are easy to implement and suitable for dissemination in school and community groups. The Youth Message Development (YMD) program is a brief, four-lesson, in-person curriculum that aims to prevent youth substance use through the development of youth media literacy. Specifically, YMD aims to increase understanding of advertising reach and costs, along with the techniques used to sell products; develop counterarguing and critical thinking skills in response to advertisements; and facilitate application of these skills to the development of youth-generated antisubstance messages. Although YMD has demonstrated evidence of success, it is limited by its delivery method and focus on alcohol and smoking.Study objectives were two-fold: (1) to adapt the YMD curriculum to a self-paced, interactive, electronic-learning (e-learning) format and expand its content to cover alcohol, combustible cigarettes, e-cigarettes, smokeless tobacco, marijuana, and prescription drugs, and (2) to test the feasibility of the adapted curriculum in partnership with a national youth organization.An iterative process was employed in partnership with the 4-H youth development organization and a technology developer and consisted of six phases: (1) focus groups to guide adaptation, (2) adaptation to an e-learning format renamed REAL media, (3) pilot-testing of the REAL media prototype to determine feasibility and acceptability, (4) program revisions, (5) usability testing of the revised prototype, and (6) final revisions. Focus groups and pilot and usability testing were conducted with 4-H youth club members and adult club leaders.Focus group feedback guided the build of an e-learning prototype of REAL media, which consisted of five online levels and interactive content guided by a mix of narration and on-screen text. Results of a pilot test of the prototype were neutral to positive, and the program was refined based on end-user feedback. An independent usability test indicated that youth 4-H members felt favorably about navigating REAL media, and they reported high self-efficacy in applying skills learned in the program. Additional refinements to the program were made based on their feedback.The iterative build process involving the end user from the outset yielded an overall successful technology-driven adaptation of an evidence-based curriculum. This should increase the likelihood of effectively impacting behavioral outcomes as well as uptake within community organizations. National survey results show a significant increase in prevalence rates of youth substance use during high school, which typically spans ages 14 to 18 in the United States . For exaGiven that youth experimentation with substance use begins in early adolescence (or middle school), the majority of evidence-based substance use prevention efforts target youth of this age to prevent further escalation of use -9. Yet, Youth Message Development (YMD) is one example of an evidence-based program targeting youth in middle adolescence ,14. YMD Initial research on YMD focused on smoking-specific ads, and results indicated a positive impact on beliefs about smoking as well as intentions to smoke among youth who received YMD relative to controls ,19. A foTo overcome these challenges of YMD, this paper focuses on adapting YMD from an in-person to an electronic-learning (e-learning) program, as well as broadening its focus to include other commonly used substances in adolescence. Much of human interaction and information seeking is migrating to electronic format, particularly among this age group, and substance use prevention efforts are no exception ,21. ThisA six-phase process was employed to adapt the YMD curriculum from an in-person, paper-based curriculum to an e-learning program, and subsequently test the feasibility see : (1) focParticipants for our user feedback phases  consisted of 4-H members (n=76) and leaders (n=16) recruited from clubs in New Jersey and Maryland . Input was also solicited regarding the new program name. All focus group sessions were audio recorded and later transcribed. Detailed notes were taken for each session by the cofacilitator and circulated to the lead facilitator after each focus group to confirm accuracy.The YMD curriculum was adapted to an e-learning prototype in a collaborative effort between the research team and technology developer.The YMD curriculum consists of four lessons, is approximately 90 minutes in length, and is delivered in-person by a trained facilitator either all at once or in multiple sessions. Lessons focus on (1) media reach and strategies advertisers use to sell products, (2) claims in advertisements and counterarguments to those claims, (3) production techniques advertisers use to get attention , and (4) the application of content learned in lessons 1 to 3 to the development of a drug prevention message in the form of a poster ,14.The development of the e-learning version of YMD, named REAL media based on focus group input and project team discussions, was an iterative process. Guided by the YMD curriculum, as well as feedback from the focus groups, the research team developed scripts for each lesson to guide the translation of content to e-learning format including on-screen text, narration, and interactive components. The Web development team offered its own expert feedback, and the entire team worked together to clarify the vision for REAL media. After each lesson was developed, the research team would provide feedback and the Web development team would make additional modifications. This process continued until the content was accurate and any observed technical glitches were resolved.Five 2-hour pilot-testing sessions were conducted with 4-H leaders and members to assess the feasibility and acceptability of the REAL media prototype.Each participant was provided with a laptop computer with a wireless internet connection, mouse, and headset. Participants were asked to complete, at their own pace, as much of the REAL media program as possible during the 2-hour session. Participants were not asked to create an antisubstance use message—the final component of the REAL media program—due to time constraints, but they were provided with sample files to upload to test the program’s functionality. Participants also completed a brief survey to evaluate the performance of and their engagement in each level, what they liked and did not like, as well as demographic items. This was followed by a participant debrief led by the research team.The 4-H members’ engagement at the end of each level was assessed with 12 items adapted from the Audience Engagement Scale  and NarrChanges were made to the prototype based on feedback from the pilot-testing sessions.The research team communicated the suggested changes to the programmer, who in turn made the edits before the usability test. The team verified the changes and performed internal testing before moving onto the next phase.After revisions to the prototype were made, an independent usability test was conducted with a sample of 4-H members and leaders with no prior knowledge of the REAL media adaptation process.Procedures for the usability test were similar to the pilot-testing procedures. Each participant was provided with a laptop computer with a wireless internet connection, mouse, and headset, and asked to complete levels 1 to 4 of the REAL media program during the 2-hour session. Participants were not asked to complete level 5 due to time constraints. Participants completed a brief survey to evaluate the usability after each level and the overall program.Program usability of each level was assessed with 10 items adapted from the System Usability Scale (SUS) . Sample A list of changes to be made to REAL media before conducting a large-scale evaluation were identified.After the usability test, the research team analyzed the findings and documented a list of recommended changes to be made upon securing funding to finalize the program and evaluate its efficacy. This list also included any lingering issues identified in the pilot test that could not be accommodated before usability testing.Focus group feedback was solicited on curriculum content and format with considerable time spent brainstorming suggestions for adapting the face-to-face content to a Web-based platform. Feedback was consistent across both youth and leaders, and it was organized by type .Much of the content-specific discussion focused on the advertisements depicted in the curriculum, including their appropriateness and relevance to the audience. Youth participants were able to identify sample ads they liked as well as ads that did not resonate with their age group. One of the concepts in the curriculum focuses on ads that use sex appeal as a strategy to sell products. Youth participants acknowledged the importance of including this imagery because it was an accurate representation of what they were exposed to, but also noted the need to avoid showing “overly” sexual images for the more conservative members and leaders and/or younger club members. For example, showing a male celebrity in an underwear advertisement was deemed acceptable only if the image was cropped above the waist.Another focus of the content-specific conversations related to the need for acronyms that resonated with the audience, and to adjust existing labels for key concepts accordingly. For example, the original acronym for the four strategies advertisers use to sell products was FUGE, which stood for fun with the group, unexpected/humor, glamor/sex appeal, and endorsement techniques, respectively. Youth participants suggested swapping the “G” to an “S” to produce FUSE, which would be easier to remember and could even be represented with an animated fuse. They also suggested relabeling the “S” from sex appeal to style, to avoid terminology that might be deemed inappropriate.Participants had many suggestions for how to adapt the curriculum content to a Web-based platform, including the use of narration to minimize on-screen text, interactive activities or games to maximize user engagement, positive feedback to encourage users when completing activities, the need for user flexibility in navigating the content, and age-appropriate communication. For example, they suggested avoiding a narrator with an adult voice, but also cautioned against someone who sounded too youthful or kid-like. Similarly, they wanted the narrator to offer positive feedback for correct answers that was simple  without being overly enthusiastic. Examples of suggested interactions were to allow the user to be able to manipulate images or ads to better highlight relevant lesson concepts and/or maximize contrasts and to offer a variety of interaction types. Participants also suggested that some users might want additional information on topics of interest and asked that additional content be made available to those seeking it. Labels were offered for program features that would avoid a school-like feel. For example, “lessons” could be “levels” and a “quiz” at the end of each lesson could be called a “challenge.” Finally, participants suggested potential names for the curriculum.Guided by the focus group results as well as the iterative feedback process described in the Methods, a five-level  e-learning course was produced. From submission of the first script to completion, the build process took approximately 12 weeks. The main concepts from the face-to-face curriculum were retained and presented via a mix of on-screen text and narration, the latter of which was recorded by a younger, college-aged female. Although the face-to-face YMD curriculum consisted of four lessons, the first lesson was split into two separate lessons for REAL media to achieve a better balance of content and timing throughout. Each lesson was projected to take approximately 20 minutes to complete. In addition to the on-screen text, there were several interactions per level, including drag and drop, multiple choice, fill in the blank, sliders, and hover/reveal. Each level concluded with a brief “challenge” in which users were asked to apply the core concepts illustrated in the level. Finally, to offer interested users the option to see additional content, levels included “optional depth” segments in which users could learn more about the topics covered in that level. Additional resources were also included with each level. The inclusion of optional depth and resources allowed the user to “customize” or “personalize” their experience, a technique found to increase engagement and effectiveness .Once built, the five levels were hosted on the technology company’s Learning Management System that was programmed so that users would be able to log in to their own personal account page and access the program. Each level was locked, meaning that participants had to proceed through the program sequentially by level; they did not gain access to the next level until the prior one was completed. To accommodate the less tech-savvy users, written instructions and a video were placed on the Learning Management System home screen to guide users through the log-in process.Ratings on indicators of realism, interest, and identification for the audience were neutral to positive overall as reported by both members and leaders see . AverageRatings on indicators of personal reflection on the impact of advertising and substance use, perceived novelty, and critical thinking about advertisements also were positive, with most averages for members close to agree (=4) for novelty and critical thinking. Member ratings for personal reflection were closer to the neutral point (=3), suggesting that not all members thought a lot about their own substance use after using the program.Given the small sample of leaders, formal comparisons of their responses with those of members were not conducted. Nonetheless, means for leaders’ perceptions of personal reflection and critical thinking were higher than mean scores of members. Conversely, realism scores were higher for members than leaders.Positive comments focused on content and technical aspects of the program. For example, specific to content, one user noted the examples were realistic and that made the program more relatable for teens. Comments related to technical aspects were focused on visuals, audio, interactive features, and overall program pace. Users reported the interactive features helped them to feel more engaged, the imagery helped keep their attention, the sound effects were positive, and the program was conducted at a nice pace.Negative comments also focused on both content and technical aspects. Specific to content, participants noted words and concepts that were hard to understand. On the technical side, participants noted frustration with load time and other technical glitches. Other comments related to overall user experience focused on increasing flexibility, such as making it easier to go back to concepts they wanted to review. Although some participants noted they liked pacing, others noted it felt rushed. Other suggestions were to increase font size in certain places and to use more narration and less text.Based on feedback from the pilot test, edits were made following a procedure similar to the original development such that the research team reviewed the changes and offered feedback until minimal glitches were observed and the revised content was satisfactorily incorporated. Specific edits made included adding additional content to the log-in page to aid users in navigating the program , adding a feature so users could enlarge images, uploading higher quality versions of embedded video content, updating screens with new images, fixing minor content issues such as errors in grammar, fixing minor technical glitches , and adjusting contrast of text and/or background colors to enhance the visual experience. Means for all constructs assessed during the usability test are presented in Mean SUS scores were favorable, with scores falling between agree (=4) and strongly agree (=5) for all levels see . Scores Ratings for confidence in using the information learned in REAL media to create counterarguments, use those arguments to convince others about media messages, use lessons to create advertisements, and change other people’s behavior via self-created advertisements were all high, with average scores falling between agree (=4) and strongly agree (=5) see .Similar to the pilot-testing feedback, comments from users on what they liked and did not like related to both technological aspects of the program and content. From a positive standpoint, participants liked the narration overall , visuals , and the overall ease of use or navigation of the program . Related to content, participants reported the program was informative, fun, and engaging, and liked the interactions.Technological areas to improve included a more appealing sign-in page, as well as ongoing timing issues  and loading errors . Some features were confusing and needed additional explanation . Participants also noted the overall program could be shortened, particularly for a 2-hour session. From a content perspective, participants noted some sections were challenging , and they suggested refining the content presented to make concepts covered in this level easier to understand. Participants also requested more feedback from the program when they provided an answer .Based on the feedback from both the usability and pilot, the research team identified further modifications for potential implementation in phase 2, the goal of which is to evaluate the impact of REAL media on user behavior prospectively and test the conceptual model guided by the Theory of Active Involvement as described previously.From a technical perspective, a user-friendly log-in system is needed. Minor issues include edits to slide timing, transitions in content, and load time for each level.From a content perspective, the youth requested more voiceovers, so there would be less on-screen text for users to read. In addition, we plan to streamline some of the repetitive content (particularly in level 3), which will be helpful for individuals who complete the program in one sitting. We will also offer more examples of challenging concepts, including claims, missing claims, and counterarguments (level 3). Finally, we plan to add a social media contest as an outlet for youth to share their antisubstance messages with their family and peers once they complete the e-learning curriculum.Adolescent substance abuse remains a significant public health concern ,2, and mOne of the main implications of these findings is the need for high levels of engagement with prevention materials. The potential for an online curriculum to meet these needs, and the challenges in doing so, are amply illustrated. This work identified both content and technical techniques for engaging the user. At the same time, the need for customization and personalization was also illustrated. For example, users want flexibility to navigate the program on their own terms, explore additional content if they choose, and receive personalized feedback on interactive elements including open-ended responses and some responses with feedback .One of the benefits of this innovative, brief, intervention component is that while potentially affecting substance use alone or in combination with a comprehensive intervention, it encourages the development of higher-order critical thinking skills valued in high school curricula under current teaching standards in most states. Favorable ratings on self-efficacy to use the skills REAL media aimed to build were notable and should only improve once additional edits are made particularly around the challenging concepts in level 3. This complementarity with curriculum standards should foster dissemination.It is worth noting that the iterative process employed to adapt content to REAL media  allowed the research team and technology developers to gain greater understandings of each other’s desires and limitations. Although extensive discussions occurred prior to initiation, it is our experience that differences in language, culture, and expectation require continued adjustments (and patience). Even though the technology company had considerable experience in training and implementation development, the media literacy approach was new to them and required considerable adjustments. Further, much discussion was required for both parties involved to understand respective goals. For example, our technology partner did not initially grasp our need for extensive program data capture to support self-report measures.Overall, the curriculum demonstrated sound functionality and engagement through the iterative adaptation process. Although initial engagement ratings reported during the pilot test were not as high as we would have liked, improvements were made before usability testing. and the self-reported usability scores were very positive. We hope that additional changes scheduled to be made in response to usability feedback, as well as pilot feedback that was not able to be incorporated due to a short timeline, will further improve engagement. There may also be a ceiling effect for engagement in any intervention that has educational and prevention goals. It is an empirical question whether further improvements in engagement would yield increases in program outcomes.We would also be remiss if we did not note the partnership model adopted throughout this research. Although the obvious partnership is between program developers (research team) and the technology developers, one cannot underestimate the importance of including the end user, in this case the 4-H organization, from the start. As opposed to the “build it, and they will come” model of intervention development, the research team adopted a partnership model that incorporates the end user from inception and has been applied successfully to other curricula adaptations . As a reFinally, there were a few limitations of note. Given the pilot nature of the study, we did not assess teen participants’ expectancies related to and actual use of substances. It is possible youth substance use tendencies and beliefs could influence their perceptions of the intervention. Accordingly, future feasibility studies on this topic should consider including these measures. In addition, participants were not selected at random. Thus, it is possible the individuals who self-selected into the study are not representative of 4-H participants as a whole. The sample was also small in size and predominately white. All these factors together potentially limit the generalizability of the results. The small sample size also limited our ability to look at differences in results between participating clubs.In conclusion, this paper describes an iterative development process for adapting evidence-based, face-to-face, manualized prevention programming to an online format whereby the end user is involved from the outset. Both challenges and triumphs were experienced throughout the process, and efforts were generally successful overall. We believe that entering the field with a more fully developed curriculum increases the chances of effectively impacting behavioral outcomes as well as the likelihood of uptake.