Since 2008, the McMaster University Royal College Emergency Medicine residency training program has run practice Short Answer Question (SAQ) examinations to help residents test their knowledge and gain practice in answering exam-style questions. However, marking this type of SAQ exam is time-consuming.To help address this problem, we require that senior residents help mark at least one exam per year alongside faculty members. Examinees’ identities are kept anonymous by assigning a random number to each resident, which is only decoded after marking. Aggregation of marks is done by faculty only. The senior residents and faculty members all share sequential marking of each question. Each question is reviewed, and exemplar “best practice” answers are discussed. As novel/unusual answers appear, instantaneous fact-checking  and discussions occur allowing for real-time modification to the answer keys as needed.A total of 22 out of 37 residents (post graduate year 1 to post graduate year 5 (PGY1 to PGY5)) participated in a recent program evaluation focus group. This evaluation showed that residents feel quite positive about this process. With the anonymization process, residents do not object to their colleagues seeing and marking their answers. Senior residents have found this process informative and have felt that this process helps them gain insight into better “examsmanship.”Involving residents in marking short-answer exams is acceptable and perceived as useful experience for improving exam-taking skills. More studies of similar innovations would be required to determine to what extent this may be the case. Examinations are still markers of final success for medical learners. Even as we progress to a competency-based framework for postgraduate medical education, it is unlikely that we will entirely abandon this age-old tradition. Additionally, the role of examinations as a part of an assessment portfolio for determining and fostering competence can be better integrated. Testing in medical education has been well-described previously -3, with Progress testing is a form of formative testing that harnesses the properties of test-enhanced learning and manifests them into a specific practice. As a result, progress testing has been utilized in medical education, both as a method of encouraging retrieval practice and also as a way of allowing learners to gain external insight into their performance compared to their peers, possibly spurring further engagement and interest in studies via social motivation. At McMaster University, we have had a long tradition of frequent progress testing for undergraduate students -5, but iSince 2008, McMaster Emergency Medicine has run quarterly, locally-developed progress tests that are aligned with our curricular blocks (dubbed “Block Exams”). The purpose of these progress tests is to encourage continuous studying and review of the curriculum topics and to provide opportunity for retrieval practice. Moreover, these exams consist of short answer questions (SAQs), mirroring the style of both the national in-training exam (CITE) and the terminal examination given by the RCPSC.Each block, half of our week’s academic time is allocated for a test (75 minutes), usually consisting of seven to 10 SAQs. The answer key is generated by the exam question author and peer reviewed by our associate program director (KS) before use. Questions are aligned to the main topics covered in the preceding three months’ academic time, although there is usually one question per block exam that is meant to force more remote information retrieval.When this program began in 2009, our program had only 20 residents; however, in the subsequent years our program nearly doubled in size (37 residents at present). As such, in five short years the human resources required to mark and provide good quality feedback to the learners on their practice tests significantly increased.To help address this problem, our program decided to pilot and then implement a program that involved senior residents in exam marking. Initially this program was volunteer-based; however, as residents participated and saw merit, our program shifted to make this a mandatory experience for all senior residents each year. Currently, we require senior residents to help mark at least one exam per year alongside faculty members. Figure The senior residents and faculty members all share sequential marking of each question. Each question is reviewed and exemplar “best practice” answers are discussed. If novel or unusual answers occur, instantaneous fact-checking  and discussions occur allowing for real-time modification to the answer keys as needed.We conducted a focus group-based program evaluation about this program. We received an exemption from the Hamilton Integrated Research Ethics Board for this program development initiative, and we did receive informed verbal consent from our participants. After receiving an exemption from our institutional review board, we conducted a series of focus groups with our residents using a non-MD (SAL) interviewer. These sessions were recorded and subsequently transcribed and then thematically analyzed with two members of our team (an MD (TC) and a non-MD (SAL)). We used an interpretive descriptive technique  for our Twenty-two out of 37 (60%) residents from our local program participated in our focus groups. Residents were interviewed in their postgraduate year cohorts. Participants ranged from postgraduate year (PGY) 1 to 5. Generally, the progress testing was viewed positively. Our residents noted that the progress testing was beneficial in providing a source of motivation to study and a measure of performance that could be trended overtime and compared to their PGY cohort. Details of the themes, subthemes, codes and quotes that emerged in the focus groups are displayed in Tables Residents discussed the usefulness and barriers for progress testing. The grades from the block exams provided a trend for professional progression in the residency program, which afforded the resident the opportunity for personal reflection on their performance to date. Residents also used grades as a barometer to compare their progress with other peers in the same cohort. The block exams also helped engage residents to identify learning gaps and validate the knowledge and skills acquired during residency. Residents saw block exams as a motivator, specifically for encouraging the review of recently learned content, ensuring performance conformity among their peers, and managing their own identity as a physician in training. Even though residents viewed block exams as a source of stress, they perceived the block exams as a very helpful motivator to review the learned material. Our learners noted variable motivation levels in the various years of residency, with increasing motivation as residents approached their final examinations. Junior residents noted that they were more motivated to learn by clinical encounters and assessments.We also inquired about their perceptions about the peer marking experience. With the anonymization process, residents did not object to their colleagues seeing and marking their answers. Factors that facilitated acceptability were the use of a standardized marking key and that most residents felt that their identities were protected by the anonymization system (e.g. numbered identifiers). Only one senior resident thought he could identify a peer’s handwriting. One junior resident especially felt that senior residents provided better feedback, since they perceived that these near-peers could better explain how to improve. Furthermore, junior residents perceived senior residents as more reliable and having the ability to provide a more thorough review. Senior residents found that participating as a marker of these exams was informative, many of them remarking that this process helped them gain insight into better exam-taking skills (“examsmanship”). For instance, senior residents felt that having exposure to their peers’ exams during marking would widen their perspective in terms of how certain questions are answered. Overall, the residents agreed that there is educational value in marking exams because it creates an opportunity to discuss and elaborate on the answers with other peers and faculty members.  In the age of competency-based medical education, it is likely that single, final, high-stakes exams may become de-emphasized as we integrate more continuous monitoring of performance. That said, it is likely that there may still be some component of testing in most jurisdictions, as it is simply a very efficient way to ensure knowledge acquisition or retention. The merits of test-enhanced learning have also shown that simply integrating more progress testing may be of benefit for our mid-stream learners who are still developing knowledge and skills.As such, strategies such as this innovation may allow educators to both more efficiently mentor junior residents through exam-taking skills (i.e. expanding the pool of possible teachers by including senior residents), and also to help prepare senior residents to assume more educational/administrative roles. Not only this, but our program evaluation suggests that substantial adjunctive learning may occur in the senior residents themselves when they are forced to reflect on the answers of others. This may allow senior residents to better calibrate and combat misalignment between self-perceptions of competence and actual efficacy (e.g. the Dunning-Kruger effect) . Indeed,One of the main limitations of our present study is that the evaluation of our innovation is restricted to a single-centre and a single specialty. We used a constructivist epistemology; however, we are not purporting to attempt to generalize our findings. That said, we feel that the attitudes towards senior-resident development via marking peer resident exams is akin to other literature around near peer mentoring, and hence, we feel that when considering other literature our findings are in alignment with previous work. As such, we feel that there is likely some level of transferability that can be inferred from our findings. Other limitations would be our sample size and convenience sampling. It is possible that we did not reach full data saturation; we did, however, feel our data reached a level of sufficiency .Annually in Canada, the RCPSC EM residents have a short answer question based in-training examination for our specialty. Instead of a centralized marking centre, each residency program is charged with marking their own examinations and then reporting the data. That said, it is often a difficult task to find enough core faculty members who are able to donate their time. Our work suggests there may be opportunity to allow senior residents to mark peer residents' exams during this process, allowing us to expand our evaluation of a peer-based test-marking experience.Including residents in the marking of local progress tests can be educationally beneficial for all parties involved. Peer-marking has been deemed to be acceptable or desired by the residents who receive the peer-feedback and therefore should not pose a barrier to implementation. It may be a useful adjunctive educational experience for centers lacking the core faculty to sustain a regular, in-house progress testing system.


The goal of this study was to describe the construction process and contentvalidity evidence of an early and preventive intervention program forstimulating executive functions (EF) in Elementary School children withinthe school environment.The process has followed the recommended steps for creatingneuropsychological instruments: internal phase of program organization, withliterature search and analyses of available materials in the classroom;program construction; analysis by expert judges; data integration andprogram finalization. To determine the level of agreement among the judges,a Content Validity Index (CVI) was calculated.Content validity was evidenced by the agreement among the experts withregards to the program, both in general and for each activity. All stepstaken were deemed necessary because they contributed to the identificationof positive aspects and possible flaws in the process.The steps also helped to adapt stimuli and improve program tasks andactivities. Methodological procedures implemented in this study can beadopted by other researchers to create or adapt neuropsychologicalstimulation and rehabilitation programs. Furthermore, the methodologicalapproach allows the reader to understand, in detail, the technical andscientific rigor adopted in devising this program. These interventions constitute a tertiary level of prevention.Neuropsychology, a segment of neuroscience, is applicable in different professionalsettings, such as clinical and educational. Many studies and systematic reviewsseeking to structure a body of knowledge in the field of neuropsychologicalrehabilitation and provide a higher level of evidence have been conducted.9 These interventions represent theincorporation of neuropsychology into the educational context.Yet, despite the primary interest in rehabilitating cognitive deficits, another typeof intervention aimed at preventing or improving levels of health and well-being hasbeen gaining prominence among researchers, clinicians and educators. Suchinterventions, through preventive and promotional actions, are destined tostrengthen and improve cognitive and emotional processes among children with typicaldevelopment.12 These abilities include three main components: inhibitorycontrol, cognitive flexibility, and working memory.11 A large number of studies have associated theseabilities with better school performance and with social and emotional competenciesin children.17 The same studies suggest that executive deficitscan increase the risk of developing learning and behavioral issues, as well aspsychopathologies.19 The relevance of executivefunctions justifies the emphasis that researchers have been giving to interventionsthat improve these abilities during childhood, at home and at school.Executive functions (EF) are among the cognitive functions that have been targeted byearly and preventive intervention programs. They are a group of abilities thatmanage and regulate cognitive, emotional, and behavioral functions.20 Some programs have already beentested, such as the computerized program Cogmed,22 theextracurricular school program Sarilhos do Amarelo,23 and the curricula Tools ofMind.26 Despite the growing international emphasis on thistype of intervention, national studies are still incipient. In Brazil, theIntervention Program for Self-Regulation and Executive Functions (Programade Intervenção em Autorregulação eFunções Executivas – PIAFEx8) is noteworthy. It was developed, and has provenefficacy, to stimulate children at pre-school age27 and at first grade28 within the school setting.Current early and preventive interventions use numerous methods, includingcomputerized cognitive training, non-computerized games, physical activities,mindfulness, school curricula, and extracurricular programs.32 Fonseca et al.,33 for example, proposed a flowchartwith each of the stages of the delicate adaptation process of neuropsychologicalinstruments with verbal stimuli for use in Brazil. According to the authors, thereare four essential stages:Although studies provide information about the effectiveness of such programs, scantstudies have systematically described the procedures used to develop interventionprograms. Moreover, few studies report evidence on content validity. One possiblesource is to consult and follow the procedures and the theoretical, technical, andscientific rigor used in the development of standardized neuropsychologicalevaluation instruments.[1] translation;[2] analysis by non-expert judges;[3] analysis by expert judges; and[4] pilot study. Moreover, the authors of the instrumentmust analyze suggestions and adapt it based on suggestions given at eachstage.36 then an intervention program mustalso be clear about the demands of each activity. Some strategies to verify evidenceof content validity are the use of a consistent theoretical model to substantiatethe development of activities, as well as subsequent analysis performed by expertjudges.In this scenario of instrument development, it is essential to be cautious whendealing with content validity. Even though this investigation is closely tied toevaluation, it must also be applied to the development of intervention instruments.In this sense, if, in the context of evaluation, an appropriate battery must includeitems that explore a wide and representative range of the domain to beevaluated,Programa de EstimulaçãoNeuropsicológica da Cognição em Escolares: ênfasenas Funções Executivas).37 This program has an early and preventivecharacteristic, and is based on assumptions from neuropsychology and educationalpractices. The PENcE was planned to complement school curricula and targetsschool-aged children in 3rd and 4th grades of ElementarySchool. Through many cognitive and playful activities, as well as the teaching ofstrategy in a systematic and explicit way, the program stimulates and leverages EFsand correlated processes for subsequent use in other contexts.Based on the previously mentioned stages for devising and adapting neuropsychologicalinstruments, the objective of this study was to present the process of constructionand evidence of content validity of the Program of Neuropsychological Stimulation ofCognition in Students: emphasis on Executive Functions, or PENcE  with theobjective of analyzing the possibility of including the program in theschool curriculum, as well as obtaining more information about theschool environment. Before creating the program, meetings were held withthe team to think about the program's construction and to decide on nextsteps.38, analysis of otherprograms8, analysis of games known to thepublic , and neuropsychologyparadigms . Other tasks were created specifically forthe intervention. This stage also had a team to operationalize tasks comprising aneducational psychologist and two psychology students. At the end, the authorsconducted a new brainstorming session and activities were subsequently organized andsystematized so that each would have instructions and a description.Stage 2. Program construction – The group established the frequency and length ofmeetings, as well as by whom the program would be mediated. General principles andguidelines to govern the program were created, and activities developed, adapted,and selected. The authors used the bibliographic review performed in Stage1Stage 3. Analysis by expert judges – This procedure was carried outin three sub-stages:[1] Evaluation of the program as a whole: within andbetween sections, and organization of modules;[2] Evaluation of each activity of the program;[3] Evaluation of which function(s) orcognitive/neuropsychological component(s) is (are) stimulated in eachactivity. Also, using open-ended questions, judges were able to offersuggestions about changes or regarding new stimuli. For each module ofthe program, 3 expert judges gave their opinions and suggestedadjustments. Judges answered an evaluation protocol  was calculated todetermine the level of agreement between judges. According to Alexandre andColuci40, CVI allowsmeasuring of the percentage of experts who agree with respect to a certain item,when Likert scales are involved. The index is calculated by adding the items rated 3or 4 by the experts, and dividing the sum by the total number of responses in theitem's evaluation. For this to be representative in groups of five or less subjects,everyone must agree. The following formula was used:Results will be presented for each stage of the PENcE.Results of Stage 1: Internal stage of program organization. A systematicreview38 was performedthrough bibliographical review and analysis of interventions already present inliterature. In this review, 19 studies showed that EFs can be stimulated anddeveloped in children with typical development. Furthermore, there was apredominance of studies using computerized programs, which involved, for the mostpart, stimulating working memory. Other studies used tasks involving pencil andpaper, and some programs were incorporated into the school curriculum with theobjective of improving self-regulation. Depending on the program format, effects andgains vary. Methods, modes, and techniques used in these studies were an inspirationto create the PENcE. Curricular standards from the Department of Education,curricular matrix analyses, and observations of the school environment during 2months, allowed a better understanding of this setting and of the contents andmaterials used in Elementary School.Results of Stage 2: Program Construction. The program was organized in 4modules, considering the main executive abilities: Module 1 – Organization andPlanning; Module 2 – Inhibitory Control; Module 3 – Working Memory; and Module 4 –Cognitive Flexibility. Each module had the three following stages:Strategy acquisition: Psychoeducation and Modeling –students are taught about what, where, how, and why use the strategies ineach component of the EFs.Learning and Strategy Consolidation: Students are stimulatedto actively practice the strategies that were taught through playful andcognitive activities, as well as school-related tasks. For each module ofthe PENcE, this stage initially included from 6 to 8 activities, alldeveloped based on information acquired in Stage 1.Reflection and Transfer to School and Everyday Activities:Students reflect about how learning and strategies can be applied to variousaspects of life and to school activities.The program was outlined to be developed within the school environment, 3 times aweek in 50-minute sessions, for 5 months. These sessions were mediated by theteacher, and assisted by a neuropsychologist. A story was created based on the movie"A Bug's Life." In this plot, each module was represented by an "Ant" that, facingits challenges, was helped by the "Active Mind Ants League." The "Ants," along withthe "Active Mind Ants League," present the strategies to the students, encouragingthem to learn and engage in the activities. For each module, activities wereorganized and included descriptions of tasks, instructions, and materials used. Atthe end of each module, a section called "Reviewing the Modules" was added, so thatthe abilities stimulated in the previous modules could be reviewed and integrated.At the end of this process, the first version of the program was reviewed by theauthors and submitted to the evaluation by expert judges.Results of Stage 3: Analysis by Judges. Tables numbered from 2 to 9 showthe evaluation of judges after the first version of the program. The tables wereorganized by modules, with their respective activities.The overall evaluation of the program , as wellSome activities have more than one level of complexity. For these activities, judgesalso evaluated whether the levels were equivalent, or level 2 was harder than level1, or vice-versa. For all activities involving more than one level, the judgesagreed that the level initially developed as the most complex one was, in fact,harder than the previous level. The judges suggested a few changes for some of theactivities with the intent of making them clearer and easier to understand. Most ofthose changes were incorporated into the program. Some judges asked for more detailsor to add more information in the instructions, others proposed substituting thewording of some terms. After these considerations, the evaluation of judges wasintegrated into the latest version of the PENcE.Results of Stage 4: Integration of Judges' Analysis and ProgramFinalization. After the judges' analysis, adjustment and changes wereincorporated into each module:Module 1 - Activity of Fitting Numbers: one of the judgesunderstood that the activity stimulated, predominantly, inhibitory control. However,this judge also considered that the activity demanded planning and organization. Tomake the activity clearer, the following procedures were carried out: addition ofinstructions in writing, and directions for teacher to count the number of mistakesand pages that each student needed to complete the task. This helped check whichstudents were able to plan the activity well before executing it.st day: planning what isgoing to be cooked; 2nd day: execution of activity and evaluation.Activity 5: Cooking – one of the judges understood that this activity was notsuitable for the classroom setting due to time constraints. Therefore, the activitywas broken down to be performed over two days. 1Module 2 - Activity 1: Dancing – some examples were added,upon request by one of the judges.Activity 6: Not everything is what it seems – because of the understanding that theactivity was not coherent and its description not clear, it was excluded from theprogram.Module 3 - Activity 2: Numbering the sequences – rather thannumbering a sequence using a material similar to a mat, it was decided that thestimuli would be projected onto a screen.Activity 5: Differences Game – besides finding the differences between the images,students had to, at the same time, count from 50 to 20. Only after this countdowncould they write down on a piece of paper the differences that they hadobserved.Module 4 - Although the CVI was appropriate for all criteria,in activity 4, "Matching Cards," one rule was added: students had to combine thecards with images of animals. In addition to the colors of the animals and thehabitat where they live, the activity included the number of syllables of each word,with the objective of making the task more complex and challenging.40, the whole program and its activities are described indetail so they can be replicated. After reading this material, other teachers canconduct the activities.When there were disagreements with respect to changes and adaptations, they weresubmitted to one of the authors, who established a consensus. There was then areview of the whole program, finalizing it. The summarized final version is given inthe appendix of this study. In the PENcE book33.This study presented the development process and content validity evidence of thePENcE for school-aged children. The PENcE is a program for stimulating EFs that,through various cognitive and school-related activities, as well as through theteaching of strategies, aims to systematically leverage these abilities in theschool setting. Creating a rehabilitation or a neuropsychological stimulationprogram requires rigorous and well defined stages and procedures. However, there isstill no systematization of the process in the available literature. Existingprograms do not specify how they were devised or constructed and do not reportcontent validity evidence. In this scenario, the present study sought to follow thesame methodological and technical rigor used to adapt or create neuropsychologicalevaluation instruments34 proposed a model withthe steps necessary to develop a psychological instrument, which must follow asequence. These steps comprise three main axes: theoretical procedures ;empirical procedures ;and analytical procedures . In this study, the first axis was attained,providing a direction for the ensuing stages. This was the first stage of thevalidity work and of the search for effectiveness of the program. All stagescontributed to identify positive aspects, possible flaws, verify understandabilityand level of complexity, adapt stimuli, and improve tasks of the program.Pasquali11, it wasdecided to include the central abilities , which form the basis of the emergence of more complex orsuperior functions, and one complex executive function (planning and organization)because of its relevance to school performance. These definitions, along with adeeper understanding of other available studies, allowed the following steps to bedeveloped underpinned by a solid theoretical basis.The first stage included a long process of research, reflections on the theme, anddiscussions among the team. This required the exploration of national andinternational bibliographies. The school environment and curricular matrices ofElementary School were closely monitored. This was a fundamental stage because itallowed the definition of aspects and theoretical models that would become the basisof the PENcE, as well as the constructs that would be stimulated. Based on the modelcreated by Diamond41 approach,and there is evidence that such programs tend to present better results in terms ofacquiring abilities and transferring gains.43 It was alsodecided that the program would be based on the teaching of systematic and explicitstrategies of the EFs. Meltzer39holds that classroom interventions must include direct instructions of metacognitivestrategies, and that they must be structured and systematic. Thus, with the goal ofallowing more reflection and knowledge transfer to other contexts, numerous playful,cognitive, and school-related activities were proposed.In the construction stage, format, principles, and program guidelines were planned.Tasks that would compose each module, with their respective instructions anddescriptions, were also planned. As is the case for the development of anyrehabilitation program, some decisions had to be made, considering the objective ofthe program. Initially, it was decided that the program would be incorporated intothe school curriculum. Programs of curricular adaptations adopt a wider and morenaturalistic44. Also, agreement among expertsprovides confirmation of content validity 36. In the absenceof other studies investigating evidence of content validity in cognitive stimulationinterventions among healthy children, the index found here is in consistent withthose of other studies determining this type of validity for instruments forneuropsychological evaluation43.The existence of evidence of validity with respect to content makes it easier forits applicability to be tested and checked in typical samples and different clinicalgroups. These procedures allowed the derivation of content validity evidence for thePENcE. After checking all suggestions put forward by the judges, in the fourth stageof the study, changes and adjustments were made, resulting in the final version ofthe intervention program.In the third stage, the judges' analysis revealed a high level of agreement amongexperts, both for the program as a whole and each activity. In this stage, therelevance of the tasks within the program was verified, allowing tasks to beadjusted and improved. The Expert Committee has the role of consolidating thepresented version and helping produce the pre-final version of theinstrumentGiven the lack of research about the development process of rehabilitation orneuropsychological stimulation programs, this study displays an innovative approach.It can help other researchers and clinicians to develop strategies and interventionprograms, since it provides a systematization and outline of the process. This studyalso provides readers with a detailed understanding about each stage and offersinsight on all the care and rigor applied in producing the PENcE. The PENcE can be auseful tool to education and healthcare professionals, since it can provideorientation for their practices and promote benefits for school-aged children in thepromotion of EFs. Future studies should verify the effectiveness of the program forchildren in Elementary School I, and adapt the program for use in teenagers andclinical groups with potential executive dysfunctions, such as attention deficithyperactivity disorder (ADHD), and learning related disorders.


In randomized controlled trials, short message service (SMS) programs have improved adherence to HIV antiretroviral therapy (ART). In response, the World Health Organization recommended use of SMS programs to support ART. However, there is limited data on real-world implementations of SMS programs.N = 2255). Our main outcome measure was prescription coverage, defined as the presence of a valid ART prescription for each day observed. We fit generalized linear mixed models adjusted for pre-program prescription coverage, demographics, and ART duration, dosing, and regimen.We conducted a retrospective cohort study of an SMS program to improve ART adherence in a government-run HIV clinic in rural South Africa. We analyzed data from all adult patients who 1) enrolled at the clinic before the observation period (July 2013 through June 2014), 2) had ≥1 ART prescriptions in the observation period, and 3) had data on phone number availability (P < 0.001) compared with non-exposure, although the absolute increase in prescription coverage was small (4.7 days of ART prescription coverage per average patient per year). Among a subset of patients (n = 725) whose pre-program prescription coverage was <100%, the corresponding mean expected absolute increase in prescription coverage was 8.2 days per year.Exposure to the SMS program was independently associated with greater prescription coverage (AOR = 1.23, 95% CI 1.13–1.34, Our primary finding was that an SMS reminder program implemented in routine clinical care was associated with a small increase in prescription coverage of uncertain clinical significance. Successful antiretroviral therapy (ART) for HIV depends on sustained, high adherence . HoweverSeveral randomized controlled trials (RCTs) have investigated the adherence benefits of weekly SMS programs. They have shown a mix of positive and null findings in terms of effect on adherence and viral suppression , 13, 15.The World Health Organization has recommended mobile phone text messages as an adherence support intervention for people on ART . HoweverIn this study, we evaluated a clinical program of weekly SMS reminders at a government-run HIV clinic near Durban, South Africa. Using a retrospective cohort design, we analyzed routine data collected at the clinic to determine ART prescription coverage, comparing those who were or were not sent SMS reminders.This study was conducted at the Ethembeni HIV Clinic, located in a rural area approximately 40 km from Durban, South Africa. Ethembeni is a government-run, hospital-based HIV clinic with an on-site pharmacy, which provides free ART. The adult clinic population is approximately 67% female, has a median age of 37 years, and nearly all speak isiZulu as their first language.Patients at Ethembeni receive prescriptions of varying lengths , depending on clinical needs. All prescribed medications are picked up at the on-site pharmacy. Patients with a prescription for three or fewer months generally receive all prescribed pills during the prescribing visit. Patients with a six-month prescription generally receive three months of pills at a time.The SMS adherence program was developed by clinic staff in collaboration with Sawubona Health , a US-based non-governmental organization. Clinic staff and Sawubona Health volunteers manually entered mobile phone numbers from paper charts of adult patients into the SMS program database over two discrete periods (June-July 2012 and the month of June 2013). Notably, the clinic routinely asked each patient for a contact phone number at clinic enrollment, which was an average of 3.3 (SD 1.5) years prior to the SMS program launch date. Patients were not re-approached by staff to add a phone number for capture into the SMS program database due to program resource limitations.Hi (First name), this is your clinic. Remember to take your pills on schedule in order to (One of: be strongorlive a long lifeorfeel well). Thank you.The SMS program started on September 9, 2013 and is ongoing. All SMS are sent free of charge to patients; the program costs approximately 1 USD per patient per year to run. The program is opt-out; the first SMS explaining the program was sent to each adult patient with a recorded mobile phone number. This introductory SMS also informs patients how to opt-out, which can be done at any time via a reply SMS. SMS are sent to all subscribed patients each week by rotating through the following message (in isiZulu):Starting on January 13, 2014, the program server began receiving delivery status reports of confirmed receipt of the SMS on the patient’s registered phone number. Further details on the SMS program can be found in a separate interview-based evaluation study [Data for this analysis was extracted from routinely collected pharmacy and medical record data, as well as the SMS program data. We utilized data from November 4, 2012 through June 26, 2014, the time period for which prescription data were available. The dates, duration, and medications of each prescription were extracted for all patients at the clinic during this period. The electronic medical record extraction included age, gender, ART start date, and date of transfer out (if applicable). The SMS program data included the availability of a valid mobile phone number, date the phone number was available, date of opt-out (if applicable), and delivery status reports . The datasets were de-identified except for dates and collated by linking each patient to a unique study identifier (ID). An indicator variable was created which was set to one for patients with any missing values for any non-prescription covariates, and set to zero otherwise. Missing values for non-prescription covariates were replaced with the mean of all corresponding available values in the cohort.18 years or older;At least one ART prescription during the time for which both pharmacy and mobile phone data were available – the “observation period” ;Enrolled at the clinic on or before July 1, 2013;Phone number availability assessed prior to the observation period.Figure “Exposed”: Patients who had a valid mobile phone number in their chart during the SMS program data capture process.“Unexposed”: Patients who had a chart available for review but for whom a valid mobile phone number was not identified in the chart. This situation could be because they did not have, or did not choose to provide, a mobile phone number at the time of enrollment; rare instances of erroneous recording of the phone number were also noted.“Unknown”: All remaining patients in the cohort, for whom the presence or absence of a mobile phone number was not determined due to missing charts. Common reasons for missing charts were use of the chart elsewhere in clinic or loss to follow-up.Our primary exposure of interest was inclusion in the SMS program. Clinic patients were assigned to one of the following analysis groups based on whether data from their chart had been captured into the SMS program database and on potential exposure to the SMS program  via an intention-to-treat approach. That is, group assignment disregarded whether the patients who were sent messages opted-out or failed to receive the messages due to phone number change or technical difficulties.Each patient was observed for a specific subset of the observation period, as determined by the following censoring criteria. The first day of the observation period was July 1, 2013. Within this period, observations began at the earliest of: 1) the day after the end of the patient’s first recorded prescription during the pharmacy data availability period (November 2012 through June 2014), or 2) the day of the patient’s second recorded prescription (to avoid including patients who had not clearly established care in the clinic). We right-censored patients on the earlier of their transfer-out date or June 26, 2014 . For baseline characteristics, continuous values were compared with ANOVA; categorical values were compared with the Chi-squared test.Our primary outcome was ART prescription coverage. A day was defined as covered if the patient had valid prescriptions that, if filled and taken as prescribed, allowed the patient to possess sufficient medication for the intended three-drug ART regimen on that day; otherwise, the day was considered not covered. Each day during the study period was a unit of observation, and thus each patient was weighted by their length of observation. The rates of patients achieving > 95% coverage before versus during the SMS program were compared using the one-sample McNemar’s test.The daily unit of observation was chosen because: 1) several covariates and the outcome varied through time, and thus averaging these values across extended time periods would obscure the temporal relationship between covariates and the outcome; and 2) averages across longer time periods were heavily skewed to the right and thus could not be adequately modeled by standard distributions. If a patient had days of prescription coverage remaining at the time of a new prescription, the remaining days of coverage were carried over to the next prescription, up to a maximum of 30 days per clinic policy, if there was no change in the antiretrovirals. For example, if a patient had 15 days of prescription coverage remaining, and came early to an appointment at which they received a 30-day prescription for the same antiretrovirals, they would have 45 days of prescription coverage available. Any records of prescriptions that simply re-recorded the remaining balance of a 6-month prescription  were treated as duplicates.a priori interest was in the adjusted results as described below. All models were fit using Stata version 13.1 .To account for repeated measures within an individual, we used mixed effects multivariable logistic regression models. No univariable results were determined; our P = 0.25) [a priori, to select the covariates for the executed model.In the regression model, we determined if SMS program exposure was associated with ART prescription coverage. Fixed effects were study group , time period , and factors available in the database that have been shown to influence adherence. Specifically, these factors were age , gender  = 0.25) , with thTo estimate the absolute risk reduction of SMS program, the model calculated the time-averaged prescription coverage for the Exposed patients in the during-program period. We then applied the odds ratio of coverage given non-exposure during the program (comparing the Unexposed and Exposed groups). This calculation yielded the hypothetical coverage that an average patient would have had if they had not been exposed to the SMS program. The difference between the actual coverage and the hypothetical non-exposure coverage equaled the absolute risk reduction for a patient with average coverage.The <100% baseline coverage subset: patients with pre-program prescription coverage < 100%, suggesting at least one missed prescription during that period.Recent ART initiators subset: patients who initiated ART within 2 years of the start of the SMS program, because such patients have an accelerated rate of attrition compared to patients with longer treatment histories .Given the WHO emphasis on providing differentiated care  as well N = 458), no ART prescription in the observation period (N = 33), not enrolled at Ethembeni prior to the observation period (N = 160), delayed capture of phone number into the SMS program (N = 1), and no observation time  (N = 13). The study cohort therefore consisted of 2,255 patients: 1,771 (78.5%) in the Exposed group, 316 (14.0%) in the Unexposed group, and 168 (7.5%) in the Unknown group. The introductory SMS was sent to all 1,771 patients in the Exposed group. Among these patients, 1,287 (72.7%) had at least one confirmed successful SMS delivery  and among these, 1,044 (81.1%) remained subscribed to the program for their entire time of observation.A total of 2,920 patients received ART during the pharmacy data availability period of November 4, 2012 to June 26, 2014. Of these, 665 did not meet our inclusion criteria for the following reasons: age <18 years  for the Exposed group, compared with 6% (P = 0.118) and 13% (P = 0.038) for the Unexposed and Unknown groups, respectively.The distribution of prescription coverage by patient before and during the program period is shown in Fig. After adjusting for age, gender, prescription lengths, regimen type, fixed dose combination, and pre-program prescription coverage, patients in the Exposed group had greater odds of daily prescription coverage during the SMS program  compared to those in the Unexposed group. For an Exposed patient at an average level of adherence (94.1% during the program), the absolute risk reduction of exposure to the SMS program was 4.7 days of ART prescription coverage missed per year compared to the Unexposed group.P = 0.040), but after adjustment for this factor, had lower odds of coverage during the program . As shown in Table Full cohort patients in the Unknown group (compared to those in the Unexposed group) had higher odds of pre-program coverage  and a higher proportion of missing data . The subset of patients who initiated ART within two years of the SMS program start date consisted of 333, 41, and 42 patients for the Exposed, Unexposed, and Unknown groups, respectively. There were no statistically significant differences in the baseline covariates among these groups (data not shown).P = 0.101). As shown in Table For the <100% baseline coverage subset, the SMS program was associated with an increase in prescription coverage when comparing the Exposed and Unexposed subgroups . The corresponding absolute risk reduction of exposure to the SMS program for an Exposed patient with average coverage in this subset was 8.2 days of ART prescription coverage missed per year compared to the Unexposed group. For the recent ART initiator subset, no statistically significant differences were seen in prescription coverage between the Exposed and Unexposed subgroups . The effect for an average patient with <100% baseline prescription coverage was 8.2 days per year, but there was no significant effect for patients who had initiated ART within two years of the SMS program, possibly due to the relatively small size of the subset.The clinical significance of the measured benefit of the SMS program among the full cohort is unclear. Although CD4 counts and viral loads were not consistently available for comparison in this cohort, prescription coverage has been shown to predict virologic suppression . PrescriMany of the recorded lapses in prescription coverage may represent treatment interruptions, highlighting the potential clinical significance of the measured program effect. Future study of the downstream effects of real-world SMS programs on viral suppression and other clinical outcomes will help clarify the broader public health impact of such programs.The exact mechanism of the improvement in prescription coverage is not known, but may reflect perceptions of social support and encouragement to remain in care. SMS programs have generally been well-accepted, including this one as described in a separate report . Other sThe Unknown group (approximately 7% of the cohort) had a disproportionate share of patients with adherence challenges. It included patients whose charts were outside the file room during the period of data capture, which generally indicated the presence of clinical challenges or being lost to follow-up. Given these aspects of the Unknown group, we do not believe it is an unbiased comparator to the Exposed group. These issues also emphasize the importance of capturing phone numbers at enrollment for all individuals in care to help to ensure all vulnerable patients are offered the program.While this study was not a randomized controlled trial, we believe the Unexposed group is a reasonable comparator to the Exposed group and adds value in assessing the impact of the SMS program on ART adherence. One potential source of confounding is differential phone ownership in the Unexposed group, which was not assessed at the time of the SMS program implementation. Rather, patients were asked for their contact phone numbers at clinic enrollment, which was on average 3.3 years prior to SMS program launch. However, we believe this possibility was unlikely to confound our results because, during this time, the prevalence of mobile phone ownership among South African adults was increasing and reached 90% in 2013 . Thus, wThis analysis has other limitations. First, the unit of observation of each day was powered to detect small effects. We addressed the repeated measures aspect of this approach by using a mixed model. Second, the prescription lengths were long (up to 6 months) relative to the observation time (up to about 12 months), thus limiting our ability to identify lapses in prescription coverage. Third, any errors in clinical data collection may have incorrectly identified patients as non-covered, including patients who transferred out but did not receive the appropriate documentation from the clinic.We found that exposure to the clinical SMS program had a small measurable clinical impact among the full cohort that was possibly stronger among patients with imperfect pre-program prescription coverage. Generalizability to similar government ART clinics in the region is likely high.We believe three primary avenues of future work should be pursued. First, additional studies should further explore the mechanisms by which SMS programs affect adherence, particularly their effects on social support , 29. SecIn light of the high overall adherence, the cost-effectiveness of such implementations could be enhanced by improving their designs and targeting enrollment to individuals at risk for adherence challenges.


AbstractColeoptera: Chrysomelinae) is explored using geometric morphometrics based on 36 landmarks. Principal component analysis and Canonical variate analysis indicate that changes of apical area, anal area, and middle area are three useful phylogenetic features at a subtribe level of leaf beetles. Variances of the apical area are the most obvious, which strongly influence the entire venation variance. Partial least squares analysis indicates that the proximal and distal parts of hind wings are weakly associated. Modularity tests confirm that the proximal and distal compartments of hind wings are separate modules. It is deduced that for leaf beetles, or even other beetles, the hind wing possibly exhibits significant functional divergences that occurred during the evolution of transverse folding that resulted in the proximal and distal compartments of hind wings evolving into separate functional modules.The success of beetles is mainly attributed to the possibility to hide the hindwings under the sclerotised elytra. The acquisition of the transverse folding function of the hind wing is an important event in the evolutionary history of beetles. In this study, the morphological and functional variances in the hind wings of 94 leaf beetle species ( Coleoptera (known as beetles) are the largest insect order, containing 380,000 named living species classified into more than 160 families (Archostemata), but mainly serve to form a protective cover for the hind part of the body (hind wings and abdomen)  . As a pa evolved . As the  evolved .Coleoptera, Dermaptera, and some species of Blattodea)  . In beetattodea) .Chrysomelinae beetles. Chrysomelinae beetles could be divided into two tribes, one is Timarchini which are not able to fly, hind wings are reduced or disappeared; the other is Chrysomelini which have functional hind wings  was collected to observe and analysis the hind wing variance. The typical hind wing of leaf beetles presented in the Figure cv in the middle area of hind wings  to obtain landmark data. There were 94 species, 81 genera included in this study; each genus included one to three species. There was two species with two specimens from different geographical locations. There were 96 specimens in 94 species. The leaf beetle specimens were deposited in the Institute of Zoology, Chinese Academy of Sciences. The specimens were examined and dissected to obtain hind wings using a LEICA MZ 12.5 dissecting microscope. Pictures of the wings were obtained with a D500s Nikon camera connected to a stereoscope (Zeiss Stereo Discovery V12). The collection and subtribe information of 96 leaf beetle specimens is presented in Suppl. material This study was based on hind wing images  and tpsDig  software programs were used for digitising landmarks. Figure MorphoJ (Version 1.06d) was used for landmark data analyses. MorphoJ is a software package enabling geometric morphometric analysis for two- and three-dimensional landmark data and designed for the analysis of actual biological data . Prior tPCA): PCA is one of the most widely used methods for exploratory multivariate analysis  are visualised directly as patterns of simultaneous displacements of landmarks in relation to one another.Principal component analysis : CVA is a method used to find the shape features that best distinguish among multiple groups of specimens  could be definitely divided into eleven subtribes; the left seven specimens  had no clear subtribe information  analysis: MorphoJ offers an implementation of PLS analysis between blocks of landmarks within the same configuration. This analysis identifies the features of shape variation that most strongly co-vary between the blocks and indicates their relative contribution to the total covariation between blocks  are lower (< 0.5), the covariance of the two blocks is weak. The RV coefficient provided by this procedure is the same as that in the output from tests of modularity hypotheses  was evaluated. MorphoJ can compare the strength of covariation between two partitions and either all or a large number of the possible alternative partitions with the same numbers of landmarks as in the hypothesised modules  and the other with 17 landmarks . In our case, the total number of different partitions into subsets was approximately 9×109 ; such regressions often fit the data well and the allometric shape changes tend to affect the entire structures . The ProPCA was carried out based on 96 specimens. The accumulative contribution ratio of the first three components was 68.04%, potentially indicating that the first three components represented the main shape variation of wing venation. Figure cv or whether existed. In addition, landmark 29 exhibited a long variance distance, indicating the length of AA3+4. PC3 (with a variance contribution ratio of 10.56%) primarily affected the size of the anal area by the movement of landmarks 20, 22, 25, 29–34 and 36. To different degrees, both PC2 and PC3 affected the relative size of the apical area by moving proximal part landmarks 1–6, 24, 26, and 35 and distal part landmarks 17–19. Thus, when PC1, 2, and 3 are integrated, it was concluded that there were three features of the wing variance of leaf beetles: the relative size of the apical area which could be considered the main feature (variance contribution ratio of 45.01%) to influence of the overall variance of the hind wing, the changes of cross vein cv in the middle area, and relative size of the anal area size.Based on the Procrustes fit data, PCA wing variance results, CVA was used to test whether the three features were useful on the distinguishing of the subtribe level. There were eleven subtribes (89 specimens) of Chrysomelinae involved in this study, except subtribe and have one specimen for each ; the left have 4–26 specimens for each subtribes. Iin CVA, there were nine subtribes with enough samples to do analysis. As the Figure CVA result showed, in the nine subtribes, and couldn’t be divided based on wing shape data; these two subtribes were overlapped. The other seven subtribes could be divided clearly. Especially for subtribes distributing in the edge of coordinates : , Chrysomelina, Doryphorina, and , the four subtribes showed big large variances on wing shape. The wing difference between each other could be described as qualitative features. The typical hind wing images of the four subtribes were presented as the Figure cv in the middle area; while Chrysomelina beetles have longer apical area and no cross vein cv. Doryphorina beetles have a wide anal area and no cross vein cv; while have a narrow anal area and cross vein cv in the middle area. Doryphorina and mainly distributed along the axis of CV1 which had a big variance contribution ratio 40.19%; and Chrysomelina speciemns mainly distributed along the axis of CV2 which had a second big variance contribution ratio 27.83%. It was concluded that the relative size of the apical area and anal area, and cross vein cv in the middles area, could play an important role in the division of these 4 subtribes. For subtribes Entomoscelina, Gonioctenina, and , the wing variance also focused on the apical area, anal area and cross vein cv. These three subtribes distributed in the central region of coordinates ; the difference between each other is quantitative and is hard to describe as qualitative features.Based on the PCA and CVA showed that the apical area of wings has a large variance. These results prompted us to test whether the apical area had a relatively independent shape change in hind wing shape variances. The apical and central area and radical cells of hind wings . When the RV coefficient values are higher, the covariance of two blocks is stronger. Therefore, the null hypothesis was rejected. Although high correlation was noted between two blocks, the overall strength of association between blocks was weak. From the analysis results, the following was concluded: in beetle hind wings, the distal part, or more precisely the areas relevant to transverse folding  exhibits both a certain degree of independence and a high correlation between other parts of wings in the total hind wing shape variances.PLS1 presented 87.22% of the total covariance, indicating that PLS1 represented the main covariance of two blocks. Figure PLS analysis results, a modularity analysis was performed to evaluate whether the proximal and the distal parts of beetle hind wings are separate modules. In the same manner, the landmarks 7–22 and 25 involving transverse folding were extracted as the distal part and the remaining landmarks 1–6, 23–24 and 26–36 as the proximal part, which was our hypothesised partition  but strong within-module integration . Based on the PLS analysis results, a modularity test to verify the hypothesis regarding partitioning as noted in Figure RV coefficient of all evaluated partitions does fit with our hypothesised partition: the proximal and distal parts of hind wing are separate modules.In our study, g Figure . A modulg Figure . Modulesg Figure , 2011. Fg Figure , 2014; tDrosophila’s wings using morphometric approaches have addressed the question of whether anterior and posterior wing compartments are distinct modules reflecting phenotypic and genetic variation (Drosophila. However, Various studies of ariation . Based oariation , the subRA4, RP, RP2 and radial cell, see Figure RA and radial bar, see Figure What is the nature of the modularity interactions of hind wings? Generally, it can be developmental, functional, or genetic , 2011. HPageBreakwing is a common feature of beetles. However, more data are needed to confirm this hypothesis. Here, the apical part of the hind wings of leaf beetles has an important influence on hind wing shape variance by PCA (Figure The separate modules of proximal and distal parts of Chrysomelid hind wings have been tested. The main reason of the separate modules could be attribute to its special function: transverse folding in hind wings of beetles. Although there were only 96 chrysomelid beetles considered in our study, it could be believed that other beetles feature the same modularity of hind wings, given that the transverse folding of the hind A Figure . The shaPCA, CVA, PLS analyses and the modularity test into account, it was concluded that areas of the beetle hind wing relevant to transverse folding importantly influence hind wing shape variances and are relatively independent. In addition, the proximal and distal parts of a beetle’s hind wing are separate modules. The changes of apical area, anal area and middle area of hind wings were useful features to distinguishing subtribe level of leaf beetles. For beetles, hind wing folding is a morphological and functional compromise between fore wing evolution to elytra and the maintenance of hind wing flying function. This separate function modules could allow the hind wings to be folded at rest and to unfold when flying. In addition, the separate function could explain why beetles are the most prosperous animals in evolution. Our discovery could provide the theoretical basis and a new perspective for studies of the morphological evolution of wings and wing folding mechanisms.Taking the Experiments were planned by Jing Ren, Si-Qin Ge, and Run-Zhi Zhang. Species identification was completed by Si-Qin Ge. Experiments were conducted by Jing Ren. Analysis and interpretation of the results was performed by Jing Ren, Ming Bai, Run-Zhi Zhang, and Xing-Ke Yang. The paper was written by Jing Ren, Si-Qin Ge, and Run-Zhi Zhang.All authors declare there are no potential competing interests.


Growing evidence suggests that text messaging programs are effective in facilitating health behavior change. However, high dropout rates limit the potential effectiveness of these programs.This paper describes patterns of early dropout in the HealthyYou text (HYTxt) program, with a focus on the impact of baseline motivation quality on dropout, as characterized by Self-Determination Theory (SDT).This analysis included 193 users of HYTxt, a diet and physical activity text messaging intervention developed by the US National Cancer Institute. Descriptive statistics were computed, and logistic regression models were run to examine the association between baseline motivation type and early program dropout.Overall, 43.0% (83/193) of users dropped out of the program; of these, 65.1%  did so within the first 2 weeks. Users with higher autonomous motivation had significantly lower odds of dropping out within the first 2 weeks. A one unit increase in autonomous motivation was associated with lower odds  of early dropout, which persisted after adjusting for level of controlled motivation.Applying SDT-based strategies to enhance autonomous motivation might reduce early dropout rates, which can improve program exposure and effectiveness. The proliferation of mobile phone ownership in the United States and worldwide has provided a unique opportunity to reach broad audiences with health behavior messages and interventions -3. GrowiMotivation is one factor that has been associated with engagement in health behavior change programs and longer-term pursuits of health behavior change . Traditistop to the program.HYTxt is a free text message-based resource that provides behavioral interventions to promote healthy diet and physical activity. HYTxt was developed as a component of the US National Cancer Institute’s Smokefree.gov Initiative (SFGI). SFGI is a large, national, multi-platform program that includes websites, text message programs, smartphone apps, and social media platforms that engage with 3-6 million users annually. HYTxt is a 6-week intervention program that delivers 1-4 daily texts, including standardized behavioral intervention and social support messages that target specific health behavior goals each week, via unidirectional and bidirectional messages. At the time of enrollment, users answer a series of basic demographic questions and complete a brief motivational assessment. Users then select a behavior change goal related to healthy eating or physical activity. Throughout the program, users receive text messages that provide SDT-based intervention content, and assess their status in achieving health behavior goals. Users may discontinue the program at any time by texting the word As a part of ongoing program quality assurance and improvement efforts, the sample for this project consisted of 193 consecutive individuals who enrolled in the HYtxt healthy eating stream of the text messaging program between June 2014 and December 2014, and met our inclusion criteria. Given our interest in the impact of motivation type on early dropout, we restricted our analysis to individuals with complete data for the motivation questions (described below). We did not collect information on the gender of our participants, but the program was promoted on the Smokefree Women Website  and Smokvery true (4), a little true (3), a little untrue (2), and very untrue (1). The autonomous motivation subscale was comprised of the mean of the identified and integrated questions; the controlled motivation subscale was comprised of the mean of the introjected and external questions. Higher values signified higher levels of motivational quality.The dependent variable was early dropout, which was defined as actively dropping out within 2 weeks of program enrollment. Our primary independent variable was motivation quality. Users self-reported their age, educational level, and race/ethnicity during program enrollment. Users could enroll in the HYTxt program by texting a keyword on their phone or by completing the Web enrollment form. Only individuals who enrolled using the Web enrollment were asked demographic questions; therefore, demographic data were only available for that subset of users.All analyses were conducted in SAS 9.3 . Descriptive statistics were calculated and chi-square tests were conducted to examine associations between demographic characteristics and early dropout status. Logistic regression was used to test the association between early dropout and motivation type. Separate unadjusted logistic regression models for controlled motivation on dropout, and autonomous motivation on dropout, were computed. Subsequently, a model including both controlled motivation and autonomous motivation was run. For the subset of individuals with complete demographic data (n=126) we also examined the association between autonomous motivation and early dropout, controlling for demographic characteristics .Most users had at least some college education, were predominately white, and had an average age of 34.6 years . There wOf those who dropped out before completing the program , 65.1% (54/83) of users did so within the first 2 weeks.P=.008). This association persisted after adjusting for level of controlled motivation . Controlled motivation was not associated with early dropout . Given the association between autonomous motivation and early dropout, we further explored whether this association held true after controlling for demographic characteristics , and autonomous motivation remained significantly associated with early dropout .At the beginning of the program, most users endorsed both autonomous and controlled motivations for healthy eating; however, on average users reported higher levels of autonomous motivation  than controlled motivation . Users with higher autonomous motivation had significantly lower odds of dropping out of the program within the first 2 weeks. A one unit increase in autonomous motivation was associated with a significantly lower odds of early dropout  of users who dropped out did so within the first 2 weeks. Users with higher levels of autonomous motivation at baseline were less likely to drop out early, even after adjusting for controlled motivation. This result is consistent with previous research that has demonstrated that autonomous motivation is predictive of initiating and maintaining health behaviors ,18.Participants in this study were users of the HYTxt text messaging program, and were not recruited for a particular study. Although this sample represents a self-selected group, they are likely more reflective of individuals who would use the publicly available program in general than participants of a tightly controlled randomized trial would be. Not all users completed the motivation questions, resulting in a great deal of missing data for demographic correlates; therefore, our results may not be generalizable to all users of HYTxt. This analysis focused solely on motivation related to healthy eating, and other factors  might influence dropout rates. Furthermore, we focused on those who actively opted out of the program, and cannot comment on those who may have continued to receive messages but disengaged from the program.Dropout is a significant problem in behavior change text messaging interventions. This study provides insight into a modifiable factor that influences likelihood of dropping out: autonomous motivation. The extant literature on SDT-based interventions offers potential insights into techniques that could be leveraged to promote autonomous motivation early in text messaging behavior change programs to decrease early dropout. Candidate SDT techniques include: supporting users in aligning behavior change goals with broader life goals and values; providing opportunities for choice, including opportunities to disengage from the current program and either (1) return at a later time or (2) change to another health behavior that better fits the user’s current interests and life situation; and providing content that normalizes the trial-and-error nature of the health behavior change processes, while offering strategies for dealing with failures and setbacks -20. WhilText messaging interventions have the potential to reach large numbers of individuals, especially traditionally hard-to-reach populations. However, high early dropout rates inhibit the potential effectiveness of such programs. This study demonstrated that those who had higher levels of autonomous motivation were less likely to drop out in the first 2 weeks of the program, during which dropout rates were highest. Applying SDT-based strategies early in text messaging programs (to enhance autonomous motivation) might reduce early dropout rates, which in turn can improve program exposure and effectiveness.


Background: Medical education program evaluation allows for curricularimprovements to both Undergraduate (UME) and Graduate Medical Education (GME). UMEprograms are left with little more than match rates and self-report to evaluatesuccess of graduates in The Match.Objective: This manuscript shares a novel method of program evaluationthrough a systematic assessment of Match outcomes.Design: Surveys were developed and distributed to Program TrainingDirectors (PTDs) at our institution to classify residency programs into which our UMEgraduates matched using an ordinal response scale and open-ended responses.Outcomes-based measures for UME graduates were collected and analyzed. Therelationship between PTD survey data and UME graduates’ outcomes were explored.Open-ended response data were qualitatively analyzed using iterative cycles of codingand identifying themes.Results: The PTD survey response rate was 100%. 71% of our graduatesmatched to programs ranked as ‘elite’ (36%) or ‘top’ (35%) tier. The mean totalnumber of ‘Honors’ grades achieved by UME graduates was 2.6. Data showed thatgraduates entering elite and top GME programs did not consistently earn Honors intheir associated clerkships. A positive correlation was identified between USMLE Step1 score, number of honors, and residency program rankings for a majority of theprograms. Qualitative analysis identified research, faculty, and clinical exposure asnecessary characteristics of ‘elite’ programs:. Factors considered by PTDs in therating of programs included reputation, faculty, research, national presence andquality of graduates.Conclusions: This study describes a novel outcomes-based method ofevaluating the success of UME programs. Results provided useful feedback about thequality of our UME program and its ability to produce graduates who match inhighly-regarded GME programs. The findings from this study can benefit ClerkshipDirectors, Student Affairs and Curriculam Deans, and residency PTDs as they helpstudents determine their competitiveness forspecialties and specific residencyprograms. A cycle of frequent medical education program evaluation followed by quality improvementinitiatives is crucial to educate future physicians effectively in the constantlychanging world of healthcare. Identification of educational program deficiencies throughregular comprehensive evaluation of programs allows for curricular improvements to bemade in both Undergraduate Medical Education (UME) and Graduate Medical Education(GME).Aspects of program evaluation that are routinely examined in UME ,2 includWithin UME program evaluation, the definition of a successful Match cycle may bewide-ranging. The Liaison Committee for Medical Education (LCME) requires that UMEprograms report Match rates as an outcome indicator of the educational program. UMEprograms may also query their graduates about whether they matched to their first,second, or third choice of residency programs. This data is not officially released byElectronic Residency Application System® (ERAS®), therefore UME programs must rely onstudent self-report. In addition, UME programs may track the GME institutions to whichtheir graduates match. Success may be defined as a majority of graduates matching intoresidency programs that the UME institution perceives as being of high quality.U.S. News and Reports (USNWR) listingssince the 1990s, there has not been a similar ranking system for GME residencyprograms.The definition of high-quality residency programs is not well-articulated orstandardized on a national level. Little is published on the topic of residency programrankings or comparisons despite a clear demand from prospective trainees and otherstakeholders. Unlike the rankings of the best hospitals and best medical colleges thathave existed in USNWR,released rankings of residency training programs by specialty. Doximity is a free,HIPAA-compliant, online network of physicians that facilitates social and professionalconnections and provides educational resources. Doximity rankings are based on acombination of resident survey data, reputation of program survey data, and researchoutput data hat qualities or factors need to be present in order for a GME training program tobe considered elite’ but came up minimally if at all for the question about what factorsresponders were considering when rating the programs. These factors expand beyond themore traditional residency training and ACGME competencies such as patient care andmedical knowledge . This suIn addition to providing program evaluation data, the findings from this study can alsobenefit Clerkship Directors, Student Affairs Deans, and PTDs as they help studentsdetermine their competitiveness for both specialties and specific residency programswithin those. This advice is often founded in years of experiences and gestalt, as manyresidency programs may not publish specific guidelines for how they determine whichstudents they will interview or accept to their program. Some of these drivers havecontributed to application inflation, with students applying to an increasing totalnumber of programs, causing PTDs to have even more applications to review . A perioOur study did have some limitations. A reputation-based survey will fail to adequatelyaccount for students who prioritize other factors over reputation of residency programfor various reasons. For example, students may be limited geographically to particularareas of the USA, thus restricting their access to elite institutions. Additionally,some students prefer to match at a program for a select track, area of expertise,anticipated quality of life, or specialty culture. The proportion of students with thesetypes of considerations is difficult to account for in this retrospective study. Anotherlimitation is the small number of individuals located at one institution participatingin the specialty-specific surveys. By design, we identified the PTD of each specialty asthe ideal individual to complete the survey, likely having knowledge about and insightof the other programs in their specialty across the country. Yet, we acknowledge thatconducting a national survey of PTDs in the future might improve the value of thismethod of evaluation.Medical schools seek ways to evaluate their programs in order to do continuousimprovement. One method of program evaluation is to look at our match rate and to whichinstitutions our graduates go. We incorporated an internal ranking of GME programs andfound that the vast majority (71%) of our students matched at ‘elite’ and ‘top’programs. PTD rankings were generally supported by the quantitative data we analyzedregarding medical student success (clerkship grades and Step 1 scores), though we didfind that achieving Honors in all or even most clerkships was not a necessaryprerequisite to matching at an ‘elite’ or ‘top’ program. This study provided usefulfeedback about the quality of our UME educational program and its ability to producegraduates who can match in highly-regarded GME programs.


Fidelity monitoring is essential with implementation of complex health interventions, but there is little description of how to use results of fidelity monitoring to improve the draft program package prior to widespread dissemination. Root cause analysis (RCA) provides a systematic approach to identifying underlying causes and devising solutions to prevent errors in complex processes. Its use has not been described in implementation science.Stepping On (SO) is a small group, community-based intervention that has been shown to reduce falls by 31%. To prepare SO for widespread U.S. dissemination, we conducted a pilot of the draft program package, monitoring the seven SO sessions for fidelity of program delivery and assessing participant receipt and enactment through participant interviews after the workshop. Lapses to fidelity in program delivery, receipt, and enactment were identified. We performed a RCA to identify underlying causes of, and solutions to, such lapses, with the goal of preventing fidelity lapses with widespread dissemination.Lapses to fidelity in program delivery were in the domains of group leader’s role, use of adult learning principles, and introducing and upgrading the exercises. Lapses in fidelity of participant receipt and enactment included lack of knowledge about balance exercises and reduced adherence to frequency of exercise practice and advancement of exercise. Root causes related to leader training and background, site characteristics and capacity, and participant frailty and expectations prior to starting the program. The RCA resulted in changes to the program manual, the training program, and training manual for new leaders, and to the methods for and criteria for participant and leader recruitment. A Site Implementation Guide was created to provide information to sites interested in the program.Disseminating complex interventions can be done more smoothly by first using a systematic quality improvement technique, such as the RCA, to identify how lapses in fidelity occur during the earliest stages of implementation. This technique can also help bring about solutions to these lapses of fidelity prior to widespread dissemination across multiple domain lapses. The Centers for Disease Control and Prevention (CDC) developed the “Replicating Effective Programs” (REP) framework in 1996 to guide the process by which proven interventions may be translated into practice –4, the RFidelity in implementation science is defined as the “the degree to which … programs are implemented … as intended by the program developers” . FidelitWhile there is substantial literature describing the importance of fidelity monitoring for implementation, there is little description of how to actually use results of fidelity monitoring to improve the draft program package , 15–17. Six sigma is an engineering management strategy designed to improve quality and efficiency of operational processes. Designed by Motorola in 1986, it has been widely used across a variety of industries, including health care, to improve processes . Its priIn 2007, the CDC funded a dissemination research study to prepare the Stepping On (SO) falls prevention intervention for widespread implementation. Developed in Australia, SO is a small group, community-based program that in a randomized trial decreased falls among high risk older adults by 31% . The proPrior to this study, we had determined key elements of SO using a modified Delphi Consensus. The Delphi panel identified 85 key elements across the nine domains of adult learning, program components, role of group leader, role of peer coleader, exercise (starting and advancing), training and background of group leader, qualifications of invited exercise experts, home visit, and booster session.After elucidating key elements, content experts  in collaboration with the program developer (Lindy Clemson) prepared a draft program package for U.S. implementation. This package consisted of a training manual, used by the master trainer to train new leaders, and a program manual, used by the leader to implement the program . Both maThe workshop was held in one of five senior apartment buildings owned by Lincoln Lutheran of Racine, Inc., a faith-based non-profit organization. Inclusion criteria for the workshop were age 65 and over, living in one of two adjacent apartment buildings in the apartment complex, and a history of one or more falls in the past year or a fear of falling. Exclusion criteria were cognitive impairment as judged by the Services Manager and planned absence from more than one of the sessions. Eligible seniors were invited to participate in the workshop by the Apartment Services Manager. Thirteen older adults were invited to participate, and two of these declined. Eleven seniors gave informed consent and were enrolled in the workshop. Human subjects’ approval was obtained from the University of Wisconsin Health Sciences Institutional Review Board.Fidelity of implementation was assessed for three areas: program delivery, participant receipt, and participant engagement , 9.Content experts  developed a tool to be utilized by an expert observer to measure fidelity of delivery of the intervention in each of the seven sessions, based on the key elements identified through the modified Delphi Consensus. The fidelity tool assessed whether specific program activities occurred using a yes/no scale. It also assessed the quality with which key elements were incorporated using a scale of excellent, very good, average, not adequate. For example, for the item, “The leader linked exercises to function,” it was rated for occurrence (yes/no) and if it occurred, for quality . Some key elements were judged in the context of specific activities ; others were rated for the session as a whole . One item rated the degree to which the leader was teacher-like (poor fidelity) versus facilitator-like (high fidelity) using a 10-point scale. At the end of the tool, the expert observer was asked “What, if any, sections did you feel didn’t have the time managed well? If so, why? Was anything omitted, and what? Please note here anything of concern.”To reduce burden on the expert observer, each key element was assessed for fidelity in at least one session. While some elements were assessed at multiple sessions, none were assessed at all sessions. Two expert observers, a peer coleader, and a physical therapist (PT) evaluated fidelity. The peer coleader was a retired RN who was a participant in SO 3 years prior, and who then served as peer coleader for at least one SO workshop per year for 3 years, and as a co-trainer for at least one leader training per year for 2 years. She observed fidelity of non-exercise events. A PT with professional experience working with seniors observed fidelity of exercise events.Stepping On is a multifaceted falls prevention program, with participants working on alleviating the falls risk factors that apply to them. For some, this may relate to low vision and the need to see an ophthalmologist; for others, modifications of medications may be important. However, all participants can benefit from improving balance and strength and so are expected to practice balance and strength exercises on a regular basis at home and advance them in difficulty. Because exercise enactment is important for all, we selected this element as the focus for the evaluation of fidelity of participant receipt and enactment.In SO, a guest PT attends sessions 1, 2, and 6 to teach participants seven balance and strength exercises. Participants practice the exercises as a group in each of the seven workshop sessions, advancing as they are able, with guidance from the PT and workshop leader. In addition, participants are provided with an exercise manual and instructed to practice the exercises at home, daily for balance exercises and three times per week for strength exercises, advancing the level of difficulty at home as able. They are expected to continue exercising after the workshop ends.To evaluate fidelity of participant receipt and enactment related to exercise, two trained researchers interviewed participants in the home during the week after the final session to ascertain exercise knowledge (receipt), and their adherence to home exercise practice, degree of advancement of exercise by self-report, and belief in exercise to prevent falls (enactment). The interviewer showed each participant a picture of each exercise and asked how it was helpful for them, if they were performing that exercise, and if so, how often in a week, and if not, why not. They were asked to demonstrate how they perform the exercise, and rate on a scale of 1–10, how much they thought exercise could play a role in preventing their falls.Participants were assessed before the workshop for baseline demographics, self-report of use of assistive devices, number of falls in the year prior, and physical performance on the Timed Up and Go . Also beTo identify lapses of fidelity in program delivery, data on fidelity observations of workshop sessions were reviewed by Jane E. Mahoney and Vicki Gobel. Expert observers’ notes were reviewed to gain insights on why the expert observer assigned a score of “did not occur” or “not satisfactory.” Jane E. Mahoney and Vicki Gobel each compiled lists of fidelity lapses separately then met to ensure all lapses were identified. Differences were adjudicated by jointly reviewing pertinent fidelity observations of workshop sessions. Lapse of fidelity in program delivery of a key element was defined as a score by the expert observer of “did not occur” or “not satisfactory” on the workshop fidelity tool. A leader being rated as more teacher-like than facilitator-like was also considered a lapse of fidelity. Lapses in fidelity in program delivery were categorized according to the domain of key elements to which they applied: program aspects, exercise, upgrading exercise, group leader’s role, background of group leader and peer coleader, for a total of seven domains regarding fidelity of delivery.Data from participant interviews post-session seven were used to investigate lapses in fidelity of participant receipt and enactment related to the key element domain of exercise. Each reviewer (Jane E. Mahoney and Vicki Gobel) coded the data separately to identify lapses and then met to adjudicate differences by referring back to the raw data. Lapse in participant receipt was defined as being present if 30% of participants lacked knowledge regarding correct frequency of exercises at the post-session seven interview. Lapses in participant enactment were defined as 30% of participants practicing exercises less frequently than prescribed, not practicing all the exercises, not advancing with balance and strength exercises by self-report, or not believing exercises will help. The thresholds of 30% were established by the research team’s context experts , based on Lindy Clemson’s findings from the original SO study.Following coding of fidelity data, Jane E. Mahoney and Vicki Gobel reviewed the field logs of SO leaders, notes from expert observers, and all interviews and surveys of participants, SO leaders, site coordinators, and guest experts to become familiar with the materials. These data were not coded prior to the RCA; rather they were used as raw material and referred back to during the RCA as a form of reflective validation.We used the RCA process to identify underlying causes of lapses to fidelity in delivery, receipt, or enactment. RCA is a method that is often used as part of DMAIC to address a problem from a systems approach, using the “5 whys” technique” –28. It iWe convened an RCA team of three content experts: an MD (Jane E. Mahoney), a PT (Terry Shea), and an RN (Sandy Cech), three injury prevention research experts (two of whom had conducted participant interviews), and the research coordinator (Vicki Gobel). The group met in three sessions for a total of 10 h. Prior to the RCA sessions, team members received educational materials regarding the RCA process and a summary of all identified lapses of fidelity in delivery, receipt, and engagement. The RCA process began with group consensus to determine the categories of primary causes , defined as categories that would be further analyzed to ascertain potential root causes for all fidelity lapses. Next, the group brainstormed secondary and underlying causes for lapses of fidelity. To assist with identifying root causes, Vicki Gobel and Jane E. Mahoney provided findings from the surveys, interviews, and field logs of participants, leaders, site coordinators, and guest experts. For each root cause proposed by the RCA team, Jane E. Mahoney and Vicki Gobel reviewed the primary data to verify mention of that cause. For example, a potential root cause could be “participant was too frail to benefit from group exercise,” which had been elicited from participant pre-surveys and leader and PT interviews. If the primary data did not support that as a proposed root cause, then it was deleted.The RCA process was conducted for all lapses in delivery, covering one domain at a time, until all seven domains were investigated. For example, lapses in the domain of exercise delivery could have primary causes in five different categories: participants, site and support, leader background, leader training, and the exercises themselves. For each category of primary cause, the group used the “ask why five times” technique to identify underlying root causes for lapses in fidelity of delivery in that domain. The RCA process was likewise conducted for lapses in participant receipt and engagement in the domain of exercise. Table Following elucidation of root causes, the PI (Jane E. Mahoney) met with the content experts and the program’s developer (Lindy Clemson) over the course of 1 month to develop solutions for each root cause.Characteristics of the 11 participants enrolled in the SO workshop are shown in Table Table Lapses in fidelity of participant receipt and enactment in the key element domain of exercise are also shown in Table Figure Table Table To our knowledge, this is the first application of RCA in dissemination and implementation research. Using RCA, we identified causes and developed solutions to lapses to fidelity that occurred with the first implementation of a program package for SO. Following the six sigma approach (DMAIC), we defined key elements, measured fidelity with those elements in intervention delivery, receipt, and enactment, analyzed root causes, and improved the draft package for widespread dissemination. The RCA allowed us to get beyond a simplistic primary cause  to understand the contribution of complex, interacting human and system factors. We identified that organizational knowledge and readiness, leader background and competing tasks, and participants’ levels of frailty all contributed to lapses in fidelity of delivery, receipt, and enactment.A program package for dissemination of a complex intervention may include a number of components: a provider protocol, a training program and materials for providers, recruitment criteria and guidelines, forms and materials for participants, and an implementation guide and materials for the organization hosting the intervention. While all these components are often necessary for dissemination, not all may be developed as part of the original randomized trial . Our stuMeasuring fidelity of implementation is essential to maintaining quality and effectiveness of behavior change interventions , 33, 34.The REP framework is a widely used framework to guide packaging of proven intervention for widespread dissemination through the stages of precondition, pre-implementation, implementation, and maintenance . HoweverThe DMAIC approach may be especially important when trying to bring complex interventions to scale. DMAIC and RCA approaches have been used frequently in health-care delivery systems to understand errors with complex processes and identify solutions. The RCA focuses not on active errors , but rather on latent and environmental causes  . In our vice versa. Third, the fidelity tool was used by two expert observers, each of whom examined elements within their expertise. Further testing of the tool, including inter-rater reliability testing, is necessary before widespread use. Fourth, the leader was a novice and had little chance to practice new skills. It may be that experience would negate these lapses in fidelity. However, it is more likely that the outcomes of this process, in particular the enhanced training and coaching that resulted, would serve to accelerate novice leaders to expert. Fifth, there are inherent biases in any causal analysis of adverse events  or e events , 40. To In summary, when translating complex interventions, we suggest that it is essential to use a proven quality improvement technique such as DMAIC and RCA at the pre-implementation stage, to refine the program prior to widespread use. Importantly, as can be seen in this study, the RCA allows identification of multiple domains of causes, rather than focusing on a simplistic solution of “provide more training.”JM is board certified in geriatrics and internal medicine. She is a Professor of Geriatrics in the University of Wisconsin School of Medicine and Public Health. She also serves as Executive Director of Wisconsin Institute for Healthy Aging, a non-profit organization that disseminates evidence-based prevention programs for older adults. She is Principal Investigator of the Community-Academic Aging Research Network, an NIA-funded initiative to support research collaboration between University of Wisconsin researchers and community partners from Wisconsin’s Aging Network. Dr. JM has received funding from the American Physical Therapy Foundation, the CDC, the NIA, and the State of Wisconsin for epidemiologic and clinical research on falls. She has studied risk factors for falls after hospitalization, clinical trials of community-based multifactorial falls interventions, and dissemination research on the Stepping On falls prevention program. She is currently working with University of Wisconsin’s Active Aging Research Center to help develop internet-based technologies to help older adults reduce falls and maintain independence.JM was responsible for conceptualizing the theoretical and empirical formulations of the research project, literature review, study protocol, and design, and collecting, analyzing, and interpreting data as well as manuscript preparation. VG was part of the original study group that participated in development of solutions based on findings of the RCA and provided feedback on draft versions of the paper. JJ participated in conduct of the root cause analysis and feedback on draft versions of the manuscript. TS and SC were also part of the original study group that prepared a draft program package for U.S. implementation. LC developed the program in Australia initially and helped the original study group prepare the program for U.S. implementation.The content is solely the responsibility of the authors and does not necessarily represent the official views of the Centers for Disease Control and Prevention. Presented in part at the 65th Annual Meeting of the Gerontological Association of America, San Diego, CA, USA, November 2012.JM and LC are co-authors on the Stepping On Leader Manual, Third North American Edition, Frieburg Press, Cedar Falls, IA, USA, 2011. All the other authors declare no conflict of interest.