QR updating procedure as a solution approach for linear least squares problem with equality constraints. We reduce the constrained problem to unconstrained linear least squares and partition it into a small subproblem. The QR factorization of the subproblem is calculated and then we apply updating techniques to its upper triangular factor R to obtain its solution. We carry out the error analysis of the proposed algorithm to show that it is backward stable. We also illustrate the implementation and accuracy of the proposed algorithm by providing some numerical experiments with particular emphasis on dense problems.In this article, we present a  We consider the linear least squares problem with equality constraints (LSE) tructure . The assQR factorization methods. In the method of weighting, a large suitable weighted factor γ is selected such that the weighted residual γ such that The solution of LSE problem  can be omentclass2pt{minimQR factorization with applications to LLS problem and presented updating algorithms which exploited LEVEL 3 BLAS. Yousaf [QR factorization as a solution tool for LLS problem. Parallel implementation on GPUs of the updating QR factorization algorithms presented in [QR factorization for the solutions of LSE problems.Updating is a process which allow us to approximate the solution of the original problem without solving it afresh. It is useful in applications such as in solving a sequence of modified related problems by adding or removing data from the original problem. Stable and efficient methods of updating are required in various fields of science and engineering such as in optimization and signal processing , statist. Yousaf  studied ented in  is perfoented in . Zhdanovented in  studied ented in  presenteQR factorization. It is well known that Householder QR factorization is backward stable [QR factorization of the subproblem is calculated and its R factor is then updated by appending the removed block of columns and rows, respectively, to approximate the solution of problem . Here, we briefly discuss the Householder reflection method due to our requirements. For a non-zero Householder vector  formula : \documeE considered in .The description of backsub for back-substitution procedure, respectively.Here, Algorithm 5 for a solution of LSE problem  calls upQR factorization and for the back-substitution procedure, we refer the reader to [In this section, we will study the backward stability of our proposed Algorithm 5. The mainstay in our presented algorithm for the solution of LSE problem is the updating procedure. Therefore, our main concern is to study the error analysis of the updating steps. For others, such as the effect of using the weighting factor, finding the eader to , 31. Hereader to .−8 in single precision computer arithmetic, while in double precision it is of the order 10−16. Moreover, for addition and subtraction in the absence of guard integers, we can write model 37\documeressions  which shA and B and its corresponding right-hand side vectors b and d are generated randomly using the MATLAB built-in commands QR Householder factorization with column pivoting by E, the size of the reduced subproblem (SP), value of the weighted factor ω, the relative errors This section is devoted to some numerical experiments which illustrate the applicability and accuracy of Algorithm 5. The problem matrices mentclass2pt{minimQR factorization. We updated only the R factor of the QR factorization of the small subproblem in order to obtain the solution of our considered problem. Numerical experiments are provided which illustrated the accuracy of the presented algorithm. We also showed that the algorithm is backward stable. The presented approach is suitable for dense problems and also applicable where QR factorization of a problem matrix is available and we are interested in the solution after adding new data to the original problem. In the future, it will be of interest to study the updating techniques for sparse data problems and for those where the linear least squares problem is fixed and the constraint system is changing frequently.The solution of linear least squares problems with equality constraints is studied by updated techniques based on


The development of Web-based interventions for substance abuse in Latin America is a new field of interest with great potential for expansion to other Spanish-speaking countries.This paper describes a project aimed to develop and evaluate the usability of the Web-based Help Program for Drug Abuse and Depression  and also to construct a systematic frame of reference for the development of future Web-based programs.The PAADD aims to reduce substance use and depressive symptoms with cognitive behavioral techniques translated into Web applications, aided by the participation of a counselor to provide support and guidance. This Web-based intervention includes 4 steps: (1) My Starting Point, (2) Where Do I Want to Be? (3) Strategies for Change, and (4) Maintaining Change. The development of the program was an interactive multistage process. The first stage defined the core structure and contents, which were validated in stage 2 by a group of 8 experts in addiction treatment. Programming of the applications took place in stage 3, taking into account 3 types of end users: administrators, counselors, and substance users. Stage 4 consisted of functionality testing. In stage 5, a total of 9 health professionals and 20 drug users currently in treatment voluntarily interacted with the program in a usability test, providing feedback about adjustments needed to improve users’ experience.The main finding of stage 2 was the consensus of the health professionals about the cognitive behavioral strategies and techniques included in PAADD being appropriate for changing substance use behaviors. In stage 5, the health professionals found the functionalities easy to learn; their suggestions were related to the page layout, inclusion of confirmation messages at the end of activities, avoiding “read more” links, and providing feedback about every activity. On the other hand, the users said the information presented within the modules was easy to follow and suggested more dynamic features with concrete instructions and feedback.The resulting Web-based program may have advantages over traditional face-to-face therapies owing to its low cost, wide accessibility, anonymity, and independence of time and distance factors. The detailed description of the process of designing a Web-based program is an important contribution to others interested in this field. The potential benefits must be verified in specific studies.International Standard Randomized Controlled Trial Number (ISRCTN): 25429892; http://www.controlled-trials.com/ISRCTN25429892 (Archived by WebCite at http://www.webcitation.org/6ko1Fsvym) The drug treatments with the greatest empirical validity are those involving cognitive behavioral interventions (CBIs) . Some auA recent meta-analysis of the effectiveness of Internet-based interventions based on cognitive behavioral therapy demonstrated a large effect size  as compared with other modalities, including psychoeducation  . On the Web-based interventions allow complex treatments to be delivered with consistency and minimal demands on staff time and training resources. Moreover, computerized programs may be less threatening, provide greater anonymity ,8, and rDuring the last decade, several Web-based interventions have been developed and validated in the mental health field . Those fWeb-based interventions developed in Latin America are very limited and focus on smoking , heavy dApproximately 3.3 billion people around the world use the Internet, among them 345 million in Latin America, with a growth of 1808.4% in the past 15 years . AccordiThere are few publications that describe the process of developing a Web-based intervention, the selection of strategies required, or the limitations involved in starting it up . Such inThe purposes of this paper are therefore to describe the development process of the Web-based Help Program for Drug Abuse and Depression   in MexicThe PAADD was developed in an interactive multistage process that involved design, testing, and redesign tasks, following international recommendations for the development of eHealth strategies and ethical standards for Web-based interventions .The aims of this phase were to define the structure and contents of the program. After a search of the literature on treatment in Mexico for problems of substance use and depression, 3 sources were chosen: (1) self-help manual ¿Cómo dejar de consumir drogas? (How to Stop Using Drugs) ; (2) WebThe basic structure of the intervention and the specific techniques of behavior modification were taken from the self-help manual How to Stop Using Drugs , a briefThe Web-based program ADep ,34 is a The Web-based program Beber Menos  is part On the basis of the literature review of Web-based interventions, the specific characteristics of the PAADD were defined as follows:1. The PAADD is a stand-alone intervention directed at persons with risky levels of drug use and not at those with drug dependence.2. It offers strategies to address substance abuse and depressive symptoms together.3. It includes 4 steps to establish a baseline and a treatment goal, in addition to strategies to change the pattern of use, stay focused on the goal, and prevent relapses.4. It offers contact from a counselor, which previous studies have shown produces favorable results ,45Once the outline and the core contents for each step were planned, these were shared with a focus group of 8 experts in addiction treatment from the public mental health care system in the Mexico City metropolitan area. Participants were asked to comment on the proposed techniques and the possible impact of the program and to provide suggestions for improvement and implementation.These experts expressed the opinion that the CBI strategies and techniques for changing substance use behaviors were appropriate. Their comments were useful for fine-tuning the inclusion criteria: (1) medium risk level of drug use according to the Alcohol, Smoking and Substance Involvement Screening Test  and (2) The PAADD was structured into 4 successive steps. The home page is shown in The aim of this module is to promote the awareness of personal drug use and its risks in order to increase the motivation to change. This module includes assessment instruments and behavioral strategies to establish a baseline. At the end, users receive a printable report and are asked if they are committed to participating in a process of change. A positive answer directs them to step 2; a negative answer redirects them to a feedback page to prompt reflection about the consequences of drug use.The purpose of this step is to set treatment goals. Users are provided with various tools, such as the recommendation to define a goal , to establish a reduction plan , and to set a time limit to achieve the goal. At the end of this step the user receives a printable feedback page.Step 3 is aimed at achieving behavioral change in accordance with the goals established in step 2. Users are advised to log in every day, record daily drug use, review the progress graph, and do an activity to achieve the change. They are offered tools to perform a functional analysis of substance use behavior, to develop action plans to face risk situations, to apply emotional control techniques to cope with anxiety, to generate positive self-reinforcement, and for cognitive restructuring of negative thoughts.The objective of step 4 is to reinforce changes in drug use behavior and avoid relapses. It offers strategies for recognition of success (self-reinforcement), for facing pressure to use drugs (assertiveness), and for seeking social support, among others. The counselor makes the decision to enable this step in accordance with the results from step 3.The program is designed to be completed in 8 weeks, but this period can be extended if necessary depending on the evaluation of the counselor, who might suggest specific activities that the participant may have skipped, according to the progress and needs of each user. The role of the counselor also includes monitoring the completion of the activities as well as providing written feedback and encouragement within; all these functions are performed through a messaging system within the program within 24 hours after a user logs in. This is a minimal contact scheme that differs from conducting therapy sessions via the Internet and represents an opportunity to optimize time because one counselor can monitor 2 or more patients simultaneously.The structure of the Web-based intervention was outlined as a diagram in order to establish the sequence of the steps involved. The PAADD was constructed by a commercial programming company as a Web application accessible through computers, laptops, or tablets with any browser; although it is not a mobile application, it can also be accessed with a mobile phone. It took into account 3 types of end users: administrators, counselors, and substance users. Functionality and usability tests were performed on the completed application.The purpose of this process was to ensure that the PAADD met the technical requirements set out in the functional design and to identify programming glitches. The test specifically verified the following:The program worked according to specifications.The database responded to the input and output specifications of expected information.The system followed the sequence identified in the functional design.These functionalities were tested by members of the research team (n=7), who interacted with the program and verified each one, following a checklist. Errors found were corrected and a second test was performed. This functionality testing was then followed by a usability test.Usability is defined as the ability of a software or program to be understood, learned, used, and be attractive to the end user under specific conditions of use . The objA total of 9 health professionals from Mexico City, State of Mexico, and the states Morelos, and Baja California agreed to participate. Inclusion criteria were that (1) they worked in an addiction treatment center; (2) they had at least a year of experience working in substance abuse treatment using cognitive behavioral strategies; (3) they had experience using the Internet; and (4) they were regular email users. Each was asked to invite patients under treatment (n=20) to give their opinion of the PAADD. The inclusion criteria for the patient group were that (1) they were adults; (2) they had received a minimum of 4 treatment sessions for abuse of psychoactive substances during the previous 6 months; (3) they could read and write; (4) they had experience using the Internet; and (5) they were regular email users.All participants  responded to a questionnaire about their Internet usage and received an instruction guide for usability test. The checklist for the health professionals focused on evaluating the role of the counselor , whereas that for patients focused on their perspective about language and ease of use.This procedure was approved by the Research Ethics Committee of the Ramón de la Fuente Muñiz National Institute of Psychiatry. All participants were volunteers and signed an informed consent form with the understanding that the data might be used in research. All information obtained in this program is confidential and available only to members of the research team. The questionnaires, forms, and other documents were identified by a numeric code that avoids the identification of participants.The usability test was based on a selection of tasks and pages according to the guidelines proposed by Dumas and Loring : (1) freThe tests carried out by the health professionals consisted of navigating the PAADD, following a list of specific actions, and reporting on their experience. The tests by patients were carried out at the centers where they were receiving treatment. A member of the research team accompanied them and recorded their commentaries and suggestions.Results of the usability test are presented here separately by type of participant.The group included 5 women and 3 men with an average age of 30.25 years (SD 5.33). All participants had a master's degree in psychology of addiction and had an average of 5 years of experience using the CBI approach to treat persons with problems of substance abuse. All participants were regular users of the Internet, with an average daily use of 6.1 hours, the majority from their home and workplace.The main suggestions from the health professionals focused on the content and format but not on the structure of the program, for instance, recommendations to change the color scheme, page layout, font size, arrangement of functions, inclusion of confirmation messages at the end of activities, avoiding the use of technical language and “read more” links, providing short introductory texts for every task, and creating more visible help buttons with specific explanations and tips. They found the functionalities easy to learn, although they indicated that some were not intuitive at first glance. They suggested providing feedback about every activity and including more activities and exercises within the program, as well as more tips on how to end substance abuse. They also suggested the possibility of including testimonials from persons who had benefited from the program see .The user group included 16 men and 4 women, 60% of them single, with an average age of 29.6 (SD 9.1) years. In this group 70% (14/20) had completed high school, 15% (3/20) an undergraduate degree, and 15% (3/20) elementary school; 65% (13/20) were employed, 20% (4/20) did not work, and 15% (3/20) were students. All participants were regular users of the Internet, with an average daily use of 3 hours, the majority from home (85%) and the rest using their cell phone and/or at Internet cafes.Users suggested a more dynamic design with images and colors, clearer, more concrete instructions, inclusion of empathetic, reflective feedback, and also a chart showing the names and appearance of different drugs. The majority said it was important to be able to modify the weekly goal and clarify the sequence of activities in the program see .The observations and suggestions of users and health professionals were compiled into a document that served as a guide for development of a new version of the PAADD, which was in turn put through an evaluation process.This paper describes the development and results of the usability test of PAADD, a Web-based intervention with the participation of a counselor, the first of its kind in Latin America that is designed to reduce substance use and depressive symptoms. The development of the program was systematic and rigorous. It began with a review of the literature on brief interventions and incorporated the opinions of experts in addiction treatment as well as 2 types of end users, following a procedure similar to that reported by Morrison et al . The resThe translation of cognitive behavioral strategies to the concrete activities of the PAADD was complex and time consuming. Van Voorhees et al  note thaAn important characteristic of the PAADD is that it includes the participation of counselors who provide feedback according to the motivational interviewing techniques. This element is intended to strengthen adherence, as it has been shown that the inclusion of health professionals has a positive influence on retention and motivation of participants in such programs ,54.In the usability test, drug users commented that the PAADD was professional and clear. Many felt that the information presented within the modules was relatively easy to follow, comprehensive, and of good quality. A few participants reported at times feeling impatient to receive the next module. The majority of users commented positively on the content of the program and felt that they could benefit from it. In their view, it helped principally with improved self-awareness of their pattern of drug use. They reported that the PAADD encouraged them to think about self-management techniques, how to monitor their thoughts and feelings, and how to regulate their behavior and consumption patterns. These observations suggest that the design process was successful; it resulted in a program that is easily usable by persons who wish to end their substance use and who present depressive symptomatology.The PAADD may have advantages over traditional face-to-face therapies because of its low cost, wide accessibility, anonymity, and independence of time and distance factors. It may be a good alternative for persons not receiving treatment owing to factors such as physical distance from available services, lack of qualified providers, socioeconomic condition, and stigma, although these potential benefits will have to be verified in specific studies.In future efforts it will also be important to consider that the initial investment for the design of a Web-based intervention can be high and that it is necessary to perform a cost-benefit analysis to document the viability of the project. It should also be kept in mind that technologies are developing rapidly; in order to provide increased availability and accessibility, it is necessary to plan from the beginning the possibility of using the program on diverse types of devices, such as mobile phones and tablets, and not just on computers.Because PAADD is not an open-access program, it will also be necessary to design a dissemination strategy that would include training of counselors and to promote the use of PAADD in specialized treatment centers. This will allow to document adoption and implementation in real-life scenarios.The lack of forums or chat functionalities for users to interact with each other may be a limitation in this program because having contact with others in the same situation can be a source of support. It is also possible that the small number of users participating in the usability test limited the number of opinions regarding improvements to the program. Such limitations have been reported in similar studies ,54.Data from the usability test indicate that the PAADD has the necessary features to support users in their process of reducing drug use. If this is corroborated in effectiveness studies, the program could be used as an alternative to treatment as usual.


The first formal orientation program for incoming emergency medicine (EM) residents was started in 1976. The last attempt to describe the nature of orientation programs was by Brillman in 1995. Now almost all residencies offer orientation to incoming residents, but little is known about the curricular content or structure of these programs. The purpose of this project was to describe the current composition and purpose of EM resident orientation programs in the United States.In autumn of 2014, we surveyed all U.S. EM residency program directors (n=167). We adapted our survey instrument from one used by Brillman (1995). The survey was designed to assess the orientation program’s purpose, structure, content, and teaching methods.The survey return rate was 63% (105 of 167). Most respondents (77%) directed three-year residencies, and all but one program offered intern orientation. Orientations lasted an average of nine clinical (Std. Dev.=7.3) and 13 non-clinical days (Std. Dev.=9.3). The prototypical breakdown of program activities was 27% lectures, 23% clinical work, 16% skills training, 10% administrative activities, 9% socialization and 15% other activities. Most orientations included activities to promote socialization among interns (98%) and with other members of the department (91%). Many programs (87%) included special certification courses . Course content included the following: use of electronic medical records (90%), physician wellness (75%), and chief complaint-based lectures (72%). Procedural skill sessions covered ultrasound (94%), airway management (91%), vascular access (90%), wound management (77%), splinting (67%), and trauma skills (62%).Compared to Brillman (1995), we found that more programs (99%) are offering formal orientation and allocating more time to them. Lectures remain the most common educational activity. We found increases in the use of skills labs and specialty certifications. We also observed increases in time dedicated to clinical work during orientation. Only a few programs reported engaging in baseline or milestone assessments, an activity that could offer significant benefits to the residency program. Emergency medicine (EM) residency programs commonly offer dedicated curricula designed for orientation of beginning residents. An orientation curriculum was first developed for incoming EM residents at the University of Cincinnati in 1976.Since 1995 there have been no general descriptions or studies describing EM orientation practices throughout the U.S. Additionally, we found no standards to guide program development. The literature is rich, however, with conversations about bridging the gap between undergraduate medical education (UME) and graduate medical education (GME),The purpose of this project was to profile the current state of orientation programs for entering EM residents across the U.S. At the outset, we anticipated an increase in the number of formal orientation programs and also predicted that we would find considerable variability in program characteristics, length, and goals. Further, we expected to find that residency programs had increased their use of benchmark assessments for incoming interns to determine where they were in their progress towards achieving the ACGME milestones. Finally, we hoped to find significant innovation in program activities and assessments that might be generalizable to others.We surveyed the EM residency program directors of all ACGME-accredited programs in the U.S. Survey participants were identified through three different residency program directories: The Society for Academic Emergency Medicine Residency Directory,12We adapted our survey instrument from one used by Brillman (1995).Survey developers were presented with a draft survey derived from the Brillman article. They were asked to add, modify, or delete items to create an instrument that contained only items they believed were important for profiling a modern residency orientation program. The subsequent results were fine-tuned into proper survey format and then presented to the developers as a pilot, which led to an additional round of modifications.The final instrument contained 18 items: 13 checklist items, one multiple choice, one fill in the blank and three open-ended comment items. To shorten the survey administration time, each of the checklist items was preceded with a skip logic question, which is a yes vs. no filter item that directs the respondent only to applicable subsequent checklists.We used the Dillman tailored design method (TDM) for electronic (e-mail) surveys to guide this national study.X2). The program demographics that we tested for bias included region of the country ; program length (three- or four-year program); and program size (number of residents) by percentile rank . We used descriptive statistics to profile the orientation programs for EM medicine residency programs. We analyzed electronic survey data with IBM-SPSS for Windows, Version 22.0.The overall survey return rate was 63% (105 of 167). The respondent sample was evaluated for representativeness using chi-square tests of proportion . Survey All but one of the EM residency program director respondents said that they conduct intern orientations . Orientation programs were most frequently sponsored by the Department of EM (97%), but some programs obtain additional sponsorship through the following: academic health centers (AHCs) (59%), medical schools (14%), or other affiliated hospitals (12%). One program said that their orientation program was sponsored through their Graduate Medical Education Office and that most of the orientation activities were shared with interns from other specialties’ residency programs.The length of EM orientation programs averaged 22 days (SD=11.8). Residents spent an average of 8.9 days (SD=7.3) of clinical orientation, i.e. clinical work in the ED. Non-clinical activities accounted for 13.2 orientation days (SD 9.3).We asked directors to estimate how they allocated their orientation time across various activities . DirectoThe most frequently expressed goals of orientation programs were an opportunity for interns to get to know each other (98%), familiarization with hospital and departmental policy (95%), acclimating to a new ED (93%), opportunity to get to know other members of the department (91%), and completion of administrative tasks (89%). Less frequently expressed goals were review of skills or medical knowledge learned in medical school (59% and 55%), baseline assessment of clinical skills and medical knowledge (54% and 50%), and other purposes, which included additional baseline assessment and certification courses (11%) .Specific orientation activities offered by programs included social activities (100%), lectures/didactic sessions (98%), procedure labs (95%), special certification courses (87%), high-fidelity simulation (82%), simulated patient encounters/objective structured clinical examination (OSCE) (34%), and baseline assessment (33%) .The most frequent topics included in lectures or didactic sessions were use of the electronic medical record (90%), physician wellness (75%), and chief complaint-based lectures (72%). Specific procedural or skill sessions offered by programs included ultrasound (94%), airway management (91%), vascular access (90%), suturing/wound management (77%), splinting (67%), trauma-related (62%), cadaver-based lab (25%), Head, eye, ears, nose and throat emergencies (HEENT) (17%), animal-based lab (12%), dental lab (8%), and other skills (10%) .Specialized certification courses were offered by 88% of programs. Specific specialized courses offered during the orientation curriculum were ACLS , PALS (74%), ATLS (68%), NRP or other neonatal courses (27%), and other specialized courses (12%).The most frequent social activities offered by programs were social events for both EM residents and faculty (87%), team-building activities (55%), and social events for EM interns only or EM residents only (40% and 40%). Twenty-eight percent of programs offer a formal retreat that occurs off-site and 18% offer social events that include other ED personnel such as nursing or staff.For programs that perform baseline assessment of new EM residents, baseline assessment practices focused on medical knowledge (79%), patient communication (49%), history-taking skills (42%), physical exam skills (36%), EKG interpretation (36%), emotional intelligence/personality assessments (27%), radiology interpretation (18%), learning style tests, evidence-based medicine (EBM) knowledge (9%), or other skills such as Level 1 Milestones or procedural skills (12%).We identified several themes through analysis of verbatim comments. First, EM residency program directors appreciated having time within an orientation curriculum for bonding and socialization. They also valued dedicated time to introduce their care delivery system and expectations for the program. However, there was an expressed desire to further streamline administrative requirements and tasks, continue to move away from lecture-based curriculum while placing more emphasis on interactive didactics , and incorporate more assessment of baseline skills.Survey respondents described innovations such as active learning experiences that include procedural assessment, and simulation experiences that involve breakdowns of critical steps and opportunities to train and remediate specific steps. Programs also included many special topics such as EBM skills, patient safety and quality, consultation skills, crew resource training, electronic communication-social media use, and work-life balance. Among programs that conducted baseline milestone assessment, some confirm that all Level 1 Milestones are met, while others evaluate only select Level 1 Milestones.In comparison to the 1995 survey by Brillman et al, slightly more EM programs are offering a formal orientation curriculum (2014= 99% vs. 1995=93%). Orientations now average 22 days, a 57% increase over the 14 days reported by Brillman in 1995. The difference appears to come from the 8.9 days, on average of additional clinical time working in the ED, almost triple that of Brillman’s reported 2.4 days. Similar to the 1995 survey, the activity with the most dedicated time during orientation was lecture-based didactics (2014= 34 h vs. 1995= 35 h). However, we note an overwhelming increase in the number of programs that offer procedure labs and specialty sessions during their nonclinical orientation (2014= 95% vs. 1995= 52%). As Brillman reported in 1995, EM programs continue to offer specialized courses during orientation . Considerably more programs are now offering a PALS course (2014= 71% vs. 1995= 39%). We report an increase in the proportion of dedicated time applied to formal clinical orientation , 44% in 2014 vs. 17% in 1995, as well.The three-fold increase in ED clinical work during orientation between 1995 and 2014 is perhaps best explained by the survey participants’ responses to the “purpose” for orientation. Most of the purposes provided seem to involve enculturation: Getting to know one another, familiarizing interns with the hospital and department policies, acclimation to a new ED, getting to know members of the department, and team building. Since only 33% of respondents said that they include formal baseline assessment during orientation, an alternative explanation is that the additional ED clinical work is designed for informal assessment of an intern’s baseline clinical skills.Innovations described by respondents include an increasing number of specialty topics and sessions, expanded active learning experiences, and incorporation of introductory assessment and baseline EM milestone assessment. With the increased focus on competency-based assessment introduced by the ABEM/ACGME Milestone Project,Documentation of medical student progress towards Level 1 Milestones could offer significant benefits to residents, their residency programs, and ultimately their patients. Deficiencies could be identified and remediated earlier, or customized learning plans based on milestone achievement could be developed. Competency-based assessment that document milestone progress or measure attainment of “Entrustable Professional Activities” are being developed and are beginning to surface in the literature.We demonstrated that our respondents were representative of the population as a whole, but because we did not receive a survey from every program, generalizability to all programs is not assured. Additionally, we should note the limitations common to survey research. First is the potential that selection bias occurred, which in our case would have been the tendency for residencies with no orientation program to have avoided participation in the survey. Second is the potential for recall bias among those who completed the survey. Finally, we cannot be certain that we captured the rich detail of every residency orientation program. By seeking a general profile of residency orientation, some unique and creative program details may have remained undetected.Since the last national survey of EM residency program directors about their orientation programs, much has changed. Now, nearly every program has an established orientation program for incoming residents. Overall, the duration of orientation has increased by almost 60%, which is primarily attributable to increases in dedicated clinical work during orientation. The most common activities remain didactic sessions and social activities, but with improvements in technology and simulation, there has been an increase in skill training sessions. A minority of programs implement baseline assessments of their learners, which is an opportunity for programs to develop early interventions for incoming residents not meeting minimum expectations.


The role of the “debrief” to address issues related to patient safety and systematic flaws in care is frequently overlooked. In our study, we interview surgical leaders who have developed successful strategies of debriefing within a comprehensive program of quality improvement.Semi-structured interviews of four implementation leaders were performed. The observations, beliefs and strategies of surgical leaders are compared and contrasted. Common themes are identified related to program success and failure. Quality and safety researchers performed, coded and categorized the interviews and coordinated the analysis and interpretation of the results. The authors from the four institutions aided in interpretation and framing of the results.The debriefing programs evaluated were part of comprehensive quality improvement projects. Seven high-level themes and 24 subthemes were identified from the interviews. Themes related to leadership included early engagement, visible ongoing commitment and enforcement. Success appeared to depend upon meaningful and early debriefing feedback. The culture of safety that promoted success included a commitment to open and fair communication and continuous improvement.There were many challenges to the success of debriefing programs. The loss of institutional commitment of resources and personnel was the instigating factor behind the collapse of the program at Michigan. Other areas of potential failure included communication issues and loss of early and meaningful feedback.Leaders of four surgical systems with strong debriefing programs report success using debriefing to improve system performance. These findings are consistent with previously published studies. Success requires commitment of resources, and leadership engagement. The greatest gains may be best achieved by programs that provide meaningful debriefing feedback in an atmosphere dedicated to open communication. Since the publication of “To Err is Human” in 1999, medical error has been acknowledged as a major contributor to the burden of illness in the United States (US) . The WHODespite a concerted, national effort, the burden of medical error in the US remains troublingly high . Follow-The “debrief” within a three-part surgical safety checklist is alternatively referred to as a “sign-out”. These terms are used interchangeably in most checklists and within this paper. However, the use of the term “debrief” emphasizes additional communication beyond acknowledgement of the tasks performed by the nursing team at the end of the surgical case. This additional communication typically addresses safety, equipment and efficiency issues arising during the case and identifies opportunities for improvement.In contrast to many hospitals that have struggled with checklist implementation, there are a few notable institutions have used the surgical checklist effectively to improve quality and safety and continue to do so. Many of these successful institutions have specifically focused on the debriefing aspect of the checklist to address systematic issues of safety, efficiency, and communication , 10. In Through a thematic analysis of these interviews, we explore effective strategies, the role of leadership, barriers and facilitators, as well as contextual factors that present stumbling blocks and opportunities for medical leaders looking to improve system efficiency and strengthen the role of debriefing in their hospital.This study was reviewed by the Harvard Ethics Board and judged to be exempt from ethics review.This study presents a thematic analysis of interviews of surgical leaders at four US institutions. The observations, beliefs and strategies of surgical leaders as reflected in their statements are compared and contrasted. Common themes related to program success and failure are identified. Three quality and safety researchers who are not members of the institutions under study generated the study design, performed, coded and categorized the interviews and coordinated the analysis and interpretation of the results. The authors representing the four institutions aided in interpretation and framing of the results.To identify four institutions, a series of searches of the grey literature through Google search engine, word of mouth, emails and phone calls as well as searches using medical search engines were performed. The aim of these searches was not to be exhaustive but rather to identify representative programs that had developed a debriefing protocol with a strategy of process improvement in mind. One of the four target hospitals (McLeod) was known to the lead authors of this study . This program was also asked to identify any additional programs they were aware of that had adopted a similar strategy for debriefing.Four centers that are geographically and administratively distinct were identified: Beaumont (Michigan), McLeod (South Carolina), Madigan (Washington), Memorial Health (Florida). The programs in South Carolina, Washington and Florida reported ongoing success while the program at Michigan had initial success but ultimately failed.Medical and administrative leaders were contacted by both email and telephone at each site and asked if they would be willing to undergo a formal interview by telephone. We identified one key medical leader from each site.A script for a semi-structured interview was created through an iterative process. An initial script was generated, piloted on quality and safety research colleagues and revised. The script was used as a guide for the interview but was not rigidly followed to allow for spontaneous discussion of elements not addressed within the script.The areas of assessment that were targeted in the interviews were defined by two members of the research team (MB and WB). These included: motivation/rationale for the debriefing initiative, contextual factors, role of local leaders, implementation process, functionality, evolution/sustainability, buy-in and reinforcement. In addition, the impact of the debriefing strategy on process and safety outcomes, team-work and communication was assessed. Each interview was performed by telephone using a web-based recording program and transcribed.Transcribed interviews were reviewed by the primary interviewer alongside the original recording to ensure accuracy.Using NVivo, a software program designed for qualitative analysis, the transcribed interviews were coded according to themes. These themes were identified through two-stage process including deductive themes based in part on the predetermined questions and inductive themes that emerged from the content of the interviews. Each line of transcribed text was reviewed for ideas, concepts and reflections which were categorized by thematic codes. Each new idea or concept was assigned to an existing thematic code or a new code was created. At the end of coding all the interviews, each interview was reviewed again to ensure consistent coding across interviews. Coding was performed by a single reviewer. After coding, code validation was performed by a second qualitative expert who reviewed the content and coding themes for consistency. If discrepancies in coding were determined, discussion among the authors was performed until consensus was achieved. The coded themes were reviewed and broad categories encompassing these themes were created.Narratives were collected within thematic groupings. Common observations were reviewed and discussed by the research team including the clinician leaders who had been interviewed from each site. The elements that were commonly linked to success were categorized; the challenges encountered by each program were, likewise, categorized.The program at Beaumont, Michigan, was examined as a separate case study. The threats to the success of the Michigan program that eventually lead to its collapse were identified.How these threats might be prevented or mitigated were reviewed and are presented within the discussion.All narratives are presented as verbatim. Any changes that are provided for clarification are presented in square brackets.The four implementation leaders interviewed had developed their strategies within four very different health systems.Madigan is an Army Medical Center in Washington that performs roughly 10,000 surgeries a year and has 220 inpatient beds. Madigan is staffed by 54 surgeons, 12 anesthesiologists and 49 certified registered nurse anesthetists (CRNAs). Dr. Andrew Foster is Chief of Anesthesia at Madigan and was a physician leader who took on a primary role in creating and maintaining the debriefing program at Madigan.Memorial Health is a large public health system in Southern Florida including five hospitals with a total of 1858 beds and 62 ORs staffed by 545 surgeons, 101 anesthesiologists and 53 anesthesia assistants as well as 112 CRNAs. Dr. Stanley Marks is a surgeon and Chief Medical Officer for Memorial Health, who developed a comprehensive approach to patient quality and safety that has been instituted across Memorial Health.McLeod Health is a large private health system including seven hospitals in South Carolina. McLeod regional center completes 19,000 cases a year and has 489 beds. The OR is staffed by 125 surgeons, 12 anesthesiologists and 50 CRNAs. Dr. Michael Rose is an anesthesiologist and Vice President of Surgical Services at McLeod Regional Medical Center and has helped lead a state-wide initiative to improve Surgical Safety in South Carolina.Beaumont is a large public health system including eight hospitals and performing more than 50,000 surgeries every year in Michigan. The system is staffed by 600 surgeons, 84 anesthesiologists and 145 certified registered nurse anesthetists. Dr. Robert Welsh is Vice Chief of Surgical Services at Beaumont Hospital- Royal Oak and developed the program of surgical debriefing for surfacing safety defects at Beaumont Hospital.In analyzing the interviews of these leaders, seven high-level themes were identified with an additional 24 subthemes that fell within the high-level umbrella themes.Figure The clinician leaders we interviewed all adopted primary leadership roles in initiating programs as well as promoting the importance of the program within the clinical teams and executive leadership. These leaders are passionate, articulate and deeply committed.Each of the four programs we present used a team training approach as the structural framework on which their programs were built. Different programs used different team training platforms. Two of these programs adopted the formal team training program TeamSTEPPS™   while twAlthough each interviewee reported that debriefing was part of a much larger implementation strategy at their centers, they acknowledged that the debriefing offered a very specific, crucial contribution to quality improvement. For each program, the debrief aims to identify and address flaws in the system and improve patient safety and system function. Education, piloting, stakeholder involvement and feedback are involved at all centers as strategies used to design and implement programs.The operationalization of the debrief differed from site to site. The debriefing is generally run by nurses but sometimes this responsibility rests with the surgeon. Participation of all members of the surgical team is an expectation in all programs. The physical form of the debrief itself is not consistent. While McLeod and Michigan use paper systems, Memorial uses an established electronic system and Madigan is working with an electronic system that is being revamped. Many of these systems rely on an administrator who categorizes the issues identified in the debrief, assigns tasks to address the issues and provides feedback regarding the plan and eventual resolution. These programs have all reported tremendous value from their programs in teamwork and communication, satisfaction and measures of efficiency and patient safety , 13, 14.All four interviewees identified executive leadership as important in both creating and maintaining programs Table . Early lthe army has to be your senior leaders… the senior leaders have to embrace it and have to continue to train it and live it”.At each site, the highest levels of administrative leadership played an important role at the outset of implementation. At Madigan and at MacLeod, executive leadership provided the initial impetus for program development, giving clinician leaders an opportunity to create a process with strong administrative support. In Florida, the executive was convinced by local physician leaders and subsequently made the program a priority. In Michigan, there was strong endorsement of a debriefing program that was established by clinician leaders. The leadership commitment required was described by Dr. Stanley Marks: ““His job had been with  … was coming in and creating a culture of safety…. I think they got up to something like 1,000 days without an injury before they had their first one which was relatively minor but it was the turnaround on their culture of safety and their injury rates, so he was quite an expert in general execution in the cultural aspects and the kind of seriousness and support that you needed from leadership when you took on an effort like this.”Although some administrators had never set foot in the operating theaters prior to the implementation of these programs of system improvement, many executive leaders had experience in other system-wide quality improvement strategies. Individual experiences ranged from experience in the flight industry, work within the commercial industry as well as direct involvement with medical quality and safety. Dr. Michael Rose described the quality improvement experience of one of their hospital board members:the ability to reach out to senior people, up to and including board members and have them engage at that level with staff as a point of emphasis on the importance of the work is part of the ‘secret sauce’ here”. In Memorial Health Care and McLeod, the ongoing direct involvement of the highest levels of executive staff remained an important part of the program long after the original implementation.The programs that remained successful all reported strong ongoing engagement of leadership. At McLeod and Memorial health, this engagement took the form of executive staff being physically present in the operating rooms and other areas where clinical care takes place. This physical presence performed two functions, 1) keeping the executive staff aware and more directly connected with the issues faced by the front-line staff, and 2) reinforcing to the front-line staff the genuine commitment of the institution to the process. Dr. Michael Rose summarized the importance of this onsite presence; “I believe that leadership, number one, mandates it and stands behind that mandate. If a member of the medical staff doesn’t take the course and cooperate with the policies, they can’t operate here.” Dr. Andrew Foster described a similar approach at Madigan “It was very important to us to have senior leaders there that said, ‘this is what we are going to do, and it really isn’t an option not to participate in this’.”An institutional mandate has helped to maintain the debriefing programs at Memorial Health and at Madigan. At Memorial Health, participation in team training was directly tied to the privileging of staff while the military structure at Madigan allowed reinforcement to be tied to military promotion and discipline. The leadership at McLeod engaged staff participation through repeated and direct communication between administration and frontline staff. Using an institutional mandate as a method of achieving participation was described by Stanley Marks; “Developing a culture of safety and using meaningful feedback to generate a pattern of continuous improvement are common themes that arose during the interviews Table . Creatina group of people who felt they had the least amount of respect, the least amount of ability to influence events, but once released and empowered, had a huge impact”.Each of the leaders interviewed acknowledged that the traditional hierarchical approach to communication posed a threat to patient safety. Leveling the playing field allowed all members to recognize and report threats to patient safety and improve communication. Dr. Michael Rose described this process as giving voice to “… we have empowered our nurses and techs to say ‘hey, we’re not bringing this patient back until we’ve talked; because number one I’ve been empowered to speak up if I see a safety issue; and number two I’ll go to my leadership, I’ll go to your leadership as a matter of fact.’”Empowering nurses means providing them with support from the highest levels of administration. Dr. Andrew Foster describes the structure of support at Madigan in Washington: “Even if it was a medical student in the room, I would ask the medical student [to speak first in the debrief] and half the time they had no clue what we had done; but sometimes they might have seen something; and then it was on to the next level trainee, moving all the way up in to the most experienced people; and I usually went last.”Creating an atmosphere of open communication was often described as a responsibility that rested with the surgeon; who would invite input from all members on things that could have been done better. To emphasize that everyone’s voice was valued, the surgeon would first ask those who were traditionally less likely to speak up. Dr. Robert Welsh is a clinician leader who exemplifies this approach: “they were complimentary of the process saying, ‘if I put it in the debriefing, I know it’s going to get attended to’”. Feedback provided to the executive also provided an additional way in which leadership could be further engaged in process improvement.Feedback on the issues raised in the debrief was commonly regarded by the clinical leaders as the cornerstone to successful debriefing. Providing caregivers with early and meaningful feedback on identified problems gave the participants the satisfaction of seeing their problems recognized and prioritized, allowed repetitive issues to be addressed and gave the participants faith that the debrief achieved a purpose. Dr. Robert Welsh reported that even the surgeons who were resistant, became fans of the process: “you’ve got to continually move the minimum standard to the right side of the equation so that you narrow your bell curve…. Raising the bar continuously must become an organizational imperative”.Focusing a system on continual improvement is a strong part of creating a culture that focuses on surgical safety. The process is described by Dr. Stanley Marks: “Each of the leaders interviewed spoke about challenges that they had encountered in the process of implementing the debriefing program and challenges that pose a constant threat to ongoing performance Table .Table 3T“the new CEO… came in probably around October of 2008 and just got blindsided by [the economic downturn] in November of 2008 so his emphasis now was the bottom line… he’s trying to figure out ‘how do I keep this boat afloat?’”The loss of leadership support and resources were concrete threats to the ongoing effectiveness of these programs. In Michigan, Dr. Welsh describes the change in leadership as coinciding with a change in institutional priorities: In addition, breakdowns in communication lead to the reluctance of disempowered team-members to speak up and lead team members to disengage with the process.“I think sometimes surgeons speak one language and nurses speak another and internists speak another and if you magnify that throughout the healthcare enterprise you know maybe there’s 50 languages being spoken in a hospital, all of which is presumed to be English.”Dr. Stanley Marks provided an illustration of the wide-spread issues with communication in health care systems: “when a person feels that they don’t have voice or their voice is suffocated, I think they fundamentally lack confidence and disengage. Some will just check out, but some people will actively disengage and broadcast how the place doesn’t care about anything that happens, never does anything.”Dr. Michael Rose described the feelings of disillusionment that lead to disengagement in a system that fails to achieve an atmosphere of open communication: Providing early meaningful feedback was frequently identified as a key to team members valuing the checklist. When feedback was no longer reliably provided, leaders recognized that this represented a great threat to meaningful participation with degeneration of the debriefing and the entire checklist into a “task” or complete abandonment.“if you’re in a workforce and you’re doing critical, complicated, potentially deadly stuff and you see adverse events happening and you experience them two weeks ago and the same thing happened six months ago and you predict it’s going to happen again two months from now, you can see how you begin over time to resent it, feel cynical, feel disempowered, and that you can’t make a difference.”Dr. Michael Rose describes the dangers of the lack of meaningful feedback: Beaumont hospital faced many of the same challenges that other programs faced. Unlike these programs, the system eventually collapsed. An examination of the processes that eventually led to the program’s failure illustrates the fragility of these systems Fig. .Fig. 2Sy“They don’t see that costs avoided are worth considering as much as…“how did I negotiate a contract or save money?” or “what did I do to increase the income?”… the averted cost was just not something that they were willing to recognize. When you go to them and say ‘look every surgical site infection costs the hospital 11,000 dollars’ or whatever number you want to use, they say ‘yeah, okay that’s nice but did it make any money for us?’ and they just couldn’t grasp this.”The debriefing program was not seen as a priority to the administration in a situation of scarcity and the key position of the quality and safety nurse was gradually phased out. The nurse in this position facilitated assigning responsibility for dealing with debriefing issues and providing feedback. Once this process was no longer performed adequately, those using the checklist no longer saw the value of the process and stopped participating in this. This loss of feedback was regarded by Dr. Welsh as the key aspect that led the program to collapse: “…pretty soon, no one is collecting the data anymore, and there’s no question that if you don’t get that data and then give feedback to people, they’ll just stop giving it; they don’t see any value in it anymore.”Dr. Robert Welsh identified the financial crash of 2008 as being the catalyst for the program’s collapse. This occurred at the same time as a change in executive leadership. Although executive leadership had supported the program, their involvement was at more of an arms-length than the other programs we have examined and this leadership change further distanced the leadership from the debriefing program. The economic advantages of a debriefing program were not easy illustrated and operated based on costs saved or avoided rather than revenue generated. Robert Welsh describes how the economic impact of 2008 affected the administration’s views on revenue generation and cost-savings:With the change in the executive leadership, the voiced institutional support of the program was also missing and the clinician leaders of the debriefing program no longer had the support of the administration and the culture of safety was no longer a prevalent force.Improving surgical safety through debriefing is a complex endeavor. Although the role of debriefing is acknowledged as an important part of surgical safety, debriefing is rarely performed in institutional audits of checklist compliance , 7. We hDebriefing is well-established in many high-acuity disciplines where it contributes to a culture of communication, reflection and continuous improvement. The American Heart Association strongly promotes the use of debriefing to improve the running of codes . StudiesThere is no simple answer as to why debriefing is not adopted routinely after surgical procedures. Qualitative studies through surveys and interviews suggest multiple reasons including logistical challenges and a perceived lack of value , 8. The The lack of a strong culture of surgical safety likely contributes to the devaluing of the debriefing both by the institution and by the team members who directly participate in the care of the patient.The interviewees from each of the four sites recognized that an immediate tangible value of the checklist would best ensure that the surgical team would participate in debriefing. Often this meant addressing concerns that impacted the teams’ ability to perform its job efficiently in addition to other issues related to improving performance. The fact that issues of equipment and process could be corrected and the surgical team could experience the immediate benefits of participation, gave them a greater commitment to the process.To improve the culture of surgical debriefing and create a meaningful debriefing process, interviewees acknowledged that the institutional approach should be broad-sweeping, targeting all levels of care and administration with a strong focus on empowerment of nurses and those members of the surgical team who have traditionally not had their voices heard. To create a system in which nurses are empowered, the focus needs to switch from individual performance to team success. In most situations, this requires an institutional culture change. Changing the culture of an institution requires a comprehensive strategy in which team training may play a critical role.When quality and safety tools are applied without an institutional strategy for ongoing implementation, they are likely to fail. This can be seen in studies of surgical safety checklist that do not specifically involve implementation strategies , 21. WheThere are many threats to the ongoing success. At Beaumont Hospital in Michigan, the loss of resources was devastating to the debriefing program as key positions were defunded. It is possible that if the executive administration valued the program more either economically or within a stronger institution-wide culture of safety, economic scarcity might not have had the same impact. Stanley Marks at Memorial Health in Florida reflected that the value of the program is so engrained within the institutional culture that it is not likely to be impacted by scarcity. But maintaining the institutional culture takes work. Even in programs like Memorial Health, Stanley Marks refers to the constant threat of degeneration into a “task” mentality.The value of the debriefing systems established by four surgical centers in diverse settings across the United States is tremendous. Even in Beaumont, where the system failed, there remains a strong commitment by clinician leaders to re-establish the program. The fragility of these systems exposed by the collapse at Beaumont underlines the importance of strong executive involvement, a reliable system of meaningful feedback and prioritization of communication within a system designed to constantly improve.


Purpose: The transgender population is a small yet distinctive portion of the gynecology patient population, requiring both primary care and specialty services. Recognizing the need for increased education, the Council on Resident Education in Obstetrics and Gynecology (CREOG) developed objectives specific to the care of transgender patients. This study is to assess residency program directors' knowledge about the transgender health CREOG objectives, describe how objectives are being implemented in training programs, and identify what types of educational materials would be useful if available.Methods: In May 2014, an 11-item anonymous survey was sent through e-mail to all eligible program directors of accredited obstetrics and gynecology residency programs. The short questionnaire contained questions about program demographics, approach to training residents with regard to the CREOG objectives, and opinions on tools they would like to use to train their residents on the transgender CREOG objectives.Results: Just under half (47%) of the 86 geographically diverse respondents were from hospital-based programs. The majority reported that the transgender health objectives were important (82%); however, only 70% were familiar with the objectives themselves. Most respondents (96%) felt that providing an educational activity in their training program would be beneficial for their residents' education.Conclusions: Most program directors support the CREOG transgender health objectives and are in favor of implementing educational tools to meet the objectives, suggesting that development of new tools to meet this need would be useful. Future endeavors will be made toward build a training module to facilitate obstetrics and gynecology (Ob-Gyn) programs meeting the CREOG objectives. In a 1997 survey done by the San Francisco Department of Public Health, 35% of male-to-female transgender persons tested positive for HIV, and roughly 85% of transgender patients reported verbal abuse because of their gender identity or presentation.2 The National Transgender Discrimination Survey, a 2008 survey of 6436 transgender persons, found that at least 25% of respondents reported being harassed or disrespected in a doctor's office or hospital, and 19% were refused medical care.3 More than one-quarter (28%) of transgender individuals postponed seeking healthcare, even while ill, because of discrimination. To address these issues, half of respondents acknowledged the need to teach their healthcare providers about transgender care.The transgender population is a small yet distinct portion of the gynecology patient population, requiring primary care as well as specialty gynecologic services. An estimated 0.5–2% of the general population is transgender.4 In 2011, The American College of Obstetricians and Gynecologists also echoed the need for education on transgender identity and healthcare in a committee opinion.5Many organizations are recognizing this need and vocalizing the importance of increased education to build the skills appropriate for addressing the unique health concerns of transgender individuals. In particular, the World Professional Association for Transgender Health (WPATH) has been a vocal proponent of increased access to healthcare for transgender persons as well as improved knowledge of transgender health issues among practicing physicians.Obstetrics and Gynecology (Ob-Gyn) residency training provides a timely opportunity to address transgender health with the next generation of gynecologists, allowing them to feel competent caring for these patients when they begin practicing as independent physicians. The Council on Resident Education in Obstetrics and Gynecology (CREOG) objectives  for resiTo better understand how these objectives are used in residency programs, residency program directors were surveyed to assess their knowledge of the CREOG objectives specific to transgender health, how they implement related training in their programs, and what forms of training materials they would use if they were made available. Our hypothesis was program directors were unaware of the Transgender Health CREOG objectives and were thus not implementing training tools in their programs.This study was reviewed and approved by the University of Kansas Medical Center Institutional Review Board; appropriate informed consent was obtained from all subjects.The study population included all Ob-Gyn residency program directors practicing within the contiguous United States with programs recognized by the American College of Obstetricians and Gynecologists. All program directors were invited to participate in the anonymous survey through e-mail. The e-mail addresses were identified from the circulating listserv of Ob-Gyn program directors as of May 2014.As there is no validated survey currently in circulation to assess residency program directors awareness of CREOG objectives, a team consisting of a program director, resident physician with experience in transgender health and advocacy, and an epidemiologist were used to construct the survey . While tThere were 219 Ob-Gyn residency program directors eligible to participate at the time of the study. The survey was sent out as a link in the e-mail body. Any e-mail addresses from the initial invitation that were returned as a nonworking address were placed into a separate category, and an attempt was made to determine which programs these addresses were associated with. If that determination could be made, then an Internet search was used to ascertain if a more current e-mail could be found. A general follow-up e-mail that included the updated e-mails was sent 1 week later to all program directors to encourage participation. Two weeks after the follow-up e-mail, a final e-mail was sent to a listserv of program coordinators with the request to pass along the e-mail to their current program director to encourage further responses.A statement of release of data was issued at the beginning of the survey; the participant assented to the use of data and participation in the study by completing the survey. The survey was voluntary, and there was no compensation. The short survey was administered through Survey Monkey and consisted of 11 items regarding program demographics, the approach to training residents with regard to the CREOG transgender objectives, and opinions on tools program directors would like to use to train their residents in the CREOG objectives.Data from the survey were analyzed using SAS . To test for differences between proportions, chi-square tests or Fisher's exact tests were used.Of the 219 eligible program directors, 86 responded to the survey (39.3%). The median number of residents in these programs was 5 (mean: 6), with a median range of 4–5.5 residents across CREOG regions. Overall, more than three-quarters of programs were either university based or located in a community hospital setting with a university affiliation .p=0.21) (The majority of program directors reported awareness of the CREOG transgender objectives (72.1%). While there was no statistically significant difference by region, reported awareness appeared to be lowest in Region 2 (52.9%), compared to Region 5, where 92.9% of program directors reported awareness of the objectives . Althougp=0.21) . Among rp=0.21) . HoweverDirect patient exposure was felt to be the most desirable form of education (67.4%), however, this was not widely available. Only 45.9% of respondents stated that transgender patients were seen in their clinics, while only 8.1% stated that their clinics saw more than 10 transgender patients per year. Of those who saw transgender patients, only 36.9% reported that residents participated in transgender patient care. Overall, only 23.3% stated they utilized direct patient exposure as a learning tool. After direct patient exposure, the next type of material reported to be beneficial was an online module (58.1%).Regardless of the method of exposure, there was consensus that the CREOG objectives were important. More than 80% of respondents ranked the objectives as important, with 37.5% of those stating it was very important. Only 2.3% of program directors indicated that the objectives were not important at all. While there are some resources available that can be used to educate residents about transgender health, few program directors reported being aware of them. Of the 86 programs, only 13 were familiar with the World Professional Association with Transgender Health (15.1%), and only 26 knew about the University of California at San Francisco Center for Excellence for Transgender Health (30.2%).The CREOG objectives form the basis of a well-rounded Ob-Gyn residency graduate. Despite the recognized need for training on the healthcare needs of transgender patients, only three-quarters of participating program directors reported being aware of the CREOG objectives pertaining to the care of transgender individuals. In addition, despite 81.4% of respondents indicating that the objectives were important, roughly a quarter of respondents reported that they did not have any modality for education.While there is an opportunity to increase awareness of the CREOG transgender objectives, there are few educational tools to adequately equip program directors to provide this training. While the ideal is an increase in patient interaction, there is little direct control that programs can have over increasing their transgender patient population. In addition, residents in a program with a recognized transgender patient population may still have limited access to that population. Only 36.9% of program directors reported that their residents had access to direct patient observation. An alternate method should be sought to provide education. Online models and lectures were the next most desired tools identified by respondents. Online modules can provide standardized patients, who can be optimized to provide a variety of traits and responses that practicing gynecologists may encounter in their practice. Online modules are a reasonable way to address both the unique healthcare problems of transgender patients as well as pronoun awareness and psychosocial issues associated with being transgender. These contextual concepts may be more difficult to grasp in a text-only learning environment such as a didactic presentation. By compiling information from a variety of resources , an online module can bring together a manageable learning experience for the average Ob-Gyn residents without requiring them to seek out different sources, many of which are still only known to few programs. Only 45% of respondents knew of any organizations working with the transgender population, the majority of them only knew of the Transgender Center for Excellence.6 The use of standardized patients allows residents to err without the risk of harming a patient. This is particularly relevant for transgender patients, who make up a small yet vulnerable population, with high levels of discrimination and mental health complications as a by-product.3 Insensitivity toward this community includes outright and perceived discrimination, lack of identification boxes appropriate for their sex/gender discrepancy, inappropriate pronoun usage, and lack of knowledge on the part of physicians and their healthcare affiliates. Educating residents in a safe space, such as in a standardized patient interaction, would be beneficial and a good supplement to any patient interaction they may receive in residency. When in-person standardized patient encounters are not available, an online learning module incorporating standardized patients would be a useful tool.Many in the medical field are aware that standardized patients invariably do not represent all patients in their target demographic scenario, but most recognize the value of standardized patients in providing exposure to common traits.This survey does have its limitations. While all known program directors were invited to participate, it is possible that some were missed due to changing roles and contact information. In addition, while the response rate was 39.3%, a good response for an uncompensated survey, it was not possible to compare the respondents and nonrespondents to elucidate group differences due to the anonymous nature of the survey. Responder bias is possible as those who were familiar with, and more interested in, the CREOG objectives may have been more inclined to participate in the survey, thus leading to a higher percentage of respondents stating that they were familiar with the objectives. All program data were reported by program directors, who may or may not be aware of education regarding transgender health that is happening through unofficial faculty–resident interactions. Thus, some survey items may have been underestimated, reflecting residency programs where transgender education happens but is not a formal part of the curriculum. The survey also did not assess for the presence of institutional limitations on training residents to care for transgender patients due to such issues as institutional social ideologies. This may be a barrier to assess in future surveys.7 The transgender health CREOG objectives were created to ensure that graduating Ob-Gyn residents are equipped with the medical, communication, and cultural skills to be able to provide services to the transgender population. To fully achieve these goals, objective awareness must increase. While our hypothesis that program directors were unaware of these objectives was disproved, it does not discount the obvious need for improved educational tools for their programs. Providing these high-quality educational tools concurrently fills an identified need and ensures access to consistent educational concepts.In the Washington Transgender Needs Assessment Survey, 32% of respondents listed insensitivity/hostility to transgender persons as a barrier to accessing care, as well as a fear of having their transgender status revealed.


As the field of D&I (dissemination and implementation) science grows to meet the need for more effective and timely applications of research findings in routine practice, the demand for formalized training programs has increased concurrently. The Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) Program aims to build capacity in the cancer control D&I research workforce, especially among early career researchers. This paper outlines the various components of the program and reports results of systematic evaluations to ascertain its effectiveness.Essential features of the program include selection of early career fellows or more experienced investigators with a focus relevant to cancer control transitioning to a D&I research focus, a 5-day intensive training institute, ongoing peer and senior mentoring, mentored planning and work on a D&I research proposal or project, limited pilot funding, and training and ongoing improvement activities for mentors. The core faculty and staff members of the MT-DIRC program gathered baseline and ongoing evaluation data regarding D&I skill acquisition and mentoring competency through participant surveys and analyzed it by iterative collective reflection.A majority (79%) of fellows are female, assistant professors (55%); 59% are in allied health disciplines, and 48% focus on cancer prevention research. Forty-three D&I research competencies were assessed; all improved from baseline to 6 and 18 months. These effects were apparent across beginner, intermediate, and advanced initial D&I competency levels and across the competency domains. Mentoring competency was rated very highly by the fellows––higher than rated by the mentors themselves. The importance of different mentoring activities, as rated by the fellows, was generally congruent with their satisfaction with the activities, with the exception of relatively greater satisfaction with the degree of emotional support and relatively lower satisfaction for skill building and opportunity initially.These first years of MT-DIRC demonstrated the program’s ability to attract, engage, and improve fellows’ competencies and skills and implement a multicomponent mentoring program that was well received. This account of the program can serve as a basis for potential replication and evolution of this model in training future D&I science researchers.The online version of this article (10.1186/s13012-018-0711-3) contains supplementary material, which is available to authorized users. Awareness of the need for scientific approaches to dissemination and implementation (D&I), to provide for more effective and timely ways to translate research in practical applications, has increased over the past two decades , 2. WithThe Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) program is perhaps the first to incorporate systematic mentored training for D&I research with a focus on cancer control. It builds on experiences with D&I science training programs of varying depth and formats offered by several organizations in the USA and globally , 17, witSeveral universities offer ongoing support and training through their respective Clinical and Translational Science Award Programs , 21 as wImplementation Science Editors’ recently issued call to action to build capacity for researchers to be able to conduct D&I research [Various groups have undertaken work to develop competency and curriculum lists and expectations –16. A feresearch . In thisFunded by the National Cancer Institute, MT-DIRC is an R25 grant-funded, post-doctoral education program, aimed at building capacity among early to mid-career, cancer control researchers in D&I science through supplemental training and mentorship in D&I research. During each year of the 2-year MT-DIRC program, fellows attended a 5-day summer institute at Washington University in St. Louis to receive didactic, group, and individual instruction on their research area of interest as it pertains to D&I science. Ongoing mentoring relationships occurred over the 2 years of the program, complemented by intermittent webinar sessions with topics chosen by the fellows themselves. This paper describes the various components of the MT-DIRC program and shares preliminary results from the first two trainee cohorts. The purpose of this paper is to describe the development of the MT-DIRC program and share the experience to date to inform the development of other D&I training programs with the aim of advancing the field of D&I research within cancer control and more broadly.The program was led by Program Director, Ross Brownson, a D&I scholar who has been a contributor to other similar D&I training programs . Dr. Brownson was joined by other renowned D&I scholars Table  who togeThe MT-DIRC program sought a range of fellows from across the USA and internationally to apply for the program. In order to be considered for the fellowship, applicants must have completed a doctoral-level degree and have a full-time appointment in a research setting prior to applying. All researchers whose work focused along the cancer control continuum were welcome to apply . The proTo recruit fellows, emails were sent out through various listservs and networks related to D&I science and cancer control. Flyers were made available at the annual Conference on the Science of Dissemination and Implementation each December in Washington D.C. Faculty and fellows were also encouraged to mention the program when giving presentations on their D&I work at related conferences.In each year, every core faculty member reviewed a subset of the submitted applications, with each application reviewed by three faculty members. Faculty scored each application based on eight questions regarding overall quality, demonstrated commitment to D&I science, demonstration of experience working in trans-disciplinary networks, evidence of research support and potential, likelihood for career development, appropriate methods in concept paper, appropriate topic in concept paper, and potential impact of the work proposed. The scores were compiled and averaged with additional commentary included. All core faculty met by conference call to review scores and select fellows for that year’s cohort.Each year of the grant, MT-DIRC sponsored 12 fellows covering all costs for travel to St. Louis, MO, during the 5-day summer institute, hotel accommodations, and meals for the week as well as an additional $1000 for pilot funds to be used during the first year of the program. An additional collaboration with the Veterans Administration provided funding for two additional fellows who are VA-affiliated researchers. The Cancer Research Network also sponsored one additional fellow in the 2014 and 2015 cohorts.Figure The foundation for the MT-DIRC program curriculum lies within a set of comprehensive D&I competencies. Building on prior work to define competencies for D&I science training , 19, 27,To identify any remaining gaps within this curriculum, the MT-DIRC team conducted a concept mapping activity . ConceptStarting in 2014, 5-day MT-DIRC summer institutes have been held each year at Washington University in St. Louis. The institutes consisted of didactic sessions, one-on-one mentoring, and small group breakout sessions. The focus of the week was to address each of the identified competencies with the goals of improving D&I research skills and provide hands-on support to each fellow’s project development. Each day had a specific focus  for project development. First-year fellows were asked to bring an updated version of the two-page D&I concept paper they had submitted in their application to the program. Returning fellows were asked to bring one or more “works in progress” to focus on during the week, such as in-progress manuscripts, grants in-development, and data sets. During breakout sessions, fellows took turns presenting their work and receiving feedback from mentors and peers. Breakout sessions were also used to allow work with their assigned MT-DIRC mentors, see Additional file All MT-DIRC fellows were matched with a core faculty mentor who was committed to working with 2–4 fellows across a 2-year time frame. Fellows were matched to their assigned mentor about 1 month prior to their arrival at the summer institute. Mentoring matches were done by the Program Coordinator with input from the faculty on preference, based on shared research interests or experiences.In preparation, all faculty who served as mentors engaged in mentor training which was offered in three-stages by Dr. Christine Pfund (CP), Mentor Consultant, from the University of Wisconsin-Madison. The training included an asynchronous, self-paced component , a synchThe Mentor Consultant provided continual support on the mentoring relationships throughout the duration of the program. Mentors were expected to have at least an hour of monthly contact with each of their assigned fellows (group mentor sessions could count toward that 1-hour requirement). If this expectation was not met, fellows or mentors were encouraged to contact the Program Coordinator and Principal Investigator so the situation could be addressed appropriately. On monthly calls with the core faculty, the group was invited to share facilitators or barriers to their current mentoring relationships. The core faculty tried to troubleshoot current issues as they arose as well as to provide suggestions on how to better facilitate these relationships.On the first day of the MT-DIRC institute, the Mentor Consultant presented a 1-hour session on “Evidence-Informed Mentoring for D&I Research” which included presentations of the key elements of research mentor training, background on the mentoring competencies, and overall expectations for the mentoring relationships, which are a critical element of the program. During this session, fellows were given time to identify their expectations for their relationships with MT-DIRC program mentors. Mentors and fellows were then given an opportunity to meet for the first time and lay out expectations for the week and beyond. Mentors and fellows had face-to face contact every day during the institute, and on the final day were given the opportunity to plan how the mentoring relationship would be handled during the monthly calls and the 2 years of the fellowship.Since most of the mentoring interaction between the fellows and faculty mentors was conducted at a distance, the team brainstormed ways for the entire cohort to connect between their attendance at the two summer institutes. While there were informal gatherings at the Annual Conference on the Science of Dissemination and Implementation in Health  held in To assist fellows in securing grants, the program offered $1000 in pilot funds to each fellow during their first year in the program. These funds were intended to support pilot research with the goal of helping fellows obtain preliminary results which could be leveraged for larger D&I grant applications. Fellows also used these funds to attend and present at various conferences, attend relevant meetings pertaining to D&I science, and support the effort of research assistants or a data analyst, or for participant incentives in pilot work.To assess the on-going mentoring relationship, D&I skill level, as well as the general satisfaction with the institute and the MT-DIRC program overall, a variety of surveys were conducted throughout the year via the Qualtrics online survey platform. In addition to the various surveys, fellows’ academic outputs  were tracked over the years. The project coordinator utilized these outputs for annual NCI training grant reporting purposes and the long-term assessment of fellows’ productivity.A pre-institute survey was administered to fellows 1 month before they attended their first summer institute. This 20-min survey was used to collect pertinent demographic information, intended for grant reporting purposes as well as some logistical questions in preparation of the training institute. The survey was also used to collect baseline data on each fellow’s level of D&I skills according to the 43 competencies  the importance of eight mentoring priority areas  and their level of satisfaction with each of the aforementioned priorities being met.As of this writing, all four cohorts had completed their pre-institute survey; three cohorts had completed the 6-month post-institute survey, and two had completed the 18-month post-institute evaluation. We are reporting these preliminary data at this time to make them accessible to others attempting to replicate a similar program.t tests were used to compare mentors’ self-ratings versus fellows’ ratings of mentors’ competencies across the MCA. The analysis looked at the fellows and mentor’s initial 6-month assessment of mentoring skills. Additionally, differences in mean ratings of importance and satisfaction among the eight mentoring priority areas were explored. These measurements were taken at fellows’ 6- and 18-month post-institute survey.Using descriptive statistics, we examined the demographic composition of the four cohorts. D&I skills and mentoring outcomes were analyzed for the two cohorts (2014 and 2015 cohorts) with complete data . D&I skill gains were tested using a Repeated Measures ANOVA between the three time points  with Greehouse-Geisser’s correction utilized where assumption of sphericity was violated . Cohen’sFrom 2014 to 2017, 56 fellows participated in the MT-DIRC program. Participants were recruited nationally with the most successful recruitment tactic being word of mouth from colleagues in a previous cohort. Demographics for each of the four participating cohorts are in Table D = 2.151). The skill that saw the least change between pre- and 18 months was “Identify sites to participate in D&I studied and negotiate or proved incentives to secure their involvement” . There were improvements in all summary domains. The domain that saw the largest increase in skills between pre- and 18 months was “Definitions, Background and Rationale” . The domain that saw the least increase in skill between pre- and 18 months was “Design and analysis” .A primary goal of the MT-DIRC program is to increase the skills of D&I research across a range of competencies identified by this program. Results indicate that fellows have had a statistically significant improvement across all 43 targeted competencies and that these gains are sustained at the 18-month time point. Mean skills gain score at the baseline, 6- and 18-month time points, grouped by competency domains are presented in Table D = 2.10)  Table .Table 5CMentoring is a critical element of the MT-DIRC program, and therefore, the mentoring skills of the core faculty as well as the mentoring relationships were evaluated on several levels. Table μ = 4.50), but fellows initially indicated a mismatch in rating their satisfaction with this need being met lower than other domains . However, by the 18-month point, there was alignment in ranking of fellows’ importance and satisfaction place on skill building. By the 18-month point, most of the factors had improved their ratings in terms of satisfaction to fellows and were more closely aligned to the initial fellows’ perceived importance in that mentoring area. Across the board, mentors and fellows indicated that dedicated time for each other was their biggest barrier to these relationships.Alignment of fellows’ priorities and their satisfaction of having those priorities met was assessed using five-point Likert scale about the mentoring priorities and satisfaction. This alignment was assessed at both the 6- and 18-month marks to explore how importance and satisfaction changed over time. As shown in Fig. Overall, early results from the MT-DIRC program show fellows are improving in their D&I skills and mentoring needs of the fellows are being met. Findings, discussed below, highlight the impact of the program and provide guidance for similar model programs. The emphasis on mentoring and skill development throughout the course of the program we believe has been integral to its success. However, the team has aimed to keep the program flexible to meet the needs of each cohort and the individual fellows and do not believe that all programs must contain every component in order to be successful. Rather, those interested in creating similar programs can utilize these results to inform their decision-making and help shape the content and format for their programs.All D&I skills showed significant improvement from the time of entering into the program to the 6-month post-institute, with further improvements 18-month post-institute. While fellows showed improvement from their time in the program just within the first 6 months, there was continued improvement in skill levels over the subsequent 12 months suggesting that participants were using the skills they learned at the institute over time which led to continued growth in these areas.Since the program was aimed at early-career researchers or mid-career researchers looking to switch their research focus into D&I science, it is not surprising that the largest area of improvement for skill levels were previously identified beginner skills  than their fellows rated them, consistent with previous research . This fiWhile this is a formal mentoring program, mentoring contracts with fellows were not required (though they were encouraged and a template was provided) nor were meeting duration or structures stipulated. The only requirement was that mentors connected with each of their fellows at least once per month, for a recommended time of 1 h. Mentors were given the choice to hold phone calls or skype calls (very few could arrange face to face meetings) and whether they wanted to conduct group calls with all their fellows and/or individual calls. These elements of flexibility may suggest that perceived capabilities of the mentor may vary based on the expectations set out from the initial contact between the mentor and fellow pair. As noted in other research, codifying the mentoring relationship roles, in a way in which calls upon fellows to also be active participants in articulating their needs, may enhance the quality of the relationship and improve perceived success of the relationship .μ = 4.50) coupled with the lower satisfaction among fellows (μ = 3.81). This alignment levels out by the 18-month mark, suggesting that some skills and priorities may take more time to develop and conceptualize than others. However, during the 18-month assessment of satisfaction, the mean score of each priority area shifted upward, indicating a general positive shift in all areas of the mentor priority areas. So, even if there was no exact alignment of mentoring areas between priority and satisfaction, overall, the fellows indicated higher levels of satisfaction with their mentoring toward the end of their time in the program.The questions in the Mentoring Priorities and Satisfaction section of the survey showed mismatches among the specific priority areas of skill building and sharing resources  or learning styles.The results from this program are valuable in demonstrating the effect and need for such training programs. The experience with MT-DIRC over the past few years provide a sound understanding of the landscape and a solid infrastructure that can be carried forward to fit with the new training models supported by sponsoring agencies. Results like these demonstrate promising evidence of the effectiveness of such mentored training programs and support the funding of such programs in the future. It is the hope that those who intend to replicate D&I trainings can apply many of the principles outlined to develop and evaluate related training programs.In the first several years of the Mentored Training in Dissemination and Implementation Research in Cancer program, we were able to demonstrate the effects of such a mentored approach to training program as evidenced by the increase in skill level over time and the satisfaction that fellows have demonstrated with their mentoring and the program overall. We hope that by disseminating our findings and lessons learned, other groups can replicate and improve upon this model in order to continue to train the next generation of D&I science researchers.Additional file 1:Sample summer institute agenda. (PDF 587 kb)Additional file 2:MT-DIRC D&I skills 2014 and 2015 full data. (PDF 440 kb)