This study aimed to recognize the gaps in dental education by studying the current level of geriatric oral health training of recent graduated dentists who have been admitted into an Advanced Education in General Dentistry (AEGD) program.The AEGD program was developed along with the Age-Friendly 4Ms model to enhance current dental education. We adopted the Rapid Cycle Quality Improvement model to test the effectiveness of the training for AEGD residents from 2019 to 2022. A total of 18 residents participated (6 residents each year). A 5-question survey was administered before and after the rotation and Wilcoxon signed-rank with Fisher Exact tests were conducted to compare pre- and post- rotation results.p = 0.011). Meanwhile, residents gained knowledge to apply the 4Ms framework  to their practices (p = 0.015) and provide age-friendly care for older adults.All 18 residents have completed pre- and post-program surveys. They self-reported minimal to no training in preparation to provide care to older adults with multiple chronic conditions. After the rotation, residents’ confidence in treating older adults was significantly increased (The study identified and addressed the missing link in dental education to gerontological and geriatrics education. More clinical rotations and didactic training to equip residents with competences of providing geriatric oral health are strongly recommended. Baby boThe predoctoral Commission on Dental Accreditation (CODA) Standards 2–25 requires graduates to be competent in assessing and managing the treatment of patients with special needs such as the vulnerable elderly . HoweverThe objective of this study is to evaluate the educational competency of AEGD residents as an approach to recognize the missing link between the current dental education and the real-world demand for geriatric oral health professionals. The program trains six residents every year for three consecutive years. After one-year of training, we measured the effectiveness of clinical rotation and didactic experience including Age-Friendly practices. The hypothesis was that our AEGD program could advance every resident’s competence in geriatric oral care, and our program would lay a successful foundation to address the identified missing link for design of future PGD training.2.2.1Matters, Medication, Mentation, Mobility. In response to the growing aging population, HRSA provided funding to educational, community-based, and primary care institutions to improve the geriatrics workforce that are vital to deliver care in an age-friendly practice  aiming to prepare the next generation of age-friendly practitioners . The Agepractice .The AEGD program included didactic (three 4 h seminars) and four weeks of clinical education. Three seminars consisted of reviewing the process of natural aging, stereotypes and ageism, and the changes in the function of organs. During the seminars, faculty and residents also discussed in very detail how chronic health conditions may further impact the body as individuals live longer with chronic conditions. The third topic was about medications that may raise concerns in oral health such as opioids, anticholinergic, anti-psychotic drugs, as well as drugs used for diabetes and heart disease. Additionally, through clinical case reviews, the AEGD residents learned the most common chronic conditions in older adults and their impacts on oral health. Each resident was then assigned to a four-week clinical rotation in two Community Health Clinics (CHCs), one nursing home, and one memory care facility, respectively.2.2The program was evaluated each year for three consecutive years. The assessment tool was the same for the first two years but was modified in the third year to better evaluate residents’ level of knowledge and skills.2.2.1The first group of six AEGD residents received a 5-question survey to establish a baseline for evaluation their knowledge in treating older adults with multiple health complexity. Due to the interruption of COVID-19, the residents were not able to have any hands-on clinical activities. However, they received 12 h geriatric education didactically and remotely every week. The educations were focused on science of aging, stereotypes and ageism, the impact of polypharmacy, and the physical and cognitive changes may have negative impact on oral condition. The clinical education was limited to clinical case presentations within the 4Ms framework, discussing treatment of choice and choice of dental materials, anesthetics and other person-centered clinical decisions based on each person’s needs and what Matters most to them.2.2.2The second group of 6 residents participated in the same 5 question survey except this group had a short window of opportunity to treat older adults as it was planned in two CHCs, a community nursing home and a memory care. Each resident rotated in all three locations per week and the overall rotations took one month. After RCQI analysis of the second year, we noticed that the 5-question survey is not sufficient to evaluate clinical learnings with an emphasis on health conditions and the 4Ms. It was also noticed that weekly rotation to different sites does not offer sufficient continuity to see the outcome of treatments. Therefore, we assigned each resident to 4 consecutive weeks (one month) of participation in geriatric oral health education.2.2.3The third group of six residents were enrolled to the program and participates in 10-question survey before and after the rotation. With the previous evaluation result, the project team added another five questions to the survey to become a 10-question survey. In year 3, all six residents had the entire duration of AEGD program to complete geriatric education and clinical rotation. Five additional questions allowed the project team to better evaluate the clinical advancement of the learners and the future application of the 4Ms in practice.2.3The AEGD program is a one-year post-graduate residency program to advance training in general dentistry. To be eligible for this study, residents must have graduated from an accredited dental school in the U.S or were accepted into the AEGD program as international residents. The program recruited six residents in each of 2019–2020, 2020–2021, and 2021–2022 and a total of 18 residents responded to pre- and post-rotation surveys. The study has been approved by the UTHealth Institutional Review Board.The purpose of this study is to assess and evaluate the education and clinical skills of the AEGD residents, before and after the rotation. All residents participated in a 5-question survey before any exposure to the content of the program and the same 5-question after completion of rotations. The third cohort (2021–2022) of six residents has responded to five additional questions before and after the rotation. For questions 1, 7, 8, 9, and 10, the responses range from 1 = strongly disagree to 5 = strongly agree. For questions 2, 3, 4, 5, and 6, the responses range from 1 = strongly agree to 5 = strongly disagree. In all questions, a higher grade indicates that residents’ knowledge and confidence have been improved after the rotation. Residents’ personal identifiers were removed, and only aggregated outcomes were reported in this study.2.4N = 18). Mean score, standard deviation, and p-value were presented in p ≤ .05. The project has been approved by Institutional Review Board (#229293) of the UTHealth Houston.A five-point Likert scale was used to rate each resident’s response to survey questions. We linked each resident’s responses to pre- and post-rotation surveys and then conducted Wilcoxon test with Fisher Exact modifications that better addressed our small sample . Although there is no statistical significance in changes in other questions, the mean scores were increased in all questions.p = 0.346). Q5 has the highest post-rotation score and the largest variation from 2.78 to 4.22 (p < 0.05).p = 0.491). Q7 has the largest variation from 2.00 to 4.33 (p < 0.05). Considering these findings, more training about extraction of weak teeth and geriatric oral health is highly recommended.4.4.1The AEGD program as a post-graduate training was intentionally designed to enable residents to provide treatments based on the 4Ms model to meet oral health needs of older adults . The stu4.2.These positive improvements in residents’ competence are critical because as discussed earlier, over 22 million older adults will need special care by 2030 due to medical vulnerability, chronically being in poor health, and decline in quality of life . In addiMedically vulnerable elders need special protection and different treatments from general population . Consequ4.3There is no standardized training to guide faculty members to establish a residency program integrating geriatric care and evaluate the effectiveness of such a program . The pro5.Recent general dentist graduates who enrolled in our AEGD program from various dental institutions (within or outside of the U.S) felt they are not prepared to treat older adults with multiple chronic conditions. Our study demonstrated the improvements in their confidence and knowledge to provide age-friendly care. Strengthening dental education in both didactic and clinical skills to assess and treat older adults with complex needs is highly recommended in response to the increasing number of older adults. Strengthening the CODA standard and faculty capacities to establish more geriatric oral health programs can also help close the gap.


The Good Life with osteoArthritis in Denmark (GLA:D®) program was implemented in Canada in 2017 with the aim of making treatment guideline-recommended care available to the 4 million Canadians with knee and hip osteoarthritis (OA). This report describes the GLA:D® Canada program, registry and data collection procedures, and summarizes the sociodemographic and clinical profile of participants with knee and hip OA to inform the scientific research community of the availability of these data for future investigations and collaborations.The GLA:D® program consists of three standardized components: a training course for health care providers, a group-based patient education and exercise therapy program, and a participant data registry. Patients seeking care for knee or hip OA symptoms and enrolling in GLA:D® are given the option to provide data to the GLA:D® Canada registry. Participants agreeing to provide data complete a pre-program survey and are followed up after 3-, and 12-months. Data collected on the pre-program and follow-up surveys include sociodemographic factors, clinical characteristics, health status measures, and objective physical function tests. These variables were selected to capture information across relevant health constructs and for future research investigations.At 2022 year-end, a total of 15,193  participants were included in the GLA:D® Canada registry with 7,527  and 2,798  providing pre-program data. Participants were 66 years of age on average, predominately female, and overweight or obese. Typically, participants had knee or hip problems for multiple years prior to initiating GLA:D®, multiple symptomatic knee and hip joints, and at least one medical comorbidity. Before starting the program, the average pain intensity was 5 out of 10, with approximately 2 out of 3 participants using pain medication and 1 in 3 participants reporting a desire to have joint surgery. Likewise, 9 out 10 participants report having previously been given a diagnosis of OA, with 9 out 10 also reporting having had a radiograph, of which approximately 87% reported the radiograph showed signs of OA.We have described the GLA:D® Canada program, registry and data collection procedures, and provided a detailed summary to date of the profiles of participants with knee and hip OA. These individual participant data have the potential to be linked with local health administrative data registries and comparatively assessed with other international GLA:D® registries. Researchers are invited to make use of these rich datasets and participate in collaborative endeavours to tackle questions of Canadian and global importance for a large and growing clinical population of individuals with hip and knee OA. Over 300 million individuals worldwide have symptomatic knee and hip osteoarthritis (OA)  and the There is no known cure or disease modifying therapy for knee and hip OA. First line management  is therefore primarily focused on symptom mitigation and improving function in OA. When non-surgical management approaches fail, total knee or hip arthroplasty may be the only recourse for those with end-stage knee or hip OA. In addition to symptomatic control, the goal of current management approaches is also to possibly delay or prevent the need for arthroplasty through non-surgical treatment options , 9. PatiTo address the lack of available guideline-recommended non-surgical care for knee and hip OA, the Good Life with osteoArthritis in Denmark (GLA:D®) program was developed by researchers at the University of Southern Denmark . GLA:D® GLA:D® Canada was created and piloted in 2015, making it the first country outside Denmark to implement the program. The feasibility of implementing GLA:D® in Canada was demonstrated in a pilot study , and ledThe purpose of this report is to present the GLA:D® Canada registry and participant cohort to the Canadian and international OA research community by describing data collection procedures, health and outcome measures collected, and summarizing the sociodemographic and clinical profile of participants, with the overall intent to encourage collaborative research using these available data.This study report was a cross-sectional analysis of registry data. This report conforms to the STROBE statement for reporting observational studies . Ethics Like all international GLA:D® programs, the GLA:D® Canada program for knee and hip OA consists of three standardized components: a training course for health care providers, a group-based patient education and exercise therapy program, and a patient data registry . An overThe group-based education and exercise therapy program consists of two patient education sessions  and 12 supervised exercise sessions, delivered twice a week . OverallThe clinician-supervised exercise sessions consist of a neuromuscular exercise program (NEMEX) aimed at promoting stabilization of the affected joints to help reduce pain and improve function , 36. Dethttps://gladcanada.ca/).Participants are eligible for GLA:D® if they have knee or hip joint problems as a result of OA that are sufficient in intensity to seek care in the health care system. Potential participants can access GLA:D® via referral from other health care providers or by self-referral to private clinics. There is no standardized method of how potential participants are informed about the program due to the varying provincial jurisdictions. Participants may be informed about the GLA:D® program by the general practitioner, physiotherapist, other patients, and/or via the GLA:D® Canada website  joint problems related to other conditions ; and 2) other symptoms that are more pronounced than OA symptoms . These eligibility criteria are used by all international GLA:D® programs for OA , 30. TheOnce participants are deemed eligible for the program, they are offered the ability to provide baseline and outcome data. Patients who consent to provide data are enrolled in the registry by the clinician. Patients who consent to provide data agree to make their data available for program evaluation and research purposes through the GLA:D® Canada registry. However, consent to provide data to the GLA:D® Canada registry is not required for participation in the GLA:D® program.GLA:D® clinicians register participants in the GLA:D® Canada registry and input information on the clinic where the program will be administered, participant index joint (knee or hip) for which they are seeking care, participant sex, health card number , and conduct the objective functional tests (described below). All other data in the GLA:D® Canada registry are collected via electronic surveys sent directly to participants via the Data Driven Outcome System (DADOS). Electronic pre-program surveys are sent to patients at the time of registration by the GLA:D® clinician (prior to beginning the program) and then 3- (coinciding with the approximate end of the program) and 12-months after the pre-program survey date. Changes to the original 2016 electronic survey were made in 2019 and 2022 to include additional questions and measures.Two tests, the 30-second chair stand test (repetitions) and 40-meter walk test (collected in seconds and converted to meters/second), are conducted by the GLA:D® clinician prior to the participant starting the program and written results are provided to the participant to be inputted into their electronic pre-program survey. These objective measures of physical function are recommended for use by the Osteoarthritis Research Society International as performance-based tests for people with knee and hip OA .2]); 3) current marital status (married/living with partner/single/divorced or separated/widowed); 4) highest level of education (elementary school/high school/trade or community college/university); and 5) current employment status (working full-time/working part-time/disability leave/unemployed/retired/other).Prior to starting the GLA:D® program, participants provide data on a number of sociodemographic factors, including: 1) age; 2) height and weight , including cardiovascular disease, high blood pressure, high cholesterol, lung disease , diabetes , kidney disease, liver disease, anemia or other blood disease, stomach or intestinal diseases , depression, cancer , and rheumatological diseases  are collected and free-text fields are also available for participants to identify any other comorbidities not captured in the list. Clinical characteristics related to the index joint are collected, including: 1) duration of joint symptoms (years); 2) additional symptomatic knees and hips (yes/no for each knee and hip joint); 3) comorbid back pain (yes/no); 4) previous injury to the index joint (yes/no); 5) previous index joint surgery (yes/no); 6) desire for index joint surgery (yes/no); 7) current physical activity level (days per week); and 8) pain medication use (yes/no).Knee or hip pain intensity in the last week is assessed using the Numeric Rating Scale (NRS), scored from 0 (no pain) to 10 (worst pain imaginable) . Knee- oSix new questions and two additional measures were added to the pre-program survey in 2019. At the time of participant registration, GLA:D® clinicians are also asked to provide the payment source (private/public) for the patient. In the electronic patient survey, participants are also asked: 1) previous OA diagnosis by a health professional (yes/no/unsure); 2) currently waitlisted for surgery (yes/no); 3) problems walking due to their knee/hip problem (yes/no); 4) work leave in the past year due to knee/hip problem (yes/no); and 5) pain frequency in knee/hip . Participants are also asked to complete the University of California at Los Angeles (UCLA) Activity Scale measuring their level of physical activity (1 low to 10 high)  and the Two new questions related to previous imaging of the index joint were added to the pre-program survey in 2022. Participants are also asked: 1) previous radiograph of knee/hip (yes/no); and 2) if yes, radiograph showed OA (yes/no/unsure).During the final exercise session, GLA:D® clinicians repeat the 30-second chair stand and 40-meter walk tests and give the written results to the participants to enter in their 3-month survey. These objective physical function tests are not recollected at the 12-month time-point. Other pre-program measures are recollected in both the 3- and 12-month surveys, including: pain NRS, KOOS-12 or HOOS-12, EQ-5D-5L, and ASES-8. The UCLA Activity Scale, OKS, and OHS were also added to both follow-up surveys in 2019. Other clinical questions such as pain medication use and desire for joint surgery are also asked in the follow-up surveys.Participants are asked a set of additional questions related to patient experience during the program (i.e. not included in pre-program survey). In the 3-month survey only, participants are asked their level of satisfaction with GLA:D®  and willingness to pay thresholds for GLA:D® ($100 or less to $301 or more). In both the 3- and 12-month surveys, participants are asked their level of benefit from the program  and how often they use what they have learned in GLA:D® . Since 2022, a 7-point scale comparing their current knee/hip problem to before the program (1 much better to 7 much worse) is also included in the 3- and 12-month survey.Patients in the GLA:D® Canada registry from program inception (2017) until 2022 year-end were included in this cross-sectional analysis of pre-program data. The number of participants who did not complete the pre-program survey were calculated but not included in the analysis. Data was accessed on 03 January 2023 for this analysis. The number of participants enrolled by year and proportions by geographical region since program inception were calculated. Pre-program characteristics of knee and hip participants were described separately. Proportions were reported for dichotomous and categorical data. The mean and standard deviation (SD) were reported for normally distributed continuous data. The median and inter-quartile range were reported for non-normally distributed continuous data. All data analyses were conducted in R version 4.2.1 .15,193  participants registered in the GLA:D® program from inception in 2017 to 2022 year-end . ParticiThe profile of patients entering the program since inception are presented in The average pain intensity (pain NRS) rating is 5 out of 10 for both knee and hip participants. Mean scores for knee- and hip-related pain , function , and quality of life , health-related quality of life (EQ-5D-5L Utility and VAS), and self-efficacy in managing their OA were likewise similar for participants with knee and hip OA. Knee and hip participants could complete an average of 12 rises in the 30-second chair stand test and had an average walking speed of 1.3 m/s on the 40-meter walk test.Pre-program characteristics for the new data captured effective 2019 and 2022 are presented in The GLA:D® Canada registry comprises one of the largest datasets with a comprehensive set of characteristics and measures from a primary care cohort of patients with knee or hip OA seeking care in a nation-wide patient education and exercise therapy program. Here we have provided detailed information on the data collection procedures and participant profiles from this large cohort, with a goal of encouraging research collaborations that will make use of these data to advance OA research and improve OA care.Our findings show that GLA:D® Canada participants with knee or hip OA are remarkably similar to participants in the GLA:D® Denmark and Australia programs , 30, 34.The GLA:D® Canada registry was designed with the intent to address OA research questions that consider patients or the impact of a program in a real-world clinical setting. It is the hope that making these already collected data available to the larger research community will foster opportunities for primary and collaborative OA research, for trainees to established investigators, with the ultimate goal of improved care and quality of life for the millions of individuals living with hip and knee OA. There are many examples of the use of international GLA:D® datasets by researchers undertaking OA-related investigations , 48–53, The lack of specific eligibility criteria in GLA:D®, namely the lack of requirement for radiologically-defined OA, can raise questions about participants truly having knee or hip OA. Despite the weak correlation between patient symptoms and imaging findings reported in the literature and international consensus that radiographic evidence is no longer needed to diagnose knee or hip OA , 56–58, The strengths of a real-world and national patient cohort registry come with some trade-offs. The program is administered in and registry data are collected from numerous clinical settings, unlike a highly controlled research setting. This may give rise to concerns about treatment fidelity, selection bias due to voluntary enrolment in the program and registry, and the equal application of patient eligibility and enrolment criteria across all clinical sites. Likewise, completion of the data collection surveys is not mandatory, although highly encouraged, for participation in the GLA:D® Canada program, which could lead to suboptimal data completion. For example, approximately 1 in 3 GLA:D® participants in this cohort did not provide pre-program data once registered and enrolled. Unfortunately, no data is available to enable comparisons of those who do and do not complete the pre-program survey. For the same reasons, missing data is also a risk at follow-up time-points. A thorough program evaluation would help to clarify potential concerns and highlight areas for improvement in the implementation process. Despite these limitations, the GLA:D® Canada registry offers the unique opportunity to conduct research on a large sample of patients managed in real-world clinical settings.The GLA:D® Canada registry contains detailed data on participant profiles, across a wide range of measures, for two large cohorts of patients seeking care for knee or hip OA, respectively. Patient enrolment continues in these programs and treatment outcome data are also available. Through their broad availability, researchers are invited to make use of these rich datasets and participate in collaborative endeavours to tackle questions of Canadian and global importance for a large and growing clinical population of individuals with hip and knee OA.S1 Checklist(DOCX)Click here for additional data file.


To address issues related to suboptimal insight in outcomes, fragmentation, and increasing costs, stakeholders are experimenting with value-based payment (VBP) models, aiming to facilitate high-value integrated care. However, insight in how, why and under what circumstances such models can be successful is limited. Drawing upon realist evaluation principles, this study identifies context factors and associated mechanisms influencing the introduction of VBP in stroke care.Existing knowledge on context-mechanism relations impacting the introduction of VBP programs  was summarized from literature. These relations were then tested, refined, and expanded based on a case study comprising interviews with representatives from organizations involved in the introduction of a VBP model for integrated stroke care in Rotterdam, the Netherlands.Facilitating factors were pre-existing trust-based relations, shared dissatisfaction with the status quo, regulatory compatibility and simplicity of the payment contract, gradual introduction of down-side risk for providers, and involvement of a trusted third party for data management. Yet to be addressed barriers included friction between short- and long-term goals within and among organizations, unwillingness to forgo professional and organizational autonomy, discontinuity in resources, and limited access to real-time data for improving care delivery processes.Successful payment and delivery system reform require long-term commitment from all stakeholders stretching beyond the mere introduction of new models. Careful consideration of creating the ‘right’ contextual circumstances remains crucially important, which includes willingness among all involved providers to bear shared financial and clinical responsibility for the entire care chain, regardless of where care is provided. Healthcare systems around the world are currently facing the challenges of suboptimal (insight in) outcomes , fragmen4A factor that is widely considered as contributing to these issues, are fee-for-service (FFS) payment systems that are used across many healthcare systems . These sAn increasingly applied form of VBP is bundled payment (BP). Instead of paying providers separately for each discrete care service provided (as in FFS), BP comprises a single, prespecified amount for providers assuming accountability for all services related to a certain medical condition, over a certain period. Ideally, BP covers all care that is necessary for treatment and management of the condition, regardless of where and by which provider care is provided when multiple providers are involved. To prevent a one-sided focus on efficiency and spending reduction, BP programs often also contain additional pay-for-performance incentives for high-quality outcomes. Although there is some evidence suggesting that BP has the potential to save costs while at least maintaining quality 141516, c14151718It is well-established that, for a variety of reasons, the introduction of BP in practice is highly complex 21. This 20Drawing upon realist evaluation principles, the goal of this study is to identify context-mechanism relations that facilitate or inhibit the introduction of an ongoing BP program in stroke care. This program, labelled PAying for Value in IntegrAted Stroke care (PAVIAS), was introduced on January 1st, 2019, in Rotterdam, The Netherlands. The program entails a BP contract with routine collection of patient-relevant outcomes and two-sided risk sharing between a large health insurance company and multiple healthcare providers , aiming to facilitate and financially stimulate value improvement and integrated care delivery for ischemic stroke patients. Given the background and knowledge gaps described above, an in-depth analysis of the introduction of this program is expected to yield valuable insights and lessons because the program was successfully introduced and covers both short-term hospital care and longer-term rehabilitation care provided by multiple providers involved in the stroke care chain. Based on literature-informed interviews with directly involved stakeholders, we present an overview of context-mechanism relations with respect to the introduction of this BP program in stroke care and formulate key lessons for current and future (V)BP programs.In this study we were particularly interested in providing information on how an outcome  might generate different outcomes under different circumstances. Therefore, we drew upon the principles of realist evaluation (RE). In contrast to other forms of theory-driven evaluations , RE focuGiven our objectives, RE was selected as the suitable framework for this study. Drawing on RE principles, we investigated context-mechanism relations that influenced the outcome ‘introduction of the PAVIAS program’. This outcome was defined as the VBP contract having been signed and the program having commenced. Specifically, it refers to the stage where the program has been initiated on both an intra and inter-organizational level among providers. In this stage, stakeholders aim to establish data sharing systems, implementation of payment mechanisms and coordination among different healthcare professionals like care coordinators and clinicians.Appendix 1.To develop an initial understanding of the factors and mechanisms that influence the implementation of VBP programs, we first conducted a literature review on VBP implementation. The focus of this narrative review was to identify various CMO configurations regarding the introduction of VBP programs . The review informed the design and contents of an interview guide and enabled us to contextualize our qualitative findings. Combining the terms value-based payment and implementation (and synonyms or strongly related keywords), we identified and synthesized key findings from thirteen included articles. From these articles, a total of fifteen CMO configurations were identified. Six articles focused on VBP in general (N = 6), five focused specifically on the introduction of BP (N = 5), and two concerned pay-for-performance (N = 2). A detailed description of the literature review and the identified CMO configurations is provided in The primary objective of this study was to examine the PAVIAS program through a case study approach. By conducting interviews with representatives from all stakeholders involved, we aimed to identify the contextual factors (C) that influenced the introduction of this program (O) and understand the mechanisms (M) by which these factors operated.Appendix 2. In total, thirteen non-public  documents with information on the program’s goals, bundle definition, stakeholders involved, allocation of financial risk, and collection of data on outcomes and costs were obtained and reviewed [We used documentation obtained from PAVIAS’ stakeholders to provide a detailed description of the program. This description is provided in reviewed . SubsequAppendix 3) using the CMO configurations derived from literature , aiming to expand, refine and revise these configurations during the interviews.We created an interview guide  and respondent reaching a consensus on the interview goals, defining key terms, and clarifying the respondent’s perceived role in the PAVIAS program and its introduction. The respondents were then asked open-ended questions about the introduction process, factors that facilitated or hindered the introduction, and the reasons behind them. Questions focused on the perceived positive or negative aspects of the program’ introduction and were asked about the most important factors, main barriers, how these barriers were partially overcome, and lessons learned. All interviews were conducted in an end-to-end-encrypted video-meeting using Microsoft Teams software. The average interview duration was 50 minutes (range 40–70 minutes).The audio-recordings were transcribed verbatim and thematic analysis was performed on the transcripts. The lead author coded the interview data, which resulted in an initial list of 42 codes each referring to a context-mechanism relation that was believed to have impacted the introduction of the PAVIAS program. This list was then discussed and adjusted in several meetings with four authors, eventually leading to 28 codes each representing a specific CMO configuration. Finally, these codes were pragmatically grouped in six overarching themes based on similarity in mechanisms triggered by context factors. For example, the theme ‘Trust, relations, and support’ contains context factors that triggered mechanisms related to feelings of shared commitment, fear, and/or scepticism.Following the coding process, all respondents were approached for a member check . Specifin and Mn.This section discusses all context-mechanism relations that were identified as having been influential during the introduction of PAVIAS , albeit on a coarse level. Shared goals were described as: striving towards higher value of stroke care through defragmentation, improvement of interprofessional communication, and contributing to the value-based healthcare (VBHC) evidence base. Motivations among stakeholders were driven by profound feelings of frustration with the current situation (M1) in which progress was perceived to be hampered by the predominant FFS payment system. Relatedly, they were experiencing a shared sense of urgency for change (M2). Furthermore, most respondents (n = 7) noted that the potential reward was perceived as being worth the time and effort (M3). As one respondent outlined:Across stakeholders, It was not only a personal drive to improve care. In general, the scientific substantiation for value-based healthcare is thin. In that regard I am willing to contribute to innovative programs like this. It is important that we contribute to science by doing so.” – Respondent 5“substantially different views on how to achieve and operationalize these shared goals (C2), which appears to have contributed to perceived tension between short and long-term goals (M4) and demotivating conflicts of interest that undermine a shared rationale (M5). For example, concrete plans or agreements on how to improve value were absent or ill-defined in advance, although sometimes – as mentioned by two respondents – this was a deliberate strategy to prevent delay by too much focus on specific goals about which reaching consensus may be difficult . When asked whether goals were concretized prior to introduction, one respondent replied:Although stakeholders generally had similar overarching goals, according to the respondents they had “Yes and no. Defining goals is an iterative process. I think it is naïve to assume that the specific stakeholder goals align. But I think it is realistic to assume that aggregated goals are aligned, and that should be emphasized. Whenever things get more concrete, misalignment becomes more likely. It’s a process of interaction.” – Respondent 6Trust, relations, and support). An example is the explicit goal of the insurer to limit spending, while some providers (n = 2) wanted to spend more to improve care. Remarkably, all respondents except the representatives from the insurer expressed that they did not see a (short-term) financial benefit from participating in the program because stroke patients are seen as an unprofitable population, or they expected to incur more (short-term) costs due to allocation of resources needed for introducing the program.Nevertheless, the fact that specific goals or operationalizations thereof were not always shared may have led to conflicts among or within stakeholder organizations  due to limited patient volumes and time required to make significant changes to healthcare delivery . Triggered mechanisms that negatively impacted motivations in this respect were perceived tension between short and long-term goals (M4) and scepticism about whether meaningful change could be realized in a reasonable time (M6).Some respondents (n = 3) expressed lacking evidence on positive effects of VBP and limited experience with integrated payment hardly affected their motivations to contribute to the program (C4); as noted by the respondents, BP-contracts such as PAVIAS are new to the Dutch healthcare system and evidence from other countries is likely to have limited applicability in the Dutch context. Mentioned mechanisms were again profound feelings of frustration with the current situation (M1) and a shared sense of urgency for change (M2). As one respondent summarized:For most respondents (n = 9), “I had zero experience with VBP, and others had very little. However, this program was one of a kind anyway.” – Respondent 9motivational leadership of individuals from different organizations (C5). Such leadership entailed setting deadlines, showing clear dedication to meet these deadlines, and persuasion of other people to introduce the program, all of which bolstered the feeling of having a shared commitment to make the program work (M7).When asked about the factor(s) that contributed most to the introduction of the PAVIAS program, most respondents (n = 7) mentioned the existence of good historic working relation and pre-existing trust among stakeholders with a good reputation (C6) was a crucial contributor to the introduction of the program. As a result, stakeholders felt comfortable in making investments (M8) and experienced a feeling of ‘being in it together’ (M9). As one respondent explained:All respondents acknowledged that “An important element was pre-existing trust. The Rotterdam Stroke Service, for example, already exists for 25 years. We have been working together intensively for a long time and it was not the first time we were gathered around the table when we conceptualized this program. We all expressed a desire to do this together.” – Respondent 5Strong organizational support (C7) was an often-mentioned facilitator, although some respondents representing the hospital added that more pro-active support could have prevented delay. As a result of the perceived support, stakeholders felt comfortable in making investments (M8) and had limited fear of (severe) repercussions during trial and error (M10). As one respondent exemplified:“Management made it possible by not blocking anything, although I think that things would have gone quicker if the board would have had an attitude like ‘we back your plans, and we will make efforts to expedite the process.”’ – Respondent 9conflicting interests between and within organizations (C8). They noted that this contributed to scepticism about each other’s motives (M13) and perceived suboptimal inter- and intra-organizational relationships (M14), which shifted focus away from shared goals. As explained by one respondent:All respondents did mention some degree of “In our organization, one board member is responsible for VBHC whereas another is responsible for IT or finance, while you need all those disciplines at the table. Unfortunately, that proved to be very hard due to differing degrees of priority given to the program by the different board members.” – Respondent 9lack of a shared responsibility for (the costs of) all care in the bundle (C9). Some stakeholders (n = 4) only considered responsibility for care delivered by their own organizations, whereas others (n = 3) stressed the importance of joint responsibility for all care in the bundle, including care provided by other organizations. Mentioned mechanisms were a high perceived importance attached to autonomy (M13), a perceived loss of control over responsibilities (M14), and professional obstination (M15). One respondent noted:In addition, there appeared to be a “An important factor for this program to be successful is that you must let go of some autonomy to bear shared responsibility. You must be willing to compromise.” – Respondent 5the decision to make an outline agreement  was beneficial (C10). This was mentioned to limit the perceived complexity and enhance experienced control over the program (M16). As one respondent explained:The introduction of the program was experienced to be complex. Some respondents (n = 3) noted that in dealing with this complexity, “An important lesson I’ve learned is that too much discussion about financial and contractual details may be a cause of failure for such programs.” – Respondent 10the choice for a multi-year contract with no accountability for financial losses in the first year (C11) was mentioned (n = 6) as a contributing factor. This reduced reluctance and uncomfortable feelings of being exposed to too much risk from the outset (M17) among stakeholders. As one respondent described:Additionally, “For the first year we agreed there would be no shared losses if outcome measures were collected and reported. Accountability for losses would go into effect in a later stage. I think that such ‘phasing’ contributed to mitigating reluctance among providers.” – Respondent 4the degree of financial risk under the program varied heavily among stakeholders (C12). In turn, the experienced potential benefit of participating in the program may not have been perceived as being worth the effort to a similar degree by all stakeholders (M3). Depending on the extent to which this is the case, stakeholders might lose or gain interest in the program (M18).Although the reluctance to take on financial risk was generally low , respondents (n = 3) did mention that compatibility of the BP model with the existing FFS reimbursement rules and billing system facilitated the introduction of the program (C13). Compatibility in this context means that the existing FFS architecture was left intact and that FFS claims made during the year would be retrospectively reconciliated with the virtual bundle price to determine savings or losses. The fact that the contract could be executed under existing payment regulations limited the perceived complexity and enhanced control over the program (M16) One respondent summarized this as follows:Respondents from the involved insurer (n = 2) noted that “Our principle was that this program should be compatible with the current reimbursement system. I truly believe that letting that principle go would be a recipe for disaster. Because of this, complex interventions, such as standardization of financial systems among stakeholders were not necessary.” – Respondent 1privacy and anti-trust legislation as a barrier, especially with respect to data exchange among competing organizations (C14). Mentioned mechanisms were the perceived complexity and loss of control over the program (M16), the reduced experienced possibilities for care coordination among stakeholders (M19), and scepticism about the possibilities for improving care (M20) for which free exchange of data is deemed crucial. This barrier was partly overcome by involving a trusted third party (TTP) . One respondent summarized:In contrast, all respondents viewed existing “It is very bothersome that we must adhere to rules that don’t benefit patients. I get why legislation and regulations exist, but these are insufficiently geared towards healthcare trends and coordinating care around patients” – Respondent 5the degree to which resources were made available and the level of leadership was generally proportional to the size of the respective stakeholder organizations (C15). Respondents mentioned that this contributed to feelings of fairness (M21) and perceived equality in workload (M22).According to all respondents, a lack of continuity in personnel and project groups (C16) as a barrier leading to delays. Examples are people in key positions leaving to other employers and premature disbandment of project groups without follow-up. Mechanisms triggered by this factor as mentioned by the respondents were insufficient perceived support and cooperation (M23) and feelings of demotivation (M24). As one respondent noted:However, multiple respondents (n = 6) identified “A clear barrier was that employees come and go during the introduction of such a program. Every time that happens you must bring new people up to speed. That significantly delayed the process.” – Respondent 6Insufficient human and financial resources frustrating effective program management (C17) was also mentioned (n = 3) as a barrier. Mentioned mechanisms were a high perceived workload (M25), feelings of stress (M26), and feelings of dissatisfaction (M27). One respondent remarked:“Every healthcare professional is already trying hard and cannot spare time to work on this program. That could be achieved by reorganizing and making one person responsible for a certain task, but that sort of creative thinking is not happening yet. – Respondent 9involvement of a TTP for data management (C18) as a contributing factor. Three reasons were provided for this. First, it reduces the perceived likelihood of data manipulation. Second, a TTP partly overcomes regulatory issues such as exchange of sensitive personal data . Third, the TTP assisted in overcoming the challenge of defining shared and standardized quality and financial metrics, which were deemed crucial by all stakeholders. Triggered mechanisms were reduced perceived complexity and increased control over the program (M16) due to centralized data management as well as confidence and trust in the validity of data (M29).Six respondents mentioned the In this study we identified and analysed context-mechanism relations that influenced the introduction of a VBP program in integrated stroke care in Rotterdam, the Netherlands. Using literature-informed semi-structured interviews with representatives from key stakeholders, 18 context factors and 28 related mechanisms were identified. Most context-mechanism relations found in literature were also identified by at least one interview respondent to some degree. Below, we discuss the key findings and derive lessons for the introduction of future VBP programs.Several factors clearly contributed to the program’s introduction. First, the good pre-existing working relations and trust among stakeholders were identified as important contributors. The intensive collaboration required for cross-sectoral VBP programs such as PAVIAS requires a solid foundation for stakeholders to feel comfortable with investing in payment and delivery system reform. This factor was also identified in previous studies as a crucial determinant of the success or failure of VBP implementation 3031. A s30Third, respondents highlighted the decision to build the new payment model on the existing FFS architecture as a key factor that likely has prevented many demotivating issues and delays that would be involved with replacing the current payment and billing system. This factor was also often mentioned in the literature, sometimes even as having contributed to the failure of VBP programs 14212232333435.323334Several key inhibiting CM-relations are also worth discussing. Although these apparently did not prevent the eventual introduction of the program, they did cause issues and delays, and might hamper future success if not addressed. First, although all stakeholders are willing to take on financial risks, reaching agreement on financial-risk sharing remains an unresolved issue mainly due to differences in the proportion of stroke-related revenue relative to total revenue. This has negatively impacted a balanced interest in the program, with stakeholders with a larger proportion potentially opting-out due to too high perceived financial risk relative to other participating providers. This issue was also often-mentioned in literature, though without insight in related mechanisms 142022303133.223031In contrast to the findings of prior work 2135373839,353738Our study yields several key lessons for VBP reform involving collaboration between multiple provider organizations, particularly in the field of stroke care but likely also for other conditions.prior to introducing a new payment model. This is expected to significantly increase stakeholder acceptance and comfort in making joint investments, as well as assist in respecting each other’s (often differing) motives and interests.First, good trust-based working relations between all intended contract-partners are ideally established Second, defining clear goals for the short and long run  among and within stakeholder organizations is important. This may prevent future conflicts of interest and demotivating discussions which could ultimately result in program failure.Third, involving a TTP for data management is advisable. Although a TTP is unlikely to be able to match the benefits of a fully integrated electronic health record , it can assist in the collection of data that are trusted by all stakeholders, overcoming potential legal issues regarding data exchange, and standardization of quality and financial metrics.prior to introduction. Such data would increase the likelihood of the bundle price accurately reflecting the costs of the current standard of care as it can be based on the most recent data. Additionally, it would better enable rigorous evaluation of the impact on outcomes and spending.Fourth, to enable representative bundle contents, it is recommended to accumulate several years of patient and financial baseline data A fifth lesson is that long-term commitment to the program of all involved organizations is crucial. Stakeholders should explicitly assign a high priority to the program, which includes showing willingness to allocate sufficient resources to it, forgo some organizational and professional autonomy, and accept shared responsibility for spending and quality outcomes beyond their full control.Sixth, in designing the payment contract it is advisable to allow time for providers to adapt to integrated payment and bearing of financial risk, for example by a ‘soft’ replacement of FFS using a retrospective payment methodology and without downside risk in the first year(s).Finally, payment and delivery system reform are clearly not finished after signing a contract. It is crucial to acknowledge that additional steps and considerations are necessary for successful reform. For instance, in the current contract, the decision was made to exclude primary care as a domain due to the complexity already involved with the existing number of stakeholders. While too many variables could potentially lead to failure of VBP programs, the inclusion of primary care is desirable for future expansion. Therefore, successful reform requires long-term commitment from all stakeholders, in which healthcare professionals ultimately have a key position. This requires time, resources, a constructive regulatory environment, and inspiring leaders as well as continuous efforts in making progress explicit, which is crucial for keeping professionals engaged in realizing the goal of increasing value for patients.and related mechanisms regarding the introduction of a VBP program for multiple care provider organizations. Insight into how context impacts complex interventions and through which generative mechanisms is valuable to better understand the causal path to certain outcomes. Another strength is that we drew upon realist evaluation principles and used literature on the introduction of VBP programs for in-depth interviews with representatives from all relevant stakeholders, which allowed us to provide a comprehensive picture of influencing factors and mechanisms.A key strength of this study is that it is one of the very few that examined both contextual factors introduction of the PAVIAS program. Further research focusing on uncovering CM-relations that impact its further implementation and success in terms of changes in patient outcomes and spending is required to assist providers and policymakers further in realizing successful payment reform. Second, although we believe our results provide valuable insights for (future) VBP programs for stroke as well as other conditions, the generalizability of our findings to other (inter)national settings is uncertain. Third, the reliance on retrospective responses from the sampled group of respondents may have biased our results due to their (shared) perception of success and the influence of reflecting with hindsight on their experiences. Future research could explore the perspectives of individuals who were not initially involved in the development and introduction of the VBP contract – such as patients, managers, clinicians, and other caregivers – but who are affected by it in practice. Finally, we acknowledge that the inherently subjective nature of defining and delineating contextual factors and generative mechanisms may to some degree have impacted the validity and reliability of the identified C-M configurations. In this paper we used the definition that context refers to observable surrounding conditions and factors that influence the implementation of an intervention, while mechanisms represent the unobservable underlying processes and reasonings through which the context factors result in the outcome.However, several limitations should also be mentioned. First, we only focused on CM-relations regarding the Several important preconditions and facilitators were in place that aided the introduction of a value-based payment program in integrated stroke care. Among the most important factors were good pre-existing and trust-based working relations, a strong motivation for change among all stakeholders due to shared dissatisfaction with the status quo, motivational leadership to keep everyone engaged and committed, simplicity and regulatory compatibility of the payment contract, and the involvement of a trusted third party for data monitoring and management. Despite substantial barriers both within and between stakeholder organisations, these did not prevent the program’s introduction.Nonetheless, going forward several issues will need intensive attention if the program is to fulfil its promise of facilitating integrated high-value stroke care. These issues include friction among stakeholders caused by tension between short- and long-term goals, unwillingness to let go of some professional or organizational autonomy, discontinuity in available financial and human resources, and limited access to real-time data for effective feedback and input for improving care delivery processes. Finally, and most crucially, all providers should be willing to bear shared financial and clinical responsibility over the entire stroke care cycle, regardless of where care is provided.This study brought together data obtained upon request and subject to restrictions from several different sources. The database is not publicly available due to the  sensitive nature of the data.The additional files for this article can be found as follows:10.5334/ijic.7566.s1Appendix 1.CMO configurations derived from literature.10.5334/ijic.7566.s2Appendix 2.Integrated care case description.10.5334/ijic.7566.s3Appendix 3.Interview guide.


In the United States (US), many obstetrics & gynecology (OB-GYN) trainees feel unprepared to care for lesbian, gay, bisexual, transgender, queer, and intersex (LGBTQI) individuals, but interest in this topic is rising. Residency program websites are one way that directors can advertise whether this training is offered within their program. We aimed to describe the extent to which LGBTQI content is currently featured on OB-GYN residency websites across the country.We identified all OB-GYN residency programs in the United States using a publicly available database. We systematically searched for select LGBTQI keywords on program websites. We collected data on mentions of LGBTQI didactics and rotations. We also searched whether LGTBQI keywords were included in diversity, equity, and inclusion (DEI) statements. We used multivariable logistic regression to compare the characteristics of programs that do and do not include this content. We used STATA SE Version 16.0 for all analyses and set the level of significance at 5%.We included 287/295 US OB-GYN residency programs in our analysis (97.3%) and excluded 8 that did not have websites. We identified any LGBTQI content on 50 program websites (17.4%), and specific mention of didactics or rotations on 8 websites (2.8%). On multivariable analysis, programs in the West were more likely to include any LGBTQI content compared to programs in the South , as were programs with 1 or 2 fellowships  or 3 or more fellowships  compared to those without fellowships. Programs in departments led by female chairs were also more likely to include LBTQI content .Academic programs, West Coast programs, and those with departments led by female chairs are more likely to mention LGBTQI keywords on their websites. Given the increasing interest in LGBTQI education for OB-GYN trainees, program directors should consider providing training opportunities and including this content on their websites.The online version contains supplementary material available at 10.1186/s12909-023-04624-3. Lesbian, gay, bisexual, transgender, queer, and intersex (LGBTQI) individuals account for approximately 7.1% of the population in the United States . Many faOne recent study found that less than half of board-certified OB-GYNs reported having received any training in LGBTQI health during residency . Other sThis increasing interest in expanding LGBTQI-related education and training in OB-GYN residency is a promising first step in preparing OB-GYNs to care for this patient population. Medical students who are applying to residency and desire training in LGBTQI health-related issues can use program websites to learn whether programs offer training in this area. Residency websites are publicly accessible and may reflect training priorities and institutional values. During the COVID-19 pandemic, the residency interview process radically changed, as virtual interviews replaced travel to in-person interviews. In this context, online informational material about programs is likely to play an even more important role in the residency selection process.We undertook a review of United Stated-based OB-GYN residency program websites to determine the proportion of programs that include a mention of training in LGBTQI health on their website. We also compared the characteristics of residency programs that do and do not explicitly mention training in LGBTQI health.This is a cross-sectional analysis of all OB-GYN residency program websites in the United States. We systematically examined each residency program website from November 1st to November 22nd, 2021, searching for specific LGBTQI keywords. We excluded programs with no existing website from the analysis.We identified all OB-GYN residency programs in the United States using the American Medical Association’s Residency & Fellowship Programs Database (FREIDA). Two researchers then examined each program website separately to search for LGBTQI content. A third reviewer resolved any conflicts or discrepancies between reviewers. We searched the website home page and all additional sub-pages using the following LGBTQI-specific keywords: LGBT/LGTBQ/LGBTQI, transgender, trans health, queer, lesbian/gay/bisexual. We selected similar keywords to those used in existing research assessing LGBTQI healthcare content online .Using our keyword searches, we determined whether programs websites had any mention of LGBTQI health in any portion of the website; mention of LGBTQI-specific rotations or didactics, mention of LGBTQI health in the program’s diversity, equity, and inclusion (DEI) statement, or other mentions of LGBTQI health. We also collected quotes from the websites that reflected this content.We collected various program characteristics such as location  and program type  from FRIEDA . We alsoWe created a dichotomous variable for any mention of LGBTQI health  versus none (primary outcome). An example of the “other” category is mention of LGBTQI research. We also created a dichotomous secondary outcome that only included mention of LGBTQI health didactics or rotations, to ensure that we had specific information about training.We conducted descriptive statistical analyses of variables reflecting program characteristics. To compare the characteristics of programs that did and did not include LGBTQI content, we initially used the Chi Square or Fisher’s Exact tests. To examine the predictors of having any LGBTQI content on the program website (primary outcome), and of having LGBTQI training , we constructed two multivariable logistic regression models. To do so, we initially included all variables with p < 0.1 on univariable analyses. We then build multivariable regression models using a stepwise, backward selection approach. We first included all covariates with p < 0.25 on bivariate analysis and then sequentially removed covariates with the highest p value until all variables were significant.We conducted all analyses using STATA Version 16.0 and set the significance level at 5%.This research study does not involve human subjects. As such, the New York Medical College Institutional Review Board deemed the study exempt from review.We identified 295 OB-GYN residency programs through the FREIDA database. Of these, we excluded eight programs that did not have a website. Only 50 17.4%) program websites included any LGBTQI content. Only six program websites (2.1%) mentioned LGBTQI-related didactics, 18 (6.3%) mentioned LGBTQI-specific rotations, 28 (9.8%) contained LGBTQI content in their DEI statements, and 16 (5.6%) had mention of LGBTQI content in another context . Almost half of programs were university-based n = 124, 43.2%), while some others  were community-based university-affiliated. Most programs did not have any religious affiliation declared on their websites . Approximately two thirds of program directors were female , while only 28.6% (n = 82) of department chairs were female. Table 24, 43.2%On univariable analysis, region (p = 0.037), program type (p = 0.009), website extent (p = 0.009), number of residents p = 0.019), number of fellowships (p = < 0.001), sex of program director p = 0.023), and sex of department chair (p = < 0.001) were significantly associated with mention of LGBTQI health [Table 3, and se9, numberTable On multivariable analysis, the only significant predictor of mentioning LGTBQI didactics or rotations specifically (secondary outcome) was number of fellowships. Programs with one or two fellowships had 12.1 times the odds of including this content compared to those with no fellowships , while programs with 3 or more fellowships had 18.72 times the odds .In this review of LGBTQI content on US OB-GYN residency program websites, we found that less than a fifth of OB-GYN residency websites had any content related to the health of LGBTQI communities. In order to understand how these findings may relate to education opportunities, we also looked at the mention of LGBQI specific rotations and didactics, which were even more rare. Although residency program websites do not necessarily reflect all training opportunities, they are important for applicants who are deciding where to train, as has been demonstrated in other specialties , 11. OurWe also found that certain program characteristics are associated with mentions of LGBTQI content on program website: program region (West Coast particularly), number of fellowships, and female sex of the department chair. We interpreted the number of fellowships as a marker of academic status. These results are new to the literature, but they are not surprising. A review of online directory for gender affirmation surgery found that a disproportionate percentage of surgeons providing these services practice in the West (30.5%) and Northeast (25.6%), despite this geographic distribution not matching the distribution of the patient population that would benefit from these services . A natioOur study has several limitations. First, we are unable to assess how website content correlates with actual training experiences. It is difficult to establish how many programs offer LGBTQI training because this may not always be advertised on program websites. Residency websites may also be outdated and may not reflect new training opportunities. In a 2016–2017 survey of OB-GYN residency program directors, 51% stated that their program offered transgender health education or training. However, only 61% of those invited to complete the survey did so, and it is possible that nonrespondents were less likely to offer this training . If thisAnother limitation is that we conducted our search at a specific point in time and our findings may not reflect changes that have occurred since we completed data collection. Finally, we did not assess programs’ social media platforms (such as Instagram or Facebook) for LGBTQI-related content. A study of ENT programs found that higher ranked programs are more active on social media, have more followers, and adopt social media earlier . AnotherOur study also has several strengths. First, our approach is novel as to our knowledge there have been no systematic assessments of US OB-GYN residency program websites. Second, we systematically reviewed all residency programs in the United States using established keywords. Lastly, we collected all the data in a short time period, and were therefore able to obtain a snapshot of all US programs at a specific point in time.In conclusion, we found that LGBTQI content exists on a small minority of US OB-GYN residency program websites. Currently, all OB-GYN residency recruitment efforts are virtual as means to improve access to residency interview opportunities for applications. Therefore, medical students applying to residency can use websites to learn whether programs offer training in LGBQTI health. Many applicants are using websites to understand and reflect on program and institutional values. The lack of LGBQTI health training mentioned on program websites as demonstrated by our study may reflect a lack of interest within the institution to highlight this training, even if it is available. In order to better serve the LGBQTI and recruit residents interested in providing healthcare to this community, programs may consider marketing their complete educational experiences and more thoroughly documenting their training experiences in LGBQTI on websites to aid in recruitment efforts. Future studies may use other aspects of training experiences within the residency program  as a control group and compare to our findings in order to account for the possibility that some aspects of our findings may be due to marketing limitations and strategies of residency program websites. More importantly, future studies could focus on determining the importance of LGBQTI training to residency applicants, how residency website content correlates with training experiences, and on describing the role of social media platforms in the residency recruitment process.Below is the link to the electronic supplementary material.Supplementary Material 1


Introduction: The coronavirus disease 2019 (COVID-19) pandemic forced international faculty development programs in medical education to forgo in-person activities and transition to online learning. We sought to examine changes in international medical educators’ evaluations of our faculty development program as it transitioned due to the pandemic.Methods: We compared survey responses from participants in our International Medical Educators Program between 2019 (in-person) and 2020 (online). The 43-item survey addressed aspects such as program evaluation and self-assessment of curriculum development and teaching skills. We analyzed data using t-tests to compare means and chi-square test for categorical variables, and performed thematic analysis of open-ended responses.Results: We found that trainees in both cohorts rated the program highly with regard to overall program quality and self-assessed learning outcomes, but the 2019 group reported stronger relationships with peers and instructors. Some scores for self-assessed outcomes were lower for the 2020 class, but no statistically significant differences were found in pre- and post- training scores between the two cohorts. Four themes emerged from the feedback: positive program utility, IMEP as an example of good curriculum design, timing issues, and online learning environment challenges.Conclusions: Despite pandemic challenges, the transition to online faculty development was favorably evaluated, with high confidence in the applicability of learned skills. Future efforts should focus on fostering community and optimizing interaction times to enhance learning experiences. The study contributes insights for global medical education communities in pandemic circumstances. With social distancing as the most effective way to prevent transmission of infections in lieu of a vaccine, in-person activities were abandoned in favor of remote learning when possible, and medical schools have had to adapt students’ learning experiences based on the availability of PPE and local transmission rates. Likewise, faculty development programs have had to forgo in-person activities and transition to online learning.The coronavirus disease 2019 (COVID-19) pandemic dramatically altered the landscape of medical education worldwide in 2020. These issues were exacerbated by uneven institutional support, influenced by educators’ individual technological literacy, inadequate resource planning, and usability problems like screen resolution and audio quality. Despite these hurdles, perceptions of online teaching demonstrated a broad spectrum: while less than half of educators and learners from one study thought online training could match the quality of traditional face-to-face courses, a significant portion saw its potential beyond the pandemic. Notably, this shift also highlighted emerging pedagogical opportunities, as educators called for increased support in formats such as online lectures, collaborative work, live broadcasts, and chats.This abrupt transition to online learning brought a set of considerable challenges. Educators grappled with delivering clinical skills, teaching remotely, and handling large groups of students in online settings, the degree of participation from trainees with online workshops, and the ease of bringing faculty from widespread regions together. However, the impact of the online transition of international faculty development programs on training outcomes has not been discussed.Literature on the pandemic’s influence on faculty development in medical education is limited. Some studies have addressed facets of online education like reduced resistance from faculty to online programs. This collaborative effort culminated in the creation of the International Medical Educators Program (IMEP) in 2018. IMEP was designed to provide Wuhan faculty members with an immersive two-week training program in medical education at the University of Chicago campus. However, due to the emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in Wuhan and ensuing pandemic in 2020, this on-site training program was suspended, and the program was quickly transitioned to an online format delivered throughZoom.The University of Chicago, United States, has provided assistance for faculty development for Wuhan University, China, in the reformation of its medical curricula since 2008The transition to a virtual platform in 2020 necessitated a few adaptations to our training program. We managed to keep the core of the program consistent with the 2019 version, except for the unavoidable exclusion of hands-on topics such as simulations and teaching rounds observations. However, the consistency of the program was upheld as the instructors and their respective course content remained the same across cohorts. To accommodate the 13-hour time difference and ongoing responsibilities of physician educators during the pandemic, we restructured our schedule to only hold 1–2 one-hour sessions a week. Consequently, the total interactive face-to-face time was reduced from 60 hours to 28 hours.The purpose of this study was to examine changes in international medical educators’ perceptions of one faculty development program as it transitioned from in-person to online due to the COVID-19 pandemic. Specifically, we aimed to analyze program quality and self-assessed training outcomes to provide insights applicable to global medical education communities.th August 2009 (IRB09126B). The University of Chicago Institutional Review Board approved an amendment to the study on 1st September 2022 (IRB09126B). Signed consent and oral consent were waived.The Wuhan University Health Science Center first approved this study on 12IMEP began as a two-week intensive program at University of Chicago, with an aim to improve curriculum development and teaching skills for international medical educators. However, the necessity for social distancing and the challenge of a 13-hour time difference led to the transition into a 15-week program. Given the reduced interactive time, we adapted the curriculum development section of the program into video modules and homework assignments, which allowed participants to progress at their own pace outside the confines of group sessions. To facilitate discussion and feedback on participants’ individual curriculum development projects, the group convened on Zoom every two weeks.Initially, IMEP used a broad range of teaching methods including lectures, observation, and interactive experiences. The advent of the pandemic led to the removal of interactive simulation and observation experiences from the program. However, the extension of the program to 15 weeks enabled longitudinal mentorship for individual curriculum projects. Our lecture topics encompassed a wide array of subjects, including learning theory, competency-based medical education, the one-minute preceptor approach, providing feedback, PowerPoint design, lecture skills, role modeling, and reflection. These lectures were delivered via Zoom for a duration of one hour, followed by a 15-minute breakout session for small group discussions.To ensure smooth communication and understanding, applicants underwent an English proficiency screening before matriculation.DocumentStar, a Chinese survey platform. The survey was not tested before the study. The survey was conducted on 19th July 2019 for the 2019 cohort and from 20th December 2020 to 22nd February 2021 for the 2020 cohort. Reminders were sent by the program administrator via group text and email, until the response rate on DocumentStar reached 100%. Participants’ email and ID for texting were obtained when they registered for the training program. The survey asked participants to rate 36 statements pertaining to program evaluation and self-assessment of curriculum development and teaching skills . Self-assessment items included both retrospective pre- and post-training assessment. The remaining items asked for demographic information and open-ended responses. We defined academic seniority as associate professor or professor. The survey can be found asExtended data .Upon completing the program, all trainees from the 2019 and 2020 IMEP cohorts were provided a link via text message to a 43-item anonymous survey hosted onWe used paired sample t-test and independent sample t-tests to compare means and chi-squared test to compare categorical variables. The data were analyzed using Stata MP 16 for Windows . We also conducted a thematic analysis of the responses to open-ended questions, which involved an iterative process of reading and coding the responses to identify themes.et al., 2022). Demographic characteristics are inThe survey response rate for IMEP both in 2019 and 2020 was 100% (10/10 and 19/19 respectively) ."IMEP as an example of good curriculum design: A theme emerged emphasizing IMEP as an example of good curriculum design. The trainees commended the balance of educational theory and practical techniques, and they valued the application of these concepts to their own professional contexts. Participants described IMEP as a "standard model of future curriculum development" and as a "beacon of effective curriculum design". One trainee encapsulated these views, saying, "The training program itself shows us how to design courses, implement courses, choose teaching strategies, evaluate, etc. which is very helpful for us to understand the design of medical courses."Timing concerns: Many comments pointed to issues related to timing of the classes, suggesting it affected their ability to fully engage with the material and discussions. Seven participants expressed issues with late-night classes due to time-zone differences, especially after the end of daylight savings time. For example, one trainee wrote, "too late at night because of the time difference."Online learning environment: Participants had mixed feelings about the online learning environment. While some appreciated the necessity of it due to pandemic circumstances, others highlighted challenges, including issues with engagement and focus. A few participants expressed a desire for more interactive and practical components, and one commented on the inefficiency of online group discussions. One participant noted, "being online is a little bit hard to focus, especially when we are still being bothered by a lot of things in the clinic." Another wrote, “All the professors gave us wonderful talk although on-line course is lack of interaction between instructors and learners (sic).”The COVID-19 pandemic led to the unforeseen transition of our international faculty development program from an in-person format to an online environment. This study explores the less examined area of the pandemic’s influence on international faculty development in medical education, and offers critical insights for the wider medical education community.. However, it is worth noting the lower scores may indicate a trend that needs monitoring.In spite of the shift to an online platform, participants rated the overall program quality favorably, echoing the high evaluations from the previous year’s cohort. Trainees indicated that the program’s educational content and instructional methods met their needs. Their confidence in using curriculum development and teaching skills was further demonstrated by high scores across all 13 self-assessment items. Although some scores in the 2020 class were slightly lower than those of the 2019 class, the comparison of pre- and post-program scores revealed no statistically significant differences. These findings are consistent with existing literature suggesting minimal or negligible differences between online learning and traditional approaches, and these findings underscore the importance of further optimizing online teaching methods to facilitate interactive engagement, foster meaningful relationships, and stimulate future collaborations among participants and instructors. Despite its importance, building virtual communities of practice is seldom discussed in health professions and deserves more focus.The analysis of trainees’ feedback also revealed certain challenges within this new format. Trainees’ perception of the strength of their education community was weaker in the online group compared to the in-person group. This was particularly evident in the lower ratings given to the quality of relationships with instructors and prospects for future collaboration with other trainees (statements 6 and 8) in 2020. Some individuals also expressed in their open-ended responses a desire for a higher degree of interactivity and engagement. While the 2020 cohort of medical educators maintained confidence that they would be better educators due to the training program, they were slightly less confident compared to their 2019 counterparts. These insights highlight the potential challenges of fostering an online learning environment that encourages peer-to-peer and mentor-mentee interactions. Intentional community building is increasingly seen as an integral component of successful longitudinal faculty development, yet the pandemic did not allow for this. Nearly all of our 2020 trainees preferred training to be in-person, whether in China or Chicago. In addition, over one third of trainees thought classes were too late, even though there was only one feasible time slot for face-to-face interaction given the time zone difference. Community was even more difficult to achieve with overall face-to-face time halved from the previous year. For subsequent training years that are virtual as well, we plan to allocate more face-to-face time from lecture to discussion, and assign group projects to encourage more interaction and mutual learning.Logistical considerations may also have affected the cohesion of the group. Although the socially distanced virtual setting allowed us to accommodate more trainees with ease, it posed other challenges. Some of the most successful international models use a short in-person period to foster community growth before moving online for the rest of the year.Another consequence of the pandemic was that we increased the duration of the program to 15 weeks, which allowed participants more time to process training content, as well as receive longitudinal mentorship for their individual education projects. However, this did not translate to increased perception of learning for any of the survey items compared to 2019, and participants were just as confident in implementing their education projects as the previous year. Notwithstanding the self-assessment data, we will be following up with trainees to evaluate the quality of the projects after they are implemented and compare differences between the cohortsBased on our experiences with the International Medical Educators Program (IMEP), we outline a set of best practices for transitioning an international faculty development program to an online format:1. Consider Time Zone Differences: Adjust the schedule to consider the time zone differences for international participants to maximize active engagement.2. Increase Interaction: Implement more interactive and engaging activities to foster active learning, even in a virtual setting. This might include group discussions, breakout rooms for small group work, and interactive quizzes or polls during sessions.3. Foster Community: Implement strategies to foster a sense of community among participants, despite not being physically present. This might involve setting up virtual social events, allowing for casual conversation before and after sessions, and using collaboration tools for group projects.4. Asynchronous Learning: Make use of asynchronous learning tools and resources, such as recorded lectures, discussion boards, and learning management systems. This can supplement synchronous sessions and accommodate participants in various time zones.5. Flexibility: Be flexible and ready to adapt the program as needed, based on feedback from participants, changes in public health conditions, and any technical challenges that might arise.6. Evaluate and Iterate: Regularly collect and analyze feedback from participants to understand their learning experience, identify any issues or challenges, and make necessary improvements to the program.By implementing these practices, we believe that international faculty development programs can successfully transition to an online format, maintaining high levels of learning, engagement, and community, even amidst challenging circumstances of social distancing such as the COVID-19 pandemic.A major limitation of this study is that it only includes data from a single faculty development program, and validity of the data is limited by small sample sizes. Additionally, the variety of changes to the program due to disruption from the pandemic limits the interpretation of results in other contexts. Another limitation was that the 2020 course was not modified to address new skills required of medical educators, such as the use of online teaching tools and online communication skills.The COVID-19 pandemic forced this international faculty development program to transition urgently from in-person to online. Overall, trainees were satisfied with the program and rated learning as high as the previous year, although they reported weaker relationships with instructors and peers. When planning international faculty development programs during a pandemic, educators need to consider optimizing limited interactive time due to time zone differences, while also fostering a sense of community. For future virtual iterations of IMEP, we will continue using asynchronous materials to supplement our face-to-face virtual sessions, but we plan to increase time for discussion and assign group projects to help build community among our trainees and instructors. The authors have corrected the missing parts now.Have any limitations of the research been acknowledged?NoIs the work clearly and accurately presented and does it cite the current literature?PartlyIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?NoIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?PartlyAre sufficient details of methods and analysis provided to allow replication by others?NoReviewer Expertise:Medical education, program evaluation, contextualization of programsI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The authors have made the amendments based on the comments.Have any limitations of the research been acknowledged?YesIs the work clearly and accurately presented and does it cite the current literature?PartlyIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?YesAre the conclusions drawn adequately supported by the results?YesAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:1. Obstetric and gynaecilogy 2. Community 3. Women health 4. Midwifery 5. Nursing EducationI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The work, while done very meticulously, offers very limited generalizability of the results. The study question needs to be revised for this international portal and to suit its current readership.  The section under methods that describes the training program looks more like a context description that I think would better suit the introduction piece and leave this section for an actual description of how the program is currently conducted. The work is an evaluation study of a faculty development program in its new form that has been shifted to online and has extended its duration for the sake of compliance. I was looking for a governing criteria that individualized this program and that can be taken as a take home message for readers who are embarking on designing their own faculty development program but unfortunately I could not find this.  In order to make this piece better utilizable by readers I wonder if a section on limitations and best practices can be added for example.Have any limitations of the research been acknowledged?NoIs the work clearly and accurately presented and does it cite the current literature?PartlyIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?NoIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?PartlyAre sufficient details of methods and analysis provided to allow replication by others?NoReviewer Expertise:Medical education, program evaluation, contextualization of programsI confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above. The inclusion and exclusion criteria for the study need to be discussed here.Write the implications of the study.Write the strengths of the study.How do the findings add to the body of scientific knowledge on the issue?References 8, 9, 10, 13, 14,and 15 are more than 10 years old, it is better to replace them with updated references.Have any limitations of the research been acknowledged?YesIs the work clearly and accurately presented and does it cite the current literature?PartlyIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?YesAre the conclusions drawn adequately supported by the results?YesAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:1. Obstetric and gynaecilogy 2. Community 3. Women health 4. Midwifery 5. Nursing EducationI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Although the design of the study and the tool used (and the high response rate) would all be appropriate under normal circumstances, as a result of the COVID pandemic and the rapid switch to online teaching and pivot in how students’ learn even in the clinical environment, many new challenges were faced in medical education. Some of these are outlined in these recent publications.et al. (2022).Chanet al. (2022).Liu et al. (2021).Motte-Signoretet al. (2022).Bastoset al. (2021).Herrmann-Werner   There are a number of limitations which aren’t mentioned.Use of technology and online teaching platforms/toolsOnline communication skillsCreation of online learning resourcesSynchronous online teaching and online use of learning activities What isn’t stated is whether the same content was delivered in the 2020 online course, or whether it was modified to address the new skills required of medical educators. These include the following: Also, the trend for all scores for both the General feedback and the Curriculum development skills statements, was that they were lower in 2020. In the general feedback, there were significantly lower scores for statements 2, 6 and 8. and For Curriculum development skills statements, 5 statements, there was a significant difference. It is quite telling that the lowest response was for the statement around formation of meaningful relationships (the next was plans for collaboration), which may reflect a failing of the transition to this online professional development program. There was also a significant lowering of the score for the self-confidence in delivering a lecture that helps students learn effectively. This is only discussed tangentially in relation to poorer community building. It is impossible to speculate on the causes for this without knowing whether the content has been changed at all or how well skilled the facilitators of the new online course were themselves in the skills listed above. In 2020, many of us were still using new techniques for the first time. This may well have affected the delivery of this course. This study would benefit from free text qualitative feedback if this was collected. They mention that the students found the timing difficult and that they preferred face to face, but only in the discussion, not as a thematic review of qualitative data (if this was done). Some pluses of this mode of delivery, such as enabling bigger cohorts. this is mentioned, but in this group it looks as if the same number (29) were enrolled. So, this needs to be clarified. This size may well require a different approach, such as the creation of smaller online breakout groups to encourage interaction and enable the creation of ‘community’ and more meaningful relationships with colleagues.Have any limitations of the research been acknowledged?PartlyIs the work clearly and accurately presented and does it cite the current literature?PartlyIf applicable, is the statistical analysis and its interpretation appropriate?PartlyAre all the source data underlying the results available to ensure full reproducibility?PartlyIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?NoAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:medical education, professional development, online learningI confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.