Are certain types of applicants disadvantaged by the current state of communications between applicants and residency programs?In this survey study, most applicants reported that they  reported sending communications to residency program directors. There were notable differences between MD, DO, and IMG applicants, as well as between applicants with different racial and ethnic backgrounds, with White and Asian applicants having more than double the rate of faculty advocating to residency programs on applicants’ behalf compared with Black or Latinx applicants.These results suggest that, because the current state of communications may increase inequities in application processes, a centralized means for applicants to convey their interest to residency programs is urgently needed. This survey study estimates racial and ethnic disparities in rates at which applicants to obstetrics and gynecology residency programs contact residency program directors and assesses how directors report being influenced by these communications. In order to equitably improve the residency application process, it is essential to understand the problems we need to address.To determine how obstetrics and gynecology (OBGYN) applicants and faculty communicate applicants’ interest to residency programs, and how program directors report being influenced by these communications.This survey study was conducted with email surveys of OBGYN application stakeholders in 2022. Included participants were OBGYN applicants, clerkship directors, and residency program directors in medical education associations’ email listservs.Surveys sent by the American Association of Medical Colleges, Association of Professors of Gynecology and Obstetrics, and Council on Resident Education in Obstetrics and Gynecology.2 tests were used to analyze differences.Whether applicants themselves, or faculty on their behalf, communicated to residency programs, and the influence program directors reported placing on these communications for their decision-making. Descriptive statistics and χP = .02). There were also differences in type of medical school, with 377 of 427 MD applicants (88.3%), 109 of 125 DO applicants (87.2%), and 67 of 87 International Medical Graduate applicants (77.7%) reporting sending communications (P = .02). Approximately one-third (254 applicants [35.7%]) had faculty reach out to programs on their behalf. White (152 [40.1%]) and Asian (37 [43.0%]) applicants were more likely to have faculty reach out compared with Black (6 [11.1%]) and Latinx (12 [29.3%]) applicants (P = .01). Program directors reported that preinterview communications from faculty they knew (64 [32.2%]) and other program directors (25 [12.6%]) strongly influenced their decisions, and otherwise rarely reported that communications strongly influenced their decisions.A total 726 of 2781 applicants (26.1%) responded to the survey and were included in analysis . The self-reported racial and ethnic demographics of the 726 applicant respondents were 86 Asian (11.8%), 54 Black (7.4%), 41 Latinx (5.6%), 1 Native Hawaiian or Pacific Islander (0.1%), 369 White (52.2%), 45 with multiple racial identities (6.2%), and 91 (21.5%) preferring not to answer. The majority of applicants (590 [82.9%]) sent communications at some point in the application process. Applicants who identified as White (336 [88.7%]) or Asian (75 [87.2%]) were more likely than those who identified as Black (40 [74.1%]) or Latinx (33 [80.5%]) to reach out to programs (The current state of communications may increase inequities in residency application processes; differences between faculty communications for applicants from different racial and ethnic backgrounds are particularly concerning given that program directors are more likely to weigh communications from faculty in their decision-making. A centralized, equitable means for applicants to signal their interest to programs is urgently needed. In the 2021-2022 application cycle, there were 2781 total OBGYN applicants registered through the American Association of Medical Colleges (AAMC) Electronic Residency Application Service (ERAS).3 In the past, when applicants submitted 10 to 20 applications, the application itself was a signal of interest, but this is no longer the case, with each OBGYN applicant submitting an average of 72 applications.1 Programs now struggle to ascertain which applicants are genuinely interested in their program and use easy-to-filter metrics to decrease their pools of applications to review to manageable numbers.5 Applicants have recognized that in order to increase the chance that their application will be reviewed, they  may need to communicate with the residency program to signal their genuine interest.While many factors contribute to the increased competitiveness of the specialty, marked application inflation creates unique challenges for both applicants and residency programs.6 and these applicants may receive varying degrees of mentoring and advocacy compared with MD applicants. At this time, it is unknown how many applicants and faculty are reaching out to programs and whether different groups of applicants are disadvantaged through this informal signaling. This information is especially important as specialties consider the implementation of formal centralized signaling mechanisms. Preference or program signaling was first implemented by otolaryngology in the 2020-2021 application cycle,7 and 16 specialties are now offering it in the 2023 application season.8 Applicants can send formal signals to residency programs around the time they submit their application through the ERAS system. The aim of this study was to assess how OBGYN applicants and faculty are currently communicating their interest to residency programs, and how program directors report being influenced by these communications.The application process should be carefully examined to understand how to create meaningful interventions to address inequities that may create disadvantages for some applicants. The OBGYN specialty has a substantial number of residents from DO (15.6%) and IMG backgrounds (7.3%),Surveys were created for OBGYN applicants, OBGYN clerkship directors, and OBGYN residency program directors by the leadership team of the Right Resident, Right Program, Ready Day One initiative led by the Association of Professors in Gynecology and Obstetrics (APGO) and funded by an American Medical Association Reimagining Residency grant. The team consisted of medical school and residency educators, medical students, and residents from osteopathic (for DO applicants) and allopathic (MD) medical schools in the US, as well as international medical schools (IMG). Content validity was established through a review of the literature and response process validity was established through piloting of the survey with stakeholder groups.For the applicant and clerkship director surveys, questions asked respondents if, when, and how they communicated with residency programs (eAppendices 1 and 2 in the AAPOR) reporting guidelines.9The applicant and clerkship director surveys were deemed exempt from the need for informed consent from participants by the University of Michigan institutional review board. Likewise, the program director survey was deemed exempt by the Health Media Lab independent review board. All surveys were deemed to carry no more than minimal risk to respondents since the responses were anonymously collected. These studies followed the American Association for Public Opinion Research . χOf the 2781 total OBGYN applicants registered through ERAS, 726 responded, for a response rate of 26.1%. Based on self-reported race and ethnicity, 54 respondents (7.4%) identified as Black, 86 (11.8%) as Asian, 41 (5.6%) as Latinx or Hispanic, 1 (0.1%) as Native Hawaiian or Pacific Islander, and 379 (52.2%) as White . Of the P = .02). In addition to personally reaching out themselves, more than one-third of all respondents (254 [35.8%]) had faculty contact programs on their behalf. Less than half (323 respondents [45.4%]) did not contact programs, 116 (16.3%) were not sure, and 19 (2.7%) did not answer this question. Notably, there were marked differences between groups based on type of medical school. Nearly half of MD applicants (189 [44.3%]) had faculty reach out, compared with 26 DO applicants (20.8%), and 20 IMG applicants (23.0%) (P < .001). There were also notable differences between self-identified racial or ethnic groups. Applicants identifying as White (336 [88.7%]) or Asian (75 [87.2%]) were more likely than applicants identifying as Black (40 [74.1%]) or Latinx (33 [80.5%]) to contact programs directly (P = .02). White and Asian applicants were also 2 to 3 times more likely to have faculty reach out to programs on their behalf (P = .01) (Most respondents (590 [82.9%]) sent communications signaling interest to residency programs at some point during the application and interview season. Different applicant groups had different rates of communication, with 377 MD applicants (88.3%), 109 DO applicants (87.2%), and 67 IMG applicants (77.7%) reporting that they reached out to programs (P = .01) .Applicants’ methods of communication with residency programs were predominantly via email (578 respondents [81.2%]), with 101 respondents (14.2%) calling the program and 71 (10.0%) writing a physical letter. For faculty communication with programs, 197 respondents (27.7%) reported that faculty emailed, 83 (11.7%) that faculty called, and 5 (1.0%) that faculty wrote a physical letter to the residency program.Many applicants reported reaching out to a substantial number of residency programs in an effort to secure an interview: 44 (6.6%) reached out to 1 program, 117 (17.5%) reached out to 2 or 3, 98 (14.6%) reached out to 4 or 5, 126 (18.8%) reached out to between 6 to 10, 86 (12.8%) reached out to between 11 to 20, and 83 (12.4%) reached out to more than 20 programs. Only 116 respondents (17.3%) did not reach out to any programs. Forty-two respondents (6.3%) did not answer this question.After interviews were completed, respondents reached out to a fewer number of programs, with 209 (31.2%) reaching out to only 1 program, 228 (34.0%) reaching out to 2 or 3, 77 (11.5%) reaching out to 4 or 5, 43 (6.4%) reaching out to between 6 and 10, 12 (1.8%) reaching out to between 11 and 20, 7 (1.0%) reaching out to more than 20 programs, and 94 (14.0%) reaching out to no programs. Forty-two (6.3%) respondents did not answer this question.Among clerkship directors, 79 of 249 responded, for a response rate of 31.7%. Of these, more than half (41 respondents [51.9%]) reached out to residency programs on behalf of their students during the application and interview process . ClerkshAmong program directors, 200 of 280 responded to the survey, for a response rate of 71.4%. Most respondents were not strongly influenced by any of the communications from different groups when making decisions about which applicants to interview. However, they were influenced most by communications from faculty that they knew, or from fellow program directors . NotablyOur study highlights the many inequities in the current residency application processes. Applicants and faculty are already trying to send many signals to residency program directors. The notable differences between MD, DO, and IMG applicants, as well as applicants from different racial and ethnic backgrounds, are alarming. The fact that White and Asian applicants had more than double the rate of faculty advocating to residency programs on their behalf is especially notable given that program directors reported being more influenced by communications from faculty and fellow program directors. While clerkship directors are also spending considerable time reaching out on behalf of their students, these communications may not be helpful if they do not know the program director—highlighting yet another potential inequity for applicants with faculty who may be less connected to other medical educators in OBGYN. Finally, the amount of effort applicants spend contacting programs with questionable benefit is remarkable, highlighting an opportunity cost that could have been spent on service, scholarship, and preparing for residency.10 further limiting access for these applicant groups to mentors and faculty outside their school who could reach out to residency programs on their behalf. Applicants from majority backgrounds may feel more entitled or comfortable asking faculty to send communications for them. It is important to highlight how these inequities can work synergistically to disadvantage applicants from already marginalized backgrounds. Because of application inflation, program directors use filters such as clerkship grades, Alpha Omega Alpha medical honor society status, and standardized test scores.11 These metrics are known to be vulnerable to racial and ethnic bias and to favor White applicants.13 Our findings show that White and Asian applicants are further advantaged at this point by having faculty who reach out to program directors, which may subsequently enable their applications to be more holistically reviewed. The percentage of Black OBGYN residents has been decreasing in the past decade,15 and these additive inequities in the current processes are likely one of the many contributors to this troubling decline. Our study demonstrates how existing systems are further propagating societal inequities in the application process and highlights the need for a centralized, equitable way for applicants to signal their interest in programs. Instead of needing to rely on faculty relationships that are out of an applicant’s control, a formal program signaling system should work to level this uneven playing field.An equitable application process should empower applicants to have fair access to transparent selection systems. Our findings reveal that this is not the case. The proverbial “smoke-filled back rooms” have been replaced by emails and calls from colleagues that can influence program directors’ decision-making. DO, IMG, and applicants from racial and ethnic backgrounds underrepresented in medicine might not have equal access to faculty to advocate on their behalf. This varying access could be due to medical school structure, varying resources, or the lack of faculty mentorship for individuals from underrepresented backgrounds. This has been further exacerbated during the COVID-19 pandemic as visiting rotations have been discontinued or have reduced availability,16 Of the 16 specialties that are implementing program signaling in the 2022-2023 application, most (14 of 16) allow applicants to send up to 5 or 10 signals. In OBGYN, applicants will be able to send out a total of 18 signals (3 highest priority and 15 high priority) to programs in the 2023 application cycle. Our study suggests that a higher number of signals may be beneficial to applicants given that more than half of our survey respondents reached out to more than 4 programs. However, it is challenging to infer whether these applicants were reaching out to these large numbers of programs because of a genuine interest in the program, or whether they were simply hoping to increase their chances of getting residency interviews. It is also important to note that a substantial number of applicants are registered through ERAS who ultimately do not count as active applicants for the Match through NRMP. What happens to these applicants and how many of them dual-apply into a second specialty is unclear at this time. Increased transparency and sharing of data available through ERAS and NRMP will be essential as change measures such as program signaling are implemented and evaluated.The inequitable faculty advocacy for different types of applicants is especially relevant given that the first 2 years of the otolaryngology preference signaling experience demonstrated that the impact of the signals was greatest for applicants who struggle to receive many interview offers.17 The purpose of this communication is notably different from that of the preinterview communication since the applicant has presumably been able to express their interest in the program during the interview day.Our study highlights that applicants are also trying to communicate their interest in residency programs after interviews have been completed. While it is notable that applicants are sending communications to fewer numbers of programs after interviews, this data questions whether these communications should continue at all given how little value program directors seem to place on these emails, as well as the inequity considerations. This is consistent with previously published work that has questioned the value of these postinterview communications.This study had several limitations. The response rates were lower for applicants and clerkship directors, but we were still able to capture representative samples from stakeholders. The response rate was higher for program directors because the survey questions were included in an annual survey of program directors.We did not ask questions about what content was included in communications from applicants, faculty, and clerkship directors. However, our study highlights that, in the midst of application inflation, program signaling is needed in order for applicants to equitably communicate their interest to residency programs.1 While it is tempting to treat a symptom of the bigger problem when working to achieve a more equitable residency application process, there remain a number of inequities when considering the burden of time and cost on individual applicants. Program signaling, especially in those specialties using a high number of signals, may provide more clarity on the feasibility of potential application capping in the future.Finally, it is important to note that program signaling does not resolve the underlying problem of application inflation. If anything, it might increase the number of applications submitted per applicant.The myriad of consequences created by application inflation has led to a worsening of inequities in the residency application process. This study highlights the need for central program signaling, especially given otolaryngology data that suggests that centralized signaling may be the most beneficial for applicants who struggle to receive many interview offers.


Precision-cut tumor slices (PCTS) have recently emerged as important ex vivo human tumor models, offering the opportunity to study individual patient responses to targeted immunotherapies, including CAR-T cell therapies. In this review, an outline of different human tumor models available in laboratory settings is provided, with a focus on the unique characteristics of PCTS. Standard PCTS generation and maintenance procedures are outlined, followed by an in-depth overview of PCTS utilization in preclinical research aiming to better understand the unique functional characteristics of cytotoxic T cells within human tumors. Furthermore, recent studies using PCTS as an ex vivo model for predicting patient responses to immunotherapies and other targeted therapies against solid tumors are thoroughly presented. Finally, the advantages and limitations of the PCTS models are discussed. PCTS are expected to gain momentum and be fully utilized as a significant tool towards better patient stratification and personalized medicine. The recent advances in immunotherapies, such as immune checkpoint modulators and adoptive T-cell transfer, open new opportunities for the treatment of cancer. With this broad spectrum of new therapeutic agents available, however, the need for robust, predictive preclinical models to minimize translational failures in immuno-oncology is increasing.Several preclinical mouse models have been developed throughout the years to model human cancer and predict patient responses. These mouse models carry important limitations, as they are unable to fully recapitulate human tumor genetics and histology as well as the human tumor microenvironment (TME) ,2. For tThe ex vivo generation and culture of live precision-cut tumor slices (PCTS) deriving from various human solid tumor sites has lately emerged as a potential tool for successful personalized drug screening ,4. PCTS,This review delves into the important role of PCTS slices in modern cancer research with a focus in immunotherapy, as well as in the progress of patient stratification and personalized medicine.Within laboratory settings, several standardized human tumor models have been widely used throughout the years to study the biology of cancers . The mosThe culture of tumor cell lines in specialized collagen or other gel-like matrices allows for tumor cells to obtain a more 3D, spherical-like shape ,9. These. They are used in various applications, including the preclinical testing of clinically available therapies to determine optimal combinations for individual patients, predictive biomarker discovery, drug screening and the elucidation of mechanisms leading to drug resistance [PDX models are developed via subcutaneous or organ-specific implantation of fresh patient tumor tissue or cells in immune-deficient mice. The engraftment of primary tumor cells or fragments results in the development of nodules that retain the phenotypic characteristics of the original tumorsistance . Despitesistance ,15. Furtsistance ,15. Mostsistance .PCTS are an ex vivo platform able to recapitulate the high in vivo intratumoral complexity more accurately than other existing models ,16. PCTSPCTS are generated using vibrating microtomes, known as “vibratomes”. Microtomes, used extensively in diagnostic histopathology, are relatively simple instruments comprising of a blade and a blade holder, an object clamp used to hold and secure objects, an advancement mechanism, and a mechanism for section thickness adjustment that allows for precise control . The wayVibratomes incorporate vibration to the microtome blade, resulting in less pressure and stress to the tissue sample than a stationary blade, and are therefore ideal for fresh tissue slicing . In the The generation of PCTS involves the acquisition of adequate volume of fresh tissue, usually provided as a surplus to diagnostic procedures. The production of PCTS occurs within a few hours post-surgery to promote tissue survival and retain high levels of tissue viability ,18.Fresh tissue samples are placed in ice-cold tissue storage buffers  immediately following surgical excision. Once in the laboratory and using a biopsy punch, several cores can be obtained from the tissue sample ,25. AlteTissue slicing speed (mm/s) and vibration amplitude (mm) are selected according to tissue type, density, consistency and integrity . Once PCPCTS can usually retain their viability as well as the same histological, architectural, and phenotypical characteristics as the original tumor for an average of 6–7 days, depending on tumor type ,31,33,34PCTS offer the opportunity to study the localization and function of the several immune cell types within the TME, and how the latter behave following treatment. With CD8+ cytotoxic and CD4+ helper T cells playing a central role in anti-tumor immunity, several PCTS-based studies have focused on understanding the localization and function of tumor infiltrating lymphocytes (TILs) in solid tumors, including lung, breast and melanoma ,45,46,47CD103+ CD45RO+ CD8+ tissue resident memory T cells (Trms) have recently emerged as a highly cytotoxic CD8+ T-cell subset localized within mucosal tissues and playing a key role in anti-tumor immunity in several solid tumors ,44,45,46In a similar study by the same group, the motility and migration patterns of CD8+ TILs within lung and ovarian cancer PCTS were explored . ImmunosData from such studies can be helpful in identifying factors that block spatial distribution of TILs within tumor cell-rich areas, as well as factors with a role in TIL anergy, ultimately contributing to the discovery of biomarkers of response to immunotherapies.PCTS use in CAR-T cell research is still in its infancy. A 2021 study used EGFR-targeting CAR-T cells in co-cultures with either PCTS derived either from a BxPC3 EGFR-expressing pancreatic cancer PDX model or from PCTS derived from NSCLC and ccRC . ConfocaThe potential of PCTS in predicting patient response has been under evaluation for several years. Most studies to date have used PCTS to assess patient responses to chemotherapy and targeted therapy ,48,49. UMore recently, PCTS from nine hepatic metastatic colorectal cancer patients were treated with clinically relevant oxaliplatin (chemotherapy), cetuximab (EGF inhibitor) and pembrolizumab (PD-1 inhibitor) . It was Seo et al. generated PCTS from surgical resections of patient PDA tumors  and demoIn a 2020 study, researchers treated PCTS derived from clear cell renal carcinoma (ccRC) patient surgical resections with either emsirolimus (mTOR inhibitor blocking cancer cell proliferation), pazopanib (angiogenesis blockade), or sunitinib (blockade of angiogenesis and tumor cell proliferation) to explore differences in drug sensitivity of different patients’ PCTS . Indeed,A recent study by Horowitz et al. aimed at developing a more automated, digitally manufactured microfluidic platform, allowing for multiplexed drug testing using intact glioblastoma (GBM) PCTS derived from patients or PDX models . This miPCTS are highlighted as the ex vivo model that fully retains the real tumor 3D landscape, along with its TME, including stroma and immune cells. They are relatively easy to produce and, within their viability timeframe of up to a week, can offer a platform to test tumor responses of immunotherapies, such as checkpoint inhibitor antibodies and CAR-T cells. As CAR-T cells are a “living drug”, ex vivo PCTS models can be used to study CAR-T cell kinetics, spatial distribution, and migration patterns, additional to their function. Thus, within the next decade, more preclinical PCTS-based research studies on CAR-T cells are expected to emerge.PCTS present with several limitations, such as the inability to be cultured long-term and to be biobanked due to current limitations in the PCTS freezing processes . Another important limitation is that PCTS are low throughput systems. Tissue dissociation leading to tumor and immune cell isolation allows for high-throughput assays, such as transcriptomics, multi-omics, single-cell profiling, multi-color flow cytometry and multi-panel drug testing. In contrast, PCTS allow for limited drug testing each time , and for imaging and immunohistochemistry techniques that usually permit the testing of only a small number of fluorescent markers. The development of commercial microfluidic systems in the future could lead to platforms capable of supporting multiplexed drug testing in live tumor slices.Although PCTS cannot be used directly for multi-omics analyses, they can still be dissociated or PCTS lysates can be generated and can subsequently be used for transcriptomic, proteomic and metabolomic assays ,55,56. AThe biggest limitation of PCTS is the heterogeneity between slices of the same sample, which reflects tumor heterogeneity. To date, there is no standardized approach to cultivating tumor PCTS and studies are limited by their minimal biochemical characterization of the impact of slice preparation and cultivation conditions. It is difficult to systematically compare different slicing techniques and cultivation methods due to the inherent heterogeneity between tumor histology, patients and individual slices. A recent study of 108 resected primary or metastatic liver tumors revealed significant variability in growth and proliferative activity among different tumors, and significant metabolic heterogeneity at baseline between slices cut from the same tumor . AdjacenAside from inter- and intra-tumoral heterogeneity at baseline, studies show that the process of slicing and cultivating PCTS may induce changes in the tissue that render it biologically different from the original tumor. A study to comprehensively characterize the transcriptional changes in PCTS during cutting and culturing using RNA sequencing was recently published . By compA similar study looking at the effect of culture on gene transcription was performed in pancreatic ductal adenocarcinoma , where bThese two studies highlight the transcriptional changes that occur during the slicing and culturing of PCTS. The changes appear to be both disease and organ specific, with different organs demonstrating different susceptibility to changes over time. Interestingly, even PCTS generated from the same organ, but of different patients, show variability in genomic changes during culture. The greater the number of changes, the less comparable PCTS are to their tissue of origin. The supplementation of culture media and manipulation of culture conditions have been proposed as mechanisms to overcome some of these changes ,61,62, aBy retaining the landscape of the original tumor, PCTS are a physiologically relevant and important culture system for the study of solid tumor immunobiology. Data to date have also shown that this approach can be a relevant model across several tumor types for the individualized testing of drug susceptibility to improve clinical response rates. Having identified several limitations in the use of PCTS, recent and ongoing work from our team and others aims to better understand the potential of this platform as well as to optimize culture condition protocols to fully utilize it. Nevertheless, we anticipate that further work will eventually make PCTS useful for personalized clinical immunotherapy, especially in the case of adoptive cell therapy.


The AGGME requires faculty to participate annually in faculty development sessions. Barriers to this requirement include faculty having a lack of time and not perceiving benefits to participating. Effective evaluation and feedback are integral to resident training. Faculty often feel ill prepared to deliver feedback, and residents find accepting and recognizing feedback challenging. We provided faculty with a spaced education program via email that used cognitive theory of multimedia learning solutions in instructional design.The 14-week program consisted of one microlecture and 13 skills-based teaching tips. One tip reinforcing knowledge and skills from the microlecture was emailed each week for faculty to practice in the clinical environment with trainees. Participants completed a short quiz, course evaluation, and self-reflection. The new world Kirkpatrick model was used for program evaluation.Fifty-two physician participants received credit for participating; 34 completed the entire course. Of the 34, 32 (94%) identified at least one effective feedback technique, and 27 (79%) were able to define evaluation and recognize observation as the cornerstone of evaluation. Out of the 15 effective feedback characteristics taught, 13 (87%) were identified. Fifty-one participants (98%) rated the program as good/excellent, 52 (100%) wanted more Tuesday's Teaching Tips programs, and the majority recognized change in knowledge and/or skills.Participants rated the spaced education program as good/excellent and were able to meet the course objectives. This teaching strategy for faculty development was well received, as it was easily accessible and implemented in the clinical learning environment with trainees. Effective evaluation and feedback enable resident growth in terms of formative assessment and assist with achievement of clinical competence (summative assessment).9 However, both faculty and residents report widespread dissatisfaction regarding faculty skills in effective evaluation and feedback and resident recognition and acceptance of each when presented.15 Additionally, faculty continue to report discomfort in giving feedback and feel ill prepared to accomplish this task.22The ACGME requires faculty development programs to improve the knowledge, skills, and behaviors of faculty as educators.23 and Pernar and colleagues.24 The Matzie study provided spaced education to surgical residents as part of a residents-as-teachers program to improve feedback to medical students, and the Pernar study sought to improve surgical faculty's teaching skills with medical students. In both studies, the target population received weekly emails in two cycles over 9 months with a brief statement and, in some instances, an example for implementation. Spaced education is a type of distributive practice by which information or procedures are presented and repeated in subsequent reviews to enhance long-term retention of knowledge and transfer of skills.25 A MedEdPORTAL search using the term evaluation and feedback yielded over 1,000 publications, most of which described workshops or online modules, but none that appeared to involve an email approach to faculty development. Similarly, a MedEdPORTAL search for the term spaced education did not reveal relevant publications utilizing a spaced education approach to faculty development.In an effort to address faculty development regarding evaluation and feedback and to take into account faculty time constraints, we developed an innovative faculty development program entitled Tuesday's Teaching Tips—Evaluation and Feedback (TTT) that built upon programs by Matzie and colleagues26 Mayer and others have suggested the importance of taking into account cognitive load to address the finite quantity of cognitive processing the mind can accomplish in both the verbal and visual channels in working memory.29 Yet there is a balance in being able to hold the learner's attention and draw connections back to previous knowledge and experiences or information just learned . The use of visual cues is a CTML solution that utilizes coloring, arrows, borders, highlighting, bolding, and other visual cues to capture or guide the learner's attention and has been shown to be effective in learning.26 For example, whereas the Matzie and Pernar studies’ emails consisted of text (words) only  but also to incorporate a microlecture  establishing foundational knowledge upon which to base the spaced education emails and, further, to utilize cognitive theory of multimedia learning (CTML) solutions to address cognitive load concerns.24 The EdD also designed the microlecture and emails utilizing CTML solutions. The microlecture  a short postmicrolecture quiz  levels 1–3 to analyze data collected from partcipants.ure quiz  at the bure quiz  consistiFifty-two physicians across 13 specialties received partial or full CME credit, with 34 (65%) completing the entire program and receiving a course certificate. The 52 physician participants included two from anesthesiology, seven from emergency medicine, one from family practice, two from internal medicine, three from neurology, two from OB/GYN, two from ophthalmology, four from orthopedics, 13 from pediatrics, four from psychiatry, three from radiology, seven from surgery, and two from urology. Forty-eight percent were female. Of the 34 who completed the program, there were 11 different specialties represented, including one participant from anesthesiology, four from emergency medicine, one from family practice, two from internal medicine, two from neurology, one from OB/GYN, four from orthopedics, 11 from pediatrics, one from radiology, five from surgery, and two from urology, with 53% being female.n = 12, 23%) or excellent , 51 (98%) reported information gained would enhance patient care or medical education , 51 (98%) had high  or moderate  confidence in their ability to implement changes in their teaching , and 52 (100%) stated they would like to see more TTT on other topics. On the question whether the COVID pandemic had affected respondents’ ability to participate, 21 (40%) answered either yes  or somewhat . Of those who answered a follow-up question on the ways the COVID pandemic had affected participation, the majority noted reduced contact with trainees due to canceled educational experiences or virtual interactions with trainees. One respondent had gone on military deployment, and another expressed experiencing burnout. When asked for recommendations for improvement, one participant recommended adding videos or simulated demonstrations, and two participants responded they would have liked a small, laminated card with salient teaching points as a quick reference.On the microlecture posttest quiz assessing whether participants were able to define evaluation and recognize observation as the cornerstone of evaluation, 41 of the 52 participants (79%) answered all questions correctly. On the course evaluation, 51 (98%) rated the program as good (n = 34) were asked to reflect on the course by answering the question “Please describe how this program has improved your overall knowledge and skills related to evaluation and feedback when working with your trainees,” with a 500-word limit. and instead of but.”To further assess the ability to describe methods of evaluation and list characteristics of effective feedback, participants who had completed the entire program and received a course certificate  the format of TTT and (2) the perceived knowledge and/or skills gained. We identified subgroupings within these two categories. Subgroupings for the format category included the following: (1) The course format was useful for learning feedback, (2) the series of tips helped participants use the tips in busy clinical environments, and (3) participants found it helpful to learn one tip per week. Subgroupings for the knowledge/skills gained category included participants (1) citing specific techniques and principles they had learned and (2) seeing themselves as more effective providers of feedback after the course . Partici24 which found that faculty did not perceive the emailed tips as helpful, our program evaluation found that participants had high satisfaction with the format and were able to identify specific techniques for effective feedback with trainees. Faculty were very receptive to this teaching strategy, as it was time efficient and easily accessible, eliminated the need to go to a training, and offered simple skills-based strategies to practice in the clinical learning environment. Furthermore, 100% of participants reported wanting more TTT instructional sessions in the future as a method for faculty development. It is possible that our use of a foundational microlecture on which to build the spaced education reinforcement and/or the incorporation of CTML solutions encouraged faculty to practice the tips, increased their knowledge and skills, and led to perceived improvements. Additionally, our program was well received by physicians across multiple specialties and was completed over a shorter period of time than the program described in the Matzie and Pernar studies (14 weeks vs. 9 months).24We developed and implemented an innovative spaced education faculty development program on evaluation and feedback that provided weekly emailed tips utilizing CTML solutions to reduce cognitive load and focus the learners on salient content. Unlike the Pernar study,Given the reliance on technology, being proactive was important to ensuring a smooth implementation of the program. Setting up the tip emails in the email system in advance allowed them to be sent at the designated time each week (Tuesday at 7 a.m.). Practice runs were instrumental in discovering technical glitches with the process, such as with our use of Outlook and our network system. We discovered that if our network system rebooted overnight, it erased emails set up in advance; therefore, we had to ensure each Tuesday morning that the emails had indeed been sent. Moreover, we felt an introductory email describing how the program was designed was imperative to set expectations for the program. Second, we learned to be creative in terms of calculating the time participants would spend on this nontraditional educational activity to earn CME credit. We took into consideration the 15-minute microlecture and determined that to read the email (maybe more than once), practice the tip on one's own, and then utilize the tip with various learners and at different times during the week, 30 minutes each week was reasonable. We also allowed for partial credit if the microlecture had been completed. Even though faculty signed up for the program, if they did not watch the microlecture as foundational information, they did not receive the TTT emails (the spaced education).Our evaluation was based on participants’ self-report and not on observations of their teaching or resident evaluation of their teaching before and after implementation of the program (level 4 of NWKM) for educational outcomes. In addition, our evaluation approach was somewhat limited by the use of our institutional CME system. The quiz after the microlecture did not accurately capture correct and incorrect questions individually but rather a pass/fail; therefore, we included only 100% passing scores to assess objectives 1 and 2. There could have been more correct answers for each question if we had individual question data. For the postprogram evaluation, we utilized Google Forms to include the required CME questions and were able to add the self-reflection question and attestation. We were mindful of keeping the evaluation feasible for busy clinicians. Satisfaction questions were based on all 52 participants who received CME credit as all 52 completed the course evaluation. However, only those who both completed the course and earned the course certificate received the self-reflection questions. We did not assess participants’ prior experience with or training in evaluation and feedback, which may have impacted the interpretation of the program evaluation results. Finally, we did not do a true qualitative thematic analysis of the reflective comments but rather grouped them into broad categories.TTT is an approach that addresses faculty development barriers of faculty time constraints, convenience in attending, and perceived benefits. The program was easy to implement, required minimal staff time, and used the Outlook email system for program delivery as well as an Excel spreadsheet to track participation. The program has broad applicability across multiple medical specialties in an academic medical center with no obvious challenges to generalizability. The tips can be integrated into an existing faculty development curriculum or utilized as a stand-alone program. Next steps involve a qualitative analysis of the participants’ reflective statements. Future research or evaluation should include the behavioral impact of TTT on faculty skills as well as the impact of TTT on learners.Evaluation and Feedback Microlecture.m4vEmailed Tips.pptxProgram Announcement.pptxRegistration Form.docxProgram Directions.docxPreparatory Email.docxCertificate of Completion.docxPostmicrolecture Quiz.docxPostprogram Evaluation.docxAll appendices are peer reviewed as integral parts of the Original Publication.


While females make up more than half of medical school matriculants, they only comprise about one third of emergency medicine (EM) residents. We examined EM residency cohorts with entering years of 2014–2017 to estimate the ratio of males to females among residents and program leadership to determine what correlation existed, if any, between program leadership and residency gender distributions.We identified 171 accredited EM residency programs in the United States with resident cohorts entering between 2014–2017 with publicly available data that were included in the study. The number of male and female residents and program directors were counted. We then confirmed the counts by contacting the programs directly to confirm accuracy of the data collected from program websites.Within the included 171 programs, the overall male to female EM resident ratio was 1.78:1. Individual program ratios ranged from 0.85–8.0. Only eight programs (5.6%) had a female-predominant ratio. Among program directors, the overall male to female ratio was 2.17:1. TThe gender of the program director did not have a statistically significant correlation with the male to female ratio among its residents (P = .93).Within 171 residency programs across the US with entering cohorts between 2014–2017, the average male to female ratio among residents is nearly 2:1. No significant correlation exists between the gender distribution among a program’s leadership and its residents. While females make up more than half of medical school matriculants, they comprise only about one third of emergency medicine (EM) residentsWe identified 171 Accreditation Council for Graduate Medical Education (ACGME)-accredited EM residency programs in the United States in 2017.We used publicly available data from individual program websites to determine resident gender distribution. We manually counted the gender distribution of residents at each program with publicly available pictures of the program’s residents and faculty on their websites. We contacted programs without publicly available data via email to determine their gender distribution. Once the data was collected, we contacted program directors and coordinators via email to verify the accuracy of the manually collected data. We allowed 60 days for response, with follow-up emails to those with no response.A total of 47 programs verified the data, which were entered into a matrix-style data collection instrument for analysis. By using this approach, we found that most of the data collected from publicly available program websites was accurate; so, we chose to also include both verified and unverified data in our analysis. Public data was determined to be relatively accurate with a 1.65% error on the total number of residents reported, a 2.08% error on the number of male residents reported, and a 3.73% error on the number of female residents reported. We calculated the percent error using datasets from programs that confirmed our manually collected data. The calculation was performed by subtracting the estimated  number of total residents or male residents only or female residents only from the actual (confirmed) data for these categories, and then dividing by the actual data. Percent errors were then averaged for each category.We identified program directors, and associate and assistant program directors using similar methods as above. We confirmed gender distribution among those identified using email, and the gender makeup of program leadership was confirmed with the program coordinator. A total of 47 programs verified their residency leadership data.We analyzed the data using simple ratios to determine the overall gender distribution within all identified EM programs. The ratios for individual residency cohort years 2014–2017 were compared against one another in addition to comparison against the cumulative ratio for the four-year period. Additionally, we analyzed individual programs to determine the variation from the mean. The data from individual programs was compared to that of the program leadership to determine whether there was a correlation between the gender of the leadership and the overall makeup of the residency cohorts with respect to the mean.Public data were available for 171 ACGME-accredited EM residency programs in the US. Of the 7,185 residents identified, 4,598 (64%) were male compared to 2,587 (36%) female, giving an overall male to female (M:F) ratio of 1.78:1. This is similar to 2017 Association of American Medical Colleges (AAMC) data for all EM residents . We examined individual programs and found gender ratios among residents to range from 0.85–8.0 . Of the Gender distribution among program directors was similar to that of residents, showing 117 males and 54 females (2.17:1 ratio) and slightly less dichotomous than a previous report.Throughout this paper, we use the term “gender” to refer to male or female. We recognize and respect that gender exists on a non-binary spectrum. Since the first stage of our data collection was created from publicly available information on program websites, we focused on determining only the male:female resident and program leadership ratios across the residency programs that existed in 2017. We recognize this binary categorization as a limitation of our study. Additionally, the AAMC reports the breakdown of residents by gender as “male” or “female,” categorized in a binary manner.Other residency resources such as the American Medical Association’s FREIDA database for residency programs by specialty also categorize programs’ current resident distribution as male or female.The AAMC releases biennial data on gender distribution; however, this study provides further information on how the gender distribution varies across EM residencies. While the overall AAMC reported breakdown of gender in EM was 1.82:1 at the time of this data collection,DeFazio published similar findings in 2017 regarding gender distribution of residents and program director.The use of publicly available data as a proxy to estimate the ratio of males to females in EM residencies is limited by only partial confirmation of the identified programs. Additionally, there were some instances in which a program’s director changed from 2014–2017, which altered the gender ratio of program leadership.While approximately half of medical school graduates are female, females comprise only about one third of EM residents. Current data shows a clear gender discrepancy within the specialty of EM, both in resident and leadership populations. This study identified the current discrepancies in EM residencies across the country showing an overall ratio of 1.78:1. We identified only eight programs with a female-predominant ratio of residents out of a cohort of 171 EM residency programs. Among program leadership, the number of males again predominated with a slightly higher than two-thirds to one-third ratio. The distribution of male and female residents does not appear to differ significantly when compared to the gender distribution of program leadership. Further studies are warranted to determine which program characteristics influence medical students’ decisions in choosing a residency program and whether existing gender distributions play a role in an applicant’s decision-making process.What do we already know about this issue?While females make up more than half of medical school matriculants, they comprise only about one third of emergency medicine (EM) residents.What was the research question?We sought to determine what correlation existed, if any, between program leadership and residency gender distributions.What was the major finding of the study?The average resident male:female ratio is nearly 2:1. No significant correlation exists between the gender distribution of a program’s leadership and its residents.How does this improve population health?Understanding gender make-up of programs provides a baseline for future studies, to determine whether existing gender distributions play a role in an applicant’s decision-making process.


Patients with persistent physical symptoms presenting in primary care are often affected by multiple symptoms and reduced functioning. The medical and societal costs of these patients are high, and there is a need for new interventions tailored to both the patients and health care system.This study aimed to examine the usability of an unguided, self-help treatment program, “My Symptoms,” developed to assist patients and general practitioners in symptom management.In all, 11 users  participated in web-based thinking-aloud interviews involving the performance of predefined tasks in the program. Thematic analysis was used to categorize the severity of usability issues. General usability heuristics were cross-referenced with the usability issues.The analysis identified important usability issues related to functionality, navigation, and content. The study shows how therapeutic knowledge in some cases was lost in the translation of face-to-face therapy to a digital format. The user testing helped uncover how the functionality of the digital elements and general navigation of the program played a huge part in locating and accessing the needed treatment. Examples of redesign to mediate the therapeutic value in the digital format involving health care professionals, web developers, and users are provided. The study also highlights the differences of involving patients and laypeople in the interviews.Taking the experience of common symptoms as a point of departure, patients and laypeople contributed to finding usability issues on program functionality, navigation, and content to improve the program and make the treatment more accessible to users. The experience of physical symptoms is a normal phenomenon. Most symptoms are self-limiting, but they may become persistent and lead to frequent contacts with health care providers. In general practice, 17% of patients are affected by persistent physical symptoms (PPS). These patients have an increased risk of disability and mental comorbidity impacting quality of life, health care use, and the ability to work .Symptoms span a continuum from mild to severely disabling. This paper refers to symptoms that lead to repeated contact with general practice and may be associated with some degree of functional disability but do not reach the severity of a disease such as a functional disorder . The preAccording to previous studies, internet-based self-help interventions may contribute to symptom alleviation and improved quality of life ,9. To asThe overall framework of making the “My Symptoms” program lent itself to ideas from the participatory design research paradigm within health care ,11. HereThe objectives of the present study were (1) to investigate the usability of the “My Symptoms” program with a specific focus on how to improve functionality and navigation and (2) to explore how users could help improve the intuitiveness and user-friendliness of the program.From 2020 to 2021, the last part of the development phase 2  commenceThe project group developing the self-help program consisted of HCPs , anthropologists, techno-anthropologists, and web developers. The techno-anthropologists conducted the thinking-aloud sessions, whereas the whole group was involved in the processing of results.All participants were informed orally and in writing about the study, and all participants gave their consent to participate. The study was approved by the Danish Data Protection Agency (J. no. 1-16-02-16-19). The Danish Act on Research Ethics Review of Health Research Projects is not applicable to qualitative studies. Therefore, ethical approval was not required from the Committee on Health Research Ethics in the Central Denmark Region.We included a convenience sample of primary care patients and laypeople. In all, 6 GPs identified and invited 4 patients aged 18-65 years with PPS. These potentially eligible patients were informed orally and in writing about the project by the GPs and gave their consent to be contacted by a researcher from the project team. A project member screened consenting patients according to the selection criteria and finally included or excluded patients .Inclusion criteriaAge 18-65 yearsAffected by persistent physical symptoms according to their general practitioner“Somewhat bothered” by at least 4 of 25 symptoms  checklist )Speak and understand DanishExclusion criteriaSevere mental disorderSick leave for more than 8 consecutive weeksWe aimed for the inclusion of approximately 12 patients, considering the number of 5-8 participants proposed by Nielsen  and VirzTo investigate usability, we applied the thinking-aloud method. The aim of this method was to “capture” the users’ thoughts as they navigated the “My Symptoms” program to gain insight into how they experienced the program in the context of actual use and what they found easy or difficult to do or understand . The proDue to the restrictions caused by the COVID-19 pandemic, we conducted the thinking-aloud sessions on the web . We usedRounds 2 and 3 focused more on testing predefined, specific elements in the program rather than core functionality. From the first round of testing, we observed that laypeople and patients interacted similarly to buttons, sliders, and other interactive web elements, which was why we included more laypeople than patients for these rounds. Immediately after thinking aloud, a follow-up interview ,23 was cAll audio and video recordings were transcribed. Based on transcripts and notes taken during the thinking-aloud sessions, we identified usability issues and rated the severity of these as minor, medium, or critical. Encountered bugs were also flagged. The GitHub platform   was usedUsing a thematic analysis approach , all usaThe thinking-aloud sessions gave insight into functionality and navigation. Additionally, the sessions revealed potentially problematic phrases and wordings that hindered the usability of the program. Inductively, a third category concerning content emerged, leaving us with 3 overall categories of issues. In In the development phase, the interactive elements and exercises of the “My Symptoms” program required close collaboration between web developers and the project group as common therapeutic tools were translated into digital equivalents. The usability test revealed that the intended purpose of the therapeutic content in some instances had been lost in this translation. General examples of core issues that suffered from translational issues were missing information on how to perform exercises, the need for additional text or content to explain exercise use, generic examples on changing habits were not translated by the user and used as is, and confusion on how to use core tools such as the “Goal Staircase.” Most of these issues concerned exercises that had been translated from a physical context into digital entities. Such translational problems were especially evident with the “Sorting of Values”  exerciseIn the “Sorting of Values” exercise, the user was supposed to select statements about the life values most important to him or her, such as “I value a healthy diet,” “I look for challenges,” and “My goal is to live in harmony with nature.” The exercise was then to sort the statements into the columns of “agree” or “disagree.” Finally, the user was prompted to choose 3 to 5 of the agreed statements into a new column to identify the most important values. all the value-statements had been sorted into the 2 above columns. Only then the user was able to sort the 3 to 5 most important statements into the previously hidden column, and the program would store these for later use. In the “Sorting of Values” exercise, only 1 of 7 users was able to complete the exercise. Completing this exercise was critical since the rest of the module depended on the “results” gained by completing it. This 1 user was able to access a “hidden” column that would only appear when In the physical version of the exercise, the patients would hold a deck of cards in their hands that would give them tactile feedback on how many cards or statements were left to sort. With the example of the “Sorting of Values” exercise, we noticed that one reason for stopping the sorting of values into the first 2 columns was the absence of knowledge on how many statements have to be sorted and how long it would take. Thus, the users figured it would be okay having some of the statements sorted into the 2 columns and moved on to the next page. The heuristic “visibility of system status”  reminds Based on the findings, the project group developed a solution where the third column was now shown all the time  to displThe “Sorting of Values” exercise was one example of several issues on not getting enough information at the right time to complete the exercises. Other issues were, for example, related to missing tactile information such as the “Sorting of Values” exercise and missing guidance from a therapist in the digital format.When investigating navigation, core issues were regarding too many pages and submodules, the placement of navigation buttons, navigating back to known content, and too much blank space. These navigational issues were especially frustrating to new users trying to find their way through the program.An example of a major navigational issue identified was navigating the different levels of the program. Knowing what page the user was on in relation to the rest of the program proved difficult. This was important if the user wanted to repeat or go back to an exercise in a new sitting. The thinking-aloud sessions demonstrated how most people tried to remember the location of the exercises by recalling the modules and the title on subpages. As a module could hold up to 3 levels and 10 subpages, this strategy proved difficult. Furthermore, the subpages of the modules were navigated by the next and previous page buttons bound to the top of specific pages. This made the navigation buttons disappear when scrolling down.To alleviate the users’ need for recalling the different locations of the exercises, a “sticky” navigation bar was developed. This was done in reference to, for example, the heuristics “recognition rather than recall” and “visibility of the system status” . The newInitially, the program was aimed at allowing the users to navigate freely according to their symptoms and to let them decide what content would be appropriate or helpful to explore. However, the usability tests revealed a need to help the users better navigate between the overall modules. They needed a better framework for their journey in the program. Therefore, we decided to let the first 3 modules open gradually before providing free access to all modules. When a new module opens, the users are prompted with a text message on their telephone.Analysis of the thinking-aloud and interview data revealed that specific program content could potentially hinder adequate program use. Core issues in this category were related to pages being text heavy, the perception of program legitimacy, and the use of language directed at the user. These issues could in many instances be attributed to the lack of therapeutic assistance to mediate program use, thus compensating for it with more text and content. These issues were especially evident when comparing feedback from patients and laypeople. Patients were more focused on the framing of the content in the program than laypeople. For example, in content, quotation marks were used to stress some of the medical terms. Most patients responded poorly toward this usage and figured the terms were made up, making them question the legitimacy of the program and their own experience with their symptoms. This was not an issue with laypeople. Patients also spent more time investigating content as means of navigation and interaction, even though we explicitly stated that reading the content thoroughly was not necessary during the usability testing. Contrarily to patients, laypeople found it bothersome to read too much text as a guide for navigation and interaction. Instead, they preferred being guided by different design cues. Patients, however, found it necessary to read most of the text to make sure they were on the right path while being guided by design cues. Patients who experienced symptoms fatigue and lack of concentration were unable to adhere to the thinking-aloud tasks to the same degree as other users, which they attributed to the pages being too text heavy. With patients tending to read most texts thoroughly, the program may be experienced as being too bothersome, thus decreasing motivation for use.what elements they would use, and patients spoke more in terms of how they would use them.In the follow-up interviews, we found that users related to the content in different ways. For example, when we asked participants, “Are there elements in the program you would like to return to and if so which ones?” , laypeopThe purpose of this study was to investigate the usability of the “My Symptoms” program by exploring functionality and navigation from a user perspective. Investigating the functionality of the program mostly revealed issues related to the translation of face-to-face therapeutic material into digital exercises. This was especially in relation to issues regarding the lack of transparency on how to complete exercises and lack of continuous feedback on exercise progression. With regard to navigation, most of the usability issues were about the number of subpage levels and lack of markers when users wished to go back to exercises or content. By reducing the number of levels of subpages and using different design cues, we sought to evoke the recognition of symbols rather than recall of page titles.The content of the program was not the target of our investigation but became a stepping stone for understanding how content also guides navigation and functionality. This was especially evident from the patients’ feedback. Although, differences in the inclusion of patients and laypeople were found, including laypeople was helpful when there was a lack of patients or patients were challenged cognitively—especially since the experience of bothersome physical symptoms is a common phenomenon. Laypeople and patients draw on the same type of IT-related schemata, for example, knowledge of browsing the web, using social media, and more. Moreover, they also draw on the experience on common symptoms.Designing behavior change exercises requires a lot of attention to the communication between the HCPs and web developers. The process of improving the functionality of the “Sorting of Values” exercise was attributed to the iterative process of continuously testing the exercise with users. Testing the exercise multiple times helped us make a digital therapy that makes sense at all levels: therapeutic model, technical capacity, and user-friendliness. Not testing the usability would have made the exercise, and the rest of the “My Values” module, inaccessible to most users. Here, a pragmatic approach of understanding both the therapeutic models and the system capability was necessary. For example, the iterative process allowed different versions of the same exercise to be investigated multiple times by different participants, making the users, web developers, and HCPs all a part of the designing process ,28. The In this study, laypeople acted as surrogates for patients. Knowing when and how to include laypeople is valuable when there is a lack of the intended end user. The use of surrogates can be beneficial in getting rid of the most critical usability issues before gaining access to a limited end user group . When usThrough the thinking-aloud tests, we observed user behavior, and through the interviews, we obtained comments and suggestions for design changes. These data were cross-referenced with Nielsen’s heuristics . Using tAlthough the web-based tests and interviews made for easy observation and recording, it limited our contextual understanding of the practical setup and the environment of the users in which they might use the program. Likewise, our aim for consistency in testing solely with a desktop setup did not necessarily match our patients’ use cases, pointing toward further investigations into the use of, for example, tablets and smartphones, when using the “My Symptoms” program. Thus, future research on improving the “My Symptoms” program and other studies alike could benefit from focusing on contextual inquires  with patIn usability engineering, it is well known that by understanding the user’s mental model, or schemata, we are able to come closer to a conceptual model  by whichAlthough the small sample size could be seen as a limiting factor, using the thinking-aloud method to highlight usability problems of immediate use is known to be an effective tool when having a small sample size. Because of the amount of detailed data the method provides, only 5-8 users are necessary to detect 80% to 85% of the usability problems ,17. FurtInterviewing the users before and after the thinking-aloud test enabled us to understand to which degree the users were accustomed to using web applications and IT in general. The interviews also gave us a chance to know how the patients were bothered by symptoms, helping us understand how the symptoms might affect the patients’ in-program behavior. Furthermore, the interviews helped us dive into issues that occurred during the thinking-aloud test, giving the users a way to suggest design changes retrospectively.Creating a digital self-help treatment program demands special attention to user-friendliness and intuitiveness. Program usability can make the difference between getting the needed treatment or not. User feedback helped improve the usability of the program and revealed how therapy sometimes was lost in the digital translation. This was especially relevant in relation to the themes: functionality, navigation, and content. Here, reducing the number of subpages, providing users feedback on tasks, and being sensitive to the framing of the content increased user satisfaction. Using the thinking-aloud method and heuristics in combination enabled us to upscale the specific design iterations into more broadly defined design statements that are applicable throughout the program. Furthermore, the usability testing helped facilitate knowledge sharing between different professions as a precondition for a successful program development facilitated by an iterative development process.


Then the characteristics of the impact of different earthquake-induced geohazards on vegetation were discussed, and a vulnerability-resilience assessment system concerning the seismic intensity was proposed. The results show that the NPPmax and NPPmean values in Jiuzhaigou Valley first decreased and then increased and were 151.5–261.9 gC/m2 and 54.6–116.3 gC/m2, respectively. The NPP value of more than 70% area was 90–150 gC/m2 in July. In August, the NPPmean values decreased, and the areas with lower values became larger; the NPPmean values of most areas affected by geohazards were 60–150 gC/m2. During the earthquake, the NPPmean values of areas hit by geohazards sharply declined by 27.2% (landslide), 22.4% (debris flow) and 15.7% (collapse) compared with those in the same month in 2016. Vegetation in debris flow zones showed a stronger recovery, with a maximum NPP value increase of about 23.0% in September 2017. The vegetation gradually recovered after the earthquake, as indicated by the uptrend of the NPP values in the corresponding period in 2018 and 2019. In general, the reduction magnitude of NPP values decreased year by year in comparison to that in 2015 and 2016, and the decrease slowed down after the earthquake. The vulnerability and resilience index corresponding to the three seismic intensity ranges were 0.470–0.669 and 0.642–0.693, respectively, and those of Jiuzhaigou Valley were 0.473 and 0.671, respectively. The impact coefficient defined to represent the impact of the earthquake on NPP was 0.146–0.213. This paper provides a theoretical reference and guidance for the impact assessment of earthquakes on the ecosystem.In order to assess the impact of the M The strong earthquake and geohazards induced by it, such as landslides, debris flows, and avalanches, seriously damaged the ecosystem and affected over 30% of vegetation there [Net primary productivity (NPP), an important metric of ecosystem growth, is greatly affected by natural events  and human activities . NPP wason there . The tres8.0 Wenchuan earthquake and found that the earthquake and the geohazards induced by it had a more obvious impact on the vegetation NPP in a short period. Although there were many studies on the relationship between NPP and climate change or human activities, the response of NPP to earthquakes has not been extensively investigated.Accurate estimation of terrestrial vegetation NPP is difficult due to the complex effects of various factors. With the development of the terrestrial ecosystem model (TEM) , the equ(1)The distribution and characteristics of NPP before and after the earthquake estimated by the CASA model were analyzed.(2)Based on the NPP results above, earthquake-induced geohazards  identified by field survey were considered for grasping the dynamic NPP changes caused by earthquake-induced geohazards. The features of the impact of each type of geohazard on the ecosystem were preliminarily discussed.(3)In accordance with the assessment methods of Hu et al. , the assIn this study, the data of three types of geohazards induced by the earthquake in Jiuzhaigou Valley were integrated to analyze the NPP response according to the following procedures:The study region was determined, and the earthquake-induced geohazards were identified through field surveys. In addition, relevant data were integrated, the estimation method and the impact factors were determined. The workflow of the research is presented in 2 and a vegetation coverage rate of 85.5%. The vegetation types include meadows, shrubs, coniferous forests, and broadleaf forests. With the terrain high in the south and low in the north, the elevation varies from <2000 m above sea level (a.s.l.) to around 4900 m a.s.l  occurred along the N-S-trending Minjiang faults [s7.0. Since Jiuzhaigou park is far from the epicenters, it was hardly influenced by the five strong earthquakes. However, the Jiuzhaigou earthquake induced landscape fragmentation and frequent geohazards. For example, one of the natural landscape heritage sites, Panda Lake, was seriously damaged by the earthquake-induced geohazards, with vegetation destruction . Static vegetation parameters , reclassSeveral multi-temporal high-resolution remote sensing images, including unmanned aerial vehicle (UAV) photography, Sentinel-2, Rapideye satellite imagery, and Google Earth history imagery (Landsat series), were acquired to identify the earthquake-induced geohazards. DEM data interpolated from large-scale contour maps were obtained from a previous study , and thehttps://zenodo.org/record/4417810 [http://data.cma.cn/ accessed on 25 September 2020). The NDVI data derived from the MODIS data can be downloaded from the Level-1 and Atmosphere Archive & Distribution System Distributed Active Archive Center (LAADS DAAC) with a spatial and temporal resolution of 250 m and 16 days (MOD13Q1) (https://ladsweb.modaps.eosdis.nasa.gov/search/ accessed on 25 October 2022). Multi-temporal data were obtained in July, August and September from 2015 to 2019 with view of the earthquake’s occurrence. To obtain the data of earthquake-induced geohazards, the satellite imageries were processed to eliminate interference from clouds, atmosphere, and solar elevation angle, among others; UAV remote sensing images were generated and calibrated. In order to intuitively analyze the five-year spatial-temporal change of vegetation NPP in the region struck by the Jiuzhaigou earthquake, the data in the Albers coordinates need to be Kriging-interpolated, extracted, and modified into the same element size . Since the methods of acquiring these data were described, they will not be specifically covered here.The preprocessing is necessary to fill missing data , deal with inconsistency (differences in raster), and eliminate noise (errors or outliers) on account of the heterogeneous datasets. The land cover data were obtained from China Land Cover Dataset (CLCD) (/4417810  accessed2) and the light energy utilization efficiency , which were as follows: x, t represent the pixel and month, respectively; SOL represents the total solar radiation of pixel x in t month obtained from meteorological data; FPAR represents the fraction of photosynthetically active radiation, defining a linear function of NDVI; SR, transformation of NDVI, represents the simple ratio, and the value of maxSR is determined by vegetation type and is related to SR when all downwelling solar radiation is intercepted [minSR represents the SR of unvegetated land areas remaining at 1.08 [optT (°C) represents the optimum temperature for vegetation growth during the month of maxNDVI at a certain extent; E1T (°C) and E2T (°C) denote the stress function of low temperature and high temperature to light energy utilization, respectively, among which E1T means the limitation of photosynthesis caused by plants’ biochemical effects at high and low temperatures, while E2T shows the decreasing trend of vegetation light energy utilization efficiency from Topt to critical temperature; T is the mean temperature of pixel x in t month; EW denotes the water stress coefficient varying from 0.5 to 1 that reflects the effect of available water conditions on light energy utilization efficiency (E); ET (mm) and pET (mm) represent estimated evapotranspiration and potential evapotranspiration obtained from MYD16A2 that are resampled into 1 km spatial resolution; maxΕ (gC/MJ) means the largest light energy utilization efficiency under ideal conditions proposed by Field et al. [The CASA model was a process-based remote sensing model driven by the global climate, radiation, soil, and vegetation index datasets. The model was not only suitable for simulating potential vegetation distribution and estimating NPP change but also appropriate for analyzing the relationship between them . In thisercepted , while S at 1.08 ; Topt and resilience (R) representing the degree of vegetation damage degree and restoration, which could be quantified by NPP reduction during the earthquake and the ratio of NPP increment in 2018 to dur-earthquake level, respectively. In addition, taking into consideration the advocation of the Jiuzhaigou Administration that “ecosystem natural restoration as the primary, manual intervention as an adjunct”, without considering the artificial intervention and hemeroby index mentioned by Winter et al. [I) of quake-induced geohazards on vegetation without self-recovering was proposed, and it can be calculated by Formula (9).A function of vulnerability and resilience of the sites where earthquake-induced geohazards occurred could be applied for NPP assessment. Thywissen  listed vr et al.  and Zinnr et al. , the imps7.0 earthquake caused instantaneous damage to the ecosystem and was devastating. Therefore, this research principally analyzed the driving factor that affected NPP in a short time, namely seismic intensity (SI). The SI could be determined by referring to “the distribution map of the Ms7.0 Jiuzhaigou earthquake and the characteristics of the earthquake damage” released by the Sichuan Earthquake Administration, from which we knew that the seismic intensity in the whole study region is VII+.Scholars have shown that NPP change was mainly attributed to natural factors such as temperature and precipitation ,34,35, wNPP(x), NPP*(x) and NPP**(x) are the NPP change values of pixel x during the month when the earthquake occurred, in the month after the earthquake and in the month before the earthquake, respectively; N*(x) is the normalized NPP change value of pixel x; maxN and minN are the maximum and minimum value of NPP change within the whole region.Vegetation NPP changes ) within one month after the earthquake were mainly attributed to earthquake damage, seasonal turnover, and self-restoration, among which earthquake damage and seasonal turnover had negative effects on NPP, while self-restoration exerted positive effects. Thus, we reclassified the NPP change into positive part ). The calculating method for the resilience index R was almost the same as that for S, but there were differences in the month selection. The computed normalized values were divided into 0–0.2, 0.2–0.4, 0.4–0.6, 0.6–0.8, 0.8–1 (5 groups), with the gradient being 0.2, and then the area proportion of each group was calculated. Due to the principle of class mid-value, the median value  was taken as coefficient 2) and study period, the data, each of which equally accounted for 30 gC/m2, were divided into nine groups for refining values of pixels.Corresponding data were imported into the CASA model to estimate the NPP value of the study region, and values in July, August and September of 2016, 2017 and 2018 are displayed in From As the NPP in the west and south of the study region changed significantly during the earthquake and had a broad range within adjacent pixels that existed in the high-altitude area, the area proportion represented by vegetation NPP of the whole study region was significant for evaluating the ecosystem of Jiuzhaigou Valley. Therefore, the NPP values were classified based on the number of pixels, and the area proportions are shown in 2 accounted for nearly 50% of the whole study region, and the maximum value range was found in July 2017 and 2018. In addition, the NPP level in July 2018 is equivalent to that before the earthquake, indicating that the ecosystem was recovering continuously after the earthquake. The NPP value of more than 70% of the study region was 90–150 gC/m2, which was consistent with the finding of Zhang et al. [2, and the color block indicating the maximum range did not appear. However, a larger value range appeared in August the next year, which proved that grasslands and arbor forests with higher productivity showed stronger resilience [2 in September 2018.From g et al. . In Augusilience . In Septsum represented the total NPP value of the vegetation ecosystem. Owing to the diversity of the ecological system, NPPsum values are comprised of maximum (NPPmax) and minimum values in the region according to zonal analysis. The NPPmax value was taken to reflect the maximum intensity of vegetation productivity. Moreover, NPPmean, as the average value of the ecological productivity quality, was used to reflect the overall productivity level of plants within a specified zone. The NPPmax and NPPmean values from 2015 to 2019 are shown in NPPmax showed an overall decreasing trend with seasonal change in each year. During the earthquake, the NPP reached the lowest value of 199.9 gC/m2 and only showed an increase of 2.8 gC/m2 during the month after the earthquake, which was attributed to the aftershocks. The NPPmean value was around 110.1 gC/m2 in July, 98.4 gC/m2 in August and 71.9 gC/m2 in September. Its variation was similar to that of NPPmax. The NPPmean value, less affected by earthquake-induced geological disasters, also quantitatively represented the damage and restoration of vegetation before and after the earthquake and can reveal the change law of vegetation NPP. Compared with the NPP values in 2015 and 2016, those in the same month in 2018 and 2019 were higher, and the integral decline decreased year by year. It is indicated that the vegetation was reconstructed by natural restoration (restoration of soil fertility and species diversity) and artificial intervention  [max value reached the highest level (261.9 gC/m2) in August 2019, suggesting that the ecosystems were rebuilt. The NPP values show an overall downward trend and then an upward tendency.From ivities) , which cGeohazards in gullies and valleys can further cause vegetation destruction. Geohazards are classified in terms of their causes. Landslide is attributed to the instability of rock-soil mass of a high-steep slope and overall sliding along the weak surface. Debris flow is caused by heavy rainfall, after which the slope runoff carries the loose soil with large pores and gradually converges into a flood with fast speed and large flow. Collapse is a rolling phenomenon that often occurs on steep slopes or reverse slopes. Based on the classification above, 128 geohazards identified by field survey that caused damage to vegetation were divided into landslides, collapses and debris flows.In light of the area where geohazards occurred, landslide and collapse were integrated, and debris flows mainly occurred in Shuzheng Gully, Zharu Gully, Heye Gully, Zezhawa Gully and Xiajijiehai Gully. By combining the DEM data, the areas affected by representative geohazards are shown in The study shows that the area affected by debris flow is larger than those affected by collapse and coseismic landslide. According to the land cover database, the land cover type in five debris-flow gullies in the study region was mostly needle-leaved evergreen forest with a small proportion of alpine and sub-alpine meadow and plain grassland in Zharu Gully. The land cover type in areas affected by collapse and coseismic landslide was the needle-leaved evergreen forest. In addition, the land cover type in the central and northern areas was needle-leaved evergreen forest, where the NPP value in the same year shows a rising tendency with time.mean change caused by the three representative geohazards from 2015 to 2019 was analyzed . Additionally, the NPPmean values of landslide and collapse zones were generally higher than the values of debris flow zones. During the earthquake, the values sharply declined by 27.2% (landslide), 22.4% (debris flow) and 15.7% (collapse) compared with those in the same month in 2016, and the largest decline was caused by landslides. After the earthquake, the NPPmean values began to rebound and vegetation generally recovered, and the values became close again during the study period. The vegetation in debris flow zones showed a stronger recovery than the vegetation in landslide and collapse zones, with a maximum NPPmean increase of about 23.0% compared with the corresponding value in August 2017. Through the analysis of the above results, it could be preliminarily concluded that, to some extent, geohazards had a negative correlation with vegetation NPP change.From NPP values corresponding to each seismic intensity were obtained, and vulnerability indices were calculated, as shown in In accordance with the designed assessment system, the quantitative change of NPP before and after the earthquake was calculated by Formula (10). After normalization (Formula (11)), the R) one month after the earthquake was 0.642–0.693, and the resilience (R) of the whole study region was 0.671. At an intuitive ecological level, resilience was determined by the natural restoration of the ecosystem, and the results had a high referential value. However, the vulnerability (S) was 0.470–0.669, with the range slightly broader than that of resilience, but the vulnerability (S) of the whole study region was 0.473 and was within the normal range. The impact coefficient (I) calculated by Formula (9) was 0.146–0.213. Also, the vulnerability and resilience indices corresponding to a single seismic intensity range remained similar, and the indices corresponding to the IX+ intensity were markedly different from those corresponding to other intensity ranges. In fact, the seismic intensity was affected by the epicentral distance and the smaller the epicentral distance, the greater the intensity [From ntensity . The magntensity ,40, and In the process of studying the damage degree and recovery level of vegetation before and after the earthquake, it was found that almost all (98.98%) vegetation in IX+ seismic intensity zones was damaged; the vegetation in VIII–IX seismic intensity zones was slightly less damaged (94.84%); VII–VIII seismic intensity had the least impact on NPP (82.79%). Vegetation in IX+ seismic intensity zones recovered best (93.12%) . It coulThe possible reason why landslides and collapses cause more serious damage is that large plants with greater productivity are more able to grow in areas with relatively dense soil structures. As to the stronger recovery of vegetation in debris flow zones, by checking the relevant land cover database, it was found that the major vegetation types in the five debris-flow gullies are shrub and grassland . It is ds8.0 Wenchuan earthquake in 2008 caused enormous damage to vegetation and GPP of the severely afflicted area dropped by 22% but then recovered [s7.0 Jiuzhaigou earthquake triggered various geohazards, which destroyed ground surface plants and decreased NPP value by around 23%. Fortunately, the vegetation in debris-flow gullies and landslide-and-collapse areas after the earthquake was restored despite the aftershocks, and the NPP value returned to a nearly normal level after one year. Furthermore, as a strong dynamic indicator of terrestrial vegetation coverage [In recent years, little attention was paid to the relationship between NPP and earthquakes. However, as an indicator of respiration and photosynthesis of ecological vegetation, vegetation gross primary productivity (GPP), comprised of autotrophic respiration of plants and NPP, was combined with MODIS data, and it was found that the Mecovered . Nine yecoverage , NDVI alcoverage ; GPP andcoverage . From thcoverage ,47; the coverage  and the maxSR and maxΕ required in this model were determined in the research of Zhu et al. [With the NPP estimation method adopted in this research, the impact of the earthquake was considered, various types of subsequent hazards and their influencing areas were identified, and combined with the CASA model, the impact of the earthquake on NPP change could be preferably analyzed. However, there are some limitations concerning data precision. The maximum composition of 16-day MODIS NDVI data was synthesized from two sets of original NDVI data of one month, and the final NDVI data of one month was synthesized from two sets of original NDVI data of that month, which cannot accurately represent the NDVI data at the time of the earthquake. As to the interpolated meteorological data, the smooth operation of the Kriging method requires quantities of data, which avoid the problem that the semi-variogram could not be estimated. In this paper, u et al. , where tu et al. ,50. But Despite the limitations, this research was valuable as it quantitatively assesses the impact of earthquake-induced geohazards on NPP. Future research could explore more factors, such as NDVI influencing vegetation vitality and NDWI  influencing vegetation water content . Finally(1)2 in July. In August, the value showed an overall decrease, and the areas with lower values became larger; the NPP value of most areas was 60–150 gC/m2.The NPP value of more than 70% area was 90–150 gC/m(2)max and NPPmean were 151.5–261.9 gC/m2 and 54.6–116.3 gC/m2, respectively, in July, August and September from 2015 to 2019. The vegetation gradually recovered after the earthquake. The NPP value showed an uptrend in the corresponding periods in 2018 and 2019. The integral decline amplitude decreased by years in comparison to that in 2015 and 2016. The decrease slowed down after the earthquake.NPP(3)During the earthquake, compared with the NPP value in the same month in 2016, the value in areas affected by geohazards sharply declined by 27.2% (landslide), 22.4% (debris flow) and 15.7% (collapse). The vegetation in debris flow zones showed a stronger recovery, with a maximum increase of about 23.0% in September 2017.(4)In the aspect of the assessment system, the resilience index corresponding to the seismic intensity ranges one month after the earthquake was 0.642–0.693, and that of the whole study region was 0.671. The vulnerability index corresponding to the seismic intensity ranges was 0.470–0.669, and that of the whole study region was 0.473. The impact coefficient defined to represent the impact of the earthquake on NPP was 0.146–0.213.From an ecological and geographical perspective, this study combined the NPP value of seismic areas obtained by the CASA model with the spatial distribution data of geohazards collected by field surveys to explore the effect of seismic intensity on vegetation destruction. The conclusions are as follows: