Herein, we focus the analysis on the 2007 Trichonis Lake, the 2016 Western Crete, the 2021–2022 Nisyros, the 2021–2022 Thiva and the 2022 Pagasetic Gulf earthquake swarms. Using the seismicity catalogs for each swarm, we investigate the inter-event time (T) and distance (D) distributions with the Q-exponential function, providing the Tq and Dq entropic parameters. The results show that Tq varies from 1.44 to 1.58, whereas Dq ranges from 0.46 to 0.75 for the inter-event time and distance distributions, respectively. Furthermore, we describe the frequency–magnitude distributions with the Gutenberg–Richter scaling relation and the fragment–asperity model of earthquake interactions derived within the NESP framework. The results of the analysis indicate that the statistical properties of earthquake swarms can be successfully reproduced by means of NESP and confirm the complexity and non-additivity of the spatiotemporal evolution of seismicity. Finally, the superstatistics approach, which is closely connected to NESP and is based on a superposition of ordinary local equilibrium statistical mechanics, is further used to discuss the temporal patterns of the earthquake evolution during the swarms.Greece exhibits the highest seismic activity in Europe, manifested in intense seismicity with large magnitude events and frequent earthquake swarms. In the present work, we analyzed the spatiotemporal properties of recent earthquake swarms that occurred in the broader area of Greece using the Non-Extensive Statistical Physics (NESP) framework, which appears suitable for studying complex systems. The behavior of complex systems, where multifractality and strong correlations among the elements of the system exist, as in tectonic and volcanic environments, can adequately be described by Tsallis entropy ( M 1.0 to M 5.0 and rarely produce earthquakes with large magnitudes, i.e., M ≥ 6 [The clustering of earthquakes in both time and space without a prominent large-magnitude earthquake is commonly referred to as an earthquake swarm. Swarms can last for several days, weeks or months, registering many earthquakes within a small volume. According to , when th., M ≥ 6 . In thesGreece is located on a tectonically active plate boundary at the convergence of the Eurasian and African lithospheric plates south of Crete, forming the Hellenic arc . The Hel2 emissions as the 2012–2014 microseismic activity in Florina [Apart from the background seismic activity and the occurrence of strong events accompanied by pronounced aftershock sequences, frequent earthquake swarms also occur in the area of Greece. Some characteristic cases come from the active continental Corinth Rift, such as those of the 2001 Agios Ioannis swarm ,18, the  Florina ,26. Furt Florina .Q-exponential function in agreement with NESP, as proposed in a number of cases [Tq, Dq, Mq). We demonstrate that NESP is an adequate and methodological tool for analyzing complex systems, such as the spatiotemporal evolution and the FMD of earthquake swarms. In addition, based on the observed scaling properties of seismicity, we describe the temporal evolution of the swarms using superstatistics [In the present work, we study the physical characteristics of recent earthquake swarms that have occurred in the broader area of Greece and in different seismotectonic settings, such as the earthquake swarms of Trichonis Lake (8 April 2007–2 July 2007), Western Crete (1 February 2016–25 March 2016), Nisyros (7 April 2021–30 June 2022), Thiva (10 July 2021–1 July 2022) and Pagasetic Gulf (9 May 2022–23 June 2022) . Particuof cases ,35,36 foof cases ,38,39,40of cases . This statistics ,43,44 thNon-extensive statistical physics (NESP) is a generalization of Boltzmann–Gibbs (BG) statistical physics and has been used to describe complex dynamic systems that exhibit scale-invariance, (multi)fractality, long-range interactions and long-term memory effects, leading to broad distributions with power-law asymptotic behavior ,45,46,47Sq) which is expressed as:Bk is Boltzmann’s constant and q the entropic index that signifies the non-extensivity of the system. In the following, we address the NESP theory for a continuous variable X that may express the inter-event times (T), i.e., the time intervals between the successive seismic events, or the inter-event distances (D), i.e., the three-dimensional Euclidean distance between the foci of successive seismic events. In 1988 ,46, Tsalq = 1, then Sq = BGS and the approach reduces to well-known Boltzmann–Gibbs (BG) entropy. Despite the fact that Sq and BGS have many similar properties in common, such as non-negativity, expansibility and concavity, there is a distinctive difference between the two entropies. The Boltzmann–Gibbs entropy is additive, meaning that the entropy of a combined system is the sum of the entropy of the separated parts, whereas the Tsallis entropy Sq is non-additive. According to this property and for two probabilistic independent events A and B, the total entropy Sq of the system A + B satisfies:q < 1, sub-additivity if q > 1 and additivity when q = 1 (Boltzmann–Gibbs statistics) where the last term on the right-hand side of Equation (2) vanishes, and the additivity property is recovered. This has been recently discussed in terms of the entropy defect [In the case of y defect .p(X) of the continuous variable X  is acquired by maximizing the non-extensive entropy Sq under appropriate constraints [Zq refers to the q-partition function defined as:Xq is a generalized scaled inter-event time or inter-event distance, while the q-exponential function is defined as:For seismic events, the probability distribution straints  using th expqX=0 .CDF) P (>X) should be obtained upon integration of the escort probability distribution qP (X) [P (>X) is estimated by the integration of physical probability p(X) (Equation (3)) instead of the escort probability distribution qP (X), then:Q-exponential function, defined for q < 2, with q and X0. Moreover, the variable X refers to the inter-event times (T) or distances (D), while the q-value describes the spatiotemporal evolution and the degree of correlations, with Tq > 1, for the inter-event times, and Dq < 1 for the inter-event distances, respectively, according to [The corresponding to Equation (3) cumulative distribution function (n Pq (X) ,46,47,49 q=2−1/Q ,51,52,53rding to ,35. Q-exponential function (Equation (7) for X > 0) leads to the Q-logarithmic function [q > 1, the Q-exponential function presents asymptotic power–law behavior, while for q < 1 a cut-off appears in the tail of the distribution. In the limit when q = 1, the Q-exponential and the Q-logarithmic recover the ordinary exponential and logarithmic expressions, respectively.The inverse of the function ,49,54:) is a straight line, with slope −1/X0 [The latter Equation implies that after the estimation of the appropriate pe −1/X0 .M represents the earthquake magnitude, N(>M) is the cumulative number of earthquakes with a magnitude equal to or greater than M and a, b are positive scaling parameters. Parameter a describes the regional level of seismicity and shows significant variations from one area to another. Parameter b, also known as the b-value, is the slope of the frequency–magnitude distribution (FMD) and describes the size distribution of the earthquake events. The estimated b-value usually has typical values close to 1 [b-values (up to 3) associated with volcano areas [The Gutenberg–Richter scaling relation  is one oose to 1 ,56,57, eno areas .–asperity model of earthquake interactions, which describes the earthquake generation mechanism within the NESP context. This model takes into account the interaction of two rough profiles (fault blocks) and the fragments filling the area in between them, originating from the local breakage of the tectonic plates. This interaction explains how earthquakes are triggered. These authors [–asperity model has been used in a variety of applications, including regional and local seismicity, as well as volcanic seismicity [Sotolongo-Costa and Posadas  develope authors  applied ismicity ,60,61.M) is related to the relative seismic energy and by taking into account the threshold magnitude cM [M is the earthquake magnitude, cM is the threshold magnitude, A is proportional to the volumetric energy density and Mq is the entropic index [Mq indicate that the physical state of a seismic area moves away from equilibrium [Telesca  revised itude Mc , proposeitude Mc :(10)logNic index . Temporailibrium .b-value calculation using the maximum-likelihood approach b=2−h  is rathon of M0 .M ≥ 5.5 [In the present work, we analyze the spatiotemporal scaling properties and the frequency–magnitude distribution of earthquake swarms in terms of Tsallis Entropy. The location of the five earthquake swarms that we study in this work are illustrated in M ≥ 5.5 . The stuwM 5.6), whereas on 21 December 1975 an wM 5.5 event was followed by another one on 31 December 1975 with a magnitude of wM 6.0 [Trichonis is the largest natural lake in western Greece which strikes WNW–ESE for a distance of about 32 km and has a width of about 10 km . More spf Mw 6.0 . Other sf Mw 6.0 .wM 4.9, 4.9 and 5.2, respectively. The seismic activity continued for a month with smaller magnitude events forming a swarm [wM 4.8. It was shown that this seismic activity did not correlate with any of the two fault zones at the northern and southern edges of the lake, but with two unmapped NNE-SSW and NW-SE faults along its eastern shore [SL-NKUA) for the period between 8 April 2007 to 2 July 2007, counting a total of 1309 events. The minimum and maximum magnitude and depth of the catalogue is LM 1–5.2 and 0–19 km, respectively. In the southeastern part of the lake, an intense seismic sequence, with a series of relatively strong earthquakes, started in April 2007. The sequence initiated with small events on 8 April and two days later the three strongest events of the entire sequence occurred. More analytically, on 10 April three strong earthquakes occurred at 03:17:56, 07:13:03 and 10:41:00 UTC with magnitudes  a swarm ,70. Anotrn shore ,70,71. Arn shore , the exiThe island of Crete is located in a fore-arc position above the active northward-directed subduction zone of the African plate beneath the Aegean plate , where tLM) varies from 1 to 4.8, whereas the depth of the events is 0–26 km.The data for Western Crete were retrieved from the Institute of Physics of the Earth’s Interion and Geohazards. In this work, we studied the seismicity in Western Crete, near the village Platanos between 1 February 2016 and 25 March 2016. For the study period, the catalogue contains a total of 653 events. The strongest earthquake took place on 12 March 2016 with a magnitude of 4.8. The magnitude  to study the seismicity in the northwest area of Nisyros between 7 April 2021–30 June 2022. For this period the catalogue involves a total of 1567 events. The first of the strongest earthquakes took place on 13 April 2021 (LM 5.2), whereas on 21 June 2021 an LM 5.7 event was followed by another on 1 August 2021 with a magnitude of LM 5.4. The minimum and maximum depth of the events is 2–23 km. The earthquake catalogue was extracted from the Geodynamics Institute of the National Observatory of Athens , located at the transition zone between the Corinth Gulf in the south and Evia in the east. Two major rift structures, oriented WNW—ESE and NW-SE, respectively, dominate the tectonics of the area . The lan in 1914 .wM 4.3, 4.1, 4 and 4.4, respectively. The earthquake catalogue was obtained from the Seismological Laboratory of the National and Kapodistrian University of Athens (SL-NKUA). For the above period, the catalogue contains a total of 4695 events with depths from 0 to 15 km. The 2020 seismic events mainly took place at the eastern part of the study area, on a system of normal faults. On 10 July 2021, seismic activity started at the western part of the swarm and the next seismic events present a general tendency for spatiotemporal migration towards ESE. The evolution of the swarm is related to stress triggered by its major events and facilitated by pore-fluid pressure diffusion [In this part, we studied the seismicity in the area of Thiva between 10 July 2021 and 1 July 2022. On 2 December 2020 at 10:54:56 UTC, an earthquake with magnitude 4.5 took place east of Thiva, which was followed by a prolific seismic swarm until recently. The next strongest events were on 11 July 2021, 20 July 2021, 2 September 2021 and 10 April 2022, with magnitudes iffusion .The Pagasetic Gulf is a semi-enclosed gulf located in the northwestern part of the Aegean Sea and is connected with North Evoikos Gulf and the Aegean Sea through the channel of Trikeri. The study area is north of the island of Evoia and southeast of the city of Volos and Ayia Kiriaki, as it is shown in GI-NOA) and consists of 283 events. This swarm lasted less than two months between 9 May 2022 and 23 June 2022, with the strongest seismic event on 11 May 2022 with a magnitude LM 3.8. The depth of the events ranges from 2 to 20 km. The earthquake catalogue was extracted from the Geodynamics Institute of the National Observatory of Athens (cM) using the FMD in terms of the Gutenberg–Richter scaling relation. The measurements were made by applying the Zmap software package, version 7.0, [a and b values of the Gutenberg–Richter relation and the completeness magnitude (cM) in each catalogue. The estimated values are given in M ≥ Mc. In this paragraph, using the catalogues of each earthquake swarm, we estimated the magnitude of completeness  − t(i), where t(i) is the time of occurrence of the ith event, i = 1, 2, …, N − 1 and N is the total number of events, whereas the inter-event spatial distance is defined as the 3D Euclidean distance between the successive hypocenters. After the estimation of the cumulative inter-event time distribution P(>T), the corresponding fitting with the Q-exponential function, up to the value cT indicates the value Tq. The parameter cT  denotes the transition points between the non-additive and additive behavior. In most of the cases that we study, we observe the deviation from the Q-exponential function for high values of T, with T > cT. The crossover points between the non-additive and additive behavior are indicated by the deviation from linearity at T > cT in the Q-logarithmic functions lnQP(>T). Following the same procedure, we estimate the cumulative interevent distances distribution P(>D) that provides the Dq parameter value. After the calculation of the appropriate q that describes the observed distributions P(>T) and P(>D), the Q-logarithmic functions lnQP(>T) and lnQP(>D) are linear with T (T < cT) and D, respectively.The theory of NESP, as previously described, is herein applied to the inter-event time and distance distributions, as well as to the seismic energy distributions of the earthquake swarms. The catalogues were updated to include all earthquake events with P(>T) of the inter-event times T for each earthquake swarm. The Q-exponential fitting uses the generalized expression of entropy, Equation (1) describes the observed distributions quite well, leading to the parameters Tq and qT (Q-logarithmic distribution lnQ(P(>T)) for each earthquake swarm is shown in the middle panels of T) as a function of time (t) is shown in the right panels of cT value) shows that most inter-event times have T values less than cT, indicating that the Tsallis entropic mechanism is predominant in the main part of the swarm. However, as time passes the characteristics of the earthquake swarm such as those of finite degree of freedom and long-range memory, related to a non-extensive statistical physics description are not any more predominant and the Boltzmann-Gibbs statistical physics is recovered .The results of the analysis are presented in T and Tq . The traQ-exponential distribution describes well the observed scaling behavior for the values of Tq = 1.44 and qT = 1805 s and up to a characteristic time cT = 6.8 × 104 s, where a fall-off in the distribution appears, probably due to the finite size of the swarm, limiting the occurrence of sporadic events and hence the emergence of long inter-event times (finite-size effect). Similarly, for the 2016 Western Crete and the 2021–2022 Nisyros swarms, the Q-exponential distribution describes quite well the observed P(>T) for the values of Tq = 1.53, qT = 633 s, cT =4.3 × 104 s and Tq = 1.58, qT = 1625 s, cT = 1.7 × 105 s, respectively. In addition, for the 2021–2022 Thiva and the 2022 Pagasetic Gulf swarms, the Q-exponential fitting using Equation (7) describes the observed data well, leading to Tq = 1.47, qT = 1736 s, cT = 2.9 × 105 sec and Tq = 1.57, qT = 829 s, cT = 1.1 × 105 s, respectively. The higher than unity Tq-values indicate asymptotic power-law behavior and long-range correlations in the temporal evolution of the earthquake activity [Q-logarithmic distributions for each earthquake swarm, describe the observed distributions with high correlation coefficients, as shown in the middle panels  of inter-event distances D are presented, showing a good fit with the Q-exponential function for Dq < 1. The q and qD values estimated from the analysis are presented in Q-logarithmic distributions lnQ(P(>D)) are also plotted in the middle panel of Q-exponential function that approaches linearity with a ρ correlation coefficient. The estimated Tq or Dq parameters give the best linear fitting. In Q-exponential distributions for the 2007 Trichonis Lake, the 2016 Western Crete and the 2021–2022 Nisyros swarms, describe well the observed scaling behavior for the values of Dq = 0.75, qD = 7.5 km and Dq = 0.46, qD = 8.62 km and Dq = 0.48, qD = 16.67 km, respectively. Furthermore, for the 2021–2022 Thiva and the 2022 Pagasetic Gulf swarms, the Q-exponential distributions describe well the observed P(>D) for the values of Dq = 0.67, qD = 5.47 km and Dq = 0.46, qD = 8.46 km. The interevent distances of the seismic activity deviate from the exponential function, indicating organization rather than random spatial occurrence [Q-logarithmic distributions approximate linearity with high correlation coefficients, indicating the goodness-of-fit between the model and the data. According to our results, the cumulative distribution functions of inter-event times and distances are quite well described with the Q-exponential function, with an entropic parameter q greater than one (q > 1) for inter-event times and less than one (q < 1) for the inter-event distances, further confirming the results of [The currence . The corsults of ,82,83.–2022 Nisyros, the 2021–2022 Thiva and the 2022 Pagasetic Gulf earthquake swarms, while the fitting of Equation (10) to the observed data leads to Mq = 1.49, Mq = 1.48, Mq = 1.48, Mq = 1.45, Mq = 1.48, respectively  to the normalized cumulative magnitude distribution of each earthquake swarm , for theectively . In Figuntervals . The aboMq values between 1.45–1.49 for the studied earthquake swarms. These results suggest that the areas are away from equilibrium in a statistical physics sense, which means that the fault planes and fragments filling the gap between them are not in equilibrium, leading to an increased seismic activity to be expected [Mq are observed when small-magnitude earthquakes occur, as in the case of Thiva (Mq = 1.45). This could reveal that the order within the system of faults is decreasing, and the amount of accumulated stress is not yet enough to initiate a correlated behavior of the whole system [M are found in the regions where large earthquakes have occurred, which is in agreement with previous results [Mq [Mq = 1.49. The distributions reflect sub-extensive systems, where long-range interactions are significant ,84. The expected . The dece system . High va results ,84,86,87Q-exponential function (Equation (1)). In the case of inter-event times, a deviation from the Q-exponential function appears for high values of T (cT > T), where cT indicates the transition between the non-additive and additive behavior. For each earthquake swarm, we estimated the entropic parameter Q by fitting the Q-exponential function to the observed data up to a value close to cT and then we calculated the parameter q from the equation Q-exponential distribution and the corresponding Dq values are less than unity Dq < 1, whereas the cumulative distribution of the inter-event times is described by the Q-exponential function with Tq > 1. They also proposed the relation: P(>T) and inter-event distances P(>D) are adequately described with the Q-exponential function, with T > 1q and D < 1q, respectively. The latter indicates a correlated process in time and space that deviates from the random case and the exponential distribution. In addition, the inter-event times exhibit an asymptotic power-law behavior, whereas for inter-event distances a cut-off appears. In this last case, a cut-off in the inter-event distances distribution seems to be the appropriate scaling behavior for real data [q < 1 has been verified in previous earthquake studies [q-values that characterized the observed inter-event time distributions are within the range of 1.44–1.58, whereas for inter-event distances are from 0.46 to 0.75. The entropic parameters Tq,Dq,qT and qD along with their 95% confidence intervals and the critical parameter Tc, are summarized in Tq and Dq parameters of the inter-event time and distance distributions for each earthquake swarm is approximately The concept of Non-Extensive Statistical Physics (NESP) is herein applied to the inter-event time and distance distributions of recent earthquake swarms that have recently occurred in Greece. We observed that the cumulative distribution functions of the inter-event times and distances between the successive earthquakes for all the studied swarm sequences are well described by the d Suzuki ,35, the nd Japan ,35 and Ind Japan , and vernd Japan ,90. Furtnd Japan ,83,91. Oeal data  due to teal data . This sc studies ,35,83,91Q-exponential function, indicating asymptotic power–law behavior and long-term correlations in the spatiotemporal evolution of seismicity. In addition, the q-values of Tq > 1 and Dq < 1 are in agreement with the q-values found for aftershock sequences [Tq > 1 suggests a sub-additive process, leading to the conclusion of long-range memory in the evolution of earthquake swarms for c.T < T The framework of NESP that was used in the present study describes well the spatiotemporal scaling properties of the earthquake swarms, as well as the seismic energy distributions. Hence, the application of non-extensive statistical physics represents a quite useful tool in investigating such phenomena, exhibiting scale-free nature and long-range memory effects. The cumulative inter-event time and distance distribution functions for each earthquake swarm are well described with the equences ,83,93,94equences , for theequences ,61 and fequences ,84. In aequences ,18,26,95equences ,59. MoreQ-exponential function for high T values could be explained by introducing two mechanisms that drive the earthquake swarms. The first one is governed by Tsallis entropy and is dominant for inter-event times with cT < T, whereas the second is observed for cT > T and is defined by an exponential function . Thus, in order to include a crossover from non-additive (q ≠ 0) to additive (q = 1) behavior, we apply a generalization of anomalous equilibrium distributions described in [C is a normalization factor and for positive qβ and β1, p(T) decreases monotonically with increasing T. In the case where q-exponential, The observed deviation from the ribed in ,97, wherbehavior .Q-exponential behavior of the inter-event times can be observed in terms of superstatistics, which are based on a superposition of ordinary local equilibrium states, controlled by an intensive parameter (β) that fluctuates on a relatively large spatiotemporal scale and is supplementary to NESP [T|β)T|β) expressing the conditional probability density of the inter-event times given that it has an average value T|β and the intensive parameter β that fluctuates on a relatively large scale, leading the exponential model to become superstatistical. If the parameter β is distributed with probability density f(β), then the probability distribution is given as:n Gaussian random variables X1, ..., Xn of the same variance due to various relevant degrees of freedom in the system [β is to square the Gaussian random variables and sum them up. Hence, as a result Many complex systems exhibit inhomogeneous spatiotemporal dynamics that can be characterized by a superposition of numerous generalized statistics on different scales, called ’superstatistics’, which is complementary to NESP ,43,44. S to NESP ,42,43,44β is X2-distributed with n degrees of freedom:β0 is the average of β.The probability density of C is a normalization factor and The integration in Equation (14) using Equation (15) has as a result the generalized canonical distribution of NESP:n degrees of freedom that are influencing the value of β, can be performed according to [Tq value of each earthquake swarm  behavior, such as in the case of Trichonis Lake with n = 3, whereas the low degrees of freedom deviate from the random case and indicate a correlated process in time and space.The calculation of the rding to : (17)n=2ke swarm , acquireke swarm . The degQ-exponential function up to a characteristic inter-event time cT. The range of the Tq parameter is found between 1.44 and 1.58, supporting the idea of the presence of long-range memory effects in the evolution of seismicity. In addition, the inter-event distances distributions are described with the Q-exponential function with Dq-values below unity and in the range of 0.46–0.75, leading to a cut-off in the tail of the distribution. The sum of these parameters that describe the inter-event times, Tq > 1, and distances, Dq < 1, distributions, is Mq in the range of 1.45 to 1.49, reproduces the complexity in the earthquake energy distribution quite effectively. Lastly, we applied a superstatistic approach, which is based on a superposition of ordinary local equilibrium statistical mechanics with an appropriate intensive parameter β that fluctuates as χ2 distribution on a rather large temporal scale. The superstatistical approach leads to the conclusion that in each earthquake swarm, the temporal evolution is described by low degrees of freedom, indicating a high level of organization, hierarchy and non-additive characteristics. In the present work, the spatiotemporal scaling properties and the frequency-magnitude distribution of earthquake swarms that have recently occurred within the area of Greece were investigated using the framework of Non-Extensive Statistical Physics (NESP). The analysis and results indicate power-law asymptotic behavior in the inter-event time distributions that scale according to the


Effective interventions are needed to address suicide risk following discharge from the hospital emergency department or inpatient setting. Studies that examine follow-up contact methods show promise, but little is known about how follow-up programs are implemented in the real world and who is benefitting. The purpose of this formative evaluation and analysis was to gain insight about the usefulness and value of a partnered suicide prevention follow-up program  from the standpoint of psychiatry resident physicians providing direct care and suicide prevention center crisis counselors making follow-up outreach telephone calls to patients.A qualitative thematic analysis was conducted with focus group data from a convenience sample of psychiatry residents who performed consultations in the emergency department setting and counselors at the suicide prevention center crisis follow-up program. Focus group sessions, using semi-structured question guides, were completed at each participant group’s workplace. Grounded theory techniques were used to guide coding and analytic theme development.Analyses resulted in four overarching themes: valuing the program’s utility and benefit to patients, desiring to understand what happens from emergency department discharge to program follow-up, having uncertainty about which patients would benefit from the program, and brainstorming to improve the referral process. Psychiatry residents appreciated the option of an “active” referral service (one that attempts to actively engage a patient after discharge through outreach), while suicide prevention crisis counselors valued their ability to offer a free and immediate service that had potential for fostering meaningful relationships. Both participant groups desired a better understanding of their partner’s program operations, a uniform and smooth referral process, and awareness of who may or may not benefit from program services.Results revealed the need for improved communication and implementation, such as expanded inter-agency contacts, consistent provider training, more documentation of the requirements and rules, a consistent message about program logistics for patients, and coordination between the program elements. In the United States (U.S.), suicide is a leading cause of death across the lifespan, accounting for nearly 46,000 deaths in 2020 . The NatThe high prevalence of suicide during this critical period has created a demand for effective interventions following hospital and ED discharge. A variety of interventions, including ED-initiated follow-ups, have been proposed or implemented in limited settings. Follow-up communication has included a variety of methods including phone calls, postcards, in-person, apps, e-mail, and text messaging , 26, 35.Building on this approach, this article describes a formative, qualitative evaluation of an ongoing suicide prevention partnership that was initiated in 2013. The partnership was formed between an academic hospital ED and an independent regional suicide prevention center (SPC), which established a follow-up program for individuals at high risk for suicide in 2010 through seed funding from the US Department of Health and Human Services Substance Abuse and Mental Health Services Agency (SAMHSA). The goal of the partnership is to provide follow-up services for patients who are at risk for suicidal ideation and/or behaviors during the period following hospital or ED discharge via telephone outreach from counselors at the suicide prevention center’s crisis follow-up program (SPC program). Figure The inter-agency  nature of this partnership and the referral system is unique in comparison to prior studies. Several prior documented programs have focused on follow-up programs that are owned and operated with the hospital’s own staff. Potential benefits of the inter-agency model include improved patient outcomes due to being connected to specialty trained counselors in the community based SPC program, decreased hospital readmission rates, decreased programmatic costs to hospitals, as well as knowledge to improve scalability to other facilities. Potential risks include errors in communication, lack of continuity of care, and challenges in monitoring data and outcomes.Evaluation of the partnership began as a quality improvement (QI) project using an inter-agency system framework that combined QI and Exploration, Preparation, Implementation, and Sustainment (EPIS) to inform implementation sciences and partnership of systems . This st1) Whether or not services addressed specific population needs,2) How the inter-agency suicide prevention follow-up program was implemented and suggestions for improvement,3) Exploration of program operations to inform formative program evaluation and suggestions for modifications.For this qualitative study, a six-phase thematic analysis , 6 usingPurposeful sampling was used to recruit participants for parallel focus group sessions including one series of focus groups for psychiatry residents and another series for SPC counselors. Psychiatry residents were eligible to participate if they were working at the academic hospital ED between 2017 and 2019 (recruitment ended October 2019), had been working there for at least 6 months, and had managed at least 15 ED patients with suicidal ideation and/or behaviors. The SPC counselors were eligible to participate if they worked in the follow-up program for at least 6 months between 2017 and 2020 (recruitment ended February 2020) and had participated in at least three ED follow-up cases.A semi-structured question guide was used in focus group sessions that were conducted at each participant group’s workplace. A total of 5 focus group sessions were completed . One of the 5 focus group sessions included 3 SPC counselors and it extended to a second session, however, two new participants joined for the second session. A total of 18 participants participated overall in the focus group sessions. Each session was digitally recorded and lasted approximately 60 min.The semi-structured question guide included open-ended questions that were designed to be exploratory to gain each group’s perceptions about the value of services offered, understanding of and experience with running the program, and how program operations and services could be improved. At the completion of each session, individual participants received a US $25 gift card. All digital recordings were uploaded to the university-approved, secure, HIPAA-compliant cloud-based storage service. Audio files were transcribed verbatim by a professional and secure transcription service. The team verified accuracy and de-identified transcripts in preparation for analysis.Thematic analysis was employed in 6 phases . First, the analysis team familiarized themselves with the data by reading and re-reading the transcripts while making notes about analytic ideas for potential themes. Line numbers were used on transcripts to heighten organization of data and notes. In phase 2, every transcript was coded line-by-line , 6, 9 byA total of 9 psychiatry residents and 9 SPC counselors consented and participated in the study. Psychiatry residents were between the ages of 28 and 35 years of age, and SPC counselors were between the ages of 24 and 59 years of age. All focus group sessions included male and female participants Table .Table 1This analysis resulted in 4 themes: valuing the program because it is useful and benefits patients, having a mutual desire to understand what happens from the ED to discharge to program follow-up, uncertainty about which ED patients would benefit from program services, and brainstorming to improve the referral process. See the main themes and subthemes outlined in Table “active” referral service meaning it was provider initiated instead of patient initiated  system so it would be smooth and time saving. One resident suggested:A hundred percent. It should be an order in [EHR System]. If that were a possibility, that’s like an obvious improvement to this. Because I remember recently the [EFU program] number changed, and then you're just like...so if you could search your email—I mean, we are slammed busy in this ED. This is not something you want to be wasting time on, and it’d be really nice if it was part of the discharge order set or something.Another resident agreed saying:I don't know, if we had a list here we could at least put it together and have the right spelling and have the right contact number. Because we often don't get callback numbers, either. So, if we had questions for the doctor about the patient, we don't know how to get in touch with them. I mean, we know they're with the hospital, but we don't—it would make it easier if we had a number.SPC counselors indicated their desire for more comprehensive contact information from psychiatry residents so they could discuss pertinent questions regarding their patients. One counselor offered a potential solution in having a list of ED doctors’ names and callback numbers available at the call center:“the basics like name, age” and “discharge time”. Further, they wanted to hear about patients’ relevant psychological history. One emphasized the practical value of this information saying, “… it would help to know if the person has been previously diagnosed with any kind of mental disorder and if they're receiving treatment for that, because that gears our conversation.” Another wished they could know a physicians’ appraisal of how receptive a patient might be to the EFU program:I mean, I would also argue the presentation of the person, of the patient. Because if we know, okay, patient is hesitant towards program, we may approach how we speak to them for the first time differently versus someone who's eager to—even that little tidbit of information would be able to gear our first contact and how our voice tone … is, how many times potentially we go ahead and try to make contact, as well.Relatedly, several SPC counselors lamented a lack of uniformity in the amount or types of patient information physicians left in their voicemail messages. They discussed a need for vital patient information, including Finally, SPC counselors emphasized the importance of being informed if the patient actually consented to the referral. They wished referring psychiatry residents would give more context when they provided alternate contact numbers for a patient. Counselors admitted it was at times difficult to navigate conversations and maintain confidentiality when they were unable to speak directly with the patient on a follow-up call.This qualitative exploration describes how healthcare providers perceived the process and effectiveness of a real-world suicide prevention telephone follow-up program that was formed jointly between an academic ED and a regional suicide prevention center. The 4 key themes that emerged indicate ED psychiatry residents and SPC counselors regarded the EFU program as valuable, desired to understand how the “other” side of the program operates, felt uncertain about which ED patients would benefit, and embraced ideas for improving the referral process. Within each theme, findings uncovered the nuanced experiences and perspectives of both the psychiatry residents making referrals to the EFU program and the SPC counselors making the outreach follow-up calls to patients. Application of the EPIS framework to this analysis highlighted that while there was enthusiasm for the EFU program, there was a need for improving communication, collaboration, and system effectiveness within and between agencies to inform the sustainment phase of the process . For exaThe SPRC produced a consensus guide for emergency departments in caring for adult patients with suicide risk, which included a recommendation that EDs “consider establishing an agreement with a local crisis center that allows its staff to make caring contacts with recently discharged patients” , 25, 33.Our findings suggest that partnerships between EDs and suicide prevention centers that specialize in suicide risk screening, assessment, triage, and management, may potentially benefit patients, communities, and health systems if efforts are invested to assure that the system collaboration is patent. Psychiatry residents valued the EFU program since it was an active referral, having an advantage over standard of care passive referrals that require patients to initiate follow-up. This feature could be crucial for high-risk patients who may not have the capacity or desire to make contact. SPC counselors embraced the role of being a transitional facilitator and saw themselves as well suited to their role, viewing their anonymity as an advantage. Phone, versus in-person services, helped counselors feel they could build trust within a judgment-free space, as well as foster honest interactions. The insights provided by psychiatry residents and SPC counselors indicate features of the EFU program that may be key to its efficacy.While interventions are needed to address suicide risk following discharge from an ED or hospital setting, our qualitative findings indicate that gaps in communication between partners may have negative consequences for providers and patients. Psychiatry residents considered the EFU program to be a “safety net” that may greatly benefit patients at risk for suicide after discharge. However, fluid communication between partners, timely referral, and reliable follow-up is needed to help those most in need of follow-up. SPC counselors in this study reported challenges with the enrollment process, stating some patients failed to understand what the EFU program was, making their extended follow-up services feel more like a “cold call”. It is not immediately clear whether this communication gap is due to hand off between agencies (via a recorded phone message), and whether this problem would persist even in an in-house program operated by the hospital/ED. Efforts to determine how to fortify the existing system or eradicate this communication gap may benefit from additional studies focused on patient perspectives on the flow from the ED to the EFU.The findings of this study suggest that EFU providers may question the worthiness or utility of the EFU program for different types of patients, as opposed to simply referring all patients who meet the criteria regardless of diagnosis or condition. In particular, psychiatry residents questioned the motives (whether conscious or unconscious) of patients with borderline personality disorder who may articulate suicidal ideation as instrumental or for secondary gain . While they voiced awareness of their own countertransference, they simultaneously expressed concern about the re-enactment of patients’ transference, attention-seeking, or maladaptive behavioral patterns with the SPC counselors. Similarly, counselors suggested a nuanced view of some patients potentially benefitting more than others. For example, they suggested that patients range from those with no previous experience creating a safety plan, to those who are chronically suicidal, and that interest in creating a meaningful safety plan and/or the program may be higher for patients with new-onset suicidality but lower for those who are chronically suicidal or chronically relapsing patients . Thus, the recognition and understanding of the needs and desires of the range of suicidal patients who may present to the ED, warrants not only the attention of clinicians, but from researchers too. More evidence-based insight is needed on whether it is important to differentiate within suicidal populations, and whether specialized approaches, training, or support systems may be useful within the 988 Lifeline system .When psychiatry consultants in busy and often overcrowded EDs have an established relationship with a patient, curating a referral to an outside system can be time consuming for providers and patients/families. Streamlining procedures to allow follow-up referrals by interdisciplinary care providers, and leveraging technology such as websites, may have potential to reduce barriers and expedite referrals. Training providers on ED teams, such as social workers and nurses, could help foster a more seamless referral process across the inter-agency partnership. Furthermore, embedding follow-up referral activities within an existing and well-established ED electronic health record (EHR) system increases likelihood of implementation, documentation and potential success . This waOn the other hand, systems of automatic referral that foster efficiency may have the downside of hampering the transmission of some critical information. To this point, SPC counselors desired greater context for the referral, for example, more awareness of the patient’s level of interest vs. hesitancy towards the program which could help the counselor to tailor their approach when contacting the patient. Challenges with handoffs between agencies is likely a product of the structure of 24/7 referrals that are transmitted telephonically in voicemail rather than live telephonic contact between the psychiatry resident and SPC counselor. This mode of information transmission supports a 24/7 function and is efficient but may fail in providing enough data, especially given the complexity of many patients with suicidal ideation or behaviors. Efficiency of referrals could perhaps be supported by electronic messaging between agencies with more information on patient/family context. This could include mutually agreed upon data-sharing information that could help maximize SPC counselors’ ability to engage and assist patients with increased suicide risk following discharge.Since SPC counselors had concerns about maintaining confidentiality when they were unable to reach a patient directly on a follow-up call, further work is needed to optimize balance between efficiency of making inter-agency referrals on a 24/7 basis, with the need for more detailed patient data and context. Hospitals or ED’s that operate their own caring contact program may have an advantage with greater ease of access to electronic health information, with the trade-offs of needing to train their own SPC counselors and manage the costs of the EFU program, an important issue for future research and quality improvement.Nevertheless, there was a disconnect experienced by both agencies, concerning inadequate knowledge of the other, absence of feedback on referral outcomes, and feeling challenged about lack of a clean handoff between the ED and follow-up services. Enhanced technology for the program and deeper investment in the development of the structured partnership will require more rather than less effort in the future from both partners. Specifically, the desire for residents and SPC counselors to know each other could potentially be addressed by more regular recurring meetings or in-services between EFU program leadership and/or providers.Partners on both sides of the EFU program lacked substantive feedback about program outcomes and effectiveness. SPC counselors had anecdotal evidence that patients appreciated their extra support while waiting for linkage to ongoing mental health care. However, psychiatry residents lacked any feedback, keeping them in the dark about services used and whether or not the EFU program improves patient outcomes. This lack of a feedback loop bred skepticism among residents, which led to the reluctance of referring patients into the program, and calls for efforts to provide detailed information to both partners. Innovative approaches to enhance communication are warranted such as collaborative quarterly virtual meetings, the making and sharing of videos about each site, or a website with photos and descriptions that is made for the staff of the ED and the program that provides needed details about each partner and the program process.The 988 Suicide and Crisis Lifeline offers best practice guidelines for follow-up programs . RecommeQualitative data were from the perspective of providers involved in a partnered suicide EFU program for an academic institution and community outreach program. Therefore, results from the study are limited to this inter-agency partnership and not generalizable to other suicide prevention programs. Furthermore, we recruited a convenience sample of psychiatry residents and SPC counselors who were currently working in the ED and regional suicide center. Future exploration of the perspectives of other ED providers , other suicide prevention program employees (supervisor or director), and patients/families could broaden understanding of program benefits and operations. Finally, patients discharged from the inpatient psychiatric setting were also offered follow-up services. Future investigation should explore perspectives of inpatient psychiatrists and other providers, since differences in EFU program referral practices, such as time available to prepare and refer patients, could vary greatly between ED and inpatient psychiatrists.Programs for reducing risk of suicide or self-harm following discharge from the emergency department are critically important. “Caring contacts” are brief but highly valuable communications with patients after discharge from the ED or hospital, and have been endorsed by multiple agencies, including AHRQ, SAMHSA, and the Suicide Lifeline. Partnerships between hospitals and local suicide prevention centers have many potential benefits for patients and the healthcare system alike, however these programs require careful study to assure that the people in need are effectively getting connected with follow-up care and not falling through the cracks, so that the system is operating as efficiently as possible. Careful analysis is also needed to document and appropriately justify the costs of such programs.This study provides a qualitative appraisal of such a program. The results demonstrated that there are benefits to this community partnership that effectively provide a “caring contact” hand-off. Psychiatry residents in our sample appreciated how such referrals reduced the burden of patient responsibility and was an improvement above the typical standard of care of providing a list of referrals that otherwise require the patient to act upon and initiate their own follow-up. SPC counselors appreciated the ability to develop meaningful relationships with patients over an extended period of time and valued their role as a facilitator of transition of care from the ED discharge.However, this study identified a number of important challenges and areas of opportunity for future work. The results highlight the need for improvement in the quality of information transmitted between the ED and the SPC, which would enable the counselors to approach the patient in a more sensitive and fine-tuned manner. Additionally, further study is needed regarding the potential for difference in patients with chronic and/or recurrent suicidal ideation compared to new-onset suicidal ideation. It remains unclear if the program benefits certain populations more than others, or if specialized training based on these differences would be beneficial.While efficacy of workflows was highly valued by the ED team, including recommendations to develop automated transmission of the referral to the Suicide Prevention team through the electronic health record, such efforts would need to keep in mind that counselors valued more thorough patient information and clinical context. Both psychiatry residents and counselors desired to have a better connection or awareness of the other, as well as more formal feedback on the success or failure of the referral to the outreach team. This may further inform if the outreach should be internal or external to the ED system, another important future research effort.Overall, this study provides qualitative evidence that this partnered program is valued as a whole. Improved communication, implementation, consistent provider training, efficient and accurate outcome reporting mechanisms, and coordination between partner systems would likely lead to a better provider and patient experience. Such a program is potentially sustainable, particularly with consistent funding for ongoing collaboration.


How do administrators and faculty perceive the impact of National Institutes of Health (NIH) funding requirements for Medical Scientist Training Programs (MSTPs) on the racial and ethnic diversity of matriculants?In this qualitative study including 69 administrators and faculty at 12 MSTPs, NIH funding requirements for diversity fostered urgency among MSTP leadership to bolster matriculant diversity, enabled program administrators to secure additional investments in diversity, promoted holistic review, and encouraged rigorous evaluation of the effectiveness of recruitment efforts.Findings suggest that funding requirements, such as those instituted by the NIH, may encourage impactful institutional practices to increase recruitment efforts and improve the representation of diverse candidates in training programs and, ultimately, the biomedical science workforce. Since 1964, the National Institutes of Health (NIH) has funded the Medical Scientist Training Program (MSTP) MD-PhD program at medical schools across the US to support training physician-scientists. Recent studies have suggested that MSTPs have consistently matriculated more students from racial and ethnic backgrounds historically underrepresented in science than MD-PhD programs without NIH funding; however, the underlying basis for the increased diversity seen in NIH-funded MSTPs is poorly understood.To investigate how administrators and faculty perceive the impact of MSTP status on MD-PhD program matriculant racial and ethnic diversity.This qualitative study used a positive deviance approach to identify 9 high-performing and 3 low-performing MSTPs based on the percentage of students underrepresented in science who matriculated into the program between 2014 and 2018. This study, a subanalysis of a larger study to understand recruitment of students underrepresented in science at MSTPs, focused on in-depth qualitative interviews, conducted from October 26, 2020, to August 31, 2022, of 69 members of MSTP leadership, including program directors, associate and assistant program directors, and program administrators.The association of NIH funding with institutional priorities, programs, and practices related to MD-PhD program matriculant racial and ethnic diversity.The study included 69 participants . A total of 51 participants (74%) were in administrative roles, and 18 (26%) were faculty involved in recruitment. Five themes emerged from the data: (1) by tying MSTP funding to diversity efforts, the NIH created a sense of urgency among MSTP leadership to bolster matriculant diversity; (2) MD-PhD program leadership leveraged the changes to MSTP grant review to secure new institutional investments to promote recruitment of students underrepresented in science; (3) MSTPs increasingly adopted holistic review to evaluate applicants to meet NIH funding requirements; (4) MSTP leadership began to systematically assess the effectiveness of their diversity initiatives and proactively identify opportunities to enhance matriculant diversity; and (5) although all MSTPs were required to respond to NIH criteria, changes made by low-performing programs generally lacked the robustness demonstrated by high-performing programs.This study suggests that NIH funding requirements may be a powerful incentive to promote diversity and positively affect representation of students underrepresented in science in the biomedical scientific workforce. This qualitative study investigates how administrators and faculty perceive the impact of MSTP status on MD-PhD program matriculant racial and ethnic diversity. Although they comprise only 4% of medical school graduates, these physician-scientists receive nearly 50% of National Institutes of Health (NIH) research funding grants awarded to physicians.3 generate higher-quality research,5 and increase participation in clinical trials by underrepresented populations.6 Despite these benefits, in 2020, only 2% of NIH Research Project Grants (R01 grants) were awarded to Black scientists, and only 5% were awarded to Latinx scientists.3The benefits of diversity within research teams are well documented and include a greater ability to solve complex problems,7 Prior studies have reported that MD-PhD programs with MSTP status have consistently matriculated a higher percentage of students historically underrepresented in science (URiS) by race and ethnicity10 than MD-PhD programs without MSTP status. Notably, URiS students are nearly 4 times more likely to matriculate at longstanding MSTPs than non-MSTPs.8 Nevertheless, the mechanism for this higher level of diversity is poorly understood.Since 1964, the NIH has funded the Medical Scientist Training Program (MSTP) at medical schools across the US to support the training of physician-scientists obtaining an MD-PhD.To address this knowledge gap, we conducted in-depth interviews with MSTP leadership among a diverse cohort of MD-PhD programs nationally to better understand URiS student recruitment practices and how MSTP status was associated with matriculant diversity with regard to race and ethnicity.12 In deviant case sampling,12 the aim is to focus on examples “rich in information” that are unique in some way, which allows us to learn lessons that can be applied to other settings. We selected programs based on the percentage of URiS students13 who matriculated into the program between 2014 and 2018, using data from the Association of American Medical Colleges , while low-performing programs were drawn from the bottom quartile   reporting guideline.20This analysis is a component of a larger parent study that primarily sought to understand URiS student recruitment practices at MD-PhD programs nationally. The parent study used a positive deviance approach, comparing programs with differing levels of URiS student matriculation to generate hypotheses about factors that distinguished them.Colleges .9 Becauss, ≤11%) . We enroBox 1. American Medical College Application Service data provided by the Association of American Medical Colleges for academic years 2014-2018Mean percentage of URiS matriculants per classStratification into quartiles; top quartile (n = 12) identified as high-performing programs and bottom quartile (n = 12) as low-performing programsFinal sample size: 9 high-performing programs and 3 low-performing programsIn-depth interviews with 162 participants at identified high- and low-performing programsOverall thematic analysis to identify URiS student recruitment best practices and generate hypotheses; this includes a subanalysis of the whole data set, using 69 program leaders to understand the effect of NIH requirements for funding as associated with diversityQualtrics survey with items generated from in-depth interviewsTargeted 162 respondentsTest the extent to which recruitment strategies identified are implementedAbbreviations: NIH, National Institutes of Health; URiS, underrepresented in science.22 within the program to identify others with knowledge of efforts to increase matriculant diversity. In the parent study, we interviewed 162 individuals with diverse roles. For the present study, we focused on a subset of 69 interviews with administrators and faculty involved in recruitment because they were most likely to be knowledgeable of efforts to increase URiS student recruitment.Directors of MD-PhD programs at selected institutions were contacted by the principal investigator (D.B.) via email to participate in the parent study. Of the 13 institutions that we invited to participate, 12 responded (1 institution did not reply); 9 institutions were high performing, and 3 were low performing. After providing informed consent, program directors worked with the study team to provide an initial set of potential key informants in administrative, faculty, and student roles. Key informants included program directors, chief diversity officers, deans of diversity, deans of admissions, deans of students, MD-PhD faculty involved in recruitment, faculty of racial and ethnic minority groups, and URiS MD-PhD students. Once connected with key informants, we further recruited participants via “snowball” samplingFrom October 26, 2020, to August 31, 2022, we conducted in-depth individual interviews with MSTP leadership via Zoom. Our multidisciplinary team had expertise in public health, emergency medicine, psychology, and journalism; all had experience conducting in-depth interviews. Interviews typically lasted 1 hour, followed a structured interview guide , working in alternating teams of 2, coded transcripts independently and met frequently to discuss new ideas, refine the code structure, and reach consensus on divergent views. We refined the code structure until no new concepts emerged, finalizing after 15 rounds of revision . We used standard techniques to ensure that data collection and analysis were systematic and verifiable, including discussion guides for all interviews, audiotaping and independent transcription, standardized coding and analytic procedures, multiple diverse coders, and an audit trail to document analytic decisions.25We analyzed interview data using the constant comparison method,The study included 69 participants . Participants represented individuals most involved in implementing changes to URiS student recruitment practices at their respective MSTPs. Informants consisted of administrators  and faculty involved in recruitment. A total of 51 participants (74%) were administrators, and 18 participants (26%) were faculty involved in recruitment .27 Five major themes .r themes  emerged Box 2. We were actually docked a lot of spots because they [NIH] didn’t feel that we had done a good enough job in recruiting underrepresented minorities into the MD-PhD program. That was a very specific criticism of our application, and we had to make sure we had some way to address that…. So that was largely the impetus to try something…. When I took it over, I think I had to be a lot more proactive in trying to make it work .Before I started working here NIH came at [the program] and said, “You have to do better with your URM recruitment in the context of the MD-PhD program.” I think at that point is when recruitment in that particular area really started being taken care of and more attention paid and intensified .The dean was there in the room when we were being site-visited by NIH…. And so that was part of the reason that I could then kind of go to the dean, and say, “Look. It’s really important for you to give money and support these things like the undergraduate research program because NIH is paying attention, and they don’t think [the program] is doing a good job…. So, in a way, the emphasis at NIH was very helpful” .One of our MSTP NIH reviews wanted us to be more innovative in terms of bringing more underrepresented [students] in. So we were able to show them that and get their commitment to support the program…. We had to recently re-request funding where we showed them our data that it’s been a very successful program…. So I think some of the impetus was probably poked by the comments from the NIH grant .We got a summary statement back on our [year] renewal, that said, “It’s wonderful that the institution is doing these things to promote diversity and equity in admissions. But what is the MD-PhD program doing?” In response…we turned around and saw that our [dean] was going to historically Black colleges on a regular basis. He was in charge of an outreach program to [a Hispanic-serving institution]. We made him an associate director of our program as well and included him in all of the decision-making regarding the program, including him in all of the administrative meetings, the interviews, everything, so that the outreach he was doing on the part of the graduate school also became outreach on behalf of the program .How do you evaluate a student that…may have come a long distance—may have been the first person in the family to go to college, let alone…medical school…. And how do you balance that in the admissions process?…I think it’s getting better. I think everywhere we’re going to much more holistic admissions—getting away from using standardized tests as a measurement…. Looking at the distance traveled, is this somebody that really is interested in coming here and moving the needle and commit to the career?…I do think that the admissions process has gotten more holistic and probably more about the entire arc of the applicant, which has helped .When I took over the program, we had about 10% underrepresented students in the program, just like every other MSTP. And the first couple of years I was director, I focused on MCAT [scores] and GPA, and we weren’t particularly [successful] in changing the numbers. And I realized that if we don’t consider other factors and do more holistic review of the applicants than just the numbers, we’re never going to change the needle .The other thing that we’ve done is in recognition of the fact that there is historical racism in the American educational system that starts in kindergarten and is very hard to undo by the time you’re in college…we don’t really put as much weight on the MCAT [scores] as we used to. We used to have admissions committee meetings where someone would say, “I really like this student, but.” And the but was always a mention of a red flag somewhere in the scores on the MCAT. And now we don’t discuss the MCAT [scores] after we meet the students in person. If we bring them in for in-person interviews, it’s understood that all of those numeric scores, your past, you’ve moved beyond those now, and now we’re looking for other qualities in the applicants that would make them good members of our MD-PhD community…. We are trying to make sure that we don’t sort of miss those diamonds in the rough .The number of underrepresented minority medical students has gone through the roof. When I became the director, it was about a third women and two-thirds men. Now it’s just about 50-50. I doubled the number of underrepresented minority students during my time from about 10 to 20-something percent. And we were patted on the back, lauded by the NIH for having such a high percentage of underrepresented minority students for such a selective program…. Traditionally it’s been much easier for us to recruit Latino, Latinx underrepresented minority students, [but not] African American students…. But we’ve been chipping away at that and doing better with it (site 8 participant 1).We know we are receptive to some of the things that NIH needs. Of course, we’re not just doing it for that, but we have to be aware of that because then otherwise we would lose our own funding. So it’s a balance of who we want to attract and have in our program and mentor them, plus who do we need to answer to make sure that our program is considered top notch and sustained long term .I think our success in recruiting individuals from diverse and marginalized groups has not been as good as I have hoped. We’re sort of holding our own. But we’re not making huge strides in that…. I think we’ve got a long way to go. I think we’ve got a lot more work .It is something at the forefront of our mind. I mean, we want to make sure that we are meeting NIH benchmarks and our reporting is accurate…. It’s something we’re aware of. If we have a year where we don’t have any underrepresented minorities, it’s something that is obvious, something that we talk about, maybe why they didn’t accept our offer, what we can do next year to successfully recruit those applicants .Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; MSTP, Medical Scientist Training Program; NIH, National Institutes of Health; URiS, underrepresented in science; URM, underrepresented in medicine.By tying MSTP funding to diversity efforts, the NIH created a sense of urgency among MSTP leadership to bolster matriculant diversity. Members of MSTP leadership reported that the NIH’s linking of MSTP funding to diversity efforts created an immediate impetus to prioritize the recruitment of URiS students.We have to provide a diversity report of recruitment with our application…it was not a very good report for the previous 5 years…a couple of years there were no students that fit the diversity category. The grant went in…about a month later, we got this letter that said, “We’re disallowing the award because we think your diversity record and your plans for diversity are inadequate for our funding.” So therein began a number of changes in the program .Shifts in NIH funding evaluation criteria realigned program priorities to focus on increasing diversity. As 1 assistant program director stated,NIH is asking lots of tough questions about diversity. When we go to renew our training grant, they’re asking for all these numbers and asking us what do we do to try to increase diversity in our program .New diversity funding criteria forced MD-PhD program leadership to thoroughly reflect on and document their prior work to support diversity, which created a mechanism of accountability that did not previously exist. One MSTP administrator noted,I had to write this report to NIH about what we had done with minority students…. That was probably the most important thing because I realized that we had not done as much as I thought we should have done… .Some MD-PhD program leadership shared receiving negative feedback from the NIH regarding low matriculant diversity on MSTP grant applications. In a few cases, MD-PhD programs were given probationary standing because of a consistent lack of diversity.Many of the questions that they [NIH reviewers] had were about our ability to recruit minority students to our program, to retain them in training, and to have successful outcomes… We only had probationary funding, and it was contingent upon our improving our record .The risk of losing funding spurred institutional activity to enhance URiS student recruitment. Respondents at programs placed on probation noted a change in their program’s willingness to pursue diversity initiatives after their NIH funding was jeopardized. One program administrator described being “threatened with having the program not funded anymore” as an event that “crystallizes one’s thoughts and actions” . For example, some institutions describe implementing action plans to support increased enrollment of URiS candidates.When I became director…I realized that NIH actually had put us on probation for our lackluster performance with respect to minority students…when I took over, it was very traumatic… I laid out an action plan for how we could overcome these problems .Leadership of MD-PhD programs leveraged the changes to MSTP grant review to secure new institutional investments to promote URiS student recruitment. Pressure from the NIH to improve program diversity promoted buy-in for recruitment strategies among administrators and yielded financial investments in human capital and program activities to support diversity. One administrator described the confluence of the program director’s goal of improving diversity with the provision of resources to hire an individual to support those efforts.When we had one of our reviews for our NIH grant there was a comment that we needed to increase the diversity of the program. Then that was the goal of the directors as well. So, between the NIH, “You need to up the game a little bit,” and the interest on the directors’ part, it became a more formalized process when they hired the [URiS recruiter] .At the institution described, the administration allocated money for an individual to attend conferences geared toward supporting URiS students, such as the Annual Biomedical Research Conference for Minoritized Scientists (ABRCMS). In addition, this individual received financial resources to start college tours, which included visiting smaller schools and historically Black colleges and universities (HBCUs) to recruit students. At other institutions, administrators were able to lobby for increased funds to establish institutional infrastructure that could help increase and develop the URiS candidate pool, such as summer research programs.28Medical Scientist Training Programs increasingly adopted a holistic review to evaluate applicants to meet NIH funding requirements. To increase the number of URiS students, programs altered their approach and developed strategies to evaluate candidates holistically. With a holistic review of candidates, programs evaluate applicants’ capabilities in a way that allows for a balanced consideration of their experiences, academic performance, and characteristics that may contribute positively to their success as students and physician-scientists.One of the official complaints of the NIH reviews of our program…was that we didn’t have a very diverse student body. [Program director] came on board and really moved the admission process and considerations of the applications to a much more holistic review that included URM [underrepresented minority] status, disadvantage status, and this distance traveled criterion .29 and what populations are considered URiS, resulting in expansion of the demographic characteristics that they target for recruitment.Programs made efforts to create applicant review teams that were also more diverse and reflected the makeup of candidates that they aimed to recruit. In addition, programs reported the responsiveness to changes in the NIH’s definition of diversityThe NIH expanded the definition to include now a low socioeconomic status… One of the things that I did…when I became program director is I changed the formula that we use for evaluating applications .Medical Scientist Training Program leadership began to systematically assess the effectiveness of their diversity initiatives and proactively identify opportunities to enhance matriculant diversity. As they complied with NIH standards, many programs described initiating rigorous assessments of the effectiveness of their recruitment strategies. As 1 program director identified, they noted disparities in sex and among specific racial and ethnic URiS groups, some of which they had improved, and others they knew had yet to be fully addressed.We have this database so we can look and see how are we doing…we have a column to identify, are we talking to underrepresented students in this group? What’s the percentage in this group? Is the institution Hispanic-serving or Hispanic Association of Colleges and Universities or HBCU? …We have those identifiers…track what are you doing…then you can find areas to grow…we do that in a purposeful way .30 so have the ways in which institutions track the makeup of their classes. For example, 1 assistant program director stated,Furthermore, as NIH diversity reporting criteria have evolved,NIH requires a trainee diversity report with every annual progress report that we provide…those numbers have to be tracked in order for the NIH partner support to be completed. The last 2 years have been really going back to rigorous data standards to be able to track those demographic data. Because they weren’t rigorously tracked over time .Although all MSTPs were required to respond to NIH criteria, changes made by low-performing programs generally lacked the robustness demonstrated by high-performing programs. Some low-performing institutions differed from high-performing MSTPs in the intensity of their activity to recruit URiS students. Administrators at low-performing institutions described being “receptive” to NIH requirements and being “aware” otherwise they risk losing their funding .However, while the NIH standards promoted an understanding of a need to increase diversity and encouraged recruitment activity, some administrators at these MD-PhD programs acknowledged the lack of an institutional “why”  as a hindrance to progress. Efforts at low-performing programs seemed to be more perfunctory without complete buy-in to promote diversity.Historically, we’ve gone to these recruitment fairs because we are supposed to, because as an NIH-funded grant program, we’re supposed to. But it’s really kind of separating what we should be doing vs why do we want to do this? I think that’s kind of been a challenge. All these policies were in place just because we had to have policies. But I think the institution and our program is really realizing that you need to have the “why” within you, or else, it’s not going to work… I think that’s been a challenge, getting the genuineness to move us forward .Although some low-performing programs struggled with establishing a strong commitment to improving URiS student recruitment activities, this was not the case for all, with some noting it as important regardless of NIH funding requirements. At these programs, administrators see URiS student recruitment as “something [they] put a lot of effort into” and “don’t necessarily need a financial incentive for” but acknowledged being “pragmatic… to keep NIH funding” .Overall, while low-performing programs responded to NIH funding requirements, they were not as proactive as high-performing institutions where administrators realized they had to “stay ahead of the curve” . These institutions were more likely to rely on fewer activities, such as attendance at ABRCMS, and had only more recently started collaborations that would allow them to recruit students through initiatives such as pipeline programs. This differed from high-performing institutions that had more established initiatives, which potentially demonstrate increased investment of human and financial resources.We’re trying to start a pipeline program. We’re hoping over time, maybe we can open it up broader than [our school] and attract students from other campuses .Finally, some low-performing programs differed from high-performing programs in their reliance on traditional application metrics, such as standardized test scores, for admission as opposed to more holistic reviews of candidates. Some described these metrics as a challenge to their recruitment efforts.We found that changes to MSTP grant review linking funding to diversity compelled programs to adopt efforts to increase matriculant diversity. Efforts included redefined programmatic priorities geared toward improving diversity, the allocation of financial and human resources to bolster matriculant diversity, the use of holistic review to enrich the applicant pool, and assessment to determine effectiveness of recruitment strategies.31 Although MSTPs have consistently achieved higher rates of racial and ethnic diversity compared with non-MSTPs,32 this study is the first, to our knowledge, to describe the ways in which NIH funding requirements have been associated with MD-PhD program efforts to improve diversity. This includes required detailed documentation of diversity efforts and outcomes as part of MSTP grant funding review.33These findings are consistent with studies that demonstrate an association between medical school accreditation standards and subsequent gains in matriculant diversity.35 Given the potential of MSTPs to successfully recruit URiS matriculants, expanding the number of programs could present a significant opportunity to increase diversity among physician-scientists. Furthermore, increasing the number of training slots at current MSTPs may be another strategy to increase MD-PhD program diversity.The study findings have implications for national funding agencies and MD-PhD programs. National Institutes of Health–funded MSTPs account for approximately 44% of all MD-PhD programs in the US.36 Given this substantial financial support, the NIH could extend its influence to promote diversity beyond MSTP funding to all grant awards, creating a strong impetus for academic medical centers to broadly bolster diversity.Behavioral and economic incentives to promote diversity within MSTPs created by NIH funding criteria, if applied to other NIH grant types, could lead to increased diversity in the entire biomedical scientific research enterprise. Medical schools receive nearly 67% of all NIH funding.5 These efforts could be evaluated in the investigators section of NIH grant priority scoring.The NIH could link medical school diversity efforts to federal funding by several means. The research environment is a key component of evaluation for most NIH grants. The NIH could require grant applicants to report institutional measures of diversity, equity, and inclusion, such as faculty compositional diversity, promotion and attrition rates by race and ethnicity, and pay equity, which could be assessed as part of the research environment in grant priority scoring. Similarly, the NIH could require grant applicants to describe efforts to assemble a team of investigators from diverse backgrounds, especially given prior literature demonstrating that diverse teams may produce higher-quality science.Although the NIH’s linking of diversity to MSTP funding may have a beneficial, forcing-function influence on programs and may encourage diversity recruitment activities, study findings suggest that this policy may not be sufficient to fully mitigate the disparities that we see in the biomedical research workforce. Some programs may adhere to the criteria with perfunctory or “checking the box” behavior that may not lead to sustainable parity in representation. The diversity policy alone may also be insufficient to promote a sincere belief in the importance of diversity in science and may result in programs meeting minimum standards only to maintain NIH funding. Beyond the findings of this study, other considerations exist that may affect the impact of the NIH’s diversity policy. For example, future studies should examine how judicial rulings related to affirmative action anticipated at the Supreme Court may affect the NIH’s ability to promote diversity in the physician-scientist workforce.37 To mitigate these factors, we included interview probes designed to elicit both positive and negative experiences from participants,38 provided assurances of confidentiality, and interviewed a wide range of individuals from all levels of the institutions to ensure we had diverse perspectives.This study has some limitations, and these data must be interpreted in the context of the study design. First, our criteria for selecting MSTPs that are high and low performing, as determined by the number of URiS matriculants, may be associated with other factors beyond the programs’ control, such as size of the institution, geographic location, and perceived institutional reputation. Second, our sample is limited to NIH-funded MSTP MD-PhD programs. Although NIH-funded MSTPs comprise two-thirds of all MD-PhD students, our study does not explore how NIH funding opportunity review criteria may have been indirectly associated with programs without NIH support. Third, we acknowledge that there may be recall bias and social desirability that may have affected participants’ responses.In our qualitative study of MSTPs, we found that the NIH’s linking of diversity efforts to MSTP funding created a powerful impetus for program leadership to enhance the recruitment and retention of URiS students. The NIH could consider linking diversity efforts to additional funding opportunities to increase the diversity of the biomedical research enterprise.