The virological meaning of the different patterns of serology in COVID-19 has been little examined in clinical settings. Asymptomatic subjects with IgM-spike (S) and IgG-nucleocapsid (N) determinations by chemiluminescence were studied for SARS-CoV-2 shedding in respiratory secretions by transcription-mediated amplification (TMA). In subjects showing IgM-S positive and IgG-N negative, IgG-S was determined by lateral flow assay. A total of 712 individuals were tested: 30.0% presented IgM-S(+)/IgG-N(−), 25.8% had IgM-S(+)/IgG-N(+) and 44.2% had IgM-S(−)/IgG-N(+); the proportion with TMA(+) were comparable in these three groups: 12.1, 8.7 and 10.5%, respectively. In individuals with IgM-S(+)/IgG-N(−), IgG-S(+) was detected in 66.5%. The frequency of IgM-S(+)/IgG-S(−) in the total population was 10.0%, of whom 24.1% had TMA(+); the chances for TMA(+) in subjects with an IgM-S(+) alone pattern were 2.4%. Targeting of the same SARS-CoV-2 antigen seems to be better for the characterization of IgM/IgG patterns of response. IgM-S(+) alone reactivity is rare, and a small proportion is associated with viral shedding. Direct diagnosis of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is today based on antigen and molecular tests . SerologClassically, the IgM and IgG responses have been used to determine the different phases of viral infections, namely acute, convalescent or chronic. Guidelines indicate that subjects with IgM reactivity but with negative IgG should be tested to discard active SARS-CoV-2 infection ,11, althThe seroSOS study is a SARS-CoV-2 serological point survey of most sociosanitary centers in the Community of Madrid. All asymptomatic residents and workers in these centers were offered to participate with a serology in blood and a nasopharyngeal swab as a molecular test for SARS-CoV-2. We selected for this study all consecutive participants for whom IgM plus IgG serology was available.® SARS-CoV-2 assay, that detects anti-S IgM (IgM-S) and anti-N IgG (IgG-N) by chemiluminescent microparticle immunoassay (CMIA) with peak sensitivities of 96 and 99.6%, respectively [® Diagnostics Co., Zhengzhou, China [The Abbottectively , was useu, China ) was use®, Hologic-Grifols®) [Detection of SARS-CoV-2 RNA from nasopharyngeal swabs was undertaken by transcription-mediated amplification  .®, Microsoft®). Comparison of proportions was done by Chi-2, with Yates’ correction if needed, with Epidat® software . The Regional Ethics and Investigation Committee of the Community of Madrid approved the study.Clinical information was recorded in electronic databases (Formsp = 0.007).Serum samples from 712 asymptomatic individuals were available for the determination of IgM-S and IgG-N responses. A total of 449 subjects were identified as residents  and 263 subjects as workers  in nursing homes. History of past COVID-19 was registered in medical records in 156 (35.3%) residents and 68 (26.3%) workers (Positive IgM-S was found in 439 (61.7%) and positive IgG-N in 498 (69.9%) individuals. Patterns of antibody response were as follows: IgM-S positive plus IgG-N negative in 214 (30.0%) subjects, IgM-S positive plus IgG-N positive in 184 (25.8%) subjects and IgM-S negative plus IgG-N positive in 314 (44.1%) .p = NS).Nasopharyngeal swabs, all sampled at the time of blood test, were available for 688 individuals, 407 residents (90.6%) and 241 workers (91.6%). Positive TMA was detected in 72 (10.5%) individuals, similarly distributed among 56 (12.5%) residents and 24 (9.1%) workers. Molecular tests were available in 149 subjects showing an IgM-S positive alone pattern, in 184 subjects with both positive IgM-S and IgG-N, and in 314 subjects with an IgG-N positive alone pattern. No significant differences in terms of proportion of individuals with positive TMA were found among those three serological patterns, 12.1% for IgM-S positive alone, 8.7% for IgM-S plus IgG-N positive and 10.5% for IgG-N positive alone . Historyp = 0.03) , while in 66.5% cases the second IgG-S result was positive . The chances for detection of positive TMA were 24.1% among IgM positive plus IgG negative for samples tested against S antigen, while this figure was 12.1% if IgM against S but IgG against N were used ( = 0.03) .Although serology tests for viral infections mostly inform of past or chronic infection, detection of IgM reactivity with negative IgG sets an indication to discard active infection ,11. Our ®) failed to detect IgM-S in more than 90% of subjects that were IgM-S positive alone by CMIA. Previous studies have shown that, even in patients with history of positive PCR for SARS-CoV-2, colloidal-gold immune assays have much lower accuracy for IgM-S (sensitivity of 47%) [Lateral flow rapid test (Autobio of 47%)  when com of 47%) . The IgM of 47%) ,23,24.Differences in the dynamics of S versus N antibodies may in part account for the discrepancies observed in our study with respect to IgG responses. A rapid decline in IgG-N titers has been shown previously, while levels of IgG-S remain detectable for longer ; the meaWe feel that the failure of the N-based IgG test to detect memory response is supported by the fact that viral shedding was twice as frequent in IgM-S positive plus IgG-S negative (24.1%) than in IgM-S positive plus IgG-N negative samples (12.1%). A recent study showed that S-based serology was able to detect IgG activity in up to 69% of subjects with negative IgG against N antigen . If IgM Targeting S antigens better reflects neutralizing immune activity and may better indicate immune protection than N antigens ,35. AddiIn conclusion, our analyses support the utility of targeting S-protein for the monitoring of antibody response against SARS-CoV-2. The use of IgM is of little interest given that both IgM and IgG have a very similar kinetics. Targeting N-protein is needed to establish that immunity was acquired after natural infection rather than just by vaccination.


Introduction: There are scant data to demonstrate that the long-term non-pharmaceutical interventions can slow the progression of motor and non-motor symptoms and lower drug dose in Parkinson's disease (PD).Methods: After randomization, the Exercise-only  group completed an initial 3-week-long, 15-session supervised, high-intensity sensorimotor agility exercise program designed to improve the postural stability. The Exercise + Maintenance  group completed the 3-week program and continued the same program three times per week for 6 years. The no exercise and no maintenance control  group continued habitual living. In each patient, 11 outcomes were measured before and after the 3-week initial exercise program and then, at 3, 6, 12, 18, 24, 36, 48, 60, and 72 months.Results: The longitudinal linear mixed effects modeling of each variable was fitted with maximum likelihood estimation and adjusted for baseline and covariates. The exercise program strongly improved the primary outcome, Motor Experiences of Daily Living, by ~7 points and all secondary outcomes . In E group, the detraining effects lasted up to 12 months. E+M group further improved the initial exercise-induced gains up to 3 months and the gains were sustained until year 6. In C group, the symptoms worsened steadily. By year 6, levodopa (L-dopa) equivalents increased in all the groups but least in E + M group.Conclusion: A short-term, high-intensity sensorimotor agility exercise program improved the PD symptoms up to a year during detraining but the subsequent 6-year maintenance program was needed to further increase or sustain the initial improvements in the symptoms, quality of life, and drug dose. Parkinson's disease (PD) impairs mobility, cognition, and quality of life –3. The dThus, there is a need to demonstrate that high-intensity is in fact not harmful in moderating the disease-symptoms in PwPD and to document how long the beneficial effects of exercise last when it is stopped. Further, it is unclear if exercise is continued as a maintenance program, it would in fact prolong symptom-moderation at a functionally meaningful level , 31, 32.This study aimed to determine the immediate and lasting effects of a 3-week-long, high-intensity, and high-frequency Exergaming agility program with and without a 6-year-long high-intensity Exergaming agility maintenance program, on the motor and clinical symptoms in PwPD. Based on our previous experience and the extant data, we expected that PwPD would tolerate the short-term 3-week initial high-intensity Exergaming agility program which would produce the favorable and lasting effects during detraining. In addition, we expected that the maintenance program would further slow the symptoms-progression and reduce the increase in L-dopa equivalent levels , 32.n = 8) and others declined to participate (n = 16). The remaining 67 patients were randomized to: Exercise + Maintenance group ; Exercise only no maintenance group , and to a no exercise and no maintenance control group , a full neurological exam, and a mobility evaluation. In a separate visit, a neuropsychologist evaluated cognitive function of the patients. The neurologist and neuropsychologist were blind to the group assignments. The exclusion criteria were: MRI-based brain abnormalities; Mini Mental State Examination score <24; a Beck Depression Inventory (BDI) score >40; severe cardiac disease; uncontrolled diabetes; a history of stroke; traumatic brain injury; seizure disorder; and past or current deep brain stimulation, vestibular/visual dysfunction limiting locomotion or balance, or current participation in a self-directed or formal group exercise program. All the patients remained “on” medication that the patients took before exercise or assessment.The assessors included two experienced physical therapists and a physical therapy assistant who were blind to the group assignments of the patients. In the familiarization session, the patients practiced each physical test and watched the Xbox Kinect Exergaming programs. The patients gave written informed consent to participate in the study. The Ethics Committee of University Hospital approved the study protocol (IKEB2020/05) and the trial was registered (NCT04559997).The primary outcome was M-EDL, which is sensitive to the changes in a wide array of the PD symptoms . The cha−1) were computed to determine if interventions reduced the required dose; Schwab and England Activities of Daily Living Scale (SE ADL); EuroQol five-dimension health-related quality of life questionnaire (EQ-5D); the Parkinson's Disease Questionnaire   agility intervention, detailed and illustrated by using the video clips . Brieflyrg scale .After the 3-week-long, high-intensity E, E+M continued the M-program 3 × /week for 72 months in the physical therapy gym of the hospital using the same exercises used in the 3-week exercise program. The three therapists supervised each small-group session at about the same time of the day for 1 h. The aim of the maintenance program was to determine if the patients could endure a high-intensity rehabilitation program for an extended time-period and if such a program could slow the symptom-progression.This group did not participate in the E nor in the M program. We instructed these patients to continue their habitual activities. Thus, they participated in their normal social activities outside the study. They participated in the periodic testing sessions  and receCIs. All the values were recorded for each patient at each time point. Because the PD drugs are not adjusted over a short time-period, the L-dopa values at 3-week and 3-month follow-up were imputed with the baseline value. The analyses were performed using the built-in XT  for specific comparisons among the groups at the given follow-up moments. As a sensitivity analysis, we also combined the longitudinal linear mixed effects modeling with a semiparametric Cox regression to model survival by assuming the shared random intercepts .In a separate analysis, we compared those patients at the baseline who deceased over the 72 months with those who completed the trial. For this analysis, the variables not normally distributed were transformed. The comparisons were done with an unpaired p < 0.05 and adjusted by Bonferroni correction for multiple comparisons in The changes over time and the differences between the groups were further characterized by Cohen's effect sizes. We further quantified the within-group changes over time and between-group differences by Cohen's effect sizes . We computed Pearson's correlations between the changes in the primary and secondary outcomes to explore the potential mechanistic links underlying the improvements in the mobility and clinical symptoms of the patients. The level of significance was set at n = 9) dropped out after 24 months. 2 (df:18) = 209.0, p < 0.001]. The 3-week E-program improved M-EDL by 6.3 points  in E + M (d = 1.53) and by 8.4 points  in E . These changes exceeded the 0.7-point  change in C . After 3 months, the M-program did not further improve but sustained these initial gains until month 72 . At 72 months, the difference in M-EDL between E + M vs. C was 16.6 points (d = 6.52), 10.4 points between E + M vs. E (d = 3.22), and 6.1 points between E vs. C (d = 2.21) in favor of E + M or E (p < 0.001).After the E-program, the M-program in E+M further improved M-EDL for 3 months by 3.5 points  = 32.2–117.3, p < 0.020 to p < 0.001] but not for SE . There were no changes in any of the outcomes in C over the 3-week control period while E + M and E completed the E-program, but declined (p < 0.001) in all the variables by month 72 except in TUG . By month 72, E + M lost 4.5 kg , 2.9 and 3.7 kg more than C and E. Exergaming improved the PDQ similarly in E + M and E by 15.3 points . E+M maintained this improved score for 72 months, which was then ~25 points clinically better (d = 4.55) than the score in E or C , with a difference of ~6 points  in favor of E + M vs. E and C at 72 months  and E+M further improved BDI by 7.1 points  at month 72. The effects of E-program lasted 36 months. At 72 months, the difference in BDI between E + M vs. C was 13.3 points (d = 5.92), 8.8 points between E + M vs. E  in favor of E + M or E . Exergaming improved TUG by 6.7 s . The patients in the group E vs. group C completed TUG still ~5 s faster  at month 36 and E + M sustained its 6.7-s improvement and completed TUG ~7 s faster than the other two groups at month 72  in the four static standing tests. E + M maintained but did not improve this reduction at month 72 when sway path was ~20 cm shorter in E + M vs. E and C . Sway path in the four measures averaged, was still shorter by ~5 cm  in E vs. C at month 36. The L-Dopa equivalents increased least in E + M by 317.0 mg·d−1  compared with increases in E by 427.2 mg·d−1  and in C by 765.2 mg·d−1  and at 72 months r = −0.696 . As the primary outcome in E + M plateaued after 3 months during the M-program (p > 0.05).The baseline M-EDL correlated with Exergaming-induced changes in M-EDL at 3 weeks -program , we compn = 5), neoplasms (n = 4), falls (n = 3), ischemic heart disease (n = 1), pulmonary embolism (n = 1), and PD (n = 1). p < 0.05) between these two groups in M-EDL and PD years.Unrelated to the study, 1, 5, and 9 patients, died in E + M, E, and C groups, respectively . Cause oWe examined the immediate and lasting effects of a 3-week-long, high-intensity and high-frequency Exergaming agility E-program with and without a 6-year-long high-intensity Exergaming agility maintenance M-program, on the motor and clinical symptoms in PwPD. We found that the short-term E-program improved the PD symptoms up to a year during detraining but the subsequent 6-year-long M-program was needed to further increase or sustain the initial improvements in the symptoms, quality of life, and drug dose. The L-Dopa equivalents increased over 6 years but the increases were less in E+M than in the other two groups. The drug doses at the baseline and the changes in dosing did not correlate with the changes in other outcomes.The Exergaming E-program improved M-EDL by 7.4 points or 37%  above thd = 0.82–5.92, 14.6–39.3%) (d = 4.48). However, the changes in PDQ, EQ-5D, and BDI were not inter-related , providing no mechanistic links among the changes in these outcomes. Our BDI data strengthen the conclusion of a review reporting favorable effects of exercise on depression in PwPD  change reported after a 12-week-long, weight-shift-focused Qigong therapy  . PDQ mea.3% . PDQ6–39.3%)  and the  in PwPD . However in PwPD . The imp in PwPD  and stan in PwPD , 18, 47.d = 1.35, p < 0.001). Afterward, the three times weekly M-program sustained but did not further increase these gains at month 72  which was registered at All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Intensive interdisciplinary pain treatment (IIPT) involves multiple stakeholders. Mapping the program components to its anticipated outcomes  can be difficult and requires stakeholder engagement. Evidence is lacking, however, on how best to engage them. Logic analysis, a theory‐based evaluation, that tests the coherence of a program theory using scientific evidence and experiential knowledge may hold some promise. Its use is rare in pediatric pain interventions, and few methodological details are available. This article provides a description of a collaborative logic analysis methodology used to test the theoretical plausibility of an IIPT designed for youth with pain‐related disability. A 3‐step direct logic analysis process was used. A 13‐member expert panel, composed of clinicians, teachers, managers, youth with pain‐related disability, and their parents, were engaged in each step. First, a logic model was constructed through document analysis, expert panel surveys, and focus‐group discussions. Then, a scoping review, focused on pediatric self‐management, building self‐efficacy, and fostering participation, helped create a conceptual framework. An examination of the logic model against the conceptual framework by the expert panel followed, and recommendations were formulated. Overall, the collaborative logic analysis process helped raiseawareness of clinicians’ assumptions about the program causal mechanisms, identified program components most valued by youth and their parents, recognized the program features supported by scientific and experiential knowledge, detected gaps, and highlighted emerging trends. In addition to providing a consumer‐focused program evaluation option, collaborative logic analysis methodology holds promise as a strategy to engage stakeholders and to translate pediatric pain rehabilitation evaluation research knowledge to key stakeholders.  According to the Medical Research Council, a complex intervention is described as one that contains several interacting components, requires various behaviors to be exhibited by both those delivering and those receiving it, incorporates different groups and organizations, and includes many different outcomes, all the while exhibiting flexibility or tailoring.Theory‐based evaluation is an approach that may facilitate stakeholder engagement.In an attempt to broaden the application of this evaluation approach in pediatric health and rehabilitation, this article aims to provide details on the logic analysis methodology including the strategies targeting stakeholder inclusion, the data collected, and the analyses used. To do so, we will present an example of its application in a preliminary evaluation of an implemented IIPT for youth with pain‐related disability and share the findings assessing whether this IIPT was theoretically designed to achieve its desired outcomes.22.1With funding from a large philanthropic donation, the IIPT in Western Canada was conceived in response to a growing number of youth presenting with pain‐related disability. This cohort‐based IIPT was influenced by the day‐hospital model described by Logan et al.2.2To determine whether the core intervention components and critical contextual conditions were present to produce the desired outcomes, a direct logic analysis was used.2.3An expert panel of representatives from stakeholders involved in the treatment designed for youth with pain‐related disability was identified by facility leadership and recruited via email invitation. The 13‐member panel consisted of five clinicians, a program coordinator, and healthcare manager, all of whom had experience (range 2‐15 years) treating youth with pain and/or disability . Also included were two teachers with over 10 years of experience academically supporting youth with an array of physical and mental health conditions, two youth managing pain‐related disability, and their parents. As no standards exist to guide the appropriate number of stakeholders to engage in a panel, guidance was gleaned from the consensus building literature, where a diverse group of 5‐15 participants is recommended.2.4To foster an environment conducive to stakeholder engagement, several activities preceded the evaluation process. First, a charter of the role and responsibilities was created and, once agreed upon, was signed by all expert panel and research team members. Additionally, educational resources and training sessions associated with the logic analysis methodology were provided . The 3‐step logic analysis process described by Brousselle & ChampagneTable 2.4.1In this first step of the 3‐step logic analysis methodology, three data collection methods were used to generate the data required to construct a stakeholder representation of the logic model. These included document analysis, stakeholders’ surveys, and group discussions. All available historical documents , be related to self‐management, self‐efficacy, and/or participation in meaningful activity ), and have a multi‐ or interdisciplinary focus. Retrieved titles and abstracts were screened by two reviewers for relevance. Entire manuscripts were then examined. Reference lists were inspected, yet no additional studies were identified. Once completed, original manuscripts cited in the review studies were scanned for additional relevant information.A data extraction form  described program activities as a function of the disciplines (see Appendix 3.1.3(Clinician 1)Our target population should include parents and the school, but also others in their community environment. At the first expert panel group meeting, a new program objective drafted and distributed prior to the meeting was validated. The program objectives became “To provide youth with pain‐related disability and their parents the knowledge, skills, and tools to self‐manage their pain, build their self‐efficacy, and promote their participation in meaningful activities, despite their pain.” Furthermore, based on expert panel discourse as per the member below, the program reach was extended to include school and community personnel.(Youth 2)I think two things are absolutely fundamental in this program: the education group sessions and the connections you have with the other participants. Some activities and processes were omitted, while others were added, or further detailed. Program activities, which provided support, most valued by parents and youth were underscored.(Youth 1)In terms of long‐term outcomes, it should be how much knowledge is retained. Because if you can refine the application of that knowledge; and once you build routines, you’ve found a way to make it work for you. Youth also recognized activities that should be added to further improve their outcomes. Such activities focused on self‐advocacy and the need to facilitate their transition back to their community following the program. The expected outcomes were adjusted and further elucidated based on panel member's experience.Finally, contextual factors believed to be essential for program success were discussed, and agreement was reached. These factors were associated with the preprogram screening, access to specialized health human resources, and participant characteristics. Figure 3.23.2.1Table 3.2.2Table Self‐management, defined as a person's ability to acquire and apply the skills and knowledge to manage their symptoms, is learned with the support of one's family, community members , and healthcare professionals.Self‐efficacy, defined as a youth's confidence in their ability to function effectively while in pain,Participation, defined as one's involvement in life situations , is an important pediatric rehabilitation outcome.Contextual conditions essential for program success were also found in the literature. Admission criteria across IIPT programs worldwide are similar, of which, pain impacting function, and youth and parent commitment to a self‐management approach dominate.3.3When detailed IIPT components, their links, and anticipated outcomes were systematically compared to the conceptual framework, generally speaking, the scientific evidence supported the program theory plausibility. Furthermore, interconnectivity between the three IIPT program objectives was illustrated. Below the IIPT program, strengths are presented, followed by recommendations for improvements.3.3.1Regarding refining the self‐management intervention for youth, our IIPT intervention aligned well with the evidence contained in the conceptual framework. As per the evidence, psychoeducation was acknowledged as a valued tenet of the program. Many teaching approaches  recognized as effective were incorporated in the program group activities and included opportunities for practice in real‐life environments . These peer‐learning moments were highly valued by expert panel parent and youth members and recognized as pivotal in achieving positive outcomes. However, a need to incorporate additional community‐focused transition opportunities was underscored by both parents and youth, and by the scientific evidence reviewed.In relation to building self‐efficacy, our IIPT program also performed well against the scientific evidence of the conceptual framework. In addition to family counseling and individual psychological interventions, many targeted activities identified as beneficial  in the evidence were already incorporated in the IIPT. Moreover, the inclusion of community‐based activities  in the IIPT, designed to foster problem‐solving, decision‐making, and self‐management skills and their generalization to real life, was strongly supported by the scientific evidence and the experiential knowledge of the youth expert panel members. However, youth panel members also requested even further guidance on the safe return to such activities postdischarge.With respect to fostering participation in meaningful activity, the IIPT included several components deemed effective based on the evidence. Sports, recreation and leisure counseling, advocacy education, and youth and parental training in activity and environment modifications were activities already incorporated in the IIPT and for which conceptual framework scientific support existed. Transition meetings with school personnel, part of the current program discharge process, were acknowledged by youth and parent expert panel members as an opportunity to foster collaboration with teachers, which coincided with the conceptual framework evidence. Youth expert panel members not only valued these meetings, they requested additional tools to further facilitate their ongoing advocacy initiatives in this context postdischarge.Finally, concerning creating an ideal context to achieve the anticipated program outcomes the IIPT fulfilled many of the prerequisite conditions identified in the conceptual framework. When compared, the IIPT admission criteria, key program features, and team memberships shared many similarities with studies included in the conceptual framework.3.3.2When comparing the logic model to the conceptual framework, three main areas of improvement associated with the reach, activities, and processes of the evaluated IIPT were presented to the expert panel for consideration. First, the importance of adopting a developmental lens to the acquisition of knowledge and skills aligned with the expectations of different age groups was recognized. Although the IIPT integrates school‐based, sports, leisure, and recreation activities, the evidence supported incorporating sessions addressing topics such as vocation and work, independent living , and the management of intimate relationships, for older youth . Youth expert panel members also advocated for postprogram support associated with the quickly changing responsibilities and mounting societal expectations inherent to this age group. To incorporate this empirical and experiential knowledge, the inclusion of developmental goals to the already existing goal‐setting process was suggested. The conceptual framework also highlighted the need to expand the reach of the program to include youth's broader social networks. Enhancing peer support through educating classmates and school personnel on pain‐related disability and on how to support to those suffering from this condition was recommended. Expert panel clinicians, youth, and parents’ members alike acknowledged this missing pillar in the IIPT. Finally, the conceptual framework highlighted emerging evidence supporting the use of the web and application technology. Although the technological trials have been limited to one or two of the IIPT components , these technologies hold promise for families for whom access to trained professionals, distance from care facilities, and long waiting times are major barriers. However, web‐based expansion of any of our program component was not acknowledged or recognized as a gap by our expert panel. Upon review of these IIPT improvement recommendations and in light of the organizational constraints raised by the health manager expert panel member, the panel provided the following recommendations to the hospital leadership team: (a) expand information provided to older adolescents to incorporate vocation, work, independent living, and relationships; (b) incorporate self‐management goals tailored to the developmental spectrum; and (c) broaden the psychoeducation to involve peers and school personnel.4The purpose of this article was to detail the logic analysis methodology and to share the findings of the program theory testing of an IIPT using this approach. As a collaborative IKT approach, this evaluation methodology proved helpful in many ways. First, logic analysis provided an opportunity to create a shared understanding of the complexity of IIPT among stakeholders, highlighting previously unidentified intervention and context interactive mechanisms. Stakeholder engagement was critical in ensuring the accuracy, validity, and the integrity of the implemented IIPT description. Furthermore, stakeholders’ reflections, in particular those of youth and their parents, were crucial in establishing those causal mechanisms and activities most valued. Through this value‐based process, mechanisms were identified where interactions between the invention and the context occurred. Complex interventions, like IIPT, are built on a number of components, which may be dependent and interdependent, and where interactions between the intervention and the context exist.Secondly, the logic analysis process assisted in unveiling health professionals’ beliefs about the causal mechanisms thought to contribute to the achievement of the anticipated outcomes. It provided an opportunity to weigh these assumptions against two important sources of validity: scientific evidence and youth and caregivers’ experiential knowledge and values. More importantly, both these sources failed to confirm clinicians’ assumptions of discipline and activity‐specific mechanisms. In evaluation research, it has been recognized that the mechanisms of change are not so much linked to the interventions per se, but instead to the participants’ reasoning and responses generated by the activity and the context which lead to the outcomes of interest.Thirdly, the conceptual framework used a recognized evidence review method and presented a synthesis of current evidence to the expert panel members. This evidence‐informed framework stimulated practice reflection and comparison with experiential knowledge and values. As such, logic analysis presented an innovative way to integrate IKT, addressing the persisting knowledge‐to‐practice gap in pediatric rehabilitation. Discovering scientific evidence to support many of the causal mechanisms of the evaluated program and gaining awareness of those components most valued by youth and their families were noted by clinician expert panel members to be most enlightening part of this collaborative process. Whether this reflective process and increased awareness of the evidence prompt behavior and practice change in clinicians will require further investigation.Engaging stakeholders in logic analysis has been previously recommended.Specific evidence‐informed practices and strategies to foster stakeholder engagement were incorporated into this logic analysis methodology. Targeted activities included (a) choosing a sample of parents and youth who have used the services,Despite our best efforts, this study should be interpreted with some limitations in mind. First, the nonequivalent numbers in each of our stakeholder groups on our expert panel may have biased our results and may have created a power imbalance in favor of clinicians in the group discussions. A variety of data collection methods were, however, used, incorporating anonymous strategies  to ensure authentic perspective were expressed by expert panel member, decreasing social desirability biases. Second, despite expansive recruitment efforts, limited diversity was evident in our expert panel membership. Although youth and parents were representative of the population using this program, other recruitment strategies should be explored if this methodology is expanded to interventions servicing a more cultural and ethnic diverse population. Third, the inclusion of expert panel members into the conceptual framework development could be enhanced. In previously described logic analysis processes, the conceptual framework phase was completed by the evaluator only. Although the expert panel members were included in many stages of the conceptual framework construction, incorporating stakeholders in the data extraction and theming processes of the scoping review could be added if appropriate oversight was provided.Theory‐based evaluation provided an opportunity to further detail the causal path of IIPT rehabilitation intervention, leading to a better understanding of these interventions, and evaluated the plausibility of the program theory in achieving its anticipated outcomes. Stakeholders were implicit to this process. The methods presented in this article, where scientific and experiential knowledge were weighed in a similar manner, provided a collaborative, pragmatic, and realistic approach, representative of the clinical environment in which most healthcare providers conduct evaluation. Engaging stakeholders, including parents and youth, in the logic analysis represents a catalyst for better understanding complex of pediatric pain rehabilitation interventions, such as IIPT, and their evaluations. Furthermore, it represents a novel IKT method to narrow the ongoing knowledge‐to‐practice gap existent in the field.The authors have nothing to declare.This study was conducted by Karen Hurtubise as a requirement for the Doctor of Philosophy Degree. Karen Hurtubise's contribution to this manuscript included its conceptualization, the literature and scoping review, the data analysis, creation of the conceptual framework, the presentation of the findings, and the writing of this manuscript. Dr Astrid Brousselle provided methodological evaluation expertise in the development of this study, supervision, and guidance during the various stages of the study. She also viewed all results and provided feedback on the manuscript prior to submission. Dr Chantal Camden provided expertise in stakeholder engagement processes and integrated knowledge translation strategies throughout the study, validated the results at the various stages of the study, reviewed, and provided feedback on the manuscript prior to submission. She also provided supervision and guidance throughout the study.Supinfo 1Click here for additional data file.Supinfo 2Click here for additional data file.Supinfo 3Click here for additional data file.


Analysis of the robustness and vulnerability of metro networks has great implications for public transport planning and emergency management, particularly considering passengers’ dynamic behaviors. This paper presents an improved coupled map lattices (CMLs) model based on graph attention networks (GAT) to study the cascading failure process of metro networks. The proposed model is applied to the Shanghai metro network using the automated fare collection (AFC) data, and the passengers’ dynamic behaviors are simulated by GAT. The quantitative cascading failure analysis shows that Shanghai metro network is robust to random attacks, but fragile to intentional attacks. Moreover, there is an approximately normal distribution between instant cascading failure speed and time step and the perturbation in a station which leads to steady state is approximately a constant. The result shows that a station surrounded by other densely distributed stations can trigger cascading failure faster and the cascading failure triggered by low-level accidents will spread in a short time and disappear quickly. This study provides an effective reference for dynamic safety evaluation and emergency management in metro networks. With the development of the metro network, it has become the main means of transportation in mega-cities. For example, Shanghai has a metro system with 303 stations and 350 tunnels over 617 km [As the metro system plays an increasingly important role in the urban mobility, if the metro system breaks down, the consequences will be extremely serious and unbearable. In the metro system operation, the negative result is usually caused by cascading failure, which can be triggered by the signal failure of infrastructure, natural disasters, terrorist attacks and other emergencies . For exaThis paper provides an advanced and accurate analytical model and theoretical support for the safety operation and the emergency management of metro systems. The main contributions of this study are as follows. (1) An improved CMLs model based on GAT is proposed, which can better reflect the cascading failure of metro networks than the previous coupling models. (2) Based on the coupling strength trained by GAT, the passenger flow redistribution during the cascading failure is dynamically simulated. (3) A case study with the AFC data of the Shanghai metro system demonstrates the topological characteristics of the Shanghai metro, the failure degree under intentional attacks (IAs) and random attacks (RAs) and the influence of the propagation process on the robustness. The rest of this paper is organized as follows. In In the early stage, the network-based analysis approach ,7 providHowever, the above network-based studies do not offer a rigorous comparison among metro networks. In this respect, the topological characteristics of the world’s 20 largest metro systems  and the In addition, as there exists the dynamic passenger flow in the metro networks, some studies loaded dynamic passenger flow data into the metro networks based on the static network topologies ,15,16. TSome empirical studies on the cascading failures in different networks , such asCompared with the load-capacity model, the CMLs model  has the However, the bidirectional flow characteristics of the metro network are not considered. The CMLs cascading failure algorithm based on the actual case of the Nanjing metro was optimized by considering the two-way metro problem and the passenger flow redistribution . In factTo deal with the above two problems (the two-way metro problem and the passenger flow redistribution), further research needs to accurately quantify the correlation between nodes, including the passenger flow correlation and topology correlation. Deep learning can accurately extract the relationship between data objects through learning the right representation for the data. To learn a vectorized representation containing sufficient information from complex graph data, the concept of graph neural networks (GNNs) was first proposed , which wFor a metro network, the graph-based deep learning system can explore the internal relationship between nodes (or lines) and passenger flow in different time periods to provide reference for transportation planning and management. Han  proposedHowever, GCNs still have two major limitations. One is that it is unable to handle inductive learning, which means that the images processed by training and testing are different. The other is that it is difficult to assign different weights to different neighbors for directed graphs. Therefore, recently, more variants of spatial-based GCNs have been developed ,38. AmonIn this paper, we aim to train passenger flow correlation between stations in the metro system with GAT, and simulate passenger flow redistribution based on this. Based on the application of CMLs model, it can compensate for the assumption that all nodes have the same coupling relationship, and capture the complex non-stationary time dynamics and spatial dependence among metro stations.As shown in We use this data to construct the adjacency matrix of the Shanghai metro network and the graph dataset that will be learned by GAT. Moreover, RAs and IAs are used to simulate attack or failure, and the cascading failure process of the Shanghai metro system is analyzed based on the proposed improved model. Three important stations are chosen to illustrate how IAs work. They are Century Avenue (CA) with a maximum node degree of 7, Shanghai Railway Station (SRS) with maximum in-flow of 6095 and Xinzhuang Station (XZ) with maximum out-flow of 3307 in the first Wednesday of our dataset. The three chosen stations are also labelled in The research flow diagram is shown in The core of the attention mechanism is to assign weight to the given information, and information with high weight means that attention should be focused. As the sole layer throughout all the GAT architectures, a single graph attentional layer will first be described in this study. As with all attention mechanisms, a single graph attentional layer can be mainly divided into two parts: calculation of correlation coefficient and aggregation.N is the number of nodes, and F is the number of features in each node. The output will be a new set of node features, i and its neighbor j and map the spliced high-dimensional features by a shared attentional mechanism i and node j. In this step, we only compute In our experiments, the whole metro network is a graph and each station is a graph node with operation data  as its node features. The input to graph attentional layer is a set of node features, After the calculation of flow coupling strength, we aggregate the features of each node by applying a nonlinearity activation function K independent attention mechanisms, and then concatenate their features (Equation (4)):k-th attention mechanism, and k-th input shared linear transformation’s weight matrix.We also employ multi-head attention to stabilize the learning process in our experiments. This step will be executed by i . The first step is to transform the characteristic information of metro stations through a linear layer and calculate the attention coefficient When a metro station fails because of an accident, the passenger flow of the current time step will be redistributed to other lines connected with the station according to the demand of the passenger flow itself. These redistributed passenger flows will gradually spread to the entire metro network, leading to load stress at other stations and even cascading failure. The flow coupling strength trained in i fails, the flow i will be shared by other neighbors of station j. Since the link between two stations with a larger flow coupling strength means that it has stronger passenger flow carrying capacity and passenger flow preference in most cases, we assume that it will have a larger proportion of passenger flow. Then, If station The passenger flow redistribution model enables the relevant management agencies to obtain the transfer rules of passenger flow when a failure occurs based on the large-scale learning of the natural passenger flow movement. This indicates that the transfer of passenger flow has preference consistency in a certain period of time. Generally, this rule exists in real metro systems because when a station or line suffers a disaster or attack, passengers are more likely to choose a nearby transit service to escape chaos and accidents as quickly as possible.CMLs have been widely used in the research of temporal–spatial traits of complex systems. Most of these researches assumed that different pairs of stations had a regular coupling topology. However, in a real metro network, the synchronization of coupling topology on CMLs does not exist due to the heterogeneity between different station pairs. Additionally, the cause of cascading failure is not only related to its topology attributes, but also related to the passenger flow transfer rules.i at the t-th time step. i happens. A larger R indicates a more serious attack or failure. If node i has not encountered an attack, We first consider a CML of i fails because of the passenger flow overload at time step s, the state i will still be affected by s + 1. In case the state of one or more neighbors of the failed station is larger than threshold value 1, a new round of failures will be triggered.In this work, i.If we add the passenger flow coupling strength, Equation (7) can be rewritten as Equation (8):In order to better simulate the real metro network topological characteristics, we consider the bidirectional metro problem and improve the CMLs model with both heterogeneity between different station pairs and passenger flow coupling strength (Equation (9)):We can assume t-th time step, which is the ratio of the number of station failures over the metro network with N stations. We then use This formula derivation is reasonable because the out-flow of one station will not affect its state at the next time step. We use The correlation between the scale of cascading failures and the external perturbation under different attacks;The speed of cascading failures in a discrete time series under different attacks;The difference between this improved model and existing CMLs model.With this improved model, the problems needed to be studied are as follows: R to different stations. In R in the four chosen Wednesdays of the experiment dataset. As shown in R increases, the curve of CA will first make a mutation, and then that of XZ, SRS and RA. The results show that the large-scale cascading failure of IA is more easily triggered than RA because only a relatively small attack is needed to realize the rapid spread of cascading failures over the network. Thus, the Shanghai metro network is more robust to RFs than to IAs. This mutation effect also shows that for important stations, even a very small increase in the perturbation value R can quickly trigger large-scale cascading failure of metro stations. The results reveal the inherent vulnerability of the metro as a special mean of transportation.We first investigate the thresholds to trigger maximum cascading failures under different attacks. Different levels of attack have been simulated by adding different I of XZ is held constant, there is a relatively large decrease as R increases. This result indicates that when the value of R reaches a certain threshold, the importance of CA and SRS to the whole metro network is higher than that of XZ. However, this phenomenon is not significant in From I of each station increases firstly and then decreases instead of increasing gradually to 1. The maximum value of balanced I is close to 0.8, which demonstrates that after adding the topology coupling strength and passenger flow coupling strength, the Shanghai metro system is impossible to fail completely. Previous studies whose experiments guaranteed complete failure of the whole metro system as long as the perturbation R is large enough generally assume that Moreover, we find that in all the subgraphs of =ε2=0.25 ,20,21,23This means that as long as the I always increases fast when R than that of IAs.By comparing R, and it also changes with time, however, the perturbation in the station that leads to steady The critical perturbation Rs at different time are shown in t. When R is small, for example, In Compared with the curves of XZ shown in R. For instance, the curves with Compared with the curves of RA shown in t, which is verified by the Kolmogorov–Smirnov normal distribution test in p value >0.05, the data is normally distributed. When R is small, the peak times of curves under both RA and IAs are always within two steps. It indicates that the cascading failure triggered by a slight accident or attack will be limited in a small-world network and terminate quickly. However, the peak time will significantly be delayed as R increases to a certain value. It implies that the propagation of cascading failures is mainly reflected in the early stage. For example, It can be seen from R are almost similar with each other, which indicates that SRS has the predictability to perturbations, at least when R changes. When attacking a random station on a metro system, the attack often needs to be more intense, which means that a larger R should be added. This echoes the previous conclusion.In As the classical algorithms of GNN, both GCN and GAT aggregate the features of neighbor nodes to the central node by learning the new node feature expression. The difference is that GCN uses the Laplacian matrix, while GAT uses the attention coefficient. In The prediction of this study is passenger flow characteristics, and the dataset includes every 5 minutes’ passenger flow. The correlation between different stations’ passenger flow can be trained by using historical passenger flow to predict the specified station passenger flow in the future, which is passenger flow coupling strength. In order to solve the problem of outlier data, this study takes the average by sliding window to make passenger flow data more stable .Compared with GCN, GAT performs better in predicting the characteristics of the Shanghai metro passenger flow, which is shown in Since topology and passenger flow coupling strength describe two kinds of interactions between different pairs of stations in a network, it is necessary to consider them to investigate their influence with cascading failure. The topology coefficients and passenger flow coefficients of the chosen three stations have been shown in Shen  proposedε1=ε2 , while tAs shown in In this work, an improved CMLs model based on GAT to investigate the cascading failure process of metro networks is proposed, which provides a theoretical reference for the emergency management when the metro system encounters an accident.By learning the real passenger flow data of the Shanghai metro network, we obtain the passenger flow coupling coefficients that can best represent the passenger flow distribution of the entire Shanghai metro network. The practical meaning of the passenger flow coupling coefficient is the proportion of passenger flow allocated from one station to adjacent stations. The dynamic OD requirements are considered to accurately simulate the network state when cascading failures occur. By analyzing the Shanghai metro topology network and passenger flow data, three key stations are selected as the targets of IAs. The simulation result of the cascading failure shows that the Shanghai metro network is more robust to Ras than to IAs. For IAs, when the degree of the perturbation increases to a threshold, the mutation effect occurs, which means large-scale cascading failure of metro stations is triggered. Previous studies insisted that a large enough perturbation or attack can cause the complete cascading failure of the whole metro network. However, in this study, the Shanghai metro system is impossible to fail completely after considering the topology coupling strength and passenger flow coupling strength, which shows the simulation is more realistic. The scale of cascading failure changes with time increasing, however, the perturbation in the station which leads to steady state is approximately a constant. Small perturbations will spread in a short time and disappear quickly, which means a low-level accident on a station will cause limited other metro stations to fail in a short time. A station surrounded by other densely distributed stations will trigger cascading failure faster. It is easier for cascading failure to be predicted in the beginning and be increasingly unmanageable over time because the destruction of the metro network will restrict the passenger flow evacuation. There is an approximately normal distribution between instant cascading failure speed and time step and the propagation of cascading failures is mainly reflected in the early stage. Different attacks can significantly influence the cascading failure process. The transfer of passenger flow between each station and its neighbors is indeed unevenly distributed. Increasing the coupling strength of one station will reduce the robustness of the whole network.The method proposed in this paper can simulate the cascading failure in other metro systems more accurately, not only the Shanghai metro system. The simulation considers both topology attributes and passenger flow transfer rules. There are still several meaningful research aspects. For example, further research can focus on the cascading failure triggered from multiple nodes. The passenger flow redistribution algorithm should be improved by considering alternative modes of transportation. Moreover, the difference of cascading failures caused by different failure scenarios is worth analyzing.


We aimed at describing for the first time peripheral small-fiber neurotoxicity and pain sensitization in survivors of pediatric acute lymphoblastic leukemia after stem cell transplantation (SCT).In a cross-sectional, retrospective, single-center study, we assessed 25 relapse-free long-term survivors  using a reduced version of the pediatric-modified total neuropathy score for clinical assessment and Quantitative Sensory Testing (QST). Inclusion criteria: p < 0.0038). Except for one, all survivors showed at least one abnormal QST parameter. When using QST, signs of small and large fiber dysfunction were present in 22 (88%) and 17 (68%) survivors, respectively. More than half of all survivors were found to experience pathologic sensitization to pain.Nine patients (36%) had peripheral neuropathy as defined by the clinical red-pmTNS (≥ 4). The QST parameters mechanical pain sensitivity, mechanical detection threshold, thermal sensory limen, vibration detection threshold and pressure pain threshold were significantly abnormal in the survivor cohort  contains supplementary material, which is available to authorized users. Acute lymphoblastic leukemia (ALL) is the most common malignancy in childhood  and hemFirst-line polychemotherapy of ALL includes neurotoxic components such as vincristine (VCR) and methotrexate Board  and the Peripheral polyneuropathy is subdivided into damage of large fibers, small fibers and a combination of both. Large-fiber neuropathy (LFN) affects Aα-and Aβ-fibers and is clinically characterized by loss of vibration perception, proprioception and motor control  are the most accurate diagnostic methods for large fiber CIPN  especially SFN besides LFN using a questionnaire for symptoms and clinical signs for the assessment of peripheral neuropathy as well as QST, (2) patients with signs of pain sensitization with QST, and (3) clinical risk factors for peripheral neuropathy. We hypothesized to reveal abnormal somatosensory function and pain sensitization in a substantial percentage of survivors of pediatric ALL after SCT.The Ethics Committee of Charité-Universitätsmedizin Berlin approved our study (EA2/105/16) in accordance with the Declaration of Helsinki. We identified survivors from the database of our pediatric SCT program at Charité-Universitätsmedizin Berlin.2 (range 0–22.5 mg/m2). Cyclosporine A, prednisone, mycophenolate mofetil, and if needed corticosteroid pulses, were administered. Standard criteria defined acute or chronic Graft versus Host Disease (GvHD) .Results of QST were analyzed according to the standardized protocol using the published reference data for gender, age and body region . Our assessed cohort did not differ significantly from non-participants except for an increase of males and patients treated with SCT in first remission Table .Table 1CThe median of the total reduced pediatric-modified Total Neuropathy Score was 2 (0–10)  survivors showed at least one significant abnormal parameter  in QST, 17 (68%) at least two and 14 (56%) three or more parameters  and mechanical pain . Only few survivors showed decreased nociceptive thermal perception for cold  and warm . Nine (32%) survivors showed paradox heat sensations (PHS). A patient was diagnosed with the phenomenon of pain sensitization when showing at least one abnormal finding such as hyperalgesia to pressure, mechanical stimuli, allodynia or when testing wind-up ratio. This meant that the perception of a stimulus that is not painful in the healthy age-matched population was already perceived as painful. Half of our cohort fulfilled this criterium (13/25), i.e., that survivors had at least one abnormal finding in the four above-mentioned tests. Among them, seven survivors showed pain sensitization in one QST parameter, five in two and one in three, respectively.Isolated positive symptoms like hyperalgesia to thermal or mechanical stimuli, allodynia and PHS were found in 4 patients (16%). Isolated negative symptoms like hypoesthesia and hypoalgesia to thermal or mechanical stimuli were detected in 7 patients (28%). Thirteen survivors (52%) presented a combination of positive and negative symptoms. The median number of abnormalities in QST was 3 .When grouped into the LoGa scheme of losses and gains of function, the most frequent somatosensory deficit combination was L3G2 found in 7 patients (28%) , whereas 7 patients (28%) revealed a dominant SFN. A combination of LFN and SFN was found in 15 patients (60%).We identified no significant clinical risk factor for abnormal red-pmTNS score and only one for QST parameters. Using Pearson’s correlation shorter recovery time after SCT was associated with increased sensitivity to pinprick stimuli  and decreased VDT .The red-pmTNS uncovered signs and symptoms of functional impairment in about one out of three survivors. In 2013, Gilchrist introduced the pmTNS and demonstrated its validity and reliability in pediatric cancer patients Below is the link to the electronic supplementary material.


Resident achievement data is a powerful but underutilized means of program evaluation, allowing programs to empirically measure whether they are meeting their program aims, facilitate refinement of curricula and improve resident recruitment efforts. The goal was to provide an overview of available metrics of resident achievement and how these metrics can be used to inform program aims.A literature search was performed using PubMed and Google Scholar between May and November of 2020. Publications were eligible for inclusion if they discussed or assessed “excellence” or “success” during residency training. A narrative review structure was chosen due to the intention to provide an examination of the literature on available resident achievement metrics.57 publications met inclusion criteria and were included in the review. Metrics of excellence were grouped into larger categories, including success defined by program factors, academics, national competencies, employer factors, and possible new metrics.Programs can best evaluate whether they are meeting their program aims by creating a list of important resident-level metrics based on their stated goals and values using one or more of the published definitions as a foundation. Each program must define which metrics align best with their individual program aims and mission. Every residency program desires to offer truly excellent training to their learners. However, excellence represents different things to different physicians, institutions, communities, and patients.The Accreditation Council of Graduate Medical Education (ACGME) requires programs to develop aims that are reflections of the program’s mission statement as part of their Self-Study.The author group is made up of practicing U.S. emergency medicine physicians from multiple academic institutions and includes four members of the residency leadership team and two Medical Education Fellows.A narrative design was chosen in order to examine the literature regarding possible metrics of assessing resident achievement. The scope of the review was designed to focus on actionable ideas for programs.Individual searches were conducted by the authors using the Google Scholar and PubMed databases for relevant keywords, including “achievement,” “success,” “resident,” “physician,” “training,” and “graduate medical education”. From the list of articles generated, a list of 17 potential metrics was generated through virtual discussion between experienced scholars using the telecommunications software Zoom  . Other mOur literature search revealed 57 unique papers that met inclusion criteria for the review.The ACGME outlines Milestones that provide a framework for assessing resident performance.The use of faculty assessment data has been demonstrated previously to predict future success in residency.Peer assessments can be another source of important feedback on resident performance. Faculty members were shown in one study to score residents higher than their peers in several sub-competency categories,While resident self-assessment data can be useful for program evaluation, historically, learners have difficulty determining the areas in which they are deficient.20A portfolio assembled by the resident, highlighting key examples where the resident believes they have demonstrated strength as well as weaknesses that they are working on could be used by programs in conjunction with other more traditional forms of assessment, with faculty assessing the resident’s portfolio.Fellowship training results in the attainment of specialized knowledge and skills beyond those of graduates pursuing general practice. The number of residents deciding to pursue fellowship could be a benchmark of success for programs, particularly those affiliated with academic institutions where training future leaders in the specialty is valued. This has been used previously as a means of evaluating general surgery programs.22The number of residents serving in leadership roles within residency programs, medical schools, or healthcare administration near the beginning of their post-residency career could represent an important focus for program evaluation.Residency programs have considered the number and/or quality of scholarly works produced over the course of training as a marker of excellence given that this represents one of academia’s most widely accepted currencies.Objective measures, such as the in-training exam (ITE) or USMLE Step 3, have been shown to correlate well with future passage of specialty board exams.The need for remediation implies a deficiency in one or more ACGME core competencies, most commonly medical knowledge, patient care, and professionalism.While clinical performance metrics hold promise as an objective measure of excellence for a residency program, it can be challenging to generate meaningful performance metrics for resident physicians that are free from significant confounders. Markers of efficient care delivery  generated), as well as measures of care quality  rapid return visits  or ICU upgrades, are attractive metrics that could be defined by each program or institution. Higher resident case volumes correlated with better performance in diagnostic radiology,37Resident patient satisfaction scores may be another useful tool by which to benchmark program excellence. While this metric is similar to what attending physicians are measured on, it is controversial whether patient satisfaction scores are appropriate markers for quality of care received and physician performance,37The successful completion of the procedures without complications is necessary for independent physician practice and is patient-centered, making this a potentially attractive target as a program aim. This is often measured by the absolute number of procedures completed through a portfolio or logbook,42A more recent potential measure of residency program success is adaptability. With change accelerated by the COVID-19 pandemic, adaptability in an ever-changing clinical environment has become a focus, as new facets of care, such as virtual healthcare visits, are being implemented rapidly.46The number of graduates choosing to practice in a local community may be critically important for certain residency programs. Success in this regard could encompass not only practicing medicine in these communities, but also serving in other ways, such as health literacy training.While interpersonal skills, communication, and professionalism are measured by the Core Competencies and Milestones, empathy has not previously been a target of significant performance assessment. Empathy has been rated as critically important to successful physicians and has been demonstrated to increase patient satisfaction and improve treatment outcomes.With multiple public health crises ongoing in the U.S., such as racism, COVID-19 and gun violence, some residency training programs may consider advocacy work to be an important measure of success in their graduates. While well-studied metrics to measure the impact of these programs are currently lacking, curricula for residents have been introduced successfully into training programs.Well-being is an important component of success as a physician, as physician burnout has been associated with increased medical errors and decreased adherence to best practices.Given the wide array of potential metrics for residency programs to define excellence, each program should thoughtfully determine which metrics are most meaningful in their environment and for the trainees that they want to produce.Key departmental stakeholders should be included in the selection of appropriate metrics for program aims. Defining new metrics will often cause a change in how resources are deployed, so buy-in from the department chair, as well as the residents that will be directly affected by any changes, will be especially crucial. Benchmark performance levels should be set based on historical performance in the chosen area; if a residency program has never sent a graduate to fellowship, it may be unrealistic to achieve a goal of 80% of graduates choosing this option in the next few years. As the program grows and changes, the excellence metrics may need to evolve also. A program with an interest in developing a track record of resident scholarly publications for example, may want to turn its attention to academic leadership positions after several years of success.Programs may also wish to carefully consider how their definition of excellence is used during the selection of residents to ensure they are meeting their goals while also seeking out residents who can push the program in new directions.This review does have limitations, many of which are inherent to the narrative design of the review. First, some relevant studies may have been missed. Given the broad range of metrics included and the lack of standardized terminology, it is difficult to ensure that all the related literature was assessed for inclusion. However, this was not designed to be an exhaustive search from its conception. Second, it is possible that there was bias in the inclusion or exclusion of certain studies given the non-systematic nature of the review. We attempted to ensure quality studies were included based on assessment by the authors who are experienced medical educators. Finally, more rigorous consensus methodology could have been employed to enhance the content validity of the review rather than the more informal discussion between the author group.Further work will also be required to determine if the introduction of novel metrics, such as wellness or empathy scores, can be effectively used to improve outcomes via program evaluation. Additionally, future work could focus on the long-term impact of program-level metrics; for example, do physicians who initially start in an underserved community stay there, or are patient satisfaction gains sustained even after the program begins to focus in other areas.There are a variety of possible resident-level metrics which can be used for program evaluation, many of which target different aspects of performance beyond clinical skills. Each program should assess the available metrics and decide collectively on those that they consider most aligned with their program’s mission statement, aims and individual and institutional goals and use those to create measurable targets for program evaluation.