Is a national antibiotic stewardship program for ambulatory care associated with a decrease in antibiotic prescribing?In this cohort study comprising members of 389 US ambulatory practices, including clinicians and staff, the Agency for Healthcare Research and Quality Safety Program for Improving Antibiotic Use Program addressed attitudes and culture that challenge judicious antibiotic prescribing and incorporated best practices for the management of common infections. Between September 2019 and November 2020, antibiotic prescribing at clinic visits decreased from 18% to 9%, and antibiotic prescribing at acute respiratory infection visits decreased from 39% to 25%.In this study, the Safety Program appeared to provide a model for stewardship implementation in ambulatory care practices. The Agency for Healthcare Research and Quality (AHRQ) Safety Program for Improving Antibiotic Use aimed to improve antibiotic prescribing in ambulatory care practices by engaging clinicians and staff to incorporate antibiotic stewardship into practice culture, communication, and decision-making. Little is known about implementation of antibiotic stewardship in ambulatory care practices.To examine changes in visits and antibiotic prescribing during the AHRQ Safety Program.This cohort study evaluated a quality improvement intervention in ambulatory care throughout the US in 389 ambulatory care practices from December 1, 2019, to November 30, 2020.The AHRQ Safety Program used webinars, audio presentations, educational tools, and office hours to engage stewardship leaders and clinical staff to address attitudes and cultures that challenge judicious antibiotic prescribing and incorporate best practices for the management of common infections.The primary outcome of the Safety Program was antibiotic prescriptions per 100 acute respiratory infection (ARI) visits. Data on total visits and ARI visits were also collected. The number of visits and prescribing rates from baseline  to completion of the program  were compared.Of 467 practices enrolled, 389 (83%) completed the Safety Program; of these, 292 (75%) submitted complete data with 6 590 485 visits to 5483 clinicians. Participants included 82 (28%) primary care practices, 103 (35%) urgent care practices, 34 (12%) federally supported practices, 39 (13%) pediatric urgent care practices, 21 (7%) pediatric-only practices, and 14 (5%) other practice types. Visits per practice per month decreased from a mean of 1624  at baseline to a nadir of 906  early in the COVID-19 pandemic (April 2020), and were 1797  at the end of the program. Total antibiotic prescribing decreased from 18.2% of visits at baseline to 9.5% at completion of the program . Acute respiratory infection visits per practice per month decreased from baseline (n = 321) to a nadir of 76 early in the pandemic (May 2020) and gradually increased through completion of the program (n = 239). Antibiotic prescribing for ARIs decreased from 39.2% at baseline to 24.7% at completion of the program .In this study of US ambulatory practices that participated in the AHRQ Safety Program, significant reductions in the rates of overall and ARI-related antibiotic prescribing were noted, despite normalization of clinic visits by completion of the program. The forthcoming AHRQ Safety Program content may have utility in ambulatory practices across the US. This cohort study examines antibiotic prescribing rates in ambulatory care practice during a stewardship intervention implemented through the Agency for Healthcare Research and Quality Safety Program for Improving Antibiotic Use. In response to the need for an AS implementation framework in outpatient practices,13 the Agency for Healthcare Research and Quality (AHRQ) established the AHRQ Safety Program for Improving Antibiotic Use  in ambulatory care. This 1-year program was available at no cost to primary and urgent care practices across the US. Through the development of a comprehensive toolkit, the Safety Program assisted participating practices with implementing sustainable AS.Despite national encouragement by government agencies, accreditation organizations, and specialty societies to expand AS to outpatient settings,14 Earlier cohorts of the Safety Program focused on AS implementation in acute15 and long-term care.16 The objective of the present work was to describe the implementation of the Safety Program targeting outpatient practices and measure changes in antibiotic prescribing patterns in participating practices from baseline  to completion of the program . This period also largely overlapped with early stages of the COVID-19 pandemic.The Safety Program approach was adapted from the Comprehensive Unit-Based Safety Program, a method to improve patient safety and quality by emphasizing health care worker teamwork and the science of safety.STROBE) reporting guideline.We performed a cohort study evaluating a quality improvement intervention in a national collaborative of ambulatory practices. The ambulatory care Safety Program was conducted December 1, 2019, to November 30, 2020, across the US. The Safety Program team included the investigators, with input from a technical panel comprising experts in AS, primary care, urgent care, implementation research, patient safety, and patient advocacy. Primary care, urgent care, student health, community-based health centers, and specialty medical practices providing primary care were recruited through strategies such as listservs; social media; outreach to relevant professional societies, state health departments, local medical societies, and area health education centers; attendance at professional conferences; federal agency engagement; quality improvement organization contacts; and insurers. Participating practices were required to use electronic health records. The project was deemed nonhuman participants research and exempt from informed consent by the Johns Hopkins School of Medicine Institutional Review Board. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology  whether antibiotics are necessary, (2) what diagnostic testing is needed, (3) the safest and most effective antibiotic, and (4) ensure patients have an appropriate follow-up plan. Additional educational content included audio presentations, 1-page summary documents, patient handouts, and discussion guides  assist practices with establishing AS infrastructure, (2) assist practice members with understanding best practices in antibiotic prescribing for common infectious conditions seen in ambulatory care, and (3) improve antibiotic prescribing teamwork, communication, and safety culture among the practice members and with patients and family members. During the 12-month Safety Program, there were 14 webinars . Webinars were focused on the diagnosis and management of infectious diseases and used the Four Moments of Antibiotic Decision Making framework .17 This Box. Does my patient have an infection that requires antibiotics?Have I ordered appropriate cultures before starting antibiotics? What empiric therapy should I initiate?A day or more has passed. Can I stop antibiotics? Can I narrow therapy or change from intravenous to oral therapy?What duration of antibiotic therapy is needed for my patient’s’ diagnosis?17Adapted from Tamma et al, 2019.The clinical and administrative AS leads were encouraged to meet with all staff monthly to accomplish the following: (1) review any updates since the last meeting and seek feedback, (2) introduce the educational topic for the month and associated material in the toolkit  and webinar dates, (3) summarize key points from the 1-page document on the monthly topic, and (4) review questions in the discussion guide with the practice . The AS leads summarized comparative, practice-level feedback data received from the Safety Program and were asked to continue the AS activities after completion of the Safety Program, and resources were provided on sustainability.Participants completed a practice assessment describing their current AS infrastructure at the beginning and the end of the Safety Program. Practices submitted monthly data from the baseline period  through the Safety Program intervention . Monthly data consisted of the number of visits, the number of antibiotic prescriptions written categorized by antibiotic class, the number of ARI visits, and the number of antibiotics prescribed for ARI visits. The list of oral and intramuscular antibiotics included is presented in eTable 2 in the International Statistical Classification of Diseases, 10th Revision codes (ICD-10)  to the period of initial widespread recognition of the COVID-19 pandemic (March to May 2020), and to program end (November 2020) were estimated. Data analyses were conducted using Stata, version 16.1 (StataCorp LLC). Findings were considered significant at α < .05 with 2-sided testing.A total of 467 practices enrolled in the Safety Program. Of these, 389 (83%) remained in the program until its completion and 292 (75%) submitted complete data for analysis, including 6 590 485 visits to 5483 clinicians. Common reasons for withdrawal from the Safety Program included COVID-19–related competing interests (36 [46%]), data submission challenges (19 [21%]), and practice closures (14 [24%])  primary care practices, 103 (35%) urgent care practices, 34 (12%) federally supported practices, 39 (13%) pediatric urgent care practices, 21 (7%) pediatric-only practices, and 14 (5%) other practice types . ParticiP < .001). The percentage of practices that developed local guidelines for common bacterial conditions increased from 48% to 61% (P < .001)  to a minimum at the beginning of the pandemic  and returned to slightly more than the baseline rate at the end of the Safety Program  . The panICD-10 codes, decreases in acute bronchitis, and very few influenza visits  through the beginning of the pandemic  and gradually increased but remained below baseline by the end of the Safety Program  . Antibio3 Studies have reported that specific interventions may decrease ambulatory antibiotic prescribing, including educational interventions; electronic decision support; multimodal interventions; and behavioral economics nudges, including commitment posters, peer comparison feedback, and justification alerts.22 The decreases in antibiotic prescribing seen in the Safety Program are similar to or greater than those shown in earlier trials.26 These earlier studies were smaller and provided more practice support than the current project. We believe that the similar success, despite the intervention’s pragmatic rather than controlled nature, shows the feasibility of our approach.In this cohort study, a year-long national program focused on AS implementation in ambulatory care practices was associated with decreases in total antibiotic prescriptions and antibiotic prescriptions for ARI visits. Antibiotic prescriptions decreased by 9 per 100 visits and antibiotic prescriptions for ARIs decreased by 15 per 100 ARI visits. The AHRQ Safety Program appears to meet an important need in ambulatory care settings where total antibiotic prescriptions are high and AS implementation strategies have been limited.8 and associated toolkits.28 However, implementation strategies have been less well defined. We noted that approaches to improve patient safety 14 by engaging clinicians and staff in communicating around antibiotic prescribing, implementing AS interventions, accessing and reviewing antibiotic prescription data, and providing evidence-based guidelines were essential components of ambulatory AS. Quality improvement experts who regularly reached out to individual practices may have also improved continued engagement.15 Safety Program participants appeared to have ongoing perseverance and program engagement despite significant clinical challenges during the COVID-19 pandemic.The Safety Program is the first national program to provide ambulatory practices guidance in AS implementation. Evidence-based interventions for decreasing ambulatory antibiotic prescribing have been incorporated into guidance, such as the core elements of outpatient antibiotic stewardship,31 Similarly, as COVID-19 mitigation measures reduced the rates of visits for common respiratory viral infections,33 visits for non–COVID-19 ARIs decreased.34 We similarly observed decreases in total and ARI visits between March and May 2020. Two studies of US retail pharmacy data also showed a decrease in antibiotic prescribing during that period; however, this decrease was followed by an increase through December 2020.36 National pharmacy data did not reflect clinic visits, so a reduction in ambulatory care visits likely contributed to some of this national decrease in antibiotic prescribing.29 A single-institution study also reported that ambulatory care antibiotic prescriptions decreased at the beginning of the COVID-19 pandemic (March to May 2020), but stabilized in the subsequent months.34 Meanwhile, in our study, antibiotic prescribing decreased through November 2020; this decrease was in antibiotic prescribing per visit, so was not as affected by visit numbers. Furthermore, by the end of the program, practices demonstrated improvements in the infrastructure needed to sustain AS. We believe that these findings provide support for the Safety Program playing a role in a reduction in antibiotic use.Implementation of the Safety Program coincided with early stages of the COVID-19 pandemic, which may have affected antibiotic prescribing. Physical distancing guidelines led to a national decrease in primary care visits between March and May 2020.37 It is possible that low ARI visits but high overall visits reflect visits related to COVID-19, for which ICD-10 codes were not captured in the data abstraction tool. Patients with COVID-19 may have been less likely to have antibiotics prescribed and more likely be seen in urgent care clinics rather than in primary care settings owing to previsit screening protocols.Urgent care practices had a more substantial decrease in both overall and ARI visits compared with other practice types, especially between March and May 2020. At completion of the Safety Program, urgent care ARI visits remained low while overall visits returned to near baseline, consistent with national trends.36 Over the past decade, antibiotic prescribing overall in children has decreased while remaining stable in adults,38 possibly because pediatricians may spend a longer time with patients and reassuring the caregiver when not prescribing antibiotics.39Pediatric practices also had a steeper decrease in antibiotic prescriptions than adult practices. This finding is consistent with national data showing a decrease in antibiotic use to be steepest among pediatricians.40 Second, the Safety Program depended on practices submitting accurate data. Practices were contacted to verify out-of-range data to improve the accuracy of the information. Third, because data abstraction tools were built before the COVID-19 pandemic, data were not collected about COVID-19 diagnoses and no ICD-10 codes were included for COVID-19. Fourth, the focus was on prescriptions, so delayed prescriptions 8 would have been counted.25 However, it is unlikely that this practice changed during the intervention because it was not discussed in the Safety Program. Fifth, called-in and handwritten prescriptions would have been missed because only electronic prescriptions were captured. However, because all practices used electronic health records, most prescriptions would have been sent electronically.This study has limitations. First, despite the findings, it is difficult to distinguish the association of the COVID-19 pandemic with antibiotic prescriptions from the association of the program with antibiotic prescriptions. No control group was used, because training practices in how to perform data abstraction was a key AS intervention, which would have led to the control group being exposed to the intervention. National antibiotic prescribing data from pharmacies are not visit-based so are not directly comparable with practice-based prescribing data. We were therefore unable to compare directly with control practices. Instead, the data were qualitatively compared with published reports.Sixth, baseline and intervention periods were only 15 months, which precluded our ability to address seasonality beyond September to November data over 2 years. Seventh, we did not collect patient-level or clinician-level antibiotic prescribing data or granular data necessary for analysis by specific diagnosis. We made these decisions to increase the feasibility and simplicity of data collection for the many practices that were not accustomed to data collection, allow for the practices to use the data in their own audit and feedback or peer comparison interventions, and allow practices to participate without individual institutional review board approval . Eighth, although the focus was to investigate a feasible AS intervention in ambulatory practices, the practices that joined the collaborative were those that were already interested enough in AS to have garnered leadership support and identified AS leads. Our intervention may not have been as successful among practices that did not want to implement AS. Ninth, some practices dropped out of the program and others struggled to upload enough data for analysis. However, we successfully retained most practices in this pragmatic, intensive AS intervention without financial incentives or external requirements for participation, and competing demands from COVID-19, practice closures, and practice ownership changes were among the most common reasons for leaving the Safety Program.In this cohort study of US ambulatory care practices, the AHRQ Safety Program successfully launched and retained engagement in AS in a national collaboration of ambulatory care practices, despite challenges posed by the COVID-19 pandemic. There was a significant decrease in antibiotic prescribing per 100 visits and per 100 ARI visits. Addressing behavior and communication and empowering frontline staff to take part in AS was an important consideration when establishing ambulatory AS. The forthcoming AHRQ Safety Program content may have utility in ambulatory practices across the US.


Emerging evidence supports that determination of intrathecal immunoglobulin M (IgM) synthesis (ITMS) and neurofilament light (NfL) concentration in cerebrospinal fluid (CSF) may be clinically useful as disease severity biomarkers in relapsing-remitting multiple sclerosis (RRMS).IF), and by qualitative determination of oligoclonal IgM bands (OCMB). Univariable and multivariable models were performed to predict Evidence of Disease Activity-3 (EDA-3) status within 24 months from onset, and the risk of Expanded Disability Status Score (EDSS) ≥3 and ≥6.Monocentric observational longitudinal cohort study in which prospectively collected data were retrospectively retrieved. Included were patients with RRMS (n=457) who had a diagnostic investigation including analysis of ITMS and CSF neurofilament light (cNfL). ITMS was calculated with the linear index formula, the intrathecal fraction of IgM according to Reiber . Combining IgM-index>0.1 or OCMB with increased cNfL were strong predictors of EDSS≥3  and EDSS≥6 .All investigated methods to calculate ITMS significantly predicted evidence of disease activity (EDA-3) within 24 months. IgMIn a real-world setting, ITMS was a useful biomarker in early RRMS to predict disabling MS and its prognostic value was even stronger in combination with cNfL. Our data suggest that determination of ITMS and cNfL should be included in the diagnostic work-up of RRMS for prognostic purposes and in decisions of disease-modifying therapy. Multiple sclerosis (MS) is an inflammatory and degenerative disease of the central nervous system (CNS). The clinical course of MS is heterogeneous, including patients with benign course with essentially preserved functions to those with highly aggressive course , often cAccumulating data on the role of diagnostic intrathecal IgM synthesis (ITMS) suggest that it may serve as a marker of an aggressive MS course and a recent literature review summarized the majority of these studies and concluded that there is strong evidence that ITMS is a negative prognostic marker . NumerouIF) according to Reiber’s formula  in CSF indicates dissemination in time according to the 2017 revised McDonald criteria , and the formula , 12, 17, formula , 21, as  formula , 22. HowOther cerebrospinal fluid (CSF) biomarkers previously have shown valuable prognostic information. Glial fibrillary acidic protein (GFAP), a biomarker of astrogliosis  has beenThe aim of this study was to validate ITMS in a real world setting as a predictor of a more aggressive disease course as determined by rapid disability progression –31 and thttp://www.msreg.net)  patients who underwent a routine diagnostic MS investigation between 2001-2018 at the MS centre, Sahlgrenska University Hospital, Gothenburg, Sweden Figure 1reg.net) . Data abreg.net)  and exporeg.net) . Only paThree study endpoints were used for the purpose of our analyses. The first endpoint was evidence of disease activity 3  within 6 months (6-CDW), and new T1 gadolinium-enhanced lesions/new/newly enlarging T2-weighted [T2W] lesions) , 35 withAlb) was calculated as CSF albumin (mg/L)/serum albumin (g/L) and the IgM-index was calculated as [(CSF IgM (mg/L)/serum IgM (g/L)) ÷ QAlb]. IgM-index>0.1 was considered elevated (IF) was calculated according to the formula by Reiber  diluted in PBS with Tween®20 0,05% overnight. After four washes with PBS the blot was developed with SigmaFAST™ BCIP®/NBT (Sigma cat# B5655), the developed membrane was scanned while wet and then dried. Both the scanned image and the dry membrane were evaluated, and CSF-specific OCMB were defined as ≥2 IgM bands present in CSF which were not present in the matching serum sample. On each gel a CSF sample with known OCMB was included as quality control.Matched CSF and serum samples were obtained during routine diagnostic work-up and analysed consecutively. Levels of albumin and IgM in serum and CSF were measured on a cobas c module instrument (Roche) with the ALBT2, IGM-2 and IGM-C reagent cassettes. The albumin quotient . The IgG-index was calculated as [(CSF IgG (mg/L)/serum IgG (g/L)) ÷ QBoard-certified laboratory technicians, who were blinded to the clinical status, using strict procedures for quality control and run-approval, performed the analyses.® ELISA kit; UmanDiagnostics AB, Umea, Sweden; catalogue # 10-7001 CE) and age-adjusted upper limits of the reference range utilized in clinical practice were used in order to determine whether cNfL levels were elevated or normal, as previously described  method .Descriptive statistics are presented as median and interquartile ratio (IQR) or range (minimum-maximum). Categorical variables are described using relative and absolute frequencies. The measurement of agreement between the different methods to evaluate ITMS was determined with the kappa statistic.IF or OCMB and cNfL, and performed cox proportional hazards regression with the same endpoints and adjustments as above. Due to the exploratory nature of the study, no adjustments for multiple comparisons were made.To investigate the ability of ITMS to predict a worse disease course, the following endpoints were assessed: the association of ITMS with EDA-3, and the disability progression endpoints EDSS≥3 and EDSS≥6. For these endpoints, cox proportional hazards regression models were performed and the adjusted hazard ratios (aHR) along with corresponding 95% confidence intervals (CI) were calculated, as well as univariable Kaplan-Meier survival analyses with corresponding logrank tests. Kaplan-Meier curves are presented to visualize the results. For the EDA-3 endpoint, we adjusted for the following potential confounding factors: age at debut, sex, disease duration at diagnosis, T2W lesion burden at the time of diagnosis, as well as exposure to DMTs (first-line/second-line). In the case of EDA-3, the total follow-up time was 24 months. Patients who experienced EDA-3 during the follow-up period were censored at the time of the first signs of EDA-3. Those who fulfilled NEDA-3 at the end of the 24-month follow-up period were censored at 24 months. In the analyses of disability progression endpoints, patients were censored either at the time of reaching the investigated milestone or at the time of the last visit in case EDSS was kept <3 and <6 respectively. For the disability milestones, we adjusted for age at the time of diagnosis, sex, disease duration, baseline MRI T2W lesion burden, exposure to DMTs, and whether subjects escalated therapy during the follow-up. Time to EDA-3, EDSS≥3, and EDSS≥6 was analysed with Kaplan-Meier survival analysis and the logrank test. In order to investigate whether ITMS and cNfL have an additive predictive value, we then computed a new variable that combined positivity for IgM-index, IgMThe same analyses utilising cox proportional hazards models were performed independently and separately for IgG-index and cNfL. All statistical analyses were performed with IBM SPSS version 28.0.1.0 . Figures were created in GraphPad prism version 9.1.0.IF>0%, and 182 (96.7%) patients with OCMB had also OCGB. Demographic and clinical characteristics are presented in The study population included 457 RRMS patients, of which 316 (69%) were female, with a median (IQR) age at clinical onset of 35 (28–44). Males had significantly higher  IgM-index values  compared with females . Age did not influence ITMS. Median (IQR) follow-up time from lumbar puncture (LP) until the last follow-up visit was 7 years (4–12). CSF specific IgG-OCBs were present in 430 (94.1%) patients, in which 289 (97.3%) patients with IgM-index>0.1, 148 (98%) patients with IgMIF>0%. The measurement of agreement between IgM-index>0.1 and IgMIF>0% according to the kappa statistic was ϰ=0.42 (p<0.001). Increasing the cut-off-value of IgM-index to ≥0.18 increased the measurement of agreement substantially to ϰ=0.87 (p<0.001). Likewise, 188 patients (44.7%) had OCMB, while 36 patients (7.9%) were missing a measurement of OCMB. The measurement of agreement between IgM-index and IgMIF≥0% with OCMB was fair .In 297 patients (65%), elevated IgM-index>0.1 was noted, whereas 151 (33%) patients had IgMIF were moderately associated with a higher EDA-3 hazard , 14 (11.3-16.7), and 18 (16.8-19) months respectively. Patients with IgG-index>0.7 had an aHR of 1.6 . Age was the only other covariate with a significant predictive effect in which rising age was slightly associated with a protective effect .In our cohort, 178 patients (38.9%) did not fulfill NEDA-3 within 24 months. Both IgM-index and IgMEDSS≥3 was reached in 136 patients (29.8%) whereas 37 patients (8.1%) reached EDSS≥6 within a mean (range) follow-up period of 8.3 years (2-17). In a multivariable analysis, only patients with IgM-index>0.1 had a statistically significant higher risk to reach EDSS≥3  as well as prediction of EDSS≥3  and EDSS≥6 .(i)-), 83 patients (18.2%) had elevated cNfL but IgM-index ≤ 0.1 (cNfL+/IgM(i)-), 174 patients (23.9%) had normal cNfL levels and IgM-index>0.1 (cNfL-/IgM(i)+), and 108 patients (38.1%) had elevated levels of both cNfL and IgM-index (cNfL+/IgM(i)+). As expected, in a multivariate analysis, patients with cNfL-/IgM(i)+ exhibited higher risk for EDA-3 + showed an even higher risk for experiencing disease activity according to NEDA-3  were negative for both cNfL and IgM-index , ganglion cell-inner plexiform layer (GCIPL), and higher rates of EDSS≥3 and EDSS≥4 . FurtherIF, OCMB, and LS-OCMB and investigated their ability to predict a second relapse in patients with CIS as well as achieving the disability milestones EDSS≥4, EDSS≥6, and conversion to SPMS . Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.IR and JL contributed to conception and design of the study. IR organized the database, performed the statistical analysis, and wrote the first draft of the manuscript. SR wrote sections of the manuscript. IR, MA, LN, KB, HZ, and JL had major role in data acquisition. All authors contributed to manuscript revision, read, and approved the submitted version.The study was supported by grants from the Swedish State Support for Clinical Research , Regional FoU grant Västra Götalandsregionen (260 101), NEURO Sweden, NEURO Gothenburg, Edith Jacobson’s Foundation and Helena Ahlin’s Foundation.KB is supported by the Swedish Research Council (#2017-00915), the Alzheimer Drug Discovery Foundation (ADDF), USA (#RDAPB-201809-2016615), the Swedish Alzheimer Foundation , Hjärnfonden, Sweden (#FO2017-0243 and #ALZ2022-0006), the Swedish state under the agreement between the Swedish government and the County Councils, the ALF-agreement (#ALFGBG-715986 and #ALFGBG-965240), the European Union Joint Program for Neurodegenerative Disorders (JPND2019-466-236), the National Institute of Health (NIH), USA, (grant #1R01AG068398-01), and the Alzheimer’s Association 2021 Zenith Award (ZEN-21-848495). HZ is a Wallenberg Scholar supported by grants from the Swedish Research Council (#2018-02532), the European Research Council (#681712 and #101053962), Swedish State Support for Clinical Research (#ALFGBG-71320), the Alzheimer Drug Discovery Foundation (ADDF), USA (#201809-2016862), the AD Strategic Fund and the Alzheimer’s Association , the Olav Thon Foundation, the Erling-Persson Family Foundation, Stiftelsen för Gamla Tjänarinnor, Hjärnfonden, Sweden (#FO2019-0228), the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 860197 (MIRIADE), the European Union Joint Programme – Neurodegenerative Disease Research (JPND2021-00694), and the UK Dementia Research Institute at UCL (UKDRI-1003).IR has received compensation for lectures from biogen. MA has received compensation for lectures and/or advisory boards from Biogen, Genzyme, and Novartis. LN has received honoraria for lecture from Biogen, Novartis and Teva, and for advisory boards from Merck. KB has served as a consultant, at advisory boards, or at data monitoring committees for Abcam, Axon, BioArctic, Biogen, JOMDD/Shimadzu. Julius Clinical, Lilly, MagQu, Novartis, Ono Pharma, Roche Diagnostics, and Siemens Healthineers, and is a co-founder of Brain Biomarker Solutions in Gothenburg AB (BBS), which is a part of the GU Ventures Incubator Program, all unrelated to the work presented in this paper. HZ has served at scientific advisory boards and/or as a consultant for Abbvie, Alector, Annexon, Artery Therapeutics, AZTherapies, CogRx, Denali, Eisai, Nervgen, Novo Nordisk, Pinteon Therapeutics, Red Abbey Labs, Passage Bio, Roche, Samumed, Siemens Healthineers, Triplet Therapeutics, and Wave, has given lectures in symposia sponsored by Cellectricon, Fujirebio, Alzecure, Biogen, and Roche, and is a co-founder of Brain Biomarker Solutions in Gothenburg AB (BBS), which is a part of the GU Ventures Incubator Program (outside submitted work). JL has received travel support and/or lecture honoraria and has served on scientific advisory boards for Biogen, Novartis, and Sanofi Genzyme; and has received unconditional research grants from Biogen and Novartis.The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Embedding research and evaluation into organizations is one way to generate “practice‐based” evidence needed to accelerate implementation of evidence‐based innovations within learning health systems. Organizations and researchers/evaluators vary greatly in how they structure and operationalize these collaborations. One key aspect is the degree of embeddedness: from low embeddedness where researchers/evaluators are located outside organizations  to high embeddedness where researchers/evaluators are employed by organizations and thus more deeply involved in program evolution and operations. Pros and cons related to the degree of embeddedness (low vs high) must be balanced when developing these relationships. We reflect on this process within the context of an embedded, mixed‐methods evaluation of the Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program. Considerations that must be balanced include: (a) low vs high alignment of goals; (b) low vs high involvement in strategic planning; (c) observing what is happening vs being integrally involved with programmatic activities; (d) reporting findings at the project's end vs providing iterative findings and recommendations that contribute to program evolution; and (e) adhering to predetermined aims vs adapting aims in response to evolving partner needs. DoE has progressed to become an integral part of the broader Innovation Ecosystem in the VHA. The objectives of the DoE program are to: (a) identify promising clinical and administrative practices successfully developed and tested by individuals at the frontline of care; (b) pilot these promising practices in new facilities and provide implementation facilitation support for implementing teams; and (c) support the scale‐up and spread of successful practices across the health system.DoE is a nationwide program that employs implementation strategies for promoting adoption, implementation, and spread of promising practices across VHA. Frontline staff members apply for the chance to pitch their practices during a national “Shark Tank” competition; VHA facility directors serve as “Sharks” who place bids. The winning practices are designated DoE Promising Practices. Winning Shark bidders receive 6 to 9 months of external implementation consultation at their facility. DoE then provides national support for diffusing successful practices across the VHA in cooperation with national program offices. Diffusion support may take one of three different paths, including: (a) direct facilitation by DoE staff for a small number of DoE National Diffusion Practices; (b) initial assistance with diffusion and training of practice developers through a VHA Diffusion Academy for a second set of DoE Promising Practices; or (c) promotion of organic diffusion across the VHA through packaged information for the final group of DoE Promising Practices.The DoE program has also developed an online Diffusion Marketplace that provides information about these practices so that facilities seeking to address specific challenges can search the Diffusion Marketplace for potential solutions that have shown positive impact in VHA. The Diffusion Marketplace also helps to track diffusion and implementation of practices across the VHA.DoE has had broad reach in the VHA. Over the course of six Shark Tanks, there have been 2671 applications from across the VHA, which has led to the designation of 69 DoE Promising Practices. As of the end of fiscal year 2020, DoE estimates that eight National Diffusion Practices have been implemented at numerous facilities across the VHA and impacted an estimated 570 000 veterans in fiscal years 2019 and 2020. Example DoE Promising Practices include (a) enhancing oral hygiene of inpatients with the goal of preventing non‐venerator associated pneumonia3At an early stage of program development, DoE leaders recognized the benefits of rigorous evaluation to describe and measure the program's reach, effectiveness, adoption, implementation, and maintenance .Per regulations outlined in VHA Program Guide 1200.21, the SHAARK evaluation has been designated a non‐research quality improvement activity. Although QUERI funds teams of research investigators and staff across VHA facilities nationally, these projects are funded using clinical dollars, not as congressionally appropriated research. QUERI funded the initial SHAARK evaluation as well as select “DoE Promising Practices” identified through the Diffusion of Excellence Shark Tank process and by VA national leaders as top priority for national diffusion. Funding for later phases of these evaluations has been shared between QUERI and DoE through funding from the VHA Office of Rural Health (ORH) as well as other national VHA program offices, which provided an indication of the perceived operational value of the evaluation. For this analysis, we report on the experience involving the global evaluation of the DoE program through the national SHAARK evaluation.Unlike one‐time evaluations of programs , the SHAARK team built a long‐term relationship with DoE leaders. Transitions in core DoE leadership and staff have occurred several times over the past years, thus the longevity of the evaluation team's relationship has been particularly valuable. SHAARK team members have worked alongside successive DoE leaders to develop and refine key questions, establish rationale for methods, interpret findings, and iteratively integrate these findings into strategic planning and programing activities. To complete a meaningful evaluation, research team members have balanced responsiveness within a valuable collaborative relationship with the responsibility of maintaining objectivity.The initial goals of the SHAARK evaluation were developed with original leaders of the DoE program, whose focus was on rapidly standing up a high‐visibility program (the Shark Tank), supporting participants, and institutionalizing the program. Aims focused on explaining program participation, motivations, and processes used for bidding in the Shark Tank, and implementation effectiveness in sites receiving facilitated implementation support as a result of having a “winning” Shark Tank bid.As described above, DoE has expanded its facilitation role over the past 5 years toward promoting spread of successful innovations across the VHA through a combination of direct staff support, training, and active tracking spread across the healthcare system. As a result, SHAARK has also pivoted from a focus primarily on understanding the Shark Tank and early facilitated replication of innovation toward understanding how to help DoE support innovations once they have completed the initial replication stage. The goal is to consider how to help the VHA as a whole take maximum advantage of DoE Promising Practices as it seeks to serve veterans across the country.4Below, we discuss considerations that must be balanced in order to maximize the benefits of embedded research or evaluation. Figure 4.1Balancing the alignment of goals required a clear understanding of the goals of the DoE program, without losing sight of the aims of the evaluation itself. Evaluation team members worked to track changing program priorities, to stay connected through leadership changes, and to navigate changing roles and responsibilities assigned to contracted consultants who provide day‐to‐day support for DoE activities. Even as we worked to stay abreast of DoE goals, the evaluation team members maintained a distinct identity, seeking to contribute expertise in implementation science where applicable. Evaluation team members supported close collaboration between DoE and the national QUERI office and served as faculty for DoE events that highlighted the links between implementation science and quality improvement.4.2SHAARK evaluation findings have informed strategic planning, influencing the evolution of DoE. Early in the evaluation timeline, the SHAARK team found that key DoE participants including Sharks , practice developers, and pilot implementation teams lacked a clear understanding of their roles and expectations for others within DoE. In response, DoE leaders, in collaboration with the SHAARK evaluation team, drafted descriptions of roles and responsibilities that were shared with participants and participation agreements with sites receiving facilitated replication of DoE Promising Practices, leading to programmatically important indication of greater understanding of roles in the DoE process.4.3The SHAARK team has had to navigate the appropriate level of involvement with programmatic aspects of DoE. Through structured observations and semi‐structured interviews, evaluators identified challenges experienced by some local implementation teams that could be traced to poor fit of the practice with local conditions.4.4The evaluation team's focus on building strong partnerships with stakeholders such as DoE leadership, VHA program offices, VHA facility leaders, and participants in DoE programs has prompted the delivery of frequent reports, recommendations, and discussions. As illustrated above, the provision of iterative findings has influenced DoE's evolution far more than would have been expected from an evaluation that provided a single report after several years' study.4.5There has been continued negotiation as to whether and how evaluation aims should evolve. The partnered nature of the work inevitably leads to changes. To account for these changes, the SHAARK team has renegotiated project scope within available resources with the goal of more fully addressing both the changing nature of the DoE structure and needs of new leadership for specific types of information. Allowing aims to evolve over time has been an effective way to accommodate the needs of the DoE partners, but also poses challenges for publishing evaluation findings within traditional scientific structures . The non‐research/quality improvement designation of this evaluation enabled a formative approach where subsequent aims were added based on findings from earlier aims and feedback to partners.5The SHAARK evaluation team has worked to clarify roles and expectations between the embedded evaluation team and DoE partners while recognizing that we have different perspectives and varying incentives. This is challenging work that takes frequent communication and negotiation. The process has included: frequently meeting formally and informally in‐person (including multiple day‐long planning meetings) to build mutual understanding and trust; evaluators shadowing in‐person and virtual DoE events; regular (bi‐weekly) conference calls with program leaders and contracted consultants; open discussion of both programmatic and evaluation activities; and rapidly addressing any miscommunication or need for clarification.In many ways, this is the reality of any lasting relationship of any kind where the parties are not required to participate. SHAARK has had an initial grant and two subsequent extensions. Ultimately the reason we continue to work together is that there is an opportunity to help the VA enhance innovations that can positively impact veterans' lives while enjoying the process of working together. Through this process, we have summarized a number of lessons that have allowed for the calibration of the degree of embeddedness. These lessons, which are described in more detail in Table 6To be effective, embedded research and evaluation efforts rely on strategic alliances between organizational partners.The potential rewards of these interorganizational strategic alliances are considerable. Researchers extend their networks to ensure relevance and applicability of their work within a learning healthcare system; operational leaders benefit from researcher's methodologic expertise in addressing operational challenges.Overall, the embedded research/evaluation approach that we describe here has helped to inform VHA's novel DoE program in ways that benefit VHA and that can be shared with outside systems as well. The strong partnership between the SHAARK evaluation team and DoE leaders has relied in equal parts on dedicated efforts by both parties. Together, evaluators with a research background and operational leaders have achieved this partnership through continuous dialogue, frequent re‐examination of the evaluator's level of objectivity, and careful attention to the balancing that characterizes embedded research. These principles are key to the type of partnered research required of both implementation science and achieving learning health system goals of utilizing research and operational expertise to develop innovative strategies to enhance the health and health care of the people they serve.Authors have no conflicts to declare. All authors are employees of the United States Department of Veterans Affairs. No authors received compensation for preparation of this manuscript outside of their salaries.


This economic evaluation investigates the association between cuts to the Patient Protection and Affordable Care Act (ACA) navigator program and private sector television advertising in the health insurance marketplace. What is the association between the 80% cut in funding for the Patient Protection and Affordable Care Act (ACA) navigator program between 2017 and 2019 and private sector advertising in the ACA individual health insurance marketplace?This economic evaluation of 2435 counties in 33 US states using a difference-in-difference analysis found no significant change in the number of private sector advertisements aired targeting marketplace health insurance or other non-Medicare, non-Medicaid health insurance associated with the funding cuts.These findings suggest cuts to funding for the ACA navigator program were not associated with changes in the number of advertisements aired by health insurance companies or other private sector sponsors; the findings can inform policy debates about the extent to which the private sector adjusts in response to changes in government outreach. Many individuals eligible for coverage in the Patient Protection and Affordable Care Act (ACA) marketplace remain unenrolled because of information barriers. Whether the private sector or the public sector should conduct outreach to address these barriers is a topic of active debate.To determine whether cuts to the funding of the ACA navigator program were associated with changes in the volume of private sector advertising.Using data from the 2015 to 2019 open enrollment periods, this economic evaluation analyzed the changes in advertising associated with 2017 to 2019 cuts to navigator program funding. A difference-in-difference analysis was used to compare outcomes before and after the cuts in counties with higher and lower exposure to the navigator program. Health insurance advertising was measured using data from Kantar/Campaign Media Analysis Group in collaboration with the Wesleyan Media Project, the most comprehensive data available on local broadcast and national cable advertising. The data set included all counties that met the eligibility criteria for the navigator program from 2015 through 2019. Data were analyzed from August 2021 to May 2022.Counties were classified as having higher or lower exposure to the navigator program according to the intensity of program activity in 2016, before the funding cuts. Counties served only by statewide navigator programs were categorized as lower exposure, while those also served by local navigator programs were categorized as higher exposure.Number of privately sponsored television advertisement airings for the ACA individual health insurance marketplace during the 2015 to 2019 open enrollment periods in each county, adjusted for population.All counties in 33 states that met the eligibility criteria for the navigator program from 2015 through 2019 were included in the analysis (2435 counties). Cuts to the navigator program were not associated with changes in the number of privately sponsored health insurance advertisements aired. Results were similar under several alternative approaches including an event study specification.In this study of the association between television advertising and navigator funding in the ACA marketplaces, private sector entities did not increase their advertising to compensate for declines in government-sponsored navigator activity. This finding can inform policy debates about the extent to which the private sector adjusts in response to changes in government outreach, and thus improve the design of state waivers and federal funding allocations. The federally facilitated health insurance marketplace, known as HealthCare.gov, is a public-private partnership that provides a clearinghouse exchange where private insurers offer regulated health insurance plans to individuals. However, barriers to marketplace coverage persist; in 2021, an estimated 12 million people remained uninsured despite being eligible for marketplace coverage, including 6 million who could have received free coverage.3 Media coverage of the ACA rarely referenced the available financial assistance.4 In 2018, there remained an estimated 30 million uninsured people, and two-thirds of uninsured adults had not visited the marketplace to see their coverage options.5 Furthermore, many uninsured adults are not familiar with key health insurance terms, such as deductible and premium.8 These gaps in health insurance literacy are important because plans in the ACA marketplace vary widely in terms of benefits, networks, and premiums.10 Complex choices and the large number of available options increase the difficulty of selecting a marketplace plan.15A lack of awareness about relevant options may be a barrier to gaining coverage. In 2014, nearly two-thirds of uninsured adults reported being exposed to little or no information about financial assistance for marketplace coverage.17 to direct enrollment assistance.18 Since 2014, each state was required to establish assister programs providing free, impartial, one-on-one advice and enrollment assistance.19 During the first 3 years these programs were available, more than 28 000 assisters aided more than 20 million potential enrollees.22 In states that used the federally facilitated marketplace, these required public sector assisters were funded via navigator grants from the federal government.23To address these information barriers and increase marketplace coverage, public and private sector entities have conducted outreach ranging from advertising26 After the inauguration, the Trump administration canceled the television and radio advertising scheduled for the final week of the 2017 open enrollment period.27 Just before the beginning of the 2018 open enrollment period, the administration eliminated all federal funding for television advertising for the marketplace and reduced marketing funding by 90%.28 In addition, the Centers for Medicare & Medicaid Services (CMS) unexpectedly cut navigator grants by approximately 40% in 2017.29 The following year, CMS decreased navigator program funding even further, resulting in an approximately 80% decline between 2016 and 2018; this reduced funding level remained consistent for the next 2 years.32 Cross-county comparisons suggest that these cuts in funding for local navigator programs substantially reduced coverage among many groups, including individuals with low incomes, those younger than 45 years, those with low English proficiency, and individuals who identify as Hispanic.25During the Trump administration, public sector outreach efforts changed in several ways relevant to marketplace enrollment.33 Navigators and advertising both plausibly increase the awareness and potential engagement of individuals, while navigators can also aid in navigating the process of choice and enrollment. Thus, for insurers seeking to raise profits by enrolling more people in marketplace coverage, advertising may be a substitute for public funding of the navigator program, albeit an imperfect one. Despite evidence of the association between outreach and health insurance enrollment, including direct enrollment assistance via navigators36 and television advertising,40 little research has explored the interplay between public and private sector sources of outreach. To assess the repercussions of recent developments and inform policy decisions, this study examined the association between the Trump-era cuts in navigator program funding and the volume of private sector advertising.Identifying the association between navigator funding and private sector advertising is important for informing future marketplace design, navigator assistance funding, and public sector advertising allocations. In a precaution adoption process model for insurance enrollment, individuals proceed from being unaware, to aware but unengaged, to deciding to enroll or not enroll in coverage, to acting on their decision about enrollment.STROBE) reporting guideline.41 We hypothesized that cuts in navigator funding would lead to an increase in private sector health insurance advertising as a compensatory response. We leveraged the approximately 80% reduction in funding for navigator programs between 2017 and 2019 under the Trump administration as a natural experiment  according to the service areas of navigator grantees in 2016. Counties not served by any navigator program or served only by a statewide program  were classified as lower exposure, while counties served by both statewide and nonstatewide  navigator programs  were classified as higher exposure.Following Myerson et al,We used difference-in-differences models to estimate the association between navigator program funding cuts and insurers’ advertising decisions. The first difference is the change in advertising volume before and after the funding cuts. The second difference is the relative change in advertising volume in lower exposure and higher exposure counties. This design allowed us to control for concurrent state-level and national-level policy changes that may influence the results. Advertising specifically targeted to Medicare, a health insurance market not directly affected by cuts to the navigator program, was included as a placebo test. For further confidence in the robustness of our approach, we performed several event studies with varying specifications.16 In addition, advertisements were categorized according to the open enrollment period in which they aired. Open enrollment periods begin in the autumn before the plan year, which starts on January 1, and end either in the December before the plan year or at some point early in the plan year.42 The advertising data from each open enrollment period were crosswalked from the media market level to the county level by assigning each county to a single media market in which it had the largest share of its population because all areas within a given media market are exposed to the same television advertising; this method assigns all counties in the market to the same advertising volume. We excluded advertisements that could not be categorized (0.4%). Interrater reliability was high for Medicare coding (κ = 0.85) and Medicaid coding (κ = 0.7).The primary outcome was the number of privately sponsored non-Medicare and non-Medicaid health insurance television advertisements airings during the 2015 to 2019 open enrollment periods in each county. Kantar/Campaign Media Analysis Group supplied the Wesleyan Media Project with the airing-level advertising data, including information on the media market in which each advertisement aired as well as the date, time, and sponsor of the advertisement. We used a method described in previous work to categorize advertisements by whether they were privately sponsored and by product category .43 Detailed records are available because CMS funds the navigator program via grants to specific organizations, rather than grants to states. Additional details on the classification method are available in prior work.25The main variable was the county’s exposure to the navigator program in 2016. To classify counties as higher exposure or lower exposure, we used publicly available CMS records that detail which counties were served by each navigator organization.The analytical data set included counties in the 33 states that met the eligibility criteria for the navigator program throughout the study period of 2015 to 2019 . As a result of this exclusion criterion, all states in the analytical sample were observed for the same number of periods both before and after funding cuts.44 Regressions controlled for key determinants of advertising strategy that might vary within counties over time, including the current population younger than 65 years, the benchmark premium, the spread between the benchmark premium and the least expensive silver plan for a single 40-year old, and the number of carriers participating in the marketplace (because competition and prices may be associated with purchasing decisions).48To identify the association between the cuts to the navigator program and advertising outcomes, we used county-level differences in prior navigator funding within each state as the source of identification. The first cuts to the navigator program were announced and implemented in the summer of calendar year 2017, just before the open enrollment period for 2018. Accordingly, the 2018 open enrollment period was the start of the postcut period in the primary analysis. All regressions included state-by–open enrollment period and county indicator variables to control for time-invariant differences across counties, other state-level and national-level policy changes, and secular trends unrelated to the funding cuts.30 In all analyses, 2-sided t tests were used, and a significance level of P < .05 was used to determine statistical significance. The eAppendix in the Heteroskedasticity-robust SEs were clustered by state and treatment group to account for correlation of the error terms according to exposure to the navigator program.We conducted several additional sensitivity analyses to assess the validity of the methodological approach and results. The research design assumes that outcomes would have been similar in higher-exposure and lower-exposure counties if there had been no reduction in navigator funding. We conducted event study models to test for violations of this parallel trends assumption.49 Third, we conducted additional analyses that did not include the number of insurance carriers as a control variable because recent evidence suggests that insurers are slow to modify advertising plans according to changes in competition levels.37 Finally, we assessed whether a single state, including Iowa, which adopted alternate Farm Bureau insurance plans during our study period, might be driving the results by systematically dropping states 1 at a time and reestimating the models. The eAppendix in the We assessed the sensitivity of the results to several alternate specifications. First, we assessed the sensitivity of the results to the timing of the posttreatment period using event study models that allowed open enrollment 2018 to be a transition period and open enrollment 2019 to be the postcut period. The rationale of this analysis was that some advertisers were notified about the navigator funding cuts too late to alter their advertising strategy for the 2018 open enrollment period. Second, we estimated alternate models that used 2016, rather than 2015, as the first year of the analysis, because 2016 was the first year in which insurers could rely on more than a year of individual marketplace experience to inform rates and strategy.The final analytical sample contained 2435 counties. 51 has linked insurance-related advertising to increases in enrollees’ perceptions of being informed about the ACA as well as increases in shopping for and enrollment in coverage. Navigator programs, which provide outreach and one-on-one assistance, have been linked with coverage gains among marginalized groups.52 Despite the importance of these 2 forms of outreach for enrollment outcomes, few prior studies have examined how public and private sector actors conducting outreach react to one another’s efforts.Prior researchThis economic evaluation tested whether private sector advertising volumes changed in response to large cuts in the navigator program, 1 of the key avenues for public sector outreach in the ACA marketplace. We were unable to detect any change in privately sponsored advertising volumes associated with the nearly 80% decrease in navigator funding implemented by the Trump administration. Furthermore, the point estimates of the change in the primary outcome of interest were small . Event studies of prior trends supported the validity of the analysis, and supplemental analyses suggested the findings were robust to alternate specifications. In sum, the data do not support the hypothesis that private sector actors increased advertising to compensate for declines in navigator activity.53 Instead, private sector entities such as brokers, agents, and other enhanced direct enrollment entities would determine eligibility, estimate subsidies, and effectuate enrollment.54 As part of this change, the state would end the public sector navigator program in Georgia, under the assumption that private sector activity would compensate for its absence.This evidence of a lack of responsiveness of television advertising to navigator program cuts can inform the administration of outreach in the marketplace. For example, Georgia applied for a Section 1332 waiver that would eliminate the use of the federal HealthCare.gov website for enrollment.56 For example, prior studies40 of health insurance advertising suggest private sector outreach is targeted to manage risk and maximize profit. In contrast, government-funded navigator programs target difficult-to-reach, low-income, or marginalized individuals,56 populations that may or may not be profitable to insurers. Indeed, 1 study57 found that individuals who earned between 100% and 138% of the Federal Poverty Level had the highest expenditures controlling for plan actuarial value and risk adjustment score, suggesting low-income enrollees may be less profitable to enroll than higher income enrollees.Our findings are consistent with prior evidence suggesting that private sector actors target outreach to individuals who are likely to be profitable, whereas public sector actors often use other criteria to prioritize outreach activities.58 Third, although we have the most comprehensive data available on local broadcast and national cable television advertising, the data do not include advertising on local cable channels, and we were unable to observe or quantify nontelevision advertising. Fourth, the measure of advertising volume includes ads for nonmarketplace plans and non–ACA-compliant plans , as we were unable to identify these separately from advertisements for marketplace plans. Fifth, our findings may not represent the experience of states without county-level variation in the presence of local navigator programs . Sixth, although the point estimates of the change in the primary outcome of interest were small, the width of the 95% CIs imply we could not rule out changes in private sector advertising of up to 7% of the baseline level. Additionally, we were unable to assess the timing between the notification of initial navigator funding cuts and commitment deadlines for television advertising purchases.This study has limitations. First, potential confounding from unobserved, concurrent changes that varied by county may have impacted the results. Second, the estimates are specific to the group of states in the sample  and may not generalize to other states. States with state-based marketplaces fund and operate their own advertising and enrollment assistance programs.In this observational economic evaluation study of the association between television advertising and navigator funding in the ACA marketplaces, we found no association between cuts in navigator funding and changes in private sector television advertising. Advertising was not a substitute for a change in federally funded enrollment assistance. Information pathways for individuals who sought insurance are complex and channels such as privately sponsored advertising and federally funded navigator programs may not have interacted with each other. This lack of interaction can inform the design of state waivers and federal funding allocations for enrollment and outreach efforts.


Dual-task training is an emerging field used for people with Parkinson disease (PD) to improve their physical and cognitive well-being, but the patients’ acceptability, safety, and adherence to such training in online settings are unknown.This study aims to evaluate the acceptability of a dual-task cognitive and motor online training program for people with PD as a group online community program.People with PD were invited to participate in an online program (PD3 Move) consisting of physical and vocal exercises in response to different cognitive challenges displayed as dynamic backgrounds on Zoom. The program ran twice per week for 16 weeks. Patient acceptability was assessed at 4 months by monitoring attendance rates and feedback from an exit questionnaire emailed to all participants assessing satisfaction, perceived benefit, safety, and willingness to continue and recommend to others.The online program was delivered to 15 participants  with a diagnosis of PD, a mean age of 69.4 (SD 9.3) years, and Hoehn and Yahr (H&Y) stages I-IV. The attendance rate was high, with participants coming to more than 13 (81%) of the sessions. Participants were very satisfied  or satisfied  with the program. Participants reported that what they most liked were the new cognitive physical challenges. The 3 main facilitators to participating were perceiving the benefits, instructor’s flexibility and engagement, and the social interaction moments with others. The 3 main difficulties were dealing with motor fluctuations , difficulties in using technology , and difficulty hearing instructions due to hearing loss . Patients had favorable perceived benefits of the program, with 14 (93%) considering it very useful for the current management of health and 1 (7%) moderately useful. No adverse events were reported, and all participants said that they were willing to continue the program and recommend it to others.Our findings suggest that the online cognitive and motor program was well received, safe, and perceived to be of benefit to this group of medically stable people with PD in H&Y stages I-IV. Access to specialized care and enhancement of long-term adherence to regular exercise can be achieved with online community group programs. Parkinson disease (PD) is considered 1 of the fastest-growing neurological disorders in the world ,3. ImpaiThere is growing evidence suggesting that nonpharmacological interventions, such as exercise/physiotherapy ,9,10 andInitial research in this area of dual-task training concluded that 30 minutes once a week for 3 weeks of multiple-task gait training is feasible in 5 people with PD  , with suGiven the recent need for online solutions associated with the coronavirus pandemic and with the advances in technology facilitating access to specialized care -25, onliHere, we aim to assess the acceptability and safety of delivering such dual-task programs in an online group format with people with PD in the early to late stages of PD.Program acceptability was evaluated with mixed methods quantitative and qualitative assessments . To ensuThe sampling method selection was nonprobabilistic by convenience. Recruitment took place for 3 months from October 2021. All people with PD registered at the Portuguese Parkinson Disease Patient Association were invited to join the program online. People were included if they had (1) a diagnosis of PD (self-reported by patients), (2) H&Y stages I-IV, (3) the ability to connect online via Zoom  safely or have a care partner to assist if needed, (4) the ability to communicate with the investigator to understand and comply with the study procedures, and (5) the willingness and ability to provide written informed consent to participate and understand the right to withdraw their consent at any time without prejudice toward future medical care.Participants were excluded if they had self-reported severe cognitive difficulties and significant active psychiatric problems that would incapacitate them from participating.A therapist from the patient association clinic carried out the selection process and included participants that met the study criteria via email. The PD3 Move program was provided as an online community exercise program offered by the patient association.The program was led by a physiotherapist with expertise in PD and cognitive training, as well as with experience in building and implementing community dual- and multitask programs in PD. The program was delivered in 2 (1-hour) group sessions per week for 16 weeks. The program consisted of combining cognitive exercises projected on a dynamic Zoom background, with participants responding with physical and vocal exercises. The use of different types of Zoom backgrounds to support the cognitive challenge, as well as the use of voice to respond, was expected to be associated with higher levels of engagement, learning benefits, and exercise adherence.Dual-task training was defined as the capacity to simultaneously perform 2 or more tasks during transfers, ambulation, and other movement-related activities . The perThe cognitive exercises selected targeted the 4 main cognitive domains  increasing the physical or vocal challenge or (2) manipulation of features on the cognitive exercises . Gamification principles were also adapted to some exercises to act as motivational drivers for the participants and enhance social interaction . An examA pretest was first applied to a group of 5 individuals with a diagnosis of PD. Researchers questioned these individuals to understand their perceptions about the online questionnaires. The instrument was considered sufficiently clear, objective, and comprehensive and did not present questions that could be ambiguous or equivocal. This procedure allowed researchers to determine that the survey was suitable for this study.Before attending the program, all participants completed a questionnaire collecting general information about demographics, clinical problems, past medical conditions, current exercise habits, and perceived facilitators/barriers to exercise. Importantly, when patients reported falls in the assessment questionnaire, they were asked to take safety precautions, namely have a care partner present, always exercise in the sitting position, and signal to the therapist to monitor ongoing risk.A postassessment anonymous online questionnaire was sent to participants immediately after the program. This questionnaire assessed patients' overall satisfaction, preferences, and barriers and facilitators to participating and any type of problem or adverse events that occurred during the program.The overall satisfaction with the online program was assessed using a 5-point Likert scale . Patients were also asked whether they would recommend the program to a friend  and how likely they are to return to a similar program .To assess the program design, patients were asked which part of the session they preferred: 1=warm-up with simple amplitude-based movements; 2=physical exercises on their own ; 3=cognitive exercises with movements ; 4=social interaction moments before, during, and after the session; 5=explanation of the teacher to Parkinson-related questions or reasons for the specific exercise; 6=final relaxation with soft music and breathing exercises; 7=preferred all the parts; and 8=did not prefer any parts.The instructor assessed patient presence and feedback at the end of each session and prepared a monthly report regarding difficulties or learning points from the application of the program. In addition, the therapist made follow-up phone calls, when needed, to assess people’s feedback, concerns, or any questions. To ensure that the data collected by the instructor remained pertinent to the aims of the study, an interview guide was developed, comprising a set of open-ended questions focused on the difficulties in and facilitators to participating in PD3 Move. Examples of questions used in the guide are:Are there any factors regarding technology or other issues limiting your participation in the program?Do you have someone to help you during the classes?Do you feel the exercises are being delivered at a good speed for you?Do you take resting periods, when needed?What do you think would assist or facilitate you participation in the exercises?Descriptive statistics was performed using the IBM Statistic Package for the Social Sciences software . Textual data from open-ended questions and the instructor’s notes were analyzed using the QDA Miner Lite database. The Braun, Clarke, Hayfield, and Terry (2019) process of content analysis was applied. This method of analysis focuses on identifying recurring common themes, ideas, and patterns of meaning in data. This method comprises 4 stages: preanalysis, encoding, categorization, and interpretation of data.Researchers followed the principles of the Declaration of Helsinki. The study protocol was approved by the Egas Moniz Research Ethics Board . Prior to starting the program, all participants received information regarding the study procedures and provided their written informed consent.The online survey was set up so that participants were free to decide not to answer any question, change or review their responses, or voluntarily quit at any time. To comply with the ethical principles of anonymity and confidentiality, all data collected were free of any personally identifying information, including any form of electronic identifiers.The archive of essential documents was carried out in a locked file, ensuring their prompt availability, upon request, to competent authorities. All digital data were coded and stored on a password-protected computer. All data will remain locked in a file cabinet at Egas Moniz University for 5 years. After this retention period, all data will be destroyed.No individual data will be available.In total, 15 individuals with a diagnosis of PD participated in the program in 16 group sessions of 1 hour each performed twice a week. Participants were mainly female  with a mean age of 69.4 (SD 9.3) years and H&Y stages I-IV. The participants’ demographics and clinical characteristics can be found in Regarding exercise habits, of the 15 participants, 8 (53%) were doing some form of exercise besides the online program. All participants considered that it is important to exercise regularly, having identified several factors that influence them to exercise, namely the benefits of exercise for their health , feeling exacerbation of their symptoms when they do not exercise regularly , having fun when exercising , and feeling guilty when not exercising .The attendance rate was high, with participants coming to more than 13 (81%) of the sessions. People did not attend some sessions, but none totally dropped out of the program. Reasons for not attending reported by participants varied but included medical appointments, health-related issues, and family events. Adherence with the exercises throughout the sessions was also perceived as high by the instructor, with participants continuously engaging in the exercises .In the exit questionnaire, all participants reported having no major problems during the sessions. Several participants  had supervision from family or care partners during the sessions. Based on the instructor’s notes, all participants completed the program, with no report of severe problems during the sessions and some reporting only some fatigue.After the program, participants provided favorable feedback, with 8 (53%) feeling very satisfied and 7 (47%) feeling satisfied with the program. In addition, all participants reported their willingness to attend future online classes and said they would recommend it to another person with PD. Patients had favorable perceived benefits of the program  for the current management of exercise/health habits, with 14 (93%) considering it very useful and 1 (7%) moderately useful.The frequency of 2 sessions per week was considered ideal by 8 (53%) participants. The remaining participants expressed a preference to participate 3  to 4  times a week in this type of sessions. Regarding the type of physical and cognitive exercises, only 2 (13%) of the participants were familiar with this type of exercises performed. For the remaining 13 (87%) participants, the combination was something new.Considering all activities carried out, 9 (60%) participants reported enjoying all these activities in general. All participants expressed a preference for 2 components: (1) performing cognitive exercises, responding verbally, and movements and (2) moments of explanation by the health professional regarding Parkinson-related issues or about the reason for the specific exercise. The less preferred components were (1) performing physical movements  and (2) relaxing at the end with soft music .The social moments that allowed the exchange of experiences and ideas was considered useful by 14 (93%) participants, with 1 (7%) participant expressing neutrality in relation to this component of the program.Instructor notes highlighted that some time had to be allocated to the logistics of the program , and this was critical for program success. However, the time requirements necessary decreased as the participants became more agile with the technology and program. Additionally, the instructor reported that the most stressful aspect was the need for constant modification of the exercises during the sessions. This included adjustments to the type of physical activities, the length of the exercise, the use of verbal feedback, the time for learning in a group setting, and resting periods suitable for all. The instructor noted that alerts for the risk of falling and abnormal postural behaviors were needed but occurred mainly at the outset of the sessions and were gradually less needed. Of the 15 participants, 1 (7%) demonstrated increased dyskinesia when dual-tasking, but this was nondisabling. In addition, 3 (20%) participants reported feeling some mental fatigue in the first sessions and additional resting periods were included, alongside less cognitively demanding activities introduced to reduce this discomfort.Of the 15 participants, 10 (67%) expressed having difficulties throughout the program occasionally, 3 (20%) reported experiencing difficulties frequently, and only 2 (13%) were totally comfortable with the program, without feel difficulties. The difficulties most frequently mentioned by the participants were dealing with fluctuations in their health status , difficulties in mastering and understanding the technologies necessary to participate , difficulties with not being able to listen to instructions due to hearing loss , initially feeling confusion and disturbance due to everyone talking at the same time , lack of personal motivation , and frustration for not being able to perform the exercises .The factors most frequently identified by the participants as facilitators to participating in the online exercise program were (1) the dynamics, experience, and professionalism of the health professional who performed the sessions ; (2) perceiving the benefit of participating and feeling that it improves their well-being ; (3) feeling motivated to exchange experiences with the group and people with a similar disease ; (4) mastering the use of technologies ; (5) suitable timing of the sessions ; and (6) having the support of family members or caregivers during the sessions .There are recognized potential benefits of dual-task exercise for people with PD and calls to include cognitive exercise as a component of comprehensive physiotherapy care ,20. HoweAttendance and satisfaction was high, with people attending 81% of the sessions. This was concordant with previous feasibility studies on online dance therapy programs for patients with PD showing a 100% attendance rate .When compared to in-person programs, online programs can be a way to provide personalized and timely care to more people with PD. This may ultimately impact the change in care models as online therapies can complement in-person care as a means of ongoing and easy access to specialized care.We believe that online training will be a valid option for disease management in the future as the development of key enabling technologies will allow health professionals to provide training guidance and monitor movement, behavior, and cognitive/motor learning. This state of play will allow real-time assessment of patients' posture and performance, enabling health care professionals to provide timely feedback so that online training contributes more effectively to treating PD ,35.Several aspects of this program contributed to the high attendance and satisfaction. First, adherence may be related to highly motivated patients who have continuous encouragement to participate . The phoImportantly, participants were able to bypass common safety concerns and technological difficulties with support from care partners and the instructor. Even though 47% of the participants had reported initially having difficulties with balance and falls and 60% had walking difficulties, no one experienced major problems. This was also in concordance with previous feasibility studies on online programs for PD showing no adverse events .Important insights regarding the effects of combining dual modes of exercise in PD were identified. The physiotherapist/instructor reported that the most stressful problem/factor was the need for constant modification of the exercises during the sessions. Importantly, this constant modification may represent a barrier to replicating programs easily, and further detailed description and research on the exercises delivered should be carried out and shared with others in educational courses .Our study was not without limitations mainly intrinsic to its acceptability nature, small sample size, and single-center design, which imposed restrictions on the generalizability of the findings. Second, although primary data were collected through an anonymous online questionnaire, we should not exclude that the participants' actual reports may diverge from what they revealed due to biases, such as a lack of confidence in guaranteeing anonymity or protecting identity values or beliefs. In addition, we also must consider that if participants perceive it to be socially desirable, they might overstate the frequency of positive items. Third, we included a fairly heterogeneous group of people with PD from different backgrounds and did not speciﬁcally include people with PD with cognitive impairment. Even though it remains to be determined how such people can undergo dual-task training, given the potential benefits of this type of program on cognition due to its cognitively demanding components , these programs may be of particular interest in treating cognitive dysfunction in individuals who already display mild cognitive impairment . Future programs should attempt to include individuals with cognitive limitations that reflect the type of frontal lobe deficits that are more impaired in fallers . ImportaEven so, the use of dual-task training may already be of particular interest in preventing or delaying cognitive dysfunction in people with PD who are not (yet) affected. Additionally, people with PD in early disease stages are more amenable to these types of innovative therapeutic interventions. Any contribution to reduce the obvious pitfalls in such dual motor cognitive interventions will be a valuable adjunct to reduce the evolving needs of people with PD and their care partners. An additional limitation in group settings is the impossibility of effectively monitoring patients during the sessions and registering potential situations of wearing off, sudden offs, patients in off , etc. Reaction to the exercises and safety was the primary assessment being conducted during the sessions by the therapist. The type of medication taken by participants is another important issue as the dose and type of pharmacological treatments will highly impact patients’ overall functionality.Dual-task training is an emerging field for PD, but access to such specialized care remains limited. Online community-based dual-task exercise programs, such as the PD3 Move program, specifically adapted for people with PD may be an excellent tool to facilitate access to previously reported benefits of dual-task training. It can provide a safe and enjoyable way to reduce physical and cognitive inactivity commonly seen in PD. Yet, the design, type of visual display, type of sessions, and participants included will need further reflection. Sharing the concept of such a program implementation may fuel the development of future research and similar community exercise care services for PD that incorporate the complexity of the cognitive challenges in PD. This ultimately may lead to—at least partly—initial treatment suggestions for those who decide to start such an online program for PD and be useful to guarantee safety and better care to the target population.


NUP214::ABL1 fusion is detected in 6% of T-cell acute lymphoblastic leukemia (T-ALL), and is very rare in B-ALL. We present a case of adolescent with B-ALL and a cryptic NUP214::ABL1 fusion which was initially missed during diagnostic screening and was detected by additional RNA sequencing. Treatment with specific ABL-inhibitor Imatinib was added later in therapy with a good effect. Initial treatment according to conventional chemotherapy was complicated by severe side effects. At the end of Consolidation, the patient was stratified to a high risk group with allogeneic hematopoietic stem cell transplantation because of insufficient response to therapy. At that time, targeted RNA sequencing detected NUP214::ABL1 gene fusion which was previously missed due to a small microduplication in the 9q34 chromosome region. Gene variant analysis revealed no TKI-resistant ABL1 mutations; therefore, treatment with Imatinib was added to target the NUP214::ABL1 fusion protein. A negative minimal residual disease was achieved, and treatment was downgraded to intermediate risk protocol. Combining routine genetic assays with next-generation sequencing methods could prevent from missing atypical gene alterations. Identification of rare targetable genetic subtypes is of importance in order to introduce targeted therapy as early as possible that may improve survival and reduce toxicity. Treatment with ABL1 inhibitor imatinib mesylate revealed as a highly effective targeted therapy against the leukemia driving protein kinase.Acute lymphoblastic leukemia (ALL) with recurrent genetic lesions, affecting a series of kinase genes, is associated with unfavorable prognosis, however, it could benefit from treatment with tyrosine kinase inhibitors (TKI).  BCR-ABL-like subgroup of tyrosine-kinase driven ALL has been associated with a poor response to chemotherapy, a high relapse risk, and unfavorable long-term outcomes  gene fusion is detected in 6% of T-ALL, whereas it is rare in B-ALL   analysis and reverse transcription polymerase chain reaction (RT-PCR) (BCR-ABL1-like ALL is known for its highly heterogeneous background (Acute lymphoblastic leukemia (ALL) is a heterogeneous disease with certain cytogenetic aberrations being long-recognized to be anticipated an unfavourable prognosis. A new outcomes . In the l entity . Its gen1 fusion . The preion NUP21::ABL1 gers (TKI) . However limited . Standar(RT-PCR) . Neverthckground .NUP214::ABL1 gene fusion which was initially missed during diagnostic screening due to unusual genetic alteration and was identified by the targeted next-generation sequencing (NGS) only. Treatment with a first-generation TKI (imatinib) was added to the chemotherapy with a good effect.We present a case of pediatric B-ALL with a cryptic 9/L, and hyperleukocytosis 464.5 109/L. Immunophenotyping confirmed the expression of B-lymphoid markers CD45, CD19, CD10, CD20, CD81, CD22, cCD22, CD24, and cCD79a. Routine genomic screening by a single nucleotide polymorphism array (SNP-A) detected normal male karyotype 46,XY without larger aberrations in size ≥5 Mb. FISH and RT-PCR did not detect any of the following recurrent rearrangements: BCR::ABL1, KMT2A, EPOR, ABL1, ABL2, RUNX1 (CSF1R), PDGFRB, E2A (TCF3), JAK2, ETV6::RUNX1, or CRLF2. Cerebrospinal fluid showed three WBC/μl and ∼5.8% of aberrant phenotype B-lymphoid cells, with no leukemic blasts in cytospin. B-ALL, CNS1 was diagnosed.A 15 year-old boy with no previous significant medical history was admitted to our pediatric department in July 2020 for high fever, petechial and hemorrhagic rash, and vomiting. The blood count showed hemoglobin 51 g/L, platelet count 34 × 10Treatment was conducted according to ALLTogether protocol Induction B with dexamethasone, vincristine, daunorubicin, pegylated-asparaginase (PEG-Asp), and intrathecal methotrexate. At the end of induction, at time point 1 (TP1), minimal residual disease (MRD) in bone marrow showed residual cells of 0.79% by flow cytometry (FC) and 0.03% by IG/TCR quantitative PCR, respectively. Discrepancy in the lab results was interpreted as a subclone of leukemic cells that was not captured by PCR, and the patient was stratified to intermediate-high risk (IR-H) due to a slow response to the therapy as per protocol . ConsoliNUP214::ABL1 gene fusion. Exon 33 of the NUP214 gene and exon 3 of the ABL1 gene were fused. RT-PCR method was used to confirm NUP214::ABL1 fusion transcript. Gene variant analysis showed no TKI-resistant ABL1 mutations; therefore, treatment with a first-generation TKI imatinib mesylate was added to the conventional chemotherapy. Complete remission (CR) was achieved within a month, and treatment was downgraded to intermediate-risk protocol , targeted RNA sequencing was performed on patient’s RNA sample using TruSight Pan-Cancer sequencing kit as described earlier . Sequencprotocol . At the The present report was conducted in accordance with the guidelines of the Declaration of Helsinki. Institutional ethical review board permission for a case report was obtained and a written informed consent was received from the patient and his parents.NUP214::ABL1 fusion is a rare genetic entity carrying kinase activating alterations and making the patients candidates for TKI treatment. Although ABL1 gene rearrangements are most commonly detected in B-ALL, NUP214::ABL1 fusion transcript is mainly described in T-ALL patients  at presentation, similarly to our case. In all reported cases, the patients had a poor initial response to therapy and were stratified to very high risk (VHR) chemotherapy (case #4) or allogeneic hematopoietic stem cell transplantation  (cases #1 and #5). Two patients received treatment with TKI after allo-HSCT and achieved negative MRD, however, subsequent disease progression in case #4 resulted in lethal outcome. One patient (#5) underwent successful allo-HSCT without additional TKI use.To the best of our knowledge, only five B-ALL cases with BCR::ABL1 positive ALL, therefore, dasatinib, nilotinib or ponatinib are preferred as first line therapy  reported a second-generation TKI dasatinib, and a choice of TKI was not specified in the cases #2 and #3. Mechanisms of resistance to imatinib are known to be related to the mutations of ATP-binding site in  therapy ,17. In oABL1 gene rearrangements by conventional genetic analysis can be a challenge  due to a small 445 Kb microduplication in the 9q34 chromosome region and was detected later by performing targeted RNA sequencing. SNP-A karyotyping missed microduplication which was smaller than 5 Mb in size  should be immediately analyzed further by NGS methods to prevent from missing atypical gene alterations.Identification of rare targetable genetic subtypes is of importance in order to introduce individualized targeted therapy as early as possible to improve survival and reduce toxicity. Combining TKI with chemotherapy for