A high DM rate of 33.3% was noted in PNI/LVI double-positive patients. Among the PNI/LVI double negative, single positive to double positive subgroups, increasing LN+, DM rates and decreasing DSS rate were observed. Among the 44 LN+ patients, PNI/LVI double positive remained associated with a markedly high DM rate of 42.9% and a poor 5-year DSS of 27.7%. PNI/LVI double positive plays important roles in prognostication and potential clinical application for T3–4 OSCC by independently predicting LN+, DM, and poor DSS, and can be used as a good marker to select DM high-risk patients for novel adjuvant therapy trials.Postoperative adjuvant therapy has been indicated by advanced T classification for T3–4 oral squamous cell carcinoma (OSCC) and the significance of perineural invasion (PNI) and lymphovascular invasion (LVI) in treatment for T3–4 OSCC remains unclear. Ninety-eight cumulative patients with T3–4 OSCC who underwent curative surgery between Jan 2002 and Dec 2010 were recruited and analyzed. Twenty-seven (27.6%) patients were PNI/LVI double positive. PNI/LVI double positive demonstrated independent predictive values for higher neck metastasis (LN+), higher distant metastasis (DM) and low 5-year disease-specific survival (DSS) rates ( Surgery is the mainstay of treatment for OSCC, and pathologic features are commonly used for risk group stratification and treatment decision. Perineural invasion (PNI) and lymphovascular invasion (LVI) are two pathologic features representing the consequence of tumor microenvironment interactions. They are established poor prognostic factors in many human malignancies6. In OSCC, either PNI or LVI has been shown to correlate with advanced tumor stage, neck metastasis and poor survival10, and are two required elements in standard pathology report for OSCC11. In the National Comprehensive Cancer Network guidelines, both PNI and LVI are adverse pathologic features indicating the need for postoperative adjuvant therapy12.Oral squamous cell carcinoma (OSCC) is the most common type of head and neck cancer15. Compared with T1–2 OSCC, T3–4 OSCC generally has a higher neck metastasis rate and poor prognosis, and aggressive combination treatment of surgery and postoperative radiotherapy (PORT) or chemoradiotherapy (POCCRT) is frequently required17. The advanced T classification has already been regarded as one adverse feature for treatment decision in treatment guidelines12, although pathologic features, such as PNI and LVI, appear at a much higher frequency in T3–4 OSCC19. Thus, the significance and clinical application of PNI and LVI in the treatment decision for T3–4 OSCC remain unclear.T1–2 OSCC can be treated by single modality of surgery, and adjuvant therapy is generally not indicated in the absence of neck metastasis or adverse pathologic features10. Many recent studies, including ours, have focused on the role of PNI or LVI in early T1–2 OSCC24, but few studies have focused on T3–4 OSCC which has more aggressive clinical courses and poor prognosis. This retrospective study aimed to investigate the role of pathologic features of primary tumor, particularly PNI and LVI, in T3–4 OSCC and explore their potential clinical applications.Most studies of PNI and LVI in OSCC included patients of all T classificationsOf the 98 T3–4 OSCC patients, 93 were male and 5 were female, with a mean age of 54 years (range 32–91 years). The clinicopathologic features are summarized in Table p < 0.001) and differentiation (p = 0.002) independently predicted LN+ after controlling for PNI, T classification and tumor thickness in multivariate analysis (Table p = 0.001). Notably, POCCRT was done at a higher percentage among patients with DM than those without DM . Trends of higher local recurrence  and regional recurrence  rates are also observed among patients with DM. The mean time of DM development was 9.0 months (range 3–24 months), and the mean survival time after DM diagnosis was 3.76 months (range 0–11 months). The 5-year disease-specific survival (DSS) and overall survival (OS) were 70.6% and 62.5%, respectively.Of the 98 T3–4 OSCC patients, neck metastasis (LN+) was shown in 44 (44.9%). We found that LVI ( p = 0.00 independp = 0.026) after controlling for PNI, LVI, T classification, differentiation, and margin. Independent predictors for DSS were PNI (p = 0.010) and LVI (p = 0.049) after controlling for T classification, thickness, differentiation, and margin. In summary, when PNI and LVI were separately considered, PNI predicted a worse DSS, LVI predicted LN+ and a worse DSS, and neither PNI nor LVI predicted DM independently.Preliminary multivariate analyses were performed for the independent roles of pathologic features of the primary tumors Table . We founp < 0.001), DM (p = 0.017) and DSS (p < 0.001) after controlling for other pathologic features of the primary tumors.Given the inconsistent predictive values of PNI and LVI for LN+, DM and DSS, further analyses were performed by combining PNI and LVI as one pathologic feature. Twenty-seven (27.6%) patients with both PNI and LVI were classified as PNI/LVI double positive. The other 71 72.4%) patients were classified as non-double positive, including 34 (34.7%) as single positive (either PNI or LVI) and 37 (37.8%) as double negative. Univariate analyses demonstrated significant and higher impacts by PNI/LVI double positive on LN+ (77.8%), DM (33.3%) and 5-year DSS (44.3%) rates, compared with the impacts by either PNI or LVI , particularly worse for the PNI/LVI double positive group   was included in the 8th edition of American Joint Committee on Cancer (AJCC) staging manual for oral cavity cancers29. The rationale of combining PNI and LVI in this study is that they are both pathologic features of the primary tumor that are related to tumor-microenvironmental interactions. Despite being less well understood, PNI has been considered a process of cancer spread distinct from LVI2. Both PNI and LVI data are widely available being two required components in pathologic reports of OSCC11. The role of PNI in predicting neck metastasis in early T1–2 OSCC has been reported30. We recently reported that the migration ability of OSCC could be enhanced by human peripheral nerves31, although the exact mechanisms require further investigation. On the other hand, LVI implies the invasion of tumor cells to blood or lymphatic vessels, a process that has long been accepted as an important step in the tumor metastatic cascade or angiogenesis32. Given the distinct metastatic mechanisms of PNI and LVI, our results suggest that combined consideration of PNI and LVI consistently contributes to independent predictive roles for LN+, DM and DSS in T3–4 OSCC.The concept of combining poor risk factors in head and neck squamous cell carcinoma (HNSCC) has been reported34. However, our data demonstrated a higher DM rate of 17.3% in an exclusive T3–4 OSCC cohort despite that POCCRT was done at significantly higher percentage among patients who developed DM. Moreover, the 52.9% isolated DM rate was also higher than the reported data34. In consistent with previous reports, most DM developed in the lung and occurred early within 2 years after treatment completion in our study. The short mean survival time of 3.76 months after DM diagnosis implies again that DM of OSCC is mostly not salvageable36. Thus, strategies of prediction and early management of patient high risk for DM before clinical presentation might be important to improve the survival for OSCC. We have reported the value of chest CT especially during follow-up period every 6 months for the first 2 years in high-risk HNSCC patients37. Our results thus suggest that PNI/LVI double positive is another high-risk factor indicating the need of chest CT follow-up after treatment completion.A lower incidence of DM approximately 10% has been reported for OSCC than for other HNSCC38, and LVI and LN+ have been demonstrated to be an essential marker for DM40. By combining PNI and LVI in this study, the 33.3% DM rate in PNI/LVI double positive patients is even higher than 27.3% in LVI-positive, or 31.8% in LN+ patients. Moreover, the impacts of PNI/LVI double positive specifically in the 44 LN+ patients remained consistent despite not statistically significant for DM due to the small patient number  and retrospective study design in our cohort. Another limitation is the use of tumor thickness instead of DOI, which was not routinely reported during the study period. Also, maintenance adjuvant therapy has not been shown to be beneficial in the literature for OSCCIn conclusion, combined consideration of PNI and LVI is important in prognostication and potential clinical application of T3–4 OSCC. PNI/LVI double positive independently predicts LN+, DM, and poor DSS in T3–4 OSCC, and can be used as a good marker to select DM high-risk patients for future novel adjuvant therapy trials with the goal of reducing DM and improving the survival for T3–4 OSCC.From the hospital database, 467 patients with newly diagnosed OSCC who underwent curative surgery at the Department of Otolaryngology- Head and Neck Surgery, Taipei Veterans General Hospital between Jan 2002 and Dec 2010 were reviewed. The patients were staged according to the 7th edition of the AJCC Cancer Staging Manual. Patients with DM at initial diagnosis, head and neck cancer history, radiation history to the head and neck, or synchronous malignancies were excluded. In total, 98 patients with pT3–4 OSCC were included in the present study. All the patients underwent curative surgery, including neck dissection with or without flap reconstruction. Adjuvant therapy was given following institutional treatment guidelines, particularly to patients with multiple neck metastases, close or positive margin, PNI, LVI or ENE. Regular follow-up investigations were arranged, including computerized tomography (CT) scan or magnetic resonance imaging of head and neck area every 3 months for first year, 6 months for the second year, and yearly thereafter. Chest radiograph was done yearly. Chest CT was arranged for high-risk patients, including those suspected for recurrence. The medical records were reviewed and updated, and patients or their families were contacted by telephone if lost to follow-up for more than 1 year. This study was reviewed and approved by the Institutional Review Board (IRB No. 2020-09-005BC) of the Taipei Veterans General Hospital. All methods were performed in accordance with the relevant guidelines and regulations. The IRB waived informed consent for this retrospective study.2. LVI was defined as the detection of tumor cell infiltration in endothelial-lined vessels or tumor nests within or attached to the endothelial cell lining of the lymphovascular space. PNI/LVI double positive indicated the presence of both PNI and LVI in the primary OSCC.Pathologic features of the primary OSCC were recorded from the pathology report. The data of DOI was not available during the study period, and tumor thickness was measured from the tumor surface or the ulcer base to the deepest point of invasion. The margins were classified as “positive” or “non-positive.” The “non-positive” margin included dysplasia, close (< 2 mm) or negative margin. A senior pathologist who was blinded to the clinical data re-reviewed all hematoxylineosin stained tissue slides, and special attention was given to PNI and LVI, which were identified throughout the whole area of tumor sections serially under 40× or 100× magnification and were confirmed under 200× magnification. PNI was defined as positive when tumor cell infiltration was identified in any layer of the nerve sheath, or tumor in close proximity involving more than one third of the nerve circumferencep value < 0.05 was regarded as statistically significant. All statistical analyses were performed using Statistical Package for Social Sciences software .The endpoints of this study were to evaluate the association between pathologic features and LN+, local and regional control, DM and survival. Comparisons between categorical data were performed using Pearson’s chi-square test or Fisher’s exact test where appropriate, and the independent effect was analyzed using binary logistic regression models. The survival rates, OS and DSS were analyzed by the Kaplan–Meier method. Cox proportional hazards model was used to test the independent significance. Disease-specific death was defined as death either from the index tumor or from treatment-related events. A


Purpose: Pipeline programs are a well-known approach to enhancing health care workforce diversity and reducing health disparities. Few evaluations of pipeline programs include long-term outcome; fewer still, if any explore perceptions of students after completing such programs, to elucidate factors that contribute to successful entry into the health professions. The authors conducted a program evaluation of three summer diversity pipeline programs in the Bronx, NY, investigating both long-term outcomes and participants' hindsight perspectives of the impact of these programs on their career trajectories.Methods: Investigators conducted a cross-sectional, long-term, mixed-methods survey study. The primary and secondary outcomes for the quantitative analysis were matriculation into biomedical programs to pursue MD or PhD degrees and Master's degrees, respectively, and associated demographic factors. Free-text questions explored the most valuable and influential components of the programs; responses were analyzed qualitatively.Results: Of 147 respondents, 107 (73%) were on-track or had entered a doctoral or master's program, achieving either the primary or secondary outcomes, respectively. Components cited as most valuable included clinical experience, mentorship, career exposure, and research opportunities. Three themes were identified from the free-text responses: (1) Dreams realized; (2) Professional identity formation; and (3) Addressing systemic inequities.Conclusions: These three pipeline programs achieved career outcomes similar to published data. Participants' insights highlight the value of relationships, direct exposure to the health professions, and the importance of such programs to address systemic barriers faced. Results can inform criteria both for participant selection, as well as benchmarks used to define individual and programmatic success. I come from a low-income family and will be the first to graduate from professional school and college. I would have never had the exposure and mentorship that has been essential in my success. This program changed my life.1–4 Yet, the diversity within the health professions is not increasing at a sufficient rate to reflect the racial, ethnic, and linguistic diversity of the current or future U.S. population.5 The Association of American Medical Colleges (AAMC) defines underrepresented minorities (URM) as “racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population.”6 While URMs represent 29% of the total U.S. population, the percentage of URMs entering medical school in 2018–2019 was only 13% .7 In response, many institutions offer “pipeline” programs, interventions addressing the educational and social barriers to successful matriculation into health professional schools faced by economically and educationally underserved and/or URM groups.8–12 Programs seek to “increase the number of well-prepared URM students (in grades K-16) motivated to enter the health professions.”2A more diverse health professions workforce contributes to reducing racial and ethnic health disparities.8 Few program evaluations describe the impact of these programs beyond short-term outcomes.14 This gap in our knowledge is further compounded by minimal reports about the personal impact they have on participants, and what elements contribute most to the students' perception of their own success pursuing a health career.15–17 Without a better understanding of the components of pipeline programs that most effectively contribute to participant success, we risk wasting opportunities and resources, an especially important risk to avoid given the current, limited funding climate.2 The federal Health Careers Opportunity Program, the major funding source for pipeline programs since 1972, has for more than a decade received less than half its FY2005 level.18To achieve their goals, pipeline programs use a variety of intervention models and approaches to academic and professional enrichment and support. Limited program evaluation data hinder program developers' ability to quantify successful attainment of stated goals: A consensus on the methods and benchmarks used to define successful programs and the factors that influence them are lacking.To address this relatively underexplored topic, we conducted a long-term follow-up study of participants in three summer diversity pipeline programs. The objectives of this study were to (1) investigate participants' long-term educational and career outcomes and (2) explore participants' perspectives of the impact of the programs on their academic and career trajectories.The Albert Einstein College of Medicine and its university hospital, Montefiore Medical Center, are located in Bronx, NY. Its three pipeline programs are the Summer Undergraduate Mentorship Program (SUMP), Montefiore Health Opportunities Program (Monte-HOP), and Diversity Student Summer Research Opportunity Program (DSSROP). Although the three summer pipeline programs differ in their selection criteria and intervention approaches, they share an overall goal of diversifying the health care workforce. A detailed side-by-side comparison of the programs is presented in 19 SCCT explains how learning experiences influence the development of education and career choice and achievement through cognitive factors, including self-efficacy and outcome expectations. The model been used for over 25 years, across numerous career settings, and within various underserved populations to explain career development.21Social cognitive career theory (SCCT) served as our conceptual framework; it informed our survey design aiming to understand how students' experiences in the programs, including mentorship and exposure to clinical and research environments, influenced their academic and career choices.We conducted a cross-sectional mixed-method long-term program evaluation. Two authors (C.M.G. and H.S.) developed an 87-item electronic survey based on a literature review and on our previous experience administering pipeline programs. There was skip logic built into the survey, therefore individual respondents answered many fewer than 87 questions. Survey items included demographic information, academic and career achievements, most beneficial components of the program, and barriers to meeting career goals; it included three questions with the opportunity to provide free-text, narrative responses. All individuals who completed any of the three pipeline programs between the years of 2002–2012 and for whom we had contact information were invited to complete the confidential survey via email; three follow-up reminders were sent each spaced 2 weeks apart during the spring of 2013.For the quantitative portion of our program evaluation, the primary outcome was matriculation into medical school or graduate school to pursue an MD or a PhD in the biomedical sciences. The secondary outcome was matriculation into a health professions school at the master's degree level. An age-neutral outcome of “on-track” was defined as progress toward the completion of the primary and secondary outcomes to assess the trajectories of younger students continuing or having recently completed their undergraduate education and recent graduates taking “gap” years. The study received exempt status by the Institutional Review Board of the Albert Einstein College of Medicine.p-value <0.05 was considered statistically significant. Analyses were conducted using SPSS v.24.Statistical analysis included descriptive statistics, including means and frequencies, of the percentage of total participants who achieved the primary or secondary outcome. Pearson's correlation and chi-square tests were conducted to identify any association between demographic variables, including sex, race/ethnicity, and academic level at the time of participation in the pipeline program, and study outcomes. A 22 Narrative responses to the remaining two open-ended questions were analyzed through thematic analysis.23 For the latter, three investigators  independently read one-third of the free-text responses line-by-line and applied categories following inductive coding to the text.22 They then met to discuss and refine their individual codes and devise an agreed upon list of codes along with their definitions to create the preliminary codebook. This codebook was independently applied by at least two investigators to the remaining free-text responses and further refined. Investigators then met to perform further analysis using a constant comparative method, a technique in which data are transformed into larger theoretical categories. Investigators began with low inference codes and discussed their meaning and potential grouping of codes to develop conceptual themes.22 Finally, the relationships between themes were identified, and consensus on representative quotes was reached by the three investigators.Responses to the first open-ended question, “Please describe what was the most valuable to you during your pipeline experience,” were analyzed by frequency coding.p>0.05 for comparisons).Of our 461 summer program graduates, there were 361 with known addresses. We reached out to these 361 and received 158 responses to the survey, resulting in a response rate of 44% (158/361). Surveys without a response to the question, “Education completed since participating in last pipeline program” and duplicate surveys were excluded from our analysis, resulting 147 complete, distinct surveys. Demographic information describing our students is summarized in N=23), time (60%) and money (52%) were the most frequently reported reasons, followed by grades (40%).Primary and secondary outcome achievements by program are listed in N=133) identified distinct aspects of the programs as most valuable, resulting in 31 distinct codes. In descending order of frequency, the students identified the overall most valuable components to be (1) clinical exposure (40 mentions); (2) mentorship (25 mentions); (3) career exposure (20 mentions); and (4) research opportunities (16 mentions). Analysis of the remaining two open-ended questions resulted in three themes demonstrating the influence of the pipeline programs on participant's academic and career trajectories.For the first open-ended question, participants  of our respondents identified as black or Hispanic/Latino/a/x. Our outcomes suggest that our programs have a significant positive impact on the academic and career trajectories of URM students. Our programs successfully prepared participants for medical and graduate school matriculation and fortified their resilience, motivating them to persist along their chosen path. Participants valued exposure to clinical and research environments, as well as learning about career options and receiving direct mentorship. Participation enabled them to pursue their dreams, see themselves in others with a similar professional identity, and mitigate the impact systemic inequities had on their individual trajectories.9 (p. 79). Our findings are in line with similar undergraduate pipeline programs serving URM and disadvantaged students. For instance, one similar program reports 69% (27/39) of graduates matriculated into or on-track to enter health professions graduate programs12; another pipeline program's longitudinal 10-year retrospective study revealed that 74% of graduates matriculated into graduate school in the biomedical sciences.16 Our study presents a unique approach to program evaluation of pipeline programs by applying a cross-sectional, long-term design to collect participants' educational and career outcomes and their hindsight perspectives of the factors which influenced these outcomes.In 2003, Grumbach et al. wrote in their review of 19 controlled studies “the most rigorously designed studies indicated that special summer enrichment programs can boost the success of URM… students in applying to medical school by approximately 25%”21 In addition, contextual supports in the form of mentors, peers, and academic resources, as well as barriers, including finances and grades, both directly and indirectly influenced career interests, choice goals, and subsequent career attainment, as evidenced by reasons for being off-track.20 In addition to the most frequently cited impactful components of the programs, our themes have implications for the design of future pipeline programs and the prioritization of components in an environment with limited resources and funding opportunities.Our qualitative analysis revealed that students valued components of the program that provided direct contact and insight into the health care profession that they would not have otherwise had the opportunity to pursue with. Participants also valued the long-term relationships they formed and opportunities to work with and learn from URM mentors, faculty, and peers. These findings align with SCCT, in that, learning experiences within the pipeline programs which increased self-efficacy and favorable outcome expectations, in turn, influenced career interests, goal choices, and intentions to pursue a health career path.21 Mentorship from URM faculty members, and its positive effects on URM professional identity formation, would therefore be a key component of a successful pipeline program. URM faculty members represent only 8% of faculty in US medical schools.24 They are often tasked with many roles beyond mentoring of URM students and trainees, colloquially called the “minority tax.”26 The importance of mentorship combined with the difficulty URM and women have in obtaining suitable mentors,25 implies that pipeline programs might benefit from incorporating willing faculty from all racial, ethnic, and gender backgrounds to serve as mentors for URM students.27 Students can still be exposed to role models from similar racial, ethnic, and socioeconomic backgrounds during their clinical experiences, through didactics, and potentially with more time-limited programs such as a “Meet the Professor” format. Creative solutions to expose students to successful role models of similar backgrounds while having the more intense relationship with a mentor of any background could still capitalize on the power of seeing role models in positions to which they aspire, as SCCT predicts.One implication involves career choice and professional identity formation. SCCT holds that self-efficacy and outcome expectations developed through social learning experiences increase one's motivation to persist toward academic and career goals.22 A diverse health care workforce enhances patient outcomes and decreases health disparities.3 URM medical students are more likely to work in minority and underserved patient populations.28 Successful pipeline programs would therefore not only benefit individual students, but have potential to benefit communities and the health of our society as a whole.29In addition to the implications for the individual students, our findings have societal implications. A more racially and ethnically diverse class benefits all students.16One final implication of our results could inform recruitment approaches taken by pipeline programs. Although we report the outcomes of how many students achieved or were on track to obtain a medical degree, further research is needed as to what percentage of students entering medical school would reflect “success” of a program. Setting a percentage benchmark arbitrarily may risk increased selection of students who would reach medical school regardless of participation in a pipeline program and whose lives would not be markedly affected by program participation. We advocate for a holistic approach and flexibility to allow programs to recruit students who have the potential and interest to matriculate into health professions schools and who need the “intangible” supports as evidenced by our qualitative results. We recognize this may lead to an overall decrease in the percentage of “successful” participants, but will likely lead to an increase in medical school matriculants from disadvantaged backgrounds and enhance overall diversification of the health care workforce in line with the social justice mission of many of these programs.9–17 a randomized controlled trial design may have been considered unethical unless we were able to offer an alternative intervention program to a control group; this was not feasible with our limited resources. Inactive email addresses limited our ability to reach many participants. The disproportionate representation of SUMP participants (57%) among respondents is likely because the SUMP program's administrator (H.S.) used a personal Facebook page to remain in contact with program alumni over time. Unlike the other two programs, SUMP also used an electronic application process, which allowed for contact data to be more easily exported. We were unable to ascertain if our sample was representative of URM students who participate in educational pipeline programs because, to our knowledge, those data are not published. Response bias may have led successful participants to disproportionately respond or others to respond with more socially desirable outcomes. However, the authors have learned through nonsystematic follow-up that many nonresponders actually entered medical or graduate school and, thus, became too busy to respond. Our findings illustrate the challenges to long-term follow-up and the need for funding to support broader data-sharing agreements.This evaluation is limited due to the lack of information about the students' baseline characteristics before participating in the program, as well as our inability to control for impactful events or experiences between the time of program participation and this follow-up evaluation. Given the existing evidence,Our pipeline program served to enhance students' self-efficacy and resulted in successful academic and career trajectories to become health professionals at the doctorate and master's degree level. Beyond the skills and academic preparation offered, the programs offered support both peer support and mentorship, to facilitate success. Given the importance of these supports on the formation of their professional identity, pipeline programs should take a holistic approach considering the existing networks of applicants for participation. Our findings can inform future research to develop outcome metrics that take these intangibles into consideration. Returning to the federal funding levels of 15 years ago could more than double the scale of the Health Career Opportunity Program and support long-term program evaluations so as to develop efficient, effective best practices to diversify the health care workforce and improve the health of all communities.


A large number of Australians experience mental health challenges at some point in their lives. However, in many parts of Australia, the wait times to see general practitioners and mental health professionals can be lengthy. With increasing internet use across Australia, web-based interventions may help increase access to timely mental health care. As a result, this is an area of increasing research interest, and the number of publicly available web-based interventions is growing. However, it can be confusing for clinicians and consumers to know the resources that are evidence-based and best meet their needs.This study aims to scope out the range of web-based mental health interventions that address depression, anxiety, suicidal ideation, or general mental well-being and are freely available to Australian adults, along with their impact, acceptability, therapeutic approach, and key features.The PRISMA  guidelines for scoping reviews (PRISMA-ScR [PRISMA extension for Scoping Reviews]) guided the review process. Keywords for the search were depression, anxiety, suicide, and well-being. The search was conducted using Google as well as the key intervention databases Beacon, Head to Health, and e-Mental Health in Practice. Interventions were deemed eligible if they targeted depression, anxiety, suicidal ideation, or general mental well-being  in adults; and were web-based, written in English, interactive, free, and publicly available. They also had to be guided by an evidence-based therapeutic approach.Overall, 52 eligible programs were identified, of which 9 (17%) addressed depression, 15 (29%) addressed anxiety, 13 (25%) addressed general mental well-being, and 13 (25%) addressed multiple issues. Only 4% (2/52) addressed distress in the form of suicidal ideation. The most common therapeutic approach was cognitive behavioral therapy. Half of the programs guided users through exercises in a set sequence, and most programs enabled users to log in and complete the activities on their own without professional support. Just over half of the programs had been evaluated for their effectiveness in reducing symptoms, and 11% (6/52) were being evaluated at the time of writing. Program evaluation scores ranged from 44% to 100%, with a total average score of 85%.There are numerous web-based programs for depression, anxiety, suicidal ideation, and general well-being, which are freely and publicly available in Australia. However, identified gaps include a lack of available web-based interventions for culturally and linguistically diverse populations and programs that use newer therapeutic approaches such as acceptance and commitment therapy and dialectical behavior therapy. Despite most programs included in this review being of good quality, clinicians and consumers should pay careful attention when selecting which program to recommend and use, as variations in the levels of acceptability and impact of publicly available programs do exist. Approximately half of the Australian population experiences a common mental disorder  at some point in their lives . Mental When people do seek help for mental health issues, 71% will contact their general practitioner (GP) at the first instance . GPs plaThere is a pressing need to bridge the gap between primary health and mental health care, particularly for Australians facing greater difficulties in accessing mental health care and who are at particular risk of suicide, such as those in rural areas. Fortunately, Australians are increasingly willing to use the internet to search for health-related information , with atAcross the globe, web-based interventions have been shown to effectively reduce the severity of symptoms of depression , anxietygeneral mental well-being and suicidal ideation, which are issues commonly encountered in clinical practice but not covered by past reviews. We note that many reviews of the evidence for web-based interventions for specific disorders  and pression ,33) haveThe purpose of this research is to identify the scope of and evidence for web-based mental health interventions addressing depression, anxiety, suicidal ideation, or general mental well-being that are currently available for free to Australian adults, and in doing so, develop a quick reference guide for GPs and mental health clinicians. It seeks to answer the following questions :What web-based interventions addressing depression, anxiety, suicidal ideation, or general mental well-being are currently freely available  in Australia?What are the key gaps?What are the characteristics, including evidence of impact, credibility, and accessibility, of the interventions that are available?This scoping review was conducted in accordance with the PRISMA-ScR  . The stuEligible interventions were those that (1) targeted depression, anxiety, suicidal ideation, or general mental well-being ; (2) were web-based; (3) had an interactive component ; (4) were designed for adults; (5) were free  and publicly available ; (6) were available in English; and (7) were based on an evidence-based therapeutic approach.that restricted access or limited program availability; or (5) included classes that had to be attended in person.Interventions were excluded if (1) they were targeted at consumers with a primary general medical condition ; (2) solely provided reading materials on one of the inclusion outcomes  or only offered a single tool  without relevant psychoeducation or explanation of a broader evidence-based therapeutic approach; (3) were targeted at health professionals for training purposes; (4) were part of a research project depression, anxiety, suicide, and wellbeing. A search log was kept to record the number of hits for each search, as well as the number of websites that were included or excluded . As the previously published scoping review on web-based interventions for depression only found 1 additional website when searching Yahoo and Bing , only Goexcluded . The seaIn addition, key intervention databases were searched by 2 reviewers (SvdK and SL). These included Beacon , Head toScreening of the search results to identify eligible web-based programs was completed in 2 stages. The first stage involved screening all search hyperlinks by 1 reviewer (SvdK) to eliminate websites that solely provided information on 1 of the outcomes or those that were clearly irrelevant . The remainder of the websites was then divided into those that directly offered web-based programs and those that linked to web-based programs. Any duplicate websites were removed. The second stage involved screening potential eligible programs against the eligibility criteria by 2 reviewers  in November to December 2020, with the reviewers independently extracting data for each website so that the data extraction was performed in duplicate. Discrepancies in coding were resolved by discussion and, where necessary, investigated by the third reviewer. Where necessary, program authors were contacted to request access to the program. For 2 websites , information was extracted from the public-facing pages where possible, and the remaining information was requested from the website authors or developers because of time and administrative constraints.To identify relevant peer-reviewed papers and thereby explore the empirical evidence behind each program, we checked the program websites and the Beacon directory and contacted the program authors. A total of 3 reviewers  then conducted a rapid review of the peer-reviewed evidence for each program. This was limited to evaluations of the complete web-based program rather than individual program elements or individual modules. In addition, the level of evidence was determined for each peer-reviewed study .yes were given a score of 1 and questions that were answered no were scored 0. Questions that could not be evaluated were given a score of 0. Questions that were not relevant to the website were excluded from the final score (listed as “NA”). The final scores were calculated as a percentage of the number of relevant items for that website. A total of 3 reviewers  independently assessed the programs against the evaluation scoring system. Discrepancies were resolved through discussion.A program evaluation scoring system was adapted from previously published guidelines for evaluating and reporting web-based intervention research , similarAuthorship detailsWere the names and credentials of authors or organizations present?Were contact details provided?Was country of origin stated?Focus and target populationWas the primary focus, goals, or objectives of the intervention stated?Was the target audience or mental health issue defined?Professional supportWas there a statement of professional support ?Multimedia channel of deliveryDid the program offer a multimedia content delivery ?Model of changeWas the model of change  defined or stated?Type and dose of interventionWere the number of modules or time to complete each module stated?Was the intervention tailored to the user or was it generic for all users ?Was the program easy to navigate ?Was the information on what is covered in the intervention modules provided ?Program evaluationWas evidence for the program provided to the user ?Has the program been empirically evaluated?Ethical issuesWas a unique username or password provided to users?Was a privacy notice specified?Were the terms and conditions specified?The results were summarized quantitatively by main categories and subcategories, as displayed in web-based interactive, and 32 (62%) programs were classified as web-based with downloadable worksheets/resources. These 2 types of programs were separated in the summary tables.As seen in We only found 2 programs  designed to address distress in the context of suicidal ideation. It should be noted that the inclusion of BeyondNow was debated between team members, as we experienced difficulty in determining whether it fully fulfills the criteria of using an evidence-based therapeutic approach. Nonetheless, it is considered a valuable tool for the management of suicide ideation; therefore, it was included in this review.tailored version of the platform for those organizations to distribute freely to their members.Of the 52 programs, 36 (69%) were available at no cost and open to all Australians, 9 (17%) were accessible via referral from a health care professional (via This Way Up), 4 (8%) required an assessment before registration (via MindSpot), and 3 (6%) required sign-up to a research project  of programs and most often required users’ names, emails, location , and age. Approximately 6% (3/52) of programs created by My Digital Health were offered as part of a research study; of these, 2 programs, Life Flex and Life Flex–LGBQ , required the completion of comprehensive demographic, mental health, and well-being questionnaires before access was granted. Approximately 17% (9/52) of programs did not require registration and allowed users anonymous access. All websites were accessible via mobile phones and did not contain advertisements, except for the 5 Living Life to the Full programs, which contained advertisements for e-books related to the course content.accessible to Australians. Unsurprisingly, most of these programs identified Australia as their country of origin , although other countries of origin were Scotland , Canada , the United States  and the United Kingdom . Contact details for the authors or developers were provided by almost all websites . An overview of program developers and website links is provided in The review was limited to programs that were Overall, 17% (9/52) of programs specifically addressed depression, 29% (15/52) of programs addressed anxiety, 25% (13/52) of programs addressed multiple issues, 25% (13/52) of programs addressed general well-being, and 4% (2/52) of programs addressed suicidal ideation. Most programs specifically stated that their target audience was the general adult population , with other  programs not specifying a target group but appearing to be designed for adults, based on content. Several programs were designed for specific target populations, including expectant mothers or new parents , tertiary students , farmers or rural communities , church-going adults , the LGBQ community , Aboriginal and Torres Strait Islander people , young adults , adults aged 26 to 65 years , and older adults .Approximately 88% (46/52) of programs provided a statement of professional support, that is, mental health clinicians or researchers were involved in the programs’ development. Of the 52 programs, 11 (21%) programs allowed users to invite their independent clinician to monitor their progress in the course, and a further 10 (19%) programs offered optional therapist support while engaging in the program. Approximately 8% (4/52) of programs  did not offer therapist support through the free version of their programs; however, therapist support was an optional inclusion for paid versions of the program licensed to organizations. Only 8% (4/52) of programs  enabled peer support, such as through forums, and in all cases, this was expert-moderated. Most programs offered multiple forms of information delivery, with text, graphics, audio or video, characters, and case scenarios as common features. Email or SMS text message reminders were available 60% (31/52) of the programs.The intervention characteristics of web-based interactive programs and web-based programs with downloadable worksheets or resources are shown in On the basis of either self-descriptions or the clinical judgment of the authors, 90% (47/52) of programs were based on cognitive behavioral therapy (CBT), either with 21% (11/52) or without 69% (36/52) other therapeutic approaches. In combination with CBT, some programs included motivational interviewing , interpersonal therapy , or mindfulness . Only 2% (1/52) of programs (ifarmwell) was based on acceptance and commitment therapy (ACT), 2% (1/52) of programs  on dialectical behavior therapy (DBT), and 2% (1/52) of programs (BeyondNow) on problem solving .tunneled, where participants were led through a series of modules or sessions in a specific order. A further 42% (22/52) of programs recommended to follow the program in a specific order but allowed users to choose the order in which they completed them. Approximately 4% (2/52) of programs were part of an RCT research project that had open access and allocated participants to 1 of 2 therapy structure conditions: mandatory (participants did not choose the order) or optional (participants were given the freedom to choose the order of completion). Approximately 6% (3/52) of programs were unstructured.In total, 48% (25/52) of programs were structured or The number of modules or activities offered within a program ranged from 1 to 14 , except for 1 program (BeyondNow), which offered a safety planner instead of modules. The recommended time to complete the modules ranged from 1 week to 3 months . Approximately 33% (17/52) of programs did not recommend a set treatment length. Self-assessments were included in 79% (41/52) programs, and it was mandatory in 56% (29/52) of programs and optional in 23% (12/52) of programs.All programs, except 4% , featured worksheets, either within the web-based intervention or as downloadable forms to be completed offline. Diaries were featured in 46% (24/52) of programs, and 75% (39/52) of programs included some type of mood or symptom monitoring.Out of 17 possible evaluation items, 75% (39/52) of programs could be evaluated against all 17 items, 15% (8/52) against 94% (16/17) of items, and 10% (5/52) against 76% (13/17) of items . In totaEvidence of the program’s usability  was provided for 52% (27/52) of programs. In many cases, where a website offered ≥1 program, these results were provided for the suite of programs listed on the website generally rather than for the specific programs . Approximately 56% (29/52) of programs were supported by some type of evaluation; 59% 17/29) had undergone effectiveness trials, and 41% 12/29) had empirical evidence of both effectiveness and acceptability . Approximately 68% (19/29) of the evaluated programs had undergone evaluation via RCT . Of the 23 programs without evaluation data, 6 (26%) are currently in the process of being evaluated. All 52 programs and the type of evaluation studies conducted, including the level of evidence, are shown in /29 had e/29 had uThe vast majority of programs provided privacy notices  and terms and conditions . Approximately 88% (46/52) of programs provided crisis links in the form of telephone numbers for emergency services, numbers of helplines or links to websites, and other resources. Only 1 website did not offer crisis links for its programs (CCI); this website was developed by the Western Australian Department of Health to provide evidence-based web-based resources to support practitioners, as well as to provide self-help materials for individuals. The terms and conditions of the website state that the information is provided for information purposes only.The purpose of this scoping review was to summarize the freely available web-based resources based on an evidence-based therapeutic approach for Australian adults seeking self-help for depression, anxiety, suicidal ideation, or general well-being. We sought to describe their characteristics, including therapeutic approaches, key features, and the quality of evidence behind them and thereby produce an accessible summary to inform clinicians’ selection of web-based interventions for their patients. This review builds on past reviews of web-based resources for depression  and anxiA total of 52 web-based programs were identified, of which 20 (39%) were classified as web-based interactive programs, and 32 (62%) were classified as web-based programs with downloadable worksheets or resources. Of these 52 programs, 29 (56%) had been empirically evaluated, and most of the evidence was assessed as level II or III studies. This is similar to past reviews, which found that 50% of the programs addressing anxiety had undergone empirical evaluation , and 38%The evaluation of the included programs showed that, on average, programs scored 85%, indicating that the included web-based programs are generally of good quality. The lowest score was for Students Against Depression (44%) as it did not include information about the developers or the evidence behind the program, privacy notices, terms and conditions. However, Students Against Depression was one of the programs that offered downloadable modules, similar to the CCI programs, and it is currently undergoing evaluation, which may lead to a higher evaluation score once completed. The CCI programs, which scored third to last but are highly regarded by clinicians, included a privacy notice and terms and conditions, and their scores ranged between 69% and 75%. The highest scores were given to the 4 programs by MindSpot, which could only be evaluated against 77% (13/17) of the items. These programs could not be evaluated on more of the items because of restrictions on accessing the program, which may have resulted in higher scores.psychological flexibility [Almost all programs reported that they were based on CBT, either with or without other therapeutic approaches. Only 1 program, ifarmwell, was based on ACT. Although traditional CBT focuses on challenging unhelpful thoughts, ACT focuses on defusing from and accepting them and finding value-consistent ways of living despite the circumstances to build xibility . A recenxibility  found 25xibility  overlappxibility ,155. It xibility .The results of the current review found that 19% (10/52) of programs provided free therapist support, and 21% (11/52) allowed users to link in their existing clinicians. Research examining whether therapist-guided versus self-guided web-based interventions are more, less, or equally effective has produced mixed results. A recent meta-analysis of 21 studies of web-based programs targeting depression found no significant differences in the effectiveness of the programs to prevent depression based on whether they were guided or not . In contThis review found that roughly half of the programs were structured or tunneled; a further 22 programs provided a recommended structure (but did not require that the user follow this), and 3 programs were unstructured, allowing the user choice in how and when they completed different modules. Research evidence is inconclusive on the manner in which web-based materials are best organized . A recenMost resources were designed for adults generally, although a few included programs were designed for specific populations such as farmers, tertiary students, and expectant or new mothers. There were no programs identified in our searches that were designed for hard-to-reach groups such as culturally and linguistically diverse populations or men, and there was only 1 for people from an Aboriginal or Torres Strait Islander cultural background. Web-based interventions provide a useful avenue for administering targeted mental health support to populations who may face barriers to accessing traditional services for cultural reasons. These results highlight the need for more research and development of programs that meet the needs of these at-risk and underserved groups.The use of a scoring system to evaluate and compare programs was based on a framework for evaluating internet-based interventions , as adapAt the time of the literature search, there were no national standards for website developers and users to refer to when considering the selection of Australian web-based interventions for mental health. However, on November 30, 2020, the Australian Government released the National Safety and Quality Digital Mental Health Standards for health care providers to refer to when selecting digital media (including web-based programs) for the delivery of high-quality mental health care and suicide prevention (counseling or peer-based support) . The purtherapy or self-help. The purpose of broadening the search in this way was to mirror the type of search that a consumer is likely to perform (focused on the issue and not the intervention type).Overall, of the 52 programs, 9 (17%) addressed depression, 15 (29%) addressed anxiety, 13 (25%) addressed general mental well-being, and 13 (25%) addressed multiple issues. This is substantially lower than the 32 programs for depression and the 34 programs for anxiety identified by Renton et al , respectA total of 52 web-based programs and web-based programs with downloadable worksheets or resources programs are currently freely available to help Australians with the management of depression, anxiety, suicidal ideation, or general mental well-being. Careful attention is needed by clinicians and consumers to determine whether the interventions they refer to, or access, are evidence-based and considered acceptable by other users, given the varied levels of acceptability and impact. This review complements existing resources by providing website summaries and a clear comparison of website features to inform clinicians and consumers and assist in the selection of the most suitable program for the individual. It also identified important gaps in the availability of free web-based interventions in Australia , which may inform future research and program development initiatives.


Mentor Up is a technology training program designed to reduce loneliness through technology training and intergenerational relationships. The program, which has similarities to Cyber Seniors, has been held at Eckerd College for four years and has traditionally been held in-person at a local Continuing Care Retirement Community (CCRC). During the pandemic we partnered with AARP to re-think the program and offer a modified version of Mentor Up on Zoom. Five one-hour one-on-one virtual technology training sessions led by 16 students were scheduled. Participants joined to ask questions about how to use features on their smartphones or how to navigate the Zoom virtual platform more effectively. The remote nature of programming allows for expanded 2021 participation, including AARP members across Florida and all three Westminster Communities of Florida in the area . Participation rates, strategies to implement intergenerational programs on a virtual platform, and lessons learned will be highlighted.


People with intellectual and developmental disabilities are at increased health-related risk due to the COVID-19 pandemic. Virtual training programs that support providers in caring for the physical and mental health needs of this population, as well provide psychological support to the providers themselves, are needed during the pandemic.This paper describes the design, delivery, and evaluation of a virtual educational COVID-19–focused Extension for Community Healthcare Outcomes program to support providers during the COVID-19 pandemic in caring for the mental health of people with intellectual and developmental disabilities.A rapid design thinking approach was used to develop a 6-session program that incorporates mindfulness practice, a wellness check, COVID-19–related research and policy updates, a didactic presentation on a combination mental health and COVID-19 related topic, and a case-based discussion to encourage practical learning. We used the first 5 outcome levels of Moore’s evaluation framework—focusing on participation, satisfaction, learning, self-efficacy, and change in practice—which were rated (out of 5) by care providers from health and disability service sectors, as well as additional reflection measures about innovations to the program. Qualitative feedback from open-text responses from participants were analyzed using modified manifest content analysis.A total of 104 care providers from health and disability service sectors participated in the program. High levels of engagement (81 participants per session on average) and satisfaction  were observed. Self-efficacy (score improvement: 19.8%), support, and coping improved. Participants also rated the newly developed COVID-19 program and its innovative components highly. Open text feedback showed participants felt that the Extension for Community Healthcare Outcomes program expanded their knowledge and competency and created a sense of being part of a community of practice; provided value for the COVID-19 innovations; supported resource-sharing within and beyond program participants; and facilitated changes to participants’ approaches to client care in practice and increased participants’ confidence in supporting clients and families.The Extension for Community Healthcare Outcomes program is an effective model for capacity-building programs with a shared-learning approach. Future iterations should include targeted evaluation of long-term outcomes such as staff burnout. Countries across the world have responded to the COVID-19 pandemic with rapid deployment of public health measures and hospital-based care for the acutely unwell. Various population groups, including people with intellectual and developmental disabilities, are marginalized and underserved in the health care system, both during and prior to COVID-19 . The COVExperiences from previous pandemics suggest the need to support health care workers by increasing their mental health awareness and encouraging self-care . Health Virtual education programs can be used to overcome barriers to healthcare and social service provider training in a pandemic situation arising from physical distancing, quarantine, and other isolation measures . ExtensiGlobally, organizations have adapted the ECHO model for their respective COVID-19–specific needs. A recent study  reportedCOVID-19 was declared a pandemic on March 11, 2020. As a result of the pandemic, social service providers and health care providers who support people with intellectual and developmental disabilities faced new and unprecedented challenges in community settings. In response, we leveraged the ECHO model to develop ECHO Ontario Adult Intellectual and Developmental Disabilities: Mental Health in the Time of COVID-19 (ECHO AIDD-COVID), a targeted virtual education program to support care providers, working together, from the disability and health sectors. The purpose of the program was to share best practices in caring for the mental health of the intellectual and developmental disabilities population during the COVID-19 pandemic and to reduce feelings of isolation and burnout by making new connections and sharing resources. In this paper, we aim to describe the rapid development and evaluation of the ECHO AIDD-COVID program. We hypothesized that this program would improve participants’ self-efficacy in supporting and managing the mental health issues of people with intellectual and developmental disabilities during the COVID-19 pandemic.The rampant increase in COVID-19 infections in the winter of 2020, and the subsequent need to enhance and strengthen the skills of care workers, led to the rapid planning and development of a COVID-19–focused ECHO program. This program was an adaptation of an existing 12-session ECHO program launched prior to the COVID-19 pandemic, ECHO Ontario Adult Intellectual and Developmental Disabilities (ECHO Ontario AIDD), which focused on caring for the mental health of people with intellectual and developmental disabilities .A rapid design thinking approach  over 2 wThe inspiration stage—the problem or opportunity at hand—comprised the challenges created for providers caring for people with intellectual and developmental disabilities by the COVID-19 pandemic, such as the negative impact on mental health, for both clients and providers alike, and the need for rapid capacity building and connection in the community.This stage involved brainstorming and refining ideas and solutions. We met several times with project leads, hub members, and members of the ECHO program team to explore ways to leverage existing operational structures and the collective expertise of the multidisciplinary ECHO Ontario AIDD team members.In this stage, potential solutions were developed and shared with target users, who provided feedback. A prototype—a description used in rapid design framework to develop best possible solutions for the identified problems ,21—progrParticipation in the ECHO AIDD-COVID program was open to all care providers, both social service providers and health care providers, working with people with intellectual and developmental disabilities in Ontario, Canada. We recruited potential participants by emailing invitation flyers to all previous participants of ECHO programs at the Centre for Addiction and Mental Health, as well to developmental service agencies, community mental health organizations, professional accrediting colleges, and primary care sites in Ontario. Providers who were interested in participating completed a web-based application form. All applicants independently providing care were accepted in to the program.We assembled a hub team that comprised a psychiatrist, psychologist, primary care physician, behavior therapist, occupational therapist, nurse, social worker, and family advisor . The strengths of the team included expertise in primary care, mental health, and intellectual and developmental disabilities; experience working directly with people with intellectual and developmental disabilities the during COVID-19; and experience managing psychological distress.The curriculum was developed by triangulating sources, including feedback from a prior needs’ assessment , evaluatWeekly 1.5-hour-long sessions were conducted from April 17, 2020 to May 22, 2020 over 6 weeks, which is half the duration of ECHO Ontario AIDD. Each session included introductions, a mindfulness exercise led by the family advisor, a wellness check, COVID-19–related research and policy updates, a didactic presentation based on the curriculum topic for the day, and a case-based discussion, in which a participant (care provider) presented an anonymized case from their practice, for which they required support, to illustrate the complexities in caring for people with intellectual and developmental disabilities in conjunction with the impact of COVID-19. Additionally, a web-based ECHO AIDD-COVID resource portal, with reference materials related to the ECHO program, was available to the participants for use during and after the sessions. Sessions had a dual focus— participant skill development, for addressing the mental health issues faced by people with intellectual and developmental disabilities, and support for the psychological well-being of participants. The innovations described earlier were integrated in to the ECHO sessions seamlessly. Evidence-based resources were shared with participants and could be accessed after the course.Our evaluation strategy was informed by the Evaluation framework for continuing professional development, specifically, levels 1 to 5  . This frBasic participant demographic information  was collected throughout the duration of the program.Participant satisfaction was measured weekly using web-based postsession satisfaction questionnaires. Statements were rated on a 5-point Likert scale, from 1 (strongly disagree) to 5 (strongly agree), and focused on expanded knowledge and skills, reduced professional isolation, addressed learning needs, recommend session to others, and overall satisfaction. We obtained qualitative feedback with open-text responses to questions that asked for suggestions (for curriculum topics) and overall comments or feedback.Perceived self-efficacy was assessed for 4 core program competencies, with respect to providing care for people with intellectual and developmental disabilities during the COVID-19 pandemic, before and after program participation, with a previously established 100-point confidence scale (a higher number indicated higher confidence) . CompeteParticipants responded, using a binary scale , to a question that asked whether participation in the program resulted in a change in their practice; participants were also prompted to provide examples with open-text feedback.Feedback in this area was collected before and after the program with 2 items—having professional support and being equipped to cope with stressors  related to the pandemic—using a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree).Reflection questions about innovations to the program  were asked after the program, with responses captured using a 5-point Likert scale. Participants were also asked to comment on how participation in the ECHO AIDD-COVID program impacted the challenges that they experienced during the COVID-19 pandemic with open-text responses.Evaluation measures and data sets generated and analyzed in this report were reviewed and deemed to be part of program evaluation at the Centre for Addiction and Mental Health.t tests or Wilcoxon signed-rank tests, as appropriate. Statistical tests were 2-sided, with a statistical significance level of 5%.Quantitative data were analyzed using either Excel (Microsoft Inc) or SPSS software . Proportions, frequencies, and percentages were calculated for categorical variables . Means and standard deviations were calculated for continuous variables . Pre- and postprogram data about experiences during the COVID-19 pandemic and self-efficacy were analyzed using either paired Preliminary modified manifest content analysis was conducted using NVivo software  to evaluate open-text responses about program participation and the impact to challenges experienced during COVID-19. A project team member uploaded all open-text responses into NVivo, and then reviewed and performed open coding on all text. The project team met on a regular basis to review and discuss coding to develop and refine a coding matrix with definitions. This coding matrix was applied to all text; references for each code were reviewed, and frequencies were calculated. Finally, all codes were summarized and organized for interpretation ,32.A total of 104 care providers, with a variety of professional backgrounds, from 56 organizations  attendedWeekly satisfaction scores were high, ranging from a mean 4.07 (SD 0.18) to a mean 4.32 (SD 0.14); the overall mean satisfaction score was 4.31 SD 0.17; .41= –9.035, P<.001; d=1.394). Analysis of individual statements (core program competencies) also showed statistically significant improvements in mean self-efficacy scores . Statistically significant differences were also seen for pre-ECHO to post-ECHO mean scores for both statements on experiences with COVID-19 . In total, 42 participants completed both pre- and postprogram self-efficacy and experiences with the COVID-19 pandemic questionnaires. Mean self-efficacy scores prior to participation in ECHO were 61.3 (SD 18.2), and after the program, self-efficacy scores were 81.1 (SD 9.8); there was a statistically significant improvement . Almost all of the participants reported their learning needs were met in the program . The reflection questionnaire around prototype innovations in the program was completed by 53 participants. An overwhelming 98% of participants (52/53) agreed ECHO AIDD-COVID made them feel supported and part of a virtual community of practice. One participant commented that it wasso nice to know that we are not alone in this strange time and share the same challengesand that they wouldmiss this weekly touch point with professionals.A similar percentage of participants  reported COVID-19 updates and resources as valuable; a participant shared,there were innovative strategies suggested in each session as well as content in the presentations that I believe helped improve my approach day to day with the clients I have been supporting.Furthermore, 94% of participants (50/53) agreed or strongly agreed that having both interprofessional health care providers and social service providers enhanced their learning. Most participants  also reported that having a family member in the hub enhanced their learning. The impact of the family perspective is best illustrated by a participant who shared that...the most powerful experiences I had was whenever the family member spoke. I think we all can hypothetically understand caregivers’ perspectives, but we cannot understand the full emotional toll or the personal thoughts and worries that caregivers have.A similar percentage  of participants appreciated the opportunity to share strategies in the community. The weekly mindfulness exercise led by the family advisor was reported to be helpful by 77% of the participants (41/53) and functioned toremind us to take care of ourselves.We described the successful development and evaluation of a COVID-19–focused ECHO program for workers caring for people with intellectual and developmental disabilities. A rapid design thinking approach was used to develop the ECHO AIDD-COVID program. Evaluation findings showed high levels of engagement and satisfaction with the program, with the majority of participants reporting changing their practice because of the program. To the best of our knowledge, this paper is the first to document use of the ECHO model and its significant improvement in perceived self-efficacy in caring for people with intellectual and developmental disabilities during the COVID-19 pandemic for a cohort that encompassed both social service providers and health care providers. Improved confidence in all the core program competencies shows the ECHO model is an effective way to improve provider skills for supporting the mental health needs of people with intellectual and developmental disabilities. Additionally, there was a cascading effect from knowledge shared by participants beyond the program—use of the program can be an effective share-and-spread strategy during the pandemic. Participant feedback from open-text responses shows that the program helped expand knowledge and facilitated changes in practice. The ECHO model was conceptually designed to develop a community of practice. This was validated by the qualitative feedback from participants. The results of the pre- and postprogram evaluations suggest that participants felt that group participation helped to support their own well-being, especially their ability to cope with COVID-19 stressors.Because of the pandemic, uncertainty and unprecedented challenges arose in social support and health care sectors, and there was a need for the swift deployment of a capacity-building program to support the needs of people with intellectual and developmental disabilities and those of the health and social service providers who work with this underserved population. The rapid design thinking framework was instrumental in incorporating lessons learned from our previous ECHO capacity-building project , to adapFeeling connected” and “we are not alone” came out strongly in the reflections, which were indicators of the program bringing professionals from disability and health services sectors together as a community. Sockalingam et al [In contrast to other ECHOs designed to strictly focus on working with a population  or addream et al  concludeam et al -40, the am et al ,42.There are some limitations to consider when reviewing our findings. First, the data and measures used in this evaluation are from a single cycle of an ECHO program that was specifically focused on mental health in intellectual and developmental disabilities populations in Ontario during COVID-19; as such, our findings may not be generalizable to other cycles or settings. Future evaluation and research efforts will seek to replicate these findings with other health conditions and settings. Additionally, findings for satisfaction, changes in confidence, and experience with COVID-19 participation were informed by data collected from individuals who completed satisfaction surveys and questionnaires. This may introduce a response bias, whereby those who participated in these data collection activities may have been more engaged and likely to respond with higher scores; however, we recognize this challenge is not unique to our evaluation and exists for anyone collecting data via surveys.expert patients as educators in the ECHO model needs further exploration.Future iterations should incorporate targeted outcome measures to evaluate the role of ECHO in addressing the mental health needs of care workers who are supporting people with intellectual and developmental disabilities in the community . The virAlthough this study addressed self-efficacy and the competency of learners, focus on retention and its medium- to long-term effects would be helpful for designing future programs. In addition, the evaluation of implementation outcomes would be helpful to understand the impact of this educational intervention. Ethical implications and dynamics involved in coproducing educational content are important future considerations . We will


The rise of major complex public health problems, such as vaccination hesitancy and access to vaccination, requires innovative, open, and transdisciplinary approaches. Yet, institutional silos and lack of participation on the part of nonacademic citizens in the design of solutions hamper efforts to meet these challenges. Against this background, new solutions have been explored, with participatory research, citizen science, hackathons, and challenge-based approaches being applied in the context of public health.Our aim was to develop a program for creating citizen science and open innovation projects that address the contemporary challenges of vaccination in France and around the globe.We designed and implemented Co-Immune, a program created to tackle the question of vaccination hesitancy and access to vaccination through an online and offline challenge-based open innovation approach. The program was run on the open science platform Just One Giant Lab.Over a 6-month period, the Co-Immune program gathered 234 participants of diverse backgrounds and 13 partners from the public and private sectors. The program comprised 10 events to facilitate the creation of 20 new projects, as well as the continuation of two existing projects, to address the issues of vaccination hesitancy and access, ranging from app development and data mining to analysis and game design. In an open framework, the projects made their data, code, and solutions publicly available.Co-Immune highlights how open innovation approaches and online platforms can help to gather and coordinate noninstitutional communities in a rapid, distributed, and global way toward solving public health issues. Such initiatives can lead to the production and transfer of knowledge, creating novel solutions in the public health sector. The example of Co-Immune contributes to paving the way for organizations and individuals to collaboratively tackle future global challenges. As the world faces a rise in the number of complex challenges that threaten the resilience of our economic, environmental, social, and health systems, we observe a shift toward more collaboration and openness in the way science and innovation is performed -3, bringImmunization is one of the most cost-effective interventions to protect oneself and others from infectious diseases  and saveYet, the annual death toll for vaccine-preventable diseases stands at 1.5 million, and large gaps in coverage persist, not only between countries but also within their territories . In partIn response, the WHO Global Vaccine Action Plan 2011-2020  committeGlobal health guidelines showcase the positive outcomes of social participation for universal health coverage , which iCitizen science is an emerging and highly diverse practice that can be broadly defined as the general public being involved in the process of doing research . ResearcOther approaches to create and develop knowledge and solutions to complex challenges are slowly entering the mainstream. In particular, hackathons, challenge-based approaches, and the participation of citizens in science have been flourishing over the last two decades , especiaHackathons are short, intensive, and collaborative events that are designed to prototype solutions addressing a specific problem. They originated in the early 2000s in digital and tech fields and have been adapted to address more complex challenges in global health -40. SuchChallenge-based approaches, which provide frameworks for learning while solving real-world issues, have also been on the rise in global health and have proven to be efficient for generating innovative solutions and for incentivizing mass community engagement . For exaDespite the numerous tools and technologies created to facilitate collaboration in citizen science projects, challenges remain. These include the issues of the complementarity, coherence, and diffusion of these initiatives  to efficTherefore, the promotion of transdisciplinarity and citizen science in an open innovation framework, coupled with methods such as hackathons, and a challenge-based approach represent an opportunity to address current complex challenges of vaccination that would overcome the limits of either solution alone. In this paper, we describe the design, implementation, and outputs of Co-Immune, a collaborative open innovation program that was run in 2019 to address vaccination hesitancy and access to vaccination.Co-Immune’s aim was to develop an environment that favors the creation and development of citizen science and open innovation projects addressing the contemporary challenges of vaccination in France and around the globe. This program had four specific objectives: (1) to foster a collaborative, open, and transdisciplinary dynamic; (2) to promote the emergence of accessible knowledge and innovative solutions; (3) to support participants in the elaboration and development of their project; and (4) to disseminate the outputs and results in an open science framework. In this study, we describe the methodology of Co-Immune and its implementation, and we present its key outcomes.The overall program duration was 10 months (March 2019 to January 2020), divided into 6 months of preparation and 4 months of rollout of activities that included offline and online events, support for the development of citizen science projects, and assessment and awards for projects participating in the challenge-based competition. The main outputs of the program were projects, categorized as leading to (1) knowledge production, if they performed data analysis or generated new knowledge, whether it was context specific, generic , or knowCo-Immune was coordinated online through the JOGL platform  and suppThe governance of Co-Immune was designed to provide freedom for projects to develop innovative solutions while ensuring their compliance with local and international regulations and consideration of ethical and scientific integrity. To this end, we constituted the independent Committee for Ethics, Science and Impact (CESI), which issued an opinion on the rules of participation in the program and validated the strategic orientation of the program. Public health priorities were identified based on a literature review and divided between two main challenges to streamline participants’ work: vaccination coverage and vaccination hesitancy. They were then validated by the CESI. In addition, through a series of semistructured interviews, experts at the 7th Fondation Merieux Vaccine Acceptance conference  identifiTo be eligible for a prize, a project was required to have created a comprehensive description of their initiative on the JOGL platform and a video pitch. This material was provided to experts in charge of the assessment.Participants were recruited through our network of partners from around the globe and social media communication. Participation was open to everyone above the age of 18 years, if they had agreed to follow the participation rules validated by the CESI. Participants could take the role of “project leader” or “contributor.”Co-Immune participants used the JOGL platform to document their projects and recruit collaborators throughout the course of the program. JOGL is a decentralized mobilization platform designed for use in collaborative research and innovation . Within The Co-Immune program was realized through an interrelated and interacting set of technological and social features . Our cooTo build the community, we contacted organizations involved in a wide range of domains before the launch of the program, thereby creating a first pool of contributing professionals and students. We also recruited participants via the organization of events, typically in the evening, aimed at creating projects, fostering collaboration among participants to address project needs, and providing mentorship. To facilitate the coordination of the community, all participants were required to use the JOGL platform to describe their projects, form teams, list their needs, and initiate collaboration.In order to create a supportive and collaborative environment for the participants, we reached out to various organizations to establish partnerships. Our intention was two-fold: (1) to facilitate the participation of the organizations’ students and employees as participants or mentors by involving their institution and (2) to enhance the sustainability of projects after the course of the program by connecting them with potential partners at the early stage of their development.The 13 partners operated in the health, technology, and social sectors, and included research, innovation, and education organizations, as well as professional networks, incubators, and communication specialists . The numWe organized 10 offline and online events between October and December 2019 . ParticiThe facilitation of the hackathon-style events relied on the use of participatory and collective intelligence design and problem-solving techniques . In partThree partners in Paris—Epitech, the Wild Code School, and the Center for Research and Interdisciplinarity (CRI)—co-organized and hosted events for their students, respectively, in their engineering, coding, and life science and education schools. Other partners—Kap Code, Excelya, and CorrelAid—mobilized their teams to act as mentors during these events. A total of 14 mentors attended events, and five came to more than one event.In addition, we organized four 1-hour online events. The first was an opportunity to share information about Co-Immune with people around the globe. Another event discussed best practices to document open science projects. Finally, two events focused on the resolution of needs of single projects  declared their job category. The community was composed of a mix of students  and workers , most of whom worked full time . We observed a high representation of data science and coding alongside biology, which, altogether, related to the technical skills emphasized during the program . The skiA total of 22 projects were created by 20 project leads, with teams of up to 11 members  aimed at contributing to the production of knowledge by facilitating the analysis of publicly available data; they did this via the use of parsing tools and the creation of repositories (n=3), the analysis of open data (n=3), the development of machine learning tools to extract and analyze Twitter data related to vaccination hesitancy (n=2), and the production of data visualizations (n=3). In particular, more than 40 data sets were identified and collected by 4 projects that were created during the data-centered events. In addition, a database of 2464 tweets, in French, posted over a period of 7 years was made available by a partner, and another data set of 89,979 tweets was gathered by the project Qualitative Analysis of Tweets on Vaccination .Out of the 15 projects above, 4 (27%) used software for knowledge transfer; for instance, the HERA  project  providedA total of 5 projects out of 22 (23%) focused on different interventions , includiOut of 22 projects, 7 (32%) provided sufficient documentation on JOGL to be assessed by the pool of independent experts. In total, 27 reviews were performed, yielding scores ranging from 18 to 32.8 out of a possible total of 45 across the different dimensions that were assessed . The average score was 25.1 (SD 6.4).HERA: A Health Platform for Refugees  was awarThe projects were more successful, globally, in terms of approach, with a mean score of 9.37 (SD 1.79) out of 15 points. Out of 7 projects that were assessed, 4 57%;  had a scThe implementation strategy score of projects was low, overall, given the early stage of the projects at the time of review. As such, only projects that existed prior to the program—HERA  and HEROFor winners in each category, JOGL awarded them physical space for showcasing their project during the 2020 ChangeNOW forum at the Grand Palais in Paris as well as tickets for the Maddy Keynote, a major innovation event in Paris. Two partners—Excelya and the Wild Code School—also provided awards to the projects of their choice. Additionally, the Qualitative Analysis of Tweets on Vaccination  project The Co-Immune program was designed to foster the creation and development of citizen science and open innovation projects addressing the contemporary challenges of vaccination in France and around the globe by reaching four specific objectives: (1) to foster collaborative, open, and transdisciplinary dynamics; (2) to promote the emergence of accessible knowledge and innovative solutions; (3) to support participants in the elaboration and development of their projects; and (4) to disseminate the outputs and results in an open science framework. Below, we discuss to what extent Co-Immune reached these objectives and highlight the challenges and facilitators in implementing such a program.First, the program succeeded in creating a collaborative and transdisciplinary environment through its three core features: the JOGL platform, the organization of events, and the contest approach. This led to forming partnerships with 13 different organizations and recruiting over 230 participants, who displayed 492 unique skills and were engaged in creating 22 projects. The use of on-site hackathons was beneficial in gathering nonacademic participants from various backgrounds. Our data show that in-person events and local outreach played a significant role in growing the community around Co-Immune. These offline events recruited 45% of the total community. Local enrollment was further strengthened by local partnerships, such as higher education organizations. However, the localization of our on-site events in Paris did not allow for the participation of people living in other parts of France or the rest of the world. Additionally, our online communication restricted the access of the online events to our realm of influence and to people with an internet connection. More inclusive participation geared toward people with diverse socioeconomic statuses and geographic situations is desirable in the future to give them agency over solving the problems that affect them. The development of new communities is usually a slow process in the absence of exogenous shocks, such as the surge in collaborative communities created by the COVID-19 pandemic . TappingSecond, two design elements of the program converged to promote the emergence of knowledge and solutions to address aspects of access to vaccines and vaccination hesitancy: (1) the identification of challenges by experts in the field and (2) the alignment of the program strategy with national and international policies by frequent consultation with public health bodies and mobilization of members of public institutions in the CESI. Yet, greater representation of people affected by poor access to vaccines and people who are hesitant would be desirable to strengthen the alignment between the solutions developed and the most pressing needs at the local level.Recently, online events have been used widely during the COVID-19 pandemic -78, suppThird, the use of the JOGL platform, the mentorship during events, the assessment and feedback from experts, and the connection with a wide range of partners supported participants in the elaboration of their project in an efficient way. The use of the JOGL platform enabled projects to gain visibility, list their needs to create interfaces for collaboration, and share open data sets, code, and tools. Indeed, online platforms can offer projects that started at hackathons a pathway to pursue their development, potentially alleviating one of the main drawbacks of such short temporal interventions . In thisFinally, the open science environment of this program was not only an asset for disseminating the outputs and results of the projects developed, but it also enabled them to replicate initiatives and, thereby, accelerate the resolution of the global health challenges they address. An example of this was given by the team from the project HERA: A Health Platform for Refugees , who opeCo-Immune showcases that short, focused programs can be efficient at mobilizing diverse communities in a rapid manner and harvesting ideas from various domains to address global health challenges. Yet, more case studies and evaluation work on similar programs are necessary to assess the full relevance of their design and the impact of the projects that are developed within them.Co-Immune highlights how open innovation approaches and online platforms can help to gather and coordinate noninstitutional communities in a rapid, distributed, and global way toward solving SDG-related issues. The Co-Immune program gathered participants and partners from various backgrounds in a newly formed community to facilitate the creation of new projects as well as the continuation of existing projects to address the issues of vaccination hesitancy and access. In an open framework, the projects made their data, code, and solutions publicly available.Through hackathons and other contest approaches, such initiatives can lead to the production and transfer of knowledge, creating novel solutions in the public health sector. The example of Co-Immune contributes to paving the way for organizations and individuals to collaboratively tackle future global challenges.