Background: Family Integrated Care (FICare) integrates parents in the direct care of their child while the healthcare personnel act as teachers and guides. To this date, most reports on the feasibility of this model refer to stable preterm infants admitted to Neonatal Intensive Care Units (NICUs).Objectives: To scale up and adapt FICare to make it suitable in level IIIC NICUs, which care for extreme prematurity and other complex medical or surgical neonatal conditions.Materials and Methods: Step 1 was the creation of the FICare implementation team (FICare-IT) and baseline analysis of current procedures for critical care to identify needs, wishes, and requirements; we aimed for protocol elaboration tailored to our cultural, architectural, and clinical context (March 2017 to April 2018). Step 2 as a dissemination strategy by FICare-IT acting as primary trainers and mentors to ensure the education of 90% of nursing staff (May 2018 to July 2018). Step 3 involved piloting and evaluation with the aim to refine the procedure (July 2018 to December 2020).Results: A rigorous but flexible protocol was edited. The FICare educational manual included two curricula: for healthcare professionals/staff (Training the trainers) and for families (Education of caregivers), the latter being categorized in two intervention levels (basic and advanced), depending on the infant care needs and parent's decision. In total, 76 families and 91 infants  were enrolled in the pilot. No differences in acceptance rate  or in the number of infant-family dyads in the program per month were observed when considering the pre- and post-Covid-19 pandemic periods. All families, except for one who dropped out of the program, completed the agreed individualized training. Mothers spent more time in NICU than fathers (p < 0.05); uninterrupted time spent by mothers in NICU was longer during the pre-pandemic period (p < 0.01). Observed time to reach proficiency by task was within the expected time in 70% of the program contents. The parents revealed educational manuals, workshops, and cot-side teaching sessions as essential for their training, and 100% said they would accept entry into the FICare program again.Conclusions: The principles of the FICare model are suitable for all levels of care in NICUs. Leadership and continuous evaluation/refinement of implementation procedures are essential components to achieve the objectives. Extremely low gestational age neonates are at risk of developing short and long-term complications that can alter their life course –5. This Family Integrated Care (FICare) is trademarked by Mount Sinai Hospital in Toronto, and it includes a four-pillar model of care proposed to foster feelings of self-confidence and competence regarding interaction with the child and the ability to be involved in their upbringing , 19–23. Given the benefits already described, our purpose was to scale up the FICare model by offering two implementation levels (basic and advanced), making it suitable across the whole spectrum of care of the high-risk neonate.The present report describes the FICare model implementation strategy used in our clinical setting, a level IIIC NICU, and evaluates the methodology followed and the effort used for its development. As one of the most determining elements of the success of the program is the uninterrupted time spent by parents in the NICU, a special focus was placed on analyzing the impact that the Covid-19 pandemic had on the workplan. In addition, at the time the project started, there were not single-family room facilities in our NICU, so that infants remained in open-ward rooms gathering a variable (from 5 to 8) number of patients.The Department of Neonatology at La Paz University Hospital holds the largest NICU in the Madrid region, and it is one of the largest in Spain. The maternal and child hospital covers a population of approximately 600,000 inhabitants, attends 6,000 deliveries per year, and is a national referral center for fetal surgery, cardiovascular surgery, extracorporeal membrane oxygenation (ECMO), or treatment of retinopathy of prematurity, among other complex processes. Of a total of 67 beds, the NICU has 24 beds for infants on mechanical ventilation plus 15 additional beds for highly dependent non-ventilated infants. More than 200 nurses and nursing assistants (33 per shift) work in the department. There is also active teaching activity among the staff that includes nurse residents (18/year) and students (16/year). The average annual activity at the Department of Neonatology is above 1,500 admissions, and 3,600 visits at the outpatient clinic. The NICU attends all kinds of medical and surgical diseases, has its own Human Milk Bank facilities, and a Royal McDonald House Charities-Family Room.Our NICU policies allow parents to remain at their infant's cot-side 24 h around the clock. However, the visitors' policies changed during the lockdown period imposed by the COVID-19 pandemic. Accordingly, from mid-March to the end of June 2020 (first wave lockdown), only one visitor per child was allowed to stay for a maximum of 2 consecutive hours at the hospital; from July 2020 to the present, no time restriction was imposed but the number of visitors per infant remained unchanged. Most of the attended population at our NICU is Caucasian or American-Hispanic, with a low proportion of other cultures/ethnicities (mainly Muslims from Morocco).The project leader (AP) conceptualized the idea and elaborated a step-wise workplan .The first step was to create the FICare implementation team (FICare-IT). The FICare-IT gathered members of the local associations of veteran parents and included a variety of professional  and non-professional partners in addition to NICU staff (10% of the nursing staff and four neonatologists). A FICare clinical staff co-ordinator was nominated (MTM). The FICare program development and implementation calendar was agreed upon by the FICare-IT in March 2017 .FICare-IT conducted an analysis of current procedures for critical care to identify needs, wishes, and requirements. As a result of the analysis, the following components of the FICare program to be implemented were defined: the educational component for staff (nurses and doctors) and family primary caregivers to support their roles; psychological support for parents, with a leading role for veteran parents pertaining to the FICare-IT, or individual interventions by hospital psychologist if requested; and the physical support, which included comfortable cot-side chairs and breast-pumps as well as the facilities provided by the Ronald McDonald Family Room . The procedures and main topics to be covered in the FICare training program, ensuring the diverse needs and challenges were properly covered; the specifications of the tools and materials required for training and for pilot's data gathering; and the questionnaires for experience and psychological data gathering were also decided. Expected deliverables were the FICare implementation protocol  and the FICare teaching material.The second step was the dissemination of the FICare program and training on FICare standards among the NICU clinical staff. The FICare-IT at this stage acted as primary trainers and mentors to ensure that the provision of nurse education covered 90% of active nursing staff in the NICU, as recommended . Groups The third step was piloting and evaluation of the FICare program before opening to all potential beneficiaries ; the latter was scheduled for when the new NICU with single-family room facilities opens. Serial FICare-IT meetings were held to evaluate, refine and enrich the preliminary protocol. Barriers and other unforeseen difficulties that were found after a running-in period were taken into account for these purposes.p < 0.05.The data were analyzed with the statistical program SAS 9.3 . Qualitative data are expressed as counts and percentages and quantitative data as mean (SD) or medians (IQR). The pre- and post-Covid-19 pandemic periods during the pilot were defined as July 2018 to March 2020 (21 months) and May 2020 to December 2020 (8 months), respectively. Enrolment during April 2020 was stopped due to lockdown constraints. Non-parametric tests (Mann-Whitney test and Kolmogorov-Smirnov) were used for comparisons. The values were considered statistically significant with A rigorous but flexible protocol tailored to the cultural, architectural, and clinical context of our NICU, with two implementation levels (basic and advanced), was edited and approved by the Research Ethics Committee of La Paz University Hospital. The scope of the program covered preterm and term neonates (and their families), admitted to NICU due to complex congenital or acquired diseases or immaturity-related issues, for which a long length of stay was anticipated , providing all inclusion criteria were fulfilled and none of the exclusion criteria applied .A strategy for recruitment was established to avoid overloading parents with information during the critical period after birth when they are most vulnerable. Whenever possible, preliminary information about the FICare program was distributed to parents of potential candidates antenatally. Perinatal committees, where complicated pregnancies are discussed among obstetricians and neonatologists, were the ideal forum to comment on potential candidates. An appointment with eligible families was scheduled according to the information gathered in these meetings. In the case antenatal information was not provided, families were approached after NICU admission. Once the families accepted to participate in the FICare program and signed the informed consent, the training tools were provided. A maximum of four infant-family dyads on FICare at a given time was proposed for the pilot. This policy was changed after a 6-month running period of the program when a maximum number accepted was scaled up to 10 infant-family dyads.The FICare training program was divided into two curricula: for healthcare professionals/staff (Training the trainers) and for families (Education of caregivers), the latter being categorized in two intervention levels (basic and advanced) depending on the infant care needs and parent's decision . The folTraining the trainers: (1) understanding the boundaries of the FICare model and how to promote FICare among families; (2) psychosocial needs of families  and communication skills (assertive communication); (3) how to involve families in NICU ; and (4) professional self-care (burnout and compassion fatigue).Education of caregivers: (1) comprehensive description of the FICare model (the strengths and training methodology) and the functional and architectural structure of the NICU; (2) family self-care ; (3) learning about infants' neurobehavior, stress, and pain; (4) taking part in baby care (basic level), where parents will be “professionalized” to become the first-line care providers of their children; (5) taking part in baby care (advanced level), specific task's training for infants who require even more specialist care; and (6) parents will be prepared for home, and a map of the social resources available at the local setting will be provided.The FICare training tools included an educational manual  on all tA teacher education system started in May 2018. The Chief Nurse and FICare co-ordinator defined the mentor-assigned trainee groups and the calendar. Face-to-face meetings took place for 2 months. By June 2018, the program and procedures were disseminated among 90% of the NICU staff  to become facilitators of FICare. In these meetings, mentors particularly focused on the relevance of training harmonization and the requested adherence to the contents and procedures as described in the educational manual . MentorsIndividualized theoretical and practical learning by tasks: two family caregivers per family were trained through cot-side face-to-face sessions, following an individualized teaching plan adjusted to the baby's clinical condition and the wish of the parents. Once proficiency was fully accredited in a given task, the family caregiver was certified by the training nurse (and registered) and was allowed to do this task autonomously.Workshops: family caregivers were invited to attend 45-min open sessions on relevant topics of the learning contents, to express their doubts and concerns, as well as to share their experiences with other families. Three meetings per week were programmed; selected topics were sequentially repeated every 4 weeks.Registry of teaching activities and task certifications in the corresponding logbook.The family training process relayed on three cornerstones that were agreed upon between mentors and the staff facilitators:In February 2020 a FICare workshop for all the NICU staff (doctors and nurses/nurse assistants) was organized to refresh the knowledge and share the experiences lived during the previous months of piloting.n = 68, 74.7%), congenital heart defects , a variety of genetic syndromes , congenital diaphragmatic hernia assisted by extracorporeal membrane oxygenation (ECMO) , kidney dysplasia , esophageal atresia , and ileal atresia . Relevant clinical data of the infants included are displayed in From July 2018 to December 2020, 88 families who fulfilled the FICare program entry criteria were approached ; we saw an 86.4% acceptance rate , resulting in a total of 76 parents and 91 infants being enrolled in the pilot. The main neonatal diagnoses of the participating infants were as follows: prematurity (–5) and 4 (2.25–5.75) for the pre- and post-COVID-19 pandemic periods, respectively (p = 0.237). Systematic surveillance of uninterrupted daily time spent by parents in the NICU started in January 2020. The observed pre-pandemic period revealed that the median time for mothers was 10 (7–10.5) h, while the median time for fathers was 7 (5.5–10) h. During the post-pandemic period, the median uninterrupted time in the NICU was 7 (4.5–8) h and 5 (4–8) h for mothers and fathers, respectively. Mothers spent more time in the NICU than fathers irrespective of the period that was observed (p < 0.05). Uninterrupted time spent in the NICU differed between periods, being significantly longer for mothers during the pre-pandemic period (p < 0.01).The median number of infant-family dyads included in the program per month was 3 . The main argument given by parents to decline workshop participation was the coincidence with the infants' feeding time.The results on the satisfaction questionnaire about the FICare program for families are depicted in Our perception is that the program helped us to foster discharge; in complex cases, such as the care of our daughter, this training enable parents to feel ready to care (for) the baby and manage all the devices the baby needs, such as oxygen, home monitoring, or feeding by tube. We have also learned how to deal (with) and respond to “scares”; we are now prepared to keep calm and react appropriately when needed. Hav(ing) learned when it is time to ask for help is a great achievement.”“Another important issue that parents refer to is decreased stress and anxiety as they are really aware pf their baby and his/her context:Knowledge is power. Knowing what our babies are like and how to address their needs was essential to reduce the stress and anxiety we had due to the entrance of our twins; it has even allowed us to enjoy this stage that we have had to go (sic).”“As a final point of the parents' evaluation, they were asked whether they would accept to enter the FICare program again, to which all the families were aligned:My answer is yes, a thousand times yes. This program was determinant to learn how to deliver the care that my baby needs (sic). The program allowed us to behave as real parents and not mere observers of our baby's life.”“During the pilot, weekly FICare rounds were held by at least three members of the FICare-IT. During these rounds, direct contact with families currently involved in the training was established. Parents were invited to report any queries, doubts, or complaints about the program. Teaching notebooks and certification registries were collated, and adjustments to the individual teaching plan were carried out. The feedback on the individual progress reported by either parents or professionals involved in the training was used to adjust and refine the workplan to overcome project barriers and strengthen dissemination .This is the first study reporting on the feasibility of FICare model implementation in a complex, level IIIC NICU, that gathers surgical and non-surgical processes involving both, the preterm and term infant. The scaling up of the FICare model explored in this work included two levels of care: training parents in the basic tasks, as reported previously –31, and Another important achievement of this workplan was developing and implementing the FICare model in an a priori unfavorable architectural configuration of the NICU, as we had no single-family room for NICU admission at the time the project started. Although not a pre-requisite, single-family rooms positively influence the expected FICare health outcomes . This kiThe experience gained during the 30 months of running the project allowed the FICare-IT to make sequential adjustments in procedures to overcome difficulties and barriers that were found. For instance, in the beginning, a maximum of four infant-family dyads on FICare at a given time was proposed: a decision based on avoiding nurse work overload and guaranteeing that the family training and accreditation by task was correct. We also thought that it was easier to group FICare patients in the same ward if the number of families being trained at a time was smaller. As time went by, we realized that this approach was wrong because, even if intended, the reality was that, for different reasons, patients were not “grouped.” In addition, it was realized that many of the teaching activities had more learning inertia if there were more people in the program at any given time. Consequently, the FICare-IT decided to increase the maximum number of infant-family dyads for training by 3-fold.The COVID-19 lockdown period imposed new visitors' policies at our hospital, limiting the number and the time allowed for visits. In spite of that, enrolment in the pilot was halted only during the peak of the pandemic in our region (April 2020), while training of parents who already were participating in the program continued as scheduled. In fact, no differences in the enrolment rates were observed when the pre- (91.2%) and post-pandemic (77.4%) periods were compared, with a similar number of infant-family dyads in the program per month. Although mothers spent more time in NICU than fathers, irrespective of the period that was observed, our understanding is that the revised visitors' policies during the post-pandemic period justify the larger uninterrupted time spent in NICU by the mothers observed during the pre-pandemic period.FICare represents a change in people that involve professionals and families as main actors influencing their attitudes, skills, expectations, perceptions, and behaviors. The participation of all health personnel is critical for a successful implementation of a project that promotes a change in the working method and in the established roles. To do so, thorough project planning is essential and should include the development of theoretical and practical support, as well as a good dissemination strategy within the NICU staff. With regards to the first cornerstone, no remarks were raised among the staff regarding the teaching material contents, as expected, because in the educational manual  all chapPlanning the time needed for task certification is critical, as it is an indicator of the nurse effort required and the moment when parents are allowed to do this task autonomously. We had no a priori experience for this, and our predictions were thus based on the theoretical complexity of the task or how frequently it was to be practiced . In 70% The way to deliver training to the parents by the nurse staff was acknowledged in general. However, attendance at workshops was much lower than expected. We think this is a weakness of our working plan, as we envisage these open interactive sessions as an opportunity to create good dynamics: a way to heal scars and strengthen knowledge. The justification given by parents was, in general, unsuitable timetables because the sessions were held in a separate room and not at their baby's cot-side. Moving to the admission wards for this purpose prevents gathering a sound number of parents at a given time, and although interaction may be improved (like intimate relations between neighbors), the reaching is lower. The parental feelings about our FICare program, in general, were very positive, and all parents would accept to participate again. Individual comments/remarks were intended to be addressed during the weekly FICare rounds routinely held by members of the FICare-IT. However, it is possible that the items included in the parents' satisfaction questionnaire were leading the participants' responses to some extent.In summary, this report describes the procedures followed to develop and implement the FICare model at a complex, level IIIC NICU that cares for all kinds of medical and surgical neonatal processes. The adapted protocol scales up the FICare model, introducing two implementation levels: basic (routine tasks addressed in previous FICare experiences) and advanced . Our results support that FICare implementation is feasible in this context, even when facing two important adverse conditions: absence of single-family room facilities for hospitalization during the prolonged NICU stay and strict visitors' restrictive policies during the Covid-19 pandemic. Even in the case of thorough planning, continuous follow-up of the protocol procedures and eventual adjustments and adaptations according to the current conditions to solve difficulties raised by both professionals and families are key elements for project success. A limitation of this study is the lack of a systematic assessment of the professional perception of the model. Future studies should address the impact of this adapted scaled-up FICare model on the infants' outcome, the parental stress profiles, the professionals' viewpoint, and the health system economy.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.The studies involving human participants were reviewed and approved by Ethics Committee for Human Studies at La Paz University Hospital. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.BM-S participated in the development of training material and FICare dissemination-implementation strategy among NICU staff, conceptualized and designed the pilot, and participated in recruitment and data analysis, drafted the initial manuscript, and approved the final manuscript as submitted. MM and MA participated in the development of training material and FICare dissemination-implementation strategy among NICU staff, participated in recruitment and training during the pilot, reviewed the manuscript, and approved the final manuscript as submitted. MS and MC participated in the development of training material and FICare dissemination-implementation strategy among NICU staff, reviewed the manuscript, and approved the final manuscript as submitted. AP conceptualized and designed the workplan and participated in the development of training material and FICare dissemination-implementation strategy among NICU staff, conceptualized and designed the pilot, drafted the initial manuscript, and approved the final manuscript as submitted. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Inappropriate antimicrobial prescribing contributes to increasing antimicrobial resistance. An antimicrobial stewardship (AMS) program in the form of quality improvement activities that included audit and feedback, clinical decision support and education was developed to help optimise prescribing in general practice. The aim of this study was to evaluate the implementation of this program (Guidance GP) in three general practices in Melbourne, Australia, between November 2019 and August 2020. Thirty-one general practitioners (GPs) participated in the program, with 11 GPs and three practice managers participating in follow-up focus groups and interviews to explore the acceptability and feasibility of the program. Our findings showed that the quality improvement activities were acceptable to GPs, if they accurately fit GPs’ decision-making process and workflow. It was also important that they provided clinically meaningful information in the form of audit and feedback to GPs. The time needed to coordinate the program, and costs to implement the program were some of the potential barriers identified. Facilitators of success were a “whole of practice” approach with enthusiastic GPs and practice staff, and an identified practice champion. The findings of this research will inform implementation strategies for both the Guidance GP program and AMS programs more broadly in Australian general practice, which will be critical for general practice participation and engagement.  There is global concern about inappropriate prescribing of antimicrobials as this can drive antimicrobial resistance and adverse events including allergic reactions, side-effects and altered microbiome without clinical benefit ,2,3,4. TInterventions such as audit and feedback, clinical decision support, and education, particularly where these interventions are electronically delivered, integrated into practice workflow, provide automated clinical decision support, and stimulate patient discussion have been showed to have effects in reducing antibiotic prescribing in general practice ,7,8,9. WOur previous work has explored general practitioners’ (GPs) use of electronic medical records (EMRs) and embedded electronic guidelines to inform decisions around antimicrobial prescribing . Our simThis program of work resulted in the development of Guidance GP, a pilot quality improvement (QI) program consisting of a clinical decision support tool to facilitate access to Therapeutic Guidelines (eTG) , which aA total of 31 GPs from three metropolitan Melbourne practices (practice size range from five to 25 GPs) participated in the pilot Guidance GP program from November 2019 to August 2020. Of the GPs who participated in the program, 11 participated in three focus groups at the end of the study period (participant number ranged from 2 to 6 in each focus group). The mean GP experience was 17.3 years, with a median of 18 years. Three practice managers, one from each practice, participated in a semi-structured interview. Demographic details of participants are summarised in At the baseline GP NAPS audit, there were 231 antibiotic prescriptions recorded in the two-week period between 14 and 17 November 2019. At the follow-up GP NAPS audit from 17 to 31 August 2020, there were 73 total antibiotics prescribed across the three participating practices. Skin and soft tissue infection was the most common indication pre- and post-intervention; amoxicillin and amoxicillin-clavulanic acid were the top prescribed antibiotics pre- and post-intervention, respectively see .The CDS tool was scheduled to be installed across the three participating practices in early February 2020. However, initial technical issues included testing and implementation of the CDS tool in each practice, followed by the COVID-19 pandemic, restricted the implementation of this part of the program into practices. These barriers were discussed in the GP focus groups and PM interviews.Themes were grouped within the Clinical Performance Feedback Intervention Theory (CP-FIT) framework into feedback variables, recipient variables and context variables . We elabGPs viewed AMS as important, and receiving audit results and feedback assisted their goal setting and reflection on their practice was considered acceptable.“…you realise, ‘Well there’s some things that I’m not going to be able to change, but within these contexts… these are the things that I’m going to target in terms of my own reflective practice’….” GP4However, GPs would prefer not to be told what to prescribe, emphasizing that GP performance should not only be about following guidelines, but should also consider patient views and beliefs. There were also concerns that data extracted from the EMR did not reflect the nuances of the clinical decision-making process.“…I’d have to admit, I think most of us here would probably prescribe by guidelines, so I think a lot of that non-compliance is attributed to personal interactions with patients and their preference.” GP1While QI programs for AMS were acceptable to most GPs, there were tensions about the goal of this study—a program to assist GPs to optimise prescribing as opposed to a tool that could be used “against practices” or to show that general practice was “not performing”. Some GPs were concerned that data could be used as key performance indicators (KPI) linked to funding.“…my biggest concern about this is the data and what’s going to happen and be used with our data, because I have a very strong sense that the final destination for this stuff is going to be a KPI for funding practices… for an accreditation or to get your PIP (practice incentive payment)… you have to demonstrate that you’ve got 85% compliance with your antibiotic stewardship...” GP4Regardless, AMS programs need to be kept front and center to modify GPs’ antibiotic prescribing behaviour.“…I think when the goal of the study is essentially behaviour change for doctors, it’s got to be about keeping antibiotic stewardship sort of front of mind. It’s about keeping it salient. So, any way that I can be reminded throughout the year, if it’s a group-based thing or if it’s a pop-up on Best Practice or if it’s part of the induction when the new doctor starts, whatever it might be, it’s just about sort of constant reminders as opposed to “yes, we’re done now and you don’t have to think about it again”. It’s just keeping in your head.” GP10Some GPs felt that the audit duration with the pop-up reminder function  should be kept short.“I certainly don’t want that big popup window any longer than two weeks…” GP3“I don’t think I needed to be whacked on the nose with the newspaper multiple times...” GP7Interestingly, GPs commented that they did not notice the GP NAPS pop up tool as much during the follow-up audit period. This might be due to the change in prescribing habit, or from the impact of COVID-19 reducing respiratory tract infection consultations in practices. Often patients with respiratory symptoms were automatically referred to respiratory clinics at the height of COVID-19 cases, further reducing the numbers of antibiotics prescribed for patients coming in for these conditions.“I don’t know if I don’t put a reason in because I wasn’t seeing the patients or because I’m much better at putting a reason in… it might have just been because we weren’t prescribing nearly as many antibiotics.” GP9The impact of COVID-19, therefore, has led to concerns that audit results did not reflect their usual practice as disease presentations changed due to the pandemic.“…there was actually a whole type of pathology we weren’t seeing at all.” GP5The feedback report provided a comparison between practices, which reinforced to some GPs that while there were room for improvement, prescribing was similar across the practices.“It [feedback report] is actually remarkable how similar all three practices actually end up. They’re pretty much within that sort of two thirds appropriate and one third less appropriate… we keep hearing over and over and over again how terrible GPs are and our prescribing of antibiotics is terrible… Sure, there’s maybe some room for improvement and that might come back to some of the data extraction… So, it’s interesting that we’re probably not as bad as we’re painted to be.” GP9GPs thought benchmarking was helpful so that practices could compare their performance with other practices. However, benchmarking should be comparing practices with similar characteristics including patient demographics, practice demographics , and practice size and billing type. There was also interest in feedback comparing individual GPs. While this was discussed in the development phase of the QI program, individual-level feedback was not possible with the small number of participants in this pilot study.Knowledge and skills of health professionals in AMS and their attitude to receiving feedback were considered important factors that would influence the perceived benefit of the program and acceptability of the intervention. In terms of knowledge and skills, GPs thought interventions would be beneficial for younger, less experienced GPs, but also for older GPs who might not be as up to date with the current guidelines.“I think it’s an interesting topic and I think it’s a really common presentation… with changing guidelines and getting stuck in habits that may have been generated 15… or 18 years ago or whatever, there probably is a lot of benefit for younger GPs who are starting off their prescribing habits, but I think equally there’s probably just as much benefit for older GPs or GPs more experienced who perhaps aren’t as up to date as they could be …” GP9GPs had different attitudes towards feedback. One GP had concerns regarding the way data were presented, feeling that the data on appropriateness were presented in a negative way. Others found the feedback helpful and inspired them to reflect on their own practice.“I kind of feel that scooping up the sub-optimal to enhance the inappropriate data, this happens all the time with GP studies. We saw it with the paediatric parent study of parent satisfaction with GPs.” GP4“Yeah, I think it’s useful to reflect and look at ways that we can improve our use of antimicrobials, yeah for sure.” GP2Practice culture, where practices are research focused, was perceived to be strongly linked to whether a QI program would be successful or not.“As a group they’re highly active and highly engaged in these sorts of programs or projects… Anything they can learn from, they’re very hungry for that and passionate about making sure things are done right.” PM2It was interesting to note that GPs’ engagement was seen as the motivation for practice managers to be involved in QI programs. While actively engaged GPs helped with practice managers’ involvement, GPs who were less engaged often left practice managers at a loss as to how to drive the program forward.“Going back to it all, I think one of the downsides to it I’ve found is that the doctors just sort of did what they did... I didn’t find that there was a lot of feedback from them in terms of letting me know what was going on…So I think there was a lack of feedback from them to me.” PM3From the GP focus groups and practice manager interviews, it emerged that practice managers’ involvement and leadership was critical to the success (or failure) of any program implementation.“Absolutely. Taking that leadership, so not just sort of saying, “Hey guys, do you want to do this quality improvement project? Here you are, this is everything, this is what you need to do.” Actually be involved and be able to actually go to them every week and say, “Hey, how are you going with that? What’s been going on? Are you having issues? What’s something that happened good today?” … to be much more of a leader in the project …” PM3There were challenges to the program implementation, including accessing computer systems at the practice sites (in person or remotely); coordinating IT support between practice and research team; practice managers’ time needed to coordinate installation process; and testing of the tools before going live. Technical issues identified during the testing phase of the CDS tool, along with the added complications of the study team not being able to engage with practices during the COVID-19 period, led to the CDS component of the QI program not being implemented into the practices.Computer access was a major barrier to the implementation of the QI program; practice managers needed to find time to minimise GPs’ workflow interruptions.“They did start to get a little bit tired of being booted off their computers… we tried to do it at lunchtimes and they would just run late… as GPs do.” PM1In addition, practice managers were not comfortable having external parties accessing their computers, and with the need to use remote access due to the pandemic, this added to the burden of the practice IT support and practice managers overseeing the installation process.“And obviously for me it was about protecting patient privacy as a practice manager. So just wanting to make sure that everything was going to be above board, which it was. We’ve had a lot of help from our IT people, especially with COVID and all that…” PM1“I think the implementation of IT, that was the biggest thing, but I wouldn’t let anyone else do it. I don’t give other people access to our IT easily, so that was certainly the most time-consuming thing… I wouldn’t feel comfortable with someone else wandering around different machines and turning them on and doing what they wanted…” PM2Cost was also mentioned as a barrier to the installation process, time taken for practice managers to coordinate the implementation, practice IT team to assist with implementing the tools, and GPs to test the system.“I think the main issue, in regards to setting it up, is that it actually costs our practice manager lots of time, and with all those IT issues, and I think this is something that’s making it difficult for general practices to participate in the high-quality research.” GP2This study highlighted many important factors when implementing a QI program in general practice. Data that accurately reflect GPs’ decision-making process; tools that assisted GPs and their practice in providing better health outcome for their patients; and audits and feedback that were robust and provide meaningful information to the GPs were seen as critical to a QI program. Undoubtedly, a positive “whole of practice” approach with good communication between GPs and practice managers; ensuring strong participant engagement throughout the program; having an engaging practice champion who would advocate for GPs and the program; reducing workflow interruptions; and minimise costs for practices could ensure the success of implementing a QI program within general practice.While the GP NAPS audit data showed an overall decrease in the number of antibiotic prescribed, a slight overall percentage increase in the number of indications documented in the EMR, and an overall percentage decrease in the indication entered in the audit pop up tool, we cannot determine the significance of these results due to the small sample size. However, GPs were responsive to the feedback report provided at the end of each GP NAPS audit period. Similarly, Meeker et al. (2014) found a significant reduction in inappropriate antibiotic prescribing for acute respiratory tract infections when using accountable justification and peer comparison as behavioural interventions . NeverthTime constraints for both GPs and practice staff, cost of activities, issues with information management and technology, and a lack of financial incentives were considered as barriers to QI programs in primary care ,17,18,19The COVID-19 pandemic during our study period posed further challenges. Melbourne experienced the first lockdown period from March to May 2020, and the second lockdown in August to November 2020. Most face to face visits at general practices became telehealth visits, and patients with respiratory symptoms were diverted to respiratory clinics. During this period, participating practices commented they did not see patients with respiratory symptoms, thereby further reducing the number of consultations and potentially reducing the number of antibiotics prescribed, leading to possible inaccurate data interpretation.The small number of participating practices was a limitation to our study. Data collected were limited to three general practices in metropolitan Melbourne, therefore our findings may not be generalizable to more diverse settings, including practices in rural or remote areas. In addition, our study may include selection bias as participants were more likely to have an interest in antimicrobial stewardship and QI programs. This study was largely conducted during the COVID-19 pandemic, which was characterised by reduced respiratory tract infections presentations to general practice, significant changes to delivery of care more broadly  and impacted the delivery and likely participation in the quality improvement activity. Despite this, all components of the intervention were delivered. We did not have access to additional funding in order to carry out additional audits after cessation of lockdown in Melbourne. While our study highlighted several challenges both researchers and practices faced when implementing a QI program, some of the enablers to the success of the implementation process was the enthusiasm of participating GPs in the QI program, and the positive attitude of the practice managers, who were the driving force for this study. They constantly liaised with the research team and their GPs enforcing the importance of having a practice champion who can coordinate, communicate effectively with GPs and advocate for the practice and the program. Other studies have also emphasized the importance of a practice champion, whether this be a GP or a practice staff such as a practice manager, as part of a key ingredient to successful implementation of programs ,18,19,20Baseline GP NAPS audit (November 2019) using passively extracted prescribing data; with a customised feedback report supported by a webinar to discuss findings with participating GPs (December 2019).Royal Australian College of General Practitioners (RACGP) accredited Quality Improvement educational activity supported by North Western Melbourne Primary Health Network .Implementation of an electronic CDS tool to provide access to the eTG at the point of care (February–May 2020).Follow-up GP NAPS audit (August 2020) with a customised feedback report supported by a webinar to discuss findings.The Guidance GP program consisted of four key components, conducted between November 2019 and August 2020:During the two-week GP NAPS audit period, a data extraction tool was used to collect data from the EMR for each antimicrobial prescription generated. The tool was embedded into the practice EMR and scanned for prescriptions of any antimicrobials. For each prescription, data fields were extracted from the EMR, these included patient height, weight, renal function (eGFR), medical history, reason for prescription, prescription information, pathology and imaging requests and results. If the ‘reason for prescription’ was not entered, a pop-up box would prompt GPs to select an indication from a live and curated indication list . This isThe results from the two-week audit period were provided to GPs in a form of a de-identified practice-specific feedback report in order to allow GPs to reflect on their prescribing practice collectively. In addition to the report, a feedback and education webinar was conducted after each audit for GPs to ask clinical questions and learn from others about potential strategies for QI around AMS.The CDS tool integrated eTG into the practice EMR. The CDS tool streamlined access to the eTG through an icon located on the main screen and prescribing screens of the EMR .GPs who completed the pilot program were provided with 40 Quality Improvement Continuing Professional Development activity points through RACGP. The program included reflection on practice and documentation of strategies that they would introduce to their practice as a result of participating in the program.Three metropolitan Melbourne general practices were recruited to participate in the pilot study. An expression of interest was sent to practices via the NWMPHN newsletter. The only inclusion criteria were that practices had to be using the Best Practice™ electronic medical record system. Interested practices contacted the researcher (RBn) and a practice visit was organised where the program was discussed with the practice manager and GPs interested in this study. Incentives for participation included a practice incentive payment (AUD $2000), provision of a 12 month subscription to eTG , and a AUD $100 gift voucher for practice managers for participating in an interview.At the end of the pilot, three focus groups (one in each participating practice) with GPs and three practice manager interviews were conducted to evaluate the acceptability and usability of GP NAPS and the CDS tool, and any implementation barriers and facilitators of the program were discussed via video conferencing in September 2020. RBn conducted all focus groups and interviews. Data from the focus groups and interviews were digitally recorded, and transcribed verbatim. Transcripts were reviewed to remove any identifying information and all GPs, practice managers, and practice names were replaced by pseudonyms or codes. All transcripts were read at least twice and compared to the recording for accuracy.The CP-FIT was used to provide a framework to analyse the focus group and interview results . This thAll focus groups and interviews were independently coded by two researchers (RBn and TM) to identify pertinent concepts and ideas by mapping to the CP-FIT domains. The first transcript was independently read by RBn and TM to generate initial codes and themes to match the framework, which were then compared and refined. Differences were resolved by negotiation and consensus, and consultation with a third researcher (JMN). A further transcript was coded using this schema, compared and refined, and this process was repeated until all transcripts were coded. Data were managed using NVivo V.12 .This study demonstrated that implementing a QI program that employed (a) a CDS tool to facilitate appropriate antibiotic prescribing by providing easy access to guidelines, and prompting for an indication in general practice, (b) auditing with automated data extraction and evaluation of appropriateness and (c) feedback, is not an easy undertaking. Barriers such as cost and practice access, program fitting into the GP workflow, data accuracy, and perception of the value of the program can hinder the success of the QI program. Many technical issues could be resolved with a more centralised approach to deployment beyond the pilot. However, an enthusiastic practice-wide approach, positive practice champions, and the passion to reduce inappropriate antibiotic prescribing can contribute to the success of the program. While the pandemic was an unexpected hindrance to this study, by ensuring all participating GPs understand the expectations of the program, allowing extra cost and time for the implementation of the tools, and identifying and involving practice champions in the development of the QI programs could address some of these barriers found and enable the successful implementation of future QI programs in general practice.


ABSTRACT IMPACT: Implement and evaluate a fellowship program to foster a new generation of entrepreneurial and collaboratively-minded team scientists, equipped with the knowledge and skills to innovate technology-based solutions for COVID-19 to advance human health OBJECTIVES/GOALS: Mount Sinai Targeted Healthcare Innovation Fellowship (THRIVE) is a 9-month program for participants from diverse professional backgrounds to develop HealthTech innovations related to COVID-19. The program is designed to provide an experiential team science platform for fellows to take an idea from concept to commercially viable innovation. METHODS/STUDY POPULATION: Following a competitive application process, 16 THRIVE fellows comprise four teams working collaboratively in an online forum with input from experts in the field. Success of the program will be evaluated by:assessing pre- and post- collaborative research orientation among THRIVE fellows using the ROI scale1using social network analysis (SNA) to investigate the social networks of THRIVE fellows to capture patterns of communication and collaboration related to innovation developmentexploring participant experiences of group formation, teamwork and collaboration related to innovation development using one-to-one semi-structured interviewdetermining team success in innovation development, measured by number of publications, funding awarded, provisional patents and viable products. RESULTS/ANTICIPATED RESULTS: Paired t-tests will determine whether collaborative orientation of THRIVE fellows changes pre- vs. post- program participation, indicating changes in attitude toward multidisciplinary team work. SNA will be used to describe structural patterns of communication that occur at individual and group levels. Network-level indices will provide insight into patterns of communication that exist in innovation development: degree centrality , betweenness centrality (number of bridges to others in a network), closeness centrality (closeness to others in a network). We will also test for associations between network characteristics and team success. DISCUSSION/SIGNIFICANCE OF FINDINGS: Understanding patterns of formal and informal relationships, interactions, and perceptions of the collaborative process among individuals in THRIVE teams will elucidate whether such a program can provide an effective forum for team science and innovation development related to COVID-19.


Purpose: To examine patients' perceptions of the feasibility, acceptability, and impact of a safety net-based community health worker (CHW) program.Methods: Semistructured interviews with patient participants diagnosed with type 2 diabetes (n=13) were analyzed using a traditional text analysis method based on grounded theory.Results: This study highlights that the CHW program can improve satisfaction in accessing health services and community resources, and overall health outcomes of patients in safety net practices.Conclusion: Patients' overall positive perception of the CHW program suggests that the intervention may be a viable solution to address the health and social needs of patients in safety net settings. Evidence from randomized clinical trials suggest CHW interventions led to improved health outcomes, better quality of care, and reduction in hospitalizations.2–4 Moreover, systematic reviews of CHW interventions suggest that CHW interventions can significantly reduce health care utilizations and cost as well as provide cost-effective interventions for certain health conditions, especially among underserved populations.6 However, despite a growing body of literature on how CHWs can improve health outcomes and decrease health care utilization, there is limited evaluations and knowledge around the effectiveness of CHW programs that is based on patients' perception. Thus, there is a need for research that examines the feasibility, acceptability, and impact of the pilot CHW program.Community health workers (CHWs) are trusted members of the community with extensive knowledge about community resources and work as an integral liaison between community members and local services.This study focuses on patients who have participated in the CHW pilot program to (1) explore patients' perspectives on the feasibility of the CHW program, including the timing and frequency, location, and attendance at sessions, (2) understand the acceptability of the CHW program, specifically with respect to the format and content of the intervention, and in comparison with other available supports, and (3) evaluate the impact of the CHW program, particularly in the areas of patients' knowledge of and access to medical and community resources, relationship with the health care team, as well as the program's impact on patients' social support, self-efficacy, and patient-stated goals.7 which is a 6-month CHW intervention where patients work with a CHW to achieve their stated health goals. The intervention included 31 adults aged 18 years or older diagnosed with type 2 diabetes (with an glycosylated hemoglobin ≥8) and indicated English or Spanish as their primary language. All patients (N=31) who participated in the CHW pilot intervention were invited to participate in a study. Patients who called the research line to opt out were excluded. This study was approved by the CHCI Institutional Review Board. A subset of eligible participants (n=13) who previously enrolled in the CHW program provided consent to participate in the study. Semistructured interviews  has adopted the outpatient Individualized Management of Patient-Centered Targets approach,terviews  A1 were 8 The construction of traditional text analysis involves identifying themes from the interview transcripts, marking up the texts that relate to the themes of the research, eliminating the texts that are not related to the subject of the research, and sorting them into thematic categories.8–10 Two researchers first coded independently, followed by mutual discussions with the principal investigator to ensure the accuracy of coding.Interview transcripts were analyzed using a traditional text analysis method based on classic grounded theory.n=13). In the present sample, the average age of participants was 56 years (standard deviation=9.2). The majority of participants identified as female (69.2%) and Hispanic or Latino (69.2%).n=13) reported that the timing and frequency of sessions were adequate. With regard to the location of the engagements, the majority (n=10) mentioned the clinic as a convenient location for them to meet. However, three participants preferred their home as the primary location for meetup given their health conditions. In addition, 11 out of 13 participants had missed meetings due to forgetfulness (n=1), work schedule (n=3), transportation issues, mobility issues, and health conditions (n=7).Participants talked about the feasibility of the CHW program, including timing of the sessions, frequency of interaction with CHW, location of meetings, and reasons behind missed session(s) . The resn=13) reported being satisfied with the program, including the usefulness and appropriateness of topics covered, as well as the structure and flow of the program. Moreover, all respondents (n=13) were satisfied with the CHW's professionalism, resourcefulness, knowledge, and engagement. When comparing the CHW program with other nonmedical supports, nine out of 13 participants mentioned they could not compare since they have not received any other supports (n=6) or the focus areas of other programs and interventions were different (n=3). In terms of recommendations for improving the program, five participants suggested the expansion of the program, which reflects the need to reach more people who could benefit from the CHW program by gaining access to the required medical and social support.Participants shared perspectives on the acceptability of the CHW program, specifically on the content, perception of the CHW, in comparison with other available nonmedical supports, and key recommendations for improving the program . All parn=10), medication adherence (n=3), and keeping medical appointments (n=2) have improved as a result of participating in the program, and contributed to the adoption of healthy behaviors. Moreover, participants shared that issues around access to medical services such as appointment scheduling (n=7), medication retrieval (n=2), and transportation (n=6) have shown improvement with the integration of the CHW program to usual care. Furthermore, related to self-efficacy, participants demonstrated the support of CHW helped reduce anxiety (n=3), boost self-esteem (n=8), and improve the overall patient experience and health outcome (n=8).When describing their perception regarding the impact of the CHW program, participants focused on three major themes: health care access and engagement, health-related social needs and resources, and self-efficacy and patient goals . Specifi1c level, whereas the remainder of the participants showed unchanged (15.4%), and increased (15.4%) A1c level. Furthermore, the attendance of the program has a positive impact on patient engagement, allowing patients to meet their stated health goals. The results displayed that the best-performing participants (61.5%) were the ones who were present during almost all meetings with CHW. Patients who were contacted over the phone due to their medical conditions or other personal issues reported no significant improvement in their A1c level regardless of their good attendance.Participation in the CHW program indicated the evidence of the effectiveness on health outcome . The maj11 and to increase access to the hard-to-reach population.12 This study suggests the effectiveness of the CHW program in the areas of increasing health knowledge and improving access to needed health services and health outcomes, which aligns with previous research.13–16This study offers qualitative insights into patients' perspectives on the feasibility, acceptability, and impact of a CHW program. A CHW program is one-of-a-kind intervention to improve patient's self-efficacy or medication adherence,17–19 By understanding patients' experiences with the CHW program, the findings advocate for improvements and expansion of the CHW program at CHCI.In addition, the results of this study demonstrate the positive effect of professionalism of the CHW in enhancing patient self-efficacy and trust. With the help of the CHW, patients acknowledged that they built up skills and confidence to achieve their target goals by modifying their lifestyle behaviors, which is consistent with earlier research that documented the benevolent relationship between CHW and community.20This study has several limitations. First, social desirability bias might be an issue since one of the data collectors has exposure to patients. This may have reported a higher patients' perception of and satisfaction with the CHW program. Comparing patients' stated goals in the semistructured interview with their baseline goals might mitigate this bias. Second, a small sample size where the majority of the participants identified as female, unreported race, and Hispanic or Latino might affect the generalizability of this study. Nonetheless, 6–12 interviews were sufficient enough to achieve thematic saturation.With these limitations in mind, the findings from this study have several implications on policy, practice, and research. First, the study highlights how integration of CHWs into clinical care teams could encourage patient self-advocacy by providing culturally appropriate health education, interpreting services, and health care system navigation. Establishing appropriate funding mechanisms such as grants, incentives, and reimbursement would allow for the sustainability and scalability of CHW programs within safety net settings. Similarly, health care practices working to address disparities and promote patient-centered care should invest in CHWs as extensions of the care team and the role of CHW needs to be recognized more for their contribution to increase health access and improve health outcomes. Further research regarding the solution to common pitfalls of the CHW program can help improve patient self-advocacy by addressing their medical and social needs, and health outcomes by strengthening the capacity of the CHW.


To verify the relationship between AETA ) electromagnetic anomalies and local earthquakes, we have performed statistical studies on the electromagnetic data observed at AETA station. To ensure the accuracy of statistical results, 20 AETA stations with few data missing and abundant local earthquake events were selected as research objects. A modified PCA method was used to obtain the sequence representing the signal anomaly. Statistical results of superposed epoch analysis have indicated that 80% of AETA stations have significant relationship between electromagnetic anomalies and local earthquakes. These anomalies are more likely to appear before the earthquakes rather than after them. Further, we used Molchan’s error diagram to evaluate the electromagnetic signal anomalies at stations with significant relationships. All area skill scores are greater than 0. The above results have indicated that AETA electromagnetic anomalies contain precursory information and have the potential to improve local earthquake forecasting. Earthquakes are a highly destructive type of natural disaster. They usually occur without warning and do not allow much time for people to react. Earthquakes can destroy a tremendous number of buildings and infrastructures. The number of deaths caused by earthquakes accounts for 54% of the total deaths from various natural disasters worldwide . On averA large number of studies have reported various pre-earthquake signals, such as radon anomalies ,4, atmosElectromagnetic (EM) phenomena possibly related to earthquakes has also been reported in many articles ,13,14,15Our team has developed a system that continuously monitors geoacoustic and electromagnetic signals and transmits data to a cloud server in real time via the Internet, and named it AETA (Acoustic and Electromagnetics to AI) . In prevIn The AETA is composed of a data processing terminal, a geoacoustic sensing probe, an electromagnetic sensing probe (EM probe), a data monitoring cloud platform, and a data analysis system . The proThe EM probe uses the same principle as the IMSC (search-coil magnetometer instrument) sensor on the French DEMETER satellite and the magnetic sensor on the Chinese electromagnetic satellite ,23. BaseIn terms of the equipment reliability, the AETA is a new type of instrument for monitoring earthquake precursors, and it was mass-produced by Zowee Technology Company after it was developed in the laboratory. The AETA was tested for performance, aging, and waterproofing before it left the factory. The reliability and stability of the equipment are guaranteed. Frequency response, sensitivity, noise level, and consistency tests of the EM probe were completed in the magnetic shielding room of the Huailai Remote Sensing Comprehensive Test Station of the Chinese Academy of Sciences. All of the above indicators met the requirement.In this article, we use data from the database, which is obtained by averaging the raw data per minute. The data for EMS station from 1 July 2017 to 1 June 2020 are shown in Superposed epoch analysis (SEA) reveals the weak but significant signals common to a multitude of events of the same type and averages out signals unrelated to the event and noise. In previous studies, this method has been used to study the relationship between local earthquakes and total electron content anomalies or ultra-low frequency (ULF) geomagnetic anomalies and the relationship between the geomagnetic storms and large-mega earthquakes ,26,27. IPrincipal component analysis (PCA) is widely used in data mining and data analysis. PCA can reduce the dimension and extract the interesting features from a data set. PCA was applied to anomaly detection of AETA electromagnetic signals and has achieved good results in the 2017 Jiuzhaigou earthquake case study ,29. On tn-day AETA data, in which a column contains the average of 1 h time intervals in 1 day. We can obtain the covariance matrix C of X, the eigenvalues of C  .(6)Es′=M and r are the magnitude of the earthquake and the epicentral distance, respectively.N earthquake events. For each earthquake event, we created an array of encoded abnormal scores for 30 days before and after its occurrence day. Then, we got a two-dimensional array with dimension N × 61. Next, we summed this array along the Y-axis and got a result with dimension 1 × 61. So far, we have obtained the SEA result in terms of the seismic electromagnetic anomaly.To make our work consistent with previous studies, earthquake events occurring within 500 km of the station and with an nsidered ,33. AccoN days within the entire analyzed period of the monitoring station instead of N earthquake days. For N days, the same procedure as described in To evaluate the statistical significance, we randomly picked To perform a statistical study of the AETA electromagnetic anomalies detected by a station, two important things are continuous long-term monitoring data and a certain number of earthquake events. The selected station should be in operation for more than 3 years and the number of earthquake events that satisfied the criterion described in The SEA method was applied to perform statistical studies of the AETA electromagnetic anomalies at 20 selected stations. The accuracy of the statistical results for each station was not affected although the same earthquake may be detected by several stations as shown in The times when the results of daily counts are significantly high at all stations are summarized in The generation mechanism of seismic electromagnetic signals is still being explored. The electromagnetic signal may be affected by various environmental factors during its propagation. The phenomenon that some observation stations do not respond strongly to observation signals exists in many earthquake detection systems . It is dAETA electromagnetic sensing probe is based on Faraday’s law of electromagnetic induction. Lightning, geomagnetic storms and power frequency signals will inevitably affect the electromagnetic field in the environment. Firstly, the frequency of the AETA electromagnetic signals is less than 200 Hz. However, lightning interference is mainly composed of electromagnetic signals with a frequency of 3 kHz to 30 kHz. Since their frequencies do not overlap, the impact of lightning on the AETA electromagnetic signal is almost negligible. Secondly, geomagnetic storms are a global and violent disturbance phenomenon of the earth’s magnetic field. The AETA electromagnetic sensing probe detects changes in the magnetic field by magnetic induction, which is related to the free charges around it. There is no clear conclusion about the generation mechanism of the seismic electromagnetic signal. It is generally believed that seismic electromagnetic signals are related to the rock fracture caused by the accumulation of underground stress. In the process of rock rupture, friction generates heat radiation, piezoelectric effect, or charge separation on the fracture surface generates a strong electric field, which ionizes air molecules in weak areas of the earth’s crust or cracks in active faults. The physical meaning of the AETA electromagnetic sensing probe monitoring data is consistent with the theory of seismic electromagnetics. Therefore, we believe that AETA electromagnetic anomalies have a high possibility to be related to local earthquakes.For most of the AETA stations, the electromagnetic anomalies a few days before earthquakes have a clear statistical significance. These anomalies can be used to forecast subsequent earthquakes. If an anomaly occurs, a prediction that an earthquake will happen in the next few days will be made. A modified version of Molchan’s error diagram was used to test the feasibility of simple earthquake forecasting based on AETA electromagnetic anomalies . D) and the length of the alarm window (L). The lead time is the time interval between the occurrence of the anomaly and the start of the alarm, and the alarm window is the duration of the alarm. Given the above results, the values of D and L vary from station to station. When the anomaly arises on Day i, it is assumed that an earthquake with sE ≥ i + D and Day i + D + L − 1, and this time period is defined as the alarm days. If an earthquake with sE ≥ N equal to the total number of days of the entire analyzed period, N1 equal to the number of alarm days, n equal to the total number of earthquakes, and n1 equal to the number of predicted earthquakes. The alarm rate can be expressed as There are two concepts involved in the description of the earthquake prediction: the lead time of the alarms (In this paper, we investigated the relationship between the AETA electromagnetic signals and local earthquakes. A modified PCA was applied to detect the anomalies and SEA was used for statistical research. The results suggest that the anomalies are more likely to appear before earthquakes at most stations. Moreover, we analyzed the feasibility of earthquake prediction based on AETA electromagnetic anomaly. The results of Molchan’s error diagram showed that AETA electromagnetic anomalies can bring information gain to improve the prediction effectiveness compared to random guess.There is still a need for long-term exploration because the question of earthquake prediction is complex and difficult. The method of short-term earthquake prediction based on AETA electromagnetic signal anomaly is worth studying in the future.


The NIH Center for Accelerated Innovations at Cleveland Clinic (NCAI-CC) was funded by the National Heart Lung and Blood Institute (NHLBI) to support academic investigators in technology development and commercialization. NCAI-CC was one of three multi-institutional Centers established in the fall of 2013. The goal of each Center was to catalyze the growth of an ecosystem of commercialization within their affiliated institutions and regions by managing a program of funding and guiding translational project development and by delivering commercialization education programs to participating investigators. NCAI-CC created and managed such a funding program, ultimately supporting 75 different projects across seven separate academic institutions and developed tailored educational content following the National Science Foundation I-Corps™ curriculum and delivered the program to 79 teams from 12 institutions. We determined early on that in establishment and implementation of projects, it is important to support the teams and principal investigators throughout the program. The support includes a change in principal investigator mindset from specific aims orientation to goals and deliverables on projects. Our skills development efforts emphasized commercialization and a deep understanding of customer needs for new technology adoption. Here, we review our experiences, outcomes, and insights, including the challenges identified in program implementation. Cleveland Clinic, along with partnering institutions Case Western Reserve University, The Ohio State University, Cincinnati Children’s Hospital Medical Center, the University of Cincinnati, the University of Michigan, and Northwestern University, comprise the NIH Center for Accelerated Innovations at Cleveland Clinic  program which was established and operated by Cleveland Clinic from 2007 through 2019. The GCIC organization, comprised of a team of industry-experienced product developers and program managers, had a well-established track record of soliciting, selecting, funding, and guiding development of cardiovascular projects all throughout the state of Ohio. GCIC’s focus, however, was on funding technologies that were being developed by early-stage companies, not projects that were being pursued within academic research or clinical institutions, to achieve economic development objectives of investment and jobs growth. The program was by all accounts successful, but it did not address the significant need for providing similar commercialization-oriented funding, industry-experienced guidance, and education to research institution-based investigators. However, the NCAI opportunity was directed exactly at that gap and with a focus on closely aligned cardiovascular, lung, and blood clinical areas. As such it provided an attractive fit and opportunity for expansion.GCIC and NCAI-CC operated together as two very synergistic and complementary programs with highly leveraged resources. Through NCAI, we dedicated product development and education programs directed to investigators and early-stage translational projects in the consortium institutions. NCAI augmented the project director staffing with individuals that have more in-depth experience in diagnostics and therapeutic drug development to address more specifically those types of development projects.Our program established a regular semi-annual technology solicitation, selection, and funding process targeting projects that aligned with the NHLBI domain that included heart, lung, blood, and sleep disorders, but which excluded certain disease states that fell under other NIH institutes such as cancer, digestive disease, neurologic disease, and diabetes.Project selection and funding were managed via a Request for Application (RFA) and review process, similar to grant funding processes with which investigators are well familiar for their research Fig. . The preProject selection was based on 1) fit of the project within the clinical domain areas of NHLBI, 2) the project phase – being to conduct development-oriented activity to achieve commercially relevant milestones, and 3) the feasibility of being able to accomplish the proposed work within the appropriate timeframe and budget. Reflecting that the institutions comprising NCAI-CC were a mix of clinical and university (academic) research institutions, funded Principal Investigators were a mix of researchers , physicians , both , or other (4%); however, applications for funding were not evaluated on Principal Investigator education, academic position, or commercialization experience per se. Participating institutions exercised some influence on project eligibility a) by providing the required project matching funds, and b) by providing technology commercialization office support for establishing intellectual property protection and licensing.Projects that passed the initial Letter of Intent review were invited for full application submission. Applications were reviewed by an External Selection Committee (ESC) comprised of prominent clinicians, scientists, industry representatives, business development experts, and members of the venture capital community, who evaluated the projects on relevant clinical, technical, and commercialization criteria in a study section style review. The proposals were reviewed by at least three ESC members, discussed and ranked, and the most meritorious projects were advanced for a final evaluation by a Technology Review Committee (TRC) convened by NHLBI. The TRC was comprised of representatives from the Food and Drug Administration (FDA), the U.S. Patent and Trademark Office (USPTO), The Centers for Medicare and Medicaid Services (CMS), a major healthcare system, and the NHLBI resident regulatory expert, entrepreneur-in-residence, and investor-in-residence. This review provided unique perspective and valuable feedback to the applicants by key decision-makers in regulatory, commercialization, payment, and healthcare delivery aspects of commercialization. Investigators received instructive feedback on evaluation of their projects at each phase of selection.The NCAI-CC application process was conducted on semi-annual basis, typically took 24 weeks from RFA to funding announcements Table , and conNCAI project directors from Cleveland Clinic engaged with prospective and funded investigators on a continuous basis throughout the program. Some of the principal investigators, upon request, had an opportunity to discuss their technology with the NHLBI regulatory expert to understand the process on a first-hand basis. This engagement was not limited to the funding round periods when active solicitation of applications was ongoing, but was a consistent engagement with all the partner institutions to scout for and cultivate candidate projects and to work with the investigators to make for more compelling applications. This involved, for example, consulting on doing the necessary research, developing lead compounds, collecting relevant data such as in vitro/in vivo efficacy data, and discussing commercialization strategy. Engagement continued throughout the funded project period – keeping focus on reaching the project milestones and extended beyond to advising on post-project continued development and commercialization activities.Value Creation in Healthcare Product Development and CommercializationIntellectual Property (IP) and Building Value through IPTarget Product Profiles and WorkflowsUnderstanding Your Regulatory PathwayMedical Market SegmentationKey Commercialization Activities and ResourcesPartnerships, Relationships, and ChannelsCost, Reimbursement, and PayersMedTech Reimbursement for EntrepreneursA major objective of the NHLBI NCAI mission was implementation of an entrepreneurship education and skills development program within each of the Centers. From the beginning, NCAI-CC focused on providing direct skills development instruction to the teams from all partner institutions through individual engagement by the Center’s project development directors. The goal of the education program was to provide investigators with commercialization-focused skills that complemented the proposal solicitation. Soon after program implementation, the efforts were enhanced with the introduction of NSF I-Corps™. NHLBI partnered with the National Science Foundation (NSF) to support implementation of I-Corps™ as a component of NCAI skills development initiatives and facilitated a special funding opportunity whereby NCAI Centers could apply for funding to establish I-Corps Hubs within their programs. The I-Corps entrepreneurial education program was designed to identify the customer needs and model the critical steps for business development. The NSF program has seen various iterations since early development and the NCAI-CC program adopted basic features while piloting different strategies to specifically tailor the I-Corps content for the unique needs of biomedical investigators. We rolled out I-Corps@CC to Cleveland Clinic and Case Western Reserve University. The program introduced biomedical-tailored I-Corps lectures on all segments of the business model canvas delivered by industry-experienced entrepreneurs and project managers, plus a significant library of supplemental readings focused on specific biomedical project considerations. The specialized topics are defined here and were also incorporated into NCAI-funded project kick-off meetings:The goal of the skills development program was implementation of entrepreneurial educational programs to investigators throughout the consortium of NCAI-CC partner institutions. To expand the implementation of I-Corps@CC to the other NCAI-CC partner institutions, we partnered with the State of Ohio NSF-sanctioned I-Corps@Ohio program. The Ohio State University was funded by the Ohio Department of Education to provide the generic I-Corps program to academic investigators at institutions throughout the state. I-Corps@CC was able to complement the I-Corps@Ohio program by providing biomedical-specific content and thereby to extend the reach to all NCAI-CC Ohio partner institutions, as well as to other Ohio institutions, and recently also to the NCATS Clinical Translational Science Collaborative program’s implementation of I-Corps@NCATS. Thus, the Center has been able to augment its direct-to-project team individual coaching engagement by project directors with curriculum-based education on a wider scale.During the course of the program, NCAI-CC conducted 14 funding rounds, in which nearly 400 letters of intent were submitted; 141 full applications evaluated; and 75 projects awarded funding. Total federal funding awarded to these projects by NCAI was $8.85 Million, to which $5.61 Million of cost share was added by local non-federal sources, yielding an average per project budget of $193,000. Of the 75 projects funded, all but three were successfully completed or are in the process of concluding. The reasons for not completing projects were primarily due to the PI relocating to another institution during the project period.At time of writing, 36 of the funded projects (48%) progressed to the stage of having been licensed or optioned to newly started or existing companies. Eight of those projects received SBIR or STTR funding to continue development through 15 separate awards. Notably, two projects have later been acquired by major pharmaceutical companies Novo Nordisk and Amgen. Total follow-on funding or financing deals commit $1.3 Billion to the continued development and commercialization of the technologies (dependent on achievement of future milestones). Major factors for success to which these results can largely be attributed include a rigorous, qualification-driven project solicitation and selection process, experienced and commercialization-directed project management, and access to technology validation funding sources to support early-stage development.Over the course of conceiving, designing, operationalizing, and managing a multi-institutional, NIH-funded program to accelerate the translation of research discoveries to development for improving patient care, we learned many lessons, overcame challenges, and believe that we established certain best practices that can be instructive to other programs. Findings along that journey can be grouped into the following categories:Perhaps the first challenge and realization were that managing a program focused on product development and commercialization in contrast to academic research and publication focus necessitated a major mindset change in the investigators to whom the program was targeted. Rather than pursuing the concept of research aims, the program was focused on defining and executing commercialization-relevant product development milestones and timelines. Despite the emphasis of these elements in funding RFAs and repeatedly in funding cycle information sessions, early in the program it was not uncommon to receive slightly modified R01 aims pages as letters of intent. It took a significant educational effort over the first several RFA cycles to establish that the program served different objectives, namely directed towards achievement of critical development milestones to advance the technology towards commercialization and to achieve specific measurable project activities that are critical to establishing commercial opportunity. Similarly, to meet stage of readiness criteria, the projects had to have already reached certain stage of development. For example, for therapeutics development projects, it was strongly recommended that a lead compound was already identified with a disease target validated by appropriate screening assays and with initial indication of in vivo efficacy. With progressive funding cycles, these criteria became more familiar within the participating institutions and among the candidate investigators, and applications reflected more mature, development-oriented projects. And a higher percentage got funded.Project directors employed by the Center played a critical role, both in cultivating potential projects for the program and in working closely with the PIs to support the development work itself. The program project directors were industry-experienced senior product development engineers with either medical device or pharma backgrounds. These experts were deployed to work with all partner institutions in the Center and engaged in all aspects of candidate project identification, qualification, application for funding, and consulting to the project before, during, and after the project period. The result and impact were that candidate projects were very often cultivated with input from the project directors, guided by them during the project period to efficiently achieve targeted development milestones, and often strategically positioned with their input for follow-on funding, investment, and licensing opportunities. Frequently, investigators cited engagement with the project directors as one of the most significant aspects of the program .The program managed federal funding to the investigators via subgrants from the main grant at Cleveland Clinic for one-half of the project direct costs and associated indirect costs, plus the requirement for 1:1 non-federal matching funds for direct costs to be supplied from other, usually institution-managed, sources. The federal funds were distributed on an invoice reimbursement basis and depended on the project making progress towards the stated project milestones and timelines, like a phase-gate process, as reviewed by the project directors. This provided a measure of accountability toward the work conducted, which was different from the way investigators were used to having their projects administered.Initially, non-federal matching funds were difficult to source. Many of the participating institutions did not have matching funds sources available a) to commit to the project during the application phase as per requirement of the project RFA and b) to be distributed to the project in coordination with the federal funds. Over time, however, the institutions did identify and allocate funding sources for this purpose and aligned the timing of their decision processes to coordinate funding awards from those sources with funding awards from the Center for the approved projects.The administrative sub-grant process for federal funds from the Center to the project institutions also had a degree of difficulty in coordination throughout the program. This resulted in delays in funds availability for targeted project start dates and consequently for project completion dates. This was an internal issue but reflected a challenge in managing a multi-institutional project funding program. The projects also ran up against NIH annual funding timelines that do not coincide with the project timelines. As such, despite best efforts to manage project timelines and to keep scheduling, these administrative delays resulted in a higher-than-expected incidence of project timeline extension requests.NCAI-CC took a primary direct-to-investigators approach to skills and entrepreneurship development. Education provided was dynamic and supported the immediate and specific needs of the investigators and their projects. To complement and expand this, the Center also adopted the NSF I-Corps curriculum and specially tailored the content to support biomedical investigators for broader education. A significant learning from the NCAI-CC program was the critical need for both individualized investigator support in entrepreneurial skills development and specifically focused programs that specialize in building a greater understanding of the value proposition of the technology. Our approach also provided significant engagement with individual investigators as they developed their technology from concept to funded project and follow-on support.A stated objective of the overall NCAI program was that participating centers would develop means to scale and sustain the program beyond the original scope and federal funding of the program. The NCAI program at Cleveland Clinic did successfully demonstrate the ability to expand the number of participating institutions by adding the University of Michigan and Northwestern University to the consortium in year 4 of the program and to extend the centrally provided project management support by the project directors to new institutions. The model of engagement with the new institutions was based on deploying the commercialization-oriented, RFA-driven funding program qualification criteria, along with in-person engagement with the Technology Commercialization offices and with candidate investigators, like with the original institutions. Greater distances from Cleveland presented some logistical adjustments and incurred additional expense but was demonstrated to be workable.The program has also demonstrated scalability in the education of researchers for technology commercialization objectives by incorporating the specially tailored I-Corps™ biomedical/bioscience curriculum developed under the NSF grant to Cleveland Clinic into the broader I-Corps@Ohio program that reaches well beyond the NCAI participating institutions.A model for designing and managing academic institution-based technology development and commercialization programs. For example, the State of Ohio has developed and manages a Technology Validation and Start-up Fund that is directed towards similar stage academic research-based development and commercialization, which has very similar qualifying criteria to the NCAI program, and is broader than just biomedical/bioscience applications. It served as the source of matching funds for many of the NCAI funded projects. Coulter Foundation and Fast Forward Innovation programs also operate at participating institutions.An approach for incorporating commercialization-relevant expertise into project management at these programs. Industry-experienced project directors/managers and regulatory advisors were already in place or have been added to the technology commercialization organizations at each of the NCAI-CC participating institutions.A model for the commercialization education of investigators at academic institutions. As described, the I-Corps curriculum has been specially tailored for biomedical/bioscience applications and is deployed on an ongoing basis to investigators at a dozen Ohio institutions, well beyond the original reach of the program.A catalyst for establishing new funds to support early-stage product development at academic institutions. The program originally proposed an objective of re-investing proceeds from licensing or commercialization successes from program-funded projects back into the institutions for deployment to new projects. Such re-investment may well be managed by the individual institutions as their projects progress and yield licensing returns, depending on their policies and practices. However, there are no current means for such proceeds to feed back to the Center for its sustainability per se, and the timelines for such returns on licensing of such early-stage projects are beyond the horizon for sustaining the original program. Nevertheless, each of the participating institutions reports that they have raised new dedicated funding sources to continue and to grow these types of research translation programs.The value of including input from the FDA, NHLBI, CMS, USPTO, and other healthcare systems, early in the technology selection process. This feedback not only helped determine which projects were ultimately funded but also helped shape the technology development program to rapidly advance the technologies to a commercial state.Sustainability of the program can be viewed from at least five perspectives.NCAI-CC was a consortium of seven prominent academic research institutions and medical centers spanning the state of Ohio and extending to Michigan and Illinois. Although each operated independently, collectively they had great influence on developing and expanding the research translation and commercialization ecosystem in their regions. Each institution has incorporated elements of the NCAI experience and practices into their ongoing programs of technology commercialization. New sources of technology development funding have been established at institution, regional, state, and federal levels to provide progressive sources and continuity of support for early-stage product development. Entrepreneurship and commercialization education programs have been established and expanded to provide investigators with ongoing resources and direction to guide their product development. Inter-institutional collaboration in these activities has grown, reflective in part of the consortium participation and engagement. All institutions acknowledge that they are stronger and that the ecosystem is more robust for having their participation in the NCAI program.