The United Nations Children’s Fund (UNICEF) published their Health Systems Strengthening (HSS) approach to meet its strategic goals of ending preventable maternal, newborn and child deaths and promoting the health and development of all children and reducing inequities in health in 2016. UNICEF commissioned the University of Melbourne’s Nossal Institute for Global Health to develop and deliver a pilot blended HSS program, involving 60 hours of online learning and 2 weeks of face-to-face teaching over a 6-month period. To assess the extent to which the HSS program had built the first 83 UNICEF 2017 graduates’ capabilities to apply HSS actions by 2017, UNICEF funded an independent evaluator from the University of Melbourne.A mixed-methods assessment was conducted using: online surveys of graduates at: enrolment, completion, 6 months post-HSS program; nine focus groups with graduates at face-to-face workshops; and interviews with purposive samples of UNICEF graduates and graduate Senior Managers 12 months post-HSS program.The HSS program content, structure and mode of delivery was positively received. Graduates reported increased confidence taking HSS actions and multiple changes in work practices . Graduates’ Senior Manager interviews revealed mixed impressions of graduates applying HSS actions, partly explained by the fit between the HSS program learnings and UNICEF’s workplace environment. Key contextual factors influencing graduates applying HSS actions included: workload; limited opportunities to apply HSS actions; limited HSS examples; and variable support to apply HSS actions. Graduate and Senior Manager suggestions to optimise applying HSS actions included: linking HSS program content with UNICEF priorities; increasing opportunities for graduates to apply HSS actions; increasing access to HSS support.The paper concludes by presenting HSS program and assessment suggestions from the 2017 UNICEF Pilot HSS program assessment and actions taken for the 2018 UNICEF staff cohorts by HSS program developers, funders and beneficiaries. The West African Ebola epidemic in 2014 drew the world’s attention to the importance of health system strengthening , 11. Res1.Institutionalise a systems approach in UNICEF country programs.2.Position UNICEF as a credible strategic partner in supporting governments’ efforts to improve health systems, particularly for the benefit of disadvantaged children and women.In 2016 UNICEF commissioned the University of Melbourne’s Nossal Institute for Global Health to develop and deliver a pilot blended HSS program. The HSS program consisted of 12 modules delivered online, each taking between 4 and 6 hours to complete, and including a module on Human Resources for Health , UNICEF funded an independent evaluator from the University of Melbourne  plan for maternal or newborn health servicesBox 4: 2017 Pilot HSS program assessment logic modelAssessment Study ParticipantsUNICEF graduates’ perceptions of HSS ProgramChanges in UNICEF graduates’ confidence with applying HSS learningHSS program impacts upon UNICEF graduatesContextual factors influencing graduates’ capacity to applying HSS learningThis section summarises the assessment findings under five headings, with illustrative quotes:, of which 83 completed all 14 HSS program modules, which is common in such programs. As can be seen from Box A total of 102 UNICEF staff enrolled in the 2017 HSS programTable To increase rigour, the assessment used multiple methods, and a transparent analysis and interpretation process. To enhance the utilisation of the assessment findings by HSS program funder and developers, participatory  and realThe online survey quantitative data were analysed using descriptive (frequencies) statistics. The online surveys (free text responses) were analysed using the constant comparative thematic analysis approach . The assBox 5: Assessment participants53 of 102 (52%) UNICEF participants completed an online survey at HSS program enrolment79 of 83 (95%) UNICEF graduates completed survey at HSS program completion73 of 83 (88%) UNICEF graduates participated in focus group discussions at the HSS program face-to-face workshops35 of 83 (42%) UNICEF graduates completed an online survey 6 months post-HSS program10 (out of 83) UNICEF graduates participated in a semi-structured telephone/Skype interview 12 months post-HSS program7 UNICEF graduates’ Senior Managers participated in a semi-structured telephone/Skype interview 12 months post-HSS programA total of 102 UNICEF staff participated in the 2017 HSS Program. Of these:Overall graduates reported that the HSS program content was good (“content of the program was good and thought provoking”), comprehensive  and aligned with key frameworks and strategies and  Building Blocks and UNICEF’s Health strategy”).Overall the participants found the course content thought provoking, comprehensive and aligned with key health system frameworks. Suggestions for improvements included diversifying the context of country case studies, reducing the time spent learning how to use a range of computer programs used in health system strengthening and putting more focus on resilient health systems in complex emergency settings.“well structured starting with framework and general concepts and then deep diving into specific components of HSS”; and Very good- the structure allowed for all elements to be covered with minimal wastage)Overall UNICEF graduates responded positively to the HSS Program structure as demonstrated by the following quotes:Graduates also commented that there was a good mix of theories with practice.UNICEF graduates offered multiple suggestions about improving the HSS program structure, including having more specific objectives and more online discussions. The face-to-face modules were perceived as having too many HSS tools and that more exposure to the HSS tools was required prior to the face-to face modules.Overall graduates commented that the program delivery had the right combination of visual and, audio, in class activity; pragmatic online modules; diverse methodology  and great presenters . Graduates responded positively to the face-to-face modules, as they valued the opportunity to interact with HSS program deliverers and other UNICEF staff trainees .UNICEF graduates offered multiple suggestions about improving the HSS program delivery, including: using university mentors to facilitate discussions during online course; to provide transcripts with videos; and reducing plenary presentations and increase floor for trainee experiences.“…the Learning [Program] was more than [HSS] tools- an opportunity to see the whole health system and understand how it works and what components and to move from an academic point of view to practice. Three key learnings. 1. understanding the health system- people, resources and data. 2. Interacting with different colleagues from different regions and different contexts and 3. using tools, but the weakest learning for me as no data. HSS tools are important but not the heart of the work—it’s the people, main issue is to understand that information is important, but need to deal with people. People are at the centre for community participation; transforming the situation and for meaningful relationships with Ministry of Health”.The UNICEF graduate interviews provided rich and broader insights into the HSS program content and context, particularly that the HSS program was more than the HSS tools as can be seen below::“HSS is a game changer—signifies paradigm shift. UNICEF was a service delivery organisation, focussed on downstream, to now more upstream, analysing health system data”.The focus group discussions with UNICEF graduates also revealed the broader HSS program context. For example, one graduate commented on the significance of the HSS programRole of UNICEF and purpose of the HSS program—several graduates raised questions at the strategic and operational level. For example: “What is the role of UNICEF in HSS?” and “What is the objective of UNICEF HSS training?”; “Is it to apply HSS Tools and Actions -as to apply HSS Tools need more technical, expert and consultancy support”Barriers for UNICEF adopting HSS approaches—graduates also raised multiple challenges for UNICEF adopting HSS approaches. For example: “Silos exist at a system, sector, organisation and community level”; “there is limited regional level quality and available data, mainly at country Level”; “there are competing priorities”; “donors drive priorities but there is variable understanding of HSS; and “the Ministry focuses on quick wins not systemic change”Collectively the assessment activities with the UNICEF graduates revealed multiple other issues relevant to the HSS program context that can be summarised into two categories:a greater proportion of UNICEF graduates reported being confident to support and to lead the HSS actionsUNICEF graduates reported the most confident to support and lead ‘Explaining universal health coverage’; followed by ‘Analysing supply chain systems’over 6 months post-program a smaller proportion of UNICEF graduate reported being ‘not confident’ with all taking the HSS actionsUNICEF graduates require further support with ‘Critiquing programmes designed to provide social protection’ and ‘Discussing health financing’.All UNICEF graduates were asked in the online surveys to rate their confidence with the examples of HSS competencies Box  over thr. UNICEF graduates reported a spectrum of examples illustrating changes in work practices as a result of participating in the HSS program. Table Evidence of the multiple HSS program impacts (i.e. observable changes in work practices) upon HSS graduates was evident from the online survey results at HSS program completion and 6 months post-HSS program“..there have been gradual shifts in my daily work, for example, relooking at the design of an on-going project on closing immunization equities in rapidly urbanizing settings in selected countries in East Asia, moving from specific programme focus to broader systematic approaches, increasing attention on the programmatic elements related to six health systems blocks, testing solutions in tackling the systematic bottlenecks associated with public financing and human resources constraints, more proactively engaging leaders of communities”.A key quote illustrating the multi-level and multi-dimensional impacts upon UNICEF graduates work practices can be seen below:HSS has really helped broaden [the graduates] and other graduates views and vision…learning has exposed [graduates] to the bigger picture and to be more affirmative…with HSS exposure they have more power and a different way to see the world- big picture lensA bigger outlook on the worldHowever, only discrete (not widespread) and mixed examples of applying HSS actions were reported. Illustrative quotes from Senior Managers are provided below:increased awareness of HSS has led to HSS issues being put onto the table, I can see more coherent discussion of HSS, indicating increased confidenceRefined planning in a systematic manner at both state and district level… facilitated assessment of initiatives, and reinforced need for data to support systemic structure and processes[HSS graduate] planned to do a nutrition conference—very technical, but returning from Melbourne they changed to planning a Nutrition Summit with key result areas, involving Ministry of Health, and a governance structureGraduate did the course, came back but never really applied what they learnt—but others who have done course have applied it- I can see it in their next workplans, using systems thinking, data—definitely it has made a differenceThe semi-structured interviews with the UNICEF graduates Senior Managers revealed mixed impressions about graduates applying HSS actions. Senior Managers confirmed that graduates had increased HSS awareness and knowledge and that the HSS program had broaden graduates worldviews, as can be seen below.I don’t have any real evidence—that HSS learning has influenced day-to-day work—more at a general discussion levelHSS requires a big vision—but I have not seen it—no evidence that HSS has influenced their strategic thinkingYes I can see how graduates has used training in her monitoring and tracking country data using HSS concepts”Several Senior Managers also commented that there was variability amongst UNICEF graduates who participated in HSS program in relation to evidence of putting HSS principles and concepts into graduates work practices—as can be seen below:The assessment activities revealed multiple contextual factors that were influencing UNICEF graduates applying HSS action that can be summarised into four key domains. Table HSS program content: Revise program content to link with UNICEF priorities and programsHSS program structure and mode of delivery: Refine structure and mode of delivery to increase opportunities for trainees to interact with program deliverers and other traineesOpportunities to apply HSS actions: increasing opportunities for graduates to apply HSS actions upon return to workplacesAccess to support: Increased access to HSS technical and coaching support at a country levelUNICEF graduates and Senior Managers offered multiple suggestions to optimise UNICEF graduates applying HSS actions that can be summarised into four domains:Masterclass for UNICEF Senior Managers… as well succinct modules on HSSFor Senior Executives—leaders like us—we can have a succinct HSS module for usNeed a management level HSS Masterclass so that we can interact with colleagues who attended HSS trainingA majority of UNICEF graduates Senior Managers interviewed also mentioned the need for a HSS Masterclass for Senior Managers, as seen below:HSS needs to be a key result area in the workplace—get it into the staff KPIs—need accountability.Senior Managers also recommended more accountability for UNICEF graduates applying HSS can be seen below:This assessment has confirmed that the HSS program can be a vehicle for building UNICEF graduates capabilities to apply HSS actions to meet UNICEF’s strategic goals of ending preventable maternal, newborn and child deaths and promoting the health and development of all children and reducing inequities in health.At an operational level, the HSS program has clear strengths including: program content, structure and mode of delivery that provides a mix of theory and practice, engaging with diverse visuals, audios and presenters.While HSS program graduates reported increased confidence with HSS actions and changes in work practices, the HSS program faces key contextual challenges: graduate workload and work foci; limited opportunity to apply HSS approaches; limited HSS examples; and varied supportive workplace environments. These challenges require us to reflect upon the fit between the HSS program learnings and UNICEF’s enabling or not enabling workplace environment for its graduates to apply HSS actions. These findings are in line with the formative evaluation findings of UNICEF’s programming in HSS .The following section contextualises and discusses the assessment findings with regard to two key domains: (1) Health Systems Strengthening policy context; (2) Transformative professional development transfer and effects and UNICEF’s Workplace Environment; and (3) the hypothesised Pilot HSS Program logic model. Reflections upon the assessment strengths and limtation are also discussed.The assessment has revealed that the HSS program has contributed to increased UNICEF graduates’ HSS knowledge, skills, confidence and capabilities to contribute to HSS in their contexts. While these findings are indicators of the HSS program effectiveness, and provide evidence to suggest that a HSS paradigm shift is occuring across UNICEF and its graduates, it cannot be attributed solely to the HSS program. Over the 2017 HSS program period UNICEF initiated multiple other initiatives to advance HSS which also involved UNICEF gradutaes, thus potentially contributing to increased awareness and understandings about HSS approaches and actions. The assessment of the HSS program needs to view its graduates within an ecosystem framework—where the organisation and interaction of the HSS initiatives in the system are as important as the initiatives themselves.The assessment findings revealed that graduates Senior Manager interviews had mixed impressions of graduates applying HSS actions upon their return to their workplaces. These findings require reflection for several reasons.Firstly, professional development programmes are recognised as a means and not an endpoint and part of a transformative change process, involving a process of ongoing change, adoption, implementation, dissemination, and sustainability of the innovation into practice .Secondly, based upon the reported challenges to applying HSS actions and the findings that UNICEF graduates requested more ongoing support, experiences and opportunities to applying HSS actions, it is important to reflect upon evidence about factors that influence learning transfer and effects.•Attendance—UNICEF graduates have attended the HSS program.•Completion—UNICEF graduates having completed the HSS program.•Knowledge recitation and retention—UNICEF graduates can recite and have retained HSS knowledge.•Decision-making competence—UNICEF graduates know what to do with HSS actions.•Task competence—UNICEF graduates use HSS knowledge.•Transfer—UNICEF graduates use HSS actions; and•Effects of transfer—training influences outcomes of UNICEF graduates work practices.Learning transfer is recognised as a complex phenomenon  which prWhile a substantial literature exists , 5, 14 fUNICEF graduates/individual characteristics: related to HSS program the following issues need reflection: graduates needs and goals related to HSS in their work; graduates competency to apply HSS; graduates motivations (intrinsic vs extrinsic) for applying HSS approaches; graduates career aspirations; graduates personality traits , and graduates perceived value of HSS training.HSS learning program design-related to HSS program the following issues need reflection: HSS training content, relevance and alignment to graduates work practices; HSS training presenter quality and engagement; provision of instructional strategies and methods to facilitate transfer of HSS approaches into practice .The short assessment time frame limits our ability to comment on intermediate and long-term Outcomes achieved so far. However, the assessment has revealed that the HSS Training Program is contributing to transformative changes (change in HSS mindsets and practices) at the individual and organisational level.The scope and short assessment time frame also limits our ability to compare or comment on HSS Training Program with other innovative learning approaches’.The evaluator recognised that the HSS program developers  and funded by UNICEF were the primary intended users of the assessment and hence required useful, specific and contextualised evidence to inform future HSS programs. A participatory and realist assessment approach guided the assessment methods, analyses and interpretations. The assessment findings reveal that the assessment approach facilitated a working partnership between the HSS program developers and the evaluator, leading to increased assessment utilisation and fostering a culture of learning between all.n = 83) nor of UNICEF graduates who participated in other assessment activities by UNICEF and/or HSS program developers—whose perceptions of the HSS program were more positive. Despite inviting via email all graduates to complete online surveys, participate in interviews, the evaluator was unable to involve all trainees in the assessment, notwithstanding many invitations and email reminder attempts. We also recognise that from the outset the assessment had a particular focus upon UNICEF graduates applying HSS actions, while the learning program had a broader HSS focus.We acknowledge that despite the assessment approaches, the assessment had limitations. The sample sizes in the assessment activities were small and not representative of the entire UNICEF HSS graduate cohort ; and HSS program beneficiaries  many of which have been acted upon for the 2018 UNICEF staff cohort. Table https://www.futurelearn.com/courses/health-systems-strengthening.A HSS MOOC has been developed by a collaboration between UNICEF and the Nossal Institute for Global Health and is available •To explore the influence of the work environment upon HSS program graduates applying HSS approaches.The 2017 assessment revealed the complex array of factors influencing UNICEF graduates applying their HSS learning, in particular, the UNICEF workplace environment. Given UNICEF’s commitment to adopting and applying HSS actions, the following research and assessment priority was suggested.At the time of the writing of this manuscript, UNICEF had commissioned a further assessment of the influence of the UNICEF work environment upon its 2018 HSS program graduates applying HSS actions, approaches and competencies in practice.


Lactobacillus plantarum J23 was developed. First, the effects of the composition of MRS broth on bacteriocin Lac-B23 production and bacterial growth were researched by a one variable at a time approach. Then, a Plackett-Burman design was used to screen significant components for production. Finally, the steepest ascent and central composite designs were used to obtain an optimum medium. The final composition of the modified MRS was much simpler than MRS broth, and the modified MRS contained only glucose, yeast extract, dipotassium phosphate, manganese sulfate monohydrate, Tween 80 and sodium acetate anhydrous. The highest bacteriocin Lac-B23 production reached 2560 activity units (AU)/mL in the modified MRS, which is nine times higher than that in MRS broth (280 AU/mL). Meanwhile, the cost per liter of the modified MRS (8.56 Ren Min Bi (RMB)/L) is 34.70% the cost of MRS broth (13.11 RMB/L), and the cost per arbitrary units of bacteriocin Lac-B23 in the modified MRS is approximately fourteen times more convenient (3.34 RMB/106 AU) than in the MRS broth (46.82 RMB/106 AU).At present, De Man, Rogosa and Sharpe (MRS) broth is the medium of choice for promoting bacteriocin production. However, this medium is expensive and not applicable for large-scale production. Therefore, a low-cost and high-efficiency culture medium for bacteriocin Lac-B23 production by  Bynthesis ,11,12. Hynthesis ,14. The Lactobacillus plantarum J23 (L. plantarum J23) from Chinese traditional fermented milk and shows antimicrobial activity against Listeria monocytogenes (L. monocytogenes) [L. plantarum J23 and production of bacteriocin Lac-B23. At first, the influence of the composition of MRS broth on bacteriocin Lac-B23 production and bacterial growth was studied by a one variable at a time (OVAT) approach. Then, Plackett-Burman, steepest ascent and central composite designs were used to obtain the best culture medium for bacteriocin Lac-B23 production. The need for developing a low-cost and high-efficiency culture medium for bacteriocin production lies in the final aim of producing a high amount of bacteriocin Lac-B23 to meet the need for further research, such as large-scale purification and research of antibacterial packaging materials. Bacteriocin Lac-B23 is a novel bacteriocin produced by togenes) . HoweverL. plantarum J23 was studied as a producer of bacteriocin Lac-B23 and was grown in MRS broth . Listeria monocytogenes (L. monocytogenes) CICC 21633, as an indicator bacteria, were grown in a brain heart infusion broth . All organisms were propagated twice in their corresponding medium at 37 °C for 24 h before use.L. plantarum J23 was inoculated in fresh MRS at 37 °C for 16h and centrifuged at 8000 rpm for 5 min. The supernatants were adjusted to pH 6.0–6.5 with 1-M NaOH and filtered through a 0.22-µm filter . Bacteriocin activity tests were used by the agar well-diffusion assay. Fifteen millimeters of 1.5% (w/v) agar were poured into a sterile plate as the bottom medium, and oxford cups of 7.8 mm were put on the above. Molten soft MRS broth agar  was inoculated with a 10-µL indicator strain (108 CFU/mL). Fifteen milliliters of the inoculated soft agar was poured into the surface of the bottom medium. After solidification, oxford cups were pulled out and 100 µL of the supernatants from L. plantarum J23 was placed into a well of 7.8 mm. The plates were incubated at 37 °C, and the zones of growth inhibitions were measured after 12 h. One bacteriocin unit was arbitrarily defined as the reciprocal of the highest dilution, which showed a clear inhibition zone and was expressed as activity units (AU) per mL.The cell-free supernatants and bacteriocin activity tests were performed by methods published from our laboratory . Briefly600) was determined at 600 nm using a UV-5100 spectrophotometer .Cell density  were added to the MRS broth without a carbon source to a final concentration of 2%, and the MRS broth without a carbon source was used as a control. To test the effect of the nitrogen source, a modified MRS1 (mMRS1) was chosen as a basal medium . The mMRS1 medium was supplemented with a mixture of multiple nitrogen sources or each individual nitrogen source . To studThe purpose of this optimization step was to identify which ingredient(s) of mMRS7 had a significant effect on bacteriocin Lac-B23 production. There were six ingredients in the mMRS7 medium, and every ingredient was set as two levels (−1 and +1) .According to the PB design, there were 12 experiments.To approach the region of the maximum bacteriocin activity, the next optimization step was carried out along the path of the steepest ascent .After steepest ascent, the next step was to determine the optimal levels of Tween 80 and sodium acetate. Therefore, the response surface approach by CCD was performed .L. plantarum J23 growth were glucose and maltose, followed by fructose, lactose and galactose  or maltose (concentrations from 30 to 35 g/L) as the only carbon sources for growth were also the most suitable carbon sources for bacteriocin Lac-B23 production (280 AU/mL) compared to lactose (160 AU/mL) and galactose (160 AU/mL). Bacterial growth can be improved by adding a yeast extract. Furthermore, the highest production (560 AU/mL) of bacteriocin Lac-B23 was observed when In addition, we can conclude that dipotassium phosphate, magnesium sulfate, Tween 80, sodium acetate and diammonium hydrogen citrate do not significantly affect the bacterial growth a,b,d–f. The Plackett-Burman design is an important technique for screening significant variables and has been used successfully by many researchers ,18. AccoP-values  were lower than 0.05 . In addition, the interaction between Tween 80 and sodium acetate, quadratic for Tween 80 and quadratic for sodium acetate, was also significant. The analysis of variance (ANOVA) showed that the p-value = 0.0174 and indicated significance for the regression model. The regression equation obtained indicated an R2 of 0.9778. This illustrates that 97.78% of the variability in the response could be explained by the model. The following regression Equation (1) was obtained:5 is the Tween 80 concentration and X6 is the sodium acetate concentration.It shows that the effect of sodium acetate was significant  medium after using the OVAT approach, PB design, steepest ascent experiment and CCD design. As shown in The composition of the medium had a very tremendous influence on the bacteriocin production and bacterial growth. The results indicated that the production of bacteriocin Lac-B23 depends strongly on the type of carbon source. Glucose was the best carbon source for bacteriocin Lac-B23 synthesis. Similar conclusions were described by Schirru . Glucose6 AU) indicated that bacteriocin Lac-B23 production in the mMRS medium was approximately fourteen times more convenient (3.34 RMB/106 AU) than in the MRS broth (46.82 RMB/106 AU). The possibility of producing increasing amounts of bacteriocin Lac-B23 by L. plantarum J23 can be considered a challenging opportunity for furthering the range of bacteriocin applications. In particular, bacteriocins can be applied in innovative food packaging. For example, bacteriocin Lac-B23 can be absorbed in a suitable biopolymer and provide an additional barrier, preventing the growth of food-borne pathogens in foods, such as meat, dairy and vegetable products.After the PB design, steepest ascent experiment and CCD design, a novel modified mMRS medium was obtained. The results confirmed that bacteriocin Lac-B23 production can be improved by growing cells in a low-cost mMRS medium. The highest production, 2560 AU/mL, was in a medium consisting of glucose, yeast extract, dipotassium phosphate, manganese sulfate monohydrate, Tween 80 and sodium acetate anhydrous. In particular, the composition of a nitrogen source in the mMRS medium was greatly simplified. Peptone and beef extract powder, as animal origin nitrogens, were completely removed in the mMRS medium. It could reduce untreatable medical conditions, such as Kreutzfeldt-Jacob disease . In addiL. plantarum J23 can be improved in a modified MRS culture medium containing glucose, yeast extract, dipotassium phosphate, manganese sulfate monohydrate, Tween 80 and sodium acetate anhydrous. Bacteriocin Lac-B23 production reached 2560 AU/mL in the mMRS, which is nine times higher than that in the MRS broth (280 AU/mL). Meanwhile, the cost per liter of the mMRS (8.56 RMB/L) was 34.70% the cost of the MRS broth (13.11 RMB/L), and the cost per arbitrary units of bacteriocin Lac-B23 in the mMRS was approximately fourteen times more convenient (3.34 RMB/106 AU) than in the MRS broth (46.82 RMB/106 AU).In this study, glucose, yeast extract, dipotassium phosphate, manganese sulfate, Tween 80 and sodium acetate were found to be the key control factors for bacteriocin Lac-B23 synthesis. However, peptone, beef extract powder, magnesium sulfate and diammonium hydrogen citrate were ineffective for bacteriocin Lac-B23 production. Therefore, a low-cost and high-efficiency culture medium for bacteriocin Lac-B23 production was developed. The results show that bacteriocin Lac-B23 production by


Undergraduate medical students seek as much information as possible as to how residency programs select candidates. The Canadian Residency Matching Service (CaRMS) website is one of their primary sources of information. Students may be more competitive in the match if they know whether scholarly activity is used in the selection process by their preferred programs, as described on the CaRMS website.For all 17 Canadian faculties of medicine, 2019 R1 entry internal medicine, family medicine and pediatrics program descriptions were reviewed on the CaRMS website looking for keywords related to scholarly activity.Forty-one percent of family medicine, 65% of internal medicine and 71% of pediatric programs explicitly stated having interest in applicants with scholarly experience. In Western Canada, 80% of internal medicine and 60% of pediatrics programs included scholarly activity in their CaRMS description of criteria considered in ranking applications. Similarly, in Ontario, 66% of internal medicine and 83% of pediatrics programs mentioned scholarly activity as a valuable quality. In Quebec 100% of family medicine and 50% of pediatrics programs include scholarly activity in their descriptions. Pediatrics and family medicine programs (100%) in Atlantic Canada mentioned scholarly activities but neither of the two Atlantic Canada internal medicine programs mentioned scholarly activities.Undergraduate medical students can use this project to prioritize extracurricular activities and scholarly work to be competitive for application to family medicine, internal medicine and pediatrics residency programs. The Canadian residency match has significant implications for medical students in terms of their future geographical location, personal lives, and specialty in which they will practice.6 This is accomplished by developing useful clinical and interpersonal skills while closely interacting with the speciality of choice6. Students who participate are more likely to attend conferences, which serve as opportunities for networking and can produce connections that open doors on the residency program interview trail. Additionally, scholarly activity pairs the student with a dedicated mentor. This connection itself can be instrumental in a student’s professional growth and development during medical school.A 2015 review of 20 published studies found scholarly activity such as research and quality improvement of health services experience increases the likelihood of a student matching into their residency of choice, regardless of what that may be.CaRMS is the corporation that runs the residency match. However, the application evaluation process that determines which candidates will be admitted to residency programs is carried out by each school’s medical program. CaRMS does not make decisions about program requirements or candidate selection criteria. CaRMS only receives the ranking data and runs the algorithm which ranks the applicants. Matches are made based on congruence of the ranking of candidates by the residency programs and the ranking of residency programs by candidates.7 it is important that candidates know which activities and attributes each residency program values most in their candidate. We do not know how many programs include participation in scholarly activities in their description to potential candidates as a desirable attribute.As provincially funded residency positions become more competitive,CanMEDS)8,9 considers scholarly activity to be an essential competency of practicing physicians. According to CanMEDS, scholarly activity should be a mandatory component of physician education in addition to the other seven core competencies: medical expert, communicator, collaborator, manager, health, advocate and professional. According to CanMEDS, scholarly activity involves being:Canadian Medical Education Directives for Specialists  reviewing published residency program descriptions for R1 entry for the 2019 match year on the CaRMS website. All 51 residency program descriptions were reviewed for internal medicine, family medicine and pediatrics in Canada’s 17 medical schools looking for specific keyword descriptors related to research and scholarly activity. The internal medicine and family medicine programs have the largest number of residency spots and pediatrics was chosen as the third program as an example of a smaller program. A review of the ethics of this study was not required as it was based on publicly available data from the CaRMS website.Residency program descriptions that specifically mentioned the keyword ‘scholarly activities’ or similar descriptors such as ‘applied health research’, ‘clinical research’, ‘quality improvement’ or ‘education research’ in their program descriptions were identified.Both French and English terms were captured. For those terms that had the same meaning in French and English, for example,’ research’ and ‘recherche’, only the English versions were reported. If French terms did not show up in the list of English terms, they were left in the list of all descriptors.The medical students were given an orientation about the goals and objectives of the project. They were instructed not to include the provincial criteria for selecting residents and criteria from the Royal College of Physicians and Surgeons of Canada and the Canadian College of Family Physicians.The students were also instructed on how to complete the Data Retrieval Form. The Form is a table with the cells used to record the descriptors.The students retrieved explicit descriptors of ‘scholarly activity’ reported in the CaRMS website for the residency programs assigned to them. The students worked in four pairs with each member of the pair first independently assessing four to five residency program CaRMS descriptions. The pairs then came together to discuss any discrepancies by comparing their list for each program description that they reviewed. These discussions between the pairs of students were used to resolve inconsistencies in their two lists of descriptors for each program they reviewed.The data were then analyzed to compile a list of all descriptors used in reference to scholarly activity and the frequency with which each descriptor appears in each residency program description on the CaRMS website. The data were summarized using a tag cloud (‘WordCloud’) which provides a visual representation of the language used to describe any requirements related to scholarly activity in the residency program descriptions. The tags are single words or phrases and the frequency of each tag (word or phrase) used by residency programs is shown with font size and colour.The frequency of scholarly activity-related language in each of the residency program CaRMS website descriptions was summarized by program and by region of Canada. This avoided having to reveal results from individual programs. We hypothesized that there would be a similar amount of variation in scholarly activity requirements across programs as in regions of Canada.6 ‘Quality and Safety’ did appear but ‘Quality Improvement’ as a scholarly activity did not appear in any of the program descriptions.The word cloud  of the family medicine, 65% (11 of 17) of the internal medicine and 71% (12 of 17) of the pediatric residency programs explicitly stated their interest in applicants with scholarly activities see .In five medical schools in Western Canada, as shown in This review of R1 residency program descriptions demonstrated that the majority of internal medicine and pediatrics programs and minority of family medicine across the country include scholarly activity in their descriptions of criteria considered in ranking candidate applications. This trend is reversed, however, in Quebec programs where all the family medicine, half of the pediatrics programs and one quarter of the internal medicine programs include scholarly activity as an attribute. The range of words used to describe scholarly activity was found to be broad and non-specific which makes it difficult for candidates to know which types of activities will be most highly valued and how they should present their interests and work to best meet the program expectations.The description of scholarly activities for select residency programs varied from program to program and generally lacked specificity. Some programs suggested an interest in research would qualify as scholarly activity. The completion of an MSc or PhD during medical school can be quite difficult or not possible unless the student is given special leave of absence or acceptance into limited MD/PhD program spots. Other programs mention graduate degrees as important demonstration of scholarly work. Indeed, some schools, such as the University of Toronto, encourage undergraduate MD students to enrol in a MSc or PhD program while studying medicine. Other undergraduate MD students may have an MSc or PhD before they begin studying medicine. Some residency programs are intentionally structured to allow for completion of graduate degrees during the residency and this may be a more productive or relevant time to pursue these studies.From this review of the 2019 CaRMS website, it appears that students would more easily be able to interpret what qualifies as scholarly activity if residency programs adhered to a uniform manner of describing scholarly activity in their CaRMS website outlines. This would be reassuring to some applicants who do not have this interest. However, the CanMEDS classification of scholarly activity as a core competency of practising physicians would demonstrate that training and skill in this area should be accomplished by all medical students. There also is a risk that a student will be disadvantaged in their residency applications by not having scholarly activity as part of their curriculum vitae when a program may be valuing these skills, despite not explicitly mentioning them in their online description. Without describing the range of scholarly activity that would be considered, including quality improvement activities or work in curriculum development, students may not consider the work they have done to be “research” and may have concerns they do not meet the program’s threshold for scholarly activity.CanMEDS, including scholarly activities.Thus, based on the review of the 2019 CaRMS website, it would benefit both applicants and residency programs to introduce standard program descriptions that include uniform descriptions of selection criteria for residency programs concerning all the essential physician competency requirements as outlined in This project supports the changes made in the 2020 CaRMS website. Residency programs are now required to report in the descriptions of their program on the CaRMS website whether “research/publications” are considered in their review of documents and whether ‘scholarship’ will be a topic covered in interviews of residents. While this has forced programs to be more explicit about these topics being considered in the admission process, the ways in which residency programs use this information can be the subject of further research.One limitation of this project is that it included only reviews of family medicine, internal medicine and pediatrics residency programs. Internal medicine and family medicine programs have the largest number of available positions in Canada. Pediatrics, a smaller residency program had a different value placed on scholarly activity in their selection criteria and this may be true with other smaller residency programs. Future reviews could examine other programs on the CaRMS website. There is a possibility that programs did not include a discussion of desired scholarly activity on their CaRMS online description despite valuing this in candidate ranking. This disadvantages students in their application. There also may be more detailed program descriptions available on the university website of each program and future studies could look at the concordance between the CaRMS program description and the university website description for individual residency programs.The results of this project support the revisions to the 2020 CaRMS website that now requires programs to comment on whether ‘research/publications’ are considered in reviewing applicant documents and whether ‘scholarship’ will be discussed in the applicant’s interview. Our project can assist students in organizing and prioritizing their MD training activities in a manner that aids them during residency applications. Also, the project results were forwarded to CaRMS and the Canadian Federation of Medical Students.Authorship: JL, AB, KS, SW and LWC contributed to the conception or design of this project VB, JB, DD, MG, HK, AK, DL and DT obtained the data for this project. Analysis and interpretation of the data involved all the authors. All the authors were involved in drafting the work or revising it critically for important intellectual content; All the authors approved the version of the manuscript to be published. All the authors take responsibility for being accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.


The focus of this session will be on the implementation of a new care management program in Chicago for low income older adults. Framed by the Exploration, Installation, Initial Implementation, and Full Implementation framework, the presenter will first discuss the process and outcomes of a needs assessment which informed the program. The presenter will share strategies and lessons learned from getting the program off the ground and initial lessons learned which informed the full program as it is operated today. In 2 years, the program has grown from 1 FTE to 4.5 FTE and to 250 clients. The presenter will provide strategies for managing growth while maintaining quality care. Finally, the presenter will provide information on the program's collaboration with a researcher to enhance evidence-based service delivery within the care management program. Participants will learn specific strategies they can take back to their own communities to implement care management programs.


First responders face multiple stressors on a daily basis. They have experienced higher rates of anxiety disorders, depression, burnout, post-traumatic stress disorder (PTSD), suicide , alcohol First responders face multiple stressors on a daily basis. For current law-enforcement officers, the National Fraternal Order of Police has described the present conditions as more stressful and difficult than it has been in more than 20 years . For a vHowever, despite this reality, we should not concede that this is acceptable or that these conditions are inevitable. Nor should there be an expectation that this situation needs to continue without efforts to enhance prevention and/or recovery. Fortunately, there is ample evidence in neuroscience, resilience, and mental health studies that certain practices can help people to develop greater mental strength, increased calmness, a greater sense of purpose, more connection with others, more realistic and positive outlooks on their lives, greater emotional control, and the ability to reappraise the situations they encounter to acknowledge the positive aspects of each, especially if the situations are ostensibly negative .This paper evaluates and outlines one specific approach to building first responder resilience and mental health. The first author developed the warr;or21 program specifically for first responders, and it has since become available to, and practiced by, the general public. The program, which will be discussed in more later in this paper, utilizes research in neuroscience, positive psychology, and resilience to provide first responders with daily practices over the course of 21 days that are practical and that specifically address the work they do. Each practice, including controlled breathing and gratitude, is evidence-based drawing from relevant research. The researchers are still collecting data on this specific program and its impact and as such, the program is still in the pilot stage. However, given that, the foundations of the practices and core concept of the program are science-based and the potential benefits of such a program are promising.It is clear that the current situation for first responders is dire; they repeatedly experience severe trauma . To thisCompared to the general public, first responders face an increased risk of distress, worry, sleep issues, difficulties in concentrating, adverse effects on personal relationships, increases in substance use, and depression . DepressPTSD affects between 15 and 30% of first responders , and it Due to the various stressors and mental health conditions that are associated with law enforcement work, police officers have an elevated risk of death due to cardiovascular disease, metabolic syndromes, and various cancers . The samReliable scientific data are sparse regarding first responder suicide rates, and Other reports have shown that emergency medical services (EMS) personnel tend to think more about suicide and to attempt it more compared to the general population . ResearcA report by the National Fraternal Order of Police in collaboration with NBC New York drew on nearly 8,000 responses from police officers across the United States . ApproxiOne approach that has been discussed in the literature  and thatAlthough limited in scope and depth, research on building resilience and positive mental health with first responders has shown some promising results. According to “The data suggest that training in resilience building and self-regulation skills could significantly benefit police organizations by improving judgment and decision making and decreasing the frequency of on-the-job driving accidents and the use of excessive force in high-stress situations. Potential outcomes include fewer citizens’ complaints, fewer lawsuits, decreased organizational liabilities, and increased community safety.”Beyond the benefits of an agency conducting this training, it clearly (and importantly) benefits the individual’s well-being.Road to Mental Readiness (R2MR), has shown promising results, with participants stating that it helps to reduce stigma and to build resiliency. They also found that the training was applicable in both their work and their personal lives (n = 14), found that resilience practices yielded positive results. The program consisted of first an introductory, 2-h session and then a 6-week program of individual learning via an app as well as four separate tele-mentoring sessions. In that study, the participants reported improvements with work and their personal lives. Similarly, a study of recruits in the Milwaukee Police Department (n = 34), produced promising results in developing resilience. This program involved a 2-h session, instructions on breathing exercises, as well as four tele-mentor sessions spaced out over approximately 3 months in total.Preliminary evaluations of training used in Australia and Canada, al lives . A studyal lives , althougpartment , also win = 143) suggested that “mindfulness-based resilience training delivered in an internet format can create improvements in adaptive resilience and related resources among high-risk workers, such as first responders.”A study conducted with first responders in Australia  who partFinally, two other programs of note are FBI National Academy Associates’ Officer Resilience training  and the As first responder agencies across the globe have embraced the fact that they must work toward the enhancement of the workforce’s well-being, mental health, and resilience, the issue that they are now faced with is how can this be achieved. The first author has developed a program that offers practices that are both practical and sustainable for first responders. The warr;or21 program is a 21-day set of various practices developed based on by previous research that seeks to achieve positive outcomes of increasing first responders’ inner strength, enhancing their resiliency, and in turn, increasing their positive mental health. The first author initially created the program for first responders, but it has also undergone adaptation, making it available to the general public.Each day over the course of 21 days, first responder participants receive a keyword to guide their practices for the day. Keyword examples include grit, calm, empathy, adapt, and gratitude. The program can work in various ways, but the primary approach is to give a 1.5-h introduction to the program through an in-person workshop and then to conduct the actual program through a private Google Classroom.During the pilot phase of the program, most cohorts participated through registering in a private Google Classroom. Upon receiving the classroom code, each participant received a daily alert on a mobile device for the day’s practices. One cohort received the daily material on their agency laptops in an electronic folder designated for the program. The majority of cohorts ran Monday through Friday, and thus the weekends did not contain any practices. During the program, participants also shared their thoughts and feedback with fellow participants by posting in the Google Classroom at different points during the 21 days. The program utilizes Google Classroom as it allows participants to access the program material at their own convenience as well as in a safe and private manner since the application had to be downloaded on their personal mobile device.The daily keyword served as the focus for each day and the daily practices followed the same format: a quote connected to the keyword, 5 min of a controlled breathing practice (instructions are available each day), a short reading detailing the importance of the keyword with links to further reading, a reflection for the day regarding the key points of the article, and an evening gratitude practice. The participants received a small black notebook in which each evening they wrote down their gratitude practice. Practicing gratitude can help to increase positive sleeping habits , and speThroughout the 21 days, the warr;or21 program focuses on four pillars of resilience: awareness, wellness, purpose, and positivity. These four, which are unique to this program, draw on previously established neuroscience and resilience research that has used similar terminology that is critical to developing resilience  and 20 females (32.8%), with a significant proportion of the sample not providing a response to this demographic question . Of the 61 participants, 36.0% identified as uniformed law enforcement officers, 19.7% were civilians, and 44.3% did not specify employment status.In terms of general global satisfaction with the program, the evaluation data reveal that program participants held very positive views of the program. Of the sample, 90% were very satisfied (60.0%) or satisfied (30.0%) with the program. Ninety-five percent of participants indicated that they would recommend the program to others in their profession. Comments made by participants included,the best thing I could have done for myself;”“it really helped me become stronger, more positive, and smarter;”“The warr;or21 program is a great tool for anyone but especially in law enforcement.”““it gave me a fresh start… it’s a line in the sand for me – myself before the course and after the course.”Participants were asked about changes in their attitude toward the war;or21 program at the completion of the program, compared to when they began the program. The majority of participants indicated that their view of the program at the completion of the program was much better (56.7%) or somewhat better (20.0%). Only one program participant indicated a negative view, indicating it was somewhat worse, however, they did not elaborate. Further, 57.1% of the sample indicated that the program exceeded their expectations and 41% indicated it had met expectations. One participant said, Data were collected relating to the perceived usefulness of the breathing exercises and gratitude practices that were taught as part of the program curriculum and the intention of participants to use these strategies following program completion see , below. …by the time I was half-way through I found myself doing the breathing exercises unconsciously.”Importantly, the majority of program participants indicated that they would continue to use breathing exercises and gratitude practices in the future. A comment made by one participant reflects how the breathing exercise can become integrated into their life, “In terms of program outcomes, participants were asked to rate how much change they perceived they had experienced as a result of undertaken the program. Perceived changes were assessed in respect to resiliency, gratitude, motivation, feelings of calmness, happiness, stress, control of emotions and actions, and overall mental health see .Based on the data, over 50% of the sample perceived substantial increases in their feelings of calmness (63.2%) and gratitude (51.7%). It can be concluded that positive changes were found across all program outcomes for the majority of program participants. When combining responses for “substantial increase” and “somewhat increase” ratings, ratings ranged from 57.2% of the sample (for reductions in stress) up to 90.8% .From the perspective of participants, the least change (feeling the same at the completion of the program compared to the start of the program) included feeling out of control (31.4%), feeling in control of actions (26.5%), stress (25.7%), and motivation (23.5). While 73.4% of the sample perceived positive changes to their overall mental health, just over a quarter of the sample (26.7%) also indicated no change.The results of the evaluation of an initial sample of individuals who undertook the warr;or21 program shows promising results. Overall the course was well-received by participants, and they were in the majority satisfied with the course, felt it had met or exceeded their expectations and would recommend the course to others. The specific activities, including breathing and gratitude exercises were both useful for participants while undertaking the program. Perhaps, most importantly, the majority of participants indicated that they intended to keep using the strategies that they had learned during the program. The results of this brief evaluation indicate that the program has most impact on feelings of calmness and gratitude. While the magnitude of change varied across program outcomes, it should be noted that positive changes were experienced by at least 50% of the sample, regardless of which program outcome was studied.The results of this preliminary evaluation are positive, however, between 18 and 31% of the participants, with exception of calmness and gratitude, did not perceive any change across a number of program outcomes. This does indicate that further evaluation of the program needs to be undertaken to ascertain the reasons why some individuals are not benefiting as much as others. This could include a review of program content and considering the mode of delivery of the content that may be more effective for some than others. Further research is needed to determine whether those that experience less perceived program success have specific characteristics that make them a unique population and what potential adaptation of the program could be undertaken for such groups.Research has shown that resilience practices, including controlled breathing, gratitude, promoting connectedness, and being able to reframe a situation in a positive perspective can help police officers and other first responders handle stressors associated with their work . The warAs the program seeks to expand within the first responder community and beyond, it also must be done strategically. For example, incorporating warr;or21 into existing recruit training curriculum should be explored as it can serve numerous benefits aside of developing resiliency. This includes countering the stigma associated with seeking mental health help .The warr;or21 program continues to grow, and further evaluation data are necessary to measure its effectiveness and impact. Further evaluations will seek to collect greater information on demographic characteristics to allow for comparative analysis, and the use of pre‐ and post‐ surveys to track changes before and after program participation. The growth of the program since writing this paper and the increase in the number of participants have been substantial. Since the analysis of the initial data for this paper, more than 600 participants have participated in the program. In addition, the program has undergone adaptation to allow non-first responder participants, and this includes people from diverse backgrounds, including physicians, attorneys, clinicians, salespeople, executives, students, retirees, public relations people, and those with other business-related backgrounds. The diversity of the participants includes their locations: Australia, Brazil, Canada, New Zealand, Spain, the United Kingdom, and the United States.Future development of the program needs to include is the use of a more sophisticated evaluation strategy to better understand the impact of the program. This program data derive from only the responses of participants upon immediately completing the program. Moving forward, evaluation should include, for example, continued follow-up to explore retention of the practices as well as pre‐ and post‐ surveys so that actual changes on key mental health outcomes can be determined, rather than relying on perceptions of participants.The program continues to develop to capture and analyze the data more effectively. This includes determining which current instruments are most effective, such as possibly the PERMA test , scales Data supporting the conclusions of this article will be made available by the authors, upon request.The studies involving human participants were reviewed and approved by Institutional Review Board, Lipscomb University. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.The two authors contributed to the analysis of the results and to the writing of the manuscript. It is original and has not been submitted elsewhere. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Hypertension and diabetes are among the most common and deadly chronic conditions globally. In India, most adults with these conditions remain undiagnosed, untreated, or poorly treated and uncontrolled. Innovative and scalable approaches to deliver proven-effective strategies for medical and lifestyle management of these conditions are needed.Integrated Tracking, Referral, Electronic decision support, and Care coordination (I-TREC) program. I-TREC leverages information technology (IT) to manage hypertension and diabetes in adults aged ≥30 years across the hierarchy of Indian public healthcare facilities. The I-TREC program combines multiple evidence-based interventions: an electronic case record form (eCRF) to consolidate and track patient information and referrals across the publicly-funded healthcare system; an electronic clinical decision support system (CDSS) to assist clinicians to provide tailored guideline-based care to patients; a revised workflow to ensure coordinated care within and across facilities; and enhanced training for physicians and nurses regarding non-communicable disease (NCD) medical content and lifestyle management. The program will be implemented and evaluated in a predominantly rural district of Punjab, India. The evaluation will employ a quasi-experimental design with mixed methods data collection. Evaluation indicators assess changes in the continuum of care for hypertension and diabetes and are grounded in the Reach, Effectiveness, Adoption Implementation, and Maintenance (RE-AIM) framework. Data will be triangulated from multiple sources, including community surveys, health facility assessments, stakeholder interviews, and patient-level data from the I-TREC program’s electronic database.The overall goal of this implementation science study is to evaluate the I-TREC consolidates previously proven strategies for improved management of hypertension and diabetes at single-levels of the healthcare system into a scalable model for coordinated care delivery across all levels of the healthcare system hierarchy. Findings have the potential to inform best practices to ultimately deliver quality public-sector hypertension and diabetes care across India.CTRI/2020/01/022723). The study was registered prior to the launch of the intervention on 13 January 2020. The current version of protocol is version 2 dated 6 June 2018.The study is registered with Clinical Trials Registry of India (registration number  The Government of India, beholden to its population of 1.3 billion, has developed an electronic “NCD Portal” that consists of an electronic case record form (eCRF) to manage non-communicable diseases (NCDs) within the government sector. The I-TREC program builds on the eCRF by integrating a proven-effective clinical decision support system for hypertension and diabetes care, accompanied by clinical training, to assist with patient management.We describe the evaluation protocol for the I-TREC multi-component strategy to improve diabetes and hypertensions care at all levels of the four-tier healthcare system in India.Lessons learned may inform optimal approaches to improve healthcare processes and health outcomes within the public sector healthcare system in India and in other similar settings.Hypertension and diabetes together affect over 275 million Indians and their families . These cIndia’s rural healthcare system is currently organized as a hierarchy of facilities that range from relatively lower-skilled personnel supported by simple infrastructure at the village level to relatively higher-skilled personnel supported by sophisticated infrastructure at the district level. This model attempts to maximize geographical coverage by allowing for “up referrals” and “down referrals” across levels of the healthcare system so that the demand of the individual patient can be met by appropriate resources, such as skilled human resources, infrastructure and services. The referral linkages between these institutions, while theoretically in place, are not implemented efficiently or cohesively. In practice, patients access any level of the healthcare system convenient for them, resulting in a mismatch between patient needs and resource availability. Challenges to the system are compounded by the heterogeneity of treatment guidelines, diagnostic modalities, and medications –17 needeIntegrated Tracking, Referral, Electronic decision support, and Care coordination (I-TREC) program. We describe the components of the I-TREC program and its evaluation design.Recognizing the growing burden of hypertension and diabetes across all segments of the population, the Ministry of Health & Family Welfare, Government of India, has taken the initiative to integrate screening and management of these conditions into primary care under its National Health Mission (NHM) and the National Programme for Prevention and Control of Cancer, Diabetes, Cardiovascular Disease, and Stroke (NPCDCS). A major component of the government strategy is to encourage universal screening for hypertension and diabetes of adults aged ≥30 years in the community and subsequent referral of potential cases to higher level facilities. Consequently, the expected volume of adults seeking care for hypertension and diabetes at government health facilities is anticipated to surge. Building upon the tremendous need and political will to identify optimal and scalable approaches to expand successful care models to manage blood pressure and diabetes within the public healthcare system, we developed the I-TREC was developed as a collaboration between the All India Institute of Medical Sciences, New Delhi (AIIMS), the Centre for Chronic Disease Control (CCDC), and Emory University. For over a decade, these three institutions have collaboratively developed and tested the combination of information technology (IT), enhanced personnel training, and workflow alterations to improve the quality of care for diabetes and hypertension in diverse settings across India –29. MostThe primary implementation partners for the program include the Department of Health and Family Welfare, Government of Punjab ; TATA Trusts ; and Dell Technologies (development of software and IT infrastructure). Monitoring and evaluation activities for I-TREC will be conducted by AIIMS, CCDC, and Emory University. I-TREC will be implemented in Mukandpur block of Shaheed Bhagat Singh Nagar district, Punjab, India, and evaluated through comparison of program indictors with those observed in the neighboring Sujjon block in the same district See Fig. . The proMany barriers to optimal hypertension and diabetes care can be alleviated through IT-based quality improvement strategies. Clinical decision support software can provide up-to-date guidance to clinicians  to managMotivated by the potential benefits IT tools offer clinicians and health systems, the I-TREC program includes: an electronic case record form (eCRF) to consolidate and track patient information; an electronic clinical decision support system (CDSS) for clinicians to provide tailored guideline-based care to patients with in-built prompts for triggering referrals across health facilities as needed; a revised workflow to ensure coordinated care within and across facilities; and enhanced training for clinicians regarding NCD medical content and lifestyle management . We chose to build upon the Government of India’s eCRF to align our program with the national effort to incorporate IT into the management of NCDs in the public sector.The CDSS generates customized evidence-based treatment advisories for patients with hypertension and diabetes. The treatment advisories are based on up-to-date national and international guidelines that were further vetted by our expert clinical investigators, and tailored to each level of health facility , for example, by taking into consideration the local availability of medications and diagnostic capability. The CDSS algorithms provide the clinician with an instantaneous advisory regarding medication titration based on patient history and current clinical examination as inputted into the eCRF. The attending clinician has the option of rejecting, partially accepting, or fully accepting the advisory to generate a final treatment plan. In addition to the treatment plan, the CDSS has in-built prompts to refer patients up or down the healthcare facility hierarchy to direct patients to the most appropriate level of care for ongoing disease management. The treatment plan, referral instructions, and lifestyle advice specific to the patient become a part of the patient’s eCRF and are also printed out on paper and given to the patient.Healthcare providers employed in both the program and comparison blocks receive refresher content training related to the etiology, behavioral counseling, and medical management of NCDs following established NPCDCS training manuals. The content training is provided over a full day in separate sessions for auxiliary nurse midwives [ANMs], staff nurses, and medical officers. The sessions include training on effective techniques for delivering behavioral and lifestyle advice counselling. Unlike routine training, this refresher training includes innovative learning methods, such as case studies and role playing to enhance trainee engagement to assure improved comprehension and retention of behavioral and lifestyle counselling approaches. In addition, staff nurses and medical officers in the program block receive IT training on the use of the eCRF and CDSS, specific to their level of expertise and the level of healthcare facility in which they are employed.Figure At Primary Health Centres and above—namely Community Health Centres and the District Hospital—nurses generate and update the patient eCRF through a web-based application on a computer tablet. At the time the eCRF is first generated, nurses record the patient’s clinical history. At future visits, the eCRF is updated with ongoing examination data so that the CDSS is responsive to the patient’s health status at a given visit. Patient data entered into the eCRF are uploaded to a secure cloud-based server once per day, and these data are synced and retrievable at all facilities to inform the CDSS and assist with clinical decisions.After the initial eCRF review and update by the nurse, patients with confirmed hypertension or diabetes are instructed to see the medical officer, who is aided by the CDSS to manage these conditions. The CDSS algorithms are tailored to the expertise, medications, and diagnostic tests available at each level of facility and customized to the clinical history of the patient over all past and present contacts with the health system. Medically complex patients, such as those who are resistant to therapy, may be referred further “up” the referral hierarchy. The referral algorithms take into account a patient’s full clinical history and current health profile, including the number of medications currently prescribed, treatment response based on laboratory investigations, and comorbidities. For example, if a patient with diabetes under treatment at a PHC has uncontrolled hyperglycemia despite being on the maximum tolerated dose of three oral hypoglycaemic agents, the CDSS will trigger a referral to a CHC. Similarly, if a patient with hypertension under treatment at a CHC level has uncontrolled blood pressure despite being on the maximum tolerated dose of two antihypertensive drugs, the CDSS will trigger a referral to the District Hospital.Once the patient achieves a stable clinical state, she or he will be referred back “down” to the lowest level of health facility (PHC or CHC) that is suitable for routine management of stable disease and dispensation of appropriate medication.Adults residing in the comparator block will continue to receive the usual care by local physicians and nurses using paper-based record systems and without the assistance of the CDSS.We will employ a two-group pre-post quasi-experimental design to conduct a mixed methods evaluation of the I-TREC program in Punjab, India. While the I-TREC program is implemented within the health system by clinicians, we will assess indicators at the levels of facilities, clinicians, patients, and communities.The evaluation protocol was reviewed and approved by the ethics committee at All India Institute of Medical Sciences (AIIMS), New Delhi, India (IEC-361/07.07.2017). Given that the Government of Punjab will be implementing I-TREC, and role of research partners is limited to program design, training, and evaluation, this study was deemed to be observational. The role of researchers at Emory University, Atlanta was deemed Not Human Subjects Research (IRB00098808). Participants from whom our research team collects data for evaluation purposes will provide written informed consent following procedures approved by the AIIMS Ethics Committee.Program indicators are guided by the Reach, Effectiveness, Adoption Implementation, and Maintenance (RE-AIM) framework . The priData will be triangulated from multiple sources, including facility and patient assessments, stakeholder interviews, community surveys, and patient-level data from the I-TREC electronic database. Where appropriate, data will be collected prior to the program launch and again following 36 months of the program. With the exception of the I-TREC eCRF-CDSS data, evaluation data will be collected by trained research staff.Pre- and post-program health facility and patient assessments include 1) a health facility form; 2) patient flow mapping; and 3) patient out-of-pocket cost of care surveys. Health facility forms will be completed at all 52 government health facilities  to describe the infrastructure, facility personnel and salaries, availability of medications, and availability of diagnostics and laboratory investigations. Health facility form completion requires a combination of observational checklists and structured interviews with administrators. Patient flow mapping entails identifying and following patients with hypertension and diabetes through typical visits to map the typical workflow, diagnostic and prescription practices, and duration of visits. Together, the health facility form and patient flow mapping will provide data to describe the resources (time and costs) associated with typical healthcare visits for patients with hypertension and diabetes with and without the I-TREC program. Patient cost surveys will be used to obtain data on expenditures related to outpatient and inpatient health care utilization in the last 3 months to understand the cost incurred by patients to manage their disease. Administration of the pre- and post-program patient cost surveys will contribute data to understand whether the program has any impact on patient expenditures related to hypertension and diabetes. Purposive sampling will be done to recruit patients for the patient flow mapping and cost surveys.Qualitative methods will include a combination of focus group discussions of the community members, key informant interviews with healthcare providers and in-depth interviews of patients to provide a richer interpretation of quantitative findings and explore the processes underlying the uptake and delivery of the I-TREC program. The qualitative research will be conducted before, during, and after program implementation. Purposive sampling will be done to recruit information-rich participants for interviews and focus group discussions. For all qualitative data analysis, the textual data  will be reviewed to identify key themes and domains of interest. A code book will then be developed to reflect these domains and include both inductive  and deductive (based on literature and theory) codes. Inter-coder reliability will be assessed, and the codebook will be finalized and applied to the data. The codebook will include codes specific to each type of data collection and shared codes across participant type. A thematic analysis will be used to describe individual- or community-level views on discussion topics including program barriers and facilitators, community barriers, views of the healthcare system, and acceptability and feasibility of the program.The community-based evaluation component will assess whether the I-TREC program has an impact on blood pressure and blood glucose awareness, treatment seeking, and control among adults in the community. This evaluation component is critical to learning the real-world impact of the I-TREC program on community-level indicators of the care continuum . Data collected in the program and comparison blocks prior to the program will be compared with data collected from these same blocks after the program using identical procedures. This design allows us to assess and address several threats to validity, including lack of temporal order, comparability across the two blocks  and secular changes unrelated to our program that affect study endpoints . Given that we will be sampling separate cross-sections of the population in each group and time point, we do not expect inference to be affected by population aging (maturation threats) over the 3-year program period.At baseline—prior to intervention—we employed a multi-stage cluster sampling design to obtain a representative sample of adults aged 30 years and older in both blocks under study. Within each block, census data were used to select villages proportionate to population size and subsequently we conducted household mapping and listing to generate a sampling frame for households. Households were selected using systematic random sampling, and one adult man and woman from each household were randomly selected using the Kish method to achieve the desired sample size. At endline, this same procedure will be repeated.The community survey sample size was determined to estimate differences in mean reduction in systolic blood pressure among those with diagnosed hypertension in the community. First, we computed the base sample size required to detect a desired effect size of 5 mmHg given the SBP standard deviation of 18.5, power = .80 and α = .05, based on the mPower Heart Study . We estiUsing patient health data from the eCRF, we will assess processes of care and changes in blood pressure and blood glucose outcomes over time among patients with hypertension or diabetes who seek care at I-TREC program facilities in a facility-based evaluation component. Because I-TREC is being integrated into the routine care in the program block under real-world conditions, we will not be assigning individual patients to treatment nor actively following patients for research visits. Rather, all adults residing in the I-TREC catchment area  will be exposed to the program and patient data will be collected every time a person chooses to receive care at a government health facility. Data from all patients visiting facilities in the program block captured in the Government of India eCRF will be de-identified and obtained by AIIMS throughout the program period for monitoring and evaluation purposes.The sample size for the facility-based evaluation is out of our control and contingent on the number of patients who seek care at government facilities. We therefore report the detectable effect size for longitudinal change in systolic blood pressure over time in patients at I-TREC facilities after setting power to 80% and α = .05. The program block, Mukandpur, has a population of 98,000. We expect that 30% of the local population will seek care at a government facility, 50% will be age-eligible (30 years and older) per the government guidelines for universal and opportunistic screening of hypertension and diabetes, and 20% will test positive for hypertension, amounting to an estimated patient pool of 2940 adults. Assuming that 50% of all enrolled patients with hypertension and diabetes make repeat visits (enabling us examine changes in outcomes), we will be able to detect a 1.35 mmHg difference in SBP.In the program block, we will examine measures of adoption and implementation of the IT tools, such as completeness of eCRF forms, acceptance  and rejection of the CDSS advisories, time stamp of data entry, the initials of the enterer, and average number of new records per day. In both the program and comparison block, data regarding the total number of patients recorded in the out-patient registry at the facility, numbers screened for hypertension and diabetes, numbers receiving medication from the pharmacy, and numbers referred to higher level facilities will be collected through a combination of paper-based registries and routine NPCDCS reports. The I-TREC evaluation team will obtain these facility-level data using abstraction forms without removing any paper records from premises. In addition, the I-TREC evaluation team will periodically conduct random, unannounced visits to directly observe the number of patients seeking care for hypertension and diabetes facilities in both blocks. Additional data on intervention fidelity measures  will also be collected through patient exit interviews and the eCRF backend data.g subscripts group ; and t subscripts the time point of data . We will estimate log-binomial models (binary outcomes) or linear models  with robust variance to compute the DiD after accounting for compositional characteristics of the community and clustering of data within villages. For each outcome indicator separately, the following model will be estimated using individual-level data:Quantitative data analysis will be performed using SAS, STATA, and R software. Descriptive analyses of the community-based data will examine socio-demographic characteristics, health indicators, and healthcare behaviors of the program and comparison block samples at baseline and end-line. The quantitative evaluation of health and healthcare endpoints will focus on assessing changes in the continuum of care indicators and mean blood pressure in the community-based surveys. We will assess changes in baseline to end-line indicators of health outcomes  and changes in continuum of care indicators  for both the program and comparison blocks; see Table The coefficient associated with the interaction term, “program group x pre-post indicator,” is the adjusted DiD estimate accounting for heterogeneity in socio-demographic characteristics. The model will be estimated using generalized estimating equations (GEE) to account for clustering of outcomes within the villages . Sub-group analyses will examine differences by gender and socioeconomic status.Data points recorded in the I-TREC system will be analyzed by month to examine trends over time and seasonality. We will also evaluate change patient outcomes over time . While several of the I-TREC platform indicators are purely descriptive measures of performance , other indicators of healthcare delivery may be compared between I-TREC facilities and comparison group facilities .This is the first study in India evaluating the composite impact of a clinical decision support system integrated with the Government of India eCRF, combined with modified patient flow and enhanced healthcare provider training. It is also an important early effort to systematically evaluate a program for integrated management of hypertension and diabetes at all levels of the public healthcare system, starting from the sub-centre up to the district hospital. Lessons learned may inform optimal approaches to improve healthcare processes and health outcomes within the public sector healthcare system in India.The I-TREC program and its evaluation have several strengths but also some limitations. Given the primacy of scalability, the role of research staff is limited to program design, training, monitoring and evaluation. Therefore, the context and conditions of implementation are beyond the control of investigators. For example, the availability of drugs, diagnostic investigations, and clinicians are likely to impact the reach, effectiveness, and implementation of I-TREC but rest in the hands of the state government. Nevertheless, we expect variations in these structural elements to affect both the program and comparison blocks similarly. In order to maintain comparability between the program and comparison block, neighboring blocks within the same district were chosen. However, this means that both blocks share the district hospital, which will have to be taken into account during the analysis. While we will be able to obtain patient data in the program block from the eCRF, no comparable source of data is available to use in the comparison block. Moreover, undue monitoring of the comparison block may inadvertently lead to compensation behaviors on the part of clinicians that could undermine our ability to measure performance differences across the two blocks. The community survey, in part, is designed to mitigate these limitations by providing a well-designed comparison of healthcare processes and outcomes as observed in the program and comparison blocks.This implementation research is intended to provide evidence of workable programs to manage chronic diseases in India and inform the evolving NPCDCS . Given tIntervention development and pre-testing were completed in August 2019. Healthcare providers in both blocks received training in December 2019, and the I-TREC program was launched in January 2020. Prior to the program launch, pre-program data collection, including facility and patient assessments, qualitative research, and community survey, were completed. Since March 2020, both intervention roll-out and research activities have been impacted by the COVID-19 pandemic. Specifically, government healthcare system resources have been diverted from NCD care to test and treat patients with SARS-COV-2, national and local lockdown measures have forced intermittent closures of lower-tier health facilities and prevented field staff from conducting routine monitoring activities. In addition, patient flow through facilities—when open—has generally declined, possibly due to fear of contracting the virus while seeking healthcare. Nevertheless, as and when healthcare facilities are operational, the intervention components are being implemented by nurses and physicians in the program facilities and program monitoring activities are underway. We expect intermittent disruptions to intervention implementation and monitoring activities to continue until the COVID-19 pandemic has been fully controlled.


Professionals who work with grandparents raising grandchildren have cross-cutting training needs that span content in gerontology, social services, child welfare and program development. To address these needs, a unique, asynchronous, online continuing education program was launched by the UMaine Center on Aging. To-date the program has 177 individual program completers with learners from across the U.S. and Hong Kong that are affiliated with a diverse set of organizations and perform a wide range of professional and lay functions. Participant data indicate that the program has appealed to a wide variety of learners including participants who serve caregivers generally (60.8%) and grandparents raising grandchildren specifically (81%). A small majority (55.6%) of the agency-based learners reported serving, on average, more than 40 grandfamilies annually. Self-reported learning levels were notable ranging from a mean low of 3.46 out of 4 points  for the volunteer recruitment and mentorship programming module to a mean high of 3.79  for the caregiver self-care module. Evaluation results from the first seven learner cohorts underscore the efficacy of program content as well as the utility of performing an initial program needs assessment to guide curriculum development. Practice implications for future continuing education efforts targeting grandfamilies professionals and lay leaders include: the need for easily accessible online education in combination with supplemental training opportunities addressing topics such as the long-term impact of substance use disorder and trauma combined with locally relevant content on grandfamilies and legal resources.