Designed in 2012 with a first implementation in 2013, NE STEM 4U is a professional development program for post-secondary students/undergraduates, and serves as a source of outreach, content knowledge generation, and STEM literacy for youth in grades kindergarten through 8th grade (ages 5–14). The model empowers post-secondary students as facilitators of inquiry-based learning within the context of an out-of-school time program. This study investigated the potential for replicating or ‘franchising’ this model by evaluating on the following: (1) Is the model replicable? And, if so, (2) what core elements are necessary for program fidelity? And (3) is there a dependency on a particular setting/participant type ?n = 16 focus group participants, n = 12 reflection surveys). These data indicated that Flexibility (21.22%), Student Engagement  (19.53%), Classroom Management  (19.31%), and Communication (15.71%) were the themes most referenced by the post-secondary student mentors in the NE STEM 4U program, regardless of site. Finally, the YPQA results demonstrate general replication of program quality in a “franchise” location.Strategic expansion of the program to different institutional types , different geographical locations , and with various school district partners  determined that program fidelity and replicability required 4 core elements or criteria: (i) intentional programming, (ii) staff quality, (iii) effective partnerships, and (iv) program evaluation and continuous improvement. Importantly, we examined emergent themes by each site, as well as in combination (These results highlight the core elements of the NE STEM 4U program for consideration of expansion (through strategic replication or ‘franchising’) as a possible international model. The findings and voices highlight the program’s trajectory toward success into environments that expand professional development for post-secondary students, and for delivering STEM opportunities for youth.The online version contains supplementary material available at 10.1186/s40594-021-00320-0. In the race for talent, all components of the workforce require access to highly skilled students earlier and earlier in their educational pathways  was founded in 2012 as a post-secondary student led, faculty-guided program. The program is designed to dually serve post-secondary student mentors as a pre-professional training program, regardless of major while also providing an afterschool education experience for youth using an inquiry-based learning approach.R2) designation. Previous studies documenting the impact of the NE STEM 4U program on youth outcomes showed evidence of gains in preferences for STEM careers among youth in 4th through 8th grade, as well as better preparation and persistence in STEM paths among college students  and post-secondary  student preparation and retention, we created the program NE STEM 4U  sought to adopt the NE STEM 4U model . The term franchise, although nearly synonymous with replication in this context, also represents that the replicating institutions share in funding the program, provide oversight, use shared lesson plans, and, to some degree, work collaboratively across replicating institutions, albeit at the time of this manuscript all replicating institutions reside in the same state within the United States. However, the concept is similar to what a business franchise might do. The ultimate goal for the initiating/founding institution, as well as the franchise sites, is to develop a strategy, process, and dissemination practice to establish a national network of NE STEM 4U programs.The NE STEM 4U program provided many positive outcomes for the originating institution in 8 years of programming thus far and is well positioned to continue. Based upon the success to date, leaders at campuses in the same university system . These activities make complex STEM concepts relatable and engaging. The mentors provide this programming in teams of 2–3, and they travel to the site of our participating schools in the out-of-school time hours  to host programming for 1 h, two times weekly throughout the year. Even during the COVID-19 pandemic (at the time of writing this manuscript), the program has not paused, it shifted to simply remote instruction. The mentors are given extensive, and consistent training to provide this instruction—namely, child safety training, culturally relevant pedagogical training, preparation with the lesson plans that they will carry out via mentored practicing and then practicing more as a team before delivery, and other timely topics via “STEMinars” to include Youth Voice, Youth Leadership, and modeling of high-impact practices through established assessment training including the David P. Weikart Youth Program Quality Assessment (YPQA) instrument, and Dimensions of Success  in the program can participate in research experiences within the NE STEM 4U program, in addition to teaching. These research studies involve mixed methods approaches and human subjects, similar to the social sciences, thereby providing research experiences for traditional STEM majors in areas that are separate from those that they might experience in a course or other apprenticeship style research experience . Specifically, undergraduates participate in discipline-based education research , which combines the knowledge of teaching/learning with discipline-specific content, further expanding their preparation to include the cognitive sciences and qualitative work, a complement to their (often) quantitative heavy degree plans.Moreover, the translation of these quantitative and qualitative skills, and the certifications and training needed to work with human subjects have helped graduates in job-seeking, graduate applications, grant applications, and have supported many mentors to be co-authors on presentations, and publications. The uniqueness of bringing translational research opportunities surrounding education, cognitive, and human subjects research have added another element of uniqueness to this program, when compared with others, and are directly in line with national calls to action NAP, .gain mentorship and give mentorship, which makes this program highly unique. The mentors gain mentorship from near-peers within the institution, and have direct access to faculty advisor support, to include frequent (at least weekly) meetings, advising, assistance with career planning, and ultimately letter preparation with letters being able to speak not only to say performance in a class, but truly to character development and the candidate’s abilities surrounding critical thinking, problem solving, leadership, collaboration, communication, and professionalism, due to the demonstrated actions of these within the NE STEM 4U program. Finally, the mentors also provide mentorship to the youth in the program, helping to serve as role models for them and helping to support their curiosity in STEM while simultaneously expanding their STEM content knowledge  is the model replicable? And, if so, (2) what core components are necessary for program fidelity? And, (3) is there a dependency on a particular setting/participant type  for program fidelity?We used several strategies to identify the core contributions that ensure program fidelity and replication, and particularly within a context of franchise-like co-ownership. First, we interviewed post-secondary mentors at both the original and the franchise NE STEM 4U locations. Second, we gathered data from faculty mentors to provide validation and context for interpreting what was learned from the mentors. Finally, we interviewed replication site leads to situate the work within their unique perspectives, goals, and experiences during the scale-up.Guided by the emphasis on perceptions, perspectives, and reflexivity in our implementation science, normalization process framework, we used qualitative sources of data  from each study site to gather data. We asked participants to describe their experiences and what they thought impacted the successes or challenges affiliated with franchising (replicating) the NE STEM 4U program at their location. We chose the term ‘replication’ or ‘franchise’ to capture the idea that there are core elements of the program that are essential for fidelity, yet each site had unique reasons to adopt the program, support it with significant ownership, and adapt it to be most relevant to local needs. As mentioned, we feel that this approach loosely parallels commercial franchises that must meet certain criteria and establish a replicated environment to be considered a franchise but also have some options that they can modify to adjust to local markets.Semi-structured interview guides with open-ended questions provided consistency and flexibility for the focus groups and interviews. The goal was to elicit students’ experiences and feelings to understand their perceptions of the program Lester, . The parOne possible barrier to adopting an existing program (becoming a franchise) is concern that it will not be the same program if the franchise makes some minor changes for the local setting. Importantly, we established nonnegotiable core elements of the program to promote fidelity while highlighting strong communication between the initiation and franchise/expansion sites to successfully scale-up. This includes defining assessment, evaluation, and programming outcomes. Ultimately, the initiation site has served as the approver of any programmatic modifications, as a way to ensure that the initial program was indeed being replicated (rather than an evolution of it) similar to how commercial franchises are operationalized. However, we set out to see if the program was robust enough to tolerate minor adaptations to meet local needs and still preserve the core elements of the initiating. For example, minor changes would include the content of lessons delivered, and leadership structure . In short, we sought a relatively broad notion of “fidelity” that could capture the essential components to reach the major goals while also allowing for some degree of localized customization.The NE STEM 4U program engages a diverse group of stakeholders, participants, collaborators and contributors, to include those within local non-profit and school leaders. Consequently, to gain feedback on the effectiveness and challenges of franchising the program, we completed semi-structured interviews and administered surveys to core participants from all three sites. The founding or initiating institution (Research 2 (R2)) is in a large , urban, metropolitan area and has about 15,000 post-secondary students. The Research I (RI) institution replication site is also in a metropolitan setting, but in a more modest sized city , has about 25,000 students, and it is designated as research intensive PhD granting institution. Finally, the predominantly teaching focused (PUI), research active institutional partner replication site is located in a smaller town  in a more rural area.n = 16 focus group participants total, n = 12 participants from the founding institution, n = 2 participants from the RI replication site, and n = 2 participants from the PUI replication site; n = 12 survey reflections from the PUI replication site) , we asked post-secondary students  in the program to participate in semi-structured phenomenological focus group interviews that lasted between 30 and 40 min or reflection surveys of similar content in early 2020 from all three replication sites , and combined  to inform the study  and reflection surveys (n = 12) were coded to include positive and negative nodes to identify preliminary emergent themes. After identification of emergent themes, we examined the coverage of these coded nodes using NVivo 12  for Mac. These data included frequency, coverage of the text, and alignment between participants within and across geographic location, respectively. Specifically, these data included frequency across the interviewees, controlling for response length. Taken together, the emergent themes  instrument from the David P. Weikart Center for Youth Program Quality to determine quality of programming , ranging from DoS to YPQA to represent the overall fidelity of the program from learning environment through youth voice.Similarly, we captured the number of youth attendees per instructional experience, an average of 6.9 attendees (range 2–20 attendees) per experience for Fall 2019 sessions, and recognize the danger of too much emphasis on purely quantitative metrics. Attendance can reflect many factors that are out of the control of the program in out-of-school time programs. For example, student illness, early pick up of youth for doctor’s visits, extra homework help, and frequently, competition with sports and extracurricular activities or not can shape attendance numbers. Rather, we implemented observations using established evaluation rubrics  write a summary of their experiences. These voices from the field were implementer observations from those that franchised the original program at their own location. These voices provide contextualization to the goals of the individual sites and regions while also identifying specific challenges and opportunities as a result of the franchise, distinct from the initiation site. We asked the three leaders to describe their vision of the program, why they wanted to replicate the program, and key challenges that they had to overcome to host NE STEM 4U at their location Table . These squality of out-of-school time programming and presence of core elements as a proxy for the ability to replicate the program. The David P. Weikart Center for Youth Program Quality instrument provided consistency and assessor calibration so that the main variable was original for franchise location. The YPQA results (Table Supportive Environment (4.51) and Interaction (3.13) compared to the original large, urban, metropolitan university (R2) were mostly similar . The replication sites had lower scores for Engagement (2.21) compared to the origin site (2.99), indicating a need for potential improvement. Even though implementation happened over multiple sites and years the quality of program delivery was mostly similar. The franchise sites maintained core program components (namely: (i) intentional programming, (ii) staff quality, (iii) effective partnerships, and (iv) program evaluation and continuous improvement). Of note, the ‘program evaluation’ component is characterized using the same assessment instruments as opposed to the same evaluator—i.e., using the YPQA and DoS instruments. Ultimately, the evidence indicates that it is possible to replicate the program and maintain quality. We explore the core components more below.Stated another way, can we take the initial, core program and ‘franchise’ or replicate it within a different context, with other program leads, and in an environment with a mission distinct from that of the initial site? To address the first research question, we utilized ts Table  demonstrWe now review some quantitative and qualitative components related to fidelity.n > 150 to-date), indicated sufficient demand for the program, as well as demonstrated buy-in from the post-secondary students. Because the original NE STEM 4U program was primarily about the professional development of post-secondary STEM majors, the ability to keep recruiting mentors is an important sign of fidelity.We interpreted that the sheer length of time a partner site was still running the program  indicated successful replication. Similarly, the recruitment of mentors  maintains an attendance sheet that provides information on how many youth members participate in clubs. The YPQA evaluation observation days each had at least 6 participants, and some had as many as 20 youth participating in NE STEM 4U on any given day. Our suggested cap on attendees is 15, thus whenever we met or exceeded this number, we interpreted this as demand for the programming. Several factors  could influence attendance so that the measure is suggestive but not definitive.To be a “franchise” or replication site with co-ownership, a location had to have post-secondary students go through training similar to the original location , use activity plans provided by or modeled on the original program , allow for evaluation , focus on community building and professional development of mentors, and develop and maintain positive relationships with Community Learning Center staff (‘effective partnerships’) Fig. .Slight variation in approach, however, was approved by the originating team. For example, one franchise had weekly clubs, where another hosts twice-weekly clubs. One site includes 4th and 5th grades and others only middle school grades. One NE STEM 4U program operated as a student run organization (temporarily) and another was funded by and led by a federal grant with heavier faculty and staff oversight. Given these baseline characteristics, the question that remained was this: can the franchises reach the level of quality or respond to feedback to achieve the level of quality of the original site? The YPQA evaluations suggest that the franchise teams could use the key elements  of NE STEM 4U to create engaging and enriching programs at each location.The data from post-secondary student mentor focus groups and reflection surveys Fig. , Table 3n = 16 focus group participants, n = 12 reflection surveys). These data indicated that Flexibility (21.22%), Student Engagement  (19.53%), Classroom Management  (19.31%), and Communication (15.71%) were the themes most referenced by the post-secondary student mentors in the NE STEM 4U program, regardless of site  19.5%, Classrn = 12 focus group participants): Flexibility (23.18%), Professional Development (20.23%), Classroom Management (18.75%), and Networking (16.26%). Importantly, the percent coverage of specific software nodes to articulate emergent themes reflects consistent responses across participants—not that only four of the twelve participants mentioned ‘flexibility’, for instance. Participants from the RI institution replication site generated the following themes (n = 2 focus group participants): Communication (25.57%), Flexibility (21.01%), Collaboration (18.29%), and Mentor Support (15.81%). Importantly, at this franchise site, given its early stage, there were only four mentors; thus, this focus group represents half of the mentors at this site (n = 2). The limitation of responses here are simply because of the short duration of replication at this site thus far. Participants from the PUI institution replication site described experiences that we codified as : Student Engagement  , Flexibility (19.46%), and Pedagogy (18.70%).Participants from the founding institution identified the following key themes , have their own funding source (unique of the initiation site), and have their own evaluation and programming, including new content foci and development  Intentional Programming, (2) Staff Quality, (3) Effective Partnerships, and (4) Program Evaluation and Improvement. Specifically, program success depends on the fidelity of specific aspects of these four elements, while other aspects are more flexible depending on site limitations and preferences. In addition, we have discovered that evaluation and improvement is particularly critical for effective lesson plan design, as this real-time feedback helps novice mentors to facilitate a positive, active learning environment, and thereby maintain quality, youth engagement, and youth voice . Moreover, the process of determining if a franchise is maintaining fidelity assisted the initiation team with determining what is ‘essential’ and what is ‘flexible’ within the programmatic replication.Through our assessment of NE STEM 4U  we have identified four success:  IntentioQuality,  EffectivThe major takeaways based on our observations herein, include the need for “win-win-win” scenarios for franchisees with the founding program and with all NE STEM 4U programs with partners . These scenarios are predicated upon an ongoing mutual respect with continuous communication. Namely, in the NE STEM 4U program, we are creatively filling multiple needs: (1) offering professional development for post-secondary students to include teaching, research, and mentorship, (2) providing K-8 participants  afterschool supervision, progressing their learning of content knowledge , post-secondary mentor participants’ key takeaways, and lead faculty perspectives and reflections. Overall, we aimed to determine, via implementation science, NPT framework with dissemination, insights about the program at the original location and at the franchise locations from the perspectives of the post-secondary/undergraduate student mentors.Flexibility (23.18%), Professional Development (20.23%), Classroom Management (18.75%), and Networking (16.26%). These data suggest that the participants have a clear understanding of the support that the NE STEM 4U program provides for professional development. For example, they see the benefits of learning to communicate with others, classroom management, and networking. For the metropolitan cohort, “peer networking” (10.98%) is important (the 4th theme) and suggests the community of mentors provides additional value. Gaining insights from the perceptions of the mentors adds to other data on the metropolitan mentors about the benefits of participating in NE STEM 4U for outcomes, such as critical thinking gains and retention in major and to graduation. There is a strong track record of degree completion (97%) and retention in a STEM career post-graduation , Flexibility (21.01%), Collaboration (18.29%), and Mentor Support (15.81%), as integral to NE STEM 4U. The research institution NE STEM 4U franchise just completed 2 years and had no mentors carry over from the first semester to the second semester, due to the cohort of student mentors being all seniors and thus graduated at the completion of that year. Through conversations with the founding NE STEM 4U team, the franchise realized that to build a professional community of mentors that will carry over semester to semester, there is a need to start with students earlier in their career  and to focus on professional development and community building among the mentors. Therefore, from the second implementation onward, sophomore-level students were included. However, of the participants, the mentors (who had participated for one semester) valued collaboration and direct support via training. We utilize the ‘train the trainer model’, and collaborative professional development workshops via Zoom (video conference) and FlipGrid™ in an effort to build community and collaboration across sites. These are publicly available non-negotiable of the dissemination of the intervention. Similarly, faculty advisors  serve as daily points of contact for post-secondary mentors, further giving them a community in which to belong and be confident in trying new things, such as outreach programs.The participants at the first expansion site, a research-intensive, flagship institution  identified Student Engagement  , Flexibility (19.46%), and Pedagogy (18.71%). It is interesting that given the teaching charge of the institution, the identified core themes are consistent with the mission. Finally, despite a unique geographical location , participants identified the same themes (flexibility and classroom management) as participants at the urban institutions, and suggesting minor influences from the institutional norms.The predominantly teaching-focused, research-active institutional partner in a smaller town  had student participants describe None of the data collected and analyzed contradicted the core elements of quality program replication identified by other programs seeking to grow . We connect mentors with in-service teachers, informal educators, active researchers in science education, policy makers, administrators within research and development, intellectual property professionals, and data analytics scientists, hopefully also opening the door to the array of fields available in STEM disciplines.In conclusion, we provided the core elements for fidelity of the replication of the NE STEM 4U program, as well as emergent themes from participants, from its initiation site to that of its expansion sites. We report four core elements that ensure the fidelity of our program are: (1) Intentional Programming, (2) Staff Quality, (3) Effective Partnerships, and (4) Program Evaluation and Improvement. Moreover, we identify emergent themes across all sites to include: Finally, and perhaps even more importantly we demonstrate through normalization process theory that this longstanding program, with demonstrable impact and quality, can be replicated or ‘franchised’, thus expanding impact. Finally, we provide rationale for the replication of an existing program as a way to expedite research at various locations, without the delay of a re-design. We have demonstrated herein that the scale-up approach for this program can be completed independent of participant type  and in geographically distinct regions, thus enhancing the potential for replicability internationally. Consequently, our findings are important and relevant to the field of STEM outreach to aid in developing and enhancing the STEM pipeline.Subsequent work will investigate, based on these identified emergent themes, if we can also follow the cognitive development of students via Chickering’s Vectors, as part of Student Development Theory. We aim to track post-secondary students over time and identify if NE STEM 4U aids in providing a ‘series of stops’ on the student development theory railway. In this, we will investigate if aspects of NE STEM 4U underlie the development of post-secondary students from both a cognitive development and workforce preparation perspective. Similarly, while we have previously reported the impact on participants 3 year post-experience (i.e., 3 years after graduation/3 years into either graduate school or the workforce, Nelson & Cutucache, Finally, we would be remiss if we did not point out the key partnerships that are common to these programs supporting initial fidelity conversations and that may have been contextually helpful in replication. Several partnerships are similar across the three locations: school systems, universities, non-profits, and agencies. All of the institutions also prioritize cultural relevancy through targeted training. Successful programs are integrated with webs of connections to partners with shared missions, and NE STEM 4U at all three locations is no exception. Future research might explore if particular partnerships and relationships are crucial to success. For example, an early attempt to create a franchise at the more rural teaching-focused institution did not persist without a dedicated university champion. Not surprising and worth emphasizing, expanding to additional locations requires a carefully crafted strategy of focused care and attention to maintaining the core features of a successful program and adaptation to local needs. However, it also has advantages in improving the whole program. Direct attention to data collection and evaluation, not only on the quality of the program but also fidelity, and when it is necessary to adapt, is vital to growth both in the lead site and elsewhere. Exact word for word translations do not always capture the essence of a phrase, and similarly, keeping focused on the essence of the program and determining what should and should not be incorporated in a new setting is valuable for dissemination and growth of quality STEM programs.Additional file 1. Table S1. Focus Group Survey Questions.


Community-based preschool meals can provide an effective platform for implementing integrated agriculture and nutrition programs. However, there is little evidence on the costs and cost-efficiency of implementing these types of multisectoral interventions.Assess the economic costs and cost-efficiency of implementing an effective integrated nutrition-sensitive intervention through a preschool platform in Malawi, including community-level contributions.The Strengthening Economic Evaluation for Multisectoral Strategies for Nutrition (SEEMS-Nutrition) framework and methods were applied to assess financial and economic costs of the intervention. A mixed-methods approach was used to measure and allocate costs for program activities and inputs using financial expenditure data combined with micro-costing. All costs were allocated to input and expenditure categories using the SEEMS-Nutrition framework. To facilitate comparisons with existing school meals programs, activities were also mapped against a standardized school feeding supply chain framework.The total annualized cost of the program was US$197 377, inclusive of both financial and economic costs. The annual economic cost of the program ranged from US$160 per preschool child to US$41 per beneficiary. The principal drivers of cost by program activity were training (46%), school meals provision (19%), monitoring and evaluation (12%), and establishing and running community groups (6.5%). Notably, community contributions accounted for 25% and were driven by food donations and volunteer labor.Cost per beneficiary estimates of implementing an integrated agriculture–nutrition intervention through an early childhood development platform compare favorably with similar interventions. Further research is needed that applies a standardized economic evaluation framework to such multisectoral interventions. Agriculture has strong potential due to the manifold pathways through which it can influence nutrition.2 A recently updated review of the contributions of agricultural programs in improving nutrition suggested that these types of programs are particularly effective at increasing intake of nutritious foods and improving diet quality when they include strong behavior change communication (BCC) and women’s empowerment interventions.3 However, the same review highlighted that despite progress in documenting agriculture and nutrition linkages, there are important gaps in the evidence on the costs of implementation.The scaling up of nutrition-specific interventions will not be sufficient to meet global targets for improving nutrition.4 One justification for using ECD platforms to deliver nutrition interventions is the potential for synergies between nutrition and child development.5 Another justification is that with the priority for nutrition interventions on the first 1000 days, preschool children (2-6 years old) are not covered by nutrition interventions until they start school. Though preschoolers may have less potential compared to younger children to benefit from nutrition interventions in terms of growth, they also face important nutritional needs, including consuming a nutritious and healthy diet that meets their nutrient requirements. In Malawi, 37% of children are affected by stunting with 4% of under-5 children having acute malnutrition.6 Although there is a lack of data or evidence on the intrahousehold distribution of food, it is reported that populations in most African countries consume diets that are nutritionally inadequate for child development.8School feeding and early childhood development (ECD) programs are two potential platforms recommended for delivering nutrition interventions to young children.9 the evidence on costs and benefits of multisectoral interventions that combine nutrition and ECD is not well established.Integrating ECD and nutrition services provides a bridge in nutrition programming among children beyond the first 2 years. However, although there are data on the costs of stand-alone ECD,10 Though school feeding interventions are not always designed with a primary objective of improving nutrition, program design can include features that enhance the potential for nutrition outcomes, including optimizing meal planning and using school meals as a platform for nutrition-related behavior change.11 A model known as Home-Grown School Feeding (HGSF), involving linking the provision of goods and services for school feeding to smallholder farmers and the community, has received renewed attention in terms of its potential to improve nutrition- and agriculture-related outcomes.4 However, the review of the evidence of school feeding jointly undertaken by the World Bank and World Food Program highlighted that “more accurate estimates of costs are an important area for future research.”12 Where cost estimates of school feeding programs exist, they generally do not capture contributions made by the community at the school level. Only one study in the literature includes full costing across the program supply chain.13 This study analyzed on-site meal programs in 4 countries in sub-Saharan Africa, namely, Kenya, Lesotho, Malawi, and the Gambia. School-level contributions to school feeding ranged from 0% in Lesotho to 15% in Kenya. The gap in the evidence on the trade-offs in terms of the costs and cost-efficiency of different school feeding models is an important limitation for program and policy stakeholders in selecting effective school feeding models within budgetary constraints.12This is particularly relevant as school feeding programs operate in nearly every country in the world and the scale-up of school feeding has been a key response to economic crises of the past decade.This study aims to (1) address the evidence gap on costs and cost-efficiency by evaluating the total incremental costs of implementing an integrated nutrition and agriculture intervention through a community-based preschool meal platform in Malawi and (2) present the incremental costs of implementing the integrated intervention through the application of a methodological approach designed for multisectoral nutrition strategies. A comprehensive cost analysis captured all program inputs, including contributions from the government, implementing agency, and the communities served by the intervention. Ascertaining economic costs allows for the identification of the main cost drivers and opportunities for cost containment. In particular, this study examines the “hidden” costs of such programs, such as the opportunity cost of the time invested in the program by parents and caregivers involved in the preschool meal service delivery.14 Given that the lack of regular food provision at CBCCs causes absenteeism and even CBCC closure, in 2015 Save the Children introduced an integrated nutrition and agriculture package into the existing ECD program. This program, or the Nutrition Embedding Evaluation Program Impact Evaluation (NEEPIE) program, aimed to help communities to improve the consistency and nutritional quality of food produced and provided at CBCCs as well as to enhance the quality of food at the household level.In Malawi, preschools known as community-based childcare centers (CBCCs) are one of the main components of Malawi’s national Early Childhood Health and Development (ECD) program (the other component being parenting groups). There are an estimated 11 000 CBCCs in Malawi, which aim to provide “safe, stimulating environments, access to health and nutrition services, and training for parents and caregivers.”Save the Children and Chancellor College  developed a CBCC-based nutrition and agriculture intervention drawing on existing nutrition, agriculture, and livelihood materials, using CBCC gardens and school meals as a platform for training and practicing new agricultural, meal preparation, and planning techniques to be reproduced at home. The control group were communities whose preschools were supported with the basic package of Save the Children’s ECD program while the treatment group communities also received this basic ECD package with the additional activities of the integrated agriculture–nutrition intervention. The key aspects of the program were training and provision of inputs. Trainings covered agricultural topics, nutrition, as well as how to manage VSL groups. The components of the program are outlined in Parents, CBCC management committee representatives, lead farmers and community agents were trained over 3 days before the 2 main planting seasons. The training focused on land preparation, selection of nutritious crops, agriculture production techniques, pest and disease management, manure making and application, harvesting, storage and processing, and chicken rearing. Participants were trained by government agriculture extension development officers (AEDOs), and the CBCC garden was used as a demonstration site for new agricultural techniques. The AEDOs visited the community once a month to check on progress and address problems. Households and CBCCs also received crop and vegetable seeds, such as amaranth and carrot, as well as sweet potato vines and poultry chicks. Village savings and loans (VSL) groups were formed to help households save and access funds to start small businesses and purchase supplies for the CBCC and for use in emergencies.The CBCC management committee members, CBCC caregivers, lead farmers, and parents received a 3-day nutrition training to help them plan and prepare nutritious meals for the CBCCs and for their households. The training focused on essential nutrition and hygiene practices, seasonal nutritious food selection, food storage, preservation and preparation, CBCC meal planning, and menu adaptation for the household and younger children. The AEDOs and nutrition assistants directed trainings, leading cooking demonstrations and small group practice sessions. The program was designed to have parents rotate preparation of CBCC meals. Thus, parents continued practicing recipes at the CBCC, which were then replicated at home.Inputs were provided as seeds for nutritious foods for household production and for CBCCs for consumption by preschool children and their families.15. The supply chain model shows how the program is linked to agricultural production, driving contributions and sourcing for preschools, where meals are prepared, distributed, and, finally, consumed. Applying this school meals supply chain framework will also allow for comparisons of the preschool meal service delivery with operational benchmarks on unit costs in the literature.The intervention can also be seen as a way of improving the community-based preschool meal service delivery, complementing the preschool meal service with BCC around agriculture and nutrition practices. Conceptually, the program shares common features with HGSF and, in particular, the integrated farm-to-school supply chain. Mapping the program activities to a standardized supply chain model for school meal programs highlights these similarities, with the main difference being that the program targets preschools rather primary schools, as shown in 16 This evaluation showed that implementing an integrated agriculture and nutrition intervention through an ECD platform benefitted children’s diets and reduced stunting prevalence by 17 percentage points (P < .05) among younger siblings of targeted preschoolers in the program.17 However, important questions remained on the NEEPIE program costs and, in particular, on costs borne by the community.A rigorous impact evaluation of the NEEPIE program was conducted by the International Food Policy Research Institute in 2015 to 2016. The Nutrition Embedding Evaluation Program-Impact Evaluation was a cluster randomized controlled trial (cRCT) designed to examine the effectiveness of using community-based ECD centers as a platform to improve preschool meal delivery by improving caregiver knowledge and nutrition practices, promoting household food production, dietary diversity, and nutrition among preschoolers and their younger siblings.1 The SEEMS-Nutrition framework provides a standardized approach for data collection, cost coding, allocation, and integration as well as for the final cost analysis.18 The second framework uses a set of standardized activity categories from a school feeding supply chain to facilitate benchmarking and comparisons with other school meal programs.15A cost analysis was conducted from the perspective of the nongovernmental organization (NGO) provider, Save the Children, and the beneficiaries. The analysis estimated the incremental financial and economic costs of implementing an integrated agriculture and nutrition intervention through the existing ECD program targeted to preschool children. Any costs related to external research, such as the program impact evaluation, were excluded from the analysis. The study was guided by two related frameworks. The first framework was developed as part of the Strengthening Economic Evaluation for Multisectoral Nutrition Strategies (SEEMS-Nutrition) initiative.20 that uses accounting methods for allocating expenditures by program activities and combines it with the ingredients method that calculates costs for all resources used, including donated or volunteer personnel time and supplies. Financial costs capture actual expenditures for program implementation on an annual basis. Economic costs include the opportunity cost of donated time by community members and other CBCC-level stakeholders involved in preschool meal provision. This method has been applied to assess the total costs of complex nutrition interventions in several settings.2124 We first identified all major program-related activities over the course of the 1-year program and captured the quantities and costs for all inputs associated with start-up and recurrent activities. Start-up activities included the development of agricultural production and training materials, the training of trainers for parenting, agricultural extension and nutrition, support for start-up program activities from Save the Children personnel , and initial community investments such as CBCC construction. Recurrent activities included community-level trainings on agricultural production, food processing, nutrition, parenting practices, and VSL. Recurrent household-level activities included agricultural training and the provision of agricultural inputs  chain typology of interventions . The conFinancial and economic cost data collection occurred throughout the duration of the NEEPIE study, spanning a multiyear period between September 2015 and October 2019. The intervention began in December 2015 and was implemented for 12 months. Data collection for the parent cRCT began in November 2015 (baseline), midline in April 2016, and endline in November 2016, which included household surveys and in-depth CBCC/community surveys. A second-year follow-up survey was also conducted as part of the cRCT 1 year after the program ended to examine the sustainability of program impacts.16). Price data for the food items used in the meals were collected using standard market-level surveys in the 4 main markets in the study area. There were 60 total CBCCs included in the NEEPIE intervention, 30 of which were treatment CBCCs and 30 of which were control CBCCs. The 60 CBCCs served 1248 preschool children (36-72 months) and 304 younger siblings (aged 6-24 months).To estimate financial costs, we collected program-level expenditures from Save the Children administrative records for the 12-month program implementation. These expenditures were then coded using predefined cost categories. To compliment the expenditure records, we conducted in-depth CBCC/community surveys periodically between October 2015 and 2017 to obtain detailed information on the quantities of resources used to implement the integrated agriculture and nutrition package. These resources included staff time allocation, infrastructure investments, meal provision, and CBCC collective gardens where food for the program was grown  to feed into the overall cost analysis. The CBCC-level questionnaire covered both cash and in-kind contributions for agriculture production and complementary nutrition education to improve consumption of nutritious foods. As per the design of the intervention, community engagement and contributions were critical to successful implementation. Households in intervention villages contributed volunteer time to CBCCs as well as in-kind donations. Volunteer labor was contributed by women from participating households in preparing school meals, tending gardens, caring for children during CBCC hours, and participating on CBCC management committees. Men from intervention households generally provided support to construct or maintain CBCC buildings, and some participated in CBCC committees. Donations for CBCC meals were made in several forms, either as foodstuffs produced at the household level or as food purchased by small household cash contributions or pooled community contributions. Despite the use of a semistructured questionnaire to gather data on costs, it was not possible to obtain accurate day-to-day expenditures at the CBCC level due to the low level of record-keeping. As such, we generated estimates of the total incremental cost of implementation considering all activities required to provide the intervention service, using data collected from the different CBCCs.Cash value in Malawian kwacha (MKW), such as cost of pots, utensils, and stoves for the CBCCIn-kind costs (MKW), such as in-kind foodstuffs contributed to the programVolunteer time costs (hours per day), for example, time spent caregiving for children at the CBCC or in the construction and maintenance of CBCC buildingsThe economic costs obtained from the survey on community contributions or donations included:26 and was adapted to include NEEPIE program-specific activity categories.At the household level, data on opportunity costs were collected using a time allocation module in the household survey from the external program evaluation. This time allocation module was based on the standard, validated module developed for the Women’s Empowerment in Agriculture IndexThe overall analytical approach was to integrate total economic costs from the CBCCs and households—in voluntary labor and other contributions—with the NGO expenditure data to arrive at total incremental costs of the program. Expenditure data included the salaries of program staff, frontline workers who provided extension and nutrition support, and government workers as well as NGO costs for agricultural inputs, training, and other supplies including vehicle use and fuel. Program staff were interviewed retrospectively to estimate the allocation of their salaried time to program activities during the implementation period.27 This wage rate was used to reflect the labor profile of the equivalent type of worker. Apprentice wages were used as the participating households are poor agricultural households with low levels of education (55.1% with a primary education and only 16.3% possessing any secondary education); thus, the average wage representing formal employment is not representative of this population. For this reason, we utilized an apprentice wage as it was more representative; however, we also modeled other wage scenarios. We conduct sensitivity analyses to model different wage rate scenarios  to provide a cost range of the lower and upper bounds (Table 7). The total time in hours donated in the program by volunteers was recorded as part of the CBCC data collection activities described above as well as through the time allocation module in the household survey.28 The CBCC survey collected volunteer contributions from both men and women for specific CBCC activities  as captured in the CBCC survey. These CBCC costs also captured start-up investments of labor such as preschool construction. The time allocation module from the household survey measured female caregiver beneficiary time over a 24-hour period during intervention activities. There was a statistically significant difference (P < .05) between the treatment and control groups in the number of minutes devoted to caregiving and other program activities by beneficiaries as compared to the control group. This additional time was valued at the same wage rate as above (50% minimum wage for unskilled apprentice). Beneficiary time contributions were allocated as 50% to the activity of establishing and running community groups and 50% to the provision of school meals.The next step was to incorporate the opportunity cost of volunteer time donated to program implementation. The opportunity cost of time invested in the program was calculated by multiplying the hours of voluntary beneficiary labor per child per year donated to the program by 50% of the minimum wage for a 19-year-old apprentice unskilled worker in Malawi, which is US$23 per month, as it has been reported that actual wages in Malawi are less than half of legal minimum wages.To estimate the annual cost of the program per child, we aggregated estimates of individual cost items per child per day based on data collected from each CBCC to arrive at an average daily cost per child per day and ultimately an annual cost per child. The following steps were used to estimate the annual cost per child enrolled in the CBCC.First, using resource and cost data collected from each participating CBCC, we obtained a daily program cost for each activity. For daily CBCC food costs, we estimated the quantity of each of the food items provided and then multiplied each item by the average price of that item obtained from the local market price survey. A summary of cost calculations can be found in Fifth, in order to understand costs by a set of standard activity codes, we retroactively mapped the NEEPIE activity and input costs to the standardized SEEMS-Nutrition cost categories. Lastly, 29 The assumed useful life of all relevant school-level capital equipment was set to 10 years . All costs were incurred in MKW and converted to USD using official exchange rates, based on an average yearly exchange rate of US$1 to 714 MKW in August 2017. Cost data were coded and analyzed using Microsoft Excel. Time allocation data from the survey were analyzed in Stata version 14.2.Capital costs were annuitized over the period of program implementation using a discount rate of 3% following World Bank recommendations. Annuitization enables an equivalent annual cost to be estimated and reflects the value in-use of capital items, rather than reflecting when the item was purchased.27 Each of these 3 wage estimates was multiplied by the total number of hours invested in the program by volunteers in the community, resulting in varying estimates for the monetary opportunity cost of time invested in the program by volunteers.Wage variations: The opportunity cost of the program in USD was calculated by obtaining an approximate hourly value of the unskilled labor wage, the legal minimum wage, and the average wage. These were then halved since it has been reported that actual wages in Malawi tend to be less than 50% of legal wages.Training variations: The cost of training communities to run the program was found to comprise a significant proportion of total program costs (46%). Trainings were run during the first and only year of the program. Since the magnitude of follow-up training required over a hypothetical 5-year program life span could vary, 3 different scenarios were considered: In the first scenario, the training was run in the first year only, with no follow-up training in the subsequent 4 years of the program life span. In the second scenario, training was run in the first year combined with yearly follow-up training that was less intensive, assumed at 75% of the first-year training in each of the subsequent 4 years of the program life span. In the third scenario, the training program was assumed to be run at the same initial yearly cost for all 5 years of the program’s duration.Beneficiary variation: The total cost of the program  was initially reported as a cost per year per child enrolled in the participating CBCCs. Since other members of the household have also been found to benefit from the program , a total program cost per beneficiary was also calculated by dividing the cost per child by the average number of members per household in the communities participating in the study.To obtain an estimate of how costs of the program could vary, a range of costs were calculated based on different scenarios, including variations of wage, training program intensity, and beneficiaries:30 and used for a linear modeling analysis of the CBCC menus for meals provided the day before the survey. Output data were combined with costs to provide estimates of unit costs or cost-efficiency, including costs per preschool child served, cost per kilocalories delivered, cost per mg iron delivered, and cost per 100 mcg vitamin A delivered. To compare with operational benchmarks for school feeding in the literature, the total cost per child served per year was standardized to a 200-feeding-day year and 700 daily planned kcal.31Process and output data covering the adequacy of the meal service delivery  were collected from CBCC and household-level surveys using standardized data collection forms. Food composition data were retrieved from a Malawi-adapted food composition tableData were collected from 30 CBCCs that received the integrated intervention and were operational during the survey period. The results of the cost analysis are summarized in Community contributions ($49460.97) accounted for US$40.08 per child per year, or 25% of total program costs, driven by food contributions  and volunteer time in caregiving, meal preparation, and other support .We also examine variability by CBCC size to show variation of CBCC costs. The mean cost per CBCC is $3360, with the lower ($1920) and upper bounds ($7360). The cost per CBCC is generated by taking the upper and lower bounds for CBCC size by enrolment and multiplying by the cost per child.When exploring how program activities map to the NSV chain typology , approxiThe evidence from the 3 alternative sensitivity analysis scenarios using varying wage rates  suggesteThe findings on the cost-efficiency metrics are summarized in 12 This gap is particularly important when examining implementation models that require contributions at the community level, where opportunity costs like volunteer time in food preparation and child care, or in providing in-kind food donations, are often overlooked.13Nutrition-sensitive approaches, such as school feeding and ECD programs, have the potential to provide platforms to scale up nutrition interventions. However, despite the recent progress in generating evidence on the effectiveness of these programs, there are important gaps on the costs of implementation. Moreover, despite the popularity of school meal programs, there is little rigorous evidence on the total incremental costs of program implementation.15 However, unlike the traditional farm-to-school model where donations are encouraged but the meals rely on government food and cash transfers, the meal provision in the NEEPIE model relies entirely on voluntary contributions from parents and other community members. As a result, community contributions accounted for 25% of total costs, equivalent to approximately US$40 per child per year. These figures are remarkable, considering that the intervention was scaled up in high poverty rural areas with chronic food insecurity. The only other study that collected data on community contributions found that these contributions ranged from 0% in Lesotho to 15% in Kenya.13 In the Malawi case, food donations accounted for 66% of total community contributions, which does not include the additional volunteer time in caregiving, meal preparation, and other support. The financing of the model implemented in Malawi is thus heavily subsidized by communities receiving the program, highlighting an opportunity for scale-up as part of the ECD policy of the Government of Malawi’s (GoM).32The integrated agriculture and nutrition intervention implemented in Malawi, an example of an integrated farm-to-preschool school feeding model, incorporates a range of innovations in program design. The intervention used preschool meals as a platform to implement a range of agriculture and nutrition activities aimed at improving production and consumption of nutritious foods. Unlike traditional school meal programs, no food or cash transfers from external agencies or government were involved. Similar to the integrated farm-to-school model that has been implemented in Cote d’Ivoire, for example, it strives to link school meals to in-kind donations of nutritious foods from the community.Subsequently, the GoM recently partnered with the World Bank to scale up this intervention model nationally under its “Investing in Early Years for Growth and Productivity in Malawi” project. This approach is important as the program targets vulnerable children in the age gap between the first 1000 days and traditional school age. The SEEMS-Nutrition consortium is currently working with the GoM to collect and analyze the costs of the government-implemented intervention.28 Regardless, these results provide a strong case for conducting economic evaluations to identify and acknowledge the significant costs of participation to communities. An economic evaluation is underway by the SEEMS-Nutrition consortium to conduct further cost–benefit analyses of the NEEPIE program.Our analysis highlights the level of contribution by communities, which was the intention of the program’s design and also a positive indicator of the investment of households in the program. However, the intensity of community engagement could also be a potential limiting factor affecting feasibility and generalizability. The high level of community engagement might not be found in all settings, particularly in urban contexts where communities may be less cohesive. Second, although participants may reprioritize or change their daily tasks while contributing to program activities, it is critical that they do not become overburdened. A separate study examined whether female participants were burdened by NEEPIE activities and found that although there was a modest increase in daily caregiving, they did not consider contributions burdensome or unmanageable.The intervention also benefitted other household members alongside the participating preschoolers receiving meals, including improved caregiver knowledge of nutrition, diet diversity and linear growth in their younger siblings, and production of nutritious foods. These additional benefits suggest that the estimates of cost per beneficiary, rather than cost per child receiving meals, may provide a more accurate metric for this particular type of intervention, thus increasing considerably its cost-efficiency.We still have much to learn by exploring the costs of multisectoral agriculture-nutrition programs. Examining costs with the SEEMS-Nutrition activity and input categories and using the NSV chain typology expands our ability to disaggregate costs and to analyze resource allocation patterns. Exploring cost components or cost shares by the SEEMS-Nutrition activity, input, and value chain intervention typology categories has the potential to enhance future cost comparability with other multisectoral interventions to improve nutrition in cases where the methods for measuring and valuing resources and costs are transparent .33, making comparisons difficult, particularly as many of these interventions are not designed to address nutritional outcomes. Comparisons across studies are also challenging due to the different methodologies involved, including on how to account for coordination costs. For example, in this study, costs associated with activities for the coordination across sectors accounted for approximately 18% of total expenditures. How these costs compare to other models of implementation remains an important question for program and policy stakeholders looking to scale up these types of programs. The Global Health Cost Consortium (GHCC) reference case on global health costing notes that resources that occur above the delivery site, whether that is the clinic or community, are often excluded. The GHCC recommends that these costs be considered in the same way as on-site costs, rather than arbitrarily omitted. When these costs are difficult to capture, the omission should be clearly stated, and any bias reported.34 A study of an integrated agriculture, health, and nutrition project implemented in Kenya estimated above-site integration and coordination activities as monthly feedback meetings and any other coordination or monitoring activities that included participation by all partners and implementing agents. They found that these above-site costs accounted for 27% of total incremental costs.24There is scant evidence of agricultural value chain interventions improving nutrition35 In programs with a deliberately stated nutrition objective such as NEEPIE, using an NSV chain framework to map costs allows for a better understanding of how program activities are aligned to value chain typologies . This analysis could aid program implementers in decision-making around future program design. For example, if a program did not have intended impacts on driving consumption and, through an examination of costs, it emerged that most costs were directed toward supply-side activities, this might suggest more resources to be directed to demand-side activities such as BCC for the consumption of nutritious foods or to child feeding practices. This mapping also could aid future comparisons with other complex, multisectoral NSV interventions, to better chart the pathways through which impact is occurring in each program and as well as to trace the flow of program resources to those activities linked with the pathways. This approach also shows relative trade-offs in how to allocate resources when investing in different NSV chain approaches.For those value chain interventions that do have nutrition objectives, outcomes have been mixed. For example, one dairy-focused NSV chain intervention, similarly targeted to preschoolers, raised hemoglobin but did not reduce anemia.Likewise, the mapping of costs to program activities highlights the drivers of cost. This is helpful for implementers—who are interested in controlling program costs and increasing efficiencies—to understand which activities comprise the majority of costs and allowing for identification of opportunities for economies of scale. For example, travel costs/per diems comprise 22% of input costs and 45% of activity costs are related to training. The program could investigate whether travel could be made more efficient for actors delivering trainings such as delivering trainings at regional hubs. Of course, such adjustments must assess potential costs to program participants, such as transferring travel costs onto beneficiaries. Policymakers also benefit from understanding cost drivers, as with the ECD GoM scale-up, to understand how program costs change over time, to pick and choose program activities for policy design, or to identify synergistic activities with extant programs .24 Another study estimated the costs over 10 years of delivering an enhanced homestead food production intervention in rural Cambodia to be US$929 per household.36 In Zimbabwe, a study found that the average costs per household reached of an intervention providing community vegetable gardens for people living with HIV were US$1890 in 2010.37 A cross-country study in Ethiopia, Nigeria, and India found that the costs per child reached of agriculture and nutrition intervention varied widely, ranging from US$2650 for a livestock transfer program to US$0.58 for a media and education campaign.38 In comparisons with school feeding programs, the evidence suggests that integrated farm-to-preschool model as implemented in Malawi is generally less cost-efficient than other school feeding program models  were estimated at US$155 per pregnant women reached or US$110 per beneficiary if both pregnant mothers and infant children were included.The approach under development through the SEEMS-Nutrition framework aims to address this challenge of comparability through standardization of methods, tools, and analysis. Future work will outline this approach as well as apply it to other multisectoral programs and contexts, building a toolbox for economic evaluation of multisectoral nutrition programs as well as growing the evidence base.15 Another important consideration involves the opportunity cost of time invested in the program by community volunteers; when this is considered in USD, it appears as a relatively low cost due to the low wages for unskilled labor in Malawi. In actual hours worked, this translates to approximately 60 h/child/yr, which perhaps provides a better reflection of the opportunity cost of the intervention. As a function of caregivers’ total available time, this represented only a small percentage. However, the intervention is intensive and involves a considerable amount of inputs. In the absence of cost-effectiveness analysis, an important gap remains in terms of understanding the value of scaling up these types of integrated interventions. However, because of positive impacts with large effect sizes on preschoolers, younger siblings, and households and how the program holds up to other benchmarks, it is suggestive that the program provides significant benefits for the cost invested. Additionally, as the benefits accrue to different groups, the cost–benefit is not straightforward and provides a challenge for economic evaluation, which will be presented in a future paper.The study findings are limited by several important considerations. First, it was not possible to obtain an accurate picture of the day-to-day expenditures at the CBCC level due to the low level of record-keeping. As such, in our analysis, we used the existing data from the different CBCCs to generate estimates of the total incremental cost of implementation considering all activities required to provide the intervention service. The data used for the estimations at community level were based on caregiver recall, which is likely to have measurement error. However, the per-child estimates of the different community-level costs compare relatively well with those from similar studies in similar contexts.By assessing the total economic costs of implementing the community-based preschool meal program, this study is the first to our knowledge to provide evidence on the total incremental costs of implementing an integrated agriculture-nutrition intervention through an ECD platform. Cost per beneficiary estimates compare favorably with similar interventions. Further research is needed that applies a standardized economic evaluation framework to these types of multisectoral interventions.


Program outcome assessment is a complex process that demands careful planning and resources in order to accurately assess higher-order thinking skills. A well-defined assessment approach provides detailed insights into program weaknesses and leads to continuous improvement. Whereas a poor assessment approach does not reflect the underlying weaknesses and may result in a useless effort. Furthermore, each accreditation body may have a different recommended outcome measurement approach. As a result, academic institutions may make adhoc choices just to satisfy accreditation requirements rather than designing a sustainable measurement approach. On the other hand, the magnitude of huge tasks for satisfying multiple accreditation bodies results in fatigue and mental stress for academic staff. ABET is a well-known international program accreditation body, and NCAAA is a local accreditation body for academic programs in the Kingdom of Saudi Arabia. In this paper, we have documented that how a sustainable outcome measurement mechanism can be designed to satisfy both ABET and NCAAA requirements. The core contribution of this paper is relevant specifically for academic programs in the Kingdom striving to meet both ABET and NCAAA requirements and is also relevant for all education programs to design an appropriate program assessment approach to ensure a sustainable process to foster better learning among students. Accreditation has become an important quality assurance tool for higher education institutions not only to improve their processes but also to use as a marketing tool to attract more students and funding. There are many local as well as international accreditation agencies in different geographical regions that accredit institutions or a particular program. The majority of the accreditations can be termed as single-tier accreditation, where a particular criterion needs to be satisfied by aspiring academic institutes/programs and there is no further classification among accredited programs. Due to the recent focus on outcome-based education, an academic program intended for accreditation needs to provide evidence that it has a comprehensive system of measuring the intended outcomes of the program. Student outcome attainment requires that higher-order thinking skills be measured to ensure that students have gained the required mastery of skills to be successful in their future professional careers.An ideal program assessment approach should be valid, lightweight, and easy to verify, however, many additional constraints are relevant, such as management support, faculty support and intention for continuous improvement . In the In this paper, we present a case study from an Information Systems undergraduate program which is required to satisfy the international accreditation, Accreditation Board of Engineering and Technology (ABET)  as well The rest of the paper is structured as follows: Oudshoorn et al. have described that recently there is an increase in acquiring ABET accreditation by computing programs. They have outlined a set of guidelines for stakeholders aiming to acquire ABET accreditation . HossainMany researchers have documented their ABET accreditation experience to help other institutions in achieving accreditation requirements. Anwar and Richards have compared the accreditation criteria of ABET and the engineering council, both of which are Washington accord signatories and highlighted similarities and differences among their criteria. They proposed the need for alignment among accredited programs of different signatory bodies of the Washington accord . BachnakProgram assessment is one of the critical tasks to verify that an academic program is able to meet the intended student outcomes. Carelli has presented a case study of an academic program’s assessment for acquiring ABET accreditation to provide guidelines for other aspiring institutions to acquire ABET accreditation . Shafi eEstablishing an institutional quality management system can help in fostering quality culture and ultimately help academic programs in achieving accreditation requirements. Almuhaideb and Saeed have proposed a set of best practices to deploy quality assurance practices to foster appropriate outcome-based learning . FurtherSince the ABET assessment process is quite complex and modern technologies have the potential to improve the data collection and reporting process. Therefore, Sabir et al. have developed an application in Microsoft access to facilitate data management of program assessment for ABET accreditation . AlhakamRecently, like all other fields’ accreditation activities are also impacted by the COVID-19 pandemic. Hussain et al. have proposed a digital quality management system for program assessment to facilitate virtual accreditation visits due to the COVID-19 pandemic. This model was applied to three engineering programs, and they recommended its usage by academic institutions and accreditation bodies in remote accreditation processes . Karimi Despite these studies, there is no systematic study that provides detailed insights on developing a program assessment strategy that is aligned with different accreditation requirements. Keeping this in view, in this paper we provide a detailed insight into how a program assessment strategy can be formulated which is in line with accreditation requirements of different accreditation bodies.The findings of this research are part of a longitudinal action research project, initiated to develop sustainable quality practices for computing academic programs, which are aligned with international and national standards. In this contribution, our core research question was that how to define a sustainable program assessment methodology that is aligned with multiple accreditation requirements. In order to answer this question, we adopted a case study approach in our research design. The case study approach is widely used in research studies to provide a rich description of settings to relate research findings in diverse contexts. Our case study is based on a Bachelor of Science in Computer Information Systems (CIS) program. The quantitative data were collected using multiple direct and indirect assessments including surveys and student performance data. To ensure data accuracy, we employed multiple reviews by different team members and during the analysis phase, we conducted a descriptive analysis of assessment data. The findings of this case study will help other academic institutions to align their teaching strategies and assessments with different local and international accreditation requirements.Our case setting (CIS academic program) is offered by the College of Computer Science and Information technology (CCSIT) at Imam Abdulrahman Bin Faisal University (IAU) . Since iCIS program at CCSIT is an ABET-accredited program. The accreditation by ABET is aimed to ensure that education provided through this program meets an acceptable level of quality. It helps in creating goals for institutional self-improvement and to look for self-regulatory alternatives. The program was accredited by ABET in 2018, and it will be assessed for ABET accreditation in 2023.Desired characteristics that are expected to be achieved from students at the end of the program are called Student Outcomes (ABET)/Program Learning Outcomes (NCAAA). To design an optimal teaching strategy, we have used a top-down strategy, where firstly we devised the intended learning outcomes of the academic program. ABET refers to the intended learning outcomes to be achieved at the completion of an academic program as Student Outcomes (SOs), whereas NCAAA uses the term Program Learning Outcomes (PLOs). In this paper we will mainly use the term SOs, however, both terms are interchangeable.ABET has proposed five generic student outcomes required for a computing-related program and one specific student outcome for an information system related degree program which are numbered as 1–6. On the other hand, NCAAA does not have any such specific recommendations, however, they recommend that program learning outcomes need to be classified into three learning domains, Knowledge & understanding, Skills and Values. Furthermore, each domain needs to have at least one program learning outcome mapped to it. When we mapped ABET proposed CIS student outcomes into learning domains of NCAAA, we found that these outcomes are mapped to only Skills and Values domains, and there is no learning outcome at the knowledge & understanding level. Therefore, we have developed an additional student’s outcome for our CIS program in the Knowledge & Understanding domain, targeting the coverage of fundamental concepts and theories in the information system domain. Since ABET has numbered its student outcomes as 1–6, so to avoid confusion we numbered the additional student outcome as 0, whereas the NCAAA recommends numbering program learning outcomes with a prefix of the respective domain, so CIS program has 1 SO (K1) for Knowledge & Understanding domain, four SOs  for Skills domain and 2 SOs  for Values domain in our CIS program.Following are the CIS program Student Outcomes/program learning outcomes (labeled with associated learning domain) which satisfy ABET revised criteria for the CIS program, as well as are aligned with NCAAA requirements.Define fundamental concepts and theories from information systems and related fields. [Knowledge & understanding]Analyze a complex computing problem and apply principles of computing and other relevant disciplines to identify solutions. [Skills]Design, implement and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program’s discipline. [Skills]Communicate effectively in a variety of professional contexts. [Skills]Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles. [Values]Function effectively as a member or leader of a team engaged in activities appropriate to the program’s discipline. [Values]Demonstrate an understanding of processes that support the delivery and management of information systems within a specific application environment. [Skills]To ensure that all aspects of a student outcome are covered in our program and required higher-order skills are integrated into the curriculum, each student outcome is further decomposed into performance indicators which are mapped with the entire curriculum. ABET recommends measuring performance indicators whereas NCAAA is only concerned with SOs/PLOs, therefore, in our program mapping we have considered both of them. Following are key performance indicators (PIs) (labeled with associated student outcomes) of the CIS program.Students demonstrate the ability to understand the fundamental concepts related to information system discipline. [SO:0]Students demonstrate the ability to understand the knowledge of supporting disciplines appropriate to the needs of the program. [SO:0]Students demonstrate the ability to decompose a task into appropriate components. [SO:1]Students demonstrate the ability to solicit and formulate requirements specifications. [SO:1]Students demonstrate the ability to estimate the resources required for the proposed solution. [SO:1]Students demonstrate the ability to design a secure computer-based system, process, component, or program to meet desired needs. [SO:2]Students demonstrate the ability to develop a computer-based solution. [SO:2]Students demonstrate competency in creating and executing test cases. [SO:2]Students demonstrate the ability to write technical reports. [SO:3]Students demonstrate the ability to deliver oral presentations. [SO:3]Students demonstrate the ability to learn new skills and apply them rationally to solve the given problem. [SO:4]Students demonstrate knowledge of professional, ethical, legal, social issues and responsibilities. [SO:4]Students demonstrate the ability to produce quality deliverables and meet the deadlines of the group. [SO:5]Students demonstrate the ability to organize themselves and function as a team. [SO:5]Students demonstrate the ability to understand information system management issues, tools and technology. [SO:6]Students demonstrate the ability to evaluate the applicability of a technology or its impact, on an organizational environment. [SO:6]Students demonstrate the application of business knowledge to facilitate the delivery and management of Information Systems. [SO:6]Students demonstrate the ability to manage security risks affecting business continuity. [SO:6]As a next step, a program study matrix was developed, so as already mentioned ABET is interested in SOs as well as PI attainment whereas NCAAA is mainly interested in SO attainment. Therefore, we designed a teaching strategy for each performance indicator, each course was mapped to relevant PIs as Introduced (I), Practiced (P) and Mastered(M). Courses from the first three common years were mapped as (I), fourth-year courses were mapped as (P) and final year relevant courses were mapped as (M). To ensure the mapping was correct, each course learning outcome (CLO) that was contributing to a particular PI was documented. As an example, if a particular course’s CLO is mapped to performance indicator 1.1, then it is shown as , highlighting that this CLO is mapping to PI 1.1 and SO1. Once all PIs are mapped to the entire curriculum, a mapping for respective CLOs was made, in case any of the PI under a particular SO is mapped to a course then we used the same mapping symbol (I/R/E) for that SO, e.g., for SO 1, a particular course is mapped as “I” for any of the PI 1.1/1.2/1.3, then we will map the SO as “I” as well. In this way, we have a program mapping for all SOs and all PIs. For ABET, we used program mapping at the PI level whereas for NCAAA documentation PI mapping was ignored and only program mapping was used at the SO level. In this way rather than maintaining two different program mapping, we developed a unified program mapping which is shown in As a next step, we identified optimal teaching strategies required to teach the curriculum content required for a CLO. As a result, we developed a set of teaching strategies being used in the entire program for each SO, as shown in In the context of the measuring approach, there were also considerable differences in the requirements of both accreditation bodies. NCAAA requires that the CLOs of each course be measured and then as per program mapping the data of each SO is calculated by averaging all the relevant CLOs. This is very exhaustive and requires that the CLO of each course in the curriculum be considered. As each course has developed an exam blueprint to demonstrate how each CLO is going to be evaluated in that course, so at the completion of each course CLO data is collected from the teaching team which reflects the student performance levels for each CLO in the respective course. Therefore, to collect program assessment data in the case of NCAAA, all the relevant CLO data, contributing to a SOs based on program mapping were averaged from the last two-year course. Only the last two years of the course were considered to collect program assessment data because in the first three years it is not possible to track students only belonging to a particular degree program, as students select the degree at the completion of year three only. In an earlier contribution, we have discussed the processes to have an end of term presentation where each instructor presents the result of CLO attainments and continuous improvement actions are planned based on the feedback of the entire department .On the other hand, for ABET, there is no such restriction, and it is possible to select representative courses to select for formative and summative assessment, so we have developed an assessment plan for ABET as well. In the assessment plan, 4th-year courses are selected for formative assessments only and 5th-year courses are selected for summative assessments. These courses are selected based on the relevance of the course to the PI requirements. For each PI, three courses for assessments were identified, one for formative assessment and two for summative assessments. The assessment plan was approved by the department in the first week of the term. This ensured that the instructors of all identified courses for program assessments know before the commencement of classes that their course will be used for ABET data collection. The rubrics of all performance indicators were defined which were also approved by the department. Attainment of SOs was collected using summative data, formative data, alumni, faculty and exit surveys. Each course selected for formative or summative assessment had a dedicated comprehensive question targeting the measuring PI in the final exam as per the defined rubric. These assessment results were shared with the quality unit and quality unit members collaborated with the instructor team in filling the summative form. In the case of surveys, relevant surveys had relevant questions relating to PIs.At the completion of the semester, each instructor filled an excel sheet and this data was collected by the program quality unit which aggregated the data to prepare the student performance for each SO. In order to ensure the validity of data peer reviews were conducted at course level data, as well as program level data.The data collection for ABET was a little complex, here for each course selected for a given PI, the student performance was categorized into four categories named Poor, Developing, Developed and Exemplary based on the defined rubrics. To ensure that assessment data covers the required rigor of higher-order thinking skills, during the data collection process program quality unit members collaborated with faculty. Initially, they conducted a workshop with faculty that how higher-order thinking skills can be measured and how rubrics were applied. Later during the data collection process, they supported in identifying students’ performance levels and data collection. This collaborative approach helped especially the new faculty with how good assessments can be designed to appropriately measure the analytical skills required by the performance indicators. All this assessment data was collected in an excel sheet for each program and peer review by program quality unit members ensured the data validity. Furthermore, random checks were conducted for PI data to backtrack the data to the course level to ensure the correctness of data. Finally, complete data was presented to the department board for verification and approval. The data was analyzed using cohort analysis, where formative and summative data helped to understand the student learning progress through their academic journey. Furthermore, the results of formative and summative data were correlated.The attainment was calculated based on the number of students reached to developed and exemplary categories and an attainment target of 70% was set. Formative and summative data was documented starting from each course belonging to a PI and then the average results of each PI and then the average for each SO is calculated. Similarly, faculty, exit, and alumni survey attainment was documented separately at each PI level and averaged at the SO level. Since survey questions use a Likert scale from 1–5, so we use the attainment formula as (Strongly Agree + Agree + (1/2 of Sometimes True) percentages). Finally, as shown in Based on this extensive program assessment exercise, a continuous improvement action plan was formulated which is shown in Although NCAAA and ABET program assessments had considerable differences, ABET data was based on selective courses whereas NCAAA approach was more exhaustive covering all courses, but still the results in Establishing a sustainable assessment mechanism requires that the assessment approach is understandable to all stakeholders. In the case of multiple accreditation requirements, it is important to have a unified program mapping to avoid confusion and assessment methods should be overlapped wherever possible to minimize effort and stress on academic staff ,32. As HWe also highlight that selection of courses from where the data should be collected in one assessment cycle should be based on the relevance of the course to the PI requirements, rather than ease in data collection. In each cycle, there should be a change in the assessed courses wherever possible to balance the load among course instructors. There is a tradeoff between the reliability of data and the effort required to collect data. Selecting all PIs data from a few courses results in less effort but may compromise the quality of data, therefore, it needs to be ensured that there is a reasonable number of courses identified for data collection in one cycle.Instructors need to ensure that the assessment questions aimed at measuring ABET SOs/PIs are in line with the rigor required in rubrics. The questions in these assessments should strictly follow the rubrics designed for PIs. The development of unified rubrics, training workshops and collaboration of a central quality unit are the main hallmarks of success to ensure that assessment practices are targeting the required higher-order skills.To ensure compliance with accreditation procedures, academic programs need to document the attainment of student outcomes. The precision of the student attainment process is vital to draw continuous improvement action plans. In many countries, there are local accreditation bodies and international accreditation bodies and normally there are differences in their approach to program assessment. In this paper, we have presented that how the computer information systems program solved this challenge by devising an assessment approach that is aligned with international accreditation (ABET) and local accreditation (NCAAA). The findings will help other academic institutions facing a similar situation to appropriately design program outcome measurement mechanisms.


A new generation of earthquake catalogs developed through supervised machine-learning illuminates earthquake activity with unprecedented detail. Application of unsupervised machine learning to analyze the more complete expression of seismicity in these catalogs may be the fastest route to improving earthquake forecasting. The past 5 years have seen a rapidly accelerating effort in applying machine learning to seismological problems. The serial components of earthquake monitoring workflows include: detection, arrival time measurement, phase association, location, and characterization. All of these tasks have seen rapid progress due to effective implementation of machine-learning approaches. They have proven opportune targets for machine learning in seismology mainly due to the large, labeled data sets, which are often publicly available, and that were constructed through decades of dedicated work by skilled analysts. These are the essential ingredient for building complex supervised models. Progress has been realized in research mode to analyze the details of seismicity well after the earthquakes being studied have occurred, and machine-learning techniques are poised to be implemented in operational mode for real-time monitoring. We will soon have a next generation of earthquake catalogs that contain much more information. How much more? These more complete catalogs typically feature at least a factor of ten more earthquakes Fig.  and prov1 that are specifically designed based on earthquake signal characteristics and that mimic the manual processing by analysts, can lead to performance increases beyond those offered by earlier models that adapted neural network architectures from other fields. Those interested in using earthquake catalogs for forecasting can anticipate a shifting landscape with continuing improvements.This next generation of earthquake catalogs will not be the single, static objects seismologists are accustomed to working with. For example, less than 2 years after the 2019 Ridgecrest, California earthquake sequence there already exist four next-generation catalogs, each of which were developed with different enhanced detection techniques. Now, and in the future, this will be the norm, and earthquake catalogs will be updated and improved—potentially dramatically—with time. Second-generation deep learning modelsWhile these improvements are impressive, the value of the extra information they provide is less clear. What will we learn about earthquake behavior from these deeper catalogs and how might it improve the prospects for the stubbornly difficult problem of earthquake forecasting?2 and thus both policy and earthquake risk reduction. A key assumption is that what we learn from the newly uncovered small earthquakes in AI-based catalogs, will inform earthquake forecasting for events of all magnitudes. The observed scale invariance of earthquake behavior suggests this is a reasonable expectation.Short-term deterministic earthquake prediction remains elusive and is perhaps impossible; however, probabilistic earthquake forecasting is another matter. It remains the subject of focused and sustained attention and it informs earthquake hazard characterization3 that describes the temporal decay of aftershock rate, the magnitude-frequency distribution, with the b-value describing the relative numbers of small vs. large earthquakes4, and the Epidemic Type Aftershock Sequence (ETAS) model5 in which earthquakes are treated as a self-exciting process governed by Omori’s law for their frequency of occurrence and Gutenberg–Richter statistics for their magnitude. These empirical laws continue to prove their utility. Just in the past few years, the time dependence of the b-value has been used to try to anticipate the likelihood of large earthquakes during an ongoing earthquake sequence6 and the ETAS model has been improved to better anticipate future large events7. So it appears that there is value in applying these longstanding relationships to improved earthquake catalogs, but our opinion is that much more needs to be done.Empirical seismological relationships have played a key role in the development of earthquake forecasting. These include Omori’s law8 to discern new relationships encoded in the seismicity.The relationships cited above date from 127, 77, and 33 years ago. The oldest of them, Omori’s Law, was developed based on felt reports without the benefit of instrumental measurements. We suggest that a fresh approach using more powerful techniques is warranted. Earthquake catalogs are complex, high-dimensional objects and as Fig. 9. Well-controlled laboratory analogs to faults lack the geologic complexity of the Earth, yet, weak natural background vibrations of a similar sort, that again were thought to be random noise, have been shown to embody information that can be used to predict the onset time of slow slip events in the Cascadia subduction zone10. Finally, unsupervised deep learning, in which algorithms are used to discern patterns in data without the benefit of prior labels, applied to seismic waveform data uncovered precursory signals preceding the large and damaging 2017 landslide and tsunami in Greenland11.There are tantalizing indications that such an approach may lead to new insights. In double-direct-shear experiments, background signals that were thought to be uninformative random noise have instead been shown to encode information on the state of friction and the eventual time of failure of faults in a laboratory setting12. Secondary triggering and background stress heterogeneity were shown to improve stress triggering models13 but were most effective when they incorporated near‐real‐time aftershock data from the sequence as it unfolded14. We note that there is no reason why more complete earthquake catalogs, developed with pre-trained neural network models, cannot be created in real time as an earthquake sequence unfolds. Finally, despite the disappointing history of the search for precursors, due diligence requires that seismologists consider the pursuit of signals that might be precursory.These examples are compelling but come with the caveat that they are not representative of the typical fast rupture velocity earthquakes on tectonic faults that are of societal concern. For such earthquakes, however, there are also indications from state-of-the-art forecasting approaches that next-generation earthquake catalogs may contain information that will lead to progress. Physics-based forecasting models, which account for changes in the Coulomb failure stress due to antecedent earthquakes that favor the occurrence of subsequent earthquakes, have shown increasing skill such that they are competitive with, and are beginning to outperform, statistical models. Coulomb failure models benefit particularly from deeper catalogs because they include many more small magnitude earthquakes. These small earthquakes add predictive power through their secondary triggering effects tracking the evolution of the fine-scale stress field that ultimately controls earthquake nucleation in foreshock and aftershock sequences. They can also be used to define the emerging active structures that comprise fault networks and by doing so clarify the relevant components of stress that would act to trigger earthquakes15 are particularly well-suited tool for that effort.We conclude that it is now possible to image the activity on active fault systems with unprecedented spatial resolution. This will enable experimentation with familiar hypotheses and enable the formulation of new hypotheses. It seems certain that the underlying processes that drive earthquake occurrence are encoded in this next generation of earthquake catalogs, but we may not find them unless we put new effort into searching for them. Unsupervised learning methods