The American Center for Reproductive Medicine's summer internship course in reproductive medicine and research at Cleveland Clinic is a rigorous, results-oriented annual program that began in 2008 to train both local and international students in the fundamentals of scientific research and writing. The foremost goal of the program is to encourage premedical and medical students to aspire toward a career as a physician–scientist. The internship provides participants with an opportunity to engage in original bench research and scientific writing while developing theoretical knowledge and soft skills. This study describes selected survey responses from interns who participated in the 2014 internship program. The objective of these surveys was to elicit the interns' perspective on the internship program, its strengths and weaknesses, and to obtain insight into potential areas for improvement.Questionnaires were structured around the five fundamental aspects of the program: 1) theoretical knowledge, 2) bench research, 3) scientific writing, 4) mentorship, and 5) soft skills. In addition, an exit survey gathered information on factors that attracted the interns to the program, communication with mentors, and overall impression of the research program.The opportunity to experience hands-on bench research and scientific writing, personalized mentorship, and the reputation of the institution were appreciated and ranked highly among the interns. Nearly 90% of the interns responded that the program was beneficial and well worth the time and effort invested by both interns and faculty.The outcomes portrayed in this study will be useful in the implementation of new programs or refinement of existing medical research training programs. The American Center for Reproductive Medicine's (ACRM) summer internship program at Cleveland Clinic  is a 7-week annual course in recent advances in reproductive medicine. The goal of the program is to offer students a closely mentored, initial foray into the dynamic world of research in medicine. This is achieved through the objectives of the internship program, which is, for participating interns to 1) discover new knowledge from topics in reproductive medicine and other selected areas of medicine, research, and soft skills, by speakers chosen for their expertise in clinical, surgical, research, and scientific fields; 2) apply skills gathered through the daily mentoring of research work by experienced physicians and/or scientists; 3) to discover the art and science of how bench research is conducted; 4) apply skills of data compilation and statistical methods of analysis in basic science research; 5) compose the writing of scientific articles and reviews in reproductive biology and human infertility; and 6) synthesize research findings into a formal oral/PowerPoint presentation, then discuss and defend these research findings to a team of experienced scientists.A key strength of the program is that during this intensive internship, scientists and clinicians take on roles as preceptors to mentor and teach interns the fundamental principles of bench research and scientific writing. As a whole, the internship course is founded upon five integral pillars – theoretical knowledge, hands-on bench research, scientific writing, mentorship, and soft skills. Through the internship activities, the interns obtain a foundation for a clinical or research career in medicine or its related sciences. The significance of this internship course is evident by the fact that within the past 5 years, Case Western Reserve University School of Medicine has recognized and awarded the program three times, for making a positive impact on medical education and careers of students.With each passing year, the applicant pool has become more competitive and selective. Students accepted into the program are academically driven and were found to demonstrate a high level of intellectual curiosity based on their performance during the course of the program. Interested candidates complete an application that requires them to submit a copy of their resume and to answer a few essay-type questions that explore a variety of areas, including their interest in reproductive medicine, prior research experience, and future career plans. Most students who apply have had some form of medical or science-related educational training and/or are considering a career in medicine or research. Prior research experience is not a requirement, and in fact, most of the past interns had no previous research training.Since the inaugural class of 2008, the program has enrolled a wide variety of students with varying academic backgrounds from all over the world. Most have been either undergraduate or medical students along with some who are completing a residency or postgraduate program. The initial course (2008) consisted of 14 students and, by the seventh annual program (2014), grew to 25. The 2014 group consisted of 3 students from the United States (12%) and 22 international students (88%).The key features of an effective training program include the ability to provide proficient mentorship, hands-on application, teaching, and learning strategies that are tailored toward the learner and construct validity . HoweverDuring the summer of 2014, a total of 12 surveys were conducted – one before the start of the internship, 10 during its 7-week duration, and one exit survey at the conclusion of the program. These survey questions were designed using questionnaires that have been administered in a comparable setting serving as a reference point. The variables chosen were those that were of our interest that would elicit student's responses about the internship in order to help improve the program. The verbiage and construction of questions have been refined through the years to be more focused on key outcome areas of the internship program. Prior to administration, all survey questionnaires were subjected to review by language and content experts who are familiar with the program.Each survey contained between 9 and 18 questions, except for the first, penultimate, and final surveys, which had 51, 27, and 56 questions, respectively. Surveys were conducted via Google Forms (except the exit survey which was printed) and composed of qualitative questions in the form of ranking , multiple choice, and open-ended questions.The surveys analyzed various facets of each pillar of the program, including its logistics, implementation, and general reflections. More specifically, interns were surveyed on the pre-internship teleconferences, orientation, research communication/presentation exercises, mentors, lectures, scientific writing, bench research, and soft skills. In addition to the surveys held as the internship progressed, the comprehensive exit survey given to each intern at the end of the program was perhaps the most important of all the surveys because of the detailed information it sought on virtually every aspect of the program. Topics surveyed in the exit survey included factors that attracted the intern to the program, past experiences with scientific writing and bench research, incorporation of lecture information into scientific writing, communication with mentors, overall impression, and recommendation of the program to future candidates.We hypothesize that students who ‘graduate’ from this internship program upon fulfilling its objectives have improved research skills because of their participation in the course. In this report, we present the outcomes of the surveys assessing various aspects of the summer internship program completed by the interns in the 2014 batch as they progressed through the 7 weeks of the internship. In addition, the goal of collecting the interns’ perspectives via these surveys was to understand the interns’ view about the program as well as to glean insight into improving the following year's program.The internship surveys were considered mandatory and, thus, required a response by all the interns within a stipulated time frame. Among the respondents, 88% (22/25) were international students who had no prior internship-type experience in the United States. All 25 (100%) interns completed each survey. Selected outcomes of the surveys, which are categorized based on the five core pillars of the program, are presented here. Following this, survey responses regarding the pre-internship teleconference calls and feedback in the exit survey that summarizes the internship program are highlighted.Over the first 5 weeks of the internship, lectures were delivered by Cleveland Clinic faculty as well as invited scientists and/or clinicians. The lecture component covered a wide range of topics, from those related to reproductive medicine (with a focus on infertility) through the basics of conducting bench research and scientific writing, to public speaking and experience sharing . The aimIn general, responses to the surveys demonstrated that the lectures were beneficial in familiarizing the interns with the multiple layers of the program. Of equal importance to the interns were the talks given by scientists and clinicians on various reproductive physiology and pathology topics. Nearly all interns found these lectures valuable in furthering their understanding of their research assignments and applying their scientific knowledge to a bench research project.Among the lectures delivered on scientific writing, ‘library resources’ and ‘how to write an informative and concise, yet interesting title, abstract, and introduction’ were found to be most useful by 64% (16/25) of the interns. When asked about the usefulness of the talk that discussed how to conduct an exhaustive literature search on a scientific writing topic, most of the interns responded that they had previously heard of the process and search engines, but lacked confidence in completing the database search on their own. However, after several training sessions from the faculty and librarians, the interns felt that the step-by-step exercise gave them more confidence and clarity in conducting the database search without assistance . Similarsee Soft Skills section).Furthermore, the lectures on presentation skills, leadership and team work, and time-management skills helped boost the interns’ confidence when presenting their respective research projects, working together as a team in a culturally diverse group, and managing the intense internship workload, respectively. These soft skills were nurtured and refined throughout the internship through its daily activities  of interns chose the opportunity to participate in an original bench research project as one of the most important factors when deciding to apply to the program . Based osee Soft Skills section).Under the direct supervision of a reproductive biologist/researcher who serves as the lead mentor for their assigned project, interns undergo vigorous training to become proficient at performing the relevant techniques pertaining to their project. Before they can qualify to assist in the actual research project and conduct the experiments, interns must have achieved a sound understanding of the project's basis and goals, and verification by their project mentor and the program director. This research opportunity encourages interns to use their background knowledge to solve current clinical problems. Furthermore, by performing bench work, interns are driven to progressively develop leadership, teamwork, and time management skills (see Mentorship section).A detailed questionnaire surveyed the interns on their bench research experience. Most noted the mentorship they received and the ability to work as a team as the two most valued qualities of the experience. Mentors trained interns in all the required research techniques, assisted throughout all aspects of the research project, and guided them in preparing for their presentations. Nearly all interns  were either more than or very comfortable in confiding in their mentors regarding the difficulties that they faced while conducting bench work . All intFurthermore, interns realized the importance of working as a team in all aspects of the project. Without the proper communication and willingness to work together, the interns believed that their bench research results would be compromised and that the day would drag on late into after hours. Teamwork was identified as the most vital aspect of the project's success. Time management also became an essential factor, particularly on bench research days. Interns found that they not only had to be prepared at all times but also manage their other tasks, such as presentations, while maintaining smooth progress with their writing assignment. Overall, interns rated the hands-on bench research experience as one of the most rewarding aspects of the program.One of the most important aspects of the scientific process is the ability to communicate knowledge in a concise and effective manner. Another major pillar of the summer internship program is scientific writing for publication. In fact, 76% (19/25) of interns chose this as one of the most important factors when deciding to apply to the program . Internssee Mentorship section).A series of lectures given in the first few weeks provided the foundation of how to research, write, and present their topic. Besides theoretical lectures on scientific writing, the interns also participated in sharing sessions with the mentors, who shared their own perspectives, experiences, and tips through talks focusing on research and publications. Most interns found these topics useful or very useful  . All intInterns felt that the related lectures, one-on-one mentoring, and opportunities to deliver oral presentations on their topics assisted them in their understanding of the subject and helped them successfully complete this important undertaking for the first time. From their first 2-min presentation, interns responded that the factors they had found most difficult to do were 1) summarizing the large amount of information on their topic within the time limit given ; 2) time management ; and 3) not fidgeting/not standing frozen in front of an audience . Only 8% (2/25) interns responded that it was most difficult for them to hear/receive feedback from all the mentors/fellow interns. One of the interns stated, ‘I am excited about the next presentation because I hope to do much better than this time’. By the end of the internship, nearly all interns responded that the ability to contribute to the medical field in some way through their writing was a great feeling and accomplishment.One of the most straightforward approaches to encourage and promote translational research is to promote the structured mentoring and training of prospective researchers . For thisee Soft Skills section).Scientific writing and bench research are two enormous tasks that are undertaken by the interns. The mentors guide and evaluate the interns throughout their scientific writing and bench research projects and for their oral presentations of their projects. These exercises provide interns with real-time constructive feedback, which helps strengthen their understanding of their research topic and stay on track with their scientific communication and in their work moving forward. Furthermore, the oral presentation exercises and subsequent personalized feedback by mentors allow the interns to improve their ability to effectively communicate their scientific findings and speak in front of an audience , as these interns are, prefer working under the supervision of those who are approachable and supportive as well as good communicators and motivators . A good Prior to presentations, the mentors played a key role in assisting interns in their preparations. Most students relied on or consulted with their mentors for assistance and suggestions. Before their bench research presentation, interns responded that presenting among their team members  and then to the mentor as a team  and receiving feedback was beneficial. The preparation with their mentors made them more comfortable in presenting and they felt that they did well for their actual presentation after practicing once with their mentor. One intern responded, ‘My mentor was really helpful in every stage. He listened to our presentation, gave feedback, helped us find resources, and was very encouraging, giving me confidence for the presentation’. Most of the interns  felt that the mentors were very helpful in guiding them for their presentations. When asked what their mentors could do to help further improve the interns’ performance, the interns’ response was ‘nothing’ and ‘just continue doing what they are doing’.Soft skills are a collective term for skills such as leadership, communication, interpersonal skills, attitude, and work ethics, which help an individual work and socialize in a positive manner. Soft skills such as these complement the individual's hard skills, which are his/her knowledge and occupational/technical skills. Effective soft skills are much sought after in the workforce for their contributory role in career growth and as a distinguishing factor of an individual's caliber among his/her peers. In research and medicine, prominence is placed on effective communication and interpersonal skills. Presentation and public speaking skills are extremely valuable commodities, particularly in medical research, as they represent an ability to disseminate scientific knowledge in a clear, concise, and methodical manner. Moreover, the concepts of professionalism and teamwork are given equal emphasis, in order to guarantee outcomes of the highest quality.The internship offers several lectures to the interns on these topics, providing them with valuable information and opportunities to build on each of these skills. The interns surveyed thought that the information on public speaking was most helpful . Although these concepts were not entirely new to the interns in this stage of their studies , there wAs was described previously, several oral and PowerPoint presentations for both scientific writing and bench research were assigned periodically during the internship. These presentations provided a summarized look at each project and helped solidify the interns’ comprehension of their research topics. During these sessions, interns presented their research topic and the progress made, while all the mentors evaluated them and individually provided suggestions for improvement. Many, if not all, interns found the feedback given to them on their presentation and delivery style to be beneficial. One student noted, ‘The feedback was spot on, as they were critical and correct. They praised all the good qualities in me but at the same time criticized qualities which I needed to improve on’. The suggestions obtained from each of the mentors were applied in the interns’ next oral presentation, thus demonstrating the continuous impact of the mentorship process. In addition to mentor feedback, fellow interns were also encouraged to share their thoughts on their peers’ performance, which was found to be helpful.Most of the interns stated that they became more comfortable with public speaking because of these presentations. Furthermore, 96% (24/25) found this to be a valuable experience. As such, all (100%) thought that having additional time to practice would have improved their performance. Hence, adjustments to allow more time for practice sessions were taken into account in the following year's internship schedule (in 2015).Personalized mentorship remains a striking feature of the internship experience. Interns and mentors must foster clear channels of communication, both in written and verbal form, in order to forge successful, lasting relationships. Efficient communication between the interns and mentors enhanced the interns’ success and experience, whether in the laboratory, during writing, or in other activities of the program. As stated by an intern, ‘Our environment (in the laboratory) is low pressure and efficient because the mentor and the interns have a good relationship’. Exchange of frequent and consistent updates (interns to mentors) and feedback (mentors to interns) was essential to ensuring that the interns successfully completed their bench and writing projects. Individual meetings with mentors on scientific writing were found to be extremely helpful as well. During these meetings, the interns updated their mentor on any findings, revisions, and plans on moving forward.Bench research makes up a significant portion of the internship curriculum. Teamwork was vital in completing the projects in an efficient and orderly manner, particularly in the bench research activities and presentations, which entailed unified work in a small group. Many interns reiterated that working as a cohesive unit and the constant involvement of their peers substantially improved teamwork as a whole.Two months prior to the start of the internship, two teleconferences were held – 1 month apart – involving all the accepted interns from their respective home cities and the faculty/mentors at Cleveland Clinic. The purpose of the first teleconference was to acclimate the interns to the structure, policies, and expectations of the program whereas the second delved into more details regarding the curriculum and day-to-day logistics. The meeting agenda was provided ahead of time to allow interns to be aware of the topics discussed and prepare any questions that they may have. Responses from the interns  indicated that this was helpful in providing a well-organized, focused, and purposeful discussion.Throughout each teleconference, students responded that they felt encouraged to freely ask questions  and convey any apprehension on their part. The interns unanimously  indicated that these calls helped to alleviate any fear and uneasiness that they may have had regarding the upcoming program. As a result of the teleconferences, all 25 interns (100%) felt that they were able to better associate themselves with their future mentors and the program's general structure.All 25 interns responded that the calls showcased the quality of their prospective mentors and helped to increase their understanding and expectations of the program. In addition, 92% (23/25) found that the teleconference calls presented a helpful avenue to connect with their fellow interns in advance of the internship.We reiterate here that communication between mentors, interns, and the course coordinators is crucial to the success of any program. The preferred methods of communication for 80% (20/25) of the interns were e-mail (uniformly Gmail) and WhatsApp. These efficient means of communication allowed for timely instructions and updates to be communicated not only throughout the internship but also prior to the program.Overall, the teleconference meetings allowed the interns to feel more comfortable  prior to their arrival and that they were going to be in good hands . Furthermore, their interaction with the faculty during the teleconference increased the interns’ confidence in their mentors  and emphasized the significance of the internship program  from early on.The first day of any educational program is a crucial one, as this is when students formulate their foremost impressions of their surroundings, colleagues, and the program itself. During the program's orientation process, a structured, hour-by-hour daily schedule was distributed along with other essentials. This schedule was carefully designed months ahead of the program's start date in order to facilitate smooth progression of activities over the course of the 7 weeks. Prior to obtaining the physical copy of the internship materials, an electronic copy was sent to the interns, which allowed them early access to important internship information. One intern stated, ‘I really liked how everything is really organized and prepared for us in advance’. The entire class  believed it was beneficial to have the internship material beforehand, as this allowed interns to prepare their thoughts and provided them with information on what to expect from the program.Prior to their arrival, interns had to also read and sign a policy document explaining the ACRM's policies and procedures. All interns  believed this to be helpful and informative as it allowed them to have a basic understanding of what was expected of them from the start. More than 95% (24/25) of the students thought that the orientation process was streamlined and well structured. One intern described the orientation experience as, ‘It was a smooth and comfortable process, and the staff was very welcoming’. Other interns were of the opinion that the orientation sessions were ‘helpful’, ‘well organized’, ‘detailed’, and ‘thorough’. This feedback helped to validate the effectiveness of the orientation process.Cleveland Clinic is a world-renowned medical institution that is consistently recognized for its commitment to excellence and leadership in both the practice of medicine and in research innovation. Interns were surveyed on aspects that contributed to their decision to apply to the program . ClevelaThe exit survey also looked at the overall experience of the student, asking them to describe the internship as excellent, very good, good, fair, or poor. A resounding number of interns  rated their experience in the category of excellent, very good, or good. Given this, it can be inferred that the common viewpoint of more than 80% of the class regarding the program was a positive one. And as with any other course that provides an educational service, one of the inherent factors to look at is the target audience's overall impression of the service offered. Thus, it is reassuring to know that 88% (22/25) of the interns found the program to be beneficial and well worth the time and dedication, on both the part of the interns and the faculty.One of the main reasons why the surveys were conducted as compulsory questionnaires throughout the course is that, in spite of the success of any program, there is always room for improvement. Based on their interview at the time of joining the course, interns expected through their participation in the internship to gain knowledge and/or skills in: topics in reproductive and other selected areas of medicine, research, and soft skills; daily mentoring of research work by experienced scientists; the conduct of bench research and data collection; writing of scientific articles or reviews in reproductive biology and human infertility; and being able to summarize, discuss, and defend their research findings, then present these findings in a formal oral/PowerPoint presentation to a team of experienced scientists. These interns’ expectations aligned well with our course objectives.Information garnered from the survey responses proved our hypothesis and demonstrated the program's effectiveness in mentoring and training students in scientific writing, bench research, and effective communication, thus fulfilling the interns’ expectations from participating in the program. Responses from these surveys gave a sense of overall impressions of the interns, allowed for the strengths and weaknesses of the program to be captured, and generated suggestions for improvement in upcoming programs. By scrutinizing the collective responses of the interns, the ACRM team was able to use the feedback to refine the program. For example, several interns had expressed dissatisfaction about having too many lectures and not enough time for self-study to work on scientific research and bench research. This feedback was taken into consideration in the planning of the 2015 ACRM summer internship program, and the number of lectures was significantly decreased. Timely feedback of the interns’ perspectives was valuable in helping lead the program along an enhanced and effective route.The advantages of collecting participant feedback are that it is cost effective, is relatively straightforward to carry out, and can potentially be completed in a timely manner by all participants. The disadvantage of these types of surveys is that the respondents may feel encouraged or obliged to provide positive answers that do not reflect their true thoughts, or rather, provide answers that present themselves in a favorable manner to the faculty. It is also very easy to have bias within a survey, depending on the respondents. A major challenge was that the interns became tired of completing the numerous surveys within a short span of time, and some required reminders to complete them on time.A proud achievement of the program is that nearly all past medically qualified international interns that went on to apply to a residency program in the United States have received a positive outcome. These interns themselves have attributed their participation in this internship program as a major determinant of their success because of the interest it generated during their interviews for residency positions. This is partially due to the fact that, to the authors’ knowledge, there is no other program currently in the country that allows for an intern to be intensively engaged in the conduct of both bench research and scientific writing during a relatively short period of time. Being a published individual is an admirable feat, and the fact that this internship provides the opportunity for an intern to participate in mentored scientific writing from conception to publication surpasses other similar programs.In conclusion, this study has described how participation in our internship provides the opportunity for interns to gain knowledge and/or skills in current topics in medicine and research from renowned speakers, experience effective daily research mentoring by resourceful and results-oriented mentors, engage in hands-on conduct of bench research and data collection, experience the writing of novel scientific research, and hone their presentation skills. Based on the responses highlighted in this study, we highly recommend that a similarly-structured mentorship program be implemented in other branches of medicine or medical sciences, in order to present more opportunities for students to gain significant exposure to research at an early stage of their careers.To the authors’ knowledge, the program's goal of offering students an insight into the vast area of research in medicine has not been accomplished in the same intensity and success by any other program. The ACRM summer internship program at Cleveland Clinic continues to grow and be refined each year, and the applicant pool has become more competitive and selective, despite an increase in the number of interns accepted per class. As the program goes into its ninth year in 2016, and more recognition is being awarded to the program, it is convincing that the program has a bright future ahead of itself.


Software testing is a very labor intensive and costly task. Therefore, many software testing techniques to automate the process of software testing have been reported in the literature. Assertion-Based automated software testing has been shown to be effective in detecting program faults as compared to traditional black-box and white-box software testing methods. However, the applicability of this approach in the presence of large numbers of assertions may be very costly. Therefore, software developers need assistance while making decision to apply Assertion-Based testing in order for them to get the benefits of this approach at an acceptable level of costs. In this paper, we present an Assertion-Based testing metrics technique that is based on fuzzy logic. The main goal of the proposed technique is to enhance the performance of Assertion-Based software testing in the presence of large numbers of assertions. To evaluate the proposed technique, an experimental study was performed in which the proposed technique is applied on programs with assertions. The result of this experiment shows that the effectiveness and performance of Assertion-Based software testing have improved when applying the proposed testing metrics technique. Software testing is a very labor intensive and costly task. Theretofore, many software testing techniques to automate the process of software testing have been reported in the literature, for example, –8. ThereIt has been reported in  that AssAlthough Assertion-Based testing is a promising approach in terms of detecting program's faults, the applicability of this approach in the presence of larger numbers of assertions may be costly. This cost is related to search time required for finding a test data to violate such a large number of assertions which may hamper the effectiveness of Assertion-Based testing and makes it impractical for industrial-size software. Therefore, software developers testing need to decide when to apply Assertion-Based testing in order to get the benefits of this approach at an acceptable level of costs. In this paper, we present an Assertion-Based testing metrics technique that is based on fuzzy logic techniques . The maiThe rest of this paper is organized as follows. Assertion-Based testing is presented in Assertions have been recognized as a powerful tool for automatic run-time detection of software errors during testing, debugging, and maintenance, for example, –17. An aA, the goal of Assertion-Based testing is to identify program input for which A will be violated. Assertion-Based testing is a goal-oriented approach [Given an assertion approach –3 that iapproach  is assocThe main aim of Assertion-Based software testing is to increase the developer confidence in the software under test. Therefore, Assertion-Based software is intended to be used as an extra and complimentary step after all traditional testing methods have been performed to the software. Assertion-Based testing gives the tester the chance to think deeply about the software under test and to locate positions in the software that are very important with regard to the functionality of the software. After locating those important locations, assertions are added to guard against possible errors with regard to the functionality performed in these locations.and, or, not) operators. There are two types of logical expressions: Boolean expression and relational expression. A Boolean expression involves Boolean variables and has the following form: e1 op e2, where e1 and e2 are Boolean variables or true/false constant and op is one of { = , ≠}. Relational expressions, on the other hand, have the following form: e1 op e2, where e1 and e2 are arithmetic expressions and op is one of {<, ≤, >, ≥, = , ≠}. For example, (x < y) is a relational expression, and  is a Boolean expression.An assertion may be described as a Boolean formula built from the logical expressions and from the  operators. In [1 op A2, where A1 and A2 are Boolean variables or true/false constant, and op is one of { = , ≠}. On the other hand, relational expression has the following form: A1 op A2, where A1 and A2 are arithmetic expressions and op is one of {<, ≤, >, ≥, = , ≠}. For example, (x < y) is a relational expression, and  is a Boolean expression. For example, the following is a sample assertion: ∗@ (x < y) and  @∗),A: (which will be translated during a preprocessing stage [  not ((x < y) and ),if then Report_Violation;where Report_Violation is a special procedure which is called to report an assertion's violation. As reported in , an assetors. In , Pascal ng stage  into theOne of the major problems faced during software testing is deciding when to stop the testing process . This isAs stated in , AssertiThe proposed testing metrics approach works as follows. Give a program P with assertions, before applying Assertion-Based testing on P, and this approach starts by classifying the P units, that is, modules or procedures, based on two main criterions: importance and confidence. The main objective of the proposed approach is to make Assertion-Based software testing more efficient and effective. To satisfy this objective, more testing resources will be directed toward those parts of the program that need to be tested more. For example, a program unit that performs dynamic computations should more important than a graphical interface unit that displays the company's logo. Because it is difficult to quantify the notion of “importance” and “confidence,” we propose to utilize the powers of fuzzy logic techniques , 26. TheAt first, the proposed approach orders each program's unit according to their importance with respect to the functionality of the program under test. Using fuzzy logic terms, this classification is mapped into a single fuzzy set , named IU, this step is achieved by asking each developer to specify a percentage value to express her confidence in the correctness of U based on her believes in the effectiveness of tests that were performed to test U previously. As with the issue of importance, it is very difficult to express a developer confidence level, in a certain unit U, within crisp boundaries agreeable by different people on various types of programs. Therefore, we create a fuzzy set named, CONFIDENCE, in order to capture a developer's confidence in the correctness of a given program's unit U.In the next step, the proposed approach measures the confidence level of the developer in the correctness of each of the program's units. Given a program unit U = {u1, u2,…, un}, where each unit, uk, has a number of program assertions: Auk = {a1, a2,…, am}. The application of the proposed approach for Assertion-Based software testing metrics on P proceeds as follows. First, all units of P are identified and classified according to their importance to the correctness of the program P. The result of this step is that each program's unit, uk, will belong to the IMPORTANCE fuzzy set with a certain membership degree. As mentioned earlier, because the notion of importance is subjective and may differ from one software to another and from a developer to another, we utilize the flexibility of fuzzy logic [S-function [uk, to the fuzzy set.For presentation purposes, let P be a program under consideration with units zy logic  in orderfunction , shown ifunction , this me S-function is as follows:Theuk, based on previous tests results applied to test uk. The data obtained from developers is mapped using the S-function to create each unit, uk, membership degree to the fuzzy set, CONFIDENCE, mentioned above with a certain membership degree, ukm.Given a set of programs units that were classified according to their importance, the second step of the proposed approach is to decide whether or not to apply Assertion-Based testing to these classified units. In order to make this decision, we need to estimate the developer's confidence on the correctness of each unit based on her experience with previous tests, for example, black-box and white-box, that were applied, previously, to test program P. As with the notion of “importance,” the notion of “confidence” is subjective and may differ from one person to another. Therefore, for the purpose of capturing developers confidence in the correctness of their programs, we asked them to express their confidence on the correctness of each program unit, S-function may be described as follows:a mathematical function that is used in fuzzy sets as a membership function;a simple but valuable tool in defining fuzzy functions such as the term “confidence”;rather than mainlining a table defining the membership function, all data may by compactly represented by a formula;α, β, and γ are parameters which may be adjusted to fit the desired membership data;S-function is flat at a value of zero for x ≤ α and at 1 for x ≥ γ. In between α and γ, the S-function is a quadratic function of x. The β parameter corresponds to the crossover point of 0.5 and is (α + γ)/2.the The In order to illustrate the functionality of the proposed technique, consider the following example. Let P to be the first program used in our experiment, “Boyer-Moore,” as shown in S-function, as follows. In order to use the S-function, we have to find what values to be used for the parameters: α, β, and γ. Based on the data obtained from the developers regarding the importance of the units of this example program, we set the values of these parameters as S-function [α represents the least value a program's unit may receive and the value of γ represents the most value a unit may receive. Specifically, the value of 10 for α parameter means that no unit of this specific program, P, is less than 10% of importance to the functionality of the whole program. Similarly, the value of 100 for γ parameter says that a unit may be extremely important to the functionality of P. Note that the values of α and γ parameters may be adjusted to fit the desired membership data. By plugging these values in the S-function of /2 = 50. Consequently, the CONFIDENCE fuzzy set is constructed as follows: CONFIDENCE = {0/Unit_1, 0.9/Unit_2, 0.5/Unit_3, 1.0/Unit_4}.In addition to the “importance” criterion, the proposed approach presented in this paper considers the “confidence” criterion to improve the performance of Assertion-Based software testing. Therefore, in the next step, this approach repeats same process of classifying program's P according to the “confidence” data obtained from the developers as follows. To obtain the “confidence” for this example program, Boyer-Moore, developers were polled to express their confidence in the correctness of this specific program. The result is shown in the first table labeled, “Boyer-Moore,” in d out in , AssertiThe values in the CONFIDENCE fuzzy set may be interpreted as follows. The developers have no confidence at all in the correctness of Unit_1. All developers are highly confident that Unit_2 and Unit_4 are fault-free and may not need further tests, while they expressed only 50% confidence in the correctness of Unit_3. Based on the results of this analysis, the proposed approach will concentrate on those programs unit that have low values of in the membership of the CONFIDENCE fuzzy set. For this example, most allotted testing resources will go to Unit_1, followed by Unit_5. Note that without this knowledge, Assertion-Based testing would have been performed equally to all the units in the program without discrimination.From this example, it is illustrated how applying the proposed technique for Assertion-Based testing metrics may provide a technical and valuable help for the developer while making her decisions about applying Assertion-Based software testing on her software. Additionally, the developer is able to direct their valuable testing resources and efforts towards those parts of the software that are considered to be mostly important and may require additional tastings. As a result of applying the proposed approach, Assertion-Based testing will only be applied to some selected units instead of applying this method to the whole software. The results of this is that the proposed approach will only explore some of the assertions found in the program as compared to exploring all assertions when Assertion-Based testing is applied blindly to the whole software. Therefore, saving valuable search resources, especially when testing programs with large numbers of assertions.To evaluate the performance of the proposed technique for Assertion-Based testing metrics, we have performed an experiment. The main objective of this experiment is to show that the proposed technique may be utilized to direct testing resources towards those spots of the programs that need more testing. This experiment is designed as follows. We have asked three programmers to write ten different programs  such thaIn the first stage of this experiment, we have seeded some program faults in these programs. After that, we subjected all of the programs to black-box and white-box testing techniques . SpecifiIn the fourth stage of this experiment, we have asked the programmers to express their confidence on the correctness of their programs according to the proposed testing metrics technique presented in As indicated by the results of this experiment, applying the proposed testing metrics technique has resulted in an improvement in the performance of Assertion-Based testing as measured by this approach ability to violate more assertions as compared to the case before applying the proposed technique. This encouraging result indicates that using the proposed testing metrics technique has enhanced the performance of Assertion-Based testing by making it utilizes allocated testing resources and to concentrate on violating assertions in certain parts of the program instead of wasting these valuable resources trying to violate all assertions found in the program.In this paper, we presented a novel software testing metric technique for Assertion-Based software testing that is based on fuzzy logic technology. The main goal of the proposed approach is to enhance the performance of Assertion-Based software testing in the presence of large number of assertions. Furthermore, this approach assists software developers while making their decisions on applying Assertion-Based software testing to their software. In order to evaluate the proposed technique, we have conducted an experimental study in which this technique was applied on a set of programs with assertions. The results of this experiment is very encouraging where applying the proposed approach has enhanced the performance of Assertion-Based testing as shown by the increase on the number of assertions violated in the programs considered in the experiment. In the future, we intend to perform more experiments to evaluate the performance of the proposed technique while applying on commercial size software with large numbers of assertions.


Midlands Province started implementing the Isoniazid (INH) preventive therapy (IPT) program in January 2013. Shurugwi and Gokwe North were the piloting district hospitals. In May 2014, four more districts hospitals  started implementing IPT. Shurugwi District decentralized the program to its rural health facilities in January 2014. A review of the Shurugwi IPT program, 2013 data, indicated that the majority of eligible clients were not started on IPT. None out of the 400 eligible clients were started on IPT in November against the 100 % target according to the World Health Organization and the National Tuberculosis (TB) Program. We conducted a study to evaluate the IPT program in Shurugwi District from January 2013 to August 2014.The logical framework approach was used to evaluate inputs, processes, outputs and outcomes of the IPT program. An interviewer administered questionnaire was used to collect data from key informants. Checklists were used to collect data from IPT program records.Sixteen health facilities were implementing IPT in Shurugwi District. All the facilities had TB screening tools and three did not have TB screening algorithms. The district experienced medicine stock outs in 2013. One formal training at district level and on job trainings in implementing health facilities were done. From January 2013 to August 2014, Shurugwi District screened 6794 antiretroviral (ART) clients for TB. Out of those screened, 5255 were eligible for IPT and 2831 (54 %) were started on IPT. A total of 700 clients had completed the IPT 6 month’s course by August 2014. The dropout rate due to INH toxicity and TB was 0.6 % (n = 18) and 0.3 % (n = 8) respectively. Fifty-three advocacy and community sensitization meetings were done. The program had no Information Education and Communication (IEC) materials.The IPT program in Shurugwi District achieved half its target. This could be due to inadequate formally trained staff, lack of IEC materials, inadequate advocacy and community sensitization, non-availability of the INH 300 mg single dose and inadequate INH 100 mg dose tablets in 2013. To improve the IPT program, there is need for routine advocacy, communication and social mobilization. Isoniazid Preventive Therapy (IPT) is the provision of isoniazid (INH) to people at high risk of developing active tuberculosis (TB) . It is aThe World Health Organization (WHO) recommends IPT as part of the three I’s for HIV/TB collaboration. These include, intensified TB case finding, INH preventive therapy and Infection control for TB in health care and congregate settings . The curAdults and adolescents living with HIV should be screened with the WHO symptom checklist for TB, to rule out active TB infection. Those reporting one or more symptoms may have active TB and should be further evaluated and then treated for TB .WHO has recently revised the guidelines on IPT that recommends the use of a simplified screening algorithm that relies on the absence of all four clinical symptoms  to identify people living with HIV (PLHIV) who have less likelihood of active TB disease and hence eligible for IPT . This siAccording to WHO, before initiating a service to provide routine IPT to PLHIV, some prerequisites should be in place. These are a strong linkage between HIV care and TB control services, adequate capacity for HIV counseling, which should include information, education and counseling about TB, health care staff well trained in pre-treatment screening to exclude active pulmonary and extra pulmonary TB, resources available for close monitoring, supervision and evaluation of IPT outcomes, TB control programme with high adherence, high cure rates (≥70 %), case detection (≥70 %) and combined default/failure rates of less than 10 % .Globally, almost a quarter of a million PLHIV, that is 242,847, were reported to have received IPT between the period 2002 to 2009, showing a 19-fold increase over the period . DespiteIn sub-Saharan Africa, where the emergence of HIV has caused dramatic increases in tuberculosis (TB) case notifications, IPT for HIV positive individuals was evaluated as a potential community-wide strategy for improving TB . ZimbabwBy the end of June 2014, the number of sites implementing ICF/IPT had expanded to 46 in total with each of the 10 provinces in the country having at least two (2) to six (6) district level institutions . The 46 Midlands Province started implementing IPT program in January 2013. Shurugwi and Gokwe North were the piloting district hospitals . In May A review of the Shurugwi IPT program, 2013 data, indicated that the majority of clients, eligible for IPT were not started on IPT. In 2013, three out 62 (5 %), zero out of 400 (0 %) and 4 out of 400 (1 %) of the eligible HIV positive clients were started in August, September and November respectively . The tarA descriptive study was conducted in Shurugwi District. The logical framework approach was used to evaluate inputs, processes, outputs and outcomes of the IPT program in Shurugwi District. Nurses in Charge (NIC) from the 15 health facilities, Opportunistic infections (OI)/ART Sister in Charge (SIC) from the district hospital, the District Medical Officer (DMO), TB coordinator, District Nursing Officer (DNO), Matron and Pharmacist were purposively recruited into the study as key informants. An interviewer administered questionnaire was used to collect data from key informants on resource availability, IPT processes being implemented both at district, health facility and community level, IPT program achievements and challenge. Checklists were used to collect data from OI/ART records, TB records and IPT registers.Ethical approval was obtained from the Provincial Medical Director (PMD) for Midlands Province, Health Studies Office (HSO), and DMO for Shurugwi District. Confidentiality was assured and maintained throughout the study. Participants were told that there were no monetary benefits for participating and that participation was voluntary. Informed written consent was obtained from each participant.Out of the 21 study participants, the majority (71.4 %) were Nurses in Charge of the 15 health facilities  offering IPT in Shurugwi District. The median years in service was 8 years   the required. In 2013 the district hospital had both IPT guidelines and TB screening tools. In 2014 all the 16 health facilities had TB screening tools and only 3 did not have IPT guidelines. The district never produced or received Information, Education and Communication (IEC) materials on IPT. All the 175 village health workers (VHW) were sensitized on IPT in 2013  ART clients for TB. Out of those screened, 5255 (77 %) were eligible for IPT and 2831 (54 %) were started on IPT. About 10 (63 %) of the health facilities conducted a total of 53 community sensitizations on IPT. However, the program had no IEC materials to compliment community sensitizations  of the clients stopped IPT due to INH toxicity and eight (0.3 %) clients developed TB during IPT.IEC materials provide key messages to educate communities on the benefits of IPT, thereby marketing the program and clearing myths and misconceptions about IPT. Shurugwi District had no IEC materials on IPT. The National TB Program did not produce IEC materials to educate the public. Indumathi et al. found that program uptake can be enhanced by adequate education at the start of the therapy and during subsequent follow ups . This imAdvocacy and community sensitization are strategies that are implemented to enhance community participation and involvement of local leadership. If communities are involved and gain the ownership of the program, uptake improves. Advocacy and community sensitization for program success was not well capacitated by the district hence low program uptake. The district conducted advocacy and community sensitizations for the IPT program mainly when the program started in 2013. We concluded that there was need to continue advocating for the program through adequate involvement of the local community leadership and the community at large in program implementation. Similar findings were reported by Durovni et al. where he recommended advocacy efforts for effective scale up of the program .The target for IPT initiation is 100 % according to the World Health Organization and the National TB Program. All eligible clients are expected to be started on IPT. In Shurugwi District, half of the eligible clients were started on IPT, however this was more than double the national picture where almost a quarter of the eligible clients were started on IPT from January 2013 to May 2014. INH stock outs were experienced in 2013 when the program started leading to failure to start some clients on IPT in that same year. In Shurugwi District it was noted that some clients declined IPT due to fear of pill burden as they were already on ART and Cotrimoxazole, making a minimum of 10 tablets per day for those who accepted IPT.All clients started on IPT are expected to complete the 6 months course. Treatment completion was high mainly because a small proportion stopped due to development of INH toxicity and TB during IPT. Completion of the IPT course implies successful control of TB among ART clients for a period of 2–3 years after IPT. Similar findings were reported by, Durovni et al. in 2010 who recorded a high adherence and completion rate among ART clients .Screening algorithms provide important information on eligibility criteria. Lack of IPT screening algorithms may result in poor program implementation. The majority of the health facilities offering IPT in this study had TB screening algorithms. Granich et al., in 2010, found out that lack of standard operating procedures (SOPs), guidelines and screening algorithms contributes to the low availability and poor uptake of IPT . WHO higIn this study, we found out that the district hospital managed to decentralize the program to 15 other health facilities which is a positive achievement. However, challenges faced were shortage of staff in some clinics. Health workers were inadequate and overwhelmed because of many other competing program activities that call for their daily attention for example Provider Initiated Testing and Counseling (PITC), TB screening and treatment, (OI)/ART, Expanded Program on Immunization (EPI), Child growth monitoring, Prevention of Mother to Child Transmission of HIV (PMTCT) and deliveries. With adequate staff, IPT processes would be adequately implemented.It was observed that the IPT program was highly effective at screening TB among ART clients, this is intensified case finding. The majority of ART clients were screened for TB at every review visit. This implies early detection and management of TB, hence reduced transmission to others. The key informants highlighted that It was also during clinic visits that clients were assessed for adherence to treatment through pill count.Velle et al. found out that the most common reason for failure to adhere to and complete treatment was not visiting the health facility frequently . In thisThe IPT program in Shurugwi District achieved half its target. This could be due to inadequate formally trained staff, lack of IEC materials, inadequate advocacy and community sensitization, non-availability of the INH 300 mg single dose and inadequate INH 100 mg dose tablets in 2013. The district should improve Advocacy, Communication and Social Mobilization (ACSM) for the IPT program.As a result of this study, we gave health education on IPT to 15 ART clients, sourced and distributed A4 size TB/IPT screening algorithms to the clinics in need. We corrected some anomalies noted in IPT implementation where some clinics were not prescribing Pyridoxine together with INH to reduce the risk of developing side effects.The study recommended the DMO to train health workers who were not formally trained on IPT. The District Health Promotion Officer to produce and distribute IEC materials on IPT for educating health workers, clients and the community at large and to conduct advocacy, communication and social mobilization (ACSM) for IPT. Nat Pharm to provide the 300 mg single dose of INH for the IPT programme in order to reduce pill burden for clients.


As part of a cross-site evaluation of the implementation of an evidence-based intervention for pediatric asthma care coordination into low-income communities, we sought to understand the factors that influenced the programs’ expected sustainability of the programs after external funding ended.We administered the Center for Public Health Systems Science’s Program Sustainability Assessment Tool, a 40-item instrument assessing 8 domains of sustainability capacity, to 12 key informants across 4 program sites. We developed open-ended probes for each domain. We examined patterns in site-specific and overall domain scores, and coded qualitative data to identify challenges and strategies in each domain.Across sites, the domains of program evaluation  and program adaptation  had the highest ratings  and funding stability had the lowest rating . Scores varied most across sites in the domains of strategic planning  and funding stability . Qualitative data revealed key challenges, including how implementation difficulties and externally led implementation can impede planning for sustainability. Program leaders discussed multiple strategies for enhancing capacity within each domain, including capitalizing on the interconnectedness of all domains, such as using evaluation and communication strategies to bolster internal political support throughout the implementation process.Findings indicating weak and strong domains were consistent with previous findings of studies that used the Program Sustainability Assessment Tool. The addition of qualitative probes yielded detailed data describing capacity strengths, weaknesses, and strategies to increase the likelihood that programs are sustained. Understanding how effective health programs can be sustained over time is of interest to researchers, practitioners, and funders ,2. SustaThe literature on sustainability is fragmented , and mosYes We Can to fit each site’s priority population, organizational context, and leadership structure. Yes We Can is a medical-social model of care that provides asthma education, links families to health and social services, and facilitates patient–provider communication (The Merck Childhood Asthma Network (MCAN) funded 4 sites from 2010 to 2014 to demonstrate the effectiveness of care coordination in communities with significant disparities in asthma illness and death. This funding was for the second phase of a 2-part initiative to improve pediatric asthma care and outcomes in underserved communities . All 4 snication –19. The nication  had diffVersion 1 of the PSAT assesses 8 domains of sustainability capacity  with 5 items each. Respondents rate the extent  to which the program has or does what the item describes . PSAT subscales demonstrated high internal consistency and reliability in a sample of 592 respondents representing 252 public health programs . BecauseIn 2014, we conducted 12 key informant interviews  by telephone with each site’s principal investigators, program managers, and (at one site) the medical director of a partner clinic. Before the interview, the respondent was emailed the PSAT and asked to 1) complete the close-ended items and 2) think about answers to the open-ended questions. In audio-recorded interviews, respondents first reported their responses to the PSAT items and then answered the open-ended questions. A research assistant listened to all audio recordings to add details to interviewer notes and capture illustrative quotations.We calculated site-specific means for each of the 40 items. Site-specific domain scores were obtained by averaging item scores within a domain. Overall domain scores were obtained by averaging the 4 site scores for each domain, and standard deviations were calculated to show variability by site. We also calculated cross-site average scores for each item.To develop a coding scheme, notes from 3 randomly selected interviews were each independently coded by 2 evaluators. A priori codes were derived from PSAT items, with subcodes indicating strategies and challenges. Other codes emerged from data, and evaluators came to consensus on a codebook. Two researchers independently coded the remaining notes, compared codes and came to consensus, modifying the codebook as needed to ensure convergence and divergence of the coding scheme .Each sustainability capacity domain is defined, and mean domain scores across sites are presented . The domOpen-ended responses provided insights into domain ratings. These are presented below in descending order of average domain score.Program evaluation . Respondents indicated that they used evaluation findings to strengthen the program and its sustainability by tailoring messages for families, continuously improving the program, and generating and maintaining support. As one respondent put it, “People have to see a value in what you are doing before they completely invest in it.” Evaluations were strengthened by allocating sufficient resources, tapping into existing systems for data collection, and seeking regular input from staff and other stakeholders. Conversely, a few respondents reported that lack of access to health record data and inadequate process evaluation limited the effectiveness of evaluation on sustainability.Program adaptation . Adaptability was seen as critical to program success and sustainability, and no respondents cited barriers to program adaptation. Respondents cited several reasons for adaptations: to adjust to shifting political environments; to work effectively with key partners; to boost participant engagement; to accommodate clinical workflows; and to leverage resources. One respondent emphasized that adapting the program to be more cost-effective promoted sustainability by “showing stakeholders that the program is in a position to adjust to the financial reality they live in.”Organizational capacity . Respondents cited numerous organizational strengths promoting implementation and sustainability, including ongoing asthma quality improvement initiatives, ability to integrate coordinators into clinical workflow and staff meetings, coordinator access to electronic health records, a vast network of school nurses serving as a referral system, and support from key staff and board members. Respondents from all programs cited the ongoing challenge of organizations’ multiple competing priorities. Other commonly cited challenges were a lack of physical space, staffing, and extant referral systems.Communications . Communication was seen as enhancing sustainability primarily by enhancing political and financial support for the program. Effective strategies were using channels such as the organization’s communications office, staff in-services, and relationships with community leaders; tailoring messages to the audience; and emphasizing positive evaluation results and the program’s immediate benefit to families. One respondent described how communication efforts garnered national recognition for the organization, increasing internal political support to continue the program. Challenges were insufficient skills, resources, and channels for communication.Partnerships . Partner organizations and community members were viewed as contributing to sustainability by facilitating successful program implementation and offering political support. For example, the endorsement of community partners who act as gatekeepers to community members was critical to successful implementation at one site. Partnerships are especially critical for care coordination work, which seeks to connect families to resources such as housing assistance and smoking cessation. As one program manager said, “We’re not trying to do it all ourselves. We know our program limitations and we know our strengths, so if there is a partner that can help us, we definitely will tap into that.” Responses highlighted that partnerships are instrumental in generating political support to continue the work. For instance, FQHC board members successfully advocated for program components to continue beyond grant funding. Challenges were overcoming partner distrust, insufficient resources for a strong community advisory board, and low visibility in the community resulting from weak communication.Strategic planning . Two sites invested considerable resources in strategic planning for sustainability. In the health system site, respondents noted that having sustainability in mind from the outset was key to their success. The program was designed so that the care coordinators were housed in-clinic and viewed as part of the health care team. This approach made it easier to make the coordinators’ staff positions permanent eventually and to have a major payer agree to cover a home visit for children with poor asthma control. In contrast, another site encountered so many implementation barriers that it switched its primary clinical partner midway through the funding period, and the demands of implementation precluded a focus on sustainability planning.Political support . Among the challenges to political support for sustainability were local government elections that resulted in the loss of powerful advocates; lack of policy incentives; community economic hardship; organizational disarray and high turnover; lack of a local external group focused on awareness and advocacy for asthma; and the challenge of garnering support for a health program within a school district. A few respondents referred to the lack of buy-in from clinicians who are overwhelmed with responsibilities, lack interest in the program, or both.Respondents described strategies that strengthened political support: demonstrating return on investment to organizational leaders; creating powerful champions by involving organizational leaders from the beginning; and tying the program to the organization’s core mission. For example, the school site emphasized the program’s impact on reducing school absences, and program support was bolstered when the local district attorney called for an end to asthma-related absences. Others built support by providing reimbursable services that generated revenue. Strategies for garnering clinician support were reducing their workloads, aligning incentives, and engaging program champions.Several respondents mentioned characteristics of asthma itself that facilitated political support: its high prevalence and its status as the most common reason for pediatric emergency department visits, significant disparities in asthma outcomes, asthma not stigmatized like sexually transmitted infections, and the relative ease with which positive outcomes can be produced, in contrast to conditions such as obesity.Funding stability . Maintaining funding stability was difficult for all sites. Most respondents cited challenges stemming from the larger economic and political context . Some reported strengthening funding stability by seeking in-kind support and cost sharing, pursuing multiple avenues of support, and being flexible about which program components are supported.As part of a broader evaluation of pediatric asthma care coordination interventions in 4 urban communities, we used the PSAT to evaluate sustainability capacity in 8 domains, adding our own open-ended probes. The relative strengths of the 8 sustainability capacity domain scores were fairly consistent across sites. The domains of program evaluation and program adaptation were rated highly at all sites. The funder required sites to have a robust evaluation component and provided resources for this expense. Similarly, the funder encouraged the sites to adapt the programs to optimize accessibility and impact of their services for hard-to-reach populations. The domains with the lowest scores across sites — funding stability, political support, and strategic planning — were also the most variable. The single item with the lowest average cross-site rating  suggested a lack of diversification in funding streams across settings. (Cross-site item averages are not shown in the Table.) Funding stability is a common challenge described in the literature; Scheirer , for exaScores for the strategic planning domain — as well as overall site scores — were significantly lower in the 2 sites where program implementation was led by academic researchers in partnership with one or more organizations, compared with the 2 sites with programs that were led and housed internally. When lead implementers are internal to an organization, they may be better positioned to garner political and financial support. External implementers may encounter more barriers to strategic planning, and academic researchers in particular may be less able to direct resources toward sustainability. The discrepancies in scores for the item “The program plans for future resource needs” indicate that less planning took place in the 2 programs led by academic researchers  than in the 2 internally led programs .Our findings mirror those from a previous assessment of 252 public health programs  that alsOur qualitative findings highlighted the interconnectedness of domains; these findings, along with those from a review of empirical literature on sustainability , supportFindings demonstrated the usefulness of the PSAT for guiding a mixed-methods evaluation of sustainability capacity. Although developers used open-ended probes during PSAT tool development, the qualitative questions we added were useful for evaluation purposes. Besides providing rich data about sustainability capacity, these questions also gave insight into how close-ended items were interpreted by respondents. This finding provides additional information on the validity of scale items, as called for by the scale developers .First, all data were self-reported by a small number of individuals at each site. Self-report assessments can be inaccurate , and theDespite the requirement by many funders that an implementing organization be able to maintain a successful program, this goal seems elusive in practice. Study findings can help inform community, public health, and health system programs throughout a program’s life cycle. In particular, the area of funding stability warrants special attention, as it continues to be the main barrier to sustainability. From the outset, program developers can deliberately design interventions that are likely to strengthen funding stability and the other domains that are consistently weak across studies. Likewise, funders, purveyors, and practitioners can promote structures and processes that strengthen factors in low-scoring domains, and all can harness the interconnectedness of the domains to leverage strong areas to boost those that are weak. Evaluators and researchers can use the PSAT to assess both program design and the strategies to promote sustainability used during the implementation process to continue to identify links between these and long-term program sustainment.


SWITCH® is an evidence-based childhood obesity prevention program that works through schools to impact parenting practices. The present study was designed as a formative evaluation to test whether an online version of SWITCH® would work equivalently as the established print version.rd grade classrooms were guided through the 4 month program by a team of program leaders working in cooperation with the classroom teachers. Children were tasked with completing weekly SWITCH® Trackers with their parents to monitor goal setting efforts in showing positive Do , View (≤2 hours of screen time), and Chew (≥5 servings of fruits and vegetables) behaviors on each day. A total of 91 parents completed a brief survey to assess project-specific interactions with their child and the impact on their behaviors.Ten elementary schools were matched by socio-economic status and randomly assigned to receive either the print (n = 5) or online (n = 5) version. A total of 211 children from 22, 3ps > .05) suggesting the equivalence in intervention effect for print and online versions of the SWITCH® program.The majority of parents (93.2%) reported satisfactory experiences with either the online or print SWITCH® program. The return rate for the SWITCH® Trackers was higher (42.5% ± 11%) from the print schools compared to the online schools (27.4% ± 10.9%). District program managers rated the level of teacher engagement in regards to program facilitation and the results showed a higher Trackers return rate in the highly engaged schools (38.5% ± 13.3%) than the lowly engaged schools (28.6 ± 11.9%). No significant differences were observed in parent/child interactions or reported behavior change (The findings support the utility of the online SWITCH® platform but school-based modules are needed to facilitate broader school engagement by classroom teachers and PE teachers. Childhood obesity prevention is a priority on the public health agenda ,2. Numerswitch what they do, view and chew” [A promising program that is consistent with these recommendations is the SWITCH® obesity prevention program. SWITCH® is a multi-component, ecologically-based program designed to work through schools to reach families. The strategy capitalizes on the coordinating structure and motivation provided through a school-based program while also minimizing the burden on teachers and schools. Parents are viewed as the primary target of the SWITCH® program since they directly influence children’s current and long-term lifestyle behaviors ; howevernd chew” . Childrend chew” .The SWITCH® program has been previously shown to be effective in promoting children’s health behaviors . The oriExperts in program dissemination emphasize the importance of systematically evaluating underlying assumptions and pre-testing different implementation methods before dissemination . With SWwww.iowaswitch.org). The creation of this online tool made it possible to directly answer the following research question: whether or not the online version of SWITCH® would work equivalently (or better) than the existing print-based version. We used a formative evaluation plan for the study since the broader goal was to determine changes to be incorporated into the more robust web-based system envisioned for the future of the program. While the present study reports on one specific program , the results have broader value for other program planners considering the relative merits of print vs online resources.The purpose of the present study was to evaluate the utility of the online SWITCH program®. To enable this work, we first developed a basic web platform (SWITCH® 1.0) which made all program-related materials available online for download  continuously since the original efficacy-based study back in 2006. The program has been valued in the community but the high cost has presented a barrier to sustainability and dissemination so the schools agreed to participate in a controlled evaluation of the newly developed online version of SWITCH®. Schools were matched by socio-economic status  and then randomly assigned into either the print (n = 5) or online (n = 5) versions of the SWITCH® program. The online version replicated all aspects of the print-based manual and the programming was delivered in a consistent way for both sets of schools. The matched samples and standardization of methods make it possible to directly compare the print vs online formats.rd grade classrooms involved, with total school enrollments ranging from 14 to 38. Parents from the participating schools were asked to complete a post-test survey to evaluate their engagement with and utilization perceptions of the SWITCH® program. Because the data we collected in this study were de-identified, the Iowa State University institutional review board (IRB) deemed the study as “exempt”, therefore, written consent from participants or their parents/guardians were also waived.Parents voluntarily enrolled in the SWITCH® program by either returning a signed enrollment form (print) or by completing a similar online registration form (online). A total of 210 children/parent dyads enrolled in the project . Schools had 2–3, 3Switch what you DO”, week 2 “Switch what you VIEW”, week 3 “Switch what you CHEW”, and week 4 “You Rule! Try all 3 goals!”. The programming in each week is enriched by activities carefully designed to promote parent/child interactions about the behaviors. For example, in a specific “Do” week parents are provided with a set of “SWITCH Activity Cards”. Each card is detailed with activity name, necessary equipment and space, and description of activities that the child can do on their own or with their parent. In this round of formative evaluation, the SWITCH® program was implemented in the same manner for the print and the online conditions. The parents in the print group received booklets for each month of the program and the same resources were released on the website for parents to browse, print, and use.The SWITCH® program operates over a 16 week period with new materials released each week. Programming is divided into 4 modules each following the same sequence of content: week 1 “The key behavior change strategy in SWITCH® is self-monitoring as children are tasked to work with their parents to complete the weekly SWITCH® Tracker sheets. The Trackers prompt children to choose a daily behavior goal , record the actual behavior result , then tally and add up the points earned if goal is attained . Children turn in the Trackers each week to accumulate SWITCH® points which are redeemable for various incentives throughout the program. District program managers hired by the local YMCA made weekly site visits to gather the Trackers and to provide the weekly and monthly incentives for participation.The link between the school and the home is an important component in the SWITCH® program since it keeps parents more involved in the program. The previous efficacy-based studies were able to systematically implement the SWITCH® program with optimal fidelity since research assistants could handle the coordination and communication. However, anecdotal observations of current SWITCH® schools indicate that there is considerable variability in the degree of school engagement. To enable broader dissemination of the program it is important to better understand the impact of school engagement on the program outcomes. While not a planned part of the study, school engagement was included as a moderating variable in the analyses.Epidemiology Assessment” in the PRECEDE Phase challenges the planner to separate out the behavioral and environmental factors that are targeted in the intervention. The subsequent “Educational and Ecological Assessment” then splits these influences into Enabling, Reinforcing and Predisposing Factors. In SWITCH®, parents are viewed as Enabling Factors since they enable, facilitate and promote behavior change in their children. Schools are viewed as Reinforcing factors since the teachers are positioned to remind and reinforce the systematic efforts with the program. The goal of the programming is to facilitate self-monitoring and behavioral skills in children and these are considered the key Predisposing Factors. The final stage of the PRECEDE phase (“Administrative and Policy Assessment”) captures variables thought to be potentially important for the implementation and sustainability of the program. Examples in the present study include school characteristics and school engagement. A second advantage of the model is that it incorporates an evaluation of the process, the impact (i.e. the intervention itself), and the final outcome in the Proceed phase. The structure of this model has been endorsed as an appropriate model for dissemination and implementation research [The evaluation of the SWITCH® program was guided by the established PRECEDE PROCEED Model . There aresearch . DetailsThe primary goal of the program was to promote self-monitoring and goal setting for changing diet, activity and screen time behaviors. Children  were tasked with filling out SWITCH® Trackers each week and bringing them to school for incentives (points redemption). Therefore, the percent of children completing and returning SWITCH® Trackers was viewed as the key process measure. Another key process measure was the overall parent satisfaction with the SWITCH® materials. Parent satisfaction was captured with a single multiple-choice item anchored on a 4-point Likert scale ranging from 1 = “very satisfied” to 4 = “very dissatisfied”.The focus of the SWITCH® programming is to facilitate parent/child interactions about healthy lifestyles. The impact of the programming was assessed with items capturing parent’s report of the quality of interactions related to the three target behaviors . For example, the item for “Do” behavior was stated as: “Did SWITCH help you talk to your child about being physically active?” The answers included 1 = yes, helped a lot”, 2 = “yes, helped somewhat”, 3 = “yes, helped a little”, and 4 = “no, did not help”. The mean of the three items was used to reflect the overall impact of the SWITCH® program on parent/child interactions.Children’s weight management behaviors were measured by 8 items, with 2 items each for the Do and View behaviors, and 4 items for the Chew behavior. Parents were asked to compare their child’s current (upon intervention) behaviors to before participating in SWITCH®. An item for the chew behavior is stated as “Compared to before your family participated in SWITCH®, does your child consume fruits?” The answers ranged from 1 = “a lot less often” to 5 = “a lot more often”. Other Chew behavior items asked about children’s consumption of vegetables, 100% fruit juice, and soft drinks. The means for each of the individual behaviors was used for the analyses.The degree of teacher/school engagement was assessed as a moderating variable using the numerical ratings conducted by two experienced program managers who had worked as SWITCH® staff for over six years. The program managers rated each teacher’s level of engagement from 1 (low) to 3 (high) based on their in-person field observations and interactions with each school. There were a total of 22 teachers  being rated by the program managers. The researchers then averaged the scores for the teachers of the same school and categorized two levels of teachers’ engagement for data analysis: 1 = “low” (≤mean), 2 = “high” (>mean).The focus of the evaluation was on direct comparisons of process, impact and outcome measures between the print and online versions of SWITCH®. For the process evaluation, we compared the rate of completion of the Trackers between print and online using graphic techniques, and between the highly engaged schools and the lowly engaged schools using descriptive analysis (Mean and Standard Deviation). It was not possible to examine these data using statistical methods since the sample sizes for the school-related outcomes were too small. We also descriptively compared the level of parental satisfaction  with the SWITCH® program between the print and online groups. Standard analytic techniques described below were used to examine the Impact and Outcome measures. Frequencies of key measures were first reported to provide an overall sense of parent reactions to the programming. Descriptive statistics (mean and standard deviation) were provided for the primary measures and these were reported for both print and online groups as well as combined. Two-way multivariate analyses of variance (MANOVAs) were used to statistically examine differences in Impact and Outcome measures between the print and online groups and between lowly and highly engaged schools. Homogeneity of variances test was performed prior to the series of inferential statistical analyses.The SWITCH® programming was carried out as planned and the return rates of the SWITCH® Trackers (obtained from the district program coordinators each week) provided valuable process data about the programming. Surveys were completed by 91 parents (43% return rate) and these data were used to capture impact and outcome measures. The majority of respondents on the survey were female (81%) and return rates were slightly higher from the print schools  compared to the online schools . All impact and outcome variables on the survey had low skewness and kurtosis absolute values , indicating no violation of normal distribution. The parent participants reported similar mean scores for these variables between the two SWITCH® versions. Levene’s homogeneity of variances test revealed that all variables had homogeneous variances between groups (p > .05).Overall, parent satisfaction was high with 93.2% of parents reporting being “satisfied” (70.5%) or “very satisfied” (22.7%) with the SWITCH® program. Most parents (68.2%) reported that they spent less than an hour per week on SWITCH® activities. However, more than a third (35.7%) reported that their child spent more than 2 hours per week on the materials. No significant differences were observed in mean parent satisfaction scores between print and online (Cohen’s d = .09). The district program managers rated 6 lowly engaged schools and 4 highly engaged schools. Reports from the SWITCH® trainers revealed large differences in the degree of school engagement across the sites. Variability in school engagement can have a profound effect on outcomes so school engagement was studied more directly as a moderating variable.The key process measure in the study was the return rate of the SWITCH® Trackers. The Trackers serve as the primary behavior change tool in the program since it necessitates direct parent/child interaction and self-monitoring of their lifestyle. The overall return rate was 34.1% ± 13.0% across the 16 weeks of the programs but rates were higher in the print schools 42.5% ± 11%) than the online schools (27.4% ± 10.9%). Involvement in the program diminished across the course of the program for both groups as evidenced by declining return rates over time  or across levels of teacher engagement . The interaction was also not significant . The effect sizes for the impact and outcome measures between print vs online SWITCH® programming were generally low  in talking to their child about being active (68.9%), eating healthy 71.1%) or watching less TV (64.7%). The descriptive results (mean and SD) for these three impact variables are shown in the top portion of Table .1% or waThe present study was designed to formatively evaluate the relative utility of a print-based program of SWITCH® and an identically configured web-based version. Since participants in the print and online conditions received identical treatments and incentives, any differences can be attributed to the form of interventions received during the program. It was hypothesized that the online version would work similarly to the more expensive and less-sustainable print version. The comparison between the two versions of SWITCH® program was made by examining differences in the designated process, impact, and outcome measures.The process evaluation revealed differences in the absolute numbers of SWITCH® trackers returned between the conditions. This makes sense considering that the print version allowed parents to simply tear out the pre-printed Tracker and fill it out. In the online version, parents had to print materials themselves before engaging with the program. The results confirmed a plausible and somewhat expected weakness of the online materials; however, a closer examination of the data revealed that school engagement explained some of the differences in both school enrollment and return rates. Specifically, schools rated as highly engaged had much higher class enrollments and much higher Tracker return rates than schools rated as lowly engaged by the district coordinators. This explains at least some of the observed differences since engagement and class sizes were higher in the print versus online SWITCH® schools. The results suggest that teacher/school engagement may have a moderating influence on parent recruitment and participation in the SWITCH® program. However, it is also possible that the Tracker rates and engagement were higher in schools that had higher enrollments. For example, if there are more kids participating, teachers may pay more attention to the program and this would, in turn create more interest in the students. Thus, it is not completely clear whether school engagement is a cause or a consequence of child involvement. Regardless of the mechanism, it is clear that building school and teacher support is an important consideration for effective implementation of the SWITCH® program. This result shows the importance of coordinating school personnel and parents and creating an ecological environment for children to enhance self-monitoring with their health-related behaviors .It is noteworthy, and somewhat expected, that weekly Tracker return rates declined trend over time since it is typical for participants to lose interest in programming over time. This was evident in both conditions, but suggests the need for creative strategies to maintain parent/child interest in SWITCH®. The trend was noted during implementation and the research team worked with the district coordinators to pilot a peer incentive system in the last month that would provide double incentives if peer-selected partners both returned their Trackers. This strategy is consistent with behavioral economics theory and previous research suggesting that peer incentives may help hold youth accountable -23. We nWhile we observed some differences in Tracker returns between conditions (and declines over time) there were no significant differences in the key Impact or Outcome variables. This indicates no difference in parent-reported SWITCH®-related interactions with their child and no differences in the observed behavior changes. These are important results since they demonstrate that the web-based platform is as effective as the more expensive print-based materials in promoting parent/child engagement. It is especially noteworthy considering that differences were evident with the return of Trackers. This suggests that the online materials may have had other subtle benefits on parent/child interactions not captured by the Trackers. With easier access to the internet in the modern era, health-enhancing interventions have started to move to the virtual world -29. Web-The formative evaluation supports the utility of the web-based platform but it is clear that the current version of the website does not provide considerable value. This is because parents still had to print copies of the Trackers as well as other SWITCH® materials and submit them manually. It was important to test this basic system first to show that it at least produced similar or equivalent outcomes. Nevertheless, web-based programs are supposed to provide a higher level of interactivity than the print materials to actively engage the users . The onlAn important observation in the present study is that the degree of school/teacher engagement had an important impact on enrollment and involvement in the program. This is a common and somewhat expected finding in effectiveness research  and addiDespite the encouraging findings, this formative evaluation was limited by the small sample and the lack of pre-test measures. While most of the parents-children dyads contributed data to the earlier efficacy study , only abIn summary, the formative evaluation demonstrates the viability of the online SWITCH® program as a more cost-effective method for dissemination. The online version yielded similar results for both the impact and outcome measures. Thus, parents and children were able to interact with the program in a similar fashion, regardless of the versions received. The differences in Tracker rates may not be relevant since we ultimately envision a system in which the participants would log SWITCH® trackers directly on the website and see accumulated points over time. The school coordinator could also view results for individuals or classes without having to laboriously track the printed sheets. Thus, the reported differences in Trackers returns may not be a major concern, when the new web platform is developed.The formative evaluation study shows that the online SWITCH® program holds great potential as an efficacious and sustainable childhood prevention program. The findings will guide the development of the more robust SWITCH® platform as well as the additional school-based modules . It is expected that the refined SWITCH® content management system will be able to more effectively impact children’s adoption of healthy lifestyles in school. The modularized system will facilitate integration of classroom, PE and lunchroom components and enable school wellness teams to carry out programming on their own. The SWITCH program provides a model consistent with CDC recommendations for Comprehensive School Physical Activity Programs (CSPAP).