A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Approach Strengths and Weaknesses <s> Metaheuristic search techniques have been extensively used to automate the process of generating test cases, and thus providing solutions for a more cost-effective testing process. This approach to test automation, often coined “Search-based Software Testing” (SBST), has been used for a wide variety of test case generation purposes. Since SBST techniques are heuristic by nature, they must be empirically investigated in terms of how costly and effective they are at reaching their test objectives and whether they scale up to realistic development artifacts. However, approaches to empirically study SBST techniques have shown wide variation in the literature. This paper presents the results of a systematic, comprehensive review that aims at characterizing how empirical studies have been designed to investigate SBST cost-effectiveness and what empirical evidence is available in the literature regarding SBST cost-effectiveness and scalability. We also provide a framework that drives the data collection process of this systematic review and can be the starting point of guidelines on how SBST techniques can be empirically assessed. The intent is to aid future researchers doing empirical studies in SBST by providing an unbiased view of the body of empirical evidence and by guiding them in performing well-designed and executed empirical studies. <s> BIB001 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Approach Strengths and Weaknesses <s> This paper presents some initial results from a twelve-month empirical research study of model driven engineering (MDE). Using largely qualitative questionnaire and interview methods we investigate and document a range of technical, organizational and social factors that apparently influence organizational responses to MDE: specifically, its perception as a successful or unsuccessful organizational intervention. We then outline a range of lessons learned. Whilst, as with all qualitative research, these lessons should be interpreted with care, they should also be seen as providing a greater understanding of MDE practice in industry, as well as shedding light on the varied, and occasionally surprising, social, technical and organizational factors that affect success and failure. We conclude by suggesting how the next phase of the research will attempt to investigate some of these issues from a different angle and in greater depth. <s> BIB002 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Approach Strengths and Weaknesses <s> JUnit test repair has been proposed as a way to alleviate the burden of maintaining the broken tests caused by evolving software. Existing techniques for JUnit test repair either focus on repairing failing assertions or fixing test case compilation errors rendered by evolving method declarations. The empirical work suggests that the synthesis of new method calls is often needed when repairing test cases in practice. In this work, we propose Test Fix, an approach to fix broken JUnit test cases by synthesizing new method calls. Test Fix reduces the synthesis to a search problem and uses a genetic algorithm to solve it. Evaluated on real world applications, preliminary experimental results show that Test Fix can repair broken tests with adding or deleting method calls. Evaluations on the performance of JUnit test repair technique using a genetic algorithm against a random search algorithm are also conducted. Experimental results indicate the clear superiority of genetic algorithms over random search algorithm. <s> BIB003 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Approach Strengths and Weaknesses <s> System testing of a GUI-based application requires that test cases, consisting of sequences of user actions/events, be executed and the software's output be verified. To enable automated re-testing, such test cases are increasingly being coded as low-level test scripts, to be replayed automatically using test harnesses. Whenever the GUI changes—widgets get moved around, windows get merged—some scripts become unusable because they no longer encode valid input sequences. Moreover, because the software's output may have changed, their test oracles—assertions and checkpoints—encoded in the scripts may no longer correctly check the intended GUI objects. We present ScrIpT repAireR (SITAR), a technique to automatically repair unusable low-level test scripts. SITAR uses reverse engineering techniques to create an abstract test for each script, maps it to an annotated event-flow graph (EFG), uses repairing transformations and human input to repair the test, and synthesizes a new “repaired” test script. During this process, SITAR also repairs the reference to the GUI objects used in the checkpoints yielding a final test script that can be executed automatically to validate the revised software. SITAR amortizes the cost of human intervention across multiple scripts by accumulating the human knowledge as annotations on the EFG. An experiment using QTP test scripts suggests that SITAR is effective in that 41-89 percent unusable test scripts were repaired. Annotations significantly reduced human cost after 20 percent test scripts had been repaired. <s> BIB004
Model-based approach + Ensure generalizability and provide tool-independent solutions BIB004 . -Requires expertise to design models BIB002 . Search-based approach + Provide the most optimized solution to the problem BIB003 . -Computationally expensive BIB001 .