A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Introduction <s> Developers often change software in ways that cause tests to fail. When this occurs, developers must determine whether failures are caused by errors in the code under test or in the test code itself. In the latter case, developers must repair failing tests or remove them from the test suite. Repairing tests is time consuming but beneficial, since removing tests reduces a test suite's ability to detect regressions. Fortunately, simple program transformations can repair many failing tests automatically. We present ReAssert, a novel technique and tool that suggests repairs to failing tests' code which cause the tests to pass. Examples include replacing literal values in tests, changing assertion methods, or replacing one assertion with several. If the developer chooses to apply the repairs, ReAssert modifies the code automatically. Our experiments show that ReAssert can repair many common test failures and that its suggested repairs correspond to developers' expectations. <s> BIB001 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Introduction <s> Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference. Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence. Method: We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings. Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4. Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners. <s> BIB002 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Introduction <s> Web applications tend to evolve quickly, resulting in errors and failures in test automation scripts that exercise them. Repairing such scripts to work on the updated application is essential for maintaining the quality of the test suite. Updating such scripts manually is a time consuming task, which is often difficult and is prone to errors if not performed carefully. In this paper, we propose a technique to automatically suggest repairs for such web application test scripts. Our technique is based on differential testing and compares the behavior of the test case on two successive versions of the web application: first version in which the test script runs successfully and the second version in which the script results in an error or failure. By analyzing the difference between these two executions, our technique suggests repairs that can be applied to repair the scripts. To evaluate our technique, we implemented it in a tool called WATER and exercised it on real web applications with test cases. Our experiments show that WATER can suggest meaningful repairs for practical test cases, many of which correspond to those made later by developers themselves. <s> BIB003 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Introduction <s> Script-based automated regression testing is widely used in industry. In this work, we focus on failed tests in a real regression test project. The causes of 197 failed tests produced in automated testing are examined and categorized based on an analysis procedure presented. The result shows that incorrect scripts, oracle mismatches, test tool bugs and misconfigurations involved in testing contribute most to failures instead of product bugs. Detecting and fixing the false positives are laborious and time-consuming. Our empirical study demonstrates that the benefits of test automation are obliterated to some extent in practical settings. To improve the effectiveness of automated regression testing further, some issues should receive more attention. <s> BIB004 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Introduction <s> System testing of a GUI-based application requires that test cases, consisting of sequences of user actions/events, be executed and the software's output be verified. To enable automated re-testing, such test cases are increasingly being coded as low-level test scripts, to be replayed automatically using test harnesses. Whenever the GUI changes—widgets get moved around, windows get merged—some scripts become unusable because they no longer encode valid input sequences. Moreover, because the software's output may have changed, their test oracles—assertions and checkpoints—encoded in the scripts may no longer correctly check the intended GUI objects. We present ScrIpT repAireR (SITAR), a technique to automatically repair unusable low-level test scripts. SITAR uses reverse engineering techniques to create an abstract test for each script, maps it to an annotated event-flow graph (EFG), uses repairing transformations and human input to repair the test, and synthesizes a new “repaired” test script. During this process, SITAR also repairs the reference to the GUI objects used in the checkpoints yielding a final test script that can be executed automatically to validate the revised software. SITAR amortizes the cost of human intervention across multiple scripts by accumulating the human knowledge as annotations on the EFG. An experiment using QTP test scripts suggests that SITAR is effective in that 41-89 percent unusable test scripts were repaired. Annotations significantly reduced human cost after 20 percent test scripts had been repaired. <s> BIB005
Testing is an important activity to assure the quality of software systems. Modern software development practices like DevOps, continuous integration and development encourage automated execution of test cases by requiring test engineers to develop test scripts. This leads to several advantages such as automated test execution, test effort reduction, efficient usage of resources and an easy to use regression test suite. Availability of test scripts that can be executed automatically improves the efficiency of testing, but can lead to significant challenges in the maintenance of the test scripts. As the system under test (SUT) evolves, a number of test scripts can fail due to changes in the SUT. Therefore, the test suite needs to evolve along with the SUT. For regression testing to be effective, the test suite must be updated to keep pace with the changes in SUT. Each introduced change may lead to failed test cases. Some tests fail due to the presence of faults or bugs in the application code. However, some tests may stop prematurely due to modifications in the application code such as repositioning or renaming of existing elements, locator and layout changes, etc. Such premature stopping of test cases due to changes in the SUT is referred to as test breakage. The broken test cases cannot be executed on the updated SUT without fixing the implementation of the test cases (the test scripts). The existing literature classifies the regression test suite as usable, unusable and obsolete test scripts [1] . The usable test cases conform to the existing functionality because they are not affected by the changes made in the evolved (modified) version of SUT. The unusable/broken test cases contain at least one statement that cannot be executed successfully. Such un-executable statement(s) may break the whole test case but the test case can be 'fixed' by applying repairing transformations to the test case implementation. Obsolete test cases fail to execute on the updated version and are not repairable, for example, they correspond to functionality that has been removed from the SUT. The changes that break test cases can be structural or logical  . Structural changes deal with the layout and structure of the application whereas the logical changes deal with modification in business logic or functionality. Discarding broken test cases after modifications highly affect the quality of the regression test suite. This reduces the size of the test suite and requires significant effort in re-writing and rerecording test scripts from scratch. Even small modifications can lead to a large number of broken test cases, in some cases up to 74% of the test suite  . Discarding the broken test cases therefore leads to a significant increase in the cost of testing and may reduce the quality of the test suite. Therefore, repairing broken test scripts is an important task BIB004 . Over the past decade, researchers and practitioners have proposed different techniques for automated repairing of broken test scripts of evolving software systems  BIB001 BIB005 BIB003 . Broadly, the test script repair approaches perform three essential steps, (i) examine and classify difference between the original and modified versions of the evolving SUT, (ii) identify the broken test scripts, (iii) repair the broken test scripts using repairing transformations. In this paper, we systematically identify, summarize and evaluate the existing literature to find gaps in the area and to position new research activities. We present a Systematic Literature Review (SLR) BIB002 in which we review 41 papers on test breakage prevention and automated repairing of test scripts. SLRs are used to investigate, categorize, and evaluate the existing literature in a particular research area by applying well-defined inclusion and exclusion techniques. The contribution of this study is twofold. First, it helps new researchers in a structured understanding of the area by indexing the existing studies and by providing new research directions. Second, it helps practitioners to understand state-of-the-art tools, techniques and their appropriate usage. More specifically, we provide the following contributions in the area of test script repair: • We identify the test case repair approaches presented in the literature and classify the studies in terms of the type of contribution made, kind of approaches, and testing frameworks used for writing and recording test scripts. • We classify the causes behind test case breakages that are presented in the literature and provide a taxonomy of commonly identified changes that can result in test breakage. • We report on the evaluation of test breakage prevention and test script repair techniques. We document any identified empirical evaluations, benchmark case studies and the widely used metrics to evaluate the quality of the proposed techniques. • We provide implications of existing test repair techniques for practitioners based on the available evidence on the application of test repair techniques and tools. • Finally, we offer new directions for future research by identifying the gaps in the area. The rest of the paper is structured as follows. Section 2 presents the research methodology and the research questions used in this study. Section 3 discusses the answers to our research questions and the results of the review. Section 4 presents a discussion on our findings and take away for researchers and practitioners. Section 5 presents the related work and section 6 discusses different threats to validity. Finally, Section 7 concludes the paper with a discussion of potential future directions.