A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Causes of test case breakages (RQ 2) <s> Software engineers often use record/replay tools to enable the automated testing of web applications. Tests created in this manner can then be used to regression test new versions of the web applications as they evolve. Web application tests recorded by record/replay tools, however, can be quite brittle, they can easily break as applications change. For this reason, researchers have begun to seek approaches for automatically repairing record/replay tests. To date, however, there have been no comprehensive attempts to characterize the causes of breakagesin record/replay tests for web applications. In this work, wepresent a taxonomy classifying the ways in which record/replay tests for web applications break, based on an analysis of 453 versions of popular web applications for which 1065 individual test breakages were recognized. The resulting taxonomy can help direct researchers in their attempts to repair such tests. It can also help practitioners by suggesting best practices when creating tests or modifying programs, and can help researchers with other tasks such as test robustness analysis and IDE design. <s> BIB001 </s> A Systematic Literature Review of Test Breakage Prevention and Repair Techniques <s> Causes of test case breakages (RQ 2) <s> Abstract The importance of test automation in web engineering comes from the widespread use of web applications and the associated demand for code quality. Test automation is considered crucial for delivering the quality levels expected by users, since it can save a lot of time in testing and it helps developers to release web applications with fewer defects. The main advantage of test automation comes from fast, unattended execution of a set of tests after some changes have been made to a web application. Moreover, modern web applications adopt a multitier architecture where the implementation is scattered across different layers and run on different machines. For this reason, end-to-end testing techniques are required to test the overall behavior of web applications. In the last years, several approaches have been proposed for automated end-to-end web testing and the choice among them depends on a number of factors, including the tools used for web testing and the costs associated with their adoption. They can be classified using two main criteria: the first concerns how test cases are developed (ie, Capture-Replay and Programmable approaches), while, the second concerns how test cases localize the web elements to interact with (ie, Coordinates-based, DOM-based, and Visual approaches), that is what kind of locators are used for selecting the target GUI components. For developers and project managers it is not easy to select the most suitable automated end-to-end web testing approach for their needs among the existing ones. This chapter provides a comprehensive overview of the automated end-to-end web testing approaches and summarizes the findings of a long term research project aimed at empirically investigating their strengths and weaknesses. <s> BIB002
Software systems undergo several changes during their evolution. Unfortunately, such changes might affect the corresponding test cases. Some studies are available in the literature which has classified the causes of test breakages. For example, BIB001 provides a taxonomy of the causes of record and reply test breakages for evolving web application. Record and Replay tools record the interaction with the web browser while performing specific tasks. However, they are vulnerable to changes and will break during the test execution BIB002 . We extract the causes of test case breakages and collect them into a single taxonomy. Our taxonomy subsumes the taxonomy of causes of test case breakages for web applications presented in BIB001 and covers both desktop applications and mobile applications based on the data extracted from 41 included studies. Such taxonomies help researchers to guide their test repair techniques for repairing maximum causes of breakages. It also helps to evaluate the maturity of approaches and clarifying key issues in the area. Without knowing the causes of broken test cases, it would not be possible to propose new approaches to repair them. Figure 9 shows the common causes of test breakages in all domains. The most common types of changes are the addition, deletion or modification of elements. We summarize all identified causes of test breakages from the existing literature and re-classify them into coarse-grained classes on the basis of similarities among the causes, as shown in Table  6 , Table 7 and Table 8 . In order to integrate and classify the existing causes of test case breakages, two of the authors of this paper studied the presented causes in selected studies (where applicable) and labeled each cause of test breakage for creating the taxonomy independently. Subsequently, these labels were then refined through multiple review and group meetings of all authors for organizing them into hierarchies. For example, S8 is focused towards repairing the changes related to the method signature, class hierarchies and addition or deletion of overridden/overloaded methods. The targeted changes from each study were identified and grouped in some high-level classes with the consensus of all authors of the study. Table 6 presents the type of code changes which can break their corresponding test cases. S6 is the only study which provides test repairs for almost all of the changes mentioned in the Table. S23 and S24 mostly deal with breakages related to method-level changes. We can conclude that "Method-Level Changes" (such as changes in the declaration of method parameters and return values, insertion and removal and type changes) are the prominent causes of test breakages for desktop applications. We did not find any work that focusses specifically code level changes for mobile and web applications. However, due to the nature of the approaches, it can be inferred that mobile and web applications will also share the same causes. Therefore, the approaches that fix and repair test breakage based on such changes should also be applicable to mobile and web applications. Table 7 shows the causes of test breakages for testing web applications. Most of the techniques provide repairs for the broken HTML locators (such as id, name and XPath) and it also shows that web locators are a prominent cause of web test breakages. Changes related to pop-up boxes, page reloading and session expiry are neglected by the web test repair techniques. Table 8 shows the causes of test breakages due to GUI evolution of software systems. A number of test repair techniques provide fixes for the structural GUI evolution such as repositioning of GUI elements, enable or disable buttons, and other GUI layout changes. We found some instances of overlap between the causes of test breakages in this category between desktop, mobile and web applications. For example, the changes classified under event-related changes are common for all three platforms. Similarly, repositioning of graphical elements is also common to all three platforms. We did not find any works that address repairing of test cases broken as a result of session related changes (for example, user inactivity time increased or decreased), changes to Java scripts pop-ups, etc., despite being common in web applications. As a consequence of classifying the reported causes in higher level classes, we hope to let researchers and practitioners infer whether a given technique may be applied on a SUT from a domain for which it was not originally intended. For example, S9 uses a model-based approach to repair broken GUI test scripts of mobile applications. This approach constructs an event sequence model (ESM) to abstract possible event sequences in a GUI and a delta ESM (DESM) to abstract the changes made to a GUI. By using delta DESM, it automatically updates test cases for the updated version. The use of modeling methodology makes it generalizable to be applied to other GUI-based desktop applications. VISTA (S37) uses computer vision-based approach to suggest repairs for broken capture and replay test scripts. As capture and replay test scripts share common characteristics like <locator, value, action>, this technique can also be used to repair other automated test scripts such as Selenium and QTP. Similarly, code level techniques such as those proposed by S4, S5, S6, S23 and S24 can also be applied mobile applications, web applications as well as desktop applications, even though the papers themselves do not provide any such application evidence. 