Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project whilst also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation.
Manual software testing is associated with problems in practice such as high cost and tediousness and error-proneness @cite_16 @cite_56 @cite_36 @cite_38 @cite_34 @cite_31 . Despite these problems, manual testing is still extensively used for system and acceptance testing in industrial practice. One reason is because state-of-practice test automation techniques primarily perform testing on lower levels of system abstraction, e.g. unit testing with JUnit @cite_15 . Attempts to apply the low level techniques for high level testing, e.g. system and acceptance tests, have resulted in complex test cases that are costly and difficult to maintain, presenting a need for high level test automation techniques @cite_6 @cite_44 @cite_55 @cite_56 @cite_2 @cite_26 @cite_45 @cite_36 . Another reason for the lack of automation is presented in research as the inability to automate all testing @cite_6 @cite_14 @cite_46 @cite_47 @cite_37 . This inability comes from the inability of scripted test cases to identify defects that are not explicitly asserted, which infers a need for, at least some level of, manual or exploratory testing @cite_23 .
@cite_38 @cite_47 @cite_26 @cite_14 @cite_37 @cite_15 @cite_36 @cite_46 @cite_55 @cite_34 @cite_56 @cite_6 @cite_44 @cite_45 @cite_23 @cite_2 @cite_31 @cite_16
1972478306 2483314354 2114647523 1481784587 2154516928
This report addresses some of our observations made in a dozen of projects in the area of software testing, and more specifically, in automated testing. It documents, analyzes and consolidates what we consider to be of interest to the community. The major findings can be summarized in a number of lessons learned, covering test strategy, testability, daily integration, and best practices. The report starts with a brief description of five sample projects. Then, we discuss our observations and experiences and illustrate them with the sample projects. The report concludes with a synopsis of these experiences and with suggestions for future test automation endeavors. Writing unit test code is labor-intensive, hence it is often not done as an integral part of programming. However, unit testing is a practical approach to increasing the correctness and quality of software; for example, the Extreme Programming approach relies on frequent unit testing. In this paper we present a new approach that makes writing unit tests easier. It uses a formal specification language's runtime assertion checker to decide whether methods are working correctly, thus automating the writing of unit test oracles. These oracles can be easily combined with hand-written test data. Instead of writing testing code, the programmer writes formal specifications (e.g., pre- and postconditions). This makes the programmer's task easier, because specifications are more concise and abstract than the equivalent test code, and hence more readable and maintainable. Furthermore, by using specifications in testing, specification errors are quickly discovered, so the specifications are more likely to provide useful documentation and inputs to other tools. We have implemented this idea using the Java Modeling Language (JML) and the JUnit testing framework, but the approach could be easily implemented with other combinations of formal specification languages and unit test tools. There is a documented gap between academic and practitioner views on software testing. This paper tries to close the gap by investigating both views regarding the benefits and limits of test automation. The academic views are studied with a systematic literature review while the practitioners views are assessed with a survey, where we received responses from 115 software professionals. The results of the systematic literature review show that the source of evidence regarding benefits and limitations is quite shallow as only 25 papers provide the evidence. Furthermore, it was found that benefits often originated from stronger sources of evidence (experiments and case studies), while limitations often originated from experience reports. We believe that this is caused by publication bias of positive results. The survey showed that benefits of test automation were related to test reusability, repeatability, test coverage and effort saved in test executions. The limitations were high initial invests in automation setup, tool selection and training. Additionally, 45 of the respondents agreed that available tools in the market offer a poor fit for their needs. Finally, it was found that 80 of the practitioners disagreed with the vision that automated testing would fully replace manual testing. Exploratory testing (ET) - simultaneous learning, test design, and test execution - is an applied practice in industry but lacks research. We present the current knowledge of ET based on existing literature and interviews with seven practitioners in three companies. Our interview data shows that the main reasons for using ET in the companies were the difficulties in designing test cases for complicated functionality and the need for testing from the end user's viewpoint. The perceived benefits of ET include the versatility of testing and the ability to quickly form an overall picture of system quality. We found some support for the claimed high defect detection efficiency of ET. The biggest shortcoming of ET was managing test coverage. Further quantitative research on the efficiency and effectiveness of ET is needed. To help focus ET efforts and help control test coverage, we must study planning, controlling and tracking ET. Since manual black-box testing of GUI-based APplications (GAPs) is tedious and laborious, test engineers create test scripts to automate the testing process. These test scripts interact with GAPs by performing actions on their GUI objects. An extra effort that test engineers put in writing test scripts is paid off when these scripts are run repeatedly. Unfortunately, releasing new versions of GAPs with modified GUIs breaks their corresponding test scripts thereby obliterating benefits of test automation. We offer a novel approach for maintaining and evolving test scripts so that they can test new versions of their respective GAPs. We built a tool to implement our approach, and we conducted a case study with forty five professional programmers and test engineers to evaluate this tool. The results show with strong statistical significance that users find more failures and report fewer false positives (p ≪ 0.02) in test scripts with our tool than with a flagship industry product and a baseline manual approach. Our tool is lightweight and it takes less than eight seconds to analyze approximately 1KLOC of test scripts.