A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Describes ATAC (Automatic Test Analysis for C), a tool for data flow coverage testing of C programs. ATAC is being used as a research instrument at Purdue and Bellcore and as a software development tool at Bellcore. The authors discuss the design of ATAC, a preliminary view of its uses in development, and its research uses. > <s> BIB001 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Object-oriented unit tests consist of sequences of method invocations. Behavior of an invocation depends on the state of the receiver object and method arguments at the beginning of the invocation. Existing tools for automatic generation of object-oriented test suites, such as Jtest and J Crasher for Java, typically ignore this state and thus generate redundant tests that exercise the same method behavior, which increases the testing time without increasing the ability to detect faults. This work proposes Rostra, a framework for detecting redundant unit tests, and presents five fully automatic techniques within this framework. We use Rostra to assess and minimize test suites generated by test-generation tools. We also present how Rostra can be added to these tools to avoid generation of redundant tests. We have implemented the five Rostra techniques and evaluated them on 11 subjects taken from a variety of sources. The experimental results show that Jtest and JCrasher generate a high percentage of redundant tests and that Rostra can remove these redundant tests without decreasing the quality of test suites. <s> BIB002 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Aspect-oriented software development is gaining popularity with the adoption of languages such as AspectJ. Testing is an important part in any software development, including aspect-oriented development. To automate generation of unit tests for AspectJ programs, we can apply the existing tools that automate generation of unit tests for Java programs. However, these tools can generate a large number of test inputs, and manually inspecting the behavior of the software on all these inputs is time consuming. We propose Raspect, a framework for detecting redundant unit tests for AspectJ programs. We introduce three levels of units in AspectJ programs: advised methods, advice, and intertype methods. We show how to detect at each level redundant test that does not exercise new behavior. Our approach selects only non-redundant tests from the automatically generated test suites, thus allowing the developer to spend less time in inspecting this reduced set of tests. We have implemented Raspect and applied it on 12 subjects taken from a variety of sources; our experience shows that Raspect can effectively reduce the size of generated test suites for inspecting AspectJ programs <s> BIB003 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> There are several problem areas that must be addressed when applying randomization to unit testing. As yet no general, fully automated solution that works for all units has been proposed. We therefore have developed RUTE-J, a Java package intended to help programmers do randomized unit testing in Java. In this paper, we describe RUTE-J and illustrate how it supports the development of per-unit solutions for the problems of randomized unit testing. We report on an experiment in which we applied RUTE-J to the standard Java TreeMap class, measuring the efficiency and effectiveness of the technique. We also illustrate the use of randomized testing in experimentation, by adapting RUTE-J so that it generates randomized minimal covering test suites, and measuring the effectiveness of the test suites generated. <s> BIB004 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> R<scp>ANDOOP</scp> for Java generates unit tests for Java code using feedback-directed random test generation. Below we describe R<scp>ANDOOP</scp>'s input, output, and test generation algorithm. We also give an overview of RANDOOP's annotation-based interface for specifying configuration parameters that affect R<scp>ANDOOP</scp>'s behavior and output. <s> BIB005 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Reducing test suites is crucial for reducing the test cost, for the test cost is directly proportional to the size of test suites. Reducing the test cost is particularly important when we need to urgently develop a stable compiler for a newly developed processor. In this paper, we propose a framework to reduce test suites for retargeted compilers using intermediate representations of test input programs. We describe that redundancy of test cases at the level of source codes exists from the viewpoint of intermediate representation and, thus, testing efficiency can be improved by excluding those redundant test cases. We also conduct a case study with RTL where we found that test suite can be considerably reduced. In addition, we show that the fault detection capability can be maintained with the reduced test suites by means of mutation testing. <s> BIB006 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Combinatorial testing consists in generating (possibly large) test suites by combining both the sequencing of several operations, and the selection of test data. The TOBIAS tool is based on this generation technique. The combinatorial part of the approach makes both its strength and its weakness. Indeed, the more tests are produced, the more confidence the user may have in his/her program. Nevertheless, simple patterns may result in millions of test cases due to the combinatorial explosion problem, leading to intractable testsuites. To overcome this weakness, TOBIAS makes it possible to connect the generator to "selectors" that choose a subset of the test suite based on various techniques or criteria. This paper presents such a test suite reduction technique, based on a stochastic approach. The selection is improved so a uniform repartition of test cases is performed w.r.t. a given criterion, namely, an abstraction of the test cases based on their "shape". The approach is implemented into TOBIAS as a generic selector, independent of the system under test. It has been put into practice on a smart card case study, for which the results are reported in this paper. <s> BIB007 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> The continuous use of the Web for daily operations by businesses, consumers, and the government has created a great demand for reliable Web applications. One promising approach to testing the functionality of Web applications leverages the user-session data collected by Web servers. User-session-based testing automatically generates test cases based on real user profiles. The key contribution of this paper is the application of concept analysis for clustering user sessions and a set of heuristics for test case selection. Existing incremental concept analysis algorithms are exploited to avoid collecting and maintaining large user-session data sets and to thus provide scalability. We have completely automated the process from user session collection and test suite reduction through test case replay. Our incremental test suite update algorithm, coupled with our experimental study, indicates that concept analysis provides a promising means for incrementally updating reduced test suites in response to newly captured user sessions with little loss in fault detection capability and program coverage. <s> BIB008 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Test-suite minimization techniques aim to eliminate redundant test cases from a test-suite based on some criteria, such as coverage or fault-detection capability. Most existing test-suite minimization techniques have two main limitations: they perform minimization based on a single criterion and produce suboptimal solutions. In this paper, we propose a test-suite minimization framework that overcomes these limitations by allowing testers to (1) easily encode a wide spectrum of test-suite minimization problems, (2) handle problems that involve any number of criteria, and (3) compute optimal solutions by leveraging modern integer linear programming solvers. We implemented our framework in a tool, called MINTS, that is freely-available and can be interfaced with a number of different state-of-the-art solvers. Our empirical evaluation shows that MINTS can be used to instantiate a number of different test-suite minimization problems and efficiently find an optimal solution for such problems using different solvers. <s> BIB009 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Test case management may make the testing process more efficient and thus accelerate software delivery. With the popularity of using JUnit for testing Java software, researchers have paid attention to techniques to manage JUnit test cases in regression testing of Java software. Typically, most existing test case management tools are based on the coverage information. However, coverage information may need extra efforts to obtain. In this paper, we present an Eclipse IDE plug-in (named Jtop) for managing JUnit test cases in absence of coverage information. Jtop statically analyzes the program under test and its corresponding JUnit test cases to perform the following management tasks: regression test case selection, test suite reduction and test case prioritization. Furthermore, Jtop also enables the programmer to manually manipulate test cases through a graphical user interface. <s> BIB010 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Test-suite reduction technique aims to find a subset of the test suite while still satisfying the original test requirements; therefore, it can save the cost of software testing. Because many test cases have been removed, the testing information is also lost. Fault localization is a technique using testing information to locate the fault and widely used by programmers to debug programs, so it suffers from the side effects of test-suite reduction. How to reduce the test-suite size in software testing with the premise of retaining or improving fault-localization effectiveness has become the hot spot in the area of software debugging recently. In this paper, we propose a new approach that selectively keeps the limited redundant test cases in the reduced set; it makes the new reduced set relatively redundant compared to the original one, and we expect that it retains or even improves fault-localization effectiveness. We also describe a framework that implements our approach and conduct a set of empirical studies for evaluation. The results show that our approach can retain or even improve fault-localization effectiveness as expected. <s> BIB011 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Abstract: With widespread use of embedded processors, there is an increasing need to develop compilers for them in a timely manner. Retargeting has been an effective approach to constructing new compilers by modifying the back-end of existing compilers. An efficient testing method for retargeted compilers using intermediate codes is proposed in this paper. The concepts of the intermediate-code-based approach are described, and it is demonstrated that the proposed approach is efficient and effective enough. We have developed an automated tool, PLOOSE, which generates test suites based on source-code coverage criteria and then reduces them based on intermediate-code coverage. In addition, case studies are presented which reveal that a test suite based on source codes can be considerably reduced using intermediate codes. Moreover, by mutation analysis, it was found that the size of the test suite has been reduced by over 90% on average, but the fault detection capability of the reduced test suites is still approximately 80% of that of the original test suites. In particular, the proposed reduction method can be useful for testing compilers in the early stage of the development of retargeted compilers. <s> BIB012 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> A program fails. What now? Taking a single failing run, we record and minimize the interaction between objects to the set of calls relevant for the failure. The result is a minimal unit test that faithfully reproduces the failure at will: "Out of these 14,628 calls, only 2 are required". In a study of 17 real-life bugs, our JINSI prototype reduced the search space to 13.7% of the dynamic slice or 0.22% of the source code, with only 1--12 calls left to examine. <s> BIB013 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> When software developers make changes to a program, it is possible that they will introduce faults into previously working parts of the code. As software grows, a regression test suite is run to ensure that the old functionality still works as expected. Yet, as the number of test cases increases, it becomes more expensive to execute the test suite. Reduction and prioritization techniques enable developers to manage large and unwieldy test suites. However, practitioners and researchers do not always use and study these methods due, in part, to a lack of availability. In response to this issue, this paper describes an already released open-source framework that supports both research and practice in regression testing. The sharing of this framework will enable the replication of empirical studies in regression testing and encourage faster industrial adoption of these useful, yet rarely used, techniques. <s> BIB014 </s> A survey on Test Suite Reduction frameworks and tools <s> Thematic Taxonomy based Comparison of Test Suite Reduction (TSR) Tools <s> Post-deployment in situ testing and validation techniques have become an important means of ensuring the reliability of mobile and embedded systems. However, these techniques do not take into consideration the amount of energy they consume, which is an issue of paramount concern for systems with limited energy budgets. In this paper we propose a new test suite minimization approach that allows developers to generate energy-efficient, minimized test suites. The approach is based on encoding minimization problems as integer linear programming problems. Our empirical evaluation shows that, compared to traditionally generated minimized test suites, the test suites generated by our approach are equally effective in terms of their test coverage, but can realize energy savings of up to 90 %. <s> BIB015
This section compares existing Test Suite Reduction (TSR) tools based on the devised thematic taxonomy (presented in Section 2) to highlight the commonalities and differences among the reported tools (depicted in Table 1 and 2). We critically analyzed current TSR tools based on several parameters: (i) approach type, (ii) testing paradigm, (iii) optimization type, (iv) coverage source, (v) execution platform, (vi) computational mode, vii) license type, (viii) evaluation, (ix) customizability, and (x) support. Current state-of-the-art TSR tools can be classified into coverage-based, search-based, similarity-based, and Integer Linear Programming (ILP) based approaches. Majority of the existing tools (Horgan and London, 1992 BIB002 BIB003 BIB004 BIB005 , Jaygarl et al., 2010 BIB014  BIB006 BIB012 BIB011 BIB013 BIB007 consider coverage of the SUT as a base to determine the reduced suite. However, Open-SourceRed BIB014 and TEMSA follows search-based techniques to find the diverse subset of a test suite. Similarly, MINTS BIB009 and EDTSO BIB015 employ ILP-based approach to determine the global optimal solution for TSR problem. In contrast, USbRed BIB008 employs similarity-based technique to find the most different test cases from a test suite. In comparison to search-based approaches, coverage-based approaches require extra computational time to determine the coverage of a program using certain coverage criteria such as statement, branch, or path coverage. Moreover, coverage-based approaches may produce optimal or near-optimal solution using greedy algorithms  . In contrast, search-based approaches employ different types of search algorithms, such as Genetic Algorithm, to determine a global optimal solution based on the defined fitness function. However, search-based approaches can be very time-consuming especially if the fitness function is evaluated in terms of increased coverage provided by the different possible solutions. The attributes of testing paradigm parameters state the type of programming languages in which the targeted SUT is developed. The TSR tools are broadly classified into structured, object-oriented, and aspect-oriented implementation platforms. The tools in (Horgan and London, 1992 BIB006 BIB012 , BIB002 BIB004 BIB005 , Jaygarl et al., 2010 BIB010 BIB014 BIB008  BIB009 BIB015 BIB011 BIB013 , and BIB003 , consider structured, object-oriented, and aspect-oriented programming platforms, to minimize the test-suite size, respectively. The TSR tools can be categorized into single-objective and multi-objective tools. The single-objective based optimization type focuses on cost such as loss in fault-detection capability or effectiveness in terms of size of the RS. Alternatively, MINTS BIB009 and EDTSO BIB015 focus on the multi-objective optimization type using ILP-based approach, which attempts to balance the tradeoffs between both cost and effectiveness measures. Multi-objective is preferable to the tester as it generates different possible solutions of considered objectives and constraints, which consequently helps the tester in the decision-making process. However, in comparison, single-objective takes less computation time, since it only considers either cost or effectiveness measure. The coverage source provides a base to determine the representative test cases. The tools in (Horgan and London, 1992 BIB004 BIB005 , Jaygarl et al., 2010 BIB010 BIB008  BIB013 BIB011 BIB014 BIB006 and BIB002 BIB003 consider source code and test execution profile, respectively. In comparison to source code, test execution profiles are expensive as they can be collected by executing the entire test suite prior to initiating the test reduction process. The execution platform considers a single server and multiple server attributes to support the reduction process. Among others, MINTS BIB009 and EDTSO BIB015 use multiple servers attribute to concurrently solve the TSR problem. As a result, the computational time taken by MINTS BIB009 and EDTSO BIB015 significantly reduced at the cost of extra hardware resources. The computational mode exploits offline (Horgan and London, 1992 BIB002 BIB003 BIB004 BIB005 , Jaygarl et al., 2010 BIB010 BIB013 BIB011 BIB014 BIB006 and online BIB008  attributes to generate the representative solution. In comparison to offline, online computational mode supports real-time testing, especially in the case of web-based applications. The attributes of license type parameters define whether the TSR tool is freely available or a commercial product. The tools in BIB001 London, 1992, Woo et al., 2007) and  are available as free and proprietary, respectively. Analysis of TSR tools reveals that most of the existing tools are products of multiple academic research projects. While commercial support for such tools is rather scarce. In contrast, most of the freely available open source tools provide full customizability (ATAC (Horgan and London, 1992) , RUTE-J BIB004 , Randoop BIB005 , Jtop (Zhang et al., 2009), Open-SourceRed (Kauffman and BIB014 ) so that desired alterations can be performed to adapt and enhance these tools according to the diverse set of requirments. While numerous other tools allow only limited (GZoltar , MINTS BIB009 , JINSI BIB013 EDTSO [41] ). It is really hard to find the available support for numerous tools. A significant hurdle in this regard is language barrier as some research groups have their web sites and support in languages other than English (RTL BIB006 , PLOOSE BIB012 , TOBIAS BIB007 ). This makes it hard to find the available support for respective tools. Moreover some proprietary academic research tools (USbRed BIB008 , CPUT  ) were only accessible through private search spaces, making it difficult to access executables and supporting materials. While considering evaluation of these tools, very few tools have been externally evaluated. This fact points to a significant research gap regarding empirical evaluation of existing TSR tools. Most of the freely available TSR tools are open source and we have provided download links for source codes and support for these tools (ATAC (Horgan and London, 1992) , RUTE-J BIB004 , Randoop BIB005 , Jtop (Zhang et al., 2009), Open-SourceRed (Kauffman and BIB014 , MINTS BIB009 ). This aggregation of information will help researchers in acquiring useful information for empirical evaluation and extension of existing tools. 