A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> Object-oriented unit tests consist of sequences of method invocations. Behavior of an invocation depends on the state of the receiver object and method arguments at the beginning of the invocation. Existing tools for automatic generation of object-oriented test suites, such as Jtest and J Crasher for Java, typically ignore this state and thus generate redundant tests that exercise the same method behavior, which increases the testing time without increasing the ability to detect faults. This work proposes Rostra, a framework for detecting redundant unit tests, and presents five fully automatic techniques within this framework. We use Rostra to assess and minimize test suites generated by test-generation tools. We also present how Rostra can be added to these tools to avoid generation of redundant tests. We have implemented the five Rostra techniques and evaluated them on 11 subjects taken from a variety of sources. The experimental results show that Jtest and JCrasher generate a high percentage of redundant tests and that Rostra can remove these redundant tests without decreasing the quality of test suites. <s> BIB001 </s> A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> Aspect-oriented software development is gaining popularity with the adoption of languages such as AspectJ. Testing is an important part in any software development, including aspect-oriented development. To automate generation of unit tests for AspectJ programs, we can apply the existing tools that automate generation of unit tests for Java programs. However, these tools can generate a large number of test inputs, and manually inspecting the behavior of the software on all these inputs is time consuming. We propose Raspect, a framework for detecting redundant unit tests for AspectJ programs. We introduce three levels of units in AspectJ programs: advised methods, advice, and intertype methods. We show how to detect at each level redundant test that does not exercise new behavior. Our approach selects only non-redundant tests from the automatically generated test suites, thus allowing the developer to spend less time in inspecting this reduced set of tests. We have implemented Raspect and applied it on 12 subjects taken from a variety of sources; our experience shows that Raspect can effectively reduce the size of generated test suites for inspecting AspectJ programs <s> BIB002 </s> A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> There are several problem areas that must be addressed when applying randomization to unit testing. As yet no general, fully automated solution that works for all units has been proposed. We therefore have developed RUTE-J, a Java package intended to help programmers do randomized unit testing in Java. In this paper, we describe RUTE-J and illustrate how it supports the development of per-unit solutions for the problems of randomized unit testing. We report on an experiment in which we applied RUTE-J to the standard Java TreeMap class, measuring the efficiency and effectiveness of the technique. We also illustrate the use of randomized testing in experimentation, by adapting RUTE-J so that it generates randomized minimal covering test suites, and measuring the effectiveness of the test suites generated. <s> BIB003 </s> A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> R<scp>ANDOOP</scp> for Java generates unit tests for Java code using feedback-directed random test generation. Below we describe R<scp>ANDOOP</scp>'s input, output, and test generation algorithm. We also give an overview of RANDOOP's annotation-based interface for specifying configuration parameters that affect R<scp>ANDOOP</scp>'s behavior and output. <s> BIB004 </s> A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> Combinatorial testing consists in generating (possibly large) test suites by combining both the sequencing of several operations, and the selection of test data. The TOBIAS tool is based on this generation technique. The combinatorial part of the approach makes both its strength and its weakness. Indeed, the more tests are produced, the more confidence the user may have in his/her program. Nevertheless, simple patterns may result in millions of test cases due to the combinatorial explosion problem, leading to intractable testsuites. To overcome this weakness, TOBIAS makes it possible to connect the generator to "selectors" that choose a subset of the test suite based on various techniques or criteria. This paper presents such a test suite reduction technique, based on a stochastic approach. The selection is improved so a uniform repartition of test cases is performed w.r.t. a given criterion, namely, an abstraction of the test cases based on their "shape". The approach is implemented into TOBIAS as a generic selector, independent of the system under test. It has been put into practice on a smart card case study, for which the results are reported in this paper. <s> BIB005 </s> A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> Test case management may make the testing process more efficient and thus accelerate software delivery. With the popularity of using JUnit for testing Java software, researchers have paid attention to techniques to manage JUnit test cases in regression testing of Java software. Typically, most existing test case management tools are based on the coverage information. However, coverage information may need extra efforts to obtain. In this paper, we present an Eclipse IDE plug-in (named Jtop) for managing JUnit test cases in absence of coverage information. Jtop statically analyzes the program under test and its corresponding JUnit test cases to perform the following management tasks: regression test case selection, test suite reduction and test case prioritization. Furthermore, Jtop also enables the programmer to manually manipulate test cases through a graphical user interface. <s> BIB006 </s> A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> A program fails. What now? Taking a single failing run, we record and minimize the interaction between objects to the set of calls relevant for the failure. The result is a minimal unit test that faithfully reproduces the failure at will: "Out of these 14,628 calls, only 2 are required". In a study of 17 real-life bugs, our JINSI prototype reduced the search space to 13.7% of the dynamic slice or 0.22% of the source code, with only 1--12 calls left to examine. <s> BIB007 </s> A survey on Test Suite Reduction frameworks and tools <s> Randomized Unit Testing based TSR Tools <s> When software developers make changes to a program, it is possible that they will introduce faults into previously working parts of the code. As software grows, a regression test suite is run to ensure that the old functionality still works as expected. Yet, as the number of test cases increases, it becomes more expensive to execute the test suite. Reduction and prioritization techniques enable developers to manage large and unwieldy test suites. However, practitioners and researchers do not always use and study these methods due, in part, to a lack of availability. In response to this issue, this paper describes an already released open-source framework that supports both research and practice in regression testing. The sharing of this framework will enable the replication of empirical studies in regression testing and encourage faster industrial adoption of these useful, yet rarely used, techniques. <s> BIB008
Majority of existing TSR tools focused on randomized unit testing, including ATAC (Horgan and London, 1992) , Rostra BIB001 , Raspect BIB002 , RUTE-J BIB003 , Randoop BIB004 , GenRed (Jaygarl et al., 2010) , Jtop BIB006 , TOBIAS BIB005 , TEMSA and Open-SourceRed BIB008 . In the following, we briefly discuss current randomized unit testing focused TSR tools.  ATAC: Horgan and London (Horgan and London, 1992) proposed a unit testing based TSR tool called ATAC that eliminates redundant test cases based on the high coverage of selected data-flow coverage metric. The main advantage of ATAC is to support the test evaluation process by implicitly facilitating in obtaining execution slices of program paths. However, ATAC lacks in detecting a significant number of crucial defects, since it only supports selective testing.  Rostra: Xie et al. BIB001 ) presented a formal object-oriented unit testing based framework, called Rostra that facilitates to evaluate a test suite's quality by finding similar unit test cases based on the equivalent objects. Although, it is argued that Rostra efficiently generates test sequences such as violating and satisfying with few ineffective test sequences. However, Rostra uses state-space exploration, which is heuristically pruned. Ultimately, the proposed tool never guarantees to determine minimal solution.  Raspect: Xie et al. BIB002 extended Rostra BIB001 to detect the redundant unit test cases for aspect-oriented programs, called Raspect that preserves structural coverage using similar states for the considered objects. The proposed tool automatically detects similar test cases, which do not exercise the new behavior of the program under test. Consequently, it requires less time for manual inspection of the computed solution. However, Raspect performance degrades significantly in the case of non-availability of program's algebraic specification, which is used for asserting the program behavior.  RUTE-J: Andrews et al. BIB003 proposed a tool, called RUTE-J to test a small portion of the program under test such as methods or classes. It uses delta debugging technique BIB007 to isolate the failure-inducing inputs of the program under test. Furthermore, it effectively manages test's storage and retrieval operations. However, RUTE-J tool interaction experience is highly required as the tester has to enter some technical inputs.  Randoop: Pacheco and Ernst BIB004 presented a deterministic execution feedback-directed tool, called Randoop that determines and eliminates the test inputs that either throw exceptions or create similar objects. However, Randoop lacks in creating a parameter graph instead it used previously generated test sequences of a component set, which are useful to create similar objects or throw exceptions. Consequently, Randoop determines a small number of redundant test cases.  GenRed: Jayagarl et al. (Jaygarl et al., 2010) proposed a new feedback-directed randomized tool, called GenRed that discards generated redundant method sequences. The authors empirically evaluated the performance of GenRed and Randoop BIB004 ) using a total testing time criterion. Although, GenRed showed good performance compared to Randoop in terms of size of the RS and achieved code coverage, however, the authors did not report required time mechanism necessary to understand the core mechanism of controlling GenRed working. BIB008 proposed an open-source framework supported by a prototype tool, called Open-SourceRed, which contains two open-source components: (i) Proteja and (ii) Modificare. Proteja accepts two main inputs including, Java Program and JUnit test suite, and generates a test-coverage report in a binary matrix format based on statement, method, and class coverage criteria. In contrast, Modificare takes test-coverage and timing information to perform TSR and also produces a modified suite file.