A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> Object-oriented unit tests consist of sequences of method invocations. Behavior of an invocation depends on the state of the receiver object and method arguments at the beginning of the invocation. Existing tools for automatic generation of object-oriented test suites, such as Jtest and J Crasher for Java, typically ignore this state and thus generate redundant tests that exercise the same method behavior, which increases the testing time without increasing the ability to detect faults. This work proposes Rostra, a framework for detecting redundant unit tests, and presents five fully automatic techniques within this framework. We use Rostra to assess and minimize test suites generated by test-generation tools. We also present how Rostra can be added to these tools to avoid generation of redundant tests. We have implemented the five Rostra techniques and evaluated them on 11 subjects taken from a variety of sources. The experimental results show that Jtest and JCrasher generate a high percentage of redundant tests and that Rostra can remove these redundant tests without decreasing the quality of test suites. <s> BIB001 </s> A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> Aspect-oriented software development is gaining popularity with the adoption of languages such as AspectJ. Testing is an important part in any software development, including aspect-oriented development. To automate generation of unit tests for AspectJ programs, we can apply the existing tools that automate generation of unit tests for Java programs. However, these tools can generate a large number of test inputs, and manually inspecting the behavior of the software on all these inputs is time consuming. We propose Raspect, a framework for detecting redundant unit tests for AspectJ programs. We introduce three levels of units in AspectJ programs: advised methods, advice, and intertype methods. We show how to detect at each level redundant test that does not exercise new behavior. Our approach selects only non-redundant tests from the automatically generated test suites, thus allowing the developer to spend less time in inspecting this reduced set of tests. We have implemented Raspect and applied it on 12 subjects taken from a variety of sources; our experience shows that Raspect can effectively reduce the size of generated test suites for inspecting AspectJ programs <s> BIB002 </s> A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> There are several problem areas that must be addressed when applying randomization to unit testing. As yet no general, fully automated solution that works for all units has been proposed. We therefore have developed RUTE-J, a Java package intended to help programmers do randomized unit testing in Java. In this paper, we describe RUTE-J and illustrate how it supports the development of per-unit solutions for the problems of randomized unit testing. We report on an experiment in which we applied RUTE-J to the standard Java TreeMap class, measuring the efficiency and effectiveness of the technique. We also illustrate the use of randomized testing in experimentation, by adapting RUTE-J so that it generates randomized minimal covering test suites, and measuring the effectiveness of the test suites generated. <s> BIB003 </s> A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> Combinatorial testing consists in generating (possibly large) test suites by combining both the sequencing of several operations, and the selection of test data. The TOBIAS tool is based on this generation technique. The combinatorial part of the approach makes both its strength and its weakness. Indeed, the more tests are produced, the more confidence the user may have in his/her program. Nevertheless, simple patterns may result in millions of test cases due to the combinatorial explosion problem, leading to intractable testsuites. To overcome this weakness, TOBIAS makes it possible to connect the generator to "selectors" that choose a subset of the test suite based on various techniques or criteria. This paper presents such a test suite reduction technique, based on a stochastic approach. The selection is improved so a uniform repartition of test cases is performed w.r.t. a given criterion, namely, an abstraction of the test cases based on their "shape". The approach is implemented into TOBIAS as a generic selector, independent of the system under test. It has been put into practice on a smart card case study, for which the results are reported in this paper. <s> BIB004 </s> A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> R<scp>ANDOOP</scp> for Java generates unit tests for Java code using feedback-directed random test generation. Below we describe R<scp>ANDOOP</scp>'s input, output, and test generation algorithm. We also give an overview of RANDOOP's annotation-based interface for specifying configuration parameters that affect R<scp>ANDOOP</scp>'s behavior and output. <s> BIB005 </s> A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> To reason about programs, dynamic optimizers and analysis tools use sampling to collect a dynamic call graph (DCG). However, sampling has not achieved high accuracy with low runtime overhead. As object-oriented programmers compose increasingly complex programs, inaccurate call graphs will inhibit analysis and optimizations. This paper demonstrates how to use static and dynamic control flow graph (CFG) constraints to improve the accuracy of the DCG. We introduce the frequency dominator (FDOM), a novel CFG relation that extends the dominator relation to expose static relative execution frequencies of basic blocks. We combine conservation of flow and dynamic CFG basic block profiles to further improve the accuracy of the DCG. Together these approaches add minimal overhead (1%) and achieve 85% accuracy compared to a perfect call graph for SPEC JVM98 and DaCapo benchmarks. Compared to sampling alone, accuracy improves by 12 to 36%. These results demonstrate that static and dynamic control-flow information offer accurate information for efficiently improving the DCG. <s> BIB006 </s> A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> Test case management may make the testing process more efficient and thus accelerate software delivery. With the popularity of using JUnit for testing Java software, researchers have paid attention to techniques to manage JUnit test cases in regression testing of Java software. Typically, most existing test case management tools are based on the coverage information. However, coverage information may need extra efforts to obtain. In this paper, we present an Eclipse IDE plug-in (named Jtop) for managing JUnit test cases in absence of coverage information. Jtop statically analyzes the program under test and its corresponding JUnit test cases to perform the following management tasks: regression test case selection, test suite reduction and test case prioritization. Furthermore, Jtop also enables the programmer to manually manipulate test cases through a graphical user interface. <s> BIB007 </s> A survey on Test Suite Reduction frameworks and tools <s> Comparison of Randomized Unit Testing based TSR Tools <s> When software developers make changes to a program, it is possible that they will introduce faults into previously working parts of the code. As software grows, a regression test suite is run to ensure that the old functionality still works as expected. Yet, as the number of test cases increases, it becomes more expensive to execute the test suite. Reduction and prioritization techniques enable developers to manage large and unwieldy test suites. However, practitioners and researchers do not always use and study these methods due, in part, to a lack of availability. In response to this issue, this paper describes an already released open-source framework that supports both research and practice in regression testing. The sharing of this framework will enable the replication of empirical studies in regression testing and encourage faster industrial adoption of these useful, yet rarely used, techniques. <s> BIB008
A significant number of existing TSR tools (53%, 10/19) focused on randomized unit testing. However, there are two key limitations associated with current tools in the category of randomized unit testing that need further research to design more effective tools. The limitations are: (i) test oracle problem, which demands manual effort to evaluate the test results, and (ii) generating infeasible test cases, since execution does not represent the actual scenario. Currently, tools are focusing on different programming domains, including structured (Horgan and London, 1992) , objectoriented BIB007 BIB002 BIB003 , Jaygarl et al., 2010 BIB008 BIB004 , and aspect-oriented BIB001 . However, there is no generalized tool, which is applicable and effective to test programs developed using various programming domains. A variety of tools for randomized unit testing has been proposed, where the majority was based on code coverage information of the program under test to determine the reduced suite (Horgan and London, 1992 BIB001 BIB002 BIB003 BIB005 , Jaygarl et al., 2010 BIB004 . However, there are a number of drawbacks of using code coverage information: (i) instrumented version of source code needs to be run to collect coverage information, (ii) coverage storage and management cost proportionally increases with respect to the program size, and (iii) previously collected coverage information becomes inconsistent due to software evolution. Consequently, current coverage-based TSR tools might be difficult to test real-world applications due to extra cost in terms of coverage collection, storage, and management. In this situation, compared to ATAC (Horgan and London, 1992) , Rostra BIB001 , Raspect BIB002 , RUTE-J BIB003 , Randoop BIB005 , GenRed (Jaygarl et al., 2010) , TOBIAS BIB004 , TEMSA and OpenSourceRed BIB008 , Jtop BIB007 ) is an effective tool, since it does not require any code coverage information. Furthermore, Jtop automatically extracts static call graph to determine a test case covering particular program elements; however, it usually contains over approximations. In other words, over approximation results significant call relationships, which would never occur in real program execution. In contrast to other randomized unit testing tools, both Rostra BIB001 and Raspect BIB002 dynamically monitor the test executions to detect and remove the redundant test cases based on equivalent object's states. To achieve this, they used bounded exhaustive generation with state-space exploration. Consequently, they required less time for test synthesis as redundant test cases are automatically detected. However, both Rostra BIB001 and Raspect BIB002 ) requires significant resources in terms of memory to keep track of all object's states. Among all, two TSR tools including, Randoop BIB005 and GenRed (Jaygarl et al., 2010) , are based on feedback-directed mechanism, which enables a limited access to the objects. For instance, if Randoop BIB005 ) is unable to locate an object of the correct type required to invoke a method in the existing set of sequences, then it will never be able to invoke such method. Moreover, Randoop BIB005 ) is easy to use but lacks in supporting multi-threaded programs. Ultimately, Randoop BIB005 ) is unable to concurrently execute the methods. Thus, in this situation, GenRed (Jaygarl et al., 2010) outperformed Randoop BIB005 by using on-demand generation to create necessary test inputs. Furthermore, GenRed (Jaygarl et al., 2010) assigns high priority to the methods with lower coverage to invoke method in the current set of method sequences. Some randomized unit testing supported TSR tools including, ATAC (Horgan and London, 1992) , RUTE-J BIB003 , Randoop BIB005 , GenRed (Jaygarl et al., 2010) , TOBIAS BIB004 , and Open-SourceRed BIB008 , require high human interventions during TSR process for their correct functioning, which ultimately needs additional testing time. For instance, ATAC (Horgan and London, 1992) requires high human interaction to create test data, test script, and evaluate test results. Similarly, TOBIAS BIB004 requires significant effort of test engineer to write test patterns based on their knowledge. In this situation, Raspect BIB002 requires less manual inspection, since it automatically detects redundant test cases using program algebraic specifications. In contrast, some tools such as RUTE-J BIB003 , Randoop BIB005 , GenRed (Jaygarl et al., 2010) , and Open-SourceRed BIB008 require high user experience regarding tool interaction in order to enter some technical inputs. For example, RUTE-J BIB003 heavily depends on user interaction experience as it has no intelligence to automatically check the correctness of the run-time entered values, including method parameters range and weighting scheme. Similarly, Randoop BIB005 requires the test engineer's interaction to set the time limits for determining the RS. In this situation, Jtop BIB007 ) outperformed other proposed tools since it does not require any user experience. Jtop automatically extracts the static call graph in order to generate nonredundant test cases. Future tools can further improve the capability of Jtop by constructing a dynamic call graph BIB006 , which extracts an exact record of program execution frequencies of call relationship based on the profile information. Ultimately, it is beneficial in optimizing overall program under test behavior. However, some cost may be required in terms of non-invasive capture tools to adequately capture the execution traces for generating profile information.