A survey on Test Suite Reduction frameworks and tools <s> Taxonomy of Test Suite Reduction (TSR) Frameworks/Tools <s> This paper presents an empirical comparison of four different techniques for filtering large test suites: test suite minimization, prioritization by additional coverage, cluster filtering with one-per-cluster sampling, and failure pursuit sampling. The first two techniques are based on selecting subsets that maximize code coverage as quickly as possible, while the latter two are based on analyzing the distribution of the tests' execution profiles. These techniques were compared with data sets obtained from three large subject programs: the GCC, Jikes, and javac compilers. The results indicate that distribution-based techniques can be as efficient or more efficient for revealing defects than coverage-based techniques, but that the two kinds of techniques are also complementary in the sense that they find different defects. Accordingly, some simple combinations of these techniques were evaluated for use in test case prioritization. The results indicate that these techniques can create more efficient prioritizations than those generated using prioritization by additional coverage. <s> BIB001 </s> A survey on Test Suite Reduction frameworks and tools <s> Taxonomy of Test Suite Reduction (TSR) Frameworks/Tools <s> Using bi-criteria decision making analysis, a new model for test suite minimization has been developed that pursues two objectives: minimizing a test suite with regard to a particular level of coverage while simultaneously maximizing error detection rates. This new representation makes it possible to achieve significant reductions in test suite size without experiencing a decrease in error detection rates. Using the all-uses inter-procedural data flow testing criterion, two binary integer linear programming models were evaluated, one a single-objective model, the other a weighted-sums bi-criteria model. The applicability of the bi-criteria model to regression test suite maintenance was also evaluated. The data show that minimization based solely on definition-use association coverage may have a negative impact on the error detection rate as compared to minimization performed with a bi-criteria model that also takes into account the ability of test cases to reveal error. Results obtained with the bi-criteria model also indicate that test suites minimized with respect to a collection of program faults are effective at revealing subsequent program faults. <s> BIB002 </s> A survey on Test Suite Reduction frameworks and tools <s> Taxonomy of Test Suite Reduction (TSR) Frameworks/Tools <s> Previous work has treated test case selection as a single objective optimisation problem. This paper introduces the concept of Pareto efficiency to test case selection. The Pareto efficient approach takes multiple objectives such as code coverage, past fault-detection history and execution cost, and constructs a group of non-dominating, equivalently optimal test case subsets. The paper describes the potential bene?ts of Pareto efficient multi-objective test case selection, illustrating with empirical studies of two and three objective formulations. <s> BIB003 </s> A survey on Test Suite Reduction frameworks and tools <s> Taxonomy of Test Suite Reduction (TSR) Frameworks/Tools <s> The design of an adequate test suite is usually guided by identifying test requirements which should be satisfied by the selected set of test cases. To reduce testing costs, test suite minimization heuristics aim at eliminating redundancy from existing test suites. However, recent test suite minimization approaches lack (1) to handle test suites commonly derived for families of similar software variants under test, and (2) to incorporate fine-grained information concerning cost/profit goals for test case selection. In this paper, we propose a formal framework to optimize test suites designed for sets of software variants under test w.r.t. multiple conflicting cost/profit objectives. The problem representation is independent of the concrete testing methodology. We apply integer linear programming (ILP) to approximate optimal solutions. We further develop an efficient incremental heuristic for deriving a sequence of representative software variants to be tested for approaching optimal profits under reduced costs. We evaluated the algorithm by comparing its outcome to the optimal solution. <s> BIB004
This section presents a taxonomy for the thematic classification of TSR frameworks and tools based on defined parameters as depicted clearly in Figure 1 . The defined parameters include: (i) approach type, (ii) testing paradigm, (iii) optimization type, (iv) coverage source, (v) execution platform, (vi) computational mode, (vii) license type, (viii) evaluation, (ix) customizability, and (x) support. The first identified parameter is the approach type, which represents the principal category of TSR approaches focused by a TSR framework. There are four main attributes for approach type: (i) coverage-based, (ii) search-based, (iii) Integer Linear Programming (ILP) based, and (iv) similarity-based. The coverage-based TSR approaches greedily select such test cases that cover maximum portions (e.g., statements or branches) of a program under test. In contrast, search-based approaches employ various search algorithms, such as Genetic Algorithm  , to find diverse test cases from the initial population. Conversely, ILP-based approaches determine minimal global solution for TSR problem based on the defined objectives and constraints BIB002 . Furthermore, ILP-based approaches achieve Pareto-optimal solution for multi-objective TSR problem BIB004 . Notice that Paretooptimal solution is a non-dominated solution, which cannot be further improved BIB003 . In contrast, similarity-based approaches focus on finding most different test cases based on the computed similarity degree between test cases  . Note that similarity-based approaches heavily depend on a similarity matrix that shows the degree of similarity for all pairs of test cases. The similarity degree is calculated using a similarity measure such as Jaccard index  . The parameter testing paradigm represents the type of implementation language of the targeted SUT. The supported attributes are structured, object-oriented, or aspect-oriented. The other important parameter is the optimization type, which is regarded as a number of optimization objectives considered by a TSR framework. The optimization type is categorized into (i) single-objective: focuses either on finding a RS or computing the FDE, and (ii) multi-objective: focuses on both RS and FDE. In comparison to single-objective optimization, multi-objective supported tools need to find a best tradeoff between the targeted objectives and constraints . The parameter coverage source is employed by the coverage supported frameworks to find a reduced test suite. The coverage source uses two main attributes: (i) source code and (ii) test execution profile. The frameworks that use the source code as a coverage source, determine the reduced suite by picking such test cases that cover maximum elements of the program source code. On the other hand, in the test execution profile-based category, first execution profiles are captured by running the entire test suite BIB001 . Next, the supported framework determines the representative test cases based on the achieved coverage score of execution profiles. The parameter execution platform represents the number of servers used by a test reduction framework to determine the RS. It can be broadly classified into: (i) single server and (ii) multiple servers. In the case of multiple servers, the TSR problem is solved by using divide-and-conquer strategy. Initially, the problem (large test suite) is divided into many sub-problems (several small test suites). Next, each sub-problem is executed on a single server with the aim to speed up the TSR process. The computational mode is the type of processing supported by a framework during TSR process and classified into two attributes: (i) online and (ii) offline. The online attribute accepts several smaller test suites in a sequential manner to determine the RS. In contrast, the offline attribute accepts the whole test suite to find an optimal solution. The parameter license type is considered as a primary characteristic for the selection of TSR tools. This parameter defines three main categories including commercial, academic research, and free tools. License of Commercial tools can be acquired through payment. Academic research tools include prototypes and tools developed by research labs/ groups, while Free tools are available free of cost. These tools are mostly open source having common licenses such as GPL, ASL, EPL and CeCILL. Evaluation parameter refers to external or internal evaluation of the selected frameworks. Internal evaluation disscusses those framework which have been evaluated by builders/design teams. While external evaluation refers to those framework which have been tested outside the environment where they have been originally built. Such external evaluation enhances the degree of confidence regarding usefulness of certain framework. Customizability can be regarded as an ability of the tool to support desired alterations. This parameter defines three categories including Full, partial and No support for customizability. Fully customizable tools support major alterations (e.g., customizing major basic functions) while partial customizability allows only minor alterations (e.g., seamless integration into a given environment). For example, most of the open source tools support full customizability. On the other hand, most of the proprietary and commercial tools provide partial or no customizability. Another significant characteristic of any tool is the support availability. Information about executables and written documentation is indispensable for a wider analysis of TSR tools. In our case, very few studies provided download links for executables and source code. So we have performed rigorous search to find the web links where tool support (executables, source code, written documentation) was actually available. We have approached personal web sites of authors and research groups in this regard. We have mainly divided available support into executables, written documentation (manuals, tutorials, instructions, examples etc.) and tool web site.