A survey of combinatorial testing <s> Test Case Prioritization <s> Event-driven software (EDS) is a widely used class of software that takes sequences of events as input, changes state, and outputs new event sequences. Managing the size of tests suites for EDS is difficult as the number of event combinations and sequences grow exponentially with the number of events. We propose a new testing technique that extends software interaction testing. Traditional software interaction testing systematically examines all t-way interactions of parameters for a program. This paper extends the notion to t-way interactions over sequences of events. The technique applies to many classes of software; we focus on that of EDS. As a proof-of-concept, we prioritize existing test suites for four GUI-based programs by t-way interaction coverage. We compare the rate of fault detection with that of several other prioritization criteria. Results show that prioritization by interaction coverage has the fastest rate of fault detection in half of our experiments, making the most impact when tests have high interaction coverage. <s> BIB001 </s> A survey of combinatorial testing <s> Test Case Prioritization <s> Regression testing is an expensive part of the software maintenance process. Effective regression testing techniques select and order (or prioritize) test cases between successive releases of a program. However, selection and prioritization are dependent on the quality of the initial test suite. An effective and cost efficient test generation technique is combinatorial interaction testing, CIT, which systematically samples all t-way combinations of input parameters. Research on CIT, to date, has focused on single version software systems. There has been little work that empirically assesses the use of CIT test generation as the basis for selection or prioritization. In this paper we examine the effectiveness of CIT across multiple versions of two software subjects. Our results show that CIT performs well in finding seeded faults when compared with an exhaustive test set. We examine several CIT prioritization techniques and compare them with a re-generation/prioritization technique. We find that prioritized and re-generated/prioritized CIT test suites may find faults earlier than unordered CIT test suites, although the re-generated/prioritized test suites sometimes exhibit decreased fault detection. <s> BIB002 </s> A survey of combinatorial testing <s> Test Case Prioritization <s> Regression testing is an expensive, but important, process. Unfortunately, there may be insufficient resources to allow for the reexecution of all test cases during regression testing. In this situation, test case prioritization techniques aim to improve the effectiveness of regression testing by ordering the test cases so that the most beneficial are executed first. Previous work on regression test case prioritization has focused on greedy algorithms. However, it is known that these algorithms may produce suboptimal results because they may construct results that denote only local minima within the search space. By contrast, metaheuristic and evolutionary search algorithms aim to avoid such problems. This paper presents results from an empirical study of the application of several greedy, metaheuristic, and evolutionary search algorithms to six programs, ranging from 374 to 11,148 lines of code for three choices of fitness metric. The paper addresses the problems of choice of fitness metric, characterization of landscape modality, and determination of the most suitable search technique to apply. The empirical results replicate previous results concerning greedy algorithms. They shed light on the nature of the regression testing search space, indicating that it is multimodal. The results also show that genetic algorithms perform well, although greedy approaches are surprisingly effective, given the multimodal nature of the landscape <s> BIB003 </s> A survey of combinatorial testing <s> Test Case Prioritization <s> Web applications have rapidly become a critical part of business for many organizations. However, increased usage of Web applications has not been reciprocated with corresponding increases in reliability. Unique characteristics, such as quick turnaround time, coupled with growing popularity motivate the need for efficient and effective Web application testing strategies. In this paper, we propose several new test suite prioritization strategies for Web applications and examine whether these strategies can improve the rate of fault detection for three Web applications and their preexisting test suites. We prioritize test suites by test lengths, frequency of appearance of request sequences, and systematic coverage of parameter-values and their interactions. Experimental results show that the proposed prioritization criteria often improve the rate of fault detection of the test suites when compared to random ordering of test cases. In general, the best prioritization metrics either (1) consider frequency of appearance of sequences of requests or (2) systematically cover combinations of parameter-values as early as possible. <s> BIB004
Test case prioritization has been well studied. The result of the prioritization is often a schedule of test cases so that those with the highest priority, according to some criterion, are executed earlier in testing. When testing is terminated after running a subset of the prioritized test suite, those test cases deemed most important are executed. Especially when the resource is limited, the important test cases should be tested as early as possible. Well-ordered testing may reveal faults early because it ensures that the important test cases are executed first. The test case prioritization problem is defined as follows. Definition 3.3.1. Given (T , , f ), where T is a test suite, is the set of all test suites obtained by permuting the tests of T, and f is a function from to real numbers, the test case prioritization problem is to find π ∈ such that ∀π ∈ , f (π ) ≥ f (π ). gives the possible prioritization of T and f is the function to evaluate the prioritization BIB001 . Prioritization can be computed in two ways: (1) reorder an existing test suite of CT based on a prioritization criteria; and (2) generate an ordered test suite for CT, taking into account the importance of combinations. By Definition 3.3.1, the key issues in test prioritization are how to define the function f , and validate the effectiveness of the prioritization. and  adapted a one-test-at-a-time greedy method to take importance of parameter pairs into account. With due consideration of seeding and constraint, they used this method to generate a set of tests in order. BIB001 explored the effectiveness of the prioritization with interaction coverage in the event-driven software. Compared with other criteria like ordering test by unique event coverage, ordering test by the length, and random ordering, prioritization with interaction coverage was found to provide the fastest rate of fault detection. BIB002 tried three code-coverage-based weighting methods and one specification-based method to prioritize the test suite, and found that the prioritized CIT test suites may find faults earlier than unordered CIT test suites, although the prioritized test suites sometimes exhibit decreased fault detection. Different prioritization methods have different performance. Better methods of test case prioritization for CT may be developed by considering additional factors. For example, we may do prioritization based on required execution time or based on incremental τ -way coverage BIB001 . The prioritization done on GUI applications was also applied to Web application BIB004 . We also need more empirical study to better understand the limitations and strengths of various proposed methods BIB003 ].