A survey of combinatorial testing <s> Metric and Evaluation <s> A trend in software development is to assemble a system from a number of components. These may be either available commercially off-the-shelf, or by the use of network-based resources. In many cases, the system is expected to function for multiple configurations of interchangeable components. The trade off that a system tester faces is the thoroughness of test component configuration coverage, versus limited resources of time and expense. This paper presents a metric that can be used to measure component interaction coverage of a set of system test configurations. It also provides a formal definition of the system test interaction problem. <s> BIB001 </s> A survey of combinatorial testing <s> Metric and Evaluation <s> Software testing plays a critical role in the timely delivery of high-quality software systems. Despite the important role that testing plays, little is known about the fault detection effectiveness of many testing techniques. We investigate "n-way" test suites created using a common greedy algorithm for use in combinatorial testing. A controlled study is designed and executed to compare the fault detection effectiveness of n-way and random test suites. Combinatorial testing is conducted on target systems that have been injected with software faults. The results are that there is no significant difference in the fault detection effectiveness of n-way and random test suites for the applications studied. Analysis of the random test suite finds that they are very similar to n-way test suites from the perspective of the number of test data combinations covered. This result concurs with other hypothetical results that indicate little difference between n-way and random test suites. While we do not expect this result to apply in all combinatorial testing situations, we believe the result will lead to the design of better combinatorial test suites. <s> BIB002 </s> A survey of combinatorial testing <s> Metric and Evaluation <s> Abstract The quality of software has been a main concern since the inception of computer software. To be able to produce high quality software, software developers and software testers alike need continuous improvements in their developing and testing methodologies. These improvements should result in better coverage of the input domain, efficient test cases, and in spending fewer testing resources. In this paper we focus on an approach for generating efficient test cases based on the special properties of Design of Experiments and developing a logistic regression model of predicting test case outcomes. Design of Experiments will be utilized to efficiently minimize the number of test cases and the logistic regression model will be used to predict software failures. This approach, in turn, would provide the software tester with a model that reduces the number of test cases, predicts test case outcomes, reduces cost, and allows better forecast of release readiness. We demonstrate our approach using two case studies (TI Interactive software and Microsoft's Pocket PC operating system). <s> BIB003
There exist two kinds of evaluation of CT: (1) measure and evaluate CT itself; and (2) measure and evaluate the quality of the SUT after CT. CT has a natural metric: combination coverage, which measures the percentage of the covered parameter value combinations relative to the total combinations. One way to measure the combination coverage is based on the k-value schemas tested relative to the total k-value schemas. This kind of CT metric can serve two purposes: (1) to set a test coverage target; and (2) to evaluate different test strategies of CT which use different test suites. BIB001 described the metric for CT and provide a formal definition of CT. BIB002 compared the failure detection effectiveness of τ -way CT and random testing. It is not easy to evaluate software quality after CT, as very few works have focused on this topic. We only find one study by  and BIB003 , in which they developed a logistic regression model of predicting software failure based on the testing result of CT. More studies should be devoted to this area. We can try to enhance existing methods of quality evaluation based on the characteristics of CT. To evolve and obtain a reasonable evaluation method, more empirical evidences are needed.