A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> Orthogonal Latin squares—a new method for testing compilers—yields the informational equivalent of exhaustive testing at a fraction of the cost. The method has been used successfully in designing some of the tests in the Ada Compiler Validation Capability (ACVC) test suite. <s> BIB001 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> The present position of the art of field experimentation is one of rather special interest. For more than fifteen years the attention of agriculturalists has been turned to the errors of field experiments. During this period, experiments of the uniformity trial type have demonstrated the magnitude and ubiquity of that class of error which cannot be ascribed to carelessness in measuring the land or weighing the produce, and which is consequently described as due to “soil heterogeneity”; much ingenuity has been expended in devising plans for the proper arrangement of the plots; and not without result, for there can be little doubt that the standard of accuracy has been materially, though very irregularly, raised. What makes the present position interesting is that it is now possible to demonstrate (a) that the actual position of the problem is very much more intricate than was till recently imagined, but that realising this (b) the problem itself becomes much more definite and (c) its solution correspondingly more rigorous. <s> BIB002 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> Robust Testing is a systematic test method that uses the OATS (Orthogonal Array Testing System) tool to convert information from product requirements or specifications into a concise set of tests. We found Robust Testing more than twice as productive as conventional testing practices for testing PMX/StarMAIL Release 2.2, a complex private mail exchange (PMX) local area network (LAN)-based electronic mail product that works with several types and versions of LAN software, operating systems, and personal computers (PCs). Robust Testing required less time, enabling us to complete testing on schedule. It found more potentially user-affecting faults, and inspired greater confidence in the quality of the delivered product. <s> BIB003 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> This paper describes how both the quality and efficiency of protocol testing were improved by using a new Bellcore tool called the Automatic Efficient Test Generator (AETG). The AETG tool is based on ideas from experimental design and it creates a test set that contains all possible pairs of involved factors. Two examples are given to illustrate this technique and compare it with traditional approaches. The improved quality of testing leads to a faster detection of nonconformances and a higher quality of products in a shorter development interval. Although the application discussed in this paper covers protocol conformance testing, the techniques for improving the quality of testing can be applied to other types of testing such as feature testing and testing between two different network elements. > <s> BIB004 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> Distributed systems consist of a number of network elements that interact with each other. As the number of network elements and interchangeable components for each network element increases, the trade-off that the system tester faces is the thoroughness of test configuration coverage vs. limited resources of time and expense that are available. An approach to resolving this trade-off is to determine a set of test configurations that test each pair-wise combination of network components. This goal gives a well-defined level of test coverage, with a reduced number of system configurations. To select such a set of test configurations, we show how to apply the method of orthogonal Latin squares, from the design of balanced statistical experiments. Since the theoretical treatment assumes constraints that may not be satisfied in practice, we then show how to adapt this approach to realistic application constraints. <s> BIB005 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> Testers face the challenge of doing as much testing as possible within the available, and usually constrained, schedule. In the hypercompetitive commercial marketplace, it is not practical to exhaustively test all combinations of system test cases. This paper discusses the author’s experience with a method that generates a small subset of test cases which provides good coverage of the test domain—the Combinatorial Design Method. It has proven to be flexible for system level testing of small, commercial satellite ground systems. Cost savings of 67% and schedule savings of 68% are shown for an example. <s> BIB006 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> Testing graphical user interfaces (GUI) is a difficult problem due to the fact that the GUI possesses a large number of states to be tested, the input space is extremely large due to different permutations of inputs and events which affect the GUI, and complex GUI dependencies which may exist. There has been little systematic study of this problem yielding a resulting strategy which is effective and scalable. The proposed method concentrates upon user sequences of GUI objects and selections which collaborate, called complete interaction sequences (CIS), that produce a desired response for the user. A systematic method to test these CIS utilizes a finite-state model to generate tests. The required tests can be substantially reduced by identifying components of the CIS that can be tested separately. Since consideration is given to defects totally within each CIS, and the components reduce required testing further, this approach is scalable. An empirical investigation of this method shows that substantial reduction in tests can still detect the defects in the GUI. Future research will prioritize testing related to the CIS testing for maximum benefit if testing time is limited. <s> BIB007 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> Performance of computer-based systems may depend on many different factors, internal and external. In order to design a system to have the desired performance or to validate that the system has the required performance, the effect of the influencing factors must be known. Common methods give no or little guidance on how to vary the factors during prototyping or validation. Varying the factors in all possible combinations would be too expensive and too time-consuming. This paper introduces a systematic approach to the prototyping and the validation of a system's performance, by treating the prototyping or validation as an experiment, in which the fractional factorial design methodology is commonly used. To show that this is possible, a case study evaluating the influencing factors of the false and real target rate of a radar system is described. Our findings show that prototyping and validation of system performance become structured and effective when using the fractional factorial design. The methodology enables planning, performance, structured analysis, and gives guidance for appropriate test cases. The methodology yields not only main factors, but also interacting factors. The effort is minimized for finding the results, due to the methodology. The case study shows that after 112 test cases, of 1024 possible, the knowledge gained was enough to draw conclusions on the effects and interactions of 10 factors. This is a reduction with a factor 5-9 compared to alternative methods. <s> BIB008 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> For ensuring the displaying effects of Web applications, it is important to perform compatibility testing for browsers under the different configurations and it is a hard task to test all. So this paper focused on the improvements for browser compatibility testing. Firstly, we introduced the related work. Then we analysed the functional speciality of the current popular browsers, and provided the preconditions to simplify problems. Next we gave an instance and brought forward the single factor covering method and the pair-wise covering design to gain testing suits. Finally, we introduced the assistant tools we have developed and the future work. <s> BIB009 </s> A survey of combinatorial testing <s> The Application of CT and Testing Procedure <s> Combinatorial testing methods address generation of test cases for problems involving multiple parameters and combinations. The Orthogonal Array Based Testing Strategy (OATS) is one such combinatorial testing method, a systematic, statistical way of testing pair-wise interactions. It provides representative (uniformly distributed) coverage of all variable pair combinations. This makes the technique particularly useful for testing of software, wherever there is combinatorial explosion: a. In system testing for handling feature interactions b. In integration testing components c. It is also quite useful for testing products with a large number of configuration possibilities. One of the fundamental assumptions behind OATS approach is that a subset covering all pair-wise combinations will be more effective than a randomly selected subset. OATS provides a means to select a minimal test set that guarantees testing the pair-wise combinations of all the selected variables. Covering pair-wise combinations has been reported to be very effective in the literature. Successful use of this technique, with 50% effort saving and improved testing with a factor of 2.6 is reported in the literature. In this paper, we report on the in-house web-based application that we designed and implemented to support customized version of OATS and our experience in piloting and getting this method used in projects. In the in-house tool we have introduced a number of additional features, that help in generation and post processing of test-cases. We have also designed a supporting process for using this method, and we discuss the steps in this process in the paper. We share details on application in feature testing of a mobile phone application. This method has also been successfully used in designing feature interaction test cases and for augmenting the regression suite to increase coverage. <s> BIB010
The second most popular topic of CT research is the study of applying CT to various types of applications, and the refinement of testing procedures. The basic concept of CT was first used in other disciplines many years ago. In 1926, Fisher pioneered interaction tests in agricultural experiments, assessing the contributions of different fertilizers to crop yield in the context of soil heterogeneity and environmental factors such as erosion, sun coverage, rainfall, and pollen BIB002 ]. Given the limited resources for testing in most cases, exhaustive testing is not feasible. Fisher applied interaction testing so that every pair of factors affecting the yield was included exactly once. Since 1926, CT has been widely used in many other disciplines , with many experiences and empirical studies reported on the testing procedure and the application of CT. For software testing, Mandl first proposed using pairwise combinatorial coverage to test Ada compiler in 1985 and generated test sets using Orthogonal Latin Squares BIB001 ]. Brownlie et al. reported a case study that tested PMX/StarMAIL system using orthogonal array in 1992 and developed the OATS system to generate test cases BIB003 . The generated test cases can detect many errors that had never been detected previously. There are two key questions on the application of CT: (1) Is there a standard procedure we can follow in using CT? (2) What is the effectiveness of the CT procedure? The big issues are how to develop a practical procedure for CT and how to evaluate it in practice. Many studies have been published in this area. BIB008 used CT for performance evaluation, and found that a small number of test cases give information of which factors, or interactions between factors, affect the performance measure. This knowledge can be used as a basis for future investigations, where, for example, other environmental factors can be added and where factors which have no effect can be removed. The information gained help to reduce the number of test cases to be run in the real environment. BIB010 reported their experience in piloting and applying CT in projects. They designed a supporting process for using CT, and shared details on application of CT in feature testing of a mobile phone application. Their proposed testing process emphasized the first step of modeling, included test prioritization, expected output and failure diagnosis, but neglected the further testing and the evaluation steps. Many papers reported the experience of using CT in practice.  examined the correlation between τ -way coverage and achieved code coverage. BIB006 used CT to test the ground system for satellite communications. BIB004 reported how both the quality and efficiency of protocol testing were improved by CT, and BIB005 described a practical strategy for the pairwise testing of network interface. These researches provided good guidelines for using CT. CT has also been applied to other applications. BIB007 proposed a method which focused on user sequences of GUI objects and selections which collaborate, called Complete Interaction Sequences (CIS), that produce a desired response for the user. An empirical investigation of this method shows that a substantially reduced test set can still detect the defects in the GUI. Future research will prioritize testing related to the CIS testing for maximum benefit if testing time is limited.  generated test cases to test an email system with AETG. We also explored the configuration testing  ] and the browser compatibility testing BIB009 with CT.  described a CT strategy for concurrent programs. From the many prior studies, it appears that CT can be applied to many types of systems. However, each research team followed its own testing procedure. We believe better testing results can be obtained by following a more effective testing procedure. Some research effort should be devoted to this topic.