A survey of combinatorial testing <s> Summary <s> Orthogonal Latin squares—a new method for testing compilers—yields the informational equivalent of exhaustive testing at a fraction of the cost. The method has been used successfully in designing some of the tests in the Ada Compiler Validation Capability (ACVC) test suite. <s> BIB001 </s> A survey of combinatorial testing <s> Summary <s> Robust Testing is a systematic test method that uses the OATS (Orthogonal Array Testing System) tool to convert information from product requirements or specifications into a concise set of tests. We found Robust Testing more than twice as productive as conventional testing practices for testing PMX/StarMAIL Release 2.2, a complex private mail exchange (PMX) local area network (LAN)-based electronic mail product that works with several types and versions of LAN software, operating systems, and personal computers (PCs). Robust Testing required less time, enabling us to complete testing on schedule. It found more potentially user-affecting faults, and inspired greater confidence in the quality of the delivered product. <s> BIB002
Since BIB001 and BIB002 first used orthogonal arrays to test software, pairwise testing and then combinatorial testing were proposed and adopted. There have been more than ten research groups working in the CT field. We summarize the contribution of various research groups in Figure 10 , with more detail given in the Appendix. Combinatorial testing is a subject that crosses both computer science and mathematics. CT has attracted interest because we can combine mathematical methods and computational methods from computer science to find solutions to the testing problem. CT also presents many new and challenging problems. Looking at Figure 10 , we can conclude that test suite generation is the most active area, and the application of CT is the second most studied area in CT. Most of the research groups contributed in these two areas. As for the evolution of this research field, from Figure 11 we can see that the research of CT begins to expand to include modeling, metric, evaluation, failure diagnosis, constraint, and prioritization. But test generation and the application continue to attract interest in recent years. We observe a steep rise in the number of publications commencing from 2002, more than tripling the average annual publications from the previous six years. Thus, the research of CT is increasing, both in breadth and in depth, and CT has become a mature field where the large-scale deployment of this technology is possible.