This paper proposes a novel column generation framework for combinatorial software testing. In particular, it combines Mathematical Programming and Constraint Programming in a hybrid decomposition to generate covering arrays. The approach allows generating parameterized test cases with coverage guarantees between parameter interactions of a given application. Compared to exhaustive testing, combinatorial test case generation reduces the number of tests to run significantly. Our column generation algorithm is generic and can accommodate mixed coverage arrays over heterogeneous alphabets. The algorithm is realized in practice as a cloud service and recognized as one of the five winners of the company-wide cloud application challenge at Oracle. The service is currently helping software developers from a range of different product teams in their testing efforts while exposing declarative constraint models and hybrid optimization techniques to a broader audience.
There is a vast literature on combinatorial software testing and the problem of determining the minimum size covering arrays has been studied extensively. We discuss only parts of this rich literature due to space limit. A comprehensive survey can be found in @cite_6 @cite_3 . Unlike our approach, some of this work are specifically designed for binary uniform domains, and or for pairwise coverage @cite_8 @cite_2 @cite_9 . The closest to our approach are declarative models that take advantage of constraint solvers. Formulations based constraint programming are given in @cite_1 @cite_4 @cite_9 . Complementary to those are heuristic approaches such as AETG @cite_0 , IPO @cite_11 . Our work falls in between complete and incomplete methods. It uses a hybrid decomposition that brings together exact MP and CP formulations. While the overall approach is not exact, it still takes advantage of declarative models instead of implementing heuristic solutions from scratch.
@cite_4 @cite_8 @cite_9 @cite_1 @cite_3 @cite_6 @cite_0 @cite_2 @cite_11
131057022 2094157810 964070544 2001663593
Abstract : Testing is an expensive but essential part of any software project. Having the right methods to detect faults is a primary factor for success in the software industry. Component based systems are problematic because they are prone to unexpected interaction faults, yet these may be left undetected by traditional testing techniques. In all but the smallest of systems, it is not possible to test every component interaction. One can use a reduced test suite that guarantees to include a defined subset of interactions instead. A well studied combinatorial object, the covering array, can be used to achieve this goal. Constructing covering arrays for a specific software system is not always simple and the resulting object may not closely mirror the real test environment. Not only are new methods for building covering arrays needed, but new tools to support these are required as well. Our aim is to develop methods for building smaller test suites that provide stronger interaction coverage, while retaining the flexibility required in a practical test suite. We combine ideas from combinatorial design theory, computational search, statistical design of experiments and software engineering. We begin with a description of a framework for greedy algorithms that has formed the basis for several published methods and a widely used commercial tool. We compare this with a meta-heuristic search algorithm, simulated annealing. The results suggest that simulated annealing is more effective at finding smaller test suites, and in some cases improves on combinatorial methods as well. We then develop a mathematical model for variable strength interaction testing. This allows us to balance the cost and the time of testing by targeting individual subsets of components. We present construction techniques using meta-heuristic search and provide the first known bounds for objects of this type. We end by presenting some new cut-and-paste techniques that merge recursive combinatorial constructions A trend in software testing is reducing the size of a test suite while preserving its overall quality. Given a test suite and a set of requirements covered by the suite, test suite reduction aims at selecting a subset of test cases that cover the same set of requirements. Even though this problem has received considerable attention, finding the smallest subset of test cases is still challenging and commonly-used approaches address this problem only with approximated solutions. When executing a single test case requires much manual effort (e.g., hours of preparation), finding the minimal subset is needed to reduce the testing costs. In this paper, we introduce a radically new approach to test suite reduction, called FLOWER, based on a search among network maximum flows. From a given test suite and the requirements covered by the suite, FLOWER forms a flow network (with specific constraints) that is then traversed to find its maximum flows. FLOWER leverages the Ford-Fulkerson method to compute maximum flows and Constraint Programming techniques to search among optimal flows. FLOWER is an exact method that computes a minimum-sized test suite, preserving the coverage of requirements. The experimental results show that FLOWER outperforms a non-optimized implementation of the Integer Linear Programming approach by 15-3000 times in terms of the time needed to find an optimal solution, and a simple greedy approach by 5-15 in terms of the size of reduced test suite. New approach is used to generate combinatorial test suitesFuzzy-based strategy developed to generate the test cases using adaptive techniqueIt is a new approach in developing Artificial Intelligent and Expert systemsThe strategy proves its efficiency, performance compared to its counterpartsThe strategy proves its effectiveness also through the case study Recent research activities have demonstrated the effective application of combinatorial optimization in different areas, especially in software testing. Covering array (CA) has been introduced as a representation of the combinations in one complete set. CAλ(N; t, k, v) is an N?×?k array in which each t-tuple for an N × t sub array occurs at least λ times, where t is the combination strength, k is the number of components (factors), and v is the number of symbols for each component (levels). Generating an optimized covering array for a specific number of k and v is difficult because the problem is a non-deterministic polynomial-time hard computational one. To address this issue, many relevant strategies have been developed, including stochastic population-based algorithms. This paper presents a new and effective approach for constructing efficient covering arrays through fuzzy-based, adaptive particle swarm optimization (PSO). With this approach, efficient covering arrays have been constructed and the performance of PSO has been improved for this type of application. To measure the effectiveness of the technique, an empirical study is conducted on a software system. The technique proves its effectiveness through the conducted case study. par>We prove some non-approximability results for restrictions of basic combinatorial optimization problems to instances of bounded “degreeror bounded “width.” Specifically: We prove that the Max 3SAT problem on instances where each variable occurs in at most B clauses, is hard to approximate to within a factor @math , unless @math . H stad [18] proved that the problem is approximable to within a factor @math in polynomial time, and that is hard to approximate to within a factor @math . Our result uses a new randomized reduction from general instances of Max 3SAT to bounded-occurrences instances. The randomized reduction applies to other Max SNP problems as well. We observe that the Set Cover problem on instances where each set has size at most B is hard to approximate to within a factor @math unless @math . The result follows from an appropriate setting of parameters in Feige's reduction [11]. This is essentially tight in light of the existence of @math -approximate algorithms [20, 23, 9] We present a new PCP construction, based on applying parallel repetition to the inner verifier,'' and we provide a tight analysis for it. Using the new construction, and some modifications to known reductions from PCP to Hitting Set, we prove that Hitting Set with sets of size B is hard to approximate to within a factor @math . The problem can be approximated to within a factor B [19], and it is the Vertex Cover problem for B =2. The relationship between hardness of approximation and set size seems to have not been explored before. We observe that the Independent Set problem on graphs having degree at most B is hard to approximate to within a factor @math , unless P = NP . This follows from a comination of results by Clementi and Trevisan [28] and Reingold, Vadhan and Wigderson [27]. It had been observed that the problem is hard to approximate to within a factor @math unless P = NP [1]. An algorithm achieving factor @math is also known [21, 2, 30, 16 .