As Software Product Lines (SPLs) are becoming a more pervasive development practice, their effective testing is becoming a more important concern. In the past few years many SPL testing approaches have been proposed, among them, are those that support Combinatorial Interaction Testing (CIT) whose premise is to select a group of products where faults, due to feature interactions, are more likely to occur. Many CIT techniques for SPL testing have been put forward; however, no systematic and comprehensive comparison among them has been performed. To achieve such goal two items are important: a common benchmark of feature models, and an adequate comparison framework. In this research-in-progress paper, we propose 19 feature models as the base of a benchmark, which we apply to three different techniques in order to analyze the comparison framework proposed by We identify the shortcomings of this framework and elaborate alternatives for further study.
There exists substantial literature on SPL testing @cite_8 @cite_32 @cite_10 @cite_27 . However, to the best of our knowledge there are neither benchmarks nor frameworks for comparing approaches. In the area of Search-Based Software Engineering a major research focus has been software testing @cite_17 @cite_18 , where there exists a plethora of articles that compare testing algorithms using different metrics. For example, Mansour @cite_11 compare five algorithms for regression testing using eight different metrics (including quantitative and qualitative criteria). Similarly, @cite_28 compare different metrics implemented as fitness functions to solve the problem of test input generation. To the best of our knowledge, in the literature on test case generation there is no well-known comparison framework for the research and practitioner community to use. Researchers usually apply their methods to open source programs and compute some metrics directly such as the success rate, the number of test cases and performance. The closest to a common comparison framework we could trace is the work of Rothermel and Harrold @cite_6 where they propose a framework for regression testing.
@cite_18 @cite_11 @cite_8 @cite_28 @cite_32 @cite_6 @cite_27 @cite_10 @cite_17
2025003613 2105438268 2039380967 2107932342 1974481336 1977321274
In the maintenance phase, the regression test selection problem refers to selecting test cases from the initial suite of test cases used in the development phase. In this paper, we empirically compare five representative regression test selection algorithms, which include: Simulated Annealing, Reduction, Slicing, Dataflow, and Firewall algorithms. The comparison is based on eight quantitative and qualitative criteria. These criteria are: number of selected test cases, execution time, precision, inclusiveness, preprocessing requirements, type of maintenance, level of testing, and type of approach. The empirical results show that the five algorithms can be used for different requirements of regression testing. For example the Simulated Annealing algorithm can be used for emergency non-safety-critical maintenance situations with a large number of small modifications. Context: Software product lines (SPL) are used in industry to achieve more efficient software development. However, the testing side of SPL is underdeveloped. Objective: This study aims at surveying existing research on SPL testing in order to identify useful approaches and needs for future research. Method: A systematic mapping study is launched to find as much literature as possible, and the 64 papers found are classified with respect to focus, research type and contribution type. Results: A majority of the papers are of proposal research types (64 ). System testing is the largest group with respect to research focus (40 ), followed by management (23 ). Method contributions are in majority. Conclusions: More validation and evaluation research is needed to provide a better foundation for SPL testing. Evolutionary algorithms are among the metaheuristic search methods that have been applied to the structural test data generation problem. Fitness evaluation methods play an important role in the performance of evolutionary algorithms and various methods have been devised for this problem. In this paper, we propose a new fitness evaluation method based on pairwise sequence comparison also used in bioinformatics. Our preliminary study shows that this method is easy to implement and produces promising results. Regression testing is a necessary but expensive activity aimed at showing that code has not been adversely affected by changes. A selective approach to regression testing attempts to reuse tests from an existing test suite to test a modified program. This paper outlines issues relevant to selective retest approaches, and presents a framework within which such approaches can be evaluated. This framework is then used to evaluate and compare existing selective retest algorithms. The evaluation reveals strengths and weaknesses of existing methods, and highlights problems that future work in this area should address. > The software product line engineering strategy enables the achievement of significant improvements in quality through reuse of carefully crafted software assets across multiple products. However, high levels of quality in the software product line assets, which are used to create products, must be accompanied by effective and efficient test strategies for the products in the software product line. The goal of this study is to understand which strategies for testing products in software product lines have been reported in the literature, enabling discussions on the significant issues, and also pointing out further research directions. A systematic literature review was carried out that identified two hundred seventy-three papers, published from the years 1998 and early in 2012. From such a set of papers, a systematic selection resulted in forty-one relevant papers. The analysis of the reported strategies comprised two important aspects: the selection of products for testing, and the actual test of products. The findings showed a range of strategies, dealing with both aspects, but few empirical evaluations of their effectiveness have been performed, which limits the inferences that can be drawn. In the past five years there has been a dramatic increase in work on Search-Based Software Engineering (SBSE), an approach to Software Engineering (SE) in which Search-Based Optimization (SBO) algorithms are used to address problems in SE. SBSE has been applied to problems throughout the SE lifecycle, from requirements and project planning to maintenance and reengineering. The approach is attractive because it offers a suite of adaptive automated and semiautomated solutions in situations typified by large complex problem spaces with multiple competing and conflicting objectives. This article1 provides a review and classification of literature on SBSE. The work identifies research trends and relationships between the techniques applied and the applications to which they have been applied and highlights gaps in the literature and avenues for further research.