Software Product Lines (SPLs) are inherently difficult to test due to the combinatorial explosion of the number of products to consider. To reduce the number of products to test, sampling techniques such as combinatorial interaction testing have been proposed. They usually start from a feature model and apply a coverage criterion (e.g. pairwise feature interaction or dissimilarity) to generate tractable, fault-finding, lists of configurations to be tested. Prioritization can also be used to sort generate such lists, optimizing coverage criteria or weights assigned to features. However, current sampling prioritization techniques barely take product behaviour into account. We explore how ideas of statistical testing, based on a usage model (a Markov chain), can be used to extract configurations of interest according to the likelihood of their executions. These executions are gathered in featured transition systems, compact representation of SPL behaviour. We discuss possible scenarios and give a prioritization procedure validated on a web-based learning management software.
There have been SPL test efforts to sample products for testing such as t-wise approaches (e.g. @cite_16 @cite_30 @cite_24 ). More recently sampling was combined with prioritization thanks to the addition of weights on feature models and the definition of multiple objectives @cite_6 @cite_9 . However, these approaches do not consider SPL behavior in their analyses.
@cite_30 @cite_9 @cite_6 @cite_24 @cite_16
2071160808 2105438268 1977497804 2061158652
Software Product Lines (SPL) are difficult to validate due to combinatorics induced by variability, which in turn leads to combinatorial explosion of the number of derivable products. Exhaustive testing in such a large products space is hardly feasible. Hence, one possible option is to test SPLs by generating test configurations that cover all possible t feature interactions (t-wise). It dramatically reduces the number of test products while ensuring reasonable SPL coverage. In this paper, we report our experience on applying t-wise techniques for SPL with two independent toolsets developed by the authors. One focuses on generality and splits the generation problem according to strategies. The other emphasizes providing efficient generation. To evaluate the respective merits of the approaches, measures such as the number of generated test configurations and the similarity between them are provided. By applying these measures, we were able to derive useful insights for pairwise and t-wise testing of product lines. Context: Software product lines (SPL) are used in industry to achieve more efficient software development. However, the testing side of SPL is underdeveloped. Objective: This study aims at surveying existing research on SPL testing in order to identify useful approaches and needs for future research. Method: A systematic mapping study is launched to find as much literature as possible, and the 64 papers found are classified with respect to focus, research type and contribution type. Results: A majority of the papers are of proposal research types (64 ). System testing is the largest group with respect to research focus (40 ), followed by management (23 ). Method contributions are in majority. Conclusions: More validation and evaluation research is needed to provide a better foundation for SPL testing. Testing software product lines (SPLs) is very challenging due to a high degree of variability leading to an enormous number of possible products. The vast majority of today's testing approaches for SPLs validate products individually using different kinds of reuse techniques for testing. Because of their reusability and adaptability capabilities, model-based approaches are suitable to describe variability and are therefore frequently used for implementation and testing purposes of SPLs. Due to the enormous number of possible products, individual product testing becomes more and more infeasible. Pairwise testing offers one possibility to test a subset of all possible products. However, according to the best of our knowledge, there is no contribution discussing and rating this approach in the SPL context. In this contribution, we provide a mapping between feature models describing the common and variable parts of an SPL and a reusable test model in the form of statecharts. Thereby, we interrelate feature model-based coverage criteria and test model-based coverage criteria such as control and data flow coverage and are therefore able to discuss the potentials and limitations of pairwise testing. We pay particular attention to test requirements for feature interactions constituting a major challenge in SPL engineering. We give a concise definition of feature dependencies and feature interactions from a testing point of view, and we discuss adequacy criteria for SPL coverage under pairwise feature interaction testing and give a generalization to the T-wise case. The concept and implementation of our approach are evaluated by means of a case study from the automotive domain. We study the problem of optimal sub-Nyquist sampling for perfect reconstruction of multiband signals. The signals are assumed to have a known spectral support spl Fscr that does not tile under translation. Such signals admit perfect reconstruction from periodic nonuniform sampling at rates approaching Landau's (1967) lower bound equal to the measure of spl Fscr . For signals with sparse spl Fscr , this rate can be much smaller than the Nyquist rate. Unfortunately the reduced sampling rates afforded by this scheme can be accompanied by increased error sensitivity. In a previous study, we derived bounds on the error due to mismodeling and sample additive noise. Adopting these bounds as performance measures, we consider the problems of optimizing the reconstruction sections of the system, choosing the optimal base sampling rate, and designing the nonuniform sampling pattern. We find that optimizing these parameters can improve system performance significantly. Furthermore, uniform sampling is optimal for signals with spl Fscr that tiles under translation. For signals with nontiling spl Fscr , which are not amenable to efficient uniform sampling, the results reveal increased error sensitivities with sub-Nyquist sampling. However, these can be controlled by optimal design, demonstrating the potential for practical multifold reductions in sampling rate.