Regression testing is an important part of quality control in both software and embedded products, where hardware is involved. It is also one of the most expensive and time consuming part of the product cycle. To improve the cost effectiveness of the development cycle and the regression testing, we use test case prioritisation and selection techniques to run more important test cases earlier in the testing process. In this paper, we consider a functional test case prioritisation with an access only to the version control of the codebase and regression history. Prioritisation is used to aid our test case selection, where we have chosen 5-25 (0.4 -2.0 of 1254) test cases to validate our method. The selection technique together with other prioritisation methods allows us to shape the current static, retest-all regression testing into a more resource managed regression testing framework. This framework will serve the agile way of working better and will allow us to allocate testing resources more wisely. This is a joint work with a large international Finnish company in an embedded industrial domain.
Another history-based method is to cluster test cases based on regression history alone or on codebase changes. For example, one can build a co-change matrix of the modified files and compute a singular value decomposition to cluster the files. Then combining the clusters with the information on test cases, the method yields a list of prioritised test cases @cite_9 . Using clustering methods, one may gain in-depth knowledge on how test cases behave, e.g., which test cases have passed and failed as a group even when that is not apparent by looking at the data alone. Downsides of clustering methods are, e.g., the required prior data and running the clustering algorithm in regular intervals to keep clusters up to date, which may become expensive in the long run.
@cite_9
2126905619 2135827170 2117799048 2136997225
During development and testing, changes made to a system to repair a detected fault can often inject a new fault into the code base. These injected faults may not be in the same files that were just changed, since the effects of a change in the code base can have ramifications in other parts of the system. We propose a methodology for determining the effect of a change and then prioritizing regression test cases by gathering software change records and analyzing them through singular value decomposition. This methodology generates clusters of files that historically tend to change together. Combining these clusters with test case information yields a matrix that can be multiplied by a vector representing a new system modification to create a prioritized list of test cases. We performed a post hoc case study using this technique with three minor releases of a software product at IBM. We found that our methodology suggested additional regression tests in 50 of test runs and that the highest-priority suggested test found an additional fault 60 of the time. Context: This paper presents an approach for selecting regression test cases in the context of large-scale database applications. We focus on a black-box (specification-based) approach, relying on classification tree models to model the input domain of the system under test (SUT), in order to obtain a more practical and scalable solution. We perform an experiment in an industrial setting where the SUT is a large database application in Norway's tax department. Objective: We investigate the use of similarity-based test case selection for supporting black box regression testing of database applications. We have developed a practical approach and tool (DART) for functional black-box regression testing of database applications. In order to make the regression test approach scalable for large database applications, we needed a test case selection strategy that reduces the test execution costs and analysis effort. We used classification tree models to partition the input domain of the SUT in order to then select test cases. Rather than selecting test cases at random from each partition, we incorporated a similarity-based test case selection, hypothesizing that it would yield a higher fault detection rate. Method: An experiment was conducted to determine which similarity-based selection algorithm was the most suitable in selecting test cases in large regression test suites, and whether similarity-based selection was a worthwhile and practical alternative to simpler solutions. Results: The results show that combining similarity measurement with partition-based test case selection, by using similarity-based test case selection within each partition, can provide improved fault detection rates over simpler solutions when specific conditions are met regarding the partitions. Conclusions: Under the conditions present in the experiment the improvements were marginal. However, a detailed analysis concludes that the similarity-based selection strategy should be applied when a large number of test cases are contained in each partition and there is significant variability within partitions. If these conditions are not present, incorporating similarity measures is not worthwhile, since the gain is negligible over a random selection within each partition. This paper presents an empirical comparison of four different techniques for filtering large test suites: test suite minimization, prioritization by additional coverage, cluster filtering with one-per-cluster sampling, and failure pursuit sampling. The first two techniques are based on selecting subsets that maximize code coverage as quickly as possible, while the latter two are based on analyzing the distribution of the tests' execution profiles. These techniques were compared with data sets obtained from three large subject programs: the GCC, Jikes, and javac compilers. The results indicate that distribution-based techniques can be as efficient or more efficient for revealing defects than coverage-based techniques, but that the two kinds of techniques are also complementary in the sense that they find different defects. Accordingly, some simple combinations of these techniques were evaluated for use in test case prioritization. The results indicate that these techniques can create more efficient prioritizations than those generated using prioritization by additional coverage. Test case prioritization techniques involve scheduling test cases for regression testing in an order that increases their effectiveness at meeting some performance goal. This is inefficient to re execute all the test cases in regression testing following the software modifications. Using information obtained from previous test case execution, prioritization techniques order the test cases for regression testing so that most beneficial are executed first thus allows an improved effectiveness of testing. One performance goal, rate of dependency detected among faults, measures how quickly dependency among faults are detected within the regression testing process. An improved rate of fault dependency can provide faster feedback on software and let developers start debugging on the severe faults that cause other faults to appear later. This paper presents the new metric for assessing rate of fault dependency detection and an algorithm to prioritize test cases. Using the new metric the effectiveness of this prioritization is shown comparing it with non-prioritized test case. Analysis proves that prioritized test cases are more effective in detecting dependency among faults.