Regression testing is an important part of quality control in both software and embedded products, where hardware is involved. It is also one of the most expensive and time consuming part of the product cycle. To improve the cost effectiveness of the development cycle and the regression testing, we use test case prioritisation and selection techniques to run more important test cases earlier in the testing process. In this paper, we consider a functional test case prioritisation with an access only to the version control of the codebase and regression history. Prioritisation is used to aid our test case selection, where we have chosen 5-25 (0.4 -2.0 of 1254) test cases to validate our method. The selection technique together with other prioritisation methods allows us to shape the current static, retest-all regression testing into a more resource managed regression testing framework. This framework will serve the agile way of working better and will allow us to allocate testing resources more wisely. This is a joint work with a large international Finnish company in an embedded industrial domain.
In agile software development one wants to select test cases based on code changes and regression testing @cite_0 to catch the flipping test cases and to decrease the feedback loop time. These methods alone are not enough, because they do not catch test cases that keep on failing. Therefore, we need, e.g., a history-based test case prioritisation (HBTP) technique as well, where test cases are prioritised based on their failure rate in the regression history. Basically, if a given test case has failed then it will most likely fail again. One technique @cite_6 gives weight to test cases depending on how many builds there have been since their last failure.
@cite_0 @cite_6
2889037615 2136997225 1984494831 1518403495
Abstract Two heuristics namely diversity-based (DBTP) and history-based test prioritization (HBTP) have been separately proposed in the literature. Yet, their combination has not been widely studied in continuous integration (CI) environments. The objective of this study is to catch regression faults earlier, allowing developers to integrate and verify their changes more frequently and continuously. To achieve this, we investigated six open-source projects, each of which included several builds over a large time period. Findings indicate that previous failure knowledge seems to have strong predictive power in CI environments and can be used to effectively prioritize tests. HBTP does not necessarily need to have large data, and its effectiveness improves to a certain degree with larger history interval. DBTP can be used effectively during the early stages, when no historical data is available, and also combined with HBTP to improve its effectiveness. Among the investigated techniques, we found that history-based diversity using NCD Multiset is superior in terms of effectiveness but comes with relatively higher overhead in terms of method execution time. Test prioritization in CI environments can be effectively performed with negligible investment using previous failure knowledge, and its effectiveness can be further improved by considering dissimilarities among the tests. Test case prioritization techniques involve scheduling test cases for regression testing in an order that increases their effectiveness at meeting some performance goal. This is inefficient to re execute all the test cases in regression testing following the software modifications. Using information obtained from previous test case execution, prioritization techniques order the test cases for regression testing so that most beneficial are executed first thus allows an improved effectiveness of testing. One performance goal, rate of dependency detected among faults, measures how quickly dependency among faults are detected within the regression testing process. An improved rate of fault dependency can provide faster feedback on software and let developers start debugging on the severe faults that cause other faults to appear later. This paper presents the new metric for assessing rate of fault dependency detection and an algorithm to prioritize test cases. Using the new metric the effectiveness of this prioritization is shown comparing it with non-prioritized test case. Analysis proves that prioritized test cases are more effective in detecting dependency among faults. Test case prioritization involves scheduling test cases in an order that increases the effectiveness in achieving some performance goals. One of the most important performance goals is the rate of fault detection. Test cases should run in an order that increases the possibility of fault detection and also that detects the most severe faults at the earliest in its testing life cycle. In this paper, we propose to put forth a model for system level test case prioritization (TCP) from software requirement specification to improve user satisfaction with quality software that can also be cost effective and to improve the rate of severe fault detection. The proposed model prioritizes the system test cases based on the six factors: customer priority, changes in requirement, implementation complexity, completeness, traceability and fault impact. The proposed prioritization technique is validated with two different validation techniques and is experimented in three phases with student projects and two sets of industrial projects and the results show convincingly that the proposed prioritization technique improves the rate of severe fault detection. Test case prioritization techniques have been shown to be beneficial for improving regression-testing activities. With prioritization, the rate of fault detection is improved, thus allowing testers to detect faults earlier in the system-testing phase. Most of the prioritization techniques to date have been code coverage-based. These techniques may treat all faults equally. We build upon prior test case prioritization research with two main goals: (1) to improve user-perceived software quality in a cost effective way by considering potential defect severity and (2) to improve the rate of detection of severe faults during system-level testing of new code and regression testing of existing code. We present a value-driven approach to system-level test case prioritization called the prioritization of requirements for test (PORT). PORT prioritizes system test cases based upon four factors: requirements volatility, customer priority, implementation complexity, and fault proneness of the requirements. We conducted a PORT case study on four projects developed by students in advanced graduate software testing class. Our results show that PORT prioritization at the system level improves the rate of detection of severe faults. Additionally, customer priority was shown to be one of the most important prioritization factors contributing to the improved rate of fault detection.