Regression testing is an important part of quality control in both software and embedded products, where hardware is involved. It is also one of the most expensive and time consuming part of the product cycle. To improve the cost effectiveness of the development cycle and the regression testing, we use test case prioritisation and selection techniques to run more important test cases earlier in the testing process. In this paper, we consider a functional test case prioritisation with an access only to the version control of the codebase and regression history. Prioritisation is used to aid our test case selection, where we have chosen 5-25 (0.4 -2.0 of 1254) test cases to validate our method. The selection technique together with other prioritisation methods allows us to shape the current static, retest-all regression testing into a more resource managed regression testing framework. This framework will serve the agile way of working better and will allow us to allocate testing resources more wisely. This is a joint work with a large international Finnish company in an embedded industrial domain.
In agile software development one wants to select test cases based on code changes and regression testing @cite_0 to catch the flipping test cases and to decrease the feedback loop time. These methods alone are not enough, because they do not catch test cases that keep on failing. Therefore, we need, e.g., a history-based test case prioritisation (HBTP) technique as well, where test cases are prioritised based on their failure rate in the regression history. Basically, if a given test case has failed then it will most likely fail again. One technique @cite_6 gives weight to test cases depending on how many builds there have been since their last failure.
@cite_0 @cite_6
2098807207 2889037615
Due to changes in the development practices at Axis Communications, towards continuous integration, faster regression testing feedback is needed. The current automated regression test suite takes approximately seven hours to run which prevents developers from integrating code changes several times a day as preferred. Therefore we want to implement a highly selective yet accurate regression testing strategy. Traditional code coverage based techniques are not applicable due to the size and complexity of the software under test. Instead we decided to select tests based on regression test history. We developed a tool, the Difference Engine, which parses and analyzes results from previous test runs and outputs regression test recommendations. The Difference Engine correlates code and test cases at package level and recommends test cases that are strongly correlated to recently changed packages. We evaluated the technique with respect to correctness, precision, recall and efficiency. Our results are promising. On average the tool manages to identify 80 of the relevant tests while recommending only 4 of the test cases in the full regression test suite. Abstract Two heuristics namely diversity-based (DBTP) and history-based test prioritization (HBTP) have been separately proposed in the literature. Yet, their combination has not been widely studied in continuous integration (CI) environments. The objective of this study is to catch regression faults earlier, allowing developers to integrate and verify their changes more frequently and continuously. To achieve this, we investigated six open-source projects, each of which included several builds over a large time period. Findings indicate that previous failure knowledge seems to have strong predictive power in CI environments and can be used to effectively prioritize tests. HBTP does not necessarily need to have large data, and its effectiveness improves to a certain degree with larger history interval. DBTP can be used effectively during the early stages, when no historical data is available, and also combined with HBTP to improve its effectiveness. Among the investigated techniques, we found that history-based diversity using NCD Multiset is superior in terms of effectiveness but comes with relatively higher overhead in terms of method execution time. Test prioritization in CI environments can be effectively performed with negligible investment using previous failure knowledge, and its effectiveness can be further improved by considering dissimilarities among the tests.