Common test generators fall into two categories. Generating test inputs at the unit level is fast, but can lead to false alarms when a function is called with inputs that would not occur in a system context. If a generated input at the system level causes a failure, this is a true alarm, as the input could also have come from the user or a third party; but system testing is much slower. In this paper, we introduce the concept of a test generation bridge, which joins the accuracy of system testing with the speed of unit testing. A Test Generation Bridge allows to combine an arbitrary system test generator with an arbitrary unit test generator. It does so by carving parameterized unit tests from system (test) executions. These unit tests run in a context recorded from the system test, but individual parameters are left free for the unit test generator to systematically explore. This allows symbolic test generators such as KLEE to operate on individual functions in the recorded system context. If the test generator detects a failure, we lift the failure-inducing parameter back to the system input; if the failure can be reproduced at the system level, it is reported as a true alarm. Our BASILISK prototype can extract and test units out of complex systems such as a Web Python SQLite C stack; in its evaluation, it achieves a higher coverage than a state-of-the-art system test generator.
A number of related works has focused on obtaining parameterized unit tests by starting from existing or generated unit tests. Retrofitting of unit tests @cite_14 is an approach where existing unit tests are converted to parameterized unit tests, by identifying inputs and converting them to parameters. The technique of Fraser and Zeller @cite_4 starts from concrete inputs and results, using test generation and mutation to systematically generalize the pre- and postconditions of existing unit tests. The recently presented tool @cite_7 generalizes over a set of related unit tests to extract common procedures and unique parameters to obtain parametrized unit tests. In contrast to all these works, our technique carves parameterized unit tests directly out of a given run, identifying those values as parameters that are present in system input.
@cite_14 @cite_4 @cite_7
1529754292 2103275664 2811006300
Recent advances in software testing introduced parameterized unit tests (PUT), which accept parameters, unlike conventional unit tests (CUT), which do not accept parameters. PUTs are more beneficial than CUTs with regards to fault-detection capability, since PUTs help describe the behaviors of methods under test for all test arguments. In general, existing applications often include manually written CUTs. With the existence of these CUTs, natural questions that arise are whether these CUTs can be retrofitted as PUTs to leverage the benefits of PUTs, and what are the cost and benefits involved in retrofitting CUTs as PUTs. To address these questions, in this paper, we conduct an empirical study to investigate whether existing CUTs can be retrofitted as PUTs with feasible effort and achieve the benefits of PUTs in terms of additional fault-detection capability and code coverage. We also propose a methodology, called test generalization, that helps in systematically retrofitting existing CUTs as PUTs. Our results on three real-world open-source applications (≈ 4.6 KLOC) show that the retrofitted PUTs detect 19 new defects that are not detected by existing CUTs, and also increase branch coverage by 4 on average (with maximum increase of 52 for one class under test and 10 for one application under analysis) with feasible effort. State-of-the art techniques for automated test generation focus on generating executions that cover program behavior. As they do not generate oracles, it is up to the developer to figure out what a test does and how to check the correctness of the observed behavior. In this paper, we present an approach to generate parameterized unit tests---unit tests containing symbolic pre- and postconditions characterizing test input and test result. Starting from concrete inputs and results, we use test generation and mutation to systematically generalize pre- and postconditions while simplifying the computation steps. Evaluated on five open source libraries, the generated parameterized unit tests are (a) more expressive, characterizing general rather than concrete behavior; (b) need fewer computation steps, making them easier to understand; and (c) achieve a higher coverage than regular unit tests. Parameterized unit testing is a promising technique for developers to use to facilitate the understanding of test codes. However, as a practical issue, developers might not have sufficient resources to implement parameterized unit tests (PUTs) corresponding to a vast number of closed unit tests (CUTs) in long-term software projects. Although a technique for retrofitting CUTs into PUTs was proposed, it imposes a laborious task on developers to promote parameters in CUTs. In this study, we propose a fully automated CUT-PUT retrofitting technique (called AutoPUT), which detects similar CUTs as PUT candidates by comparing their code structures. It then identifies common procedures and unique parameters to generate PUTs without degradation in terms of code coverage as compared with original CUTs. From the results of our case-study experiments on open-sourced software projects, we found that AutoPUT fully automatically generated 204 PUTs in 8.5 hours. We concluded that AutoPUT can help developers maintain test suites for building reliable software.