BACKGROUND OF THE INVENTION

Field of the Invention

The present invention is directed in general to field of information processing. In one aspect, the present invention relates generally to software testing.

Description of the Related Art

Computer programmers, developers, coders and/or software engineers write, test, debug, and maintain computer software or code instructions, called computer programs, which computers must follow to perform their functions. When testing a computer program, a collection of test cases (collectively referred to as a test suite) is applied to test the computer program to show that it has some specified set of behaviors. A test suite often contains detailed instructions or goals for each collection of test cases and information on the system configuration to be used during testing. However, the quality of a test suite must be evaluated to determine not only the code coverage for the test suite (e.g., what percentage of the execution paths in the computer program have been exercised during automated testing), but also the test accuracy and user case completeness of the test suite. One approach for improving the adequacy of tests to better identify defects in the code is through mutation testing (a.k.a., mutation analysis or program mutation) whereby new software tests are designed through a process where the program code is modified or “mutated” in predetermined ways and then evaluated by a test suite to determine if the test suite detects and rejects (or “kills”) the mutants. For example, mutations may be generated by using well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as dividing each expression by zero). Test suites are measured by the percentage of mutants that they kill, and new tests can be designed to kill any surviving mutants. However, while conventional mutation test tools (such as the PIT mutation testing system) can identify which mutations survived, they provide only block coverage reports, making it very difficult for a developer/tester to understand which specific part of the test suite failed and should be modified to kill the surviving mutant(s). Additional drawbacks with conventional mutation test tools is that they have significant rates of false positive reporting (e.g., 20-50%), require different plugins for different build systems (e.g., pitest maven plugin, pitest ant plugin, etc.), have limited test failure tolerance, and/or limited stability (e.g., buggy). As seen from the foregoing, the existing solutions for performing mutation coverage testing are extremely difficult at a practical level by virtue of the difficulty in using the mutation test system to accurately and efficiently identify and upgrade test suites.

SUMMARY OF THE INVENTION

A system, apparatus, computer program product, and methodology are described for efficiently identifying a test case and/or test class in a test suite that is responsible for a survived mutation to help a program developer/tester to modify the old test case/class or write a new test case in the correct test class to kill the survived mutation. Selected embodiments of the present disclosure automatically apply specified test case identification rules to a list of tests which cover a surviving mutation to identify one or more test cases or classes which are likely responsible for missing the mutation. After generating source code mutations and applying a test suite, each surviving mutation is analyzed to automatically identify which test case/method or test class is responsible for missing the mutation. In a first step of the analysis, the process assembles a list (L) of all test case(s)/method(s) which cover a given mutation, and if the list (L) contains a single test method/case, then the test method/case on the list is identified or blamed as being responsible for missing the mutation. In a second step of the analysis, the process then filters the list (L) to remove any test method(s)/case(s) that killed another mutation, and if the resulting filtered list contains a single test method/case, the that test method/case is identified or blamed as being responsible for missing the mutation. In a third step of the analysis, the process identifies a name (X) for the mutated method, and then searches through the filtered list to identify any test case/method which includes the name X as part of its name, in which case that matching test case/method is identified or blamed as being responsible for missing the surviving mutation. And if no specific test method is identified, then the analysis moves to the test class level by first determining if all of the test methods on the filtered list belong to a single test class, in which case the test class is identified/blamed as being responsible. If not, then the process searches through a list (L2) of distinct test class names for the test case(s)/method(s) remaining on the filtered list to identify any class which includes the name X as part of its name, in which case that matching test class is identified or blamed as being responsible for missing the surviving mutation. And if there is no test class which contains the name X, then the first test class on the test class list (L2) is identified or blamed as being responsible for missing the surviving mutation.

The objects, advantages and other novel features of the present invention will be apparent from the following detailed description when read in conjunction with the appended claims and attached drawings.

