In software, there are the errors anticipated at specification and design time, those encountered at development and testing time, and those that happen in production mode yet never anticipated. In this paper, we aim at reasoning on the ability of software to correctly handle unanticipated exceptions. We propose an algorithm, called short-circuit testing, which injects exceptions during test suite execution so as to simulate unanticipated errors. This algorithm collects data that is used as input for verifying two formal exception contracts that capture two resilience properties. Our evaluation on 9 test suites, with 78 line coverage in average, analyzes 241 executed catch blocks, shows that 101 of them expose resilience properties and that 84 can be transformed to be more resilient.
Ghosh and Kelly @cite_16 did a special kind of mutation testing for improving test suites. Their fault model comprises abend'' faults: abnormal ending of catch blocks. It is similar to short-circuiting. We use the term short-circuit'' since it is a precise metaphor of what happens. In comparison, the term abend'' encompasses many more kinds of faults. In our paper, we claim that the new observed behavior resulting from short-circuit testing should not be considered as mutants to be killed. Actually we claim the opposite: short-circuiting should remain undetected for sake of source independence and pure resilience.
@cite_16
1950030762 1963679610 2471516951 2135937266
Fault localization methods seek to identify faulty program statements based on the information provided by the failing and passing test executions. Spectrum-based methods are among the most popular ones and assist programmers by assigning suspiciousness values on program statements according to their probability of being faulty. This paper proposes Met allaxis, a fault localization approach based on mutation analysis. The innovative part of Met allaxis is that it uses mutants and links them with the faulty program places. Thus, mutants that are killed mostly by failing tests provide a good indication about the location of a fault. Experimentation using Met allaxis suggests that it is significantly more effective than statement-based approaches. This is true even in the case where mutation cost-reduction techniques, such as mutant sampling, are facilitated. Additionally, results from a controlled experiment show that the use of mutation as a testing technique provides benefits to the fault localization process. Therefore, fault localization is significantly improved by using mutation-based tests instead of block-based or branch-based test suites. Finally, evidence in support of the methods' scalability is also given. Copyright © 2013 John Wiley & Sons, Ltd. Many fault localization techniques operate by crosscutting coverage information of passed and failed test executions. Generally, their accuracy depends on the utilized coverage elements and on the selected test cases. This paper proposes a novel fault localization method using mutation and investigates its accuracy when using classical test selection criteria such as mutation, branch and block. A controlled experiment shows that (1) the mutation based approach is quite effective at identifying "unknown" faulty program statements. Additionally, the experimental results reveal (2) that the mutation-based test suites are significantly more effective at supporting fault localization than block or branch-based test suites. Further, (3) evidence in support of facilitating mutation alternatives, such as mutant sampling, in order to diminish mutation overheads is also given. We introduce mutation-aware fault prediction, which leverages additional guidance from metrics constructed in terms of mutants and the test cases that cover and detect them. We report the results of 12 sets of experiments, applying 4 different predictive modelling techniques to 3 large real-world systems (both open and closed source). The results show that our proposal can significantly (p ≤ 0.05) improve fault prediction performance. Moreover, mutation-based metrics lie in the top 5 most frequently relied upon fault predictors in 10 of the 12 sets of experiments, and provide the majority of the top ten fault predictors in 9 of the 12 sets of experiments. A good test suite is one that detects real faults. Because the set of faults in a program is usually unknowable, this definition is not useful to practitioners who are creating test suites, nor to researchers who are creating and evaluating tools that generate test suites. In place of real faults, testing research often uses mutants, which are artificial faults -- each one a simple syntactic variation -- that are systematically seeded throughout the program under test. Mutation analysis is appealing because large numbers of mutants can be automatically-generated and used to compensate for low quantities or the absence of known real faults. Unfortunately, there is little experimental evidence to support the use of mutants as a replacement for real faults. This paper investigates whether mutants are indeed a valid substitute for real faults, i.e., whether a test suite’s ability to detect mutants is correlated with its ability to detect real faults that developers have fixed. Unlike prior studies, these investigations also explicitly consider the conflating effects of code coverage on the mutant detection rate. Our experiments used 357 real faults in 5 open-source applications that comprise a total of 321,000 lines of code. Furthermore, our experiments used both developer-written and automatically-generated test suites. The results show a statistically significant correlation between mutant detection and real fault detection, independently of code coverage. The results also give concrete suggestions on how to improve mutation analysis and reveal some inherent limitations.