At ICSE'2013, there was the first session ever dedicated to automatic program repair. In this session, presented PAR, a novel template-based approach for fixing Java bugs. We strongly disagree with key points of this paper. Our critical review has two goals. First, we aim at explaining why we disagree with Kim and colleagues and why the reasons behind this disagreement are important for research on automatic software repair in general. Second, we aim at contributing to the field with a clarification of the essential ideas behind automatic software repair. In particular we discuss the main evaluation criteria of automatic software repair: understandability, correctness and completeness. We show that depending on how one sets up the repair scenario, the evaluation goals may be contradictory. Eventually, we discuss the nature of fix acceptability and its relation to the notion of software correctness.
Having sound evaluation methods is essential for science. In many fields, different fallacies have been described (e.g. @cite_13 in medicine). In software engineering, many authors discussed potential fallacies, such as Glass in his book @cite_1 . Recently, Bird and colleagues @cite_32 have extensively discussed the biases of datasets used in bug predication research. Posnett, Filkov and Devanbu @cite_31 have published a paper on the presence of ecological fallacies in empirical software engineering research (focusing on sample size, zonation, and class imbalance). Both papers discuss the intimate relation between the dataset construction and the conclusiveness of the evaluation.
@cite_1 @cite_31 @cite_13 @cite_32
2165516939 2518510348 2149783794 2886424491
Philosophers of science have given up on the quest for a silver bullet to put an end to all pseudoscience, as such a neat formal criterion to separate good science from its contenders has proven elusive. In the literature on critical thinking and in some philosophical quarters, however, this search for silver bullets lives on in the taxonomies of fallacies. The attractive idea is to have a handy list of abstract definitions or argumentation schemes, on the basis of which one can identify bad or invalid types of reasoning, abstracting away from the specific content and dialectical context. Such shortcuts for debunking arguments are tempting, but alas, the promise is hardly if ever fulfilled. Different strands of research on the pragmatics of argumentation, probabilistic reasoning and ecological rationality have shown that almost every known type of fallacy is a close neighbor to sound inferences or acceptable moves in a debate. Nonetheless, the kernel idea of a fallacy as an erroneous type of argument is still retained by most authors. We outline a destructive dilemma we refer to as the Fallacy Fork: on the one hand, if fallacies are construed as demonstrably invalid form of reasoning, then they have very limited applicability in real life (few actual instances). On the other hand, if our definitions of fallacies are sophisticated enough to capture real-life complexities, they can no longer be held up as an effective tool for discriminating good and bad forms of reasoning. As we bring our schematic “fallacies” in touch with reality, we seem to lose grip on normative questions. Even approaches that do not rely on argumentation schemes to identify fallacies (e.g., pragma-dialectics) fail to escape the Fallacy Fork, and run up against their own version of it. We propose a new task in the field of computational argumentation in which we investigate qualitative properties of Web arguments, namely their convincingness. We cast the problem as relation classification, where a pair of arguments having the same stance to the same prompt is judged. We annotate a large datasets of 16k pairs of arguments over 32 topics and investigate whether the relation “A is more convincing than B” exhibits properties of total ordering; these findings are used as global constraints for cleaning the crowdsourced data. We propose two tasks: (1) predicting which argument from an argument pair is more convincing and (2) ranking all arguments to the topic based on their convincingness. We experiment with feature-rich SVM and bidirectional LSTM and obtain 0.76-0.78 accuracy and 0.35-0.40 Spearman’s correlation in a cross-topic evaluation. We release the newly created corpus UKPConvArg1 and the experimental software under open licenses. Software engineering researchers have long been interested in where and why bugs occur in code, and in predicting where they might turn up next. Historical bug-occurence data has been key to this research. Bug tracking systems, and code version histories, record when, how and by whom bugs were fixed; from these sources, datasets that relate file changes to bug fixes can be extracted. These historical datasets can be used to test hypotheses concerning processes of bug introduction, and also to build statistical bug prediction models. Unfortunately, processes and humans are imperfect, and only a fraction of bug fixes are actually labelled in source code version histories, and thus become available for study in the extracted datasets. The question naturally arises, are the bug fixes recorded in these historical datasets a fair representation of the full population of bug fixes? In this paper, we investigate historical data from several software projects, and find strong evidence of systematic bias. We then investigate the potential effects of "unfair, imbalanced" datasets on the performance of prediction techniques. We draw the lesson that bias is a critical problem that threatens both the effectiveness of processes that rely on biased datasets to build prediction models and the generalizability of hypotheses tested on biased data. Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine"). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art language models to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88 ), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research.