Single token bug localization and repair.
The detection and repair of single token bugs have been explored in previous work [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}. Allamanis et al. [1]} addressed the detection and repair of variable misuse bugs (which we also considered in this work) by representing programs as graphs. Vasic et al. [3]} proposed a joint model for same task and Hellendoorn et al. [4]} explored alternative program representations. These techniques all have in common that they do not learn from real bug fixes but from artificially mutated code. In contrast, while RealiT  employs a similar Transformer-based architecture as discussed by Hellendoorn et al. [4]}, we showed that integrating real bug fixes in the training process is crucial for the localization and repair of real bugs.
More recent work [5]}, [6]}, [7]} also showed that the quality of training data is important for effective bug localization and repair. For example, employing a more realistic mutator [5]}, [6]} (i.e. a mutator that is more likely to reproduce a real bug) or learning to inject hard to find bugs [7]} can both improve the localization and repair performance. However, the integration of these approaches often increases the complexity by requiring to learn a mutation operator either prior and concurrent to the training process. With RealiT, we showed that integrating real bug fixes, while relying on simpler and easier to implement mutation operators, can be sufficient to obtain a significant improvement in real bug localization and repair performance.
Interestingly enough, a concurrent work [18]} also explored whether real bug fixes have an impact on the performance of learning-based bug detectors. Similar to RealiT, their model is pre-trained on mutants and then fine-tuned real bug fixes. Surprisingly, while the authors found that fine-tuning on real bug fixes improves precision (i.e. the number of correct programs classified as buggy), the recall (i.e. the number of real bugs detected and repaired) actually suffers. In contrast, we find that RealiT  improves the number of bugs detected and repaired significantly while training on real bug fixes can also decrease the false positive rate. We attribute the difference in our findings to significant differences to the RealiT  training process: (1) the number of real bug fixes we fine-tune on is several magnitudes larger, (2) the number of mutants generated per code snippet is significantly higher and (3) the distribution of buggy and bug-free programs is balanced both during pre-training and fine-tuning. We believe that especially (3) is key to success of RealiT. Training on an unbalanced dataset (with significant more bug-free than buggy code) risks that the model defaults to not detecting a bug (which would result in a higher precision and lower recall by design).
