Test benchmarks. We employ two test benchmarks to evaluate the
performance on the localization and repair task. To evaluate the localization and repair performance
on real bugs, we employ the PyPIBugs benchmark [1]}. The benchmark is a dataset of 2374 real-world single statement bugs and their fix derived from open source projects. The benchmark is hand-filtered and therefore it is likely that each included bug represents a real world bug. We only considered single token bugs (which excludes argument swaps) in functions where the implementation is still publicly accessibleThe benchmark consists of references to bug fixing commits. We found that not all bug fixing commits were publicly accessible at the time of writing.. This produced a real world test benchmark of 2028 real-world bugs. To avoid an overlap between train and test set, we excluded all near duplicates [2]} from our training datasets. Additionally, we also employ the test portion of the Github corpus as a mutant benchmark. For this, we extend the corpus of correct code snippets with up to 9 mutants per snippet.
<TABLE>