Although a variety of learning-based bug localization and repair models have been developed in recent years, there does not
exist a unique setup for training and benchmarking these models which is universally accepted. Therefore, even though we implemented our baselines close to the reference implementation, the resulting trained models might behave differently than in the setup they were originally developed for.
To still achieve comparable results, we designed our evaluation to replicate prior studies [1]} on neural bug localization and repair models as close as possible. For example, we adopted the same publicly accessible Github corpus ETH Py150k, a similar architectural design and similar baselines as employed by Hellendoorn et al. [1]}. To further support a wider range of bug types
which allowed us to exploit the hand validated benchmark PyPIBugs, we adjusted the architecture and mutation process similar to Allamanis et al. [3]}.
Still, our evaluation results for the baseline algorithms (in Table REF ) are slightly different than the results of prior studies.
For example, we found that while graph-based models such as GNNs and GREAT still perform better in localizing and repairing real bugs, Transformers show a surprisingly strong performance on mutants. Note however that Hellendoorn et al. [1]} anticipated this result (even though they only evaluated on variable misuses) when the models are trained for a longer duration – which we did by training on approximately 2.4x more training examples.
In contrast to the results of Allamanis et al. [3]}, we observe that graph-based models underperform in our evaluation setup which we attribute to two main differences: (1) for a fair comparison, all models only have access to the function implementation without the implementation context which prohibits the computation of type related or call structure related information exploited by the graph-based models and (2) we trained all models on a different (potentially smaller) dataset. Although integrating the type of information and training on a larger dataset would potentially benefit all baselines, the performance ranking between architectures might differ. However, since our experiments showed that the performance gain due to training on real bug fixes is unique and the effect could not be replicated by training on mutants, we
expect that adapting our evaluation setup has little to no influence on our evaluation outcome.
