The Defects4J dataset [1]} is a very useful collection of programs, bugs, and programmer-written tests. However, no collection can be all encompassing, and this collection is still only a small sample of real programs and their bugs. A technique performing well on these 17 programs might not perform particularly well on any other program (and vice versa). Second, the test cases are provided by the developers and are generally in the form of unit tests. Arguably, end-to-end system tests would be a better choice for evaluating fault localization techniques [2]}. In particular the tests fail at very low rates on the faulty program versions in the dataset. Ideally, there would be more balance between failing and passing tests. Finally, because the tests are generally unit tests they do not simulate operational input by end-users the programs.
