Should I test my algorithm using a standard dataset that I believe is flawed? I am developing an algorithm for a type of classification problem that is a small-but-growing area of AI research. Until two years ago, there was no standard benchmark for this type of problem. In 2016, a benchmark dataset was made publically available to solve this issue and most subsequent papers in this area have used it to evaluate their algorithms.

However, the dataset is of poor quality. It is very small with &lt;100 examples, many of which are slight variations on the same data. Also, the ground truth is dubious for several examples. This makes it hard to compare the performance of different published algorithms since they are nearly all within the margin of error.

I can think of four options:


Publish my algorithm using the flawed benchmark. Its apparent performance is comparable to other methods, and it has enough other advantages (fewer parameters, etc.) to be published this way, but probably not in a high-impact venue.
Publish my algorithm using a more robust benchmark that I create on my own. I could add legitimacy by implementing previously-published algorithms to compare on the new benchmark, but I feel like this would still give the impression of moving the goalposts.
Publish my algorithm using a more robust benchmark that I create on my own, in addition to the flawed benchmark. Many papers test their algorithms on additional custom datasets (though usually simulations) so this is not unusual. However, this would imply I consider the flawed benchmark valid and that would still be the results most people look at.
Separately publish a more robust benchmark dataset and validated criticism of the flawed benchmark. I could then cite this when I later publish my algorithm. However, this will delay publication of my algorithm and still depends on my new benchmark being well-received.


Because I am still a Ph.D. student and this would be my first work in this field, I am wary of bucking current practices too much. What should I do? Are there other factors I should consider?
You leave a lot unsaid here, such as how expensive (time, money) it is to run the algorithm on any given dataset and how much effort to create a new one. 

Three pieces of advice, though. If it is more or less standard to use the published data, then using that should give you some basis for evaluating your algorithm compared to others. With a new dataset you lose that opportunity. 

A new dataset also leaves you open to a charge that the "great" results you got were just an artifact of the dataset you created on your own. You should probably avoid that. At least as the first iteration. 

There is no reason that you couldn't use the existing dataset, gather your results, but also, in the same paper, discuss the shortcomings of that dataset. This also gives you an opening to future work and you might even get some advice on how to avoid the second point above. 

Ok. Four pieces of advice. If you are new at this (and not a doctoral candidate writing a dissertation) you don't need to push the boundaries too far to make a contribution. You can use the simplest case to set up your case 4. for the future. But if you buck current practice very much you need to justify it. 



On the other hand, if you are a doctoral candidate, the significance of your work is normally considered more important than the time you need to put into it and the expectation is that it is novel. But you still need to avoid the charge of tailoring the data to fit the desired results.