Reviewer questioned credibility of paper and results after first revision, which led to us finding a major error. Should we revise or withdraw? I and my team members worked on a machine learning paper to predict pass and fail on a standard academic test. The paper was submitted to a reputable journal. The first review came-back with a lot of encouragement but with few suggestions which made us change the methodology of our work especially with respect to the data - preprocessing. The changes were made the revised manuscript was submitted again. However, one thing which should have been debated before submitting was the accuracy of the results which was 100% for multiple evaluation metrics on each of the stratified folds. While the concern was raised within our team we did eventually submit it without rigorously making sure if everything was right or not.
Now the reviewer comments have come back and he has challenged the credibility of our work specifically pointing out that 100% accuracy looks too good to be true. We re-ran the models to evaluate how it behaves and found the results are quite different and not close to 100% percent. We have reviewed all the code and the steps we took to achieve the results but can-not find any flaw or mistake that may have resulted in 100% accuracy.
The current results which hover between 65 to 80% between folds look far more credible. However, I am not sure how to respond to the reviewer especially when we are not able to find our mistake that resulted in the 100% accuracy. We want to be absolutely honest and want to acknowledge that we committed a mistake in submitting the 100% accuracy and we should have been more critical of our work.
While as a group we want to be absolutely transparent with all the data files and results that we have obtained and share it with the reviewer. But I am a little concerned about how would the reviewer perceive such a big difference in results which change the conclusion or at least the interpretation.
Should we submit our responses and revise the manuscript or withdraw the paper?
Any help would be appreciated.
I suggest that you should withdraw your submission until you can figure out what's going on. Good science should be reproducible, and this is not. You did not appropriately critique your own results, and put forth a result that seemed to &quot;look good&quot;. You've now changed some part of your modeling methodology, and gotten another set of results that &quot;look good&quot;, even though you cannot articulate any theoretical or practical reason why this set of results is any more sound than the last. The first result did not pass the &quot;smell test&quot; of reasonable performance, and you kind of shot yourself in the foot by failing to investigate. Now you have a result that does pass the &quot;smell test&quot;, but the reviewer is aware that your diligence in self-criticism is somewhat lacking.
You're essentially telling the reviewer that &quot;the last result was an error, but this one is not, although we are unable to explain any difference between the two.&quot; It's a really big ask for the reviewer to have confidence in your updated methodology, if you yourself cannot explain why it is any more correct than what you did the first time around.